Im trying to get data from an sftp server. files are zipped files, around ~50 mb.
When im saving the file locally using:
It takes around ~5 minutes, which is pretty long for such a size of a file.
The main issue here, is if im trying to read the file data into memory using:
file_obj = session.open(file_path) file_data = file_obj.read()
It takes forever. waited for 20 minutes and it didnt even finish.
I did the same test with a ~1mb file size. reading into local file took about 5 seconds. reading the data into memory took about 55 seconds.
Am i doing anything wrong?
Why getting files with paramiko takes so long?
Dont know if it has anything to do with it, but files are zipped. once locally i unzip them, they are around ~10 times bigger. (which was around the same factor between those two reading options)
Source: Python Questions