I have to download a list of files from a slow ftp server, each downloading lasts around 40s.
For each of this files I have to process to tasks:
‘A’ lasts around 30-40s
‘a’ lasts less than 10s
Each time a download is finished I’m launching processes ‘A’ and ‘a’ in parallel using multiprocessing library but, because sometimes process ‘A’ lasts the same time as the downloading, I can’t assure that I will finish the job (‘A’+’a’) before the next file of the list comes.
I was wondering if there’s any way that I could give more workers to task ‘A’ to speed it up so the complete job can be done between downloads.
I’m using a Raspi 4 with 4 cores so I was thinking 1 core for downloading, 1 core for task ‘a’ and 2 cores for task ‘A’.
Is this possible? Any ideas are welcome
Source: Python Questions