Can I have more workers dedicated to an specific task in Python? Parallelization of 2 tasks with different time consuming

  multiprocessing, python

I have to download a list of files from a slow ftp server, each downloading lasts around 40s.

For each of this files I have to process to tasks:
‘A’ lasts around 30-40s
‘a’ lasts less than 10s

Each time a download is finished I’m launching processes ‘A’ and ‘a’ in parallel using multiprocessing library but, because sometimes process ‘A’ lasts the same time as the downloading, I can’t assure that I will finish the job (‘A’+’a’) before the next file of the list comes.

I was wondering if there’s any way that I could give more workers to task ‘A’ to speed it up so the complete job can be done between downloads.
I’m using a Raspi 4 with 4 cores so I was thinking 1 core for downloading, 1 core for task ‘a’ and 2 cores for task ‘A’.

Is this possible? Any ideas are welcome

Source: Python Questions

LEAVE A COMMENT