Processing NLP (coreferencing) concurrently

I understand that, generally speaking, within Python’s concurrent.futures module, the ProcessPoolExecutor is used for CPU-constrained tasks, and the ThreadPoolExecutor is used for I/O-constrained outputs.

I have a dataframe containing news articles that I’d like to apply coreference resolution with the huggingface neuralcoref package. As this is effectively a calculation, I expected this to be a CPU-constrained process and thus was trying to use the ProcessPoolExecutor. However, I’m finding that the ThreadPoolExecutor method processes the batch in half the time. It seems my assumption that this was a CPU-constrained process is incorrect. Why would this be the case?

In case this is helpful, here are the specs:

Processor: AMD FX-8350 (8-core)
Motherboard: Gigabyte 990FXA-UD3
Memory: 16GB DDR3
Storage: Samsung SSD

Source: Python Questions

LEAVE A COMMENT