Category : parallel-processing

I have a pandas dataframe and want to apply three different functions on one of the columns in parallel. For example: df = pd.DataFrame( {‘col1’: [‘LA’,’Boston’,’Phoenix’,’Toronto’], ‘col2’: [2,3,4,5]}, columns=[‘col1’, ‘col2′]) def function1(x): return(x**2) def function2(x): return(x**3) def function(x): return(x**4) I want to apply the three functions to column ‘col2’ in parallel and the results to ..

Read more

Hi I am trying to call a python function with reticulate in a parallel manner using foreach like so: library(reticulate) library(doParallel) library(foreach) library(parallel) py_install("wandb") wandb <- import("wandb") cl <- makeCluster(detectCores(), type = ‘PSOCK’) registerDoParallel(cl) foreach(i = 1:5) %dopar% { wandb$init(project = "test") } gives: Error in {: task 1 failed – "attempt to apply non-function" ..

Read more

I am trying to solve one problem related to the python3 multiprocessing package. I want to share complex objects (i.e., Z3 SMT solver instance) between more processes. Currently, I tried multiprocessing.Queue, multiprocessing.Pipe, multiprocessing.Manager and so on. I always end up on the same problem which is: ValueError: ctypes objects containing pointers cannot be pickled. This ..

Read more

I wrote a Python program which I want to parallelize using multiprocessing.Pool when calling the program (MyProgram.__call__()). The expected output is a list of dictionaries (dicts) with the same length as the input list images. However, when I test it with input with length 60 using multiprocessing.Pool of 20 cpus, I got an output with ..

Read more