Python – ipyparallel is attached to a parallel for loop to a list

I have a loop that performs a lot of CPU-intensive calculations and appends the results to the list, each iteration.

How can I get it to work in parallel. There is a concurrent container on C#, how does it work in ipyparallel?

From the ipyparallel documentation:

Python’s builtin map() functions allows a function to be applied to a sequence element-by-element. This type of code is typically trivial to parallelize.

http://ipyparallel.readthedocs.io/en/latest/direct.html#parallel-map

So it’s a matter of using the map function to get it running in parallel, but how can I append the result to the list? Is there a concurrent container here?

So, what I have now is something like this:

results = []
for element in list:
    outcome = very_heavy_computation_function(element)
    results.append(outcome)

How can I do this in parallel?

You can do this in the following ways.

Function foo represents the computation in the loop, and parameters represent the data, if any, that you want to iterate over. Although foo slept for 10 seconds, the whole loop took only 10 seconds instead of 40 seconds because I have 4 engines in my environment and the functions run in parallel on the engines. LoadBalancedView provides dynamic load balancing to distribute work evenly among engines.

from ipyparallel import Client

rc = Client()
view = rc.load_balanced_view()

def foo(param):
    import time
    time.sleep(10)
    return param*2

parameters = list(range(4))
async_results = []
for p in parameters:
    async_result = view.apply_async(foo, p)
    async_results.append(async_result)

rc.wait_interactive(async_results)

results = [ar.get() for ar in async_results]
print(results)

Output:

   4/4 tasks finished after   10 s
done
[0, 2, 4, 6]