Python Multiprocessing Tutorial
Last Updated: 1st December, 2018
Previously we’ve looked at how you can try to achieve concurrency with Python using multithreading, the tutorial of which can be found here: Python Multithreading Tutorial
Now we are going to be having a look at how we can sidestep the limiting Global Interpreter Lock that effectively prevented our multithreaded applications from being truly concurrent by using the multiprocessing Python module.
Parallelism vs Concurrency
So, before we go deeper into the multiprocessing module, it’s worthwhile ensuring you know the advantages of using multiprocessing over multithreading. The general rule of thumb is that, if you are trying to improve the performance of CPU-bound tasks, multiprocessing is what you want to use. However, if your particular task is Input/Output bound, then you’ll generally want to use multithreading to improve the performance of your applications.
A Simple Example:
Let’s start by building a really simple Python program that utilizes the multiprocessing module.
In this example, I’ll be showing you how to spawn multiple processes at once and each process will output the random number that they will compute using the random module.
from multiprocessing import Process, Queue import random def rand_num(): num = random.random() print(num) if __name__ == "__main__": queue = Queue() processes = [Process(target=rand_num, args=()) for x in range(4)] for p in processes: p.start() for p in processes: p.join()
Running this should then print out an array of 4 different decimal numbers between 0 and 1 like so:
Elliots-MacBook-Pro:Networking elliotforbes$ python processing.py 0.090906474002 0.306163229303 0.995446921388 0.0320995066016
Now, the important thing to note here, is that each of these random numbers was generated in an entirely separate Python process created with the help of the multiprocessing module.
Each of these separate processes features it’s own instance of the Global Interpreter Lock, and each of these can be run across multiple CPU cores. Now, let’s imagine we were doing something more CPU-intensive than simply generating a single random number. This is where the multiprocessing module would truly start to shine.
In the previous example, we looked at how we could spin up individual processes,
this might be good for a run-and-done type of application, but when it comes to
longer running applications, it is better to create a
pool of longer running
processes. This avoids having to create and destroy a process every time you
have a new task to execute and can subsequently improve the performance of your
import multiprocessing as mp def my_func(x): print(x**x) def main(): pool = mp.Pool(mp.cpu_count()) result = pool.map(my_func, [4,2,3]) if __name__ == "__main__": main()
Now, if we were to execute this, we’d see our
my_func being executed with the
[4,2,3] being mapped as the input to each of these function calls.
python3.6 main.py 256 4 27
Let’s expand this just a little bit to showcase some key concepts here:
import multiprocessing as mp def my_func(x): print(mp.current_process()) return x**x def main(): pool = mp.Pool(mp.cpu_count()) result = pool.map(my_func, [4,2,3,5,3,2,1,2]) result_set_2 = pool.map(my_func, [4,6,5,4,6,3,23,4,6]) print(result) print(result_set_2) if __name__ == "__main__": main()
So, we’ve added a second print statement to the
my_func function. This is to
highlight what process within our pool will be executing a particular task.
We’ve also changed from a
print(x**x) to a
return x**x so that we can view
the results of our process pool.
main() function, we’ve added a few more values to the array in a
map so that we can truly test the extent of our process pool’s workers. We’re
also printing out the results which should now be populated thanks to our newly
added returns in our
When we execute this, we should see the following output:
$ python3.6 main.py 256 <ForkProcess(ForkPoolWorker-1, started daemon)> 4 <ForkProcess(ForkPoolWorker-2, started daemon)> 27 ... 20880467999847912034355032910567 <ForkProcess(ForkPoolWorker-6, started daemon)> <ForkProcess(ForkPoolWorker-7, started daemon)> 256 46656 <ForkProcess(ForkPoolWorker-2, started daemon)> <ForkProcess(ForkPoolWorker-8, started daemon)> [256, 4, 27, 3125, 27, 4, 1, 4] [256, 46656, 3125, 256, 46656, 27, 20880467999847912034355032910567, 256, 46656]
As you can see, our each job is executed by one of the
8 workers that exist
within the pool. These processes are reused in order to prevent the costly task
of destroying and creating new processes and subsequent execution.
Passing Data Between Processes
When performing concurrent programming the best practice is usually to avoid sharing state as much as possible. However when you absolutely need to have some form of shared data then the multiprocessing module provides a couple of ways of doing so.
Queue objects are a FIFO data structure that are thread and process safe which make them perfect for passing data between different processes without potentially corrupting data. Using them is relatively simple, we can expand:
from multiprocessing import Process, Queue import random def rand_num(queue): num = random.random() queue.put(num) if __name__ == "__main__": queue = Queue() processes = [Process(target=rand_num, args=(queue,)) for x in range(4)] for p in processes: p.start() for p in processes: p.join() results = [queue.get() for p in processes] print(results)
This should output the following:
Elliots-MacBook-Pro:Networking elliotforbes$ python processing.py [0.6756465745753756, 0.4104274331895341, 0.6733748532075048, 0.9494118991646461]
Overall Python’s MultiProcessing module is brilliant for those of you wishing to sidestep the limitations of the Global Interpreter Lock that hampers the performance of the multi-threading in python.
I hope this has been helpful, if you feel anything else needs added to this tutorial then let me know in the comments section below! Also, feel free to follow me on twitter for up-to-date announcements and all the latest programming articles: @Elliot_f.