How to use Python multiprocessing module

· Category: Python Programming

Short answer

The multiprocessing module spawns separate processes to bypass the GIL and use multiple CPU cores. Use Process, Pool, or Queue to distribute work. For understanding why multiprocessing is needed, see what is the python global interpreter lock. For profiling gains, see how to profile python code for performance.

Steps

  1. Import Pool: from multiprocessing import Pool
  2. Define a worker function
  3. Create a pool: with Pool(processes=4) as pool:
  4. Map work: results = pool.map(worker, data)
  5. Handle results and exceptions

Tips

  • Process startup overhead is high; multiprocessing is best for CPU-bound tasks
  • Use if __name__ == '__main__': guard to avoid recursive spawning on Windows
  • For simpler concurrency, see python asyncio