Python Multiprocessing

01

Learning Objectives

Multiprocessing is one of the most important techniques for understanding how Python can perform CPU-intensive work using multiple processes.

By the end of this lesson, you will be able to:

Understand what a process is.
Explain multiprocessing.
Distinguish processes from threads.
Identify CPU-bound workloads.
Create processes with multiprocessing.
Use start() and join().
Pass arguments to processes.
Use multiprocessing pools.
Use map() for parallel processing.
Communicate between processes.
Use Queue and Pipe.
Understand shared values and arrays.
Use Manager for shared objects.
Protect shared resources with locks.
Handle multiprocessing exceptions.
Understand the main-process guard.
Choose between multiprocessing, threading, and asyncio.

02

What Is Multiprocessing?

Multiprocessing means running work using multiple independent processes.

Each process has its own Python interpreter and memory space. This makes processes different from threads.

from multiprocessing import Process


def worker():

    print("Worker process is running")


process = Process(
    target=worker
)

process.start()

process.join()

print("Main process finished")

The key idea

Multiple processes can execute work independently and, on a multi-core system, can execute CPU-intensive Python code in parallel.

03

Process vs Thread

Processes and threads both provide ways to perform concurrent work, but their architecture is different.

Process

Has its own memory space.
Has its own Python interpreter.
Communication requires explicit mechanisms.
Higher startup and memory overhead.
Suitable for CPU-bound parallel work.

Thread

Runs inside a process.
Threads share process memory.
Communication can be easier through shared state.
Lower overhead than separate processes.
Often useful for I/O-bound work.

Remember the GIL

In standard CPython builds, the Global Interpreter Lock historically prevents multiple threads from executing Python bytecode simultaneously within the same interpreter. Separate processes avoid sharing that interpreter lock and can therefore provide true multi-core parallelism for CPU-bound Python workloads.

04

CPU-Bound Work

A CPU-bound task spends most of its time performing calculations rather than waiting for external resources.

Examples of CPU-bound work

Large numerical calculations
Image processing
Video processing
Scientific simulations
Complex mathematical algorithms
Large-scale data transformations

These workloads can benefit from multiple CPU cores.

A useful rule is: CPU-bound → consider multiprocessing.

05

The multiprocessing Module

Python's standard library includes the multiprocessing module.

import multiprocessing

It provides tools for creating and coordinating processes.

Process
Pool
Queue
Pipe
Value
Array
Manager
Lock

06

Your First Process

A process can be created using multiprocessing.Process.

from multiprocessing import Process


def worker():

    print("Hello from worker process")


process = Process(
    target=worker
)

process.start()

process.join()

print("Done")

The target function runs in a separate process.

07

start() & join()

Processes use start() to begin execution and join() to wait for completion.

from multiprocessing import Process
import time


def worker():

    print("Worker started")

    time.sleep(2)

    print("Worker finished")


process = Process(
    target=worker
)

process.start()

print("Main process continues")

process.join()

print("Main process finished")

Without join(), the main process may continue before the worker process has completed its task.

08

Creating Multiple Processes

You can create several processes and start them independently.

from multiprocessing import Process


def worker(name):

    print(
        "Process running:",
        name
    )


processes = []


for name in [
    "Process A",
    "Process B",
    "Process C"
]:

    process = Process(
        target=worker,
        args=(name,)
    )

    processes.append(process)

    process.start()


for process in processes:

    process.join()


print("All processes finished")

Each worker runs in its own process.

09

Passing Arguments

Arguments can be passed using args.

from multiprocessing import Process


def calculate(number):

    result = number * number

    print(
        number,
        "squared =",
        result
    )


process = Process(
    target=calculate,
    args=(10,)
)

process.start()

process.join()

The comma in (10,) creates a one-element tuple.

10

Using a Process Class

Similar to threading, you can subclass Process.

from multiprocessing import Process


class Worker(Process):

    def run(self):

        print(
            "Custom process running"
        )


process = Worker()

process.start()

process.join()

The run() method contains the work performed by the process.

For simple tasks, passing a target function is usually easier to understand. A custom Process class can be useful when you need to encapsulate process behavior.

11

Process Pools

Creating a separate process manually for every task is not always convenient.

A process pool manages a group of worker processes.

from multiprocessing import Pool


def square(number):

    return number * number


numbers = [
    1, 2, 3, 4, 5
]


with Pool(
    processes=4
) as pool:

    results = pool.map(
        square,
        numbers
    )


print(results)

Output

[1, 4, 9, 16, 25]

The pool distributes work among its worker processes.

12

Pool.map()

map() is one of the easiest ways to distribute independent tasks across a process pool.

from multiprocessing import Pool


def calculate(value):

    return value ** 2


values = range(1, 11)


with Pool() as pool:

    results = pool.map(
        calculate,
        values
    )


print(results)

Each input value is passed to the function and the results are collected into a list.

Pool.map() is particularly convenient when you have many independent tasks that produce one result each.

13

apply() & apply_async()

apply() executes a function in the pool and waits for its result.

from multiprocessing import Pool


def add(a, b):

    return a + b


with Pool(2) as pool:

    result = pool.apply(
        add,
        (10, 20)
    )

    print(result)

apply_async() returns an object that can later provide the result.

from multiprocessing import Pool


def add(a, b):

    return a + b


with Pool(2) as pool:

    result = pool.apply_async(
        add,
        (10, 20)
    )

    print(result.get())

14

Multiprocessing Queue

Processes do not normally share ordinary Python variables. A multiprocessing queue provides a way for processes to exchange data safely.

from multiprocessing import Process, Queue


def worker(queue):

    queue.put(
        "Data from worker"
    )


queue = Queue()

process = Process(
    target=worker,
    args=(queue,)
)

process.start()

message = queue.get()

process.join()

print(message)

Producer and consumer

One process can produce data while another process consumes it through the queue.

15

Pipes

A Pipe creates a connection between two processes.

from multiprocessing import Process, Pipe


def worker(connection):

    connection.send(
        "Hello from worker"
    )

    connection.close()


parent, child = Pipe()


process = Process(
    target=worker,
    args=(child,)
)

process.start()

message = parent.recv()

process.join()

print(message)

Pipes are useful when two processes need a direct communication channel.

For larger producer-consumer designs, a Queue is often more natural. Pipes are useful for simpler direct communication between processes.

16

Shared Values & Arrays

Processes normally have separate memory. The Value and Array tools provide controlled ways to share simple data.

from multiprocessing import Process, Value


def increment(counter):

    counter.value += 1


counter = Value(
    "i",
    0
)


processes = []


for _ in range(5):

    process = Process(
        target=increment,
        args=(counter,)
    )

    processes.append(process)

    process.start()


for process in processes:

    process.join()


print(counter.value)

Shared data needs synchronization

Multiple processes updating the same shared value can cause race conditions. Use appropriate synchronization mechanisms when updates must be atomic.

17

Using Manager

A multiprocessing.Manager can provide shared Python-like objects such as lists and dictionaries.

from multiprocessing import Process, Manager


def worker(shared_list):

    shared_list.append(
        "Worker completed"
    )


with Manager() as manager:

    results = manager.list()

    process = Process(
        target=worker,
        args=(results,)
    )

    process.start()

    process.join()

    print(list(results))

Manager objects are convenient but involve communication overhead, so they should not be treated as a free replacement for ordinary local variables.

18

Process Locks

When processes access shared resources, synchronization may be necessary.

from multiprocessing import Process, Value, Lock


def increment(counter, lock):

    for _ in range(1000):

        with lock:

            counter.value += 1


counter = Value(
    "i",
    0
)

lock = Lock()


processes = []


for _ in range(4):

    process = Process(
        target=increment,
        args=(counter, lock)
    )

    processes.append(process)

    process.start()


for process in processes:

    process.join()


print(counter.value)

The lock ensures that only one process at a time enters the protected section.

Keep critical sections as small as practical. Holding a lock unnecessarily can reduce parallelism.

19

Exceptions in Multiprocessing

Exceptions occurring inside child processes do not automatically behave like exceptions raised in the main process.

One practical approach is to catch exceptions inside the worker and communicate the result back.

from multiprocessing import Process, Queue


def worker(number, queue):

    try:

        result = 100 / number

        queue.put({
            "success": True,
            "result": result
        })

    except ZeroDivisionError:

        queue.put({
            "success": False,
            "error": "Cannot divide by zero"
        })


queue = Queue()


process = Process(
    target=worker,
    args=(0, queue)
)

process.start()

result = queue.get()

process.join()

print(result)

For pool-based work, methods such as get() on asynchronous result objects can propagate worker exceptions back to the caller.

20

if name == "main"

The main-process guard is especially important when using multiprocessing.

from multiprocessing import Process


def worker():

    print("Worker process")


if __name__ == "__main__":

    process = Process(
        target=worker
    )

    process.start()

    process.join()

Why is this important?

On platforms that use the spawn start method, a child process starts a fresh Python interpreter. Without protecting process creation with the main guard, your program can repeatedly execute process-creation code.

Therefore, make it a habit to put process-starting code inside:

if __name__ == "__main__":

21

Real-World Applications

Multiprocessing becomes particularly valuable when computational work can be divided into independent tasks.

Image Processing

Multiple images can be resized, transformed, or analyzed independently.

Data Processing

Large datasets can sometimes be divided into independent chunks for computational processing.

Scientific Computing

Independent simulations or calculations can be distributed across CPU cores.

Video Processing

Computationally intensive transformations may be parallelized where the algorithm allows it.

Large Data Workflows

Independent CPU-heavy transformations can sometimes be distributed among worker processes.

22

Threading vs Multiprocessing vs Async

One of the most important skills at this stage of the course is choosing the right concurrency model.

Threading

Uses multiple threads inside a process.
Threads share memory.
Useful for many I/O-bound tasks.
Works well with blocking synchronous libraries.
Shared state requires synchronization.

Multiprocessing

Uses separate processes.
Processes have independent memory.
Useful for CPU-bound workloads.
Can use multiple CPU cores.
Inter-process communication has overhead.

Asyncio

Uses an event-loop concurrency model.
Uses coroutines and awaitable tasks.
Excellent for large numbers of I/O operations.
Works best with async-compatible libraries.
Does not automatically make CPU-heavy code faster.

Quick decision guide

CPU-bound → Multiprocessing

Blocking I/O → Threading

Many async I/O operations → Asyncio

These are useful guidelines, not absolute rules. Library support, architecture, workload size, and deployment environment also matter.

23

Common Multiprocessing Mistakes

Mistake 1 — Forgetting the main guard

Always protect process creation with if __name__ == "__main__": when appropriate, especially for portable code.

Mistake 2 — Expecting normal variables to be shared

A child process does not normally modify the main process's ordinary Python variables.

Mistake 3 — Using multiprocessing for tiny tasks

Creating and coordinating processes has overhead. A task should be substantial enough to justify the cost.

Mistake 4 — Sharing too much state

Heavy communication between processes can eliminate the performance benefits of parallel execution.

Mistake 5 — Ignoring synchronization

Shared resources still require synchronization to avoid race conditions.

Mistake 6 — Assuming more processes always means more speed

Performance depends on CPU cores, workload size, process overhead, memory, communication, and the structure of the algorithm.

24

Practice Exercises

Exercise 1 — First Process

Create a process that prints a welcome message.

Exercise 2 — Multiple Processes

Create four processes that each print a different number.

Exercise 3 — Process Arguments

Pass a student's name and marks to a worker process and print the student's result.

Exercise 4 — Pool.map()

Use a process pool to calculate the squares of numbers from 1 to 20.

Exercise 5 — Queue

Create one producer process and one consumer process that communicate using a multiprocessing queue.

Exercise 6 — Shared Counter

Create several processes that safely update a shared counter using Value and Lock.

Exercise 7 — Choose the Right Tool

Decide whether threading, multiprocessing, or asyncio would be most appropriate for downloading files, calculating large numerical results, and handling many network requests.

25

Mini Challenge — Parallel Sales Analyzer

Build a multiprocessing application that analyzes several independent sales datasets.

Your Tasks

Create several lists containing sales values.
Write a function called analyze_sales().
Calculate total sales for each dataset.
Calculate the average.
Find the highest sale.
Use Pool.map() to process the datasets in parallel.
Return a dictionary containing the analysis.
Print the final results.

Starter Code

from multiprocessing import Pool


sales_data = [

    [1200, 1500, 900, 2100, 1800],

    [800, 950, 1200, 1100, 1400],

    [2200, 1900, 2500, 2100, 2300],

    [600, 750, 900, 850, 1000]

]


def analyze_sales(sales):

    total = sum(sales)

    average = total / len(sales)

    highest = max(sales)

    return {
        "total": total,
        "average": average,
        "highest": highest
    }


if __name__ == "__main__":

    with Pool() as pool:

        results = pool.map(
            analyze_sales,
            sales_data
        )


    for index, result in enumerate(
        results,
        start=1
    ):

        print(
            f"Dataset {index}:",
            result
        )

Challenge extension

Once the basic version works, modify the program to process a much larger number of datasets. Then compare the design with a normal sequential loop and think about when the multiprocessing overhead becomes worthwhile.

26

Knowledge Check

Test your understanding of Python multiprocessing.

1. What is multiprocessing?

2. Which module provides Process?

3. Which method starts a process?

4. What does join() do?

5. Which type of workload is commonly suited to multiprocessing?

6. What does Pool.map() do?

7. Why can't ordinary Python variables normally be shared between processes?

8. Which tool can provide process-safe communication through a work queue?

9. Why is the main guard important?

10. Which is generally the best starting choice for CPU-bound parallel Python work?

27

Course Summary

Congratulations! You have reached the final lesson of the PyLearn Python course.

Across 73 lessons, you progressed from Python fundamentals to advanced programming concepts, data analysis, APIs, web development, concurrency, and parallel computing.

What you have learned

Python fundamentals and syntax
Variables and data types
Type conversion
Input and output
Operators
Strings and string processing
Lists, tuples, sets, and dictionaries
Comprehensions
Conditional statements and loops
Functions and advanced function techniques
Modules and packages
File handling and structured data
Error handling and exceptions
Object-oriented programming
Magic methods
Iterators and generators
Decorators
Regular expressions
Type hints and dataclasses
pip and virtual environments
Debugging and testing
SQLite databases
HTTP requests and REST APIs
Web scraping
NumPy and Pandas
Data cleaning and visualization
Data analysis
Flask and FastAPI
Async Python
Threading
Multiprocessing

You have learned Python. Now build with it.

Completing a course is only the beginning. The next stage is applying your knowledge to real projects.

Build applications, analyze datasets, create APIs, automate repetitive work, develop web applications, work with databases, and solve real business problems.

Recommended next steps

Revisit lessons where your understanding is weak.
Build small Python projects without copying complete solutions.
Practice working with real datasets.
Build at least one API project.
Build at least one database-backed application.
Create a portfolio of Python projects.
Continue learning specialized Python technologies based on your career goals.

Final principle

Do not measure your Python skills by how many tutorials you have completed. Measure them by how confidently you can take a problem, design a solution, write the code, test it, debug it, and improve it.