Building a Thread Pool from Scratch

Why Thread Pools?

Spawning a new std::thread for every task is expensive. Each thread carries its own stack (typically 1–8 MB), and the OS must schedule, context-switch, and eventually tear it down. If your program fires thousands of short-lived tasks, the overhead of thread creation can dwarf the useful work.

A thread pool solves this by pre-creating a fixed set of worker threads that pull tasks from a shared queue. The workers stay alive for the lifetime of the pool, so task submission becomes nothing more than enqueueing a callable and waking a waiting thread.

Architecture Overview

Our pool has three core components:

1. Task queue — a std::queue<std::function<void()>> guarded by a mutex and a condition variable.
2. Worker threads — a vector of std::thread objects, each running an infinite loop that waits on the condition variable, dequeues a task, and executes it.
3. Submit interface — a templated submit() that wraps any callable in a std::packaged_task, pushes it onto the queue, and returns a std::future.

Click Step to walk through the dispatch sequence: tasks leave the queue, workers pick them up, and futures become ready.

The Task Queue

The queue is the synchronization backbone. We wrap a plain std::queue with a mutex and a condition variable so that workers can sleep when the queue is empty and wake up the moment a new task arrives.

// thread_safe_queue.h
#include <queue>
#include <mutex>
#include <condition_variable>
#include <optional>

template <typename T>
class ThreadSafeQueue {
public:
    void push(T value) {
        {
            std::lock_guard<std::mutex> lock(mtx_);
            queue_.push(std::move(value));
        }
        cv_.notify_one();
    }

    // Blocking pop: waits until an item is available or stop is requested.
    // Returns std::nullopt when the queue is stopped and empty.
    std::optional<T> pop(std::atomic<bool> const& stop_flag) {
        std::unique_lock<std::mutex> lock(mtx_);
        cv_.wait(lock, [&] { return !queue_.empty() || stop_flag.load(); });
        if (queue_.empty()) return std::nullopt;
        T value = std::move(queue_.front());
        queue_.pop();
        return value;
    }

    void notify_all() { cv_.notify_all(); }

private:
    std::queue<T>  queue_;
    std::mutex      mtx_;
    std::condition_variable cv_;
};

Key design decisions:

• The push acquires the lock, moves the item in, then notifies one waiting thread. Notifying outside the lock is an option but complicates the logic for marginal gain.
• pop takes a reference to the pool’s stop flag so it can break out of the wait when shutdown is requested.
• Returning std::optional lets the caller distinguish “got a task” from “queue is stopped” without exceptions.

The Worker Loop

Each worker runs a simple loop: pop a task, run it, repeat. The loop terminates when the queue signals that no more tasks will arrive.

void worker_loop(ThreadSafeQueue<std::function<void()>>& tasks,
                  std::atomic<bool> const& stop) {
    while (true) {
        auto task = tasks.pop(stop);
        if (!task.has_value()) return;   // queue drained and stop requested
        (*task)();                        // execute the task
    }
}

There is no busy-waiting. When the queue is empty the thread sleeps on the condition variable, consuming zero CPU until a new task is submitted or the pool shuts down.

submit() and Futures

The submit() function is the public interface for enqueuing work. It must accept any callable and return a std::future so callers can retrieve the result (or catch exceptions) once the task completes.

template <typename F, typename... Args>
auto submit(F&& f, Args&&... args)
    -> std::future<std::invoke_result_t<F, Args...>>
{
    using ReturnType = std::invoke_result_t<F, Args...>;

    auto task = std::make_shared<std::packaged_task<ReturnType()>>(
        std::bind(std::forward<F>(f), std::forward<Args>(args)...)
    );

    std::future<ReturnType> result = task->get_future();

    tasks_.push([task]() { (*task)(); });

    return result;
}

The trick: std::packaged_task is move-only, but std::function<void()> requires copyability. We work around this by placing the packaged task inside a std::shared_ptr and capturing that pointer in a lambda. The shared pointer keeps the task alive until the worker executes and destroys the lambda.

Complete Thread Pool Implementation

Below is the full, compilable thread pool class. Copy it into a single header, compile with g++ -std=c++17 -pthread, and it works out of the box.

// thread_pool.h
#ifndef THREAD_POOL_H
#define THREAD_POOL_H

#include <vector>
#include <queue>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <functional>
#include <future>
#include <atomic>
#include <stdexcept>
#include <type_traits>
#include <memory>

class ThreadPool {
public:
    // Create a pool with the given number of worker threads.
    // Defaults to the hardware concurrency.
    explicit ThreadPool(std::size_t num_threads
                        = std::thread::hardware_concurrency())
        : stop_(false)
    {
        if (num_threads == 0) num_threads = 1;

        workers_.reserve(num_threads);
        for (std::size_t i = 0; i < num_threads; ++i) {
            workers_.emplace_back([this] { worker_loop(); });
        }
    }

    // Non-copyable, non-movable.
    ThreadPool(const ThreadPool&)            = delete;
    ThreadPool& operator=(const ThreadPool&) = delete;

    // Submit a callable and return a future to its result.
    template <typename F, typename... Args>
    auto submit(F&& f, Args&&... args)
        -> std::future<std::invoke_result_t<F, Args...>>
    {
        using ReturnType = std::invoke_result_t<F, Args...>;

        auto task = std::make_shared<std::packaged_task<ReturnType()>>(
            std::bind(std::forward<F>(f), std::forward<Args>(args)...)
        );

        std::future<ReturnType> result = task->get_future();

        {
            std::lock_guard<std::mutex> lock(queue_mtx_);
            if (stop_.load()) {
                throw std::runtime_error(
                    "submit() called on a stopped ThreadPool");
            }
            tasks_.push([task]() { (*task)(); });
        }
        queue_cv_.notify_one();

        return result;
    }

    // Signal all threads to finish and wait for them to join.
    // Pending tasks in the queue are executed before threads exit.
    void shutdown() {
        {
            std::lock_guard<std::mutex> lock(queue_mtx_);
            stop_.store(true);
        }
        queue_cv_.notify_all();

        for (auto& t : workers_) {
            if (t.joinable()) t.join();
        }
    }

    // The destructor calls shutdown() if the user has not done so.
    ~ThreadPool() {
        if (!stop_.load()) {
            shutdown();
        }
    }

private:
    void worker_loop() {
        while (true) {
            std::function<void()> task;
            {
                std::unique_lock<std::mutex> lock(queue_mtx_);
                queue_cv_.wait(lock, [this] {
                    return !tasks_.empty() || stop_.load();
                });

                if (stop_.load() && tasks_.empty()) return;

                task = std::move(tasks_.front());
                tasks_.pop();
            }
            task();  // execute outside the lock
        }
    }

    std::vector<std::thread>             workers_;
    std::queue<std::function<void()>>    tasks_;
    std::mutex                            queue_mtx_;
    std::condition_variable               queue_cv_;
    std::atomic<bool>                    stop_;
};

Usage Example

#include "thread_pool.h"
#include <iostream>
#include <numeric>
#include <vector>

int main() {
    ThreadPool pool(4);

    // Submit a mix of tasks.
    auto f1 = pool.submit([] {
        return 6 * 7;
    });

    auto f2 = pool.submit([](int a, int b) {
        return a + b;
    }, 100, 200);

    // Submit many tasks and collect futures.
    std::vector<std::future<int>> futures;
    for (int i = 0; i < 20; ++i) {
        futures.push_back(pool.submit([i] {
            return i * i;
        }));
    }

    std::cout << "6 * 7 = " << f1.get() << "\n";
    std::cout << "100 + 200 = " << f2.get() << "\n";

    int sum = 0;
    for (auto& f : futures) {
        sum += f.get();
    }
    std::cout << "Sum of squares 0..19 = " << sum << "\n";

    pool.shutdown();
    return 0;
}

Compile and run:

g++ -std=c++17 -pthread -O2 -o pool_demo main.cpp
./pool_demo

Expected output:

6 * 7 = 42
100 + 200 = 300
Sum of squares 0..19 = 2470

Join and Stop Semantics

The pool supports two shutdown models, both via shutdown():

1. Graceful drain. Workers keep running until the queue is empty. This is the default behavior: the condition stop_ && tasks_.empty() must both be true before a worker exits.
2. Immediate stop. If you want workers to abandon remaining tasks, change the exit condition to just stop_. Pending tasks are discarded.

// Variant: immediate stop (discard pending tasks)
void worker_loop_immediate() {
    while (true) {
        std::function<void()> task;
        {
            std::unique_lock<std::mutex> lock(queue_mtx_);
            queue_cv_.wait(lock, [this] {
                return !tasks_.empty() || stop_.load();
            });

            if (stop_.load()) return;  // exit immediately

            task = std::move(tasks_.front());
            tasks_.pop();
        }
        task();
    }
}

Performance Considerations

Contention on the Mutex

With a single queue protected by one mutex, all workers contend on the same lock. For low-to-moderate task rates this is perfectly fine. Under extreme submission rates (millions of tiny tasks per second), contention becomes the bottleneck. Solutions include:

• Per-thread local queues with work stealing (covered in the next article).
• Lock-free queues (e.g., MPMC ring buffers).
• Batching tasks before submitting them.

False Sharing

The stop_ flag and the mutex sit in the same cache line, which causes invalidation traffic even when stop_ is not modified. On hot paths, you can pad them to separate cache lines:

alignas(64) std::atomic<bool> stop_;
alignas(64) std::mutex        queue_mtx_;

How Many Threads?

For CPU-bound work, std::thread::hardware_concurrency() is a sensible default. For I/O-bound work (network calls, disk reads), you may want more threads than cores because most of them will be sleeping on I/O. Profile to find the sweet spot.

Common Pitfalls

Deadlock via Dependent Tasks

Submitting a task that blocks on the future of another task submitted to the same pool can deadlock if all workers are occupied:

// DANGER: potential deadlock with a 1-thread pool
auto outer = pool.submit([&pool] {
    auto inner = pool.submit([] { return 42; });
    return inner.get();  // blocks this worker
});

The outer task occupies the only worker and waits for the inner task, which can never run. Avoid blocking on futures inside pool tasks, or use a pool that can dynamically grow.

Exception Propagation

If a task throws, the exception is captured by std::packaged_task and re-thrown when you call future::get(). The worker thread is not affected and continues to the next task.

auto f = pool.submit([] {
    throw std::runtime_error("oops");
    return 0;
});

try {
    f.get();
} catch (const std::exception& e) {
    std::cerr << "Caught: " << e.what() << "\n";
}

Forgetting to Join

If the pool object goes out of scope without shutdown(), the destructor handles it. But relying on the destructor in a code path that might throw can delay shutdown unexpectedly. Call shutdown() explicitly when you know you are done.

Summary

• A thread pool reuses a fixed set of threads to amortize creation cost.
• The core is a mutex + condition variable guarding a task queue.
• submit() wraps callables in std::packaged_task and returns a future.
• Workers run a simple pop-execute loop, sleeping when idle.
• The destructor guarantees all threads are joined.
• For higher throughput under heavy contention, consider work-stealing designs.