Thread Pools and Async I/O Integration

The Problem with Blocking I/O in a Thread Pool

A thread pool works by keeping a fixed number of worker threads busy executing tasks. If a task performs blocking I/O (reading from a socket, waiting on a file descriptor), that worker sits idle for the duration of the I/O operation. The thread is unavailable for other tasks even though it is doing no useful work.

With enough blocking tasks, every worker in the pool can stall simultaneously, leaving CPU-bound tasks queued behind them. Throughput drops to zero until some I/O completes and frees a thread.

Reactor vs. Proactor Patterns

The two fundamental patterns for event-driven I/O are the reactor and the proactor.

Reactor (Readiness-Based)

A reactor notifies you when an I/O operation can proceed without blocking. You then perform the operation yourself.

1. Register interest in a file descriptor (e.g., “tell me when socket 5 is readable”).
2. The event loop calls epoll_wait() (Linux) or kqueue() (BSD/macOS).
3. When ready, the loop dispatches a handler that calls read() on the now-ready fd.

// Simplified reactor sketch (Linux epoll)
int epfd = epoll_create1(0);

struct epoll_event ev;
ev.events = EPOLLIN;
ev.data.fd = client_socket;
epoll_ctl(epfd, EPOLL_CTL_ADD, client_socket, &ev);

struct epoll_event events[64];
while (running) {
    int n = epoll_wait(epfd, events, 64, -1);
    for (int i = 0; i < n; ++i) {
        int fd = events[i].data.fd;
        // fd is now readable; dispatch to handler
        handle_readable(fd);
    }
}

Proactor (Completion-Based)

A proactor initiates the I/O operation and notifies you when it completes. The OS performs the read/write in the background.

1. Initiate an async read: “read 4096 bytes from socket 5 into this buffer.”
2. The OS performs the read in kernel space.
3. When complete, a completion handler is invoked with the result.

Windows I/O Completion Ports (IOCP) is the classic proactor implementation. Linux has io_uring (since kernel 5.1), which provides completion-based async I/O with a shared submission/completion ring buffer.

// Simplified IOCP sketch (Windows)
HANDLE iocp = CreateIoCompletionPort(
    INVALID_HANDLE_VALUE, NULL, 0, num_threads);

// Associate a socket with the IOCP
CreateIoCompletionPort(
    (HANDLE)client_socket, iocp, (ULONG_PTR)ctx, 0);

// Initiate async read
WSABUF buf = { buffer_size, buffer };
DWORD flags = 0;
WSARecv(client_socket, &buf, 1, NULL,
        &flags, &overlapped, NULL);

// Worker threads dequeue completions
while (running) {
    DWORD bytes;
    ULONG_PTR key;
    OVERLAPPED* ov;
    GetQueuedCompletionStatus(
        iocp, &bytes, &key, &ov, INFINITE);

    auto* ctx = reinterpret_cast<Context*>(key);
    ctx->on_read_complete(bytes);
}

Comparison

Task-Based Architectures

The ideal design separates what to do from how to schedule it. Tasks are units of work that do not care whether they run on a thread pool thread, an event loop thread, or an inline executor. This is the core idea behind task-based parallelism.

The Event Loop + Thread Pool Pattern

// Pseudo-architecture
class Server {
    EventLoop event_loop_;     // single thread, epoll/IOCP
    ThreadPool compute_pool_;  // N threads for CPU work

    void on_data_ready(int fd) {
        auto data = read_nonblocking(fd);

        // Offload CPU work to the pool
        compute_pool_.submit([this, fd,
                              data = std::move(data)] {
            auto result = process(data);

            // Schedule the response back on the event loop
            event_loop_.post([this, fd,
                              result = std::move(result)] {
                write_nonblocking(fd, result);
            });
        });
    }
};

The event loop handles I/O readiness. CPU-intensive processing happens in the pool. Responses are posted back to the event loop for non-blocking writes. No thread ever blocks on I/O.

Continuation-Passing Style

Rather than blocking on a future, tasks chain continuations:

// Conceptual (not valid C++ yet without a library)
async_read(socket, buffer)
    .then([](std::size_t bytes_read) {
        return parse_request(buffer, bytes_read);
    })
    .then([](Request req) {
        return handle_request(req);
    })
    .then([&socket](Response resp) {
        return async_write(socket, resp.data());
    });

Each .then() schedules the next step as a task, freeing the current thread immediately. This is the programming model that Boost.Asio and std::execution formalize.

Boost.Asio: The De Facto Standard

Boost.Asio (also available as standalone Asio) is the most widely used C++ async I/O library. It provides:

• An io_context that acts as both an event loop and a task scheduler.
• Async operations (async_read, async_write, async_connect) that register with the OS and invoke completion handlers.
• Built-in thread pool support: multiple threads can call io_context::run() concurrently.

#include <asio.hpp>
#include <iostream>

int main() {
    asio::io_context io(4); // internal hint for concurrency

    // Create a thread pool by running io_context on N threads
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; ++i) {
        threads.emplace_back([&io] { io.run(); });
    }

    // Post work
    for (int i = 0; i < 10; ++i) {
        asio::post(io, [i] {
            std::cout << "Task " << i << " on thread "
                      << std::this_thread::get_id() << "\n";
        });
    }

    // Asio also provides asio::thread_pool directly:
    // asio::thread_pool pool(4);
    // asio::post(pool, []{ ... });
    // pool.join();

    for (auto& t : threads) t.join();
    return 0;
}

Asio with Coroutines (C++20)

With C++20 coroutines, Asio enables a synchronous-looking coding style without blocking threads:

asio::awaitable<void> handle_client(
        asio::ip::tcp::socket socket) {
    char buf[1024];

    while (true) {
        std::size_t n = co_await socket.async_read_some(
            asio::buffer(buf), asio::use_awaitable);

        co_await asio::async_write(
            socket, asio::buffer(buf, n),
            asio::use_awaitable);
    }
}

The coroutine suspends at each co_await, freeing the thread to run other tasks. When the I/O completes, the coroutine resumes on a pool thread. No callbacks, no blocking, no thread waste.

std::execution (C++26): The Future Standard

The C++ standardization committee has been working on std::execution (formerly known as “senders and receivers” or P2300). Accepted for C++26, it provides a composable framework for describing asynchronous work.

Core Concepts

• Sender: describes work to be done, but does not start it. Think of it as a lazy future.
• Receiver: a callback that consumes the result of a sender (value, error, or cancellation).
• Scheduler: determines where work runs (thread pool, inline, GPU, etc.).

// Conceptual std::execution usage (C++26)
namespace ex = std::execution;

ex::static_thread_pool pool(4);
ex::scheduler auto sched = pool.get_scheduler();

auto work = ex::schedule(sched)
    | ex::then([] { return read_data(); })
    | ex::then([](Data d) { return process(d); })
    | ex::then([](Result r) { write_output(r); });

// Start the pipeline
ex::sync_wait(std::move(work));

Why It Matters

• Composability. Senders compose like pipes. You can build complex async graphs without callbacks or shared state.
• Scheduler-agnostic. The same sender chain can run on a thread pool, an event loop, or a GPU by simply changing the scheduler.
• Structured concurrency. Operations like when_all ensure all child tasks complete (or are cancelled) before the parent continues.
• Cancellation. Stop tokens are built into the protocol. Cancelling a parent automatically cancels children.

Practical Architecture: Combining I/O and Compute

A high-performance server typically uses this architecture:

1. I/O threads (1 per core, or fewer): run the event loop (epoll, IOCP, or io_uring). They never execute user code beyond minimal parsing.
2. Compute pool: handles CPU-bound request processing, serialization, encoding, and business logic.
3. Completion posting: when a compute task finishes, it posts the response back to the I/O thread for a non-blocking write.

// Architecture skeleton
class HighPerfServer {
    // I/O layer
    asio::io_context io_ctx_;
    asio::ip::tcp::acceptor acceptor_;

    // Compute layer
    ThreadPool compute_pool_{
        std::thread::hardware_concurrency()};

    void accept_loop() {
        acceptor_.async_accept(
            [this](asio::error_code ec,
                   asio::ip::tcp::socket socket) {
            if (!ec) handle_connection(std::move(socket));
            accept_loop();  // accept next
        });
    }

    void handle_connection(asio::ip::tcp::socket socket) {
        auto shared_socket = std::make_shared<
            asio::ip::tcp::socket>(std::move(socket));

        // Read request on I/O thread (non-blocking)
        auto buf = std::make_shared<std::vector<char>>(4096);
        shared_socket->async_read_some(
            asio::buffer(*buf),
            [this, shared_socket, buf](
                asio::error_code ec, std::size_t n) {
            if (ec) return;

            // Offload processing to compute pool
            compute_pool_.submit(
                [this, shared_socket, buf, n] {
                auto response = process_request(
                    buf->data(), n);

                // Post write back to I/O thread
                asio::post(io_ctx_,
                    [shared_socket,
                     resp = std::move(response)] {
                    asio::async_write(
                        *shared_socket,
                        asio::buffer(resp),
                        [](asio::error_code, std::size_t){});
                });
            });
        });
    }
};

Guidelines

• Never call blocking I/O from a pool thread. If you must wait on I/O, submit the wait to the event loop instead.
• Keep I/O handlers lightweight. Heavy parsing or computation should be offloaded to the compute pool.
• Use asio::strand if multiple handlers access the same connection state, to avoid data races without explicit locking.
• For disk I/O, consider dedicated I/O threads or io_uring rather than the compute pool.

Linux io_uring: The Modern Proactor

io_uring (introduced in Linux 5.1) is a high-performance async I/O interface that works with both network and file I/O. It uses two ring buffers shared between user space and the kernel:

• Submission Queue (SQ): the application pushes I/O requests.
• Completion Queue (CQ): the kernel pushes completed results.

This design eliminates most system calls from the hot path. The application writes to the SQ and reads from the CQ using plain memory operations; the kernel processes requests asynchronously.

// Simplified io_uring read sketch
#include <liburing.h>

struct io_uring ring;
io_uring_queue_init(256, &ring, 0);

// Prepare a read operation
struct io_uring_sqe* sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buffer, buf_size, 0);
io_uring_sqe_set_data(sqe, user_context);

// Submit
io_uring_submit(&ring);

// Wait for completion
struct io_uring_cqe* cqe;
io_uring_wait_cqe(&ring, &cqe);

int bytes_read = cqe->res;
auto* ctx = io_uring_cqe_get_data(cqe);
io_uring_cqe_seen(&ring, cqe);

// Process the result on a pool thread
pool.submit([ctx, bytes_read] {
    ctx->on_read_complete(bytes_read);
});

Anti-Patterns to Avoid

1. Blocking the Pool with Sleep or I/O

// BAD: blocks a pool thread for the entire duration
pool.submit([fd] {
    char buf[4096];
    read(fd, buf, sizeof(buf));  // might block for seconds
    process(buf);
});

Use non-blocking I/O or async operations instead. If you must do blocking I/O, use a dedicated I/O thread pool separate from the compute pool.

2. Unbounded Thread Growth

// BAD: defeats the purpose of a pool
void handle(Request req) {
    std::thread([req] { process(req); }).detach();
}

Detached threads are unmanageable. They consume resources without bound and cannot be joined on shutdown.

3. Mixing Event Loop and Pool Incorrectly

Calling io_context::run() from inside a pool task creates a nested event loop. Each layer should have its own dedicated threads.

4. Ignoring Backpressure

If the compute pool is full and I/O completions arrive faster than they can be processed, the completion queue overflows. Implement backpressure: stop accepting new connections or pause reads until the pool drains.

Library Comparison

Summary

• Blocking I/O in a fixed-size thread pool wastes threads and can stall the entire system.
• Reactors (epoll, kqueue) notify on readiness; proactors (IOCP, io_uring) notify on completion.
• Separate I/O handling from CPU-bound work: event loop threads for I/O, thread pool for compute.
• Post results back to the I/O layer for non-blocking writes.
• Boost.Asio is the current production standard; std::execution (C++26) is the future.
• Never block pool threads on I/O. Use cooperative async patterns instead.