Idempotency & Exactly-Once Semantics

April 2026 · 16 min read

What Is Idempotency?

An operation is idempotent if applying it multiple times produces the same result as applying it once. In mathematical notation:

f(f(x)) = f(x)

Or more precisely — no matter how many times you apply the function, the outcome is identical to applying it once:

f(x) = f(f(x)) = f(f(f(x))) = ... = f^n(x)  for all n ≥ 1

In software systems, this means: if a client sends the same request multiple times, the system processes it only once and returns the same result every time. The side effects occur exactly once, even if the request is received multiple times.

Simple real-world examples:

• Setting a thermostat to 72°F — doing it once or ten times, the temperature is still 72°F. Idempotent. ✓
• Pressing an elevator button — pressing it once or five times, the elevator still comes once. Idempotent. ✓
• Adding $50 to a bank account — doing it twice adds $100, not $50. NOT idempotent. ✗
• Toggling a light switch — each press flips the state. NOT idempotent. ✗

Why Idempotency Matters in Distributed Systems

In a perfect world, every network request succeeds exactly once. But distributed systems aren't perfect. Here's what actually happens:

The Network Is Unreliable

Retry Storms & Amplification

Modern systems aggressively retry failed requests:

// Typical client retry configuration
const retryConfig = {
  maxRetries: 3,
  initialDelay: 100,    // ms
  backoffMultiplier: 2, // exponential backoff
  maxDelay: 5000,       // ms
  retryOn: [408, 429, 500, 502, 503, 504]
};

// A single user action can generate up to 4 requests (1 + 3 retries)
// 10,000 concurrent users × 4 = 40,000 requests hitting your payment API
// Without idempotency: 40,000 charges instead of 10,000

The problem compounds across layers: the mobile app retries, the API gateway retries, the internal service retries, the message queue redelivers. A single user action can trigger 10+ identical requests through the system.

Naturally Idempotent vs Non-Idempotent Operations

Not all operations need special handling. Some are naturally idempotent — safe to retry without any additional mechanism.

HTTP Methods & Idempotency

Database Operations

-- ✅ Naturally idempotent: absolute writes
UPDATE users SET email = 'alice@example.com' WHERE id = 42;
-- Run this 100 times → email is still alice@example.com

DELETE FROM sessions WHERE user_id = 42;
-- Run this 100 times → sessions still deleted

INSERT INTO users (id, name) VALUES (42, 'Alice')
  ON CONFLICT (id) DO UPDATE SET name = 'Alice';
-- Run this 100 times → one row with name = Alice

-- ❌ NOT idempotent: relative writes
UPDATE accounts SET balance = balance + 50 WHERE id = 42;
-- Run this 3 times → balance increased by $150, not $50!

INSERT INTO orders (user_id, product_id, amount)
  VALUES (42, 101, 29.99);
-- Run this 3 times → 3 duplicate orders!

UPDATE counters SET count = count + 1 WHERE page = '/home';
-- Run this 3 times → count incremented by 3, not 1!

Idempotency Keys

An idempotency key is a unique identifier that the client attaches to each operation. The server uses this key to detect duplicates and return the cached response instead of re-processing the request.

How Idempotency Keys Work

The protocol is straightforward:

1. Client generates a unique key (typically a UUID) for each distinct operation
2. Client sends the request with the key in a header: Idempotency-Key: abc123
3. Server checks its idempotency store for the key:
   • Key not found: Process the request normally, store key → response, return response
   • Key found, processing complete: Return the cached response without re-processing
   • Key found, still processing: Return 409 Conflict to prevent concurrent execution
4. Client retries (on timeout/failure) with the same key — server returns cached result

Key Generation Strategies

// Strategy 1: UUID v4 (most common)
const idempotencyKey = crypto.randomUUID();
// "550e8400-e29b-41d4-a716-446655440000"

// Strategy 2: Deterministic from operation parameters
// Same inputs → same key → automatic dedup even across clients
const idempotencyKey = sha256(`${userId}:${orderId}:${amount}:${timestamp}`);
// Useful when the operation itself defines uniqueness

// Strategy 3: Client-generated sequence number
// Each client tracks its own monotonic counter
const idempotencyKey = `client-${clientId}-seq-${sequenceNum}`;

// Strategy 4: Stripe-style — client generates, sends in header
fetch('/v1/charges', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer sk_live_...',
    'Idempotency-Key': 'order-12345-attempt-1',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    amount: 2000,
    currency: 'usd',
    source: 'tok_visa'
  })
});

Server-Side Implementation

Redis-Based Idempotency Store

Redis is the most common choice for idempotency stores due to its speed, built-in TTL, and atomic operations:

import redis
import json
import uuid
from functools import wraps
from flask import request, jsonify

r = redis.Redis(host='localhost', port=6379, db=0)

IDEMPOTENCY_TTL = 86400  # 24 hours

def idempotent(f):
    """Decorator that makes an endpoint idempotent using Redis."""
    @wraps(f)
    def wrapper(*args, **kwargs):
        idem_key = request.headers.get('Idempotency-Key')
        if not idem_key:
            return jsonify({'error': 'Idempotency-Key header required'}), 400

        redis_key = f'idempotency:{idem_key}'

        # Step 1: Try to acquire a lock (prevent concurrent execution)
        lock_key = f'idempotency_lock:{idem_key}'
        lock_acquired = r.set(lock_key, '1', nx=True, ex=60)

        if not lock_acquired:
            # Another request with same key is currently being processed
            return jsonify({'error': 'Request in progress'}), 409

        try:
            # Step 2: Check if we already have a cached response
            cached = r.get(redis_key)
            if cached:
                response_data = json.loads(cached)
                return jsonify(response_data['body']), response_data['status']

            # Step 3: Process the request
            response = f(*args, **kwargs)
            status_code = response[1] if isinstance(response, tuple) else 200
            body = response[0] if isinstance(response, tuple) else response

            # Step 4: Cache the response
            r.setex(redis_key, IDEMPOTENCY_TTL, json.dumps({
                'body': body,
                'status': status_code,
                'created_at': str(uuid.uuid4())
            }))

            return jsonify(body), status_code

        finally:
            # Step 5: Release the lock
            r.delete(lock_key)

    return wrapper

# Usage:
@app.route('/v1/charges', methods=['POST'])
@idempotent
def create_charge():
    data = request.json
    charge = payment_processor.charge(
        amount=data['amount'],
        currency=data['currency'],
        source=data['source']
    )
    return {'id': charge.id, 'amount': charge.amount, 'status': 'succeeded'}, 201

Database-Based Idempotency Store

When you need stronger durability guarantees, use a database table within the same transaction as the business operation:

-- PostgreSQL idempotency table
CREATE TABLE idempotency_keys (
    key         VARCHAR(255) PRIMARY KEY,
    request_path VARCHAR(500) NOT NULL,
    request_body JSONB,
    response_body JSONB,
    response_code INTEGER,
    locked_at    TIMESTAMP,      -- NULL = not locked
    completed_at TIMESTAMP,      -- NULL = not yet completed
    created_at   TIMESTAMP DEFAULT NOW(),
    expires_at   TIMESTAMP DEFAULT NOW() + INTERVAL '24 hours'
);

-- Index for cleanup job
CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at)
  WHERE completed_at IS NOT NULL;

-- Cleanup job: remove expired keys
DELETE FROM idempotency_keys WHERE expires_at < NOW();

The critical advantage: the idempotency check and the business operation share the same database transaction, providing atomicity:

async function processPayment(idempotencyKey, chargeRequest) {
  return await db.transaction(async (tx) => {
    // Step 1: Try to insert or find existing key (atomic)
    const existing = await tx.query(`
      INSERT INTO idempotency_keys (key, request_path, request_body, locked_at)
      VALUES ($1, $2, $3, NOW())
      ON CONFLICT (key) DO UPDATE SET key = idempotency_keys.key
      RETURNING *
    `, [idempotencyKey, '/v1/charges', chargeRequest]);

    const record = existing.rows[0];

    // Step 2: If completed, return cached response
    if (record.completed_at) {
      return { status: record.response_code, body: record.response_body };
    }

    // Step 3: If locked by another request, return conflict
    if (record.locked_at && !existing.command.includes('INSERT')) {
      throw new ConflictError('Request already in progress');
    }

    // Step 4: Process the charge (within same transaction!)
    const charge = await tx.query(`
      INSERT INTO charges (user_id, amount, currency, status)
      VALUES ($1, $2, $3, 'succeeded')
      RETURNING id, amount, currency, status
    `, [chargeRequest.userId, chargeRequest.amount, chargeRequest.currency]);

    // Step 5: Mark idempotency key as completed (same transaction!)
    const response = { id: charge.rows[0].id, status: 'succeeded' };
    await tx.query(`
      UPDATE idempotency_keys
      SET response_body = $1, response_code = 201, completed_at = NOW(), locked_at = NULL
      WHERE key = $2
    `, [response, idempotencyKey]);

    // If the transaction commits → charge + idempotency record both persist
    // If the transaction rolls back → neither persists → safe to retry
    return { status: 201, body: response };
  });
}

TTL & Cleanup Strategies

Idempotency keys should not be stored forever. Choose TTL based on your retry window:

# Redis: TTL is automatic
r.setex(f'idempotency:{key}', 86400, response_json)  # 24h TTL

# PostgreSQL: Background cleanup job
-- Run every hour via pg_cron or application scheduler
SELECT cron.schedule('cleanup-idempotency', '0 * * * *',
  $$DELETE FROM idempotency_keys WHERE expires_at < NOW()$$
);

# Or with partitioning for high-volume systems:
-- Partition by day, drop old partitions instead of deleting rows
CREATE TABLE idempotency_keys (
    key TEXT, created_at DATE, ...
) PARTITION BY RANGE (created_at);

CREATE TABLE idempotency_keys_20260401
  PARTITION OF idempotency_keys
  FOR VALUES FROM ('2026-04-01') TO ('2026-04-02');
-- Simply DROP old partition tables instead of deleting millions of rows

Delivery Guarantees

In distributed messaging, there are three delivery semantics. Understanding these is critical because they determine your system's correctness and complexity.

// At-most-once: send and forget
producer.send(message);  // No ack, no retry
// If the network drops it... 🤷 it's gone

// At-least-once: retry until acknowledged
while (!acknowledged) {
  producer.send(message);
  acknowledged = waitForAck(timeout: 5000);
  // ACK might be lost → we retry → consumer gets it TWICE
}

// "Exactly-once" = at-least-once delivery + idempotent consumer
// The message might be sent twice, but the EFFECT happens only once
consumer.onMessage(async (msg) => {
  const deduped = await idempotencyStore.checkAndLock(msg.id);
  if (deduped) return ack(msg);  // Already processed, skip

  await processMessage(msg);     // Process once
  await idempotencyStore.markDone(msg.id);
  ack(msg);
});

Kafka Exactly-Once Semantics (EOS)

Apache Kafka achieves exactly-once semantics through two mechanisms that work together: idempotent producers and transactions.

Idempotent Producer

Without idempotency, a Kafka producer retrying a failed send() can write the same message twice to the same partition. The idempotent producer prevents this:

# Kafka producer configuration for idempotency
producer_config = {
    'bootstrap.servers': 'kafka-1:9092,kafka-2:9092,kafka-3:9092',

    # Enable idempotent producer (prevents duplicate writes)
    'enable.idempotence': True,

    # Required settings when idempotence is enabled:
    'acks': 'all',           # Wait for all replicas to acknowledge
    'retries': 2147483647,   # Retry indefinitely (Integer.MAX_VALUE)
    'max.in.flight.requests.per.connection': 5,  # Max 5 (Kafka 1.1+)

    # How it works under the hood:
    # 1. Producer gets a unique Producer ID (PID) from the broker on init
    # 2. Each message gets a monotonic sequence number per partition
    # 3. Broker tracks (PID, partition, sequence_number) tuples
    # 4. If broker sees a duplicate sequence number → it deduplicates silently
    # 5. If broker sees a gap in sequence numbers → it rejects (OutOfOrderSequence)
}

How the idempotent producer works internally:

// Internal flow (simplified):
Producer                              Broker (Partition Leader)
   |                                        |
   |-- PID=7, Seq=0, msg="order-1" -----→ |  ✓ Accept (new sequence)
   |                                        |  Store: PID=7, maxSeq=0
   |                                        |
   |-- PID=7, Seq=1, msg="order-2" -----→ |  ✓ Accept (sequence incremented)
   |                                        |  Store: PID=7, maxSeq=1
   |                                        |
   |-- PID=7, Seq=1, msg="order-2" -----→ |  ✗ Deduplicate! (seq=1 ≤ maxSeq=1)
   |    (retry due to network timeout)      |  Return success but don't write
   |                                        |
   |-- PID=7, Seq=5, msg="order-6" -----→ |  ✗ Reject! OutOfOrderSequenceException
   |    (gap: expected seq=2, got seq=5)    |  (sequence numbers must be contiguous)

Kafka Transactions

Transactions provide atomicity across multiple partitions and topics — either all messages in the transaction are committed, or none are:

# Transactional producer configuration
producer_config = {
    'bootstrap.servers': 'kafka-1:9092,kafka-2:9092,kafka-3:9092',
    'enable.idempotence': True,
    'transactional.id': 'payment-processor-1',  # Unique per producer instance
    # The transactional.id survives producer restarts:
    # - Broker uses it to fence zombie producers (old instances with same ID)
    # - New producer with same transactional.id aborts any pending transactions
    #   from the old producer (zombie fencing)
}

# Python example with confluent-kafka
from confluent_kafka import Producer

producer = Producer(producer_config)
producer.init_transactions()  # Register with transaction coordinator

try:
    producer.begin_transaction()

    # All of these writes are atomic — all or nothing
    producer.produce('orders', key='user-42', value='{"orderId": "ord-123"}')
    producer.produce('inventory', key='sku-789', value='{"delta": -1}')
    producer.produce('notifications', key='user-42', value='{"type": "order_confirmed"}')

    producer.commit_transaction()  # Atomic commit across 3 topics

except Exception as e:
    producer.abort_transaction()   # Atomic rollback — none of the 3 messages appear
    raise

The consume-transform-produce Pattern

The most powerful use of Kafka EOS is the consume-transform-produce pattern, where reading from an input topic, processing, and writing to an output topic are all part of one atomic transaction:

// Java: Exactly-once consume-transform-produce
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("group.id", "payment-processor");
props.put("isolation.level", "read_committed"); // Only read committed messages
props.put("enable.auto.commit", false);          // Manual offset management

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps);
producer.initTransactions();

consumer.subscribe(List.of("raw-payments"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, String> record : records) {
            // Transform
            String processed = processPayment(record.value());

            // Produce to output topic (within transaction)
            producer.send(new ProducerRecord<>("processed-payments",
                record.key(), processed));
        }

        // Commit consumer offsets within the same transaction
        // This is the key: offset commit + output writes are ATOMIC
        producer.sendOffsetsToTransaction(
            getOffsetsForTransaction(records),
            consumer.groupMetadata()
        );

        producer.commitTransaction();
        // If commit succeeds: output written + offsets advanced (atomically)
        // If commit fails: nothing happened — consumer re-reads same messages
    } catch (Exception e) {
        producer.abortTransaction();
        // Offsets not committed → consumer will re-read → no data loss
    }
}

Stripe's Idempotency Keys — A Production Case Study

Stripe is the gold standard for idempotency in payment APIs. Their approach is so well-designed that it's become the industry pattern.

Using Stripe's Idempotency API

import stripe
stripe.api_key = "sk_live_..."

# Create a charge with an idempotency key
# If you retry with the same key, you get the same charge — not a double charge
charge = stripe.Charge.create(
    amount=2000,           # $20.00
    currency="usd",
    source="tok_visa",
    description="Order #12345",
    idempotency_key="order-12345-charge-v1"  # Client-generated unique key
)
# Returns: {"id": "ch_1abc...", "amount": 2000, "status": "succeeded"}

# Retry with same key → same response, no new charge
charge_retry = stripe.Charge.create(
    amount=2000,
    currency="usd",
    source="tok_visa",
    description="Order #12345",
    idempotency_key="order-12345-charge-v1"  # Same key!
)
# Returns: {"id": "ch_1abc...", "amount": 2000, "status": "succeeded"}
# Same charge ID! No duplicate.

Stripe's Idempotency Behavior in Detail

How Stripe Implements It Internally

Based on Stripe's engineering blog posts, their implementation uses a state machine approach:

// Stripe's internal idempotency flow (reconstructed from public engineering blog)
// Reference: https://stripe.com/blog/idempotency

enum IdempotencyKeyState {
  STARTED,     // Key inserted, request being processed
  FINISHED,    // Request completed, response cached
}

// The idempotency record
{
  "key": "order-12345-charge-v1",
  "state": "FINISHED",
  "request_method": "POST",
  "request_path": "/v1/charges",
  "request_params_hash": "sha256(...)", // Hash of request body
  "response_code": 200,
  "response_body": "{\"id\": \"ch_1abc...\", ...}",
  "recovery_point": "charge_created",   // Last completed step
  "created_at": "2026-04-01T10:30:00Z",
  "locked_at": null
}

// Stripe uses "recovery points" for complex multi-step operations:
// 1. started         → key created
// 2. charge_created  → charge record inserted in DB
// 3. fraud_checked   → fraud model approved
// 4. settled         → funds captured from payment network
// 5. finished        → response sent to client
//
// If the server crashes at step 3, a retry with the same key
// resumes from step 3 (fraud_checked) instead of starting over.
// This is IDEMPOTENT PROCESSING, not just idempotent responses.

Edge Cases Stripe Handles

// Edge case 1: Same key, different parameters → ERROR
stripe.Charge.create(amount=2000, currency='usd',
    idempotency_key='key-1')  # ✓ Processed

stripe.Charge.create(amount=3000, currency='usd',  # Different amount!
    idempotency_key='key-1')  # ✗ 400: Keys can only be reused with same parameters

// Edge case 2: Non-retryable errors ARE cached
stripe.Charge.create(amount=-100, currency='usd',
    idempotency_key='key-2')  # ✗ 400: Invalid positive integer

stripe.Charge.create(amount=-100, currency='usd',
    idempotency_key='key-2')  # ✗ 400: Same cached error (not reprocessed)

// Edge case 3: 500 errors are NOT cached (server-side failures are retryable)
// If Stripe's server crashes during processing → key is cleaned up
// → client can retry with same key → request is reprocessed

// Edge case 4: Concurrent requests with same key
// Request A: POST /charges, key='key-3' → processing...
// Request B: POST /charges, key='key-3' → 409 Conflict (wait and retry)

// Edge case 5: Network-level retries (transparent to client)
// Stripe's client libraries handle this automatically:
stripe.max_network_retries = 2  // Retry up to 2 times on network errors
// Each retry uses the SAME idempotency key automatically

Database Upserts — INSERT ON CONFLICT

Database upserts are a powerful tool for making write operations idempotent at the storage layer. Instead of checking for existence then inserting (which has race conditions), you do it atomically.

PostgreSQL: INSERT ... ON CONFLICT

-- Basic upsert: insert or update if already exists
INSERT INTO users (id, name, email, updated_at)
VALUES (42, 'Alice', 'alice@example.com', NOW())
ON CONFLICT (id) DO UPDATE SET
  name = EXCLUDED.name,
  email = EXCLUDED.email,
  updated_at = EXCLUDED.updated_at;

-- Run this 100 times → exactly one row, always up-to-date. Idempotent!

-- Upsert with conditional update (only update if data is newer)
INSERT INTO events (event_id, user_id, payload, event_time)
VALUES ('evt-123', 42, '{"action": "click"}', '2026-04-01 10:30:00')
ON CONFLICT (event_id) DO UPDATE SET
  payload = EXCLUDED.payload,
  event_time = EXCLUDED.event_time
WHERE events.event_time < EXCLUDED.event_time;
-- Only updates if the incoming event is newer (last-write-wins with timestamp)

-- Upsert for counters (idempotent increment using event deduplication)
-- Instead of: UPDATE counters SET count = count + 1 (not idempotent!)
-- Use: event-sourced approach with dedup
INSERT INTO processed_events (event_id, counter_name, delta)
VALUES ('evt-456', 'page_views', 1)
ON CONFLICT (event_id) DO NOTHING;  -- Skip if already processed

-- Then compute the count from events:
SELECT SUM(delta) as total FROM processed_events WHERE counter_name = 'page_views';

MySQL: INSERT ... ON DUPLICATE KEY UPDATE

-- MySQL equivalent of PostgreSQL's ON CONFLICT
INSERT INTO users (id, name, email, updated_at)
VALUES (42, 'Alice', 'alice@example.com', NOW())
ON DUPLICATE KEY UPDATE
  name = VALUES(name),
  email = VALUES(email),
  updated_at = VALUES(updated_at);

-- REPLACE INTO (deletes + re-inserts): use with caution
REPLACE INTO users (id, name, email)
VALUES (42, 'Alice', 'alice@example.com');
-- Warning: REPLACE deletes the old row first, which triggers
-- DELETE + INSERT — not always what you want (cascading FKs, auto-inc)

DynamoDB: Conditional Writes

// DynamoDB conditional write — only succeeds if item doesn't exist
const params = {
  TableName: 'Orders',
  Item: {
    orderId: { S: 'order-12345' },
    userId: { S: 'user-42' },
    amount: { N: '2999' },
    status: { S: 'created' },
    processedAt: { S: new Date().toISOString() }
  },
  ConditionExpression: 'attribute_not_exists(orderId)',
  // Fails with ConditionalCheckFailedException if order already exists
};

try {
  await dynamoDb.putItem(params).promise();
  console.log('Order created');
} catch (err) {
  if (err.code === 'ConditionalCheckFailedException') {
    console.log('Order already exists — idempotent skip');
  } else {
    throw err;
  }
}

MongoDB: Upsert with $setOnInsert

// MongoDB upsert — idempotent insert-or-update
db.orders.updateOne(
  { orderId: 'order-12345' },  // filter
  {
    $setOnInsert: {             // Only set these on INSERT (first time)
      createdAt: new Date(),
      status: 'created'
    },
    $set: {                     // Always set these (safe to repeat)
      userId: 'user-42',
      amount: 29.99,
      updatedAt: new Date()
    }
  },
  { upsert: true }             // Create if not exists
);
// Run 100 times: one document, createdAt never changes, updatedAt refreshes

Deduplication at Different Layers

Idempotency can be enforced at multiple layers of your system. The right choice depends on where duplicates originate and what guarantees you need.

Defense in Depth — Layered Deduplication

Production systems use multiple layers of deduplication because no single layer catches everything:

// Layer 1: API Gateway — catch obvious duplicates early
// Kong plugin or custom middleware
if (await redis.exists(`gateway:dedup:${req.headers['idempotency-key']}`)) {
  return cachedResponse;
}

// Layer 2: Application — idempotency store with full response caching
@idempotent  // Our decorator from earlier
async function createOrder(req) {
  // Layer 3: Database — unique constraint as final safety net
  try {
    await db.query(`
      INSERT INTO orders (order_id, user_id, amount)
      VALUES ($1, $2, $3)
      ON CONFLICT (order_id) DO NOTHING
      RETURNING *
    `, [orderId, userId, amount]);
  } catch (err) {
    if (err.code === '23505') {  // Unique violation
      return await db.query('SELECT * FROM orders WHERE order_id = $1', [orderId]);
    }
    throw err;
  }
}

// Layer 4: Message consumer — dedup on message processing
consumer.on('message', async (msg) => {
  const messageId = msg.headers['message-id'];
  const processed = await redis.set(
    `consumer:dedup:${messageId}`, '1', 'NX', 'EX', 86400
  );
  if (!processed) return;  // Already processed this message
  await handleMessage(msg);
});

Trade-offs of Each Layer

// Gateway dedup
// ✓ Catches duplicates before they hit your application
// ✓ Reduces load on backend services
// ✗ Can't access business logic (doesn't know if params match)
// ✗ Adds latency to every request (Redis lookup)

// Application dedup
// ✓ Full control — can validate params, handle edge cases
// ✓ Can cache rich responses (not just "already processed")
// ✗ Added complexity per endpoint
// ✗ Multiple services = multiple dedup stores

// Database dedup (unique constraints)
// ✓ Last line of defense — if everything else fails, DB catches it
// ✓ Zero application complexity — the DB does the work
// ✗ Only catches exact duplicate primary keys
// ✗ Doesn't return cached responses (just prevents duplicate inserts)
// ✗ Race conditions with read-then-write patterns (use upserts!)

// Message broker dedup (Kafka)
// ✓ Built into the protocol — transparent to application
// ✓ Zero application code for producer-side dedup
// ✗ Only covers producer→broker path (not broker→consumer)
// ✗ Limited scope (per partition, per producer session)

Real-World Idempotency Patterns

The Outbox Pattern + Idempotency

The transactional outbox pattern combined with idempotent consumers achieves effectively exactly-once across service boundaries:

// Service A: Write to business table + outbox in ONE transaction
BEGIN;
  INSERT INTO orders (id, user_id, amount) VALUES ('ord-123', 42, 29.99);
  INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
  VALUES (gen_random_uuid(), 'Order', 'ord-123', 'OrderCreated',
          '{"orderId": "ord-123", "userId": 42, "amount": 29.99}');
COMMIT;
-- Both succeed or both fail — no "order created but event lost" scenario

// CDC connector (Debezium) reads outbox table → publishes to Kafka
// At-least-once delivery (Debezium may publish duplicates)

// Service B: Idempotent consumer handles duplicates
consumer.on('OrderCreated', async (event) => {
  await db.query(`
    INSERT INTO shipments (order_id, status, created_at)
    VALUES ($1, 'pending', NOW())
    ON CONFLICT (order_id) DO NOTHING
  `, [event.orderId]);
  // Duplicate events → no duplicate shipments!
});

Idempotency in Sagas

In a distributed saga, each step and its compensation must be idempotent because the saga orchestrator may retry any step:

// Saga: Place Order
// Each step is idempotent — safe to retry on failure

Step 1: Reserve Inventory
  → INSERT INTO reservations (order_id, sku, qty)
    ON CONFLICT (order_id, sku) DO NOTHING;
  Compensation: DELETE FROM reservations WHERE order_id = $1;
  (Deleting twice = still deleted. Idempotent.)

Step 2: Charge Payment
  → stripe.Charge.create(..., idempotency_key=f"order-{orderId}-charge")
  Compensation: stripe.Refund.create(charge_id, idempotency_key=f"order-{orderId}-refund")
  (Same idempotency key = same refund. Idempotent.)

Step 3: Create Shipment
  → INSERT INTO shipments (order_id, status)
    ON CONFLICT (order_id) DO UPDATE SET status = 'pending';
  Compensation: UPDATE shipments SET status = 'cancelled' WHERE order_id = $1;
  (Setting status to 'cancelled' twice = still cancelled. Idempotent.)

State Machine as Idempotency Guard

// Use state machine transitions to enforce idempotency
// Only valid transitions are processed; invalid ones are no-ops

const ORDER_TRANSITIONS = {
  'created':    ['payment_pending'],
  'payment_pending': ['paid', 'payment_failed'],
  'paid':       ['shipped'],
  'shipped':    ['delivered'],
  'delivered':  [],  // terminal state
  'payment_failed': ['payment_pending'],  // can retry payment
};

async function transitionOrder(orderId, fromState, toState) {
  // Atomic state transition — only succeeds if current state matches
  const result = await db.query(`
    UPDATE orders SET status = $1, updated_at = NOW()
    WHERE id = $2 AND status = $3
    RETURNING *
  `, [toState, orderId, fromState]);

  if (result.rowCount === 0) {
    // Either order doesn't exist, or it's already past this state
    const current = await db.query('SELECT status FROM orders WHERE id = $1', [orderId]);
    if (current.rows[0]?.status === toState) {
      return { alreadyTransitioned: true };  // Idempotent — already in target state
    }
    throw new InvalidTransitionError(`Cannot transition from ${current.rows[0]?.status} to ${toState}`);
  }

  return result.rows[0];
}

// Usage: calling transitionOrder('ord-123', 'paid', 'shipped') twice
// → First call: updates status to 'shipped'
// → Second call: returns { alreadyTransitioned: true } — no-op, idempotent!

Common Pitfalls

Summary

• Idempotency means an operation can be applied multiple times without changing the result beyond the first application: f(f(x)) = f(x).
• Networks are unreliable. Retries are inevitable. Without idempotency, retries cause duplicate charges, orders, and data corruption.
• Naturally idempotent operations (GET, PUT, DELETE, absolute writes) need no special handling. Non-idempotent operations (POST, increments, relative writes) need idempotency keys.
• Idempotency keys are client-generated unique identifiers. The server stores key → response and returns the cached response on retries.
• Implementation: Use Redis for speed or a database table (same transaction) for ACID guarantees. Always set a TTL for cleanup.
• Delivery guarantees: at-most-once (may lose), at-least-once (may duplicate), exactly-once (= at-least-once + idempotent consumer).
• Kafka EOS: idempotent producer (PID + sequence numbers) + transactions (atomic multi-partition writes) + read_committed consumers.
• Stripe: the gold standard — 24h TTL, parameter binding, recovery points for complex operations.
• Database upserts (ON CONFLICT) provide idempotency at the storage layer.
• Defense in depth: dedup at API gateway, application, message broker, consumer, and database layers.

In the next post, we'll shift gears to Architecture Patterns and explore Microservices — how to decompose a monolith into independently deployable services, and the distributed systems challenges (including idempotency!) that come with it.