Idempotency Keys for Money Movement: Preventing Double Charges

An idempotency key is a unique, client-generated value sent with a request. The server uses it to recognize a retry and return the original result, instead of doing the work again.

Retry safety for money has three rungs: plain retries, an idempotency key, and a key committed with the ledger entry. The third is best. This post builds each one, including a Go middleware on Postgres.

Table of Contents

Rung 1: why plain retries cause double charges

A payment request can fail in three places, and the client can’t tell them apart. Brandur Leach’s 2017 post for Stripe, Designing robust and predictable APIs with idempotency, lists them:

  1. The connection fails before the request reaches the server.
  2. The server fails midway and leaves the work in limbo.
  3. The operation succeeds, and the connection breaks before the response gets back.

From the client’s side, all three look like a timeout. The client then gives up or retries. If it gives up, a charge may have gone through that nobody tells the user about. If it retries, in the third case the card is charged twice.

Almost every client retries, because HTTP libraries ship with retries built in. Queues have the same problem in a different transport. A broker that promises at-least-once delivery will redeliver when a consumer crashes after doing the work but before acknowledging it.

So plain retries buy recovery from a lost request, and cost a double charge when the original succeeded. Duplicates will arrive. The server has to tell a duplicate from a new request. That is the whole job of the idempotency key.

I first dealt with this as CTO at Levpay, a payments company later acquired by Pagsmile. I deal with it today on crypto trading systems in Go at Questrade (more on my background). What follows is the textbook version, the one I would hand a new team.

Rung 2: an idempotency key per payment intent

The client generates the key. Stripe’s idempotent requests documentation says: “A client generates an idempotency key, which is a unique key that the server uses to recognize subsequent retries of the same request.” Stripe suggests V4 UUIDs and caps keys at 255 characters. It tells you to keep sensitive data such as email addresses out of them.

The IETF Idempotency-Key header draft from the httpapi working group agrees. A UUID or similar random identifier is RECOMMENDED, and the key “MUST NOT be reused with another request with a different request payload.”

The part people get wrong is when to generate it. A fresh UUID created inside the HTTP call wrapper is useless. The retry gets a new key, and the server sees two different requests.

Create the key when the intent is created. That’s the moment the user presses Pay, or your system decides a payout is due. Persist it with that intent before the first attempt leaves the process, so it survives a restart of your service.

In practice that is a payment_attempts row carrying a random key. Or it’s a key derived from your own identifiers, like payout-{payout_id}-attempt-{n}. Derived keys are readable in logs. Random ones leak nothing if they land in a log aggregator you don’t fully trust. I lean toward random, but either works as long as it survives your own crashes.

The header name is converging on Idempotency-Key, which the IETF draft specifies and Stripe uses. Adyen uses the same header with a 64-character limit. Square accepts a key on operations like CreatePayment.

This rung buys one winner per key. It costs a table, a state machine, a fingerprint rule, a scope and a TTL. The next five sections cover each.

How to store the keys

Whatever the backing store, each key needs the same fields.

Column Type Why it is there
account_id text Scope. Keys are unique within an account.
idempotency_key text The client’s value, at most 255 characters.
fingerprint text SHA-256 of method, path and body. Detects the same key with a different request.
status text in_progress or done. Drives the state machine.
response_code int Status code of the original response, null while in progress.
response_body bytea The original response, replayed to duplicates.
created_at timestamptz Debugging and metrics.
expires_at timestamptz When the row can be deleted.

The primary key is (account_id, idempotency_key), and that constraint is the mechanism. Two requests racing to insert the same pair produce exactly one row. The loser finds out without any lock you had to write.

Store your own response body, never raw upstream payloads. A card PAN or a full bank account number in that column is a second place where the data lives. It comes with a retention policy nobody signed off on.

How a key moves through its states

A key moves through three states.

  1. New. The row does not exist. The first request to insert it wins and does the work.
  2. In progress. The row exists, the handler is running, the response columns are null.
  3. Done. The status code and body are saved, and every later request with the same key and fingerprint gets them back.

The interesting case is a duplicate that arrives while the original is still in progress. The IETF draft says the resource “SHOULD reply with an HTTP 409 status code with body containing problem description.” Adyen documents returning 422 or 409 with its error code 704, “request already processed or in progress.” Stripe doesn’t save a result for a request that conflicts with one executing concurrently, so the client can simply retry.

You could instead block on the in-progress row and replay when it finishes. But holding connections open while you wait on someone else’s request runs you out of connections. That happens during the very retry storm you were trying to survive. I return the 409 with a Retry-After header.

Failures need a rule too. Stripe saves the result “regardless of whether it succeeds or fails,” including 500s, but only once “the execution of an endpoint begins”. A request that fails parameter validation is not saved and can be retried. I would copy that split.

  • A 400 before any side effect doesn’t need remembering.
  • A 500 after the handler started does, because nobody knows whether the money moved. The client should read the payment’s state before trying again with a new key.

The diagram usually leaves out one state: the process dies while a row is in progress. Without a lease, that key is stuck, and every retry gets a 409 until the TTL cleanup. Add a locked_until column. Or treat in_progress rows older than a threshold as abandoned. Then let one retry take them over with a conditional UPDATE. Too short a threshold reopens the door you were closing.

What to do when the same key arrives with a different body

The same key with a different body is a client bug, and the server should say so rather than guess. The IETF draft recommends 422 for this case. Stripe “compares incoming parameters to those of the original request and errors if they’re not the same to prevent accidental misuse.” Square returns an error telling you the key was used previously.

The fingerprint detects it. Hash method, path and body, store the hash with the key, and compare on every hit.

The draft allows the fingerprint to come from the whole payload or from selected elements. Selected elements matter more than they sound. Say the client puts a timestamp or a trace id in the body. Then every retry has a different fingerprint and gets a 422. Keep volatile fields in headers. Or hash only the fields that define the operation: amount, currency, source, destination, reference.

Decide this before the first client ships. Changing the definition later invalidates every key in flight.

Why keys must be scoped per account

Uniqueness has to be scoped to the caller. The draft leaves the scope to the resource owner, and Adyen documents storing keys “at a company account level.”

A global table is wrong. Two tenants can collide by bad luck, or because a client uses sequential keys. The collision hands tenant B the saved response of tenant A, which leaks data and silently drops a payment. Tenants can’t be expected to coordinate their key spaces.

The scope has to come from authentication, never from the request body. In the middleware below, account_id is a header the auth layer sets after validating the token. A client that could choose its own scope could choose someone else’s. Adyen adds that random UUIDs protect two API credentials under the same account. Neither can read the other’s responses by guessing keys.

How long to keep keys, and where

Keys need to expire, or the table grows forever. Stripe prunes keys once they are “at least 24 hours old” and treats a reused key after pruning as a new request. Adyen’s are “valid for a period of 7 to 14 days after first submission.” The draft says the resource “SHOULD define such expiration policy and publish it in the documentation.”

My rule: the TTL must exceed the longest retry window any client of yours could have. That includes a client that was down over a long weekend. For card payments 24 hours is a floor; for weekly batch payouts, a week.

There are two common stores. One is a Postgres table with a unique constraint, the other Redis with SET key value NX EX seconds.

Postgres unique constraint Redis SET NX
Atomic claim INSERT ... ON CONFLICT DO NOTHING SET ... NX returns nil if the key exists
Same transaction as the ledger write Yes, if the handler shares the connection No, two systems, two commits
Expiry Cleanup job, or partition by day Built in with EX
Durability WAL, survives a restart Depends on persistence settings; a lost key is a possible double charge
Latency Milliseconds Sub-millisecond
Operational cost A table you already back up Another system in the money path

My default for money is Postgres, in the same database as the ledger. Then the key row and the ledger entry can commit together. Redis fits high-volume deduplication where a rare duplicate is tolerable, like analytics events. It also works as a fast first filter in front of Postgres. If you put Redis in the money path, assume a failover can lose recent writes.

A Go middleware on Postgres

The schema the middleware expects:

CREATE TABLE idempotency_keys (
  account_id      text        NOT NULL,
  idempotency_key text        NOT NULL,
  fingerprint     text        NOT NULL,
  status          text        NOT NULL CHECK (status IN ('in_progress', 'done')),
  response_code   int,
  response_body   bytea,
  created_at      timestamptz NOT NULL DEFAULT now(),
  expires_at      timestamptz NOT NULL,
  PRIMARY KEY (account_id, idempotency_key)
);
CREATE INDEX idempotency_keys_expires_at ON idempotency_keys (expires_at);

A cron job runs DELETE FROM idempotency_keys WHERE expires_at < now(). At high volume, partition by day and drop old partitions instead. The middleware uses only database/sql and net/http, and passes go vet on Go 1.24. It works with any Postgres driver behind sql.DB.

package idempotency

import (
	"bytes"
	"context"
	"crypto/sha256"
	"database/sql"
	"encoding/hex"
	"io"
	"net/http"
)

type Store struct{ DB *sql.DB }

type recorder struct {
	http.ResponseWriter
	status int
	body   bytes.Buffer
}

func (r *recorder) WriteHeader(code int)        { r.status = code; r.ResponseWriter.WriteHeader(code) }
func (r *recorder) Write(b []byte) (int, error) { r.body.Write(b); return r.ResponseWriter.Write(b) }

func (s *Store) Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		key := r.Header.Get("Idempotency-Key")
		if key == "" || len(key) > 255 {
			http.Error(w, "Idempotency-Key header required (max 255 chars)", http.StatusBadRequest)
			return
		}
		account := r.Header.Get("X-Account-ID") // set by your auth layer, never trusted from the client
		body, _ := io.ReadAll(r.Body)
		r.Body = io.NopCloser(bytes.NewReader(body))
		sum := sha256.Sum256(append([]byte(r.Method+" "+r.URL.Path+"\n"), body...))
		fp := hex.EncodeToString(sum[:])

		ctx := r.Context()
		res, err := s.DB.ExecContext(ctx, `INSERT INTO idempotency_keys
			(account_id, idempotency_key, fingerprint, status, expires_at)
			VALUES ($1, $2, $3, 'in_progress', now() + interval '24 hours')
			ON CONFLICT (account_id, idempotency_key) DO NOTHING`, account, key, fp)
		if err != nil {
			http.Error(w, "idempotency store unavailable", http.StatusServiceUnavailable)
			return
		}
		if n, _ := res.RowsAffected(); n == 0 { // the key already exists: replay, wait, or reject
			var storedFP, status string
			var code sql.NullInt64
			var resp []byte
			err := s.DB.QueryRowContext(ctx, `SELECT fingerprint, status, response_code, response_body
				FROM idempotency_keys WHERE account_id = $1 AND idempotency_key = $2`, account, key).
				Scan(&storedFP, &status, &code, &resp)
			switch {
			case err != nil:
				http.Error(w, "idempotency store unavailable", http.StatusServiceUnavailable)
			case storedFP != fp:
				http.Error(w, "Idempotency-Key reused with a different request", http.StatusUnprocessableEntity)
			case status == "in_progress":
				w.Header().Set("Retry-After", "1")
				http.Error(w, "request with this Idempotency-Key is still in progress", http.StatusConflict)
			default:
				w.Header().Set("Idempotent-Replayed", "true")
				w.WriteHeader(int(code.Int64))
				w.Write(resp)
			}
			return
		}

		rec := &recorder{ResponseWriter: w, status: http.StatusOK}
		next.ServeHTTP(rec, r)
		// The client may be gone by now; finish the bookkeeping anyway.
		_, _ = s.DB.ExecContext(context.WithoutCancel(ctx), `UPDATE idempotency_keys
			SET status = 'done', response_code = $3, response_body = $4
			WHERE account_id = $1 AND idempotency_key = $2`, account, key, rec.status, rec.body.Bytes())
	})
}

Mount it only on routes that create or move something; GET and DELETE are idempotent by definition.

A few choices in there are deliberate.

  • INSERT ... ON CONFLICT DO NOTHING plus RowsAffected is the entire concurrency story. The database decides who won, and the loser reads the row to pick between replay, 409 and 422.
  • The final UPDATE uses context.WithoutCancel. A client that timed out has already closed the connection. If the request context cancels that update, the row stays in progress forever.
  • The Idempotent-Replayed header lets the access log explain why a customer saw the same payment id twice.

Two things are missing to keep the sketch readable. There is no lease on in-progress rows, so a crash mid-handler leaves a stuck key until cleanup. And the claim and the handler’s own write are separate statements. That leaves a window where the charge has committed but the row still says in progress. A retry in that window gets a safe 409, so money is not at risk. Rung 3 closes that window.

How to carry keys across service boundaries

A real payment touches more than one hop: your public API, your payments service, and the processor. Each hop needs a key. Derive the downstream key from the upstream one, or from a stable identifier such as your payment id. Never generate it fresh per attempt. If your payments service mints a new Stripe idempotency key on every retry, the double charge moves one hop down.

When a hop is a queue, the message id becomes the key, and the pattern has two halves.

  • On the producer side, the outbox pattern writes the business change and an outbox row in one transaction. A relay publishes the outbox rows at least once.
  • On the consumer side, a processed_messages table is keyed by message id. Its row is inserted in the same transaction as the consumer’s own effect. A redelivered message hits the constraint and is acknowledged without doing anything.

I covered the operational side of this in lessons from running Go microservices in production.

Rung 3: commit the key with the ledger entry

The stricter version opens a transaction and inserts the key row. It runs the handler on that same transaction, then commits the key row and the ledger entry together. That is what I would ship for the ledger service. The middleware version is good enough for the API edge in front of it. This rung needs the keys in the same Postgres database as the ledger. Redis would mean two systems and two commits.

An idempotency key still only deduplicates one request. It does nothing about these:

  • a user who presses Pay twice and generates two keys
  • a processor that captures twice on its side
  • a bug that posts a ledger entry through a different code path

Those need three more layers:

  • Business rules, such as at most one open payment per order.
  • Ledger invariants: double-entry postings that net to zero, a unique constraint on (payment_id, entry_type), and balances that can’t go negative.
  • Reconciliation against the processor’s settlement reports, to find whatever slipped past the rest.

When fintech founders ask me where to start, my order is idempotency keys first, because they are cheap. Ledger constraints come second, and reconciliation before launch. The broader version of that conversation is in fractional CTO work with Canadian and US startups.

Which rung fits your system

Rung What it buys What it costs Fits
Plain retries Recovery when a request is lost on the way A double charge when the original succeeded GET and DELETE, idempotent by definition
Idempotency key in middleware One winner per key; replay, 409 or 422 for duplicates A keys table, a TTL cleanup, a fingerprint rule; a window where the row says in progress after the charge committed The API edge
Key committed with the ledger entry, plus ledger constraints Key row and ledger entry commit together; constraints catch duplicates the key can’t see Handler runs on the key’s transaction, in the same database as the ledger The ledger service

Redis SET NX sits beside rung 2 for high-volume deduplication where a rare duplicate is tolerable. If you put it in the money path, assume a failover can lose recent writes.

How to test it

The test that matters fires concurrent duplicates. Use a real Postgres, because ON CONFLICT is the behaviour under test and a mock will happily lie about it.

  1. Wrap a fake handler that counts its calls and sleeps a few hundred milliseconds to widen the race.
  2. Launch twenty goroutines behind a sync.WaitGroup with the same key and body.
  3. Assert that the handler ran once and exactly one response is a 2xx.
  4. Assert that every other response is a 409, or a replay with the same body and the Idempotent-Replayed header.

Then the smaller cases:

  • Same key with a different body returns 422 and never calls the handler.
  • No key returns 400.
  • A key from account A sent by account B is treated as new.
  • An in_progress row with an old timestamp behaves the way your lease policy says.

If you build the transactional version, kill the handler halfway. Confirm neither the key row nor the ledger entry exists. Run the concurrency test with -race -count=50; the failures that matter show up on run thirty.

FAQ

What is an idempotency key?

An idempotency key is a unique value the client attaches to a request. The server uses it to recognize a retry of that exact request. It returns the original result instead of performing the operation again. Stripe, Adyen and Square support one on their payment APIs. The IETF httpapi working group has a draft standardizing the Idempotency-Key header. Without one, a timed-out request that actually succeeded is charged twice on retry.

Who should generate the idempotency key?

The client, at the moment the intent to pay is created. It must be persisted with that intent before the first request goes out. Stripe’s documentation states that the client generates the key and suggests V4 UUIDs. A key generated fresh inside each retry defeats the purpose, because the server sees two different requests.

How long should idempotency keys be stored?

Longer than the longest retry window any of your clients could have. That includes a client that was offline for a day or more. Stripe prunes keys that are at least 24 hours old, and Adyen keeps them valid for 7 to 14 days. So 24 hours is a reasonable floor for card payments, and a week is sensible for batch payouts. Publish the TTL in your API docs, as the IETF draft recommends.

Should idempotency keys be global or per customer?

Per account, with the account taken from authentication rather than from the request. A global key space lets a collision between two tenants return one tenant’s saved response to the other. That leaks data and silently drops a payment. Adyen documents storing keys at the company account level.

Does a database unique constraint make an API idempotent?

Only the claim step. The constraint guarantees that exactly one request wins the key, which stops concurrent duplicates from both doing the work. You still need a fingerprint to reject the same key with a different body. You need a stored response to replay to late duplicates. You also need a status column so in-progress duplicates get a 409. And you need a TTL so the table doesn’t grow forever.

Sources