Idempotency Keys for Money Movement: Preventing Double Charges

An idempotency key is a unique, client-generated value sent with a request so the server can recognize a retry of that same request and return the original result instead of doing the work again. For anything that moves money, it is the difference between a timeout being an annoyance and a timeout being a double charge. This post covers the storage model, the state machine, the edge cases, and a Go middleware on Postgres you can adapt.

Table of Contents

• Why retries cause double charges
• What an idempotency key is and who generates it
• The storage model
• The state machine
• Same key, different body
• Scoping keys per account
• TTL and where to store the keys
• A Go middleware on Postgres
• Idempotency across service boundaries
• Where idempotency keys are not enough
• Testing it
• FAQ

Why retries cause double charges

A payment request can fail in three places, and the client cannot tell them apart. Brandur Leach’s 2017 post for Stripe, Designing robust and predictable APIs with idempotency, lists them: the connection fails before the request reaches the server, the server fails midway and leaves the work in limbo, or the operation succeeds and the connection breaks before the response gets back. From the client’s side, all three look like a timeout.

The client then either gives up, and a charge may have gone through that nobody tells the user about, or retries, and in the third case the card is charged twice. Almost every client retries, because HTTP libraries ship with retries built in. Queues have the same problem in a different transport: a broker that promises at-least-once delivery will redeliver when a consumer crashes after doing the work but before acknowledging it.

Duplicates will arrive. The server has to tell a duplicate from a new request, and that is the whole job of the idempotency key. I first dealt with this as CTO at Levpay, a payments company later acquired by Pagsmile, and I deal with it today on crypto trading systems in Go at Questrade (more on my background). What follows is the textbook version, the one I would hand a new team.

What an idempotency key is and who generates it

The client generates it. Stripe’s idempotent requests documentation says: “A client generates an idempotency key, which is a unique key that the server uses to recognize subsequent retries of the same request.” Stripe suggests V4 UUIDs, caps keys at 255 characters, and tells you to keep sensitive data such as email addresses out of them. The IETF Idempotency-Key header draft from the httpapi working group agrees: a UUID or similar random identifier is RECOMMENDED, and the key “MUST NOT be reused with another request with a different request payload.”

The part people get wrong is when to generate it. A fresh UUID created inside the HTTP call wrapper is useless, because the retry gets a new key and the server sees two different requests. The key has to be created when the intent is created, the moment the user presses Pay or your system decides a payout is due, and persisted with that intent before the first attempt leaves the process, so it is still there after your service restarts.

In practice that is a payment_attempts row carrying a random key, or a key derived from your own identifiers, like payout-{payout_id}-attempt-{n}. Derived keys are readable in logs; random ones leak nothing if they land in a log aggregator you do not fully trust. I lean toward random, but either works as long as it survives your own crashes.

The header name is converging on Idempotency-Key, which the IETF draft specifies and Stripe uses. Adyen uses the same header with a 64-character limit, and Square accepts a key on operations like CreatePayment.

The storage model

Whatever the backing store, each key needs the same fields.

The primary key is (account_id, idempotency_key), and that constraint is the mechanism. Two requests racing to insert the same pair produce exactly one row, and the loser finds out without any lock you had to write.

Store your own response body, never raw upstream payloads. A card PAN or a full bank account number in that column is a second place where the data lives, with a retention policy nobody signed off on.

The state machine

A key moves through three states.

1. New. The row does not exist. The first request to insert it wins and does the work.
2. In progress. The row exists, the handler is running, the response columns are null.
3. Done. The status code and body are saved, and every later request with the same key and fingerprint gets them back.

The interesting case is a duplicate that arrives while the original is still in progress. The IETF draft says the resource “SHOULD reply with an HTTP 409 status code with body containing problem description.” Adyen documents returning 422 or 409 with its error code 704, “request already processed or in progress.” Stripe does not save a result for a request that conflicts with one executing concurrently, so the client can simply retry. You could instead block on the in-progress row and replay when it finishes, but holding connections open while waiting on someone else’s request is how you run out of connections during the retry storm you were trying to survive. I return the 409 with a Retry-After header.

Failures need a rule too. Stripe saves the result “regardless of whether it succeeds or fails,” including 500s, but only once “the execution of an endpoint begins”; a request that fails parameter validation is not saved and can be retried. I would copy that split. A 400 before any side effect does not need remembering. A 500 after the handler started does, because nobody knows whether the money moved, and the client should read the payment’s state before trying again with a new key.

One state the diagram usually leaves out: the process dies while a row is in progress. Without a lease, that key is stuck and every retry gets a 409 until the TTL cleanup. Add a locked_until column, or treat in_progress rows older than a threshold as abandoned and let one retry take them over with a conditional UPDATE. Too short a threshold reopens the door you were closing.

Same key, different body

The same key with a different body is a client bug, and the server should say so rather than guess. The IETF draft recommends 422 for this case. Stripe “compares incoming parameters to those of the original request and errors if they’re not the same to prevent accidental misuse.” Square returns an error telling you the key was used previously.

The fingerprint detects it: hash method, path and body, store the hash with the key, compare on every hit. The draft allows the fingerprint to come from the whole payload or from selected elements, and selected elements matter more than they sound. If the client puts a timestamp or a trace id in the body, every retry has a different fingerprint and gets a 422. Keep volatile fields in headers, or hash only the fields that define the operation: amount, currency, source, destination, reference. Decide this before the first client ships, because changing the definition later invalidates every key in flight.

Scoping keys per account

Uniqueness has to be scoped to the caller. The draft leaves the scope to the resource owner, and Adyen documents storing keys “at a company account level.” A global table is wrong because a collision between two tenants, by bad luck or a client using sequential keys, hands tenant B the saved response of tenant A, which leaks data and silently drops a payment. Tenants cannot be expected to coordinate their key spaces.

The scope has to come from authentication, never from the request body. In the middleware below, account_id is a header the auth layer sets after validating the token. A client that could choose its own scope could choose someone else’s. Adyen adds that random UUIDs stop two API credentials under the same account from reading each other’s responses by guessing keys.

TTL and where to store the keys

Keys need to expire or the table grows forever. Stripe prunes keys once they are “at least 24 hours old” and treats a reused key after pruning as a new request. Adyen’s are “valid for a period of 7 to 14 days after first submission.” The draft says the resource “SHOULD define such expiration policy and publish it in the documentation.” My rule: the TTL must exceed the longest retry window any client of yours could have, including one that was down over a long weekend. For card payments 24 hours is a floor; for weekly batch payouts, a week.

The two common stores are a Postgres table with a unique constraint and Redis with SET key value NX EX seconds.

My default for money is Postgres, in the same database as the ledger, so the key row and the ledger entry can commit together. Redis fits high-volume deduplication where a rare duplicate is tolerable, like analytics events, or a fast first filter in front of Postgres. If you put Redis in the money path, assume a failover can lose recent writes.

A Go middleware on Postgres

The schema the middleware expects:

CREATE TABLE idempotency_keys (
  account_id      text        NOT NULL,
  idempotency_key text        NOT NULL,
  fingerprint     text        NOT NULL,
  status          text        NOT NULL CHECK (status IN ('in_progress', 'done')),
  response_code   int,
  response_body   bytea,
  created_at      timestamptz NOT NULL DEFAULT now(),
  expires_at      timestamptz NOT NULL,
  PRIMARY KEY (account_id, idempotency_key)
);
CREATE INDEX idempotency_keys_expires_at ON idempotency_keys (expires_at);

A cron job runs DELETE FROM idempotency_keys WHERE expires_at < now(); at high volume, partition by day and drop old partitions instead. The middleware itself uses only database/sql and net/http, passes go vet on Go 1.24, and works with any Postgres driver behind sql.DB.

package idempotency

import (
	"bytes"
	"context"
	"crypto/sha256"
	"database/sql"
	"encoding/hex"
	"io"
	"net/http"
)

type Store struct{ DB *sql.DB }

type recorder struct {
	http.ResponseWriter
	status int
	body   bytes.Buffer
}

func (r *recorder) WriteHeader(code int)        { r.status = code; r.ResponseWriter.WriteHeader(code) }
func (r *recorder) Write(b []byte) (int, error) { r.body.Write(b); return r.ResponseWriter.Write(b) }

func (s *Store) Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		key := r.Header.Get("Idempotency-Key")
		if key == "" || len(key) > 255 {
			http.Error(w, "Idempotency-Key header required (max 255 chars)", http.StatusBadRequest)
			return
		}
		account := r.Header.Get("X-Account-ID") // set by your auth layer, never trusted from the client
		body, _ := io.ReadAll(r.Body)
		r.Body = io.NopCloser(bytes.NewReader(body))
		sum := sha256.Sum256(append([]byte(r.Method+" "+r.URL.Path+"\n"), body...))
		fp := hex.EncodeToString(sum[:])

		ctx := r.Context()
		res, err := s.DB.ExecContext(ctx, `INSERT INTO idempotency_keys
			(account_id, idempotency_key, fingerprint, status, expires_at)
			VALUES ($1, $2, $3, 'in_progress', now() + interval '24 hours')
			ON CONFLICT (account_id, idempotency_key) DO NOTHING`, account, key, fp)
		if err != nil {
			http.Error(w, "idempotency store unavailable", http.StatusServiceUnavailable)
			return
		}
		if n, _ := res.RowsAffected(); n == 0 { // the key already exists: replay, wait, or reject
			var storedFP, status string
			var code sql.NullInt64
			var resp []byte
			err := s.DB.QueryRowContext(ctx, `SELECT fingerprint, status, response_code, response_body
				FROM idempotency_keys WHERE account_id = $1 AND idempotency_key = $2`, account, key).
				Scan(&storedFP, &status, &code, &resp)
			switch {
			case err != nil:
				http.Error(w, "idempotency store unavailable", http.StatusServiceUnavailable)
			case storedFP != fp:
				http.Error(w, "Idempotency-Key reused with a different request", http.StatusUnprocessableEntity)
			case status == "in_progress":
				w.Header().Set("Retry-After", "1")
				http.Error(w, "request with this Idempotency-Key is still in progress", http.StatusConflict)
			default:
				w.Header().Set("Idempotent-Replayed", "true")
				w.WriteHeader(int(code.Int64))
				w.Write(resp)
			}
			return
		}

		rec := &recorder{ResponseWriter: w, status: http.StatusOK}
		next.ServeHTTP(rec, r)
		// The client may be gone by now; finish the bookkeeping anyway.
		_, _ = s.DB.ExecContext(context.WithoutCancel(ctx), `UPDATE idempotency_keys
			SET status = 'done', response_code = $3, response_body = $4
			WHERE account_id = $1 AND idempotency_key = $2`, account, key, rec.status, rec.body.Bytes())
	})
}

Mount it only on routes that create or move something; GET and DELETE are idempotent by definition.

A few choices in there are deliberate. INSERT ... ON CONFLICT DO NOTHING plus RowsAffected is the entire concurrency story: the database decides who won, and the loser reads the row to pick between replay, 409 and 422. The final UPDATE uses context.WithoutCancel because a client that timed out has already closed the connection, and if the request context cancels that update the row stays in progress forever. The Idempotent-Replayed header is there so the access log can explain why a customer saw the same payment id twice.

Two things are missing to keep the sketch readable. There is no lease on in-progress rows, so a crash mid-handler leaves a stuck key until cleanup. And the claim and the handler’s own write are separate statements, so there is a window where the charge has committed but the row still says in progress. A retry in that window gets a safe 409, so money is not at risk. A stricter version opens a transaction, inserts the key row, runs the handler on that transaction, and commits the key row and the ledger entry together. That is what I would ship for the ledger service; the middleware version is good enough for the API edge in front of it.

Idempotency across service boundaries

A real payment touches more than one hop: your public API, your payments service, and the processor. Each hop needs a key, and the downstream key must be derived from the upstream one or from a stable identifier such as your payment id, never generated fresh per attempt. If your payments service mints a new Stripe idempotency key on every retry, the double charge moves one hop down.

When a hop is a queue, the message id becomes the key and the pattern has two halves. On the producer side, the outbox pattern writes the business change and an outbox row in one transaction, and a relay publishes the outbox rows at least once. On the consumer side, a processed_messages table keyed by message id is inserted in the same transaction as the consumer’s own effect, so a redelivered message hits the constraint and is acknowledged without doing anything. I covered the operational side of this in lessons from running Go microservices in production.

Where idempotency keys are not enough

An idempotency key deduplicates one request. It does nothing about a user who presses Pay twice and generates two keys, a processor that captures twice on its side, or a bug that posts a ledger entry through a different code path. Those need business rules (at most one open payment per order), ledger invariants (double-entry postings that net to zero, a unique constraint on (payment_id, entry_type), balances that cannot go negative), and reconciliation against the processor’s settlement reports to find whatever slipped past the rest. When fintech founders ask me where to start, my order is idempotency keys first because they are cheap, ledger constraints second, and reconciliation before launch. The broader version of that conversation is in fractional CTO work with Canadian and US startups.

Testing it

The test that matters fires concurrent duplicates. Use a real Postgres, because ON CONFLICT is the behaviour under test and a mock will happily lie about it. Wrap a fake handler that counts its calls and sleeps a few hundred milliseconds to widen the race, launch twenty goroutines behind a sync.WaitGroup with the same key and body, and assert that the handler ran once, exactly one response is a 2xx, and every other response is a 409 or a replay with the same body and the Idempotent-Replayed header.

Then the smaller cases: same key with a different body returns 422 and never calls the handler, no key returns 400, a key from account A sent by account B is treated as new, and an in_progress row with an old timestamp behaves the way your lease policy says. If you build the transactional version, kill the handler halfway and confirm neither the key row nor the ledger entry exists. Run the concurrency test with -race -count=50; the failures that matter show up on run thirty.

FAQ

What is an idempotency key?

An idempotency key is a unique value the client attaches to a request so the server can recognize a retry of that exact request and return the original result instead of performing the operation again. Stripe, Adyen and Square support one on their payment APIs, and the IETF httpapi working group has a draft standardizing the Idempotency-Key header. Without one, a timed-out request that actually succeeded is charged twice on retry.

Who should generate the idempotency key?

The client, at the moment the intent to pay is created, and it must be persisted with that intent before the first request goes out. Stripe’s documentation states that the client generates the key and suggests V4 UUIDs. A key generated fresh inside each retry defeats the purpose because the server sees two different requests.

How long should idempotency keys be stored?

Longer than the longest retry window any of your clients could have, including one that was offline for a day or more. Stripe prunes keys that are at least 24 hours old and Adyen keeps them valid for 7 to 14 days, so 24 hours is a reasonable floor for card payments and a week is sensible for batch payouts. Publish the TTL in your API docs, as the IETF draft recommends.

Should idempotency keys be global or per customer?

Per account, with the account taken from authentication rather than from the request. A global key space lets a collision between two tenants return one tenant’s saved response to the other, which leaks data and silently drops a payment. Adyen documents storing keys at the company account level.

Does a database unique constraint make an API idempotent?

Only the claim step. The constraint guarantees that exactly one request wins the key, which stops concurrent duplicates from both doing the work. You still need a fingerprint to reject the same key with a different body, a stored response to replay to late duplicates, a status column so in-progress duplicates get a 409, and a TTL so the table does not grow forever.