How it's built
A payment system is judged by what it does on its worst day. Each guarantee below is paired with the test that proves it: a test that was also checked to fail when the rule is removed.
A ledger service owns every balance. Nothing else can change one.
Every movement is a journal entry whose debits equal its credits. Entries and postings are append-only: the database user the service runs as has no permission to update or delete them.
Proved by a property test that throws random postings at the ledger and requires every account to equal the sum of its postings.
A balance is changed by one conditional statement that refuses to go below zero, so two payments racing for the last yen cannot both win.
Proved by many threads spending the same wallet at once: the successes add up to exactly what was there.
A payment first reserves money, then captures it. A reservation can be released, cancelled before it lands, or left to expire, and each path returns exactly what was held.
Proved by every transition, including the ones that must be refused, and a cancel that arrives before its authorize.
An invariant checker compares balances with postings in one consistent snapshot, on a schedule and on demand from the console.
Proved by injecting each kind of corruption and requiring the checker to name it.
Networks drop responses. Phones retry. People press buttons twice.
Every request that moves money carries a key chosen by the client. The first attempt claims it; every later one gets the first one's answer.
Proved by the same payment sent from many threads at once: one charge, identical responses.
A claim is a lease with a fencing token. If the server holding it dies, another takes over and finishes the same payment, never a second one.
Proved by killing the attempt at each step and retrying with the same key.
When a call to the ledger gets no answer, the payment is not guessed at. It stays in a write-ahead state until the true outcome is found.
Proved by a fault injector that drops requests before and after the ledger applies them.
A code is single-use by a unique key in the database, not by a cache. Looking it up to see the shop and the amount spends nothing.
Proved by a second customer presenting a code that was just paid, in the API tests and again in the browser test.
Each dependency was taken away under load to see what happens.
Anything left half-done is driven forward or cleanly undone by a recovery pass every few seconds.
Proved by the ledger and the payment service each killed twice under load; every payment resolved to one outcome.
Without the idempotency store there is no protection against charging twice, so money-moving requests are refused until it is back.
Proved by freezing DynamoDB for 12 seconds under load.
Kafka being down delays events and fails nothing: an event is a row written with its state change and published afterwards.
Proved by freezing Kafka for 25 seconds: zero failed requests, every event delivered after.
A circuit breaker stops calling a ledger that is not answering, so requests get a quick refusal and not a slow timeout.
Proved by sustained failures opening it, and the ledger's return closing it.
Getting money in and out is where two systems must agree and no transaction spans both.
Each shop is paid what the business day earned it, net of fees and refunds. A payout that fails at the bank is returned to the shop's balance.
Proved by bank outages, declines and lost answers at every step of a payout.
The ledger, the settlement records and the bank's statement are compared, and every kind of disagreement has a name.
Proved by creating each kind of disagreement on purpose and requiring it to be reported.
The intent to charge, and the intent to reverse a charge, are each recorded before the bank is asked.
Proved by a bank that applies a charge and never answers: it is not charged again.
Partial refunds return a proportional share of the fee, worked out cumulatively so rounding never adds up to a yen too many.
Proved by a property test: any sequence of partial refunds that covers a payment returns exactly its fee.
Three kinds of caller, three kinds of credential, one way in.
Customers and the operator carry short-lived RS256 tokens that the edge verifies against published keys. A wallet's device holds a secret that renews its token, stored server-side only as a hash.
Proved by forged, edited and promoted tokens all being refused at the edge.
A till signs every request with HMAC-SHA256 over the method, path, time, a nonce and the body. A captured request cannot be replayed or altered.
Proved by replaying a request, changing its amount, and sending it late.
On Kubernetes, network policies admit only the traffic the design calls for. Identity is set by the edge and never read from a request body.
Proved by probing every service from inside the cluster: only the edge answers.
Pages render data as text, carry a strict content security policy, and keep secrets out of any URL a server sees.
Proved by a shop named like a script tag: the name is shown, nothing runs.
Every transaction with filters and paging, failures with their reasons, stuck work, ledger violations, mismatches, merchants and settlement.
A dashboard that answers, in order: is the money right, are payments fast, where is the backlog. Eleven alerts, and a runbook for each kind of failure.
One trace per payment, from the request through the ledger and Kafka to settlement, carried across the outbox by the event row itself.
A Helm chart with non-root, read-only containers, disruption budgets, per-service credentials and network policies, checked on a local cluster.
An optional model scores each payment in real time and can decline it. If the model is slow or down, payments go on without it.
Activity and payment lists page by position, so a payment arriving mid-scroll neither repeats nor hides a row.