Teardown · payment systems
Failure atlas · fourteen entries

Fourteen ways reconciliation breaks.

Fourteen places where the balance in your dashboard stops matching the balance at your provider and the figure in your accountant's report. For each: what the symptom looks like, and what catches it.

Drift almost never arrives as an outage. It arrives as a question from your accountant: “why are we four hundred and twenty short for March?” And then it turns out the system holds three numbers, each correct in its own way, and none of them can be called the balance.

All fourteen items below are places where two truths diverge legitimately: the provider counts one thing, the network shows another, the accountant needs a third. Without a layer that can explain the difference, the difference accumulates in silence.

Drift in the books is not an arithmetic error. It is the absence of an answer to “where did this figure come from”.

The list comes out of systems I built: a double-entry accounting core working across four networks — Bitcoin, Ethereum, Tron and TON — a payment rail with batch payouts, and a bank statement platform in daily production for three years. Ordered from what you meet every month to what fires once a year and costs the most.

01

More left the account than the payment saidFee on top of the amount, or inside it

Symptom
You sent a hundred and a hundred and three left the account. Or you sent a hundred and ninety-eight arrived. The customer calls it a bug; the system did exactly what it was told.

The same number means different things depending on who pays the fee. Both schemes are legitimate, but the system has to know which one it runs and say so out loud — before sending, not in reply to a complaint.

What catches it
Three separate quantities in the payment model: amount, fee, and the resulting debit. Never display two of the three. A payout batch dry-runs first: every row returns its own fee and its own debit before any money moves.
02

The recipient was paid twiceA retried request creates a second payment

Symptom
Two requests a second apart in the logs. An operator clicked again, or the connection dropped and the client library retried on its own.

Retries are normal in any network exchange, not an exception. A system that creates a new operation for every request it receives is guaranteed to pay twice eventually.

What catches it
An idempotency key from the caller, a unique index on it, and the previous result returned instead of a new one being created. A row's key inside a payout batch does not change between attempts — which is why re-uploading it never produces a second payment. Verified by a test that fires the same request twenty times in parallel and demands exactly one posting.
03

The balance is there and cannot be withdrawnA currency without its network

Symptom
The dashboard says “USDT: 5,000”, the withdrawal fails: only 300 sits on the network the customer wants. They see one number and don't understand the refusal.

The same token lives on several networks, and that is not one asset but several. Adding them into a single number is like adding money in your account to money in another country: you get a figure, it means nothing.

What catches it
The pair “currency plus network” becomes the key everywhere: in the account, in the posting, in the API response, in the exported row. Amount, currency and network travel together and are never separated. Addresses are validated against the format of that specific network before anything is queued.
04

The balance went negative despite the checkAccounting without holds

Symptom
The balance is below zero. There is a “sufficient funds” check in the code and it was running.

Time passes between reading the balance and debiting it. Two concurrent requests read the same value, both see enough, both debit. The check existed; atomicity did not.

What catches it
Funds are reserved as their own object — a hold — and the available balance is the balance minus active holds. Check and reserve happen inside one database transaction. In a payout batch the remaining balance is a running total: rows that no longer fit are flagged in advance rather than failing at dispatch.
05

Daily reports don't add up to the monthly oneThe day boundary and time zones

Symptom
The gap is exactly a handful of transactions, and all of them happened near midnight.

A transaction happens at an instant; a report is built on calendar boundaries. When different parts of the system understand “a day” differently — the server in one zone, the database in another, the accountant in a third — some transactions land in both reports or neither.

What catches it
Every instant is stored in one zone with the zone stated explicitly, the reporting period is an interval rather than a date, and the reporting zone is written into the report itself. Test: a transaction exactly on the boundary must land in precisely one report.
06

Turnover doubled out of nowhereA refund recorded as a new operation

Symptom
The turnover metric doubled with no growth in sales. Refunds were counted as fresh inflows.

A refund can be recorded two ways: as a separate operation in the opposite direction, or as a reversal of the original. The first is correct for money and wrong for turnover — one deal now produces two movements, and any metric that counts movements doubles.

What catches it
A refund references its original operation explicitly and carries its own type. A partial refund holds its own amount and never edits the original record. Turnover is computed over operations of a given type, not over every movement. A database constraint: refunds against an operation cannot exceed its amount.
07

One transaction, three different figuresA rate as of exactly when

Symptom
Last quarter's report built today differs from the same report built a month ago. All three versions are “correct”.

The rate moves between a payment being created, sent and confirmed. If the system takes the rate at the moment the report is built, past transactions change value retroactively — and the books stop being reproducible.

What catches it
The rate is fixed with the posting and stored inside it: not a reference to a table, but the value itself and the instant it was taken. A report for a past period, built today and a year from now, must produce identical numbers. The same requirement is what makes an audit possible.
08

Yesterday's export names the source differentlyA reference table edited retroactively

Symptom
Two exports covering the same period call the same partner by different names. Or a tariff changed and past fees recalculated themselves.

The posting references a table, the table changes, history changes with it. The quietest problem on this list: nothing breaks, the past simply stops holding still.

What catches it
Everything that affects a calculation is captured in the posting at the moment it is created: the tariff rate, the version of the rules, the name of the source. The label can change as often as you like — it is for humans. The key that totals reconcile against never changes, and those are two different fields.
09

A confirmed deposit disappearedFinality, not “confirmations”

Symptom
A deposit was credited and an hour later it isn't on the network. The customer's balance already grew and the money is already spent. Or the reverse: the transaction is visible and never makes it into a block.

Four networks run three different models of finality, and no single threshold covers them. In Bitcoin finality is probabilistic; at zero confirmations the transaction is also replaceable. In Ethereum finality is crypto-economic: before finalized a reorg across a few slots is possible. In Tron and TON finality rests on validator agreement, and waiting ten more blocks does not fix a rollback past the irreversibility boundary.

What catches it
The threshold comes from the network's model, not from a round number: in Bitcoin a count of confirmations, in Ethereum the finalized tag, in Tron block irreversibility, in TON the masterchain signature. Below the threshold the deposit sits on a separate pending account and shows to the customer as waiting. A credit is reversed with an opposing posting, never by deleting.
10

One transaction is not one operationBatches, sweeps and asynchronous transfers

Symptom
The chain watcher saw one transaction and created one posting; there were two hundred recipients inside it. Or the reverse: one payout spawned a chain of messages and part of it bounced back.

The one-transaction-one-operation assumption breaks in three places at once. A batch with many outputs is one transaction and many payments. Moving funds to a central wallet is a network movement with no customer operation behind it. And on TON a token transfer is a chain of messages, any of which can bounce midway.

What catches it
The relationship between operations and transactions is many-to-many, not one-to-one, and that is in the schema from the start. Internal movements carry their own type and never pose as customer activity. For asynchronous transfers, success means a confirmed credit on the recipient's side, not a successful send — until then the operation stays open.
11

Reconciliation fails though every entry is rightInternal movements missing from the equation

Symptom
“Sum of customer balances equals the on-chain holding” never holds, and the gap is different every day.

Not everything that moves money is in the equation. Transfers between customers inside the system never touch the network. But sweeps to the central wallet do, and so does topping addresses up with gas, and so do the fees themselves, and none of those have a customer posting.

What catches it
Postings carry a type, and reconciliation is built on the full equation: the change in on-chain holdings equals external inflows minus external outflows, minus fees, minus gas movements — with internal transfers excluded entirely, though they must sum to zero across the system. Each term is computed separately, so a discrepancy points at where to look.
12

The cost of a payout changed after sendingThe actual fee is not the estimated one

Symptom
The cost of a payout in the report doesn't match what was shown at dispatch. Sometimes by a lot.

The fee at send time is an estimate, not a fact. A transaction can stall and be accelerated, and then the cost of an already-sent payout changes retroactively. On networks with dynamic gas pricing, a gap between expected and actual is ordinary.

What catches it
Estimated and actual fees are two separate fields, and the second is filled from the network after confirmation. The difference isn't hidden — it becomes its own posting, and that posting is what shows the real cost of running payouts. Accelerating a transaction is an event with its own record.
13

The balance exceeds what can be withdrawnDust, minimum reserves and freezes

Symptom
Total system balance is larger than what can actually be sent. The difference matches no single transaction.

Three legitimate causes. Remainders smaller than the network fee cost more to move than they are worth. Some networks require a minimum reserve on an account. And a stablecoin issuer can freeze an address — the funds are there and cannot be moved, and that is neither your mistake nor your choice.

What catches it
Withdrawable balance is a separate quantity, not a synonym for balance. Both appear side by side in the report, and every part of the gap has a name: dust, reserve, frozen. A freeze is recorded as a restriction on the account, not as funds vanishing — a customer needs an explanation, not a quietly smaller number.
14

The payout went out and cost nothingA fee paid in a different asset

Symptom
The report shows a payout with zero cost. Or an asset was debited that has no business being in this operation.

A fee doesn't have to be paid in the asset being transferred. It can come out of the network's base coin, be covered by a resource rented in advance, or be paid by a third party entirely. Books where an operation has exactly one currency cannot record that cost and lose it silently.

What catches it
The fee is its own posting with its own currency and network, linked to the operation but not part of its amount. A resource bought in advance is accounted for as inventory and drawn down as it is consumed. The total cost of a payout is the sum of postings across all currencies, converted at the rates already fixed.
What all fourteen have in common

None of them is about arithmetic

All of them are about a system that at some point could not answer “where did this figure come from” — because it never stored what the answer required.

A quantity is stored together with whatever makes it meaningful.

An amount with its currency and network. A fee separate from the amount, and estimated separate from actual. A rate with the instant it was taken. A tariff with its version. A posting with its type and a link to its source. Your payment provider will not do this for you, and shouldn't — its job is to move money. The books are yours.

And a fifteenth, which everybody already knows. Money in floating point. The error on a single operation is negligible, but at volume with division it becomes visible. Fixed by a decimal type in the database and in the code. It isn't on the list because if that one is still open for you, the other thirteen haven't arrived yet.
If any of this looked familiar

Send me how your reconciliation works.

A diagram, a description in three sentences, or just a list of the systems money moves between. I'll reply in writing with the specific places it will drift and what each one costs to fix. Free, no call, usually within two days. Curious who wrote this? The systems behind it →