How I Designed a Crypto Payment Engine That Survives Crashes, Retries, and Restarts

A payment does not happen in one clean function call.

It crosses databases, blockchains, liquidity providers, settlement systems, and webhooks. Any one of them can be slow, unavailable, or ambiguous. A process can crash after moving money but before saving its new state. The same webhook can arrive twice. A blockchain event can take time to become final. A provider can accept a request and time out before returning the result.

When I started building Mayarin, a crypto-commerce payment system, I realized that the difficult part was not moving a payment forward. The difficult part was answering a more important question:

If the process stops at any instruction, can it safely continue later?

That question shaped the clearing engine. Instead of treating a payment as a request that should finish during one process lifetime, I treat it as a durable workflow that may be interrupted and replayed many times.

The result rests on three properties:

• Idempotent: repeating a step does not move value twice.
• Resumable: persisted state contains everything needed to continue.
• Auditable: every state transition leaves a durable event.

This post explains how those properties work together and the trade-offs behind them.

A payment is a state machine, not a function

The first design decision was to stop thinking about a payment as something like this:

async function pay() {
  const quote = await getQuote();
  await receiveFunds();
  await swap();
  await settleMerchant();
  await markCompleted();
}

This looks simple, but the control flow exists only in memory. If the process dies after settleMerchant() and before markCompleted(), the next process cannot know what happened by looking at this function.

Mayarin instead represents clearing as an explicit state machine:

CREATED
   ↓
QR_PARSED
   ↓
PRICE_LOCKED
   ↓
PAYMENT_PENDING
   ↓
ASSET_RECEIVED
   ↓
CLEARING
   ↓
SETTLING
   ↓
SETTLED
   ↓
SUCCESS

Any non-terminal state can also move to FAILED with a recorded reason.

The names are less important than the rule behind them: each non-terminal state has one defined forward step. The database state, not the call stack, tells the engine what to do next.

If the process restarts while a transaction is in ASSET_RECEIVED, the engine does not reconstruct a previous request or depend on hidden in-memory context. It loads the transaction and runs the work assigned to ASSET_RECEIVED.

Durable state becomes the program counter.

Advance one durable step at a time

The engine never tries to persist only at the end of the payment. It advances one state at a time and writes each successful transition.

Conceptually, the loop looks like this:

while (!isTerminal(transaction)) {
  const next = await step(transaction);

  if (next === null) {
    return { transaction, waiting: true };
  }

  transaction = next;
}

The step function exhaustively handles the current state. It either:

1. completes that state's work and persists the next state;
2. returns null because it is waiting for the outside world; or
3. throws a retryable or terminal error.

This makes progress observable. It also gives recovery a simple contract: load every non-terminal transaction and ask the same engine to advance it.

There is no separate recovery algorithm. Recovery uses the normal execution path.

The dangerous gap between an effect and a state change

The hardest detail is the ordering between external effects and persistence.

Imagine a settlement step with two operations:

1. ask a provider to pay the merchant;
2. persist SETTLED.

There is no ordinary database transaction that can atomically include both a Postgres update and an external provider call. A crash can always happen between them.

If I persist the new state first, this sequence is possible:

persist SETTLED
        ↓
process crashes
        ↓
merchant was never paid

The system now contains a dangerous lie. It claims the payment settled, so recovery skips the effect that never happened.

Mayarin uses the opposite order:

perform the idempotent effect
        ↓
persist the new state

Now the failure mode becomes:

merchant is paid
        ↓
process crashes before persisting SETTLED
        ↓
recovery repeats the settlement request

Repeating a money-moving call sounds dangerous—and it is, unless the effect is idempotent. That requirement is what makes the ordering safe.

Every repeatable effect receives a stable key derived from the clearing transaction and its current state:

${transactionId}:${state}

The ledger uses it to return the existing posting instead of writing another one. Settlement adapters use it to recognize a replay of the same request. On-chain execution also has its own replay guard: a payment intent can only be consumed successfully once.

With replay-safe effects, a crash after the effect but before persistence becomes recoverable. The next attempt repeats a no-op, then records the state that should have been recorded the first time.

This is an important distinction:

Idempotency is not merely API convenience. It is what closes the crash gap that a database transaction cannot cover.

Not every repeated operation returns the same result

Some side effects cannot be perfectly idempotent. A fresh market quote is an example: repeating the request may produce a different quote.

In the contract payment path, a crash can occur after a quote is signed but before the lock is persisted. On recovery, the engine creates a fresh lock rather than trying to discover the orphaned one.

That is acceptable because the first signed order was never exposed to the payer and was never persisted as part of the payment. It is unreachable, so creating a replacement does not duplicate value movement.

This led me to a more precise rule than “all effects must be idempotent”:

• A repeated value-moving effect must be idempotent.
• A repeated allocation must resolve to the same resource.
• A repeated ephemeral operation may produce a replacement only when the earlier result was never published or made actionable.

The real question is not whether the function returns identical bytes. The question is whether repeating it can create a second externally meaningful outcome.

Waiting is a valid state of progress

Payment systems spend much of their time waiting.

They wait for a customer to send funds, for enough blockchain confirmations, or for a settlement provider to finish. Treating that waiting as an error creates noisy retries and confused failure handling.

In Mayarin, a step can explicitly say, “I cannot move forward yet.” For example:

• PAYMENT_PENDING waits for confirmed funds or a confirmed on-chain payment event.
• SETTLING waits for the settlement provider to report an authoritative terminal status.

The engine returns the current transaction with waiting: true. A watcher, indexer, webhook, scheduled sweep, or manual retry can wake it later.

The wake-up signal does not bypass the state machine. It supplies a fact and asks the same engine to continue.

This is especially important for webhooks. A webhook should not directly mark a payment as settled. It is a signal that something may have changed. The engine wakes up and asks the provider adapter for the authoritative status.

A spoofed, duplicated, or reordered webhook therefore cannot settle a payment by itself.

Retryable errors should preserve the checkpoint

Not every error means the payment failed.

An RPC timeout, temporary database outage, or unavailable settlement provider usually means “try this state again later.” A malformed order or invalid transition means the workflow cannot continue honestly.

The engine distinguishes these cases through a typed error taxonomy. Retryable errors leave the transaction exactly where it is and propagate to the caller. A later recovery pass sees the same durable checkpoint and tries again.

Non-retryable errors move the transaction to FAILED, including a stable error code and diagnostic details.

There is one subtle exception: once a payment reaches SETTLED, the money has already moved. Failing the payment from that point would be dishonest. If the final bookkeeping step encounters a problem, the engine leaves it resumable at SETTLED rather than rewriting financial reality as a failure.

The business meaning of a state matters more than the convenience of a uniform error handler.

State and audit history must agree

Every state transition creates two values:

• the next immutable transaction snapshot;
• an event describing the transition.

The repository must persist them in the same database transaction.

transaction state + transition event = one atomic database write

A state without an event is an audit hole. An event without its state is a lie.

The transaction version also becomes the event sequence number. This gives the history an obvious order and supports optimistic concurrency: an update only succeeds if the stored version is still the one the engine loaded.

If two workers try to advance the same payment concurrently, one wins the version check. The other reloads instead of silently overwriting newer state.

This is where immutability helps. A transition returns a new transaction value rather than mutating the old object. The repository receives both the expected previous version and the new snapshot, making concurrency part of the contract rather than an implementation detail hidden inside storage code.

Recovery is deliberately boring

The recovery method is small:

async function resumeStuck(limit = 100) {
  const transactions = await repository.listResumable(limit);
  const results = [];

  for (const transaction of transactions) {
    results.push(await advance(transaction));
  }

  return results;
}

It loads a bounded number of non-terminal transactions, oldest first, and runs the normal advancement logic.

This simplicity is a consequence of the earlier constraints:

• the current state identifies the next step;
• all required facts are persisted;
• external effects are safe to repeat;
• waiting is represented explicitly;
• concurrent writes are version-checked.

Recovery does not need to guess which line of code ran before the crash. It does not inspect logs to reconstruct control flow. It does not maintain a second set of “recovery states.”

It simply continues from the durable checkpoint.

What I test

The happy path is only a small part of the test suite. The more valuable tests exercise repetition and interruption:

• Starting the same confirmed payment twice returns the same clearing transaction.
• Replaying an asset-received signal does not post to the ledger twice.
• Resuming a completed payment changes nothing.
• A payment waiting for funds remains in PAYMENT_PENDING.
• A pending settlement continues after the provider becomes complete.
• A recovery sweep finishes every resumable transaction and then finds nothing left.
• Every persisted transition has a correctly ordered event.

For external effects, I also test the idempotency boundary itself. It is not enough for the engine to send the same key; the ledger, provider adapter, and smart contract must each enforce their side of the guarantee.

An orchestration layer cannot manufacture exactly-once delivery from unreliable networks. What it can build is an effectively-once business outcome from at-least-once attempts.

Lessons I would carry into any payment system

Building this engine changed how I think about reliable workflows. These are the principles I would reuse even outside crypto:

1. Model the workflow explicitly. If a business process crosses system boundaries, its state should outlive the process executing it.
2. Make the current state sufficient to continue. Recovery should not depend on reconstructing lost in-memory context.
3. Assume every effect can be attempted more than once. Put a stable idempotency key at every money-moving boundary.
4. Do effects before claiming they happened. Then make replay safe enough to close the crash gap.
5. Treat waiting as normal. An external system being unfinished is not the same as failure.
6. Persist state and audit evidence atomically. They describe the same fact and should never disagree.
7. Respect irreversible states. Once money has moved, an internal error cannot make that reality disappear.
8. Use the normal path for recovery. A separate recovery workflow will eventually disagree with the primary one.

Closing thought

Reliable payment infrastructure is not built by preventing every crash. Crashes are inevitable. Networks time out, processes restart, and the same message arrives again.

The goal is to make interruption unremarkable.

When durable state is the control flow, effects are replay-safe, and every transition is auditable, a restart stops being a special incident. It becomes another ordinary attempt to move the payment one honest step forward.