DEV Community: Sundarapandy

Duplicate Withdrawals in Crypto Exchanges: How Idempotency Prevents Double Payouts on Retry

Sundarapandy — Mon, 26 Oct 2026 09:52:18 +0000

Imagine a user requests a cryptocurrency withdrawal from an exchange. The backend processes the request and broadcasts the transaction to the blockchain, but the API response times out. Assuming the withdrawal failed, the user clicks “Retry.” If the exchange creates a new transaction without checking the original request, the same withdrawal could be processed twice.

This is a critical problem for crypto exchanges, where duplicate payouts can cause financial losses, inconsistent wallet balances, and difficult reconciliation processes. The challenge is that a failed API response does not necessarily mean the underlying transaction failed.

Keeping duplicate withdrawals from slipping through in crypto exchanges isn’t a one-and-done fix. It calls for safeguards across the whole transaction from start to finish. Idempotency keys stop the same withdrawal from firing off twice through the API, and solid database rules plus atomic fund reservations guard against multiple requests arriving at the same time.

Idempotency is a key part of this solution. Combined with database constraints, atomic fund reservations, and blockchain transaction tracking, it allows an exchange to handle retries without creating duplicate withdrawal operations

How Duplicate Crypto Withdrawals Happen in Production

Duplicate withdrawals can occur at several stages of the withdrawal lifecycle.

API timeouts after successful processing

A server may process a withdrawal successfully but fail to return a response before the client times out. The client cannot determine whether the operation completed, so it submits another request.

Blockchain broadcast failures

An exchange sends a signed transaction to an RPC provider but does not receive confirmation that the submission succeeded. Retrying with a newly created transaction may result in a second payout if the original transaction was already accepted.

Concurrent requests

A user double-clicks the withdrawal button, or two requests arrive almost simultaneously. If both requests pass validation before either is recorded, the backend may process them independently.

Automatic retries and worker failures

HTTP clients, API gateways, and background workers can retry operations after timeouts or crashes. A worker might broadcast a transaction and crash before saving its status, leaving the next worker unaware that the transaction may already exist.

These scenarios share one underlying problem: the system cannot reliably distinguish a repeated attempt at the same withdrawal from a genuinely new withdrawal request.

Why Traditional Retry Logic Is Not Enough

Retries improve reliability, but blindly repeating financial operations introduces risk.

An HTTP timeout only indicates that the caller did not receive a response within the expected period. It does not establish whether the server processed the request or whether the blockchain accepted the transaction.

Similarly, checking the user's available balance is insufficient when multiple requests execute concurrently. Two requests could both observe the same balance and attempt to withdraw the same funds.

Database transactions also cannot automatically guarantee that an external blockchain operation and an internal database update succeed together. They are separate systems with different failure conditions.

A safe withdrawal architecture must therefore distinguish three outcomes:

Request failure: The API request did not complete as expected.
Broadcast uncertainty: The exchange cannot determine whether a transaction was submitted.
Settlement failure: The transaction failed according to the relevant blockchain or business rules.

Each outcome requires different recovery behavior. Treating every timeout as a failed withdrawal can turn a temporary network problem into a duplicate payout.

Implement Idempotency Keys for Withdrawal APIs

Idempotency handles repeated submissions of the same logical operation as if only one operation was ever submitted.

Generate and store an idempotency key

A unique key is generated for each intended operation (e.g. withdrawal) by the client.
The backend stores this key together with the corresponding withdrawal operation for the respective user or account. When a client initiates a withdrawal, it supplies a unique key identifying that intended operation.

For example:

POST /api/v1/withdrawals
Idempotency-Key: wd_8f2a91c7
Content-Type: application/json
{
  "asset": "USDT",
  "network": "Ethereum",
  "destination": "0xRecipientAddress",
  "amount": "100.00"
}

If the client retries because the response was lost, it must reuse the same key.
The backend should return the existing withdrawal record or its current status instead of creating a second withdrawal.

Validate repeated requests

An idempotency key must not allow different withdrawal instructions to be treated as the same operation. Store a request fingerprint covering relevant fields, including the asset, network, destination, and amount.

If the same key arrives with different parameters, reject the request rather than silently changing the existing withdrawal.

The backend should also enforce uniqueness for the appropriate key scope, such as the user account and idempotency key combination.

Most importantly, idempotency must be enforced server-side. Generating a key in the frontend alone does not prevent duplicate processing

Prevent Race Conditions With Atomic Database Operations

Idempotency keys work best when supported by database-level safeguards.

Enforce uniqueness

Create a unique constraint on the idempotency key and its scope so duplicates can’t spawn multiple withdrawals for the same context. If two requests race to do the same action, only one will end up creating the record. The other one should gracefully fetch the existing record and return its current status.

Reserve funds atomically

The exchange should validate the available balance and reserve the withdrawal amount within a controlled database transaction.

For example, a withdrawal of 100 USDT should reduce the amount available for other withdrawals before asynchronous processing begins. This prevents concurrent requests from reserving the same funds.

The reservation and withdrawal record must be persisted consistently. If the database transaction fails, the system should not leave a partially created withdrawal.

Funds should only be released when the system has established that the withdrawal cannot still complete. An uncertain blockchain outcome requires investigation rather than immediate release.

Use conditional state transitions

Workers should update a withdrawal only when its current state permits the transition. Conditional updates, row-level locks, or optimistic concurrency controls can prevent multiple workers from claiming the same operation.

These safeguards ensure that duplicate requests cannot bypass the business rules simply because they arrive at the same time.

Design a Retry-Safe Withdrawal State Machine

A withdrawal state machine makes recovery behavior explicit.

A typical lifecycle might include:

REQUEST_RECEIVED: Validate the request and idempotency key.
FUNDS_RESERVED: Reserve the amount and persist the withdrawal.
BROADCAST_PENDING: Prepare and submit the signed transaction.
CONFIRMATION_TRACKING: Monitor the transaction and reconcile its status.
CONFIRMED: Finalize the withdrawal according to the exchange's confirmation policy.
FAILED or MANUAL_REVIEW: Handle definitive failures or unresolved outcomes appropriately.

Each transition should have clear conditions. For example, a withdrawal should not return to its initial state simply because a worker times out.

The state machine must also distinguish a transaction that has never been submitted from one whose submission result is unknown. The latter should trigger transaction recovery and reconciliation, not automatic creation of a new payment.

The exact states and finality requirements depend on the blockchain network and the exchange's custody architecture.

Handle Blockchain Broadcast Retries Without Sending Another Payment

The toughest recovery happens when a transaction might’ve made it onto the blockchain, but the exchange never gets a solid confirmation.

Persist transaction data before broadcasting

Before sending the signed transaction, its computed transaction hash, and its corresponding withdrawal ID to the RPC provider, if the transaction architecture allows it.

This gives the system a durable reference for recovery if a worker crashes or a network connection fails.

Check the original transaction before retrying

For EVM-compatible networks, the eth_sendRawTransaction JSON-RPC method accepts a signed transaction. If the submission response is lost, the exchange should investigate the original transaction hash and relevant blockchain state before preparing another transaction.

If the original signed transaction is available, resubmitting that same transaction is different from creating a new payment with a different nonce or transaction payload.

Errors such as “already known” or “nonce too low” should trigger investigation. They do not, by themselves, prove that the intended withdrawal has completed successfully.

Handle transaction replacements carefully

A pending transaction may sometimes require a fee replacement. However, the replacement must remain linked to the same logical withdrawal and follow the exchange's transaction-management rules.

The goal is to replace or recover the original transaction where appropriate, rather than creating an extra transfer.

These principles must be adapted to each chain. Account-based networks, UTXO-based networks, and third-party custody providers have different transaction models and recovery mechanisms.

Make Background Workers and Message Queues Idempotent

Withdrawal processing often involves queues and asynchronous workers. Because jobs may be delivered more than once, consumers must be designed to handle repeated execution safely.

A transactional outbox can help coordinate database changes with job publication. The withdrawal record and outbox event are written in the same database transaction, allowing a publisher to deliver the event later without losing the operation.

Employees should utilize atomic claims, conditional updates, or leases to coordinate processing.

They should also record broadcast attempts and outcomes so that another worker can recover an interrupted operation.
Consider a worker that broadcasts a transaction and crashes before updating the database.

When the job is delivered again, the next worker should retrieve the original transaction details and reconcile its status rather than generate a fresh payment.

Queue deduplication is useful, but it is not sufficient by itself. A queue cannot guarantee that an external blockchain transaction was executed only once. The withdrawal service must enforce its own idempotency rules.

Maintain Ledger Consistency and Reconcile Withdrawals

Stopping duplicate requests is only one piece of keeping finances honest. The exchange also has to make sure its internal ledger lines up with how withdrawals actually happen.

Each logical withdrawal should have a traceable relationship with its ledger entries, fund reservation, transaction hash, and settlement status.

The system should track available, reserved, and settled balances separately. This makes it easier to determine whether funds are awaiting broadcast, associated with a pending transaction, or ready for final settlement.

Reconciliation processes should compare internal withdrawal records with blockchain transaction data. They should identify missing transactions, unexpected transaction hashes, inconsistent statuses, and mismatched ledger entries.

A missing transaction receipt does not necessarily mean a transaction was never broadcast. Depending on the chain and circumstances, the transaction may still be pending, dropped, replaced, or affected by a reorganization.

When the outcome remains uncertain, the exchange should preserve the relevant records and escalate the case for further reconciliation instead of automatically releasing funds or initiating another payout.

Test Duplicate Withdrawal Prevention Before Production

Testing should cover failure scenarios across the API, database, workers, and blockchain integration.

Use integration tests and failure injection to simulate crashes immediately before and after database commits, job publication, and transaction broadcasting.

Monitor operational metrics such as idempotency conflicts, repeated broadcast attempts, unresolved withdrawals, reconciliation mismatches, and duplicate ledger operations.

These checks help identify weaknesses before they affect customer funds.

Conclusion

The uniqueness of requests is only one aspect of financial integrity. The exchange must also verify that its internal ledger matches the actual withdrawal journey.

A solid state machine, saved transaction data, and background workers that retry safely, along with reconciliation against the blockchain, keep recovery safer when the network hiccups or a worker crashes and the withdrawal outcome isn’t clear.

Bottom line: a missing response isn’t proof that the withdrawal failed. The system should first figure out what happened to the original operation before deciding what to do next.

If your withdrawal API times out immediately after a transaction is broadcast, can your system prove that retrying the request will not send the funds twice?

Why EVM Smart Contract Integrations Fail When ABI Definitions Drift Between Deployments

Sundarapandy — Tue, 20 Oct 2026 05:20:04 +0000

A smart contract can be deployed successfully, its address can be correct, and the blockchain can be operating normally, yet an application may still fail to interact with it. One possible cause is an ABI mismatch between the contract deployed on-chain and the interface used by the application.

In EVM-compatible blockchain systems, applications rely on Application Binary Interface (ABI) definitions to encode function calls, interpret return values, and decode emitted events. When a contract interface changes but the application continues using an outdated ABI, integration failures can occur across frontends, backend services, and blockchain indexers.

These problems often appear after contract upgrades, redeployments, or changes to shared development artifacts. Understanding how ABI drift occurs, how it affects contract interactions, and how to prevent it is essential for maintaining reliable Web3 applications.

What an ABI Does in EVM Smart Contract Integrations

An ABI defines how external applications communicate with a compiled smart contract. It describes available functions, their input and output types, and event definitions that applications can use to interpret blockchain logs.

Unlike a contract's source code, an ABI does not contain the full implementation logic. Instead, it provides the interface required to interact with the contract.

For example, consider a contract function:
function deposit(uint256 amount) external;

An application using the corresponding ABI can encode a call to deposit with a specified amount. The encoded calldata includes a four-byte function selector followed by the ABI-encoded arguments.

When the transaction reaches the contract, the EVM executes the relevant function if the selector and call conditions are valid.

Applications also use ABIs to decode returned data and interpret events. If an exchange backend tracks token deposits, for instance, its indexer may depend on the correct event ABI to identify the sender, token amount, and other relevant information.

A mismatch between the expected interface and the deployed contract can therefore affect more than a single function call.

What ABI Drift Means Between Contract Deployments

ABI drift occurs when an application uses an ABI that no longer matches the interface it needs to interact with.

This can happen when developers change function signatures, modify parameter types, alter event definitions, or introduce a new contract version without updating dependent applications.

However, redeploying a contract does not automatically cause ABI drift. If the new deployment preserves the same interface and the application uses a compatible ABI, the integration may continue working normally.

The problem emerges when the application and contract evolve independently.

For example, a development team might deploy an updated contract to a test environment, generate a new ABI, and update the frontend. Meanwhile, the backend service continues using an older ABI bundled into its release. Both components may work independently during development, but their behavior can diverge in production.

Proxy-based contracts introduce additional considerations. A proxy may retain its address while its implementation changes. Applications must account for the implementation's current interface and any compatibility requirements rather than assuming that an unchanged address guarantees an unchanged contract interface.

Common Reasons Smart Contract ABI Mismatches Occur

Several development and deployment practices can introduce ABI inconsistencies.

1. Function Signatures Change

A function signature includes the function name and its parameter types. Changing either can change the function selector used to identify the call.

For example:
function withdraw(uint256 amount) external;

Changing the interface to:
function withdraw(uint256 amount, address recipient) external;

creates a different function signature and selector.

If an application continues encoding calls using the original ABI, it generates calldata for the old function. If that selector is unavailable at the target contract, the call may revert or be handled by the contract's fallback behavior.

2. Parameter and Return Types Change

Changing a parameter from one type to another, or restructuring a tuple, can affect how calldata is encoded and decoded.

Return-value changes can also create problems. An application expecting a particular output structure may fail to decode the response correctly when the deployed interface differs.

Developers should distinguish interface incompatibility from changes to internal implementation logic. A function's internal behavior can change without changing its ABI, although the application may still encounter new runtime errors.

3. Event Definitions Change

Blockchain indexers and monitoring services frequently rely on contract events to track deposits, withdrawals, trades, and other operations.

Suppose a contract changes an event's parameter types or indexed status. An indexer using the previous ABI may interpret the logs incorrectly or fail to decode them.

This is particularly important because event decoding errors do not always cause the original transaction to fail. The transaction can succeed on-chain while an off-chain service fails to recognize the resulting event.

4. Multiple Contract Versions Exist

Different environments may run different contract versions, and individual services may use separate ABI artifacts.

A frontend might target a newly deployed contract, while a backend continues interacting with the previous address. An indexer may also retain old event definitions.

Without a reliable record of contract addresses, chain IDs, versions, and ABI artifacts, diagnosing these inconsistencies becomes difficult.

How ABI Drift Causes Production Failures

The effects depend on the type of mismatch and the contract's implementation.

Failed function calls: If the application generates calldata for a function selector that the target contract does not support, the transaction may revert or be handled by a fallback function.

Transaction reverts: A call can reach a valid function but still fail because the updated implementation has different access controls, validation rules, or execution requirements. This is not necessarily an ABI mismatch, so developers must investigate the actual revert reason.

Incorrect return-value decoding: An application using an incompatible output definition may misinterpret returned data or encounter a decoding error.

Missing or incorrectly decoded events: An indexer may fail to process logs correctly, leaving an exchange dashboard, accounting service, or analytics system with incomplete data.

Application-level inconsistencies: A transaction may succeed while an off-chain service processes the result incorrectly. This can create discrepancies between on-chain activity and application records.

These failure modes demonstrate why checking only transaction status is insufficient. Reliable integrations must validate both on-chain execution and off-chain data processing.

Practical Example: An Integration Breaks After a Contract Upgrade

Consider a Web3 application that allows users to withdraw tokens from a smart contract.

The original contract interface is:
function withdraw(uint256 amount) external;

The backend generates calldata using this ABI and submits transactions to the deployed contract.

A later contract version introduces a recipient parameter:
function withdraw( uint256 amount, address recipient ) external;

The new interface allows withdrawals to specify a destination address. However, the backend still uses the old ABI.

When the backend calls withdraw(amount), it generates calldata containing the old function selector. If the target contract does not expose that function and has no compatible fallback behavior, the call fails.

The debugging process should begin by verifying the target chain and contract address. Developers can then compare the ABI used by the backend with the ABI generated for the intended deployment.

Inspecting the transaction's input data helps confirm which function selector was submitted. Transaction receipts, revert information, and execution traces can provide further evidence of the failure.

The deployment mechanism matters here. Deploying a new contract at a different address does not change the interface of the original contract. The failure occurs if the application points to the new deployment while continuing to use an incompatible ABI, or if an upgrade changes the interface at an existing address.

Code-Level Debugging: Detecting an ABI Mismatch with ethers.js

Consider an EVM application that interacts with a token vault contract. The contract was initially deployed with a withdraw(uint256) function, but a newer version introduced a recipient parameter.

The backend still uses the old ABI, so it continues generating calls for the original function signature.

Step 1: Define the Updated Contract Interface

The updated contract exposes this function:

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.20;

contract TokenVault {
    function withdraw(
        uint256 amount,
        address recipient
    ) external {
        // Withdrawal logic omitted
    }
}

For this example, assume the contract does not implement a fallback function that handles the old selector.

Step 2: Reproduce the Failure Using an Outdated ABI

The backend still contains the original interface:

const { ethers } = require("ethers");

const provider = new ethers.JsonRpcProvider(
  process.env.RPC_URL
);

const vaultAddress = process.env.VAULT_ADDRESS;

const oldAbi = [
  "function withdraw(uint256 amount)"
];

const vault = new ethers.Contract(
  vaultAddress,
  oldAbi,
  provider
);

async function testWithdraw() {
  try {
    const data = vault.interface.encodeFunctionData(
      "withdraw",
      [ethers.parseEther("1")]
    );

    console.log("Encoded calldata:", data);

    await provider.call({
      to: vaultAddress,
      data
    });

    console.log("Call succeeded");
  } catch (error) {
    console.error(
      "Contract call failed:",
      error.message
    );
  }
}

testWithdraw();

The code successfully encodes the function call because the old ABI is internally valid. However, encoding a call does not confirm that the deployed contract supports the requested function.

When the call reaches the updated contract, the old function selector is not recognized. Assuming no compatible fallback behavior, execution reverts.

This distinction matters: ABI encoding can succeed locally even when the resulting call is incompatible with the deployed contract.

Step 3: Compare the Function Selectors

Developers can calculate the selectors from both interfaces to identify the mismatch.

const oldInterface = new ethers.Interface([
  "function withdraw(uint256 amount)"
]);

const newInterface = new ethers.Interface([
  "function withdraw(uint256 amount, address recipient)"
]);

console.log(
  "Old selector:",
  oldInterface.getFunction("withdraw").selector
);

console.log(
  "New selector:",
  newInterface.getFunction(
    "withdraw(uint256,address)"
  ).selector
);

The two selectors will differ because the function signatures differ. The old selector is derived from withdraw(uint256), while the updated selector is derived from withdraw(uint256,address).

This confirms that the backend is encoding a different function call from the one exposed by the updated interface.

Step 4: Correct the ABI and Update the Call

Replace the outdated ABI and provide both required arguments

const updatedAbi = [
  "function withdraw(uint256 amount, address recipient)"
];

const vault = new ethers.Contract(
  vaultAddress,
  updatedAbi,
  provider
);

async function testUpdatedWithdraw() {
  const recipient =
    "0x1111111111111111111111111111111111111111";

  const data = vault.interface.encodeFunctionData(
    "withdraw",
    [
      ethers.parseEther("1"),
      recipient
    ]
  );

  console.log("Updated calldata:", data);

  try {
    await provider.call({
      to: vaultAddress,
      data
    });

    console.log("Call completed successfully");
  } catch (error) {
    console.error("Call failed:", error.message);
  }
}

This fixes the ABI mismatch in the example, provided the address points to the expected deployment and the contract's execution conditions are satisfied. The sample recipient is illustrative; a real application must validate the intended destination.

Because the example uses a read-only provider.call, it simulates execution rather than submitting a withdrawal transaction. A production withdrawal would require a signer and appropriate transaction handling.

What Developers Should Take Away

This debugging example demonstrates three distinct checks:

ABI validity: Can the application encode the requested function call?
Interface compatibility: Does the target contract recognize the encoded selector?
Execution validity: Does the function execute successfully under the contract's access controls and business rules?

Passing the first check does not ensure that the other two

For production systems, keep ABI artifacts tied to deployment versions and test contract interactions against the exact network and address before releasing dependent services.

How to Diagnose ABI Drift in an EVM Application

A structured debugging methodology helps isolate ABI incompatibilities and unrelated contract issues.

Step 1: Verify the Contract Address and Chain

Confirm that the application is connected to the intended EVM network and contract address. An otherwise correct ABI will not fix a call sent to the wrong deployment.

Step 2: Compare ABI Artifacts

Compare the ABI bundled into the application with the artifact generated from the intended contract version.

Look for changed function signatures, parameter types, return definitions, and event parameters. Version-control diffs can make these changes easier to identify.

Step 3: Inspect the Function Selector

For a standard external function call, the first four bytes of the calldata identify the function selector.

Compare the selector against the expected signature and verify whether the target contract supports that function. For proxy systems, inspect the relevant implementation and dispatch behavior as well.

Step 4: Examine Transaction Results

Review the transaction receipt, revert data, and execution trace where available. A decoding error in the application may occur before a transaction is submitted, while a contract-level failure requires investigation of on-chain execution.

Step 5: Validate Event Processing

If transactions succeed but application data is missing, check whether the indexer uses the correct event ABI. Review event signatures, indexed parameters, and the log-decoding pipeline.

Step 6: Reproduce the Failure

Run the same call against the intended chain and contract address using a controlled test environment. Reproduction helps determine whether the problem comes from the ABI, deployment configuration, contract logic, or application code.

How to Prevent ABI Drift Across Deployments

Preventing ABI mismatches requires treating contract interfaces as versioned deployment artifacts rather than files that can be copied between services without verification.

Version ABI Artifacts

Generate ABIs from reproducible contract builds and store them with the corresponding build artifacts. Associate each deployment with its chain ID, address, contract version, and ABI reference.

This makes it easier to identify which interface a particular service should use.

Automate Compatibility Checks

Add ABI comparison checks to the continuous integration pipeline. Flag removed functions, changed parameter types, altered return definitions, and event changes.

Not every ABI change is incompatible, so checks should distinguish between compatible additions and changes that require dependent applications to be updated.

Maintain a Deployment Registry

Keep deployment information in a structured manifest or registry. Include the network, contract address, implementation address where relevant, version, and artifact reference.

Applications can then select the correct ABI for the deployment they target rather than relying on manually maintained configuration.

Test Contract Integrations Before Release

Run integration tests against the intended contract version. Test function calls, return-value decoding, emitted events, access controls, and expected revert conditions.

For upgradeable contracts, include tests for the upgrade process and verify that dependent applications remain compatible.

Coordinate Dependent Services

A contract change can affect frontends, backend services, indexers, monitoring systems, and analytics pipelines.

Identify every consumer of the changed interface before deployment. Plan the release sequence, update dependent services, and monitor errors immediately after rollout.

Where multiple versions must remain supported temporarily, make that compatibility explicit instead of assuming every service will update simultaneously.

Production Checklist for Every Contract Deployment

Before releasing a contract change, verify the following:

Confirm the chain ID and target contract address.
Generate the ABI from the intended contract build.
Review ABI differences against the previous version.
Run integration tests for functions, return values, and events.
Verify proxy implementation details where applicable.
Update dependent applications and indexers.
Monitor failed transactions and decoding errors after deployment.
Retain previous ABI artifacts and rollback information.

These checks help teams detect integration problems before they affect users and production workflows.

Conclusion

Smart contract ABI drift is a common source of integration risk when contract interfaces and application code evolve at different speeds. A deployment can succeed while frontends, backend services, or indexers continue using outdated definitions.

The solution is not simply to replace an ABI file whenever a contract changes. Developers need reproducible build artifacts, version-aware deployment records, automated compatibility checks, and integration tests that cover both contract execution and off-chain event processing.

Before your next EVM contract deployment, ask one question: Can every application and service identify the exact contract interface it is expected to use?

Making that verification part of the release process can prevent avoidable failures and keep blockchain integrations consistent as contracts evolve.

What Happens When Two Transactions From the Same Wallet Use the Same Nonce?

Sundarapandy — Wed, 14 Oct 2026 06:16:58 +0000

Imagine a backend service sending transactions from the same Ethereum wallet. The first transaction is still pending when the application creates another one. Because the application has not updated its nonce correctly, both transactions are assigned the same nonce.

Now what happens?

Can both transactions execute? Does one replace the other? Why might the second transaction return a nonce too low or replacement transaction underpriced error?

These situations are common when applications submit multiple EVM transactions concurrently. Understanding how nonce management works is therefore important for developers building wallets, trading systems, DeFi applications, payment platforms, and blockchain automation services.

What Is a Nonce in an EVM Transaction?

A nonce is a sequence number associated with an account. For transactions sent from the same Ethereum or EVM-compatible account, the nonce determines their transaction order.

For example, a wallet might have transactions like this:
Nonce 10 → Transaction A
Nonce 11 → Transaction B
Nonce 12 → Transaction C

The transaction with nonce 10 needs to be processed before the transaction with nonce 11, and so on.

The nonce is not a global blockchain counter. It belongs to the individual sender account. This means two different wallets can have the same nonce without creating a conflict.

The problem occurs when two transactions from the same wallet use the same nonce.

How Can Two Transactions Get the Same Nonce?

A nonce conflict can happen when multiple transactions are generated almost simultaneously.
Consider an application that reads the wallet's current transaction count:

Current nonce = 15

The application creates Transaction A:

Transaction A → Nonce 15

Before Transaction A is confirmed, another process asks for the wallet's nonce. If the application does not account for the pending transaction, it may create:

Transaction B → Nonce 15

Now both transactions are competing for the same nonce.

This can happen in several ways:

Multiple backend workers send transactions from the same wallet.
Several API requests trigger blockchain transactions at nearly the same time.
The application tracks only confirmed transactions.
Nonce state is stored locally but is not synchronized between workers.
An RPC node provides a view of the account state that does not include a transaction already being managed elsewhere.

For applications sending transactions concurrently, nonce assignment needs to be treated as a coordinated process rather than something every worker calculates independently.

What Happens When Both Transactions Enter the Mempool?

When a transaction is submitted, it is propagated to nodes and may remain pending before being included in a block.

If two transactions from the same account use nonce 15, they cannot both represent separate sequential positions in the account's transaction sequence.

For example:

Transaction A
Sender: 0xABC...
Nonce: 15

Transaction B
Sender: 0xABC...
Nonce: 15

The transactions are effectively competing for the same nonce position.

EVM clients have transaction-pool rules for handling this situation. Depending on the client and transaction conditions, a second transaction may be rejected, or it may be considered as a replacement candidate for the first transaction.

This is where transaction fees become important.

A replacement transaction generally needs to satisfy the node's replacement rules, which commonly require a sufficient fee increase over the transaction already occupying that nonce in the local transaction pool.

That is why submitting another transaction with the same nonce does not automatically mean the first one will be replaced.

Can Both Transactions Be Confirmed?

No. Two transactions from the same account cannot both be included in the canonical chain using the same nonce.

Suppose the following transactions exist:

Transaction A → Nonce 15 → Send 1 ETH
Transaction B → Nonce 15 → Call Smart Contract

If Transaction A is included in the canonical chain, nonce 15 has been consumed by that account.

Transaction B cannot subsequently execute as another transaction with nonce 15.

The result can therefore look like:

Transaction A → Nonce 15 → Confirmed
Transaction B → Nonce 15 → Not executable as a second canonical transaction

This sequencing mechanism is important because it prevents multiple conflicting transactions from the same account from occupying the same transaction position.

The Replacement Transaction Scenario

There is an important distinction between an accidental duplicate nonce and an intentional replacement.

Suppose a user sends a transaction with nonce 15, but it remains pending:

Nonce: 15
Fee: Lower
Status: Pending

The user may submit another transaction using the same nonce with a higher fee:

Nonce: 15
Fee: Higher
Status: Pending

The purpose is to give the replacement transaction a better chance of being accepted by nodes and included in a block.

Wallet applications may expose this behavior through features such as speed up or transaction replacement.

A replacement does not mean both transactions will execute. The nonce still represents one transaction position.

If the replacement is accepted and eventually included in the canonical chain, the original transaction cannot later execute using that same nonce.

This is why developers should think of nonce replacement as a competition for one transaction slot rather than two independent transactions.

What Developers See When Nonce Management Goes Wrong

Nonce conflicts often become visible through RPC or wallet errors.
nonce too low

This generally indicates that the submitted nonce is already below the account's expected transaction sequence. For example, another transaction may already have consumed that nonce.

replacement transaction underpriced

This can occur when an application submits a transaction with the same nonce as an existing pending transaction but does not increase the fee enough to satisfy the node's replacement requirements.

already known

A node may return this when the transaction being submitted is already known to its transaction pool.

There can also be less obvious symptoms.

A transaction may remain pending because a later transaction depends on an earlier nonce. For example:

Nonce 20 → Pending
Nonce 21 → Waiting
Nonce 22 → Waiting

Even though transactions 21 and 22 have already been created, the account's transaction sequence cannot simply skip nonce 20.

This is one reason nonce problems can appear as broader transaction-processing failures in production systems.

How Applications Prevent Nonce Conflicts

For a simple wallet sending occasional transactions, nonce management may appear straightforward. It becomes much more important when several application processes can send transactions from the same account.

Track Pending Transactions

Applications should account for transactions that have already been submitted but have not yet been confirmed.

Looking only at the latest confirmed transaction can cause multiple workers to select the same next nonce.

Use a Nonce Manager

A dedicated nonce manager can assign each outgoing transaction a unique sequence:

Transaction Request
  ↓
Nonce Manager
  ↓
Nonce 25 assigned
  ↓
Transaction signed
  ↓
RPC submission

The next request receives nonce 26, even if transaction 25 is still pending

Serialize Transaction Creation

A transaction queue can prevent several workers from assigning the same nonce simultaneously.
For example:

Transaction Requests
 ↓
Transaction Queue
 ↓
Nonce Assignment
  ↓
Transaction Signing
  ↓
RPC Node
   ↓
Blockchain

This becomes especially useful for exchanges, automated trading systems, payment services, and other applications that generate transactions at high frequency.

Handle Replacements Carefully

If a transaction needs to be replaced, the application needs to preserve the same nonce while applying the appropriate replacement fee rules.

It should also track which transaction is currently considered active for that nonce.

A Simple Real-World Example

Consider a wallet whose next available nonce is 42.

Two requests arrive almost simultaneously:

Request A → Nonce 42 → Transfer tokens
Request B → Nonce 42 → Call a smart contract

Both transactions cannot independently execute.

If Request A is accepted into the transaction pool and Request B does not meet the replacement requirements, Request B may be rejected.

If Request B is submitted as a valid replacement with the required fee increase, it may replace Request A in the relevant transaction pool.

Eventually, only one transaction can consume nonce 42 on the canonical chain.

After that, the next transaction from the wallet needs the following nonce:

Nonce 42 → One transaction
Nonce 43 → Next transaction
Nonce 44 → Next transaction

The key point is that the nonce is tied to the account's transaction sequence, not to the individual application request.

Why Nonce Management Matters in Production

Nonce conflicts become more significant as blockchain applications scale.

A crypto exchange may have multiple services interacting with the same operational wallet. A trading bot may generate several transactions within seconds. A DeFi application may trigger automated contract interactions based on market conditions.

If all these processes independently determine the next nonce, collisions can occur.

A production transaction system therefore needs to treat nonce assignment as part of its infrastructure.

A typical architecture might include:

Application Services
  ↓
Transaction Queue
  ↓
Nonce Manager
  ↓
Signer
   ↓
RPC Provider
   ↓
Blockchain Network

This approach separates transaction generation from nonce assignment and gives the application a controlled way to track pending, confirmed, failed, and replacement transactions.

Conclusion

A nonce is more than a number attached to an EVM transaction. It determines where that transaction fits into the sequence of transactions sent by a particular account.

When two transactions from the same wallet use the same nonce, they compete for the same transaction position. Both cannot execute with that nonce on the canonical chain. Depending on the transaction-pool rules and fee configuration, one may replace the other while pending, or the second transaction may be rejected.

For developers, the bigger lesson is that nonce management needs to account for pending transactions, concurrent requests, transaction replacement, and transaction ordering.

As blockchain applications move toward higher transaction volumes and more automated workflows, reliable nonce management becomes an important part of the transaction infrastructure itself.

How Exchanges and Indexers Handle Blockchain Reorgs: Confirmation Depth, Rollbacks, and Finality

Sundarapandy — Wed, 07 Oct 2026 10:15:05 +0000

A blockchain transaction can appear confirmed one moment and become part of a different chain history later. For a normal wallet user, a short blockchain reorganization, or reorg, may go unnoticed. For an exchange or blockchain indexer, however, it can create a much bigger problem.

Imagine a user deposits 2 ETH into an exchange. The transaction is included in a block, the exchange detects it, and its internal system begins counting confirmations. A few blocks later, the network reorganizes. The block containing the deposit is no longer part of the canonical chain.
The exchange now has a question to answer: Should the deposit still be considered valid?

This is where confirmation depth, reorg detection, rollbacks, and finality become important. Exchanges and indexers cannot simply assume that the latest blockchain state is permanent. They need mechanisms to identify chain changes, reverse affected application state, and determine when a transaction is safe to treat as final.

What Is a Blockchain Reorg?

A blockchain reorganization occurs when the network replaces part of its recent chain history with another valid chain branch.

Under normal conditions, blocks form a sequence:

Block 100 → Block 101 → Block 102 → Block 103

But competing blocks can sometimes be produced around the same point:

→ Block 102A → Block 103A Block 100 → Block 101 → Block 102B → Block 103B

The network eventually selects one branch as the canonical chain. The blocks belonging to the other branch are no longer part of the accepted chain.

This means transactions included in the discarded blocks need to be reconsidered.

A transaction might:

Remain included in the new canonical chain
Become pending again
Be included in a replacement block
Disappear from the canonical chain entirely

For applications that store blockchain data in databases, this distinction matters. An indexer may have already processed an event from a block that is later removed. An exchange may have already detected a deposit that is no longer confirmed on the canonical chain.

A reorg therefore turns blockchain data from a simple append-only stream into something applications must sometimes reconcile and rebuild.

Why Reorgs Are a Problem for Exchanges and Indexers

Exchanges and indexers both consume blockchain data, but they use it differently.

An indexer continuously reads blocks, transactions, logs, and contract events and converts them into structured data that applications can query.

An exchange uses blockchain data to manage deposits, withdrawals, balances, confirmations, and settlement.

Both systems face the same fundamental issue:

A transaction being included in a block does not necessarily mean that its position in the blockchain is permanently settled.

For an indexer, a reorg can invalidate:

Indexed blocks
Transaction records
Token transfers
Smart contract events
Calculated balances
Application-specific state

For an exchange, the consequences can be even more sensitive. A deposit could be credited to a user's internal balance before the block containing that deposit becomes sufficiently stable.

This is why production systems usually separate states such as detected, confirmed, credited, and finalized instead of treating every observed transaction as immediately trustworthy.

Confirmation Depth: How Many Blocks Are Enough?

Confirmation depth is one of the simplest ways applications reduce reorg risk.

Suppose a transaction is included in Block 500:

Transaction → Block 500
        ↓
     Block 501
        ↓
     Block 502
        ↓
      Block 503

As additional blocks are added after Block 500, the transaction gains more confirmations.

An exchange might define a policy such as:

0 confirmations → Detected
1 confirmation → Confirmed
6 confirmations → Credit deposit

The exact number depends on the blockchain, asset, risk model, transaction value, and network conditions.

The basic principle is straightforward:
The deeper a transaction sits inside the chain, the less likely a normal short reorg will remove it.

However, confirmation depth does not mean absolute finality.
Waiting for additional blocks introduces a trade-off:

Fewer confirmations: faster deposits but greater reorg exposure
More confirmations: greater safety but slower user experience

An exchange therefore needs to choose a confirmation policy that balances security and usability.

What Happens During a Reorg?

Consider a deposit included in Block 502:
Block 500 ↓ Block 501 ↓ Block 502 → 2 ETH deposit ↓ Block 503
The exchange's indexer has already processed the transaction.
Now the network produces a competing branch:

Block 500 ↓ Block 501 ↓ Block 502A ↓ Block 503A

If the original Block 502 is removed from the canonical chain, the exchange's internal representation is now potentially outdated.

The application must determine:

Which block was replaced?
What is the common ancestor?
Which blocks were removed?
Which replacement blocks were added?
Which transactions and events were affected?
Does the original deposit still exist in the canonical chain?

The important point is that a reorg is not simply a "missing block" problem. It is a state consistency problem.

The application must bring its internal database back into agreement with the canonical blockchain.

How Blockchain Indexers Detect Reorganizations

A reorg-aware indexer needs more than block numbers.
It should track information such as:

Block number
Block hash
Parent hash
Timestamp
Processing status
Confirmation or finality state

Suppose the indexer stored:

`Block 105
Hash: AAA

Block 106
Hash: BBB
Parent: AAA
The node later reports:
Block 105
Hash: AAA

Block 106
Hash: CCC
Parent: AAA`

The block number is the same, but the hash has changed.
That is a strong indication that the chain has reorganized.

The indexer can then:

Detect the block-hash mismatch.
Find the latest common ancestor.
Identify blocks that are no longer canonical.
Roll back data generated from those blocks.
Process the replacement branch.
Rebuild affected derived state.

Parent-hash validation is particularly important because it allows the indexer to verify whether the block it is processing actually follows the chain it previously stored.

Rollbacks: Reverting Data After a Reorg

A rollback means reversing application state that was derived from blocks that are no longer canonical.

Imagine an indexer records:

Alice balance = 5 ETH

Deposit event:
Alice +2 ETH

Indexed balance = 7 ETH

If the block containing that deposit is removed during a reorg, simply deleting the block record is not enough.

The derived balance may still show 7 ETH.
The indexer must reverse the state:

Remove deposit
        ↓
Reverse affected event
        ↓
Restore Alice's balance
        ↓
Process replacement chain

The same principle applies to token transfers, NFT ownership, contract events, and other derived data.

This is why reorg-aware indexing requires a clear relationship between raw blockchain data and derived application state.

Designing an Indexer That Can Safely Roll Back

A reliable indexer should be designed with reprocessing in mind from the beginning.

Store Block Metadata

A blocks table might contain:

blocks

height
hash
parent_hash
timestamp
status

Transactions and events can then reference the block that produced them:

transactions

tx_hash
block_hash
block_height

events

event_id
tx_hash
block_hash
contract_address
event_type

This makes it easier to identify which records belong to a discarded branch.

Use Checkpoints

Checkpoints allow an indexer to remember where processing safely stopped.

For example:
Last checkpoint: Block 10,000
Current block: Block 10,250

If the indexer crashes or encounters a reorg, it does not necessarily need to rebuild the entire chain.

It can roll back to an appropriate checkpoint or common ancestor and continue processing.

Make Processing Idempotent

An operation is idempotent when running it multiple times does not produce incorrect duplicate state.

This matters because replacement blocks may require transactions or events to be processed again.

A robust indexer should be able to safely:
Read block

→ Process events
→ Roll back
→ Read replacement block
→ Process events again

without creating duplicate balances or duplicate records.

Database transactions can also help ensure that block processing does not leave partially updated state if something fails.

How Exchanges Protect User Funds During Reorgs

Exchanges usually apply stricter rules than a simple blockchain explorer.
A deposit may move through several states:

Detected
   ↓
Pending
   ↓
Confirmed
   ↓
Credited
   ↓
Finalized

These states allow the exchange to separate seeing a transaction from trusting that transaction.

For example, an exchange may detect a deposit immediately but wait for a predefined number of confirmations before making the funds available to the user.

If a reorg occurs while the deposit is still pending, the system can reassess the transaction without necessarily affecting the user's available balance.

For higher-value transactions, exchanges may also use stricter confirmation requirements or additional reconciliation checks.

The key principle is that blockchain monitoring should not directly equal balance crediting.

The application needs a risk-aware state machine between the two.

Confirmation Depth vs Finality

Confirmation depth and finality are related, but they are not the same thing.

Confirmation depth refers to how many blocks have been added after a transaction.

Finality refers to the point at which a transaction is considered settled under the blockchain's consensus rules and is no longer realistically expected to be reversed.

For example:

Confirmation - A transaction is included in a block
Confirmation depth - Additional blocks have been built afterward
Rollback - Application state is reversed after a chain change
Finality - The transaction reaches a sufficiently irreversible state

The distinction becomes particularly important when building systems that operate across different blockchain networks.

An application should understand the finality model of the chain it supports rather than applying the same confirmation policy everywhere.

A network with probabilistic finality may require applications to wait for additional confirmations. A network with explicit consensus finality may provide a different signal that applications can use.

A Practical Reorg-Handling Workflow

A production indexer or exchange can think about reorg handling as a sequence of checks:

New block detected
       ↓
Validate parent hash
       ↓
Compare with stored chain
       ↓
Reorg detected?
     /     \
   No       Yes
   ↓         ↓
Process    Find common ancestor
             ↓
        Roll back affected state
             ↓
      Process replacement blocks
             ↓
       Update confirmations
             ↓
          Finality

The exact implementation will vary, but the architecture should always answer three questions:

Is this block part of the canonical chain?
What application state depends on this block?
When can that state be considered final?

Keeping these questions separate makes the system easier to reason about and test.

How to Test Reorg Handling

Reorg handling should not be tested only when a production incident occurs.

Development teams can simulate scenarios such as:

Competing blocks
Removed blocks
Replacement blocks
Transactions disappearing from a branch
Transactions appearing again in a replacement branch
Duplicate events
Indexer crashes during rollback
Rebuilding state from checkpoints
Deeper-than-expected reorganizations

One useful test is deliberately simple:

If the last several blocks disappeared right now, could the system reconstruct the correct state automatically?

If the answer is no, the system may be relying too heavily on the assumption that blockchain history is permanently append-only.

A reorg-aware architecture should be able to detect the change, identify affected state, roll it back, process the canonical replacement chain, and continue without manual database correction.

Conclusion

Blockchain reorganizations are a normal engineering consideration for applications that depend on recent blockchain state.

For exchanges, reorgs can affect deposits, withdrawals, and internal balances. For indexers, they can invalidate blocks, transactions, events, and derived state.

The solution is not simply to wait for a few confirmations. Robust systems combine confirmation depth, block-hash tracking, parent validation, rollback mechanisms, checkpoints, idempotent processing, and finality-aware state management.

The most important distinction is between data that has been observed and data that can safely be treated as final.

A blockchain application should always be prepared for the possibility that recent state can change.

If the latest few blocks were reorganized today, could your exchange or indexer recover the correct state without manual intervention?

How Blockchain Indexers Rebuild Smart Contract State from Raw Event Logs

Sundarapandy — Mon, 05 Oct 2026 05:27:28 +0000

A blockchain node can tell you what happened on-chain, but it does not automatically give your application the exact data model it needs.
Suppose a DeFi protocol has processed millions of deposits, withdrawals, borrows, repayments, and liquidations. A frontend may need to answer a simple question:
What is this user's current position?

Reading the blockchain directly for every request would mean repeatedly scanning blocks, transactions, contract storage, and event logs. That approach quickly becomes expensive and difficult to scale.

Blockchain indexers solve this problem by creating a data-processing layer between the blockchain and the application.
They consume blocks through RPC endpoints, identify relevant contract events, decode ABI-encoded data, process events in canonical order, apply state transitions, and persist the resulting state in a queryable database.
At a high level:

Blockchain
↓
RPC
↓
Blocks & Logs
↓
Event Decoder
↓
State Processor
↓
Database
↓
API
↓
Application

The interesting engineering problem is not simply collecting events. It is reconstructing a consistent application state from an ordered stream of blockchain activity.
What Smart Contract State Means to an Indexer
A smart contract's state is the collection of values stored by the contract at a particular point in the blockchain.

For example, a lending protocol may maintain values such as:

user → collateral
user → debt
user → interest index
market → liquidity
market → borrow rate

The blockchain contains the transactions and contract execution that modify these values.
An indexer can build an application-oriented representation such as:

UserPosition

user_address market_address collateral_amount borrow_amount last_updated_block
The important distinction is that events are historical facts, while indexed state is a derived representation.

For example:

Deposit(user, 5 ETH) Borrow(user, 2000 USDC) Repay(user, 500 USDC) can result in: Collateral = 5 ETH Debt = 1500 USDC

The indexer derives that representation by processing events in sequence.

Getting Blockchain Data Through RPC

The indexing pipeline usually begins with an RPC connection to a blockchain node or provider.
Depending on the chain and node implementation, an indexer may use JSON-RPC methods such as:

eth_blockNumber eth_getBlockByNumber eth_getLogs eth_getTransactionReceipt

For example, an indexer can request logs for a contract over a block range:

The node returns matching logs from that range.
A production indexer normally does not request an unlimited block range. Instead, it processes blocks in manageable chunks.

For example:
100000 → 100500
100501 → 101000
101001 → 101500

Identifying the Events That Matter

An indexer rarely needs every event emitted by every contract.
Instead, it defines the contracts and event signatures it cares about.

For example, a token indexer may monitor:
Transfer(address,address,uint256)
Approval(address,address,uint256)

The event signature is represented by a topic hash.
Conceptually:
Event Definition
↓
Keccak/Event Signature
↓
Topic0
↓
Matching Blockchain Logs

When an eth_getLogs response contains a matching topic, the indexer knows that the log corresponds to an event it understands.
This filtering is important for performance. Processing irrelevant logs at scale can create unnecessary RPC traffic, CPU work, and database writes.

The exact batch size depends on the chain, RPC provider, response size, rate limits, and event density.
This is the first major engineering consideration: the indexer must reliably move through the chain without losing its position or overwhelming the RPC layer.

ABI Decoding: Turning Logs into Structured Events
Raw logs are not usually returned as convenient application objects.

A typical log contains fields such as:

address
topics[]
data
blockNumber
transactionHash
transactionIndex
logIndex

The indexer uses the contract ABI to understand how those values should be interpreted.
Consider:
event Transfer( address indexed from, address indexed to, uint256 value );
The two indexed addresses are represented in topics, while value is encoded in the event data.
A decoder transforms the raw representation into something like:
{ "event": "Transfer", "from": "0xAlice", "to": "0xBob", "value": "50000000000000000000" }

The indexer can then convert the raw integer into the appropriate application representation, such as:

50 tokens

depending on the token's decimals.
This ABI-decoding stage is critical because incorrectly interpreting event data can result in incorrect application state even when the blockchain data itself is perfectly valid.

Event Ordering Is Part of State Reconstruction

An indexer cannot simply process events in the order they arrive from different workers.
State transitions must follow blockchain ordering.

A useful ordering key is:
(block_number, transaction_index, log_index)
For example:

Block 500
 ├── Transaction 0
 │    ├── Log 0
 │    └── Log 1
 │
 ├── Transaction 1
 │    └── Log 0
 │
 └── Transaction 2
        └── Log 0

The indexer should process these in their canonical sequence.
This becomes particularly important when several events affect the same entity.

Consider:
Block 500:
Deposit +100
Transfer -40

If the indexer applies the transfer before the deposit, it could temporarily derive an impossible balance.
Therefore, parallel ingestion and sequential state application often need to be separated architecturally.

Reconstructing State from Events

Once events have been decoded and ordered, the indexer applies them as state transitions.
A simplified model is:
new_state = apply(previous_state, event)
For example:
Initial balance = 0

Mint 100
→ balance = 100
Transfer 30
→ balance = 70
Transfer 20
→ balance = 50

A simple pseudocode implementation could look like:

def process_event(event, state):
    if event.type == "Mint":
        state.balance += event.amount

    elif event.type == "Transfer":
        state.balance -= event.amount

    return state

A production indexer obviously needs considerably more logic, including address validation, token decimals, multiple accounts, transaction boundaries, database transactions, rollback handling, and failure recovery.

But the underlying principle remains:

Previous State + Event → New State

Database Schema for Indexed State

After processing events, the indexer needs to persist the resulting data.
A simplified schema for a token indexer might look like:

Keeping both raw normalized events and derived state can be useful.
The event table provides an audit trail and makes reprocessing possible, while the derived tables provide fast application queries.

For example:
SELECT balance FROM token_balances WHERE wallet_address = '0x...';

is considerably more practical for an application than scanning historical blockchain logs for every request.

Checkpoints: Knowing Where the Indexer Stopped

A production indexer also needs a checkpoint system.
Imagine the indexer has successfully processed block 2,500,000.

It can store:
last_processed_block = 2500000 last_processed_hash = 0xabc...

If the process crashes, it can resume from a known position instead of starting from the beginning.

A checkpoint table might look like:

CREATE TABLE indexer_checkpoint (
    id              INT PRIMARY KEY,
    block_number    BIGINT,
    block_hash      VARCHAR(66),
    updated_at      TIMESTAMP
);

The checkpoint should generally be updated only after the corresponding data has been successfully persisted.

Otherwise, the indexer could record a block as processed even though some of its state changes were never committed.
This creates an important consistency rule:

Persist the state and advance the checkpoint as one logical unit of progress.

Handling Reorganizations

One of the most important challenges is blockchain reorganization.
Suppose the indexer processes:

Block 100 ↓ Block 101A ↓ Block 102A

It derives application state from those blocks.
Later, the canonical chain becomes:

Block 100 ↓ Block 101B ↓ Block 102B

The events from 101A and 102A may no longer belong to the canonical chain.

A robust indexer needs to detect this situation.
That is why storing the block hash alongside the checkpoint is useful.
The indexer can compare:

Stored block hash
vs
Current canonical block hash

If they differ, the indexer knows that its local view needs reconciliation.
A simplified rollback process could be:

Detect mismatch
↓
Find common ancestor
↓
Rollback affected indexed state
↓
Remove/reverse affected events
↓
Process canonical blocks again
↓
Update checkpoint

Some architectures avoid immediately exposing very recent blocks as final application state and instead wait for additional confirmations. Others implement explicit rollback support.

The correct strategy depends on the blockchain and the application's consistency requirements.

Handling Missed Events and RPC Failures

RPC infrastructure introduces another failure surface.
For example:

Indexer → RPC Provider
             ↓
         Timeout

If the indexer simply continues without verifying the response, it could silently skip blockchain data.

Production systems commonly use:

Retry policies
Request timeouts
Block-range backfilling
Multiple RPC providers
Rate-limit handling
Failed-job queues
Data consistency checks

Suppose the indexer successfully processed:

100000 → 101000 but the RPC request for: 101001 → 101500
failed.

The indexer should not advance its checkpoint to 101500.
Instead:

Checkpoint: 101000
↓
Retry 101001 → 101500
↓
Process successfully
↓
Advance checkpoint

This simple rule prevents gaps in the indexed chain history.

A Simplified Indexer Loop

The entire process can be represented with pseudocode:
while True:

latest_block = get_latest_block()

    while checkpoint < latest_block:
        from_block = checkpoint + 1
        to_block = min(from_block + BATCH_SIZE - 1, latest_block)

        logs = get_logs(from_block, to_block)

        events = decode_logs(logs)

        events.sort(
            key=lambda e: (
                e.block_number,
                e.transaction_index,
                e.log_index
            )
        )

        begin_transaction()

    for event in events:

        `apply_state_change(event)
        store_event(event)

    save_checkpoint(to_block)

    commit_transaction()

    checkpoint = to_block`

A real implementation would need additional handling for RPC failures, duplicate events, reorgs, finality, concurrent workers, database failures, and chain-specific behavior.

Still, this loop captures the core indexing model:
Fetch → Decode → Order → Apply → Persist → Checkpoint

Why This Architecture Matters for Web3 Applications

Once the indexing layer is reliable, applications can query structured data rather than repeatedly interpreting raw blockchain activity.

A DeFi frontend can request:
GET /users/0x123/positions
and receive:
{ "collateral": "5 ETH", "debt": "1500 USDC" }

An NFT application can request the current owner of a token.
A trading application can query historical trades.
A blockchain explorer can provide transaction and contract activity through indexed tables.

In each case, the application is consuming a derived data layer, while the blockchain remains the underlying source of truth.
This separation also makes it easier to scale application APIs independently from blockchain RPC infrastructure.
The Complete Reconstruction Pipeline
Putting everything together:

Blockchain
↓
RPC Node
↓
Block / Log Fetching
↓
Event Filtering
↓
ABI Decoding
↓
Canonical Ordering
↓
State Transitions
↓
Database Persistence
↓
Checkpoint Update
↓
API
↓
Web3 Application

The indexer's job is therefore more sophisticated than simply copying blockchain data into a database.
It is maintaining a continuously updated, application-oriented representation of blockchain activity while accounting for ordering, failures, missed blocks, and chain reorganizations.

Conclusion

Blockchain indexers provide the infrastructure that turns low-level blockchain activity into data applications can actually use.

The process starts with RPC calls that retrieve blocks and logs. The indexer filters relevant events, decodes ABI-encoded values, orders them according to their position in the chain, and applies each event as a state transition.

The resulting state is persisted in database tables optimized for application queries. Checkpoints allow the indexer to resume safely after failures, while block hashes and rollback mechanisms help it recover from chain reorganizations.

The fundamental model is:

Raw Logs
↓
RPC Ingestion
↓
ABI Decoding
↓
Canonical Ordering
↓
State Reconstruction
↓
Database
↓
API
↓
Application

For developers and CTOs building blockchain applications, the key architectural question is not simply “How do we read blockchain data?”

It is:
“How do we maintain a reliable, queryable representation of blockchain state as the chain continuously changes?”

That is the core problem blockchain indexing is designed to solve.

What Happens When an API Call Fails? A Practical Guide to Timeouts, Retries, and Circuit Breakers

Sundarapandy — Mon, 21 Sep 2026 13:23:07 +0000

Your app submits a request to an external API.
Usually, everything works:

Your Application → External API → Response

But what happens when the external API becomes slow?

Or stops responding completely?

Your application may keep waiting, requests may start piling up, and eventually other parts of your system can become affected.
This is why production applications need to be designed not only for successful requests, but also for failures.

In this post, we'll look at three important mechanisms for handling unreliable dependencies:

Timeouts
Retries
Circuit breakers

We'll also look at exponential backoff and graceful degradation.

Why API Failures Are Different in Production

When developing locally, an API call often looks simple:

Application
↓
API Request
↓
External Service
↓
Response

In production, there are many things that can go wrong.
An external service might:

Take too long to respond
Return a 500 error
Become temporarily unavailable
Rate-limit your requests
Experience network problems
Return an unexpected response

For example, imagine an e-commerce application that calls a recommendation service:

User
↓
E-commerce API
↓
Recommendation Service

If the recommendation service becomes unavailable, should the entire application stop working?
Usually, no.
The application should have a strategy for handling that failure.
That's where resilience patterns become important.

Timeouts: Don't Wait Forever

The first rule is simple:

Never allow an external request to wait indefinitely.

Suppose your application sends a request to a payment service:
Application → Payment API
If the payment API doesn't respond, your application might continue waiting.
If enough requests do this simultaneously, you can end up with many requests consuming connections and resources.

A timeout places a limit on how long the application will wait.

Application
↓
Payment API
↓
No response
↓
Timeout

For example:

response = requests.get( "https://api.example.com/payment", timeout=5 )

The exact timeout value depends on the operation and system requirements, but the important principle is:
Every external dependency should have an intentional timeout.
A timeout doesn't fix the underlying failure.
It prevents one slow dependency from holding your application hostage indefinitely.

Retries: Temporary Failures Can Recover

Some failures are temporary.
A network request might fail once and succeed a moment later.
In those cases, retrying can be useful.
A basic retry flow looks like this:

Request
↓
Failure
↓
Retry
↓
Success

For example:
Attempt 1 → Failed
Attempt 2 → Failed
Attempt 3 → Success

This can be useful for transient problems such as temporary network failures or certain server errors.

However, retries should not be added blindly.
Imagine 1,000 requests reach a service that is already struggling.
If every request immediately retries three times, the service could suddenly receive thousands of additional requests. Instead of recovering, the situation can become worse. This is sometimes referred to as a retry storm.

Exponential Backoff

One way to make retries safer is to introduce a delay between attempts.
Instead of:
Retry → Retry → Retry
you can use:

Request
↓
Failure
↓
Wait
↓
Retry
↓
Failure
↓
Wait longer
↓
Retry

This is called exponential backoff.

A simplified example might look like:
Attempt 1 → Failure
Wait 1 second

Attempt 2 → Failure
Wait 2 seconds

Attempt 3 → Failure
Wait 4 seconds

The delay increases after each failed attempt.
In distributed systems, jitter is often added to these delays as well. This prevents many clients from retrying at exactly the same time.

A practical retry strategy therefore usually includes:

A maximum number of retries
Exponential backoff
Jitter
Appropriate timeout values
Rules for which errors are retryable

Don't Retry Every Request

This is an important distinction. Not every failed request should automatically be retried.
For example, retrying a read operation may be relatively safe in many cases.
But consider a request that creates a payment or places an order.
If the server processed the request successfully but the response was lost, your client may not know whether the operation actually happened.
Automatically sending the same request again could potentially create a duplicate operation.
This is why idempotency matters.

For operations that may be retried, systems often use idempotency keys or other mechanisms to ensure that repeating the request doesn't accidentally repeat the business operation.
The key question isn't:
“Can we retry this request?”
It's:
“What happens if we retry this request?”

Circuit Breakers: Stop Calling a Failing Service

Now consider a different situation.
An external service isn't failing once or twice.
It's been failing continuously for several minutes.
Should your application keep sending requests?
Probably not.
This is where a circuit breaker can help.
A circuit breaker generally has three states:

Success
┌───────────┐
│ ↓
CLOSED → OPEN → HALF-OPEN
↑ │
└─────────────────┘
Recovery

Closed

Everything is operating normally.
Requests are allowed through.
Application → External Service

Open

The dependency is consistently failing.
The circuit opens and stops sending requests to that dependency.

Application → Circuit Breaker → Fallback

This prevents your application from repeatedly calling a service that is already unavailable.

Half-Open

After some time, the circuit breaker allows a limited request through to check whether the dependency has recovered.

If the request succeeds:

Half-Open → Closed

If it fails again:

Half-Open → Open

This gives the failing service time to recover while protecting your application from repeated failures.

Graceful Degradation

Sometimes the best solution isn't retrying at all.
Instead, your application can provide a reduced version of the functionality.
Consider an online shopping application:

Product Page
↓
Recommendation Service

The recommendation service goes down.
You could make the entire product page fail.
Or you could simply hide the recommendations:

Product Page
↓
Recommendation Service ❌
↓
Show Product Without Recommendations

The user can still view the product and continue shopping.
This is called graceful degradation.
The idea is simple:
When one feature fails, don't necessarily let the entire system fail with it.
Putting the Patterns Together
These mechanisms work particularly well when combined.
A simplified architecture could look like:

            ┌───────────────┐
            │ Your Service  │
            └───────┬───────┘
                    ↓
             ┌─────────────┐
             │  Timeout    │
             └──────┬──────┘
                    ↓
             ┌─────────────┐
             │    Retry    │
             └──────┬──────┘
                    ↓
          ┌──────────────────┐
          │  Circuit Breaker │
          └─────────-────────┘
                    ↓
             External Service
                    ↓
              ┌─────────┐
              │Fallback │
              └─────────┘

The exact implementation depends on your architecture, but the concepts work together:
Timeouts prevent requests from waiting indefinitely.
Retries handle certain temporary failures.
Backoff prevents retries from overwhelming a failing service.
Circuit breakers interrupt repeated calls to an unhealthy Service.
Fallbacks allow your application to continue operating when possible.

What Should You Monitor?

Adding resilience mechanisms isn't enough.
You also need to know when they're being triggered.
Useful metrics include:

Request failure rate
Timeout rate
Retry count
Retry success rate
Circuit breaker state
External API latency
Fallback frequency
Dependency availability

For example, if your retry success rate suddenly drops from 90% to 10%, that could indicate that the dependency has a larger problem.
Without monitoring, your system may appear to be working while silently relying on retries and fallbacks.

A Practical Checklist

When your application depends on external services, consider the following:

Timeout
Set a reasonable timeout for every external request.

Retry
Retry only failures that are safe and potentially temporary.

Backoff
Avoid immediate repeated retries. Use exponential backoff and, where appropriate, jitter.

Idempotency
Make sure retrying an operation doesn't accidentally duplicate a business action.

Circuit Breaker
Stop repeatedly calling a dependency that is consistently failing.

Fallback
Decide whether your application can continue with reduced functionality.

Observability
Track failures, retries, latency, and dependency health.

Conclusion

External services will eventually become slow, unavailable, or unpredictable.
The objective is not to create a system that will never fail.
That's unrealistic.
The goal is to build a system that fails predictably, limits the impact of failures, and recovers when dependencies become healthy again.
Timeouts, retries, exponential backoff, circuit breakers, and graceful degradation are some of the building blocks that make that possible.

The next time you add an external API to your application, don't ask only:
“What happens when the API works?”
Also ask:
“What happens when it doesn't?”