Please ensure you’ve first read the spec for ABCI Methods and Types

Here we cover the following components of ABCI applications:

Connection State

Since CometBFT maintains four concurrent ABCI connections, it is typical for an application to maintain a distinct state for each, and for the states to be synchronized during Commit.


In principle, each of the four ABCI connections operate concurrently with one another. This means applications need to ensure access to state is thread safe. In practice, both the default in-process ABCI client and the default Go ABCI server use global locks across all connections, so they are not concurrent at all. This means if your app is written in Go, and compiled in-process with CometBFT using the default NewLocalClient, or run out-of-process using the default SocketServer, ABCI messages from all connections will be linearizable (received one at a time).

The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method.


The BeginBlock request can be used to run some code at the beginning of every block. It also allows CometBFT to send the current block hash and header to the application, before it sends any of the transactions.

The app should remember the latest height and header (ie. from which it has run a successful Commit) so that it can tell CometBFT where to pick up from when it restarts. See information on the Handshake, below.


Application state should only be persisted to disk during Commit.

Before Commit is called, CometBFT locks and flushes the mempool so that no new messages will be received on the mempool connection. This provides an opportunity to safely update all four connection states to the latest committed state at once.

When Commit completes, it unlocks the mempool.

WARNING: if the ABCI app logic processing the Commit message sends a /broadcast_tx_sync or /broadcast_tx_commit and waits for the response before proceeding, it will deadlock. Executing those broadcast_tx calls involves acquiring a lock that is held during the Commit call, so it’s not possible. If you make the call to the broadcast_tx endpoints concurrently, that’s no problem, it just can’t be part of the sequential logic of the Commit function.

Consensus Connection

The Consensus Connection should maintain a DeliverTxState - the working state for block execution. It should be updated by the calls to BeginBlock, DeliverTx, and EndBlock during block execution and committed to disk as the “latest committed state” during Commit.

Updates made to the DeliverTxState by each method call must be readable by each subsequent method - ie. the updates are linearizable.

Mempool Connection

The mempool Connection should maintain a CheckTxState to sequentially process pending transactions in the mempool that have not yet been committed. It should be initialized to the latest committed state at the end of every Commit.

Before calling Commit, CometBFT will lock and flush the mempool connection, ensuring that all existing CheckTx are responded to and no new ones can begin. The CheckTxState may be updated concurrently with the DeliverTxState, as messages may be sent concurrently on the Consensus and Mempool connections.

After Commit, while still holding the mempool lock, CheckTx is run again on all transactions that remain in the node’s local mempool after filtering those included in the block. An additional Type parameter is made available to the CheckTx function that indicates whether an incoming transaction is new (CheckTxType_New), or a recheck (CheckTxType_Recheck).

Finally, after re-checking transactions in the mempool, CometBFT will unlock the mempool connection. New transactions are once again able to be processed through CheckTx.

Note that CheckTx is just a weak filter to keep invalid transactions out of the block chain. CheckTx doesn’t have to check everything that affects transaction validity; the expensive things can be skipped. It’s weak because a Byzantine node doesn’t care about CheckTx; it can propose a block full of invalid transactions if it wants.

Replay Protection

To prevent old transactions from being replayed, CheckTx must implement replay protection.

It is possible for old transactions to be sent to the application. So it is important CheckTx implements some logic to handle them.

Query Connection

The Info Connection should maintain a QueryState for answering queries from the user, and for initialization when CometBFT first starts up (both described further below). It should always contain the latest committed state associated with the latest committed block.

QueryState should be set to the latest DeliverTxState at the end of every Commit, after the full block has been processed and the state committed to disk. Otherwise it should never be modified.

CometBFT currently uses the Query connection to filter peers upon connecting, according to IP address or node ID. For instance, returning non-OK ABCI response to either of the following queries will cause CometBFT to not connect to the corresponding peer:

Note: these query formats are subject to change!

Snapshot Connection

The Snapshot Connection is optional, and is only used to serve state sync snapshots for other nodes and/or restore state sync snapshots to a local node being bootstrapped.

For more information, see the state sync section of this document.

Transaction Results

The Info and Log fields are non-deterministic values for debugging/convenience purposes that are otherwise ignored.

The Data field must be strictly deterministic, but can be arbitrary data.


Ethereum introduced the notion of gas as an abstract representation of the cost of resources used by nodes when processing transactions. Every operation in the Ethereum Virtual Machine uses some amount of gas, and gas can be accepted at a market-variable price. Users propose a maximum amount of gas for their transaction; if the tx uses less, they get the difference credited back. CometBFT adopts a similar abstraction, though uses it only optionally and weakly, allowing applications to define their own sense of the cost of execution.

In CometBFT, the ConsensusParams.Block.MaxGas limits the amount of gas that can be used in a block. The default value is -1, meaning no limit, or that the concept of gas is meaningless.

Responses contain a GasWanted and GasUsed field. The former is the maximum amount of gas the sender of a tx is willing to use, and the latter is how much it actually used. Applications should enforce that GasUsed <= GasWanted - ie. tx execution should halt before it can use more resources than it requested.

When MaxGas > -1, CometBFT enforces the following rules:

If MaxGas == -1, no rules about gas are enforced.

Note that CometBFT does not currently enforce anything about Gas in the consensus, only the mempool. This means it does not guarantee that committed blocks satisfy these rules! It is the application’s responsibility to return non-zero response codes when gas limits are exceeded.

The GasUsed field is ignored completely by CometBFT. That said, applications should enforce:

In the future, we intend to add a Priority field to the responses that can be used to explicitly prioritize txs in the mempool for inclusion in a block proposal. See #1861.


If Code != 0, it will be rejected from the mempool and hence not broadcasted to other peers and not included in a proposal block.

Data contains the result of the CheckTx transaction execution, if any. It is semantically meaningless to CometBFT.

Events include any events for the execution, though since the transaction has not been committed yet, they are effectively ignored by CometBFT.


DeliverTx is the workhorse of the blockchain. CometBFT sends the DeliverTx requests asynchronously but in order, and relies on the underlying socket protocol (ie. TCP) to ensure they are received by the app in order. They have already been ordered in the global consensus by the CometBFT protocol.

If DeliverTx returns Code != 0, the transaction will be considered invalid, though it is still included in the block.

DeliverTx also returns a Code, Data, and Log.

Data contains the result of the CheckTx transaction execution, if any. It is semantically meaningless to CometBFT.

Both the Code and Data are included in a structure that is hashed into the LastResultsHash of the next block header.

Events include any events for the execution, which CometBFT will use to index the transaction by. This allows transactions to be queried according to what events took place during their execution.

Updating the Validator Set

The application may set the validator set during InitChain, and may update it during EndBlock.

Note that the maximum total power of the validator set is bounded by MaxTotalVotingPower = MaxInt64 / 8. Applications are responsible for ensuring they do not make changes to the validator set that cause it to exceed this limit.

Additionally, applications must ensure that a single set of updates does not contain any duplicates - a given public key can only appear once within a given update. If an update includes duplicates, the block execution will fail irrecoverably.


The InitChain method can return a list of validators. If the list is empty, CometBFT will use the validators loaded in the genesis file. If the list returned by InitChain is not empty, CometBFT will use its contents as the validator set. This way the application can set the initial validator set for the blockchain.


Updates to the CometBFT validator set can be made by returning ValidatorUpdate objects in the ResponseEndBlock:

message ValidatorUpdate {
  tendermint.crypto.keys.PublicKey pub_key
  int64 power

message PublicKey {
  oneof {
    ed25519 bytes = 1;

The pub_key currently supports only one type:

The power is the new voting power for the validator, with the following rules:

Note the updates returned in block H will only take effect at block H+2.

Consensus Parameters

ConsensusParams enforce certain limits in the blockchain, like the maximum size of blocks, amount of gas used in a block, and the maximum acceptable age of evidence. They can be set in InitChain and updated in EndBlock.


The maximum size of a complete Protobuf encoded block. This is enforced by the consensus algorithm.

This implies a maximum transaction size that is MaxBytes, less the expected size of the header, the validator set, and any included evidence in the block.

The Application should be aware that honest validators may produce and broadcast blocks with up to the configured MaxBytes size. As a result, the consensus timeout parameters adopted by nodes should be configured so as to account for the worst-case latency for the delivery of a full block with MaxBytes size to all validators.

Must have 0 < MaxBytes <= 100 MB.

Bear in mind that the default value for the BlockParams.MaxBytes consensus parameter accepts as valid blocks with size up to 21 MB. If the Application’s use case does not need blocks of that size, or if the impact (specially on bandwidth consumption and block latency) of propagating blocks of that size was not evaluated, it is strongly recommended to wind down this default value.


The maximum of the sum of GasWanted that will be allowed in a proposed block. This is not enforced by CometBFT consensus. It is left to the app to enforce (ie. if txs are included past the limit, they should return non-zero codes). It is used by CometBFT to limit the txs included in a proposed block.

Must have MaxGas >= -1. If MaxGas == -1, no limit is enforced.


This is the maximum age of evidence in time units. This is enforced by CometBFT consensus.

If a block includes evidence older than this (AND the evidence was created more than MaxAgeNumBlocks ago), the block will be rejected (validators won’t vote for it).

Must have MaxAgeDuration > 0.


This is the maximum age of evidence in blocks. This is enforced by CometBFT consensus.

If a block includes evidence older than this (AND the evidence was created more than MaxAgeDuration ago), the block will be rejected (validators won’t vote for it).

Must have MaxAgeNumBlocks > 0.


This is the maximum number of evidence that can be committed to a single block.

The product of this and the MaxEvidenceBytes must not exceed the size of a block minus it’s overhead ( ~ MaxBytes).

Must have MaxNum > 0.


The application may set the ConsensusParams during InitChain, and update them during EndBlock. If the ConsensusParams is empty, it will be ignored. Each field that is not empty will be applied in full. For instance, if updating the Block.MaxBytes, applications must also set the other Block fields (like Block.MaxGas), even if they are unchanged, as they will otherwise cause the value to be updated to 0.


ResponseInitChain includes a ConsensusParams. If ConsensusParams is nil, CometBFT will use the params loaded in the genesis file. If ConsensusParams is not nil, CometBFT will use it. This way the application can determine the initial consensus params for the blockchain.


ResponseEndBlock includes a ConsensusParams. If ConsensusParams nil, CometBFT will do nothing. If ConsensusParam is not nil, CometBFT will use it. This way the application can update the consensus params over time.

Note the updates returned in block H will take effect right away for block H+1.


Query is a generic method with lots of flexibility to enable diverse sets of queries on application state. CometBFT makes use of Query to filter new peers based on ID and IP, and exposes Query to the user over RPC.

Note that calls to Query are not replicated across nodes, but rather query the local node’s state - hence they may return stale reads. For reads that require consensus, use a transaction.

The most important use of Query is to return Merkle proofs of the application state at some height that can be used for efficient application-specific light-clients.

Note CometBFT has technically no requirements from the Query message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too.

Query Proofs

The CometBFT block header includes a number of hashes, each providing an anchor for some type of proof about the blockchain. The ValidatorsHash enables quick verification of the validator set, the DataHash gives quick verification of the transactions included in the block, etc.

The AppHash is unique in that it is application specific, and allows for application-specific Merkle proofs about the state of the application. While some applications keep all relevant state in the transactions themselves (like Bitcoin and its UTXOs), others maintain a separated state that is computed deterministically from transactions, but is not contained directly in the transactions themselves (like Ethereum contracts and accounts). For such applications, the AppHash provides a much more efficient way to verify light-client proofs.

ABCI applications can take advantage of more efficient light-client proofs for their state as follows:

For instance, this allows an application’s light-client to verify proofs of absence in the application state, something which is much less efficient to do using the block hash.

Some applications (eg. Ethereum, Cosmos-SDK) have multiple “levels” of Merkle trees, where the leaves of one tree are the root hashes of others. To support this, and the general variability in Merkle proofs, the ResponseQuery.Proof has some minimal structure:

message ProofOps {
  repeated ProofOp ops

message ProofOp {
  string type = 1;
  bytes key = 2;
  bytes data = 3;

Each ProofOp contains a proof for a single key in a single Merkle tree, of the specified type. This allows ABCI to support many different kinds of Merkle trees, encoding formats, and proofs (eg. of presence and absence) just by varying the type. The data contains the actual encoded proof, encoded according to the type. When verifying the full proof, the root hash for one ProofOp is the value being verified for the next ProofOp in the list. The root hash of the final ProofOp in the list should match the AppHash being verified against.

Peer Filtering

When CometBFT connects to a peer, it sends two queries to the ABCI application using the following paths, with no additional data:

If either of these queries return a non-zero ABCI code, CometBFT will refuse to connect to the peer.


Queries are directed at paths, and may optionally include additional data.

The expectation is for there to be some number of high level paths differentiating concerns, like /p2p, /store, and /app. Currently, CometBFT only uses /p2p, for filtering peers. For more advanced use, see the implementation of Query in the Cosmos-SDK.

Crash Recovery

On startup, CometBFT calls the Info method on the Info Connection to get the latest committed state of the app. The app MUST return information consistent with the last block it succesfully completed Commit for.

If the app succesfully committed block H, then last_block_height = H and last_block_app_hash = <hash returned by Commit for block H>. If the app failed during the Commit of block H, then last_block_height = H-1 and last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>.

We now distinguish three heights, and describe how CometBFT syncs itself with the app.

storeBlockHeight = height of the last block CometBFT saw a commit for
stateBlockHeight = height of the last block for which CometBFT completed all
    block processing and saved all ABCI results to disk
appBlockHeight = height of the last block for which ABCI app succesfully
    completed Commit

Note we always have storeBlockHeight >= stateBlockHeight and storeBlockHeight >= appBlockHeight Note also CometBFT never calls Commit on an ABCI app twice for the same height.

The procedure is as follows.

First, some simple start conditions:

If appBlockHeight == 0, then call InitChain.

If storeBlockHeight == 0, we’re done.

Now, some sanity checks:

If storeBlockHeight < appBlockHeight, error If storeBlockHeight < stateBlockHeight, panic If storeBlockHeight > stateBlockHeight+1, panic

Now, the meat:

If storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight, replay all blocks in full from appBlockHeight to storeBlockHeight. This happens if we completed processing the block, but the app forgot its height.

If storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight, we’re done. This happens if we crashed at an opportune spot.

If storeBlockHeight == stateBlockHeight+1 This happens if we started processing the block but didn’t finish.

If appBlockHeight < stateBlockHeight replay all blocks in full from appBlockHeight to storeBlockHeight-1, and replay the block at storeBlockHeight using the WAL. This happens if the app forgot the last block it committed.

If appBlockHeight == stateBlockHeight, replay the last block (storeBlockHeight) in full. This happens if we crashed before the app finished Commit

If appBlockHeight == storeBlockHeight update the state using the saved ABCI responses but dont run the block against the real app. This happens if we crashed after the app finished Commit but before CometBFT saved the state.

State Sync

A new node joining the network can simply join consensus at the genesis height and replay all historical blocks until it is caught up. However, for large chains this can take a significant amount of time, often on the order of days or weeks.

State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot of the state machine at a given height and restores it. Depending on the application, this can be several orders of magnitude faster than replaying blocks.

Note that state sync does not currently backfill historical blocks, so the node will have a truncated block history - users are advised to consider the broader network implications of this in terms of block availability and auditability. This functionality may be added in the future.

For details on the specific ABCI calls and types, see the methods and types section.

Taking Snapshots

Applications that want to support state syncing must take state snapshots at regular intervals. How this is accomplished is entirely up to the application. A snapshot consists of some metadata and a set of binary chunks in an arbitrary format:

For a snapshot to be considered the same across nodes, all of these fields must be identical. When sent across the network, snapshot metadata messages are limited to 4 MB.

When a new node is running state sync and discovering snapshots, CometBFT will query an existing application via the ABCI ListSnapshots method to discover available snapshots, and load binary snapshot chunks via LoadSnapshotChunk. The application is free to choose how to implement this and which formats to use, but must provide the following guarantees:

A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB), start a transaction immediately after block commit, and spawn a new thread which is passed the transaction handle. This thread can then export all data items, serialize them using e.g. Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system along with some metadata - all while the blockchain is applying new blocks in parallel.

A more advanced approach might include incremental verification of individual chunks against the chain app hash, parallel or batched exports, compression, and so on.

Old snapshots should be removed after some time - generally only the last two snapshots are needed (to prevent the last one from being removed while a node is restoring it).

Bootstrapping a Node

An empty node can be state synced by setting the configuration option statesync.enabled = true. The node also needs the chain genesis file for basic chain info, and configuration for light client verification of the restored snapshot: a set of CometBFT RPC servers, and a trusted header hash and corresponding height from a trusted source, via the statesync configuration section.

Once started, the node will connect to the P2P network and begin discovering snapshots. These will be offered to the local application via the OfferSnapshot ABCI method. Once a snapshot is accepted CometBFT will fetch and apply the snapshot chunks. After all chunks have been successfully applied, CometBFT verifies the app’s AppHash against the chain using the light client, then switches the node to normal consensus operation.

Snapshot Discovery

When the empty node join the P2P network, it asks all peers to report snapshots via the ListSnapshots ABCI call (limited to 10 per node). After some time, the node picks the most suitable snapshot (generally prioritized by height, format, and number of peers), and offers it to the application via OfferSnapshot. The application can choose a number of responses, including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent it, and so on. CometBFT will keep discovering and offering snapshots until one is accepted or the application aborts.

Snapshot Restoration

Once a snapshot has been accepted via OfferSnapshot, CometBFT begins downloading chunks from any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are spooled in a temporary directory, and then given to the application in sequential order via ApplySnapshotChunk until all chunks have been accepted.

The method for restoring snapshot chunks is entirely up to the application.

During restoration, the application can respond to ApplySnapshotChunk with instructions for how to continue. This will typically be to accept the chunk and await the next one, but it can also ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI reference for details.

If CometBFT fails to fetch a chunk after some time, it will reject the snapshot and try a different one via OfferSnapshot - the application can choose whether it wants to support restarting restoration, or simply abort with an error.

Snapshot Verification

Once all chunks have been accepted, CometBFT issues an Info ABCI call to retrieve the LastBlockAppHash. This is compared with the trusted app hash from the chain, retrieved and verified using the light client. CometBFT also checks that LastBlockHeight corresponds to the height of the snapshot.

This verification ensures that an application is valid before joining the network. However, the snapshot restoration may take a long time to complete, so applications may want to employ additional verification during the restore to detect failures early. This might e.g. include incremental verification of each chunk against the app hash (using bundled Merkle proofs), checksums to protect against data corruption by the disk or network, and so on. However, it is important to note that the only trusted information available is the app hash, and all other snapshot metadata can be spoofed by adversaries.

Apps may also want to consider state sync denial-of-service vectors, where adversaries provide invalid or harmful snapshots to prevent nodes from joining the network. The application can counteract this by asking CometBFT to ban peers. As a last resort, node operators can use P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots.

Transition to Consensus

Once the snapshots have all been restored, CometBFT gathers additional information necessary for bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers) from the genesis file and light client RPC servers. It also fetches and records the AppVersion from the ABCI application.

Once the state machine has been restored and CometBFT has gathered this additional information, it transitions to block sync (if enabled) to fetch any remaining blocks up the chain head, and then transitions to regular consensus operation. At this point the node operates like any other node, apart from having a truncated block history at the height of the restored snapshot.

Decorative Orb