Outline

    Overview and basic concepts

    ABCI 2.0 vs. ABCI

    ↑ Back to Outline

    The Application’s main role is to execute blocks decided (a.k.a. finalized) by consensus. The decided blocks are the consensus’s main ouput to the (replicated) Application. With ABCI, the application only interacts with consensus at decision time. This restricted mode of interaction prevents numerous features for the Application, including many scalability improvements that are now better understood than when ABCI was first written. For example, many ideas proposed to improve scalability can be boiled down to “make the block proposers do work, so the network does not have to”. This includes optimizations such as transaction level signature aggregation, state transition proofs, etc. Furthermore, many new security properties cannot be achieved in the current paradigm, as the Application cannot require validators to do more than executing the transactions contained in finalized blocks. This includes features such as threshold cryptography, and guaranteed IBC connection attempts.

    ABCI 2.0 addresses these limitations by allowing the application to intervene at two key places of consensus execution: (a) at the moment a new proposal is to be created and (b) at the moment a proposal is to be validated. The new interface allows block proposers to perform application-dependent work in a block through the PrepareProposal method (a); and validators to perform application-dependent work and checks in a proposed block through the ProcessProposal method (b).

    We plan to extend this to allow applications to intervene at the moment a (precommit) vote is sent/received. The applications could then require their validators to do more than just validating blocks through the ExtendVote and VerifyVoteExtension methods.

    Methods overview

    ↑ Back to Outline

    Methods can be classified into four categories: consensus, mempool, info, and state-sync.

    Consensus/block execution methods

    The first time a new blockchain is started, CometBFT calls InitChain. From then on, methods BeginBlock, DeliverTx and EndBlock are executed upon the decision of each block, resulting in an updated Application state. One DeliverTx is called for each transaction in the block. The result is an updated application state. Cryptographic commitments to the results of DeliverTx, and an application-provided hash in Commit are included in the header of the next block. During the execution of an instance of consensus, which decides the block for a given height, and before method BeginBlock is called, methods PrepareProposal and ProcessProposal, may be called several times. See CometBFT’s expected behavior for details on the possible call sequences of these methods.

    Mempool methods

    Info methods

    State-sync methods

    State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying state machine (application) snapshots instead of replaying historical blocks. For more details, see the state sync documentation.

    New nodes discover and request snapshots from other nodes in the P2P network. A CometBFT node that receives a request for snapshots from a peer will call ListSnapshots on its Application. The Application returns the list of locally available snapshots. Note that the list does not contain the actual snapshots but metadata about them: height at which the snapshot was taken, application-specific verification data and more (see snapshot data type for more details). After receiving a list of available snapshots from a peer, the new node can offer any of the snapshots in the list to its local Application via the OfferSnapshot method. The Application can check at this point the validity of the snapshot metadata.

    Snapshots may be quite large and are thus broken into smaller “chunks” that can be assembled into the whole snapshot. Once the Application accepts a snapshot and begins restoring it, CometBFT will fetch snapshot “chunks” from existing nodes. The node providing “chunks” will fetch them from its local Application using the LoadSnapshotChunk method.

    As the new node receives “chunks” it will apply them sequentially to the local application with ApplySnapshotChunk. When all chunks have been applied, the Application’s AppHash is retrieved via an Info query. To ensure that the sync proceeded correctly, CometBFT compares the local Application’s AppHash to the AppHash stored on the blockchain (verified via light client verification).

    In summary:

    Other methods

    Additionally, there is a Flush method that is called on every connection, and an Echo method that is used for debugging.

    More details on managing state across connections can be found in the section on Managing Application State.

    Proposal timeout

    PrepareProposal stands on the consensus algorithm critical path, i.e., CometBFT cannot make progress while this method is being executed. Hence, if the Application takes a long time preparing a proposal, the default value of TimeoutPropose might not be sufficient to accommodate the method’s execution and validator nodes might time out and prevote nil. The proposal, in this case, will probably be rejected and a new round will be necessary.

    Timeouts are automatically increased for each new round of a height and, if the execution of PrepareProposal is bound, eventually TimeoutPropose will be long enough to accommodate the execution of PrepareProposal. However, relying on this self adaptation could lead to performance degradation and, therefore, operators are suggested to adjust the initial value of TimeoutPropose in CometBFT’s configuration file, in order to suit the needs of the particular application being deployed.

    This is particularly important if applications implement immediate execution. To implement this technique, proposers need to execute the block being proposed within PrepareProposal, which could take longer than TimeoutPropose.

    Deterministic State-Machine Replication

    ↑ Back to Outline

    ABCI applications must implement deterministic finite-state machines to be securely replicated by the CometBFT consensus engine. This means block execution must be strictly deterministic: given the same ordered set of transactions, all nodes will compute identical responses, for all successive BeginBlock, DeliverTx, EndBlock, and Commit calls. This is critical because the responses are included in the header of the next block, either via a Merkle root or directly, so all nodes must agree on exactly what they are.

    For this reason, it is recommended that application state is not exposed to any external user or process except via the ABCI connections to a consensus engine like CometBFT. The Application must only change its state based on input from block execution (BeginBlock, DeliverTx, EndBlock, and Commit calls), and not through any other kind of request. This is the only way to ensure all nodes see the same transactions and compute the same results.

    Applications that implement immediate execution (execute the blocks that are about to be proposed, in PrepareProposal, or that require validation, in ProcessProposal) produce a new candidate state before a block is decided. The state changes caused by processing those proposed blocks must never replace the previous state until FinalizeBlock confirms that the proposed block was decided and Commit is invoked for it.

    The same is true to Applications that quickly accept blocks and execute the blocks optimistically in parallel with the remaining consensus steps to save time during block execution (BeginBlock, DeliverTx, EndBlock, and Commit calls); they must only apply state changes in Commit.

    If there is some non-determinism in the state machine, consensus will eventually fail as nodes disagree over the correct values for the block header. The non-determinism must be fixed and the nodes restarted.

    Sources of non-determinism in applications may include:

    See #56 for the original discussion.

    Note that some methods (Query, DeliverTx) return non-deterministic data in the form of Info and Log fields. The Log is intended for the literal output from the Application’s logger, while the Info is any additional info that should be returned. These are the only fields that are not included in block header computations, so we don’t need agreement on them. All other fields in the Response* must be strictly deterministic.

    Events

    ↑ Back to Outline

    Methods BeginBlock, DeliverTx and EndBlock include an events field in their Response*. Applications may respond to this ABCI 2.0 method with an event list for each executed transaction, and a general event list for the block itself. Events allow applications to associate metadata with transactions and blocks. Events returned via these ABCI methods do not impact the consensus algorithm in any way and instead exist to power subscriptions and queries of CometBFT state.

    An Event contains a type and a list of EventAttributes, which are key-value string pairs denoting metadata about what happened during the method’s (or transaction’s) execution. Event values can be used to index transactions and blocks according to what happened during their execution.

    Each event has a type which is meant to categorize the event for a particular Response* or Tx. A Response* or Tx may contain multiple events with duplicate type values, where each distinct entry is meant to categorize attributes for a particular event. Every key and value in an event’s attributes must be UTF-8 encoded strings along with the event type itself.

    message Event {
  string                  type       = 1;
  repeated EventAttribute attributes = 2;
}

    The attributes of an Event consist of a key, a value, and an index flag. The index flag notifies the CometBFT indexer to index the attribute. The value of the index flag is non-deterministic and may vary across different nodes in the network.

    message EventAttribute {
  string key   = 1;
  string value = 2;
  bool   index = 3;  // nondeterministic
}

    Example:

     abci.ResponseDeliverTx{
  // ...
 Events: []abci.Event{
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "balance", Value: "...", Index: true},
   },
  },
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: false},
    abci.EventAttribute{Key: "balance", Value: "...", Index: false},
   },
  },
  {
   Type: "validator.slashed",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: false},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "reason", Value: "...", Index: true},
   },
  },
  // ...
 },
}

    Evidence

    ↑ Back to Outline

    CometBFT’s security model relies on the use of evidences of misbehavior. An evidence is an irrefutable proof of malicious behavior by a network participant. It is the responsibility of CometBFT to detect such malicious behavior. When malicious behavior is detected, CometBFT will gossip evidences of misbehavior to other nodes and commit the evidences to the chain once they are verified by a subset of validators. These evidences will then be passed on to the Application through ABCI++. It is the responsibility of the Application to handle evidence of misbehavior and exercise punishment.

    There are two forms of evidence: Duplicate Vote and Light Client Attack. More information can be found in either data structures or accountability.

    EvidenceType has the following protobuf format:

    enum EvidenceType {
  UNKNOWN               = 0;
  DUPLICATE_VOTE        = 1;
  LIGHT_CLIENT_ATTACK   = 2;
}

    Errors

    ↑ Back to Outline

    The Query, CheckTx and DeliverTx methods include a Code field in their Response*. Field Code is meant to contain an application-specific response code. A response code of 0 indicates no error. Any other response code indicates to CometBFT that an error occurred.

    These methods also return a Codespace string to CometBFT. This field is used to disambiguate Code values returned by different domains of the Application. The Codespace is a namespace for the Code.

    Methods Echo, Info, BeginBlock, EndBlock, Commit and InitChain do not return errors. An error in any of these methods represents a critical issue that CometBFT has no reasonable way to handle. If there is an error in one of these methods, the Application must crash to ensure that the error is safely handled by an operator.

    The handling of non-zero response codes by CometBFT is described below.

    CheckTx

    When CometBFT receives a ResponseCheckTx with a non-zero Code, the associated transaction will not be added to CometBFT’s mempool or it will be removed if it is already included.

    DeliverTx

    The DeliverTx ABCI method delivers transactions from CometBFT to the application. When CometBFT receives a ResponseDeliverTx with a non-zero Code, the response code is logged. The transaction was already included in a block, so the Code does not influence consensus.

    Query

    When CometBFT receives a ResponseQuery with a non-zero Code, this code is returned directly to the client that initiated the query.

    Decorative Orb