Upgrading from CometBFT v0.38.x to v1.0

    This guide outlines the key updates and adjustments needed when upgrading from CometBFT v0.38.x to v1.0. It highlights significant changes and provides references to additional documentation for a smoother transition.

    The introduction of CometBFT v1.0 brings numerous new features. This guide outlines the most significant additions and changes.

    For additional information about changes please refer to the CHANGELOG.md and UPGRADING.md documents.

    NOTE: It is essential to emphasize that this is a major version bump upgrade (v0.38.x -> v1.0). Therefore, a coordinated upgrade is necessary. A mixed network with v0.38.x and v1.0 nodes is not supported.

    Proposer-Based Timestamps (PBTS)

    CometBFT v1.0 contains a new algorithm for generating and verifying block timestamps called Proposer-Based Timestamps (PBTS). The existing algorithm used in CometBFT releases prior to v1.0, called BFT Time is kept for backwards compatibility.

    Upgrading to v1.0 does not automatically switch the chain from BFT Time to PBTS; rather a ConsensusParam called PbtsEnableHeight was introduced and can be set to a future height to transition from BFT Time to PBTS.

    This flexible mechanism allows chains disentangle the upgrade to v1.0 from the transition in the algorithm used for block times.

    For further information, please check the PBTS documentation.

    ABCI Local Client Mutex

    CometBFT’s existing ABCI local client (used when CometBFT and the application run in the same process) is prevented from making concurrent calls to ABCI implementations by virtue of a mutex taken by the client’s implementation.

    In the v1.0 release, two additional local ABCI clients have been added.

    When using either of the new ABCI local clients, the application is now responsible for coordinating concurrent ABCI calls in order to prevent race conditions or non-deterministic behavior.

    If you are uncertain about how to ensure these guarantees in your application, it is strongly recommended to continue using the existing ABCI local client, which relies on a single global mutex.

    Protobuf Changes

    Several major changes have been implemented relating to the Protobuf definitions:

    CometBFT now makes use of the cometbft.* Protobuf definitions under proto/cometbft. This is a breaking change for all users who rely on serialization of the Protobuf type paths, such as integrators who serialize CometBFT’s Protobuf data types into Any typed fields. For example, the tendermint.types.Block type in CometBFT v0.38.x is now accessible as cometbft.types.v1.Block (see the next point in the list for details on versioning).

    All CometBFT Protobuf packages include a version whose number will be independent of the CometBFT version. As mentioned in (1), the tendermint.types.Block type is now available under cometbft.types.v1.Block - the v1 in the type path indicates the version of the types package used by this version of CometBFT. The Protobuf definitions that are wire-level compatible (but not type path-compatible) with CometBFT v0.34.x, v0.37.x and v0.38.x, where breaking changes were introduced, are available under v1beta*-versioned types.

    For example:

    All Go code generated from the cometbft.* types is now available under the api directory. This directory is also an independently versioned Go module. This code is still generated using the Cosmos SDK’s gogoproto fork.

    Several ABCI-related types were renamed in order to align with Buf guidelines. Request* and Response* were renamed to *Request and *Response (e.g. RequestQuery was renamed to QueryRequest).

    See the CometBFT Protobufs README section for more details.

    Crypto

    BLS12-381 curve

    CometBFT v1.0 adds support for BLS12-381 keys. Since the implementation needs cgo and brings in new dependencies, a build flag bls12381 needs to be used if you want to enable it.

    Mempool

    nop mempool

    CometBFT v1.0 provides an option of using a nop (no-op) mempool which, if selected via configuration, turns off all mempool-related functionality in CometBFT (e.g. ability to receive transactions, transaction gossip). CometBFT then expects applications to manage transactions and provide transactions to include in a new block when it calls the PrepareProposal ABCI method, and that application developers will use some external means for disseminating their transactions.

    If you want to use the nop mempool, change mempool’s type to nop in your config.toml file:

    [mempool]
# - "nop"   : nop-mempool (short for no operation; the ABCI app is responsible
# for storing, disseminating and proposing txs). "create_empty_blocks=false"
# is not supported.
type = "nop"

    Internal CheckTx Go API changes

    The Mempool interface was modified on CheckTx method. Note that this interface is meant for internal use only, so you should be aware of these changes only if you happen to call these methods directly.

    CheckTx’s signature changed from CheckTx(tx types.Tx, cb func(*abci.ResponseCheckTx), txInfo TxInfo) error to CheckTx(tx types.Tx, sender p2p.ID) (*abcicli.ReqRes, error). The method used to take a callback function cb to be applied to the ABCI CheckTx response and a TxInfo structure containing a sender. Now the sender ID is passed directly and CheckTx returns the ABCI response of type *abcicli.ReqRes, on which one can apply any callback manually.

    For example:

    reqRes, err := CheckTx(tx, sender)
// check `err` here
cb(reqRes.Response.GetCheckTx())

    The *abcicli.ReqRes structure that CheckTx returns has a callback to process the response already set (namely, the function handleCheckTxResponse). The callback will be invoked internally when the response is ready. We need only to wait for it; for example:

    reqRes, err := CheckTx(tx, sender)
// check `err` here
reqRes.Wait()

    RPC

    Versioned API

    The RPC API is now versioned, with the existing RPC being available under both the / path (as in CometBFT v0.38.x) and a /v1 path.

    Although invoking methods without specifying the version is still supported for now, support will be dropped in future releases and users are encouraged to use the versioned approach. For example, instead of curl localhost:26657/block?height=5, use curl localhost:26657/v1/block?height=5.

    Websocket endpoint

    The /websocket endpoint path is no longer configurable in the client or server. Creating an RPC client now takes the form:

    // The WebSocket endpoint in the following example is assumed to be available
// at http://localhost:26657/v1/websocket
rpcClient, err := client.New("http://localhost:26657/v1")

    For more information please see this PR

    Go API Surface Area

    Package Internalization

    As per ADR 109, the following packages that were publicly accessible in CometBFT v0.38.x were moved into the internal directory:

    If you depend on any of these packages and wish for them to be made public again, please submit an issue outlining your use case. We will then assess the most effective way to assist you.

    Legacy gRPC

    Broadcast API removed

    CometBFT v0.38.x offered a simplistic gRPC support with only one method equivalent to the /broadcast_tx_commit endpoint. This has been removed from CometBFT v1.0 in favor of the new Data Companion gRPC services.

    The following configuration parameters in config.toml are not valid anymore in v1.0:

    [rpc]

grpc_laddr = ""
grpc_max_open_connections = 900

    Data Companion API

    CometBFT v1.0 introduces support for a new Data Companion Pull API as specified in ADR-101

    gRPC services

    There is a whole new section in the config.toml to support the new Data Companion gRPC services.

    In the CometBFT v1.0 configuration file, if the laddr parameter in the [grpc] section is not specified, all the services will be disabled. If an address is specified, the node operator can selectively choose which services individually should be enabled or disabled.

    Pruning service

    CometBFT offers a privileged gRPC endpoint for the pruning service, which is separate from the regular gRPC endpoint and requires its own configuration and activation. These “privileged” services have the ability to manage the storage on the node. The new pruning service allows for the removal of older blocks, block results, and block and transaction indexed data.

    A crucial concept that can impact node pruning is the “retain height.” The retain height specifies the specific height from which data can be safely deleted from the node’s storage. By taking the retain height into account, nodes can efficiently manage their storage usage and ensure that they only retain the necessary data for their operations. This is crucial as storage space is a limited resource, and nodes with limited storage may struggle to keep up with the blockchain’s growth.

    To enable (or disable) and control the Pruning Service please refer the gRPC section in the config.toml file.

    Please refer to the Data Companion API documentation for additional information.

    CLI Subcommands

    Replay and Replay-console removed

    The replay and replay-console subcommands were removed (#1170).

    New ‘key-type’ flag

    Added support for all key types in gen-validator subcommand.

    The key-type flag has also been added to other subcommands such as init, unsafe-reset-priv-validator, unsafe-reset-all and start

    Database

    Pebble Database Support

    CometBFT v1.0 adds support for Pebble as a database backend.

    Default Database Change

    The default database has been changed from goleveldb to pebbledb.

    A default config.toml file will have pebbledb as the default db_backend value.

    db_backend = "pebbledb"

    Deprecated Databases

    CometBFT v1.0 is upgrading to cometbft-db v1.0.1, which deprecates cleveldb and boltdb.

    If you are currently using any of these databases, please note that we discourage their use, as we plan to discontinue support in future releases.

    Configuration File

    Review the config.toml official documentation, as several parameters have been added, modified, or deprecated.

    You may need to add, remove or modify parameters to optimize your node’s performance or properly run a node.

    New Parameters

    Check for newly introduced parameters in the CometBFT v1.0 configuration file.

    [consensus] section

    The peer_gossip_intraloop_sleep_duration has been added.

    The default value in v1.0 is "0s":

    peer_gossip_intraloop_sleep_duration = "0s"

    [grpc] section

    A new [grpc] section has been introduced to support all the parameters for the new Data Companion API services ,

    #######################################################
###       gRPC Server Configuration Options         ###
#######################################################

#
# Note that the gRPC server is exposed unauthenticated. It is critical that
# this server not be exposed directly to the public internet. If this service
# must be accessed via the public internet, please ensure that appropriate
# precautions are taken (e.g. fronting with a reverse proxy like nginx with TLS
# termination and authentication, using DDoS protection services like
# CloudFlare, etc.).
#

[grpc]

# TCP or UNIX socket address for the RPC server to listen on. If not specified,
# the gRPC server will be disabled.
laddr = ""

#
# Each gRPC service can be turned on/off, and in some cases configured,
# individually. If the gRPC server is not enabled, all individual services'
# configurations are ignored.
#

# The gRPC version service provides version information about the node and the
# protocols it uses.
[grpc.version_service]
enabled = true

# The gRPC block service returns block information
[grpc.block_service]
enabled = true

# The gRPC block results service returns block results for a given height. If no height
# is given, it will return the block results from the latest height.
[grpc.block_results_service]
enabled = true

#
# Configuration for privileged gRPC endpoints, which should **never** be exposed
# to the public internet.
#
[grpc.privileged]
# The host/port on which to expose privileged gRPC endpoints.
laddr = ""

#
# Configuration specifically for the gRPC pruning service, which is considered a
# privileged service.
#
[grpc.privileged.pruning_service]

# Only controls whether the pruning service is accessible via the gRPC API - not
# whether a previously set pruning service retain height is honored by the
# node. See the [storage.pruning] section for control over pruning.
#
# Disabled by default.
enabled = false

    [storage] section

    Experimental Key Layouts

    The experimental_db_key_layout parameter has been added to the configuration file.

    The default value is v1:

    # The representation of keys in the database.
# The current representation of keys in Comet's stores is considered to be v1
# Users can experiment with a different layout by setting this field to v2.
# Note that this is an experimental feature and switching back from v2 to v1
# is not supported by CometBFT.
# If the database was initially created with v1, it is necessary to migrate the DB
# before switching to v2. The migration is not done automatically.
# v1 - the legacy layout existing in CometBFT prior to v1.
# v2 - Order preserving representation ordering entries by height.
experimental_db_key_layout = "v1"

    Database Compaction

    Two new parameters were added to support database compaction. The compact and the compaction_interval can be customized to allow compaction and how frequent that should be triggered (number of blocks interval).

    # If set to true, CometBFT will force compaction to happen for databases that support this feature.
# and save on storage space. Setting this to true is most benefits when used in combination
# with pruning as it will physically delete the entries marked for deletion.
# false by default (forcing compaction is disabled).
compact = false

    # To avoid forcing compaction every time, this parameter instructs CometBFT to wait
# the given amount of blocks to be pruned before triggering compaction.
# It should be tuned depending on the number of items. If your retain height is 1 block,
# it is too much of an overhead to try compaction every block. But it should also not be a very
# large multiple of your retain height as it might occur bigger overheads.
compaction_interval = "1000"

    Pruning

    CometBFT v1.0 implemented support for a pruning mechanism

    A new parameter interval was added to control the time period between automated background pruning operations.

    # The time period between automated background pruning operations.
interval = "10s"

    Other parameters were also introduced to configure the Data Companion pruning service

    #
# Storage pruning configuration relating only to the data companion.
#
[storage.pruning.data_companion]

# Whether automatic pruning respects values set by the data companion. Disabled
# by default. All other parameters in this section are ignored when this is
# disabled.
#
# If disabled, only the application retain height will influence block pruning
# (but not block results pruning). Only enabling this at a later stage will
# potentially mean that blocks below the application-set retain height at the
# time will not be available to the data companion.
enabled = false

# The initial value for the data companion block retain height if the data
# companion has not yet explicitly set one. If the data companion has already
# set a block retain height, this is ignored.
initial_block_retain_height = 0

# The initial value for the data companion block results retain height if the
# data companion has not yet explicitly set one. If the data companion has
# already set a block results retain height, this is ignored.
initial_block_results_retain_height = 0

    Removed Parameters

    Some parameters have been removed in v1.0 and are not applicable anymore.

    [mempool] section

    The max_batch_bytes has been removed.

    max_batch_bytes = 0

    Changed Parameters

    Some sections changed their structure or naming conventions. Ensure you adapt your configuration accordingly.

    [p2p] section

    The flush_throttle_timeout default value has been lowered.

    Default configuration in v0.38.x:

    # Time to wait before flushing messages out on the connection
flush_throttle_timeout = "100ms"

    Default configuration in v1.0:

    # Time to wait before flushing messages out on the connection
flush_throttle_timeout = "10ms"

    [mempool] section

    The max_txs_bytes default value has been lowered.

    Default configuration in v0.38.x:

    max_txs_bytes = 1073741824

    Default configuration in v1.0:

    max_txs_bytes = 67108864

    Deprecated Parameters

    Some parameters have been deprecated in v1.0 and will be removed in future releases.

    [consensus] section

    The skip_timeout_commit has been deprecated in CometBFT v1.0. In a future release it will be removed in favor of timeout_commit=0

    # Deprecated: set `timeout_commit` to 0 instead.
skip_timeout_commit = false

    Genesis File

    The format and structure of the genesis file changed in v1.0. Review the official documentation for v1.0 to identify any new fields or modified data types that need to be included.

    After updating your genesis file, validate the file’s structure and ensure compatibility with v1.0. This step is crucial to avoid issues during node initialization.

    PTBS Synchrony Parameters

    There are two new parameters related to PBTS, the precision and the message delay. For more information about these parameters please see the Proposer-Based Timestamps section in this document.

        "synchrony": {
        "precision": "505000000",
        "message_delay": "15000000000"
    },

    ABCI Consensus Parameters Deprecated

    The ABCI consensus parameters have been deprecated.

        "abci": {
      "vote_extensions_enable_height": "0"
    }

    Feature Consensus Parameters Added

    Please use the new Feature parameters. You can specify two parameters, one related to vote extensions, and the other for PBTS (check the documentation here for more information).

        "feature": {
      "vote_extensions_enable_height": "0",
      "pbts_enable_height": "0"
    }

    Block Consensus Parameters Changes

    The Block consensus parameters default values have been updated.

    In v0.38.x, the default values were:

        "block": {
      "max_bytes": "22020096",
      "max_gas": "-1"
    },

    And in CometBFT v1.0 the default parameters are:

        "block": {
      "max_bytes": "4194304",
      "max_gas": "10000000"
    },

    For additional context on the block value updates please refer to this PR.

    Additional Resources

    By following this guide, you should be well-prepared to successfully upgrade from CometBFT v0.38.x to v1.0. Ensure you stay updated with community feedback and best practices post-upgrade.

    Release Notes

    Review the official CHANGELOG for detailed information on changes and improvements in v1.0.

    Documentation

    Consult the CometBFT v1 documentation for specifics on configuration parameters and API usage.

    Community Support

    Engage with the CometBFT community for assistance or to share experiences regarding the upgrade process.

    Decorative Orb