Metrics
CometBFT can report and serve the Prometheus metrics, which in their turn can be consumed by Prometheus collector(s).
This functionality is disabled by default.
To enable the Prometheus metrics, set instrumentation.prometheus=true in your
config file. Metrics will be served under /metrics on 26660 port by default.
Listen address can be changed in the config file (see
instrumentation.prometheus_listen_addr).
List of available metrics
The following metrics are available:
| Name | Type | Tags | Description |
|---|---|---|---|
| abci_connection_method_timing_seconds | Histogram | method, type | Timings for each of the ABCI methods |
| blocksync_syncing | Gauge | Either 0 (not block syncing) or 1 (syncing) | |
| consensus_height | Gauge | Height of the chain | |
| consensus_validators | Gauge | Number of validators | |
| consensus_validators_power | Gauge | validator_address | Total voting power of all validators |
| consensus_validator_power | Gauge | validator_address | Voting power of the node if in the validator set |
| consensus_validator_last_signed_height | Gauge | validator_address | Last height the node signed a block, if the node is a validator |
| consensus_validator_missed_blocks | Gauge | Total amount of blocks missed for the node, if the node is a validator | |
| consensus_missing_validators | Gauge | Number of validators who did not sign | |
| consensus_missing_validators_power | Gauge | Total voting power of the missing validators | |
| consensus_byzantine_validators | Gauge | Number of validators who tried to double sign | |
| consensus_byzantine_validators_power | Gauge | Total voting power of the byzantine validators | |
| consensus_block_interval_seconds | Histogram | Time between this and last block (Block.Header.Time) in seconds | |
| consensus_rounds | Gauge | Number of rounds | |
| consensus_num_txs | Gauge | Number of transactions | |
| consensus_total_txs | Gauge | Total number of transactions committed | |
| consensus_block_parts | Counter | peer_id | Number of blockparts transmitted by peer |
| consensus_latest_block_height | Gauge | /status sync_info number | |
| consensus_block_size_bytes | Gauge | Block size in bytes | |
| consensus_step_duration_seconds | Histogram | step | Histogram of durations for each step in the consensus protocol |
| consensus_round_duration_seconds | Histogram | Histogram of durations for all the rounds that have occurred since the process started | |
| consensus_block_gossip_parts_received | Counter | matches_current | Number of block parts received by the node |
| consensus_quorum_prevote_delay | Gauge | proposer_address | Interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum |
| consensus_full_prevote_delay | Gauge | proposer_address | Interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted |
| consensus_vote_extension_receive_count | Counter | status | Number of vote extensions received |
| consensus_proposal_receive_count | Counter | status | Total number of proposals received by the node since process start |
| consensus_proposal_create_count | Counter | Total number of proposals created by the node since process start | |
| consensus_round_voting_power_percent | Gauge | vote_type | A value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round |
| consensus_late_votes | Counter | vote_type | Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in. |
| consensus_duplicate_vote | Counter | Number of times we received a duplicate vote. | |
| consensus_duplicate_block_part | Counter | Number of times we received a duplicate block part. | |
| consensus_proposal_timestamp_difference | Histogram | is_timely | Difference between the timestamp in the proposal message and the local time of the validator at the time it received the message. |
| p2p_message_send_bytes_total | Counter | message_type | Number of bytes sent to all peers per message type |
| p2p_message_receive_bytes_total | Counter | message_type | Number of bytes received from all peers per message type |
| p2p_peers | Gauge | Number of peers node’s connected to | |
| p2p_peer_pending_send_bytes | Gauge | peer_id | Number of pending bytes to be sent to a given peer |
| p2p_recv_rate_limiter_delay | Counter | peer_id | Time in seconds spent sleeping by the receive rate limiter, in seconds. |
| p2p_send_rate_limiter_delay | Counter | peer_id | Time in seconds spent sleeping by the send rate limiter, in seconds. |
| mempool_lane_size | Counter | lane | Number of uncommitted transactions per lane |
| mempool_lane_bytes | Counter | lane | Number of used bytes per lane |
| mempool_size | Gauge | Number of uncommitted transactions in the mempool | |
| mempool_size_bytes | Gauge | Total size of the mempool in bytes | |
| mempool_tx_size_bytes | Histogram | Histogram of transaction sizes in bytes | |
| mempool_evicted_txs | Counter | Number of transactions that make it into the mempool and were later evicted for being invalid | |
| mempool_failed_txs | Counter | Number of transactions that failed to make it into the mempool for being invalid | |
| mempool_rejected_txs | Counter | Number of transactions that failed to make it into the mempool due to resource limits | |
| mempool_recheck_times | Counter | Number of times transactions are rechecked in the mempool | |
| mempool_already_received_txs | Counter | Number of times transactions were received more than once | |
| mempool_active_outbound_connections | Gauge | Number of connections being actively used for gossiping transaction (experimental) | |
| mempool_recheck_duration_seconds | Gauge | Cumulative time spent rechecking transactions | |
| state_consensus_param_updates | Counter | Number of consensus parameter updates returned by the application since process start | |
| state_validator_set_updates | Counter | Number of validator set updates returned by the application since process start | |
| state_pruning_service_block_retain_height | Gauge | Accepted block retain height set by the data companion | |
| state_pruning_service_block_results_retain_height | Gauge | Accepted block results retain height set by the data companion | |
| state_pruning_service_tx_indexer_retain_height | Gauge | Accepted transactions indices retain height set by the data companion | |
| state_pruning_service_block_indexer_retain_height | Gauge | Accepted blocks indices retain height set by the data companion | |
| state_application_block_retain_height | Gauge | Accepted block retain height set by the application | |
| state_block_store_base_height | Gauge | First height at which a block is available | |
| state_abciresults_base_height | Gauge | First height at which ABCI results are available | |
| state_tx_indexer_base_height | Gauge | First height at which tx indices are available | |
| state_block_indexer_base_height | Gauge | First height at which block indices are available | |
| state_store_access_duration_seconds | Histogram | method | Duration of accesses to the state store labeled by which method was called on the store |
| state_fire_block_events_delay_seconds | Gauge | Duration of event firing related to a new block | |
| statesync_syncing | Gauge | Either 0 (not state syncing) or 1 (syncing) |
Useful queries
Percentage of missing + byzantine validators:
((consensus\_byzantine\_validators\_power + consensus\_missing\_validators\_power) / consensus\_validators\_power) * 100
