This commit integrates metrics capabilities into the explorer backend. The provided metrics include the average, minimum, and maximum values for the following:
- Total Gas Usage: Represents overall gas consumption.
- WASM Gas Usage: Pertains to gas usage for WASM contract calls.
- ZK Circuits Gas Usage: Indicates specific gas usage for ZK circuit operations.
- Signatures Gas Usage: Covers gas usage for transaction signatures.
- Deployments Gas Usage: Covers gas usage for transaction deployments.
Summary of Updates
Blocks Module:
- Updated `put_block` to store metrics for each transaction found in the block
Main Binary Crate:
- Added `calculate_tx_gas_data` to the `ExplorerDb` implementation, which calculates the gas data for a given transaction
- Introduced `get_latest_metrics`, which fetches the most current metrics from the metrics store
- Added code to deploy native contracts need to calculate transaction gas data
RPC Module:
- Added `statistics.get_metric_statistics` request to the RPC request handler
RPC Statistics Module:
- Implemented `statistics_get_metric_statistics` function that returns the latest metrics for the UI
Statistics Module:
- Introduced the `MetricStatistics` structure to represent gas data metrics in the service layer
- Implemented a constructor for `MetricStatistics` to initialize it with `GasMetrics`
- Updated the `to_json_array` method to convert `MetricStatistics` into a JSON array for UI data representation
- Implemented the `get_metrics_statistics` function in the `ExplorerDb` struct to fetch the most recent metrics from the database
Transaction Module:
- Added metrics and timestamp fields to the existing `TransactionRecord` struct
- Updated the `to_json_array` method of `TransactionRecord` to include conversions for metrics and timestamp
- Replaced the previous TransactionRecord From implementation with a ExplorerDb implementation `to_tx_record`, which converts a `Transaction` to a `TransactionRecord`
- Updated all uses of `TransactionRecord::from` to use `to_tx_record`
- Added auxiliary ExplorerDb implements to support metrics:
- `get_tx_block_info`: Retrieves the `BlockInfo` associated with a given transaction hash
- `get_block_info`: Fetches the `BlockInfo` associated with a given `HeaderHash`
This commit introduces a metrics store that serves as entry point for managing metrics, providing an API for fetching, inserting, and resetting metric data backed by a Sled database. The store organizes data into separate trees, including main storage for gas metrics by a defined time interval, a tree containing metrics by height for handling reorgs, and a transaction-specific gas data tree. It uses an overlay pattern for write operations, enabling unified management of metrics by delegating tasks such as adding metrics and handling reorgs to the MetricsStoreOverlay.
Additionally, the commit introduces the GasMetrics struct, which is used for managing gas metrics across all transactions in the metrics store. This struct maintains running totals, min/max extrema, and transaction counts, allowing efficient calculations of total gas, WASM gas, ZK circuit gas, and signature gas without needing to iterate through previous transactions. With O(1) performance for calculating averages and updating min/max values, GasMetrics builds a comprehensive view of gas metrics throughout the blockchain's history. It includes methods for creating new instances, calculating average gas usage, and adding new gas data in real-time.
Furthermore, the GasMetricsKey struct is introduced to facilitate storing and retrieving metrics by time interval. The initial implementation stores metrics on an hourly basis, with flexibility for future adjustments to different intervals with minimal code changes. This struct provides methods for creating, serializing, and deserializing keys from various time representations, including [Timestamp], u64 timestamps, and &str timestamps. It also features conversion methods to ensure compatibility with the Sled database.
Test coverage has been added to validate functionality across various scenarios. Tests include inserting gas metrics, searching metrics by hour, fetching metrics by timestamp string, transaction gas usage, and resetting metrics under different conditions. Additionally, functions were added to load test data, simulating random, fixed, and reset scenarios while generating varied gas data.
Core Updates Summary:
- Created the `MetricsStore` struct to manage metric storage, providing an API for fetching, inserting, and resetting metrics backed by a Sled database.
- Developed `MetricsStoreOverlay` to facilitate write operations, including adding and reverting metrics for handling reorgs.
- Introduced the `GasMetrics` struct to encapsulate gas metrics across transactions, with methods to efficiently add data and compute extrema and averages.
- Implemented `GasMetricsKey` for storing and retrieving fee metrics based on hourly intervals.
- Introduced the `GasMetricsKeySource` trait for unified key creation using various time representations: `[Timestamp]`, `u64` timestamps, or `&str` timestamps.
Test Functions:
- `test_insert_gas_metrics`: Verifies correctness of inserted gas metrics into the main tree.
- `test_insert_by_height_gas_metrics`: Tests gas metrics insertion into the metrics by height tree.
- `test_search_metrics_by_hour`: Validates searching gas metrics by hour.
- `test_get_metrics_by_timestamp_str`: Ensures correct retrieval of metrics by timestamp string.
- `test_tx_gas_data`: Verifies transaction gas data insertion and retrieval, including handling of non-existent transactions.
- `test_reset_metrics_within_height_range`: Tests resetting metrics within a specified height range.
- `test_reset_metrics_height_to_0`: Ensures graceful handling of resetting the metrics store to height 0.
- `test_reset_metrics_beyond_height`: Validates behavior when resetting beyond available metrics.
- `test_reset_metrics_at_height`: Tests resetting metrics at the last available height.
- `test_reset_empty_store`: Confirms that resetting an empty store is handled without errors.
Test Data Load Functions:
- `load_random_metrics`: Simulates loading random gas metrics data at hourly intervals from height 0 as the genesis block.
- `load_fixed_metrics`: Loads fixed test data with timestamps.
- `load_reset_metrics`: Returns accumulated gas metrics at a specified reset height.
- `load_tx_gas_data`: Loads test transaction gas data, returning pairs of transaction hashes and gas data.
- `random_gas_data`: Generates varied gas data based on a provided seed value.
This commit introduces the initial implementation a test_util module, providing reusable testing utilities for the explorer.
The first version includes a function to initialize logging for test cases, which aids in debugging issues encountered during testing. This module will be expanded as we progress with both unit and integration testing.
Summary of Updates:
- Added the init_logger function that configures the logger based on a provided list of targets to ignore and the desired log level
This commit request introduces a sled-backed Block Explorer, replacing the SQL-based implementation.
Summary of updates:
- Transitioned the block explorer from SQL to a sled-based database implementation
- Introduced BlockExplorerDb struct that handles explorer database operations
- Established sled-based darkfid block synchronization for startup and subscription sync
- Removed rusqlite dependency
- Removed SQL create table scripts
- Updated app.py to handle not found as empty results instead of a thrown exception when calling rpc.get_block_transactions
