This commit introduces a **Contract Metadata Store**, serving as a data layer API for managing contract metadata and associated source code files. The storage utilizes a Sled database and organizes data into separate trees, including primary metadata storage for contract details and a tree for contract source code, organized by file paths prefixed with a `ContractId`. The implementation follows an overlay pattern for write operations, enabling unified management of metadata and source code by delegating tasks such as adding, updating, and deleting records to the `ContractMetadataStoreOverlay`.
Furthermore, this commit adds the `ContractMetaData` struct, which is used to store additional information for contracts that is not available on-chain. This data is indexed by `ContractId`. The struct encapsulates basic metadata fields, such as the name and description of a contract, and provides methods for creating new metadata instances.
In addition, the `ContractSourceFile` struct was added, which represents a source file containing its file path and its contents.
Finally, we added test coverage to validate functionality across various scenarios, such as metadata insertion and retrieval, source code insertion, and fetching source paths and contents.
Introduced Core Structures and Methods:
ContractMetadataStore:
- Created the `ContractMetadataStore` struct to manage contract metadata and source code storage, providing an API for fetching and inserting associated data, all backed by a Sled database.
ContractMetadataStoreOverlay:
- Developed the `ContractMetadataStoreOverlay` to facilitate write operations, including adding metadata and source code, as well as deleting existing source code.
ContractMetaData:
- Introduced the `ContractMetaData` struct, which represents contract metadata, containing additional contract information that is not stored on-chain.
ContractSourceFile:
- Introduced the `ContractSourceFile` struct to encapsulate contract source code, containing its file path and its content.
Added Test Functions:
- `test_add_contract_source`: Verifies the correctness of storing contract source code.
- `test_get_contract_source`: Validates the retrieval of a contract source code from the store.
- `test_add_metadata`: Tests the addition of contract metadata to the store.
Added Load Test Data Functions:
- `load_source_code`:Loads the source code test data.
Running Tests:
cargo test contract_meta_store::tests
This commit updates the Explorer application’s UI, providing a preliminary re-theming inspired by the Darkfi Book site. The updated design offers a familiar and visually appealing interface, along with improvements to responsiveness and usability on larger displays. These updates include changes to styling, layout, and modifications to key pages, such as the home page, block page, and transaction page.
This implementation represents the initial stage of improving the UI, making it more appealing for development and demos. It is not final, as further improvements will be made iteratively.
This commit introduces the ExplorerService, a dedicated service layer for the Explorer application that acts as a bridge between the JSON-RPC and database layers. It encapsulates application logic, orchestrates operations, and transforms database data into structured responses suitable for RPC callers, as well as other use cases.
As the Explorer application is growing in complexity, we’ve observed that blending RPC functionality with the database layer (ExplorerDb) introduces functionality into these layers that is not cohesively aligned. To address this, ExplorerService refines the application’s architecture by clearly defining and organizing application-specific responsibilities. This ensures that transformations, operation orchestration, contract source code processing, gas calculations, and other operations are performed within their appropriate layers, fostering a more cohesive and maintainable architecture over time.
This commit introduces an initial Makefile that facilitates the building and installation of the block explorer. The target directory for copying files is currently set to the blockchain-explorer directory but will be updated to the `darkfi` project directory once the explorer is integrated into the Darkfi workspace.
This comment introduces UI updates to display total network gas consumption statistics across all transactions, as well as individual transaction gas consumption. These updates enhance the understanding of gas consumption by providing insights that can help users set appropriate gas limits, identify peak usage and anomalies, manage costs, and optimize blockchain performance.
Update Summary:
- Explorer Home Page: The `index.html` file has been updated to include a new section that displays total network gas consumption statistics, covering total gas, WASM gas, ZK circuits gas, signatures gas, and deployments, along with their average, minimum, and maximum values.
- Transaction Details Page: The `transaction.html` file has been updated to include gas consumption metrics and timestamp for individual transactions.
- Index Route: The index route in `app.py` has been modified to retrieve metrics statistics.
- RPC Module: The `rpc.py` file has been updated by introducing a new function to retrieve metrics statistics from the blockchain-explorer daemon.
This commit integrates metrics capabilities into the explorer backend. The provided metrics include the average, minimum, and maximum values for the following:
- Total Gas Usage: Represents overall gas consumption.
- WASM Gas Usage: Pertains to gas usage for WASM contract calls.
- ZK Circuits Gas Usage: Indicates specific gas usage for ZK circuit operations.
- Signatures Gas Usage: Covers gas usage for transaction signatures.
- Deployments Gas Usage: Covers gas usage for transaction deployments.
Summary of Updates
Blocks Module:
- Updated `put_block` to store metrics for each transaction found in the block
Main Binary Crate:
- Added `calculate_tx_gas_data` to the `ExplorerDb` implementation, which calculates the gas data for a given transaction
- Introduced `get_latest_metrics`, which fetches the most current metrics from the metrics store
- Added code to deploy native contracts need to calculate transaction gas data
RPC Module:
- Added `statistics.get_metric_statistics` request to the RPC request handler
RPC Statistics Module:
- Implemented `statistics_get_metric_statistics` function that returns the latest metrics for the UI
Statistics Module:
- Introduced the `MetricStatistics` structure to represent gas data metrics in the service layer
- Implemented a constructor for `MetricStatistics` to initialize it with `GasMetrics`
- Updated the `to_json_array` method to convert `MetricStatistics` into a JSON array for UI data representation
- Implemented the `get_metrics_statistics` function in the `ExplorerDb` struct to fetch the most recent metrics from the database
Transaction Module:
- Added metrics and timestamp fields to the existing `TransactionRecord` struct
- Updated the `to_json_array` method of `TransactionRecord` to include conversions for metrics and timestamp
- Replaced the previous TransactionRecord From implementation with a ExplorerDb implementation `to_tx_record`, which converts a `Transaction` to a `TransactionRecord`
- Updated all uses of `TransactionRecord::from` to use `to_tx_record`
- Added auxiliary ExplorerDb implements to support metrics:
- `get_tx_block_info`: Retrieves the `BlockInfo` associated with a given transaction hash
- `get_block_info`: Fetches the `BlockInfo` associated with a given `HeaderHash`
This commit introduces a metrics store that serves as entry point for managing metrics, providing an API for fetching, inserting, and resetting metric data backed by a Sled database. The store organizes data into separate trees, including main storage for gas metrics by a defined time interval, a tree containing metrics by height for handling reorgs, and a transaction-specific gas data tree. It uses an overlay pattern for write operations, enabling unified management of metrics by delegating tasks such as adding metrics and handling reorgs to the MetricsStoreOverlay.
Additionally, the commit introduces the GasMetrics struct, which is used for managing gas metrics across all transactions in the metrics store. This struct maintains running totals, min/max extrema, and transaction counts, allowing efficient calculations of total gas, WASM gas, ZK circuit gas, and signature gas without needing to iterate through previous transactions. With O(1) performance for calculating averages and updating min/max values, GasMetrics builds a comprehensive view of gas metrics throughout the blockchain's history. It includes methods for creating new instances, calculating average gas usage, and adding new gas data in real-time.
Furthermore, the GasMetricsKey struct is introduced to facilitate storing and retrieving metrics by time interval. The initial implementation stores metrics on an hourly basis, with flexibility for future adjustments to different intervals with minimal code changes. This struct provides methods for creating, serializing, and deserializing keys from various time representations, including [Timestamp], u64 timestamps, and &str timestamps. It also features conversion methods to ensure compatibility with the Sled database.
Test coverage has been added to validate functionality across various scenarios. Tests include inserting gas metrics, searching metrics by hour, fetching metrics by timestamp string, transaction gas usage, and resetting metrics under different conditions. Additionally, functions were added to load test data, simulating random, fixed, and reset scenarios while generating varied gas data.
Core Updates Summary:
- Created the `MetricsStore` struct to manage metric storage, providing an API for fetching, inserting, and resetting metrics backed by a Sled database.
- Developed `MetricsStoreOverlay` to facilitate write operations, including adding and reverting metrics for handling reorgs.
- Introduced the `GasMetrics` struct to encapsulate gas metrics across transactions, with methods to efficiently add data and compute extrema and averages.
- Implemented `GasMetricsKey` for storing and retrieving fee metrics based on hourly intervals.
- Introduced the `GasMetricsKeySource` trait for unified key creation using various time representations: `[Timestamp]`, `u64` timestamps, or `&str` timestamps.
Test Functions:
- `test_insert_gas_metrics`: Verifies correctness of inserted gas metrics into the main tree.
- `test_insert_by_height_gas_metrics`: Tests gas metrics insertion into the metrics by height tree.
- `test_search_metrics_by_hour`: Validates searching gas metrics by hour.
- `test_get_metrics_by_timestamp_str`: Ensures correct retrieval of metrics by timestamp string.
- `test_tx_gas_data`: Verifies transaction gas data insertion and retrieval, including handling of non-existent transactions.
- `test_reset_metrics_within_height_range`: Tests resetting metrics within a specified height range.
- `test_reset_metrics_height_to_0`: Ensures graceful handling of resetting the metrics store to height 0.
- `test_reset_metrics_beyond_height`: Validates behavior when resetting beyond available metrics.
- `test_reset_metrics_at_height`: Tests resetting metrics at the last available height.
- `test_reset_empty_store`: Confirms that resetting an empty store is handled without errors.
Test Data Load Functions:
- `load_random_metrics`: Simulates loading random gas metrics data at hourly intervals from height 0 as the genesis block.
- `load_fixed_metrics`: Loads fixed test data with timestamps.
- `load_reset_metrics`: Returns accumulated gas metrics at a specified reset height.
- `load_tx_gas_data`: Loads test transaction gas data, returning pairs of transaction hashes and gas data.
- `random_gas_data`: Generates varied gas data based on a provided seed value.
This commit introduces the initial implementation a test_util module, providing reusable testing utilities for the explorer.
The first version includes a function to initialize logging for test cases, which aids in debugging issues encountered during testing. This module will be expanded as we progress with both unit and integration testing.
Summary of Updates:
- Added the init_logger function that configures the logger based on a provided list of targets to ignore and the desired log level
This commit request introduces a sled-backed Block Explorer, replacing the SQL-based implementation.
Summary of updates:
- Transitioned the block explorer from SQL to a sled-based database implementation
- Introduced BlockExplorerDb struct that handles explorer database operations
- Established sled-based darkfid block synchronization for startup and subscription sync
- Removed rusqlite dependency
- Removed SQL create table scripts
- Updated app.py to handle not found as empty results instead of a thrown exception when calling rpc.get_block_transactions
