This commit enhances the Makefile by adding the bundle_contracts_src as a phony target and adding clean-up for bundled contract code.
Summary of Updates:
- Added `bundle_contracts_src` to `.PHONY`
- Updated the `clean` target to remove the `native_contracts_src` directory
This commit introduces functionality for displaying native contract source code in the DarkFi explorer application, enabling users to view available contracts from the home page, access associated source files, and examine native contract Rust code and zk proofs. This enhancement provides accessibility to native contract information. To support these updates, we have added new pages to the UI, updated styles for rendering source-related pages, and implemented new RPC routes to facilitate contract-related code retrieval.
Additionally, we have restructured the application into Flask blueprints to address the growing complexity of the explorer and adhere to best practices, as app.py was becoming large with diverging functionality. This proactive change prepares us for future expansions, including better organization of graph functionalities. The organization also mirrors the rest of the explorer codebase by maintaining separate pages for contracts, transactions, and blocks, as well as corresponding RPC and service rust modules.
UI Support for Native Contract Source Code
- We have added source code functionality to the UI, Flask blueprints, and RPC routes defined in rpc.py, along with updated styles to effectively render source-related pages.
Style Updates:
- Expanded the color palette for rendering source files.
- Enhanced styles for displaying contract IDs and names in the native contracts table.
- Implemented layout elements for proper display of source code lists and content.
- Added styles for source file list links, including hover effects.
RPC Routes:
- New routes have been added to facilitate requests to the DarkFi daemon for retrieving native contracts, source code paths, and contract source.
UI Enhancements:
- A native contract section has been introduced on the explorer home page to display native contracts.
- A contract source list page allows users to view and click on source code files associated with a native contract.
- A dedicated contract source page renders the source for a specific path associated with a given contract.
Introducing Flask Blueprints
To better manage the growing complexity of the application, we introduced Flask blueprints that organize routes according to their functionality. This restructuring not only enhances readability but also aligns with best practices for Flask applications, facilitating future development.
App.py Updates:
- Routes have been moved to their respective blueprints.
Added Blueprints:
- `__init__.py` file initializes the explorer blueprint package and exposes various blueprints used by the application.
- `block.py` module defines a blueprint for handling block-related functionality.
- `contract.py` module defines a blueprint for managing contract-related functionality.
- explore.py` module manages general functionality related to the home page and search features.
- `transaction.py` module handles transaction-related functionality.
Python Documentation:
- We documented the Python code and updated existing documentation to adhere to Python's best practices for using docstrings, enhancing usability and integration with documentation tools.
Testing:
- Manual testing has been conducted through the UI to verify the functionality of the new RPC methods, ensuring they correctly retrieve and display contract-related information.
This commit introduces a new RPC contract module to the Explorerd implementation. The module adds three RPC methods to enhance the functionality of the explorer service:
- `contracts.get_native_contracts`: Retrieves the native contracts deployed in the DarkFi network.
- `contracts.get_contract_source_code_paths`: Fetches the source code paths for a specified contract.
- `contracts.get_contract_source`: Retrieves the source code content for a given contract and source path.
These additions will allow users to retrieve contract-related information more through the RPC interface, enhancing the overall functionality of the explorer.
### Testing
Manual testing has been performed through the UI to verify the correct functionality of the new RPC methods, ensuring they properly retrieve and display contract-related information.
We opted for manual testing because the underlying service and data layer already have reliable unit tests, providing confidence in the core functionality. This approach sufficiently validates the functionality of the new RPC methods within the overall explorer context.
This commit introduces a new module that enhances the functionality of the ExplorerService by adding functions for managing and retrieving contract-related data. The new methods include capabilities to retrieve contracts and native contracts, transforming them into ContractRecords suitable for handling by the RPC layer. In addition, we added functionality to handle contract metadata and source code.
Contract Metadata
The new module introduces functions to manage ContractMetadata, including fetching metadata for a given ContractId and adding ContractId and ContractMetaData pairs to the contract metadata store.
Contract Source Code
The module also handles operations on contract source code, such as fetching a list of source code file paths for a given ContractId, retrieving source code content for a specified ContractId and path, and adding source code for a ContractId from a provided tar file. The latter function extracts the tar archive and stores each source file with its path prefixed by the ContractId.
Auxiliary Functions
New auxiliary functions have been added for transforming contract details into a form suitable for the RPC layer, filtering contracts based on their ContractId using a provided function, and extracting source code files from a TAR archive.
Updates to Explorer Startup Sequence
The Explorer startup sequence has been updated to include the loading of native contract source code and metadata. This ensures the explorer prepared with the necessary contract data before beginning its operations.
Bundling Native Contract Source Code On Build
This commit introduces bundling of native contract source code during the build process. a new make target bundle_contracts_src to bundle native contract source code into tar files. This allows the explorer to load the latest native contract source code at startup, enhancing its ability to manage and display contract-related information. The current implementation redirects certain tar output to /dev/null to suppress verbose output. Future improvements will explore more elegant ways to manage this output in a subsequent commit.
Test Functions
We added test coverage to validate functionality across various scenarios, such as loading, storage, retrieval, and validation of contract metadata and contract source code. The primary goal is to validate the core service functionality to ensure the ExplorerService is properly handling contract-related data.
- test_add_metadata: Verifies the addition of ContractMetaData, ensuring the inserted metadata can be correctly retrieved and matches the expected values.
- test_load_native_contract_metadata: Ensures that native contract metadata can be loaded and correctly retrieved for all pre-defined native contract IDs.
- test_load_native_contracts: Validates both the loading and retrieval of native contract source code. It checks that the source paths and contents match the expected results in the corresponding tar archives.
- test_to_contract_record: Tests the transformation of contract details suitable for the RPC layer.
Auxiliary Test Functions
- setup: Sets up a temporary environment for testing by initializing an ExplorerService instance with an isolated sled database.
- verify_source_paths: Confirms that the source paths extracted from a contract tar archive match the source paths retrieved from the store a given ContractId.
- verify_source_content: Validates that the content of files from a contract tar archive matches the content retrieved from the store for a given ContractId.
Running Tests
cargo test contracts::tests
This commit introduces a **Contract Metadata Store**, serving as a data layer API for managing contract metadata and associated source code files. The storage utilizes a Sled database and organizes data into separate trees, including primary metadata storage for contract details and a tree for contract source code, organized by file paths prefixed with a `ContractId`. The implementation follows an overlay pattern for write operations, enabling unified management of metadata and source code by delegating tasks such as adding, updating, and deleting records to the `ContractMetadataStoreOverlay`.
Furthermore, this commit adds the `ContractMetaData` struct, which is used to store additional information for contracts that is not available on-chain. This data is indexed by `ContractId`. The struct encapsulates basic metadata fields, such as the name and description of a contract, and provides methods for creating new metadata instances.
In addition, the `ContractSourceFile` struct was added, which represents a source file containing its file path and its contents.
Finally, we added test coverage to validate functionality across various scenarios, such as metadata insertion and retrieval, source code insertion, and fetching source paths and contents.
Introduced Core Structures and Methods:
ContractMetadataStore:
- Created the `ContractMetadataStore` struct to manage contract metadata and source code storage, providing an API for fetching and inserting associated data, all backed by a Sled database.
ContractMetadataStoreOverlay:
- Developed the `ContractMetadataStoreOverlay` to facilitate write operations, including adding metadata and source code, as well as deleting existing source code.
ContractMetaData:
- Introduced the `ContractMetaData` struct, which represents contract metadata, containing additional contract information that is not stored on-chain.
ContractSourceFile:
- Introduced the `ContractSourceFile` struct to encapsulate contract source code, containing its file path and its content.
Added Test Functions:
- `test_add_contract_source`: Verifies the correctness of storing contract source code.
- `test_get_contract_source`: Validates the retrieval of a contract source code from the store.
- `test_add_metadata`: Tests the addition of contract metadata to the store.
Added Load Test Data Functions:
- `load_source_code`:Loads the source code test data.
Running Tests:
cargo test contract_meta_store::tests
This commit updates the Explorer application’s UI, providing a preliminary re-theming inspired by the Darkfi Book site. The updated design offers a familiar and visually appealing interface, along with improvements to responsiveness and usability on larger displays. These updates include changes to styling, layout, and modifications to key pages, such as the home page, block page, and transaction page.
This implementation represents the initial stage of improving the UI, making it more appealing for development and demos. It is not final, as further improvements will be made iteratively.
This commit introduces the ExplorerService, a dedicated service layer for the Explorer application that acts as a bridge between the JSON-RPC and database layers. It encapsulates application logic, orchestrates operations, and transforms database data into structured responses suitable for RPC callers, as well as other use cases.
As the Explorer application is growing in complexity, we’ve observed that blending RPC functionality with the database layer (ExplorerDb) introduces functionality into these layers that is not cohesively aligned. To address this, ExplorerService refines the application’s architecture by clearly defining and organizing application-specific responsibilities. This ensures that transformations, operation orchestration, contract source code processing, gas calculations, and other operations are performed within their appropriate layers, fostering a more cohesive and maintainable architecture over time.
This commit introduces an initial Makefile that facilitates the building and installation of the block explorer. The target directory for copying files is currently set to the blockchain-explorer directory but will be updated to the `darkfi` project directory once the explorer is integrated into the Darkfi workspace.
This comment introduces UI updates to display total network gas consumption statistics across all transactions, as well as individual transaction gas consumption. These updates enhance the understanding of gas consumption by providing insights that can help users set appropriate gas limits, identify peak usage and anomalies, manage costs, and optimize blockchain performance.
Update Summary:
- Explorer Home Page: The `index.html` file has been updated to include a new section that displays total network gas consumption statistics, covering total gas, WASM gas, ZK circuits gas, signatures gas, and deployments, along with their average, minimum, and maximum values.
- Transaction Details Page: The `transaction.html` file has been updated to include gas consumption metrics and timestamp for individual transactions.
- Index Route: The index route in `app.py` has been modified to retrieve metrics statistics.
- RPC Module: The `rpc.py` file has been updated by introducing a new function to retrieve metrics statistics from the blockchain-explorer daemon.
This commit integrates metrics capabilities into the explorer backend. The provided metrics include the average, minimum, and maximum values for the following:
- Total Gas Usage: Represents overall gas consumption.
- WASM Gas Usage: Pertains to gas usage for WASM contract calls.
- ZK Circuits Gas Usage: Indicates specific gas usage for ZK circuit operations.
- Signatures Gas Usage: Covers gas usage for transaction signatures.
- Deployments Gas Usage: Covers gas usage for transaction deployments.
Summary of Updates
Blocks Module:
- Updated `put_block` to store metrics for each transaction found in the block
Main Binary Crate:
- Added `calculate_tx_gas_data` to the `ExplorerDb` implementation, which calculates the gas data for a given transaction
- Introduced `get_latest_metrics`, which fetches the most current metrics from the metrics store
- Added code to deploy native contracts need to calculate transaction gas data
RPC Module:
- Added `statistics.get_metric_statistics` request to the RPC request handler
RPC Statistics Module:
- Implemented `statistics_get_metric_statistics` function that returns the latest metrics for the UI
Statistics Module:
- Introduced the `MetricStatistics` structure to represent gas data metrics in the service layer
- Implemented a constructor for `MetricStatistics` to initialize it with `GasMetrics`
- Updated the `to_json_array` method to convert `MetricStatistics` into a JSON array for UI data representation
- Implemented the `get_metrics_statistics` function in the `ExplorerDb` struct to fetch the most recent metrics from the database
Transaction Module:
- Added metrics and timestamp fields to the existing `TransactionRecord` struct
- Updated the `to_json_array` method of `TransactionRecord` to include conversions for metrics and timestamp
- Replaced the previous TransactionRecord From implementation with a ExplorerDb implementation `to_tx_record`, which converts a `Transaction` to a `TransactionRecord`
- Updated all uses of `TransactionRecord::from` to use `to_tx_record`
- Added auxiliary ExplorerDb implements to support metrics:
- `get_tx_block_info`: Retrieves the `BlockInfo` associated with a given transaction hash
- `get_block_info`: Fetches the `BlockInfo` associated with a given `HeaderHash`
This commit introduces a metrics store that serves as entry point for managing metrics, providing an API for fetching, inserting, and resetting metric data backed by a Sled database. The store organizes data into separate trees, including main storage for gas metrics by a defined time interval, a tree containing metrics by height for handling reorgs, and a transaction-specific gas data tree. It uses an overlay pattern for write operations, enabling unified management of metrics by delegating tasks such as adding metrics and handling reorgs to the MetricsStoreOverlay.
Additionally, the commit introduces the GasMetrics struct, which is used for managing gas metrics across all transactions in the metrics store. This struct maintains running totals, min/max extrema, and transaction counts, allowing efficient calculations of total gas, WASM gas, ZK circuit gas, and signature gas without needing to iterate through previous transactions. With O(1) performance for calculating averages and updating min/max values, GasMetrics builds a comprehensive view of gas metrics throughout the blockchain's history. It includes methods for creating new instances, calculating average gas usage, and adding new gas data in real-time.
Furthermore, the GasMetricsKey struct is introduced to facilitate storing and retrieving metrics by time interval. The initial implementation stores metrics on an hourly basis, with flexibility for future adjustments to different intervals with minimal code changes. This struct provides methods for creating, serializing, and deserializing keys from various time representations, including [Timestamp], u64 timestamps, and &str timestamps. It also features conversion methods to ensure compatibility with the Sled database.
Test coverage has been added to validate functionality across various scenarios. Tests include inserting gas metrics, searching metrics by hour, fetching metrics by timestamp string, transaction gas usage, and resetting metrics under different conditions. Additionally, functions were added to load test data, simulating random, fixed, and reset scenarios while generating varied gas data.
Core Updates Summary:
- Created the `MetricsStore` struct to manage metric storage, providing an API for fetching, inserting, and resetting metrics backed by a Sled database.
- Developed `MetricsStoreOverlay` to facilitate write operations, including adding and reverting metrics for handling reorgs.
- Introduced the `GasMetrics` struct to encapsulate gas metrics across transactions, with methods to efficiently add data and compute extrema and averages.
- Implemented `GasMetricsKey` for storing and retrieving fee metrics based on hourly intervals.
- Introduced the `GasMetricsKeySource` trait for unified key creation using various time representations: `[Timestamp]`, `u64` timestamps, or `&str` timestamps.
Test Functions:
- `test_insert_gas_metrics`: Verifies correctness of inserted gas metrics into the main tree.
- `test_insert_by_height_gas_metrics`: Tests gas metrics insertion into the metrics by height tree.
- `test_search_metrics_by_hour`: Validates searching gas metrics by hour.
- `test_get_metrics_by_timestamp_str`: Ensures correct retrieval of metrics by timestamp string.
- `test_tx_gas_data`: Verifies transaction gas data insertion and retrieval, including handling of non-existent transactions.
- `test_reset_metrics_within_height_range`: Tests resetting metrics within a specified height range.
- `test_reset_metrics_height_to_0`: Ensures graceful handling of resetting the metrics store to height 0.
- `test_reset_metrics_beyond_height`: Validates behavior when resetting beyond available metrics.
- `test_reset_metrics_at_height`: Tests resetting metrics at the last available height.
- `test_reset_empty_store`: Confirms that resetting an empty store is handled without errors.
Test Data Load Functions:
- `load_random_metrics`: Simulates loading random gas metrics data at hourly intervals from height 0 as the genesis block.
- `load_fixed_metrics`: Loads fixed test data with timestamps.
- `load_reset_metrics`: Returns accumulated gas metrics at a specified reset height.
- `load_tx_gas_data`: Loads test transaction gas data, returning pairs of transaction hashes and gas data.
- `random_gas_data`: Generates varied gas data based on a provided seed value.
This commit introduces the initial implementation a test_util module, providing reusable testing utilities for the explorer.
The first version includes a function to initialize logging for test cases, which aids in debugging issues encountered during testing. This module will be expanded as we progress with both unit and integration testing.
Summary of Updates:
- Added the init_logger function that configures the logger based on a provided list of targets to ignore and the desired log level
This commit request introduces a sled-backed Block Explorer, replacing the SQL-based implementation.
Summary of updates:
- Transitioned the block explorer from SQL to a sled-based database implementation
- Introduced BlockExplorerDb struct that handles explorer database operations
- Established sled-based darkfid block synchronization for startup and subscription sync
- Removed rusqlite dependency
- Removed SQL create table scripts
- Updated app.py to handle not found as empty results instead of a thrown exception when calling rpc.get_block_transactions
