Compare commits

...

26 Commits

Author SHA1 Message Date
yongkangc
7477c978e4 fix: remove dead code using non-existent RocksDBProvider::clear method
The stage/drop.rs was trying to call rocksdb.clear::<T>() but this
method doesn't exist on RocksDBProvider. Remove the dead code blocks
that would have caused compilation failures with the edge feature.
2026-01-16 09:24:27 +00:00
yongkangc
766eabce9a feat(provider): add RocksDBProvider::clear method for table cleanup
Add clear() method to RocksDBProvider and its stub implementation to
support clearing all entries from a RocksDB table. This is needed by
the stage drop CLI command to clear RocksDB history tables.

- Adds clear<T: Table>() method that iterates and deletes all entries
- Stub implementation returns Ok(()) since stub behaves as empty DB
2026-01-16 09:15:10 +00:00
yongkangc
7c1744a898 fix: update CLI commands and test utilities for RocksDB support
Update various CLI commands and test utilities for RocksDB:
- Stage drop command handles RocksDB table clearing
- Stage dump commands pass storage settings
- Test utilities provide RocksDB stubs
- Add PendingRocksDBBatches export
2026-01-16 08:37:04 +00:00
yongkangc
f775cc9b50 docs: update CLI documentation for --storage.rocksdb flag 2026-01-15 10:09:45 +00:00
yongkangc
bba7821495 fix: clippy warnings (doc_markdown, unreachable_pub, for_kv_map) 2026-01-15 10:09:45 +00:00
yongkangc
9ef853d7a1 feat(cli): add --storage.rocksdb flag for RocksDB history indices
Add a single CLI flag to enable RocksDB for all history tables:
- --storage.rocksdb enables AccountsHistory, StoragesHistory, and
  TransactionHashNumbers to be stored in RocksDB instead of MDBX

Also adds:
- `reth db settings set` subcommands for individual table control
- Updated CLI documentation
2026-01-15 10:09:45 +00:00
yongkangc
ff75aa92aa feat(storage): RocksDB support for genesis init and tx lookup stage
Add RocksDB support for:
- Genesis history initialization via EitherWriter
- TransactionLookup stage unwind operations
- RocksDB provider append_*_history_shard methods
- Metrics for RocksDB write operations
2026-01-15 10:09:26 +00:00
yongkangc
126621c5d9 feat(stages): implement RocksDB unwind for index history stages
Add RocksDB unwind support for IndexAccountHistory and IndexStorageHistory stages:
- Create RocksDB batch during unwind when storage settings enabled
- Use EitherWriter for history shard operations
- Register batches with pending_batches for atomic commit
2026-01-15 10:08:49 +00:00
yongkangc
fc776db5d8 feat(stages): add RocksDB helper functions for stage operations
Add utility functions for RocksDB integration in stages:
- make_rocksdb_provider() - creates RocksDB provider from UnifiedStorageWriter
- make_rocksdb_batch_arg() - creates RocksDB batch for EitherWriter
- register_rocksdb_batch() - registers batch with pending batches
- collect_shards_for_unwind() - shared logic for history shard unwinding
- Add RocksDBIntegrity enum variant for metadata
2026-01-15 10:08:31 +00:00
yongkangc
7a9e46f8f8 fix: use proper shard logic for history indices in RocksDB write_blocks_data
Add append_account_history_shard and append_storage_history_shard methods
to RocksDBBatch that properly handle shard boundaries. Update write_blocks_data
to use these methods instead of the naive approach that just wrote to u64::MAX
shard without checking existing shards.
2026-01-15 10:07:08 +00:00
yongkangc
5b6385940d fix(rocksdb): handle sentinel-only entries in consistency check
When RocksDB history tables have entries but ALL entries are sentinel
entries (highest_block_number == u64::MAX), the consistency check was
incorrectly returning Some(0) as the unwind target. This would trigger
an assertion failure during node startup.

Sentinel entries represent "open" shards that haven't been completed
yet, meaning no actual history has been indexed. This is equivalent to
the empty table case and should be treated as a first-run scenario.

Added tests to verify this edge case is handled correctly.
2026-01-15 10:03:52 +00:00
yongkangc
98b8c9aa4f fix(rocksdb): treat empty RocksDB tables as first-run scenario
Previously, when RocksDB tables were empty but MDBX had checkpoints > 0,
the consistency check would return Some(0), triggering an assertion
failure because unwinding to block 0 is considered destructive.

This is the expected state when RocksDB is enabled for the first time
alongside existing MDBX data. The fix treats empty RocksDB tables as a
first-run/migration scenario, logging a warning instead of requesting
an unwind. The pipeline will naturally populate the tables during sync.
2026-01-15 10:03:52 +00:00
yongkangc
f8a98e7bc8 refactor(provider): extract compute_history_rank helper to reduce duplication
- Add compute_history_rank() function for shared rank/select logic
- Simplify EitherReader::storage_history_info and account_history_info
- Simplify RocksTx::history_info by using the shared helper
- Import PhantomData directly instead of using std::marker::PhantomData
- Fix clippy doc_markdown warnings for RocksDB in stub module
- Make PendingRocksDBBatches pub(crate) to fix unreachable_pub warning

Reduces duplicated rank/select code across 3 locations while preserving comments.
2026-01-15 10:02:44 +00:00
yongkangc
ef6a6d8cbc fix: clippy warnings and fmt issues 2026-01-15 10:02:05 +00:00
yongkangc
dddc3be6ae fix: use PhantomData in EitherReader to capture lifetime 'a
When rocksdb feature is disabled, the RocksDB variant is compiled out,
leaving the lifetime 'a unused and causing E0392 error.

Add PhantomData<&'a ()> to StaticFile variant to ensure the lifetime
is always used regardless of feature flags.
2026-01-15 10:01:43 +00:00
yongkangc
7bd3fd9b48 refactor(provider): simplify EitherReader and encapsulate RocksDB logic
**Problem**
- EitherReader had unnecessary PhantomData markers
- RocksDB transaction setup was duplicated in historical.rs with cfg-gated blocks
- Addressed joshieDo's feedback about RocksDB logic leaking into historical provider

**Solution**
- Remove PhantomData from EitherReader enum variants (lifetime already captured by RocksDB reference)
- Add with_rocksdb_tx helper method to RocksDBProviderFactory trait
- Refactor historical.rs to use trait method instead of duplicated cfg-gated blocks

**Changes**
- Remove PhantomData from EitherReader enum and all constructors/match arms
- Add with_rocksdb_tx to RocksDBProviderFactory trait with default implementation
- Refactor account_history_lookup and storage_history_lookup to use with_rocksdb_tx helper
- Make RocksTxRefArg type alias public for trait method

**Expected Impact**
- Cleaner EitherReader API without unnecessary PhantomData
- RocksDB transaction setup encapsulated in trait method
- Reduced cfg-gated block duplication in historical.rs
- No behavioral changes, all existing tests pass (96/97)
2026-01-15 10:01:43 +00:00
yongkangc
38f0d2ad9f feat(storage): wire RocksDB into history lookups via EitherReader
This wires RocksDB into the history lookup paths:

- Adds account_history_info and storage_history_info methods to EitherReader
- Updates HistoricalStateProviderRef to use EitherReader for lookups
- Adds RocksDBProviderFactory trait bounds to provider impls
- Uses the rank/select pattern for efficient binary search in shards
2026-01-15 10:01:43 +00:00
joshie
851d8136fd add rocksdb writer to save_blocks 2026-01-15 10:01:43 +00:00
joshie
d8a600c4a8 add rocksdb writer to save_blocks 2026-01-14 21:38:23 +00:00
Sergei Shulepov
84bdf7158c feat(storage): add tracing spans (#21072)
Co-authored-by: Sergei Shulepov <pep@tempo.xyz>
2026-01-14 21:29:52 +00:00
joshieDo
c2ddcb284e Merge branch 'main' into joshie/par-save-blocks 2026-01-14 17:39:45 +00:00
joshieDo
fadc97aa03 feat: enable account changesets on save_blocks (#21012) 2026-01-14 17:39:27 +00:00
joshieDo
40ea5938fc fix 2026-01-13 17:47:12 +00:00
joshieDo
55e0f3d0db touch-ups 2026-01-13 16:48:46 +00:00
joshieDo
ad4998d473 rm Tx: Sync bound 2026-01-13 16:48:46 +00:00
joshieDo
cf89d57395 parallelize save_blocks 2026-01-13 16:48:46 +00:00
80 changed files with 3844 additions and 494 deletions

View File

@@ -54,6 +54,21 @@ pub enum SetCommand {
#[clap(action(ArgAction::Set))]
value: bool,
},
/// Store storages history in RocksDB instead of MDBX
StoragesHistory {
#[clap(action(ArgAction::Set))]
value: bool,
},
/// Store account history in RocksDB instead of MDBX
AccountHistory {
#[clap(action(ArgAction::Set))]
value: bool,
},
/// Store transaction hash numbers in RocksDB instead of MDBX
TxHashNumbers {
#[clap(action(ArgAction::Set))]
value: bool,
},
}
impl Command {
@@ -128,6 +143,30 @@ impl Command {
settings.account_changesets_in_static_files = value;
println!("Set account_changesets_in_static_files = {}", value);
}
SetCommand::StoragesHistory { value } => {
if settings.storages_history_in_rocksdb == value {
println!("storages_history_in_rocksdb is already set to {}", value);
return Ok(());
}
settings.storages_history_in_rocksdb = value;
println!("Set storages_history_in_rocksdb = {}", value);
}
SetCommand::AccountHistory { value } => {
if settings.account_history_in_rocksdb == value {
println!("account_history_in_rocksdb is already set to {}", value);
return Ok(());
}
settings.account_history_in_rocksdb = value;
println!("Set account_history_in_rocksdb = {}", value);
}
SetCommand::TxHashNumbers { value } => {
if settings.transaction_hash_numbers_in_rocksdb == value {
println!("transaction_hash_numbers_in_rocksdb is already set to {}", value);
return Ok(());
}
settings.transaction_hash_numbers_in_rocksdb = value;
println!("Set transaction_hash_numbers_in_rocksdb = {}", value);
}
}
// Write updated settings

View File

@@ -182,6 +182,7 @@ impl<C: ChainSpecParser> Command<C> {
}
StageEnum::TxLookup => {
tx.clear::<tables::TransactionHashNumbers>()?;
reset_prune_checkpoint(tx, PruneSegment::TransactionLookup)?;
reset_stage_checkpoint(tx, StageId::TransactionLookup)?;

View File

@@ -42,7 +42,7 @@ where
Arc::new(output_db),
db_tool.chain(),
StaticFileProvider::read_write(output_datadir.static_files())?,
RocksDBProvider::builder(output_datadir.rocksdb()).build()?,
RocksDBProvider::builder(output_datadir.rocksdb()).with_default_tables().build()?,
)?,
to,
from,

View File

@@ -39,7 +39,7 @@ pub(crate) async fn dump_hashing_account_stage<N: ProviderNodeTypes<DB = Arc<Dat
Arc::new(output_db),
db_tool.chain(),
StaticFileProvider::read_write(output_datadir.static_files())?,
RocksDBProvider::builder(output_datadir.rocksdb()).build()?,
RocksDBProvider::builder(output_datadir.rocksdb()).with_default_tables().build()?,
)?,
to,
from,

View File

@@ -29,7 +29,7 @@ pub(crate) async fn dump_hashing_storage_stage<N: ProviderNodeTypes<DB = Arc<Dat
Arc::new(output_db),
db_tool.chain(),
StaticFileProvider::read_write(output_datadir.static_files())?,
RocksDBProvider::builder(output_datadir.rocksdb()).build()?,
RocksDBProvider::builder(output_datadir.rocksdb()).with_default_tables().build()?,
)?,
to,
from,

View File

@@ -62,7 +62,7 @@ where
Arc::new(output_db),
db_tool.chain(),
StaticFileProvider::read_write(output_datadir.static_files())?,
RocksDBProvider::builder(output_datadir.rocksdb()).build()?,
RocksDBProvider::builder(output_datadir.rocksdb()).with_default_tables().build()?,
)?,
to,
from,

View File

@@ -125,7 +125,10 @@ pub async fn setup_engine_with_chain_import(
db.clone(),
chain_spec.clone(),
reth_provider::providers::StaticFileProvider::read_write(static_files_path.clone())?,
reth_provider::providers::RocksDBProvider::builder(rocksdb_dir_path).build().unwrap(),
reth_provider::providers::RocksDBProvider::builder(rocksdb_dir_path)
.with_default_tables()
.build()
.unwrap(),
)?;
// Initialize genesis if needed
@@ -328,6 +331,7 @@ mod tests {
reth_provider::providers::StaticFileProvider::read_write(static_files_path.clone())
.unwrap(),
reth_provider::providers::RocksDBProvider::builder(rocksdb_dir_path.clone())
.with_default_tables()
.build()
.unwrap(),
)
@@ -392,6 +396,7 @@ mod tests {
reth_provider::providers::StaticFileProvider::read_only(static_files_path, false)
.unwrap(),
reth_provider::providers::RocksDBProvider::builder(rocksdb_dir_path)
.with_default_tables()
.build()
.unwrap(),
)
@@ -490,7 +495,10 @@ mod tests {
db.clone(),
chain_spec.clone(),
reth_provider::providers::StaticFileProvider::read_write(static_files_path).unwrap(),
reth_provider::providers::RocksDBProvider::builder(rocksdb_dir_path).build().unwrap(),
reth_provider::providers::RocksDBProvider::builder(rocksdb_dir_path)
.with_default_tables()
.build()
.unwrap(),
)
.expect("failed to create provider factory");

View File

@@ -7,7 +7,7 @@ use reth_ethereum_primitives::EthPrimitives;
use reth_primitives_traits::NodePrimitives;
use reth_provider::{
providers::ProviderNodeTypes, BlockExecutionWriter, BlockHashReader, ChainStateBlockWriter,
DBProvider, DatabaseProviderFactory, ProviderFactory,
DBProvider, DatabaseProviderFactory, ProviderFactory, SaveBlocksMode,
};
use reth_prune::{PrunerError, PrunerOutput, PrunerWithFactory};
use reth_stages_api::{MetricEvent, MetricEventsSender};
@@ -151,7 +151,7 @@ where
if last_block.is_some() {
let provider_rw = self.provider.database_provider_rw()?;
provider_rw.save_blocks(blocks)?;
provider_rw.save_blocks(blocks, SaveBlocksMode::Full)?;
provider_rw.commit()?;
}

View File

@@ -247,7 +247,7 @@ pub async fn test_exex_context_with_chain_spec(
db,
chain_spec.clone(),
StaticFileProvider::read_write(static_dir.keep()).expect("static file provider"),
RocksDBProvider::builder(rocksdb_dir.keep()).build().unwrap(),
RocksDBProvider::builder(rocksdb_dir.keep()).with_default_tables().build().unwrap(),
)?;
let genesis_hash = init_genesis(&provider_factory)?;

View File

@@ -61,6 +61,16 @@ pub struct StaticFilesArgs {
/// the node has been initialized, changing this flag requires re-syncing from scratch.
#[arg(long = "static-files.account-change-sets")]
pub account_changesets: bool,
/// Use `RocksDB` for history indices instead of MDBX.
///
/// When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers`
/// tables will be stored in `RocksDB` for better write performance.
///
/// Note: This setting can only be configured at genesis initialization. Once
/// the node has been initialized, changing this flag requires re-syncing from scratch.
#[arg(long = "storage.rocksdb")]
pub rocksdb: bool,
}
impl StaticFilesArgs {
@@ -101,5 +111,8 @@ impl StaticFilesArgs {
.with_receipts_in_static_files(self.receipts)
.with_transaction_senders_in_static_files(self.transaction_senders)
.with_account_changesets_in_static_files(self.account_changesets)
.with_account_history_in_rocksdb(self.rocksdb)
.with_storages_history_in_rocksdb(self.rocksdb)
.with_transaction_hash_numbers_in_rocksdb(self.rocksdb)
}
}

View File

@@ -342,6 +342,14 @@ impl<ChainSpec> NodeConfig<ChainSpec> {
self
}
/// Converts the node configuration to [`StorageSettings`].
///
/// This returns storage settings configured via CLI arguments including
/// static file settings and `RocksDB` settings.
pub const fn to_storage_settings(&self) -> reth_provider::StorageSettings {
self.static_files.to_settings()
}
/// Returns pruning configuration.
pub fn prune_config(&self) -> Option<PruneConfig>
where

View File

@@ -18,7 +18,7 @@ use reth_optimism_primitives::{bedrock::is_dup_tx, OpPrimitives, OpReceipt};
use reth_primitives_traits::NodePrimitives;
use reth_provider::{
providers::ProviderNodeTypes, DBProvider, DatabaseProviderFactory, OriginalValuesKnown,
ProviderFactory, StageCheckpointReader, StageCheckpointWriter, StateWriter,
ProviderFactory, StageCheckpointReader, StageCheckpointWriter, StateWriteConfig, StateWriter,
StaticFileProviderFactory, StatsReader,
};
use reth_stages::{StageCheckpoint, StageId};
@@ -228,7 +228,11 @@ where
ExecutionOutcome::new(Default::default(), receipts, first_block, Default::default());
// finally, write the receipts
provider.write_state(&execution_outcome, OriginalValuesKnown::Yes)?;
provider.write_state(
&execution_outcome,
OriginalValuesKnown::Yes,
StateWriteConfig::default(),
)?;
}
// Only commit if we have imported as many receipts as the number of transactions.

View File

@@ -347,7 +347,10 @@ mod tests {
.with_blocks_per_file(1)
.build()
.unwrap(),
RocksDBProvider::builder(create_test_rocksdb_dir().0.keep()).build().unwrap(),
RocksDBProvider::builder(create_test_rocksdb_dir().0.keep())
.with_default_tables()
.build()
.unwrap(),
)
.unwrap();

View File

@@ -12,7 +12,7 @@ use reth_primitives_traits::{format_gas_throughput, BlockBody, NodePrimitives};
use reth_provider::{
providers::{StaticFileProvider, StaticFileWriter},
BlockHashReader, BlockReader, DBProvider, EitherWriter, ExecutionOutcome, HeaderProvider,
LatestStateProviderRef, OriginalValuesKnown, ProviderError, StateWriter,
LatestStateProviderRef, OriginalValuesKnown, ProviderError, StateWriteConfig, StateWriter,
StaticFileProviderFactory, StatsReader, StorageSettingsCache, TransactionVariant,
};
use reth_revm::database::StateProviderDatabase;
@@ -463,7 +463,7 @@ where
}
// write output
provider.write_state(&state, OriginalValuesKnown::Yes)?;
provider.write_state(&state, OriginalValuesKnown::Yes, StateWriteConfig::default())?;
let db_write_duration = time.elapsed();
debug!(

View File

@@ -1,16 +1,25 @@
use crate::stages::utils::collect_history_indices;
use super::{collect_account_history_indices, load_history_indices};
use alloy_primitives::Address;
use super::{collect_account_history_indices, load_accounts_history_indices};
use alloy_primitives::{Address, BlockNumber};
use reth_config::config::{EtlConfig, IndexHistoryConfig};
use reth_db_api::{models::ShardedKey, table::Decode, tables, transaction::DbTxMut};
use reth_db_api::{
cursor::DbCursorRO,
models::ShardedKey,
table::Decode,
tables,
transaction::{DbTx, DbTxMut},
};
use reth_provider::{
DBProvider, HistoryWriter, PruneCheckpointReader, PruneCheckpointWriter, StorageSettingsCache,
make_rocksdb_batch_arg, make_rocksdb_provider, register_rocksdb_batch, DBProvider,
EitherWriter, HistoryWriter, PruneCheckpointReader, PruneCheckpointWriter,
RocksDBProviderFactory, StorageSettingsCache,
};
use reth_prune_types::{PruneCheckpoint, PruneMode, PrunePurpose, PruneSegment};
use reth_stages_api::{
ExecInput, ExecOutput, Stage, StageCheckpoint, StageError, StageId, UnwindInput, UnwindOutput,
};
use reth_storage_api::NodePrimitivesProvider;
use std::fmt::Debug;
use tracing::info;
@@ -53,7 +62,9 @@ where
+ PruneCheckpointWriter
+ reth_storage_api::ChangeSetReader
+ reth_provider::StaticFileProviderFactory
+ StorageSettingsCache,
+ StorageSettingsCache
+ NodePrimitivesProvider
+ RocksDBProviderFactory,
{
/// Return the id of the stage
fn id(&self) -> StageId {
@@ -125,7 +136,7 @@ where
};
info!(target: "sync::stages::index_account_history::exec", "Loading indices into database");
load_history_indices::<_, tables::AccountsHistory, _>(
load_accounts_history_indices(
provider,
collector,
first_sync,
@@ -146,9 +157,40 @@ where
let (range, unwind_progress, _) =
input.unwind_block_range_with_threshold(self.commit_threshold);
provider.unwind_account_history_indices_range(range)?;
// Create EitherWriter for account history
#[allow(clippy::let_unit_value)]
let rocksdb = make_rocksdb_provider(provider);
#[allow(clippy::let_unit_value)]
let rocksdb_batch = make_rocksdb_batch_arg(&rocksdb);
let mut writer = EitherWriter::new_accounts_history(provider, rocksdb_batch)?;
// Read changesets to identify what to unwind
let changesets = provider
.tx_ref()
.cursor_read::<tables::AccountChangeSets>()?
.walk_range(range)?
.collect::<Result<Vec<_>, _>>()?;
// Group by address and find minimum block for each
// We only need to unwind once per address using the LOWEST block number
// since unwind removes all indices >= that block
let mut account_keys: std::collections::HashMap<Address, BlockNumber> =
std::collections::HashMap::new();
for (block_number, account) in changesets {
account_keys
.entry(account.address)
.and_modify(|min_bn| *min_bn = (*min_bn).min(block_number))
.or_insert(block_number);
}
// Unwind each account's history shards (once per unique address)
for (address, min_block) in account_keys {
super::utils::unwind_accounts_history_shards(&mut writer, address, min_block)?;
}
// Register RocksDB batch for commit
register_rocksdb_batch(provider, writer);
// from HistoryIndex higher than that number.
Ok(UnwindOutput { checkpoint: StageCheckpoint::new(unwind_progress) })
}
}
@@ -647,3 +689,127 @@ mod tests {
}
}
}
#[cfg(all(test, unix, feature = "rocksdb"))]
mod rocksdb_stage_tests {
use super::*;
use crate::test_utils::TestStageDB;
use reth_db_api::tables;
use reth_provider::{DatabaseProviderFactory, RocksDBProviderFactory};
use reth_storage_api::StorageSettings;
/// Test that `IndexAccountHistoryStage` writes to `RocksDB` when enabled.
#[test]
fn test_index_account_history_writes_to_rocksdb() {
let db = TestStageDB::default();
db.factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
// Setup changesets (blocks 1-10, skip 0 to avoid genesis edge case)
db.commit(|tx| {
for block in 1..=10u64 {
tx.put::<tables::BlockBodyIndices>(
block,
reth_db_api::models::StoredBlockBodyIndices {
tx_count: 3,
..Default::default()
},
)?;
tx.put::<tables::AccountChangeSets>(
block,
reth_db_api::models::AccountBeforeTx {
address: alloy_primitives::address!(
"0x0000000000000000000000000000000000000001"
),
info: None,
},
)?;
}
Ok(())
})
.unwrap();
// Execute stage from checkpoint 0 (will process blocks 1-10)
let input = ExecInput { target: Some(10), checkpoint: Some(StageCheckpoint::new(0)) };
let mut stage = IndexAccountHistoryStage::default();
let provider = db.factory.database_provider_rw().unwrap();
let out = stage.execute(&provider, input).unwrap();
assert_eq!(out, ExecOutput { checkpoint: StageCheckpoint::new(10), done: true });
provider.commit().unwrap();
// Verify data is in RocksDB
let rocksdb = db.factory.rocksdb_provider();
let count =
rocksdb.iter::<tables::AccountsHistory>().unwrap().filter_map(|r| r.ok()).count();
assert!(count > 0, "Expected data in RocksDB, found {count} entries");
// Verify MDBX AccountsHistory is empty (data went to RocksDB)
let mdbx_table = db.table::<tables::AccountsHistory>().unwrap();
assert!(mdbx_table.is_empty(), "MDBX should be empty when RocksDB is enabled");
}
/// Test that `IndexAccountHistoryStage` unwind clears `RocksDB` data.
#[test]
fn test_index_account_history_unwind_clears_rocksdb() {
let db = TestStageDB::default();
db.factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
// Setup changesets (blocks 1-10, skip 0 to avoid genesis edge case)
db.commit(|tx| {
for block in 1..=10u64 {
tx.put::<tables::BlockBodyIndices>(
block,
reth_db_api::models::StoredBlockBodyIndices {
tx_count: 3,
..Default::default()
},
)?;
tx.put::<tables::AccountChangeSets>(
block,
reth_db_api::models::AccountBeforeTx {
address: alloy_primitives::address!(
"0x0000000000000000000000000000000000000001"
),
info: None,
},
)?;
}
Ok(())
})
.unwrap();
// Execute stage from checkpoint 0 (will process blocks 1-10)
let input = ExecInput { target: Some(10), checkpoint: Some(StageCheckpoint::new(0)) };
let mut stage = IndexAccountHistoryStage::default();
let provider = db.factory.database_provider_rw().unwrap();
let out = stage.execute(&provider, input).unwrap();
assert_eq!(out, ExecOutput { checkpoint: StageCheckpoint::new(10), done: true });
provider.commit().unwrap();
// Verify data exists in RocksDB
let rocksdb = db.factory.rocksdb_provider();
let before_count =
rocksdb.iter::<tables::AccountsHistory>().unwrap().filter_map(|r| r.ok()).count();
assert!(before_count > 0, "Expected data in RocksDB before unwind");
// Unwind to block 0 (removes blocks 1-10, leaving nothing)
let unwind_input = UnwindInput {
checkpoint: StageCheckpoint::new(10),
unwind_to: 0,
..Default::default()
};
let provider = db.factory.database_provider_rw().unwrap();
let out = stage.unwind(&provider, unwind_input).unwrap();
assert_eq!(out, UnwindOutput { checkpoint: StageCheckpoint::new(0) });
provider.commit().unwrap();
// Verify RocksDB is cleared (no block 0 data exists)
let rocksdb = db.factory.rocksdb_provider();
let after_count =
rocksdb.iter::<tables::AccountsHistory>().unwrap().filter_map(|r| r.ok()).count();
assert_eq!(after_count, 0, "RocksDB should be empty after unwind to 0");
}
}

View File

@@ -1,15 +1,22 @@
use super::{collect_history_indices, load_history_indices};
use super::{collect_history_indices, load_storages_history_indices};
use crate::{StageCheckpoint, StageId};
use alloy_primitives::{Address, BlockNumber, B256};
use reth_config::config::{EtlConfig, IndexHistoryConfig};
use reth_db_api::{
cursor::DbCursorRO,
models::{storage_sharded_key::StorageShardedKey, AddressStorageKey, BlockNumberAddress},
table::Decode,
tables,
transaction::DbTxMut,
transaction::{DbTx, DbTxMut},
};
use reth_provider::{
make_rocksdb_batch_arg, make_rocksdb_provider, register_rocksdb_batch, DBProvider,
EitherWriter, HistoryWriter, PruneCheckpointReader, PruneCheckpointWriter,
RocksDBProviderFactory, StorageSettingsCache,
};
use reth_provider::{DBProvider, HistoryWriter, PruneCheckpointReader, PruneCheckpointWriter};
use reth_prune_types::{PruneCheckpoint, PruneMode, PrunePurpose, PruneSegment};
use reth_stages_api::{ExecInput, ExecOutput, Stage, StageError, UnwindInput, UnwindOutput};
use reth_storage_api::NodePrimitivesProvider;
use std::fmt::Debug;
use tracing::info;
@@ -46,8 +53,13 @@ impl Default for IndexStorageHistoryStage {
impl<Provider> Stage<Provider> for IndexStorageHistoryStage
where
Provider:
DBProvider<Tx: DbTxMut> + PruneCheckpointWriter + HistoryWriter + PruneCheckpointReader,
Provider: DBProvider<Tx: DbTxMut>
+ PruneCheckpointWriter
+ HistoryWriter
+ PruneCheckpointReader
+ NodePrimitivesProvider
+ StorageSettingsCache
+ RocksDBProviderFactory,
{
/// Return the id of the stage
fn id(&self) -> StageId {
@@ -116,7 +128,7 @@ where
)?;
info!(target: "sync::stages::index_storage_history::exec", "Loading indices into database");
load_history_indices::<_, tables::StoragesHistory, _>(
load_storages_history_indices(
provider,
collector,
first_sync,
@@ -139,7 +151,44 @@ where
let (range, unwind_progress, _) =
input.unwind_block_range_with_threshold(self.commit_threshold);
provider.unwind_storage_history_indices_range(BlockNumberAddress::range(range))?;
// Create EitherWriter for storage history
#[allow(clippy::let_unit_value)]
let rocksdb = make_rocksdb_provider(provider);
#[allow(clippy::let_unit_value)]
let rocksdb_batch = make_rocksdb_batch_arg(&rocksdb);
let mut writer = EitherWriter::new_storages_history(provider, rocksdb_batch)?;
// Read changesets to identify what to unwind
let changesets = provider
.tx_ref()
.cursor_read::<tables::StorageChangeSets>()?
.walk_range(BlockNumberAddress::range(range))?
.collect::<Result<Vec<_>, _>>()?;
// Group by (address, storage_key) and find minimum block for each
// We only need to unwind once per (address, key) using the LOWEST block number
// since unwind removes all indices >= that block
let mut storage_keys: std::collections::HashMap<(Address, B256), BlockNumber> =
std::collections::HashMap::new();
for (BlockNumberAddress((bn, address)), storage) in changesets {
storage_keys
.entry((address, storage.key))
.and_modify(|min_bn| *min_bn = (*min_bn).min(bn))
.or_insert(bn);
}
// Unwind each storage slot's history shards (once per unique key)
for ((address, storage_key), min_block) in storage_keys {
super::utils::unwind_storages_history_shards(
&mut writer,
address,
storage_key,
min_block,
)?;
}
// Register RocksDB batch for commit
register_rocksdb_batch(provider, writer);
Ok(UnwindOutput { checkpoint: StageCheckpoint::new(unwind_progress) })
}
@@ -664,3 +713,117 @@ mod tests {
}
}
}
#[cfg(all(test, unix, feature = "rocksdb"))]
mod rocksdb_stage_tests {
use super::*;
use crate::test_utils::TestStageDB;
use alloy_primitives::{address, b256, U256};
use reth_db_api::{models::StoredBlockBodyIndices, tables};
use reth_primitives_traits::StorageEntry;
use reth_provider::{DatabaseProviderFactory, RocksDBProviderFactory};
use reth_storage_api::StorageSettings;
const ADDRESS: Address = address!("0x0000000000000000000000000000000000000001");
const STORAGE_KEY: alloy_primitives::B256 =
b256!("0x0000000000000000000000000000000000000000000000000000000000000001");
/// Test that `IndexStorageHistoryStage` writes to `RocksDB` when enabled.
#[test]
fn test_index_storage_history_writes_to_rocksdb() {
let db = TestStageDB::default();
db.factory.set_storage_settings_cache(
StorageSettings::legacy().with_storages_history_in_rocksdb(true),
);
// Setup storage changesets (blocks 1-10, skip 0 to avoid genesis edge case)
db.commit(|tx| {
for block in 1..=10u64 {
tx.put::<tables::BlockBodyIndices>(
block,
StoredBlockBodyIndices { tx_count: 3, ..Default::default() },
)?;
tx.put::<tables::StorageChangeSets>(
BlockNumberAddress((block, ADDRESS)),
StorageEntry { key: STORAGE_KEY, value: U256::ZERO },
)?;
}
Ok(())
})
.unwrap();
// Execute stage from checkpoint 0 (will process blocks 1-10)
let input = ExecInput { target: Some(10), checkpoint: Some(StageCheckpoint::new(0)) };
let mut stage = IndexStorageHistoryStage::default();
let provider = db.factory.database_provider_rw().unwrap();
let out = stage.execute(&provider, input).unwrap();
assert_eq!(out, ExecOutput { checkpoint: StageCheckpoint::new(10), done: true });
provider.commit().unwrap();
// Verify data is in RocksDB
let rocksdb = db.factory.rocksdb_provider();
let count =
rocksdb.iter::<tables::StoragesHistory>().unwrap().filter_map(|r| r.ok()).count();
assert!(count > 0, "Expected data in RocksDB, found {count} entries");
// Verify MDBX StoragesHistory is empty (data went to RocksDB)
let mdbx_table = db.table::<tables::StoragesHistory>().unwrap();
assert!(mdbx_table.is_empty(), "MDBX should be empty when RocksDB is enabled");
}
/// Test that `IndexStorageHistoryStage` unwind clears `RocksDB` data.
#[test]
fn test_index_storage_history_unwind_clears_rocksdb() {
let db = TestStageDB::default();
db.factory.set_storage_settings_cache(
StorageSettings::legacy().with_storages_history_in_rocksdb(true),
);
// Setup storage changesets (blocks 1-10, skip 0 to avoid genesis edge case)
db.commit(|tx| {
for block in 1..=10u64 {
tx.put::<tables::BlockBodyIndices>(
block,
StoredBlockBodyIndices { tx_count: 3, ..Default::default() },
)?;
tx.put::<tables::StorageChangeSets>(
BlockNumberAddress((block, ADDRESS)),
StorageEntry { key: STORAGE_KEY, value: U256::ZERO },
)?;
}
Ok(())
})
.unwrap();
// Execute stage from checkpoint 0 (will process blocks 1-10)
let input = ExecInput { target: Some(10), checkpoint: Some(StageCheckpoint::new(0)) };
let mut stage = IndexStorageHistoryStage::default();
let provider = db.factory.database_provider_rw().unwrap();
let out = stage.execute(&provider, input).unwrap();
assert_eq!(out, ExecOutput { checkpoint: StageCheckpoint::new(10), done: true });
provider.commit().unwrap();
// Verify data exists in RocksDB
let rocksdb = db.factory.rocksdb_provider();
let before_count =
rocksdb.iter::<tables::StoragesHistory>().unwrap().filter_map(|r| r.ok()).count();
assert!(before_count > 0, "Expected data in RocksDB before unwind");
// Unwind to block 0 (removes blocks 1-10, leaving nothing)
let unwind_input = UnwindInput {
checkpoint: StageCheckpoint::new(10),
unwind_to: 0,
..Default::default()
};
let provider = db.factory.database_provider_rw().unwrap();
let out = stage.unwind(&provider, unwind_input).unwrap();
assert_eq!(out, UnwindOutput { checkpoint: StageCheckpoint::new(0) });
provider.commit().unwrap();
// Verify RocksDB is cleared (no block 0 data exists)
let rocksdb = db.factory.rocksdb_provider();
let after_count =
rocksdb.iter::<tables::StoragesHistory>().unwrap().filter_map(|r| r.ok()).count();
assert_eq!(after_count, 0, "RocksDB should be empty after unwind to 0");
}
}

View File

@@ -10,9 +10,10 @@ use reth_db_api::{
use reth_etl::Collector;
use reth_primitives_traits::{NodePrimitives, SignedTransaction};
use reth_provider::{
BlockReader, DBProvider, EitherWriter, PruneCheckpointReader, PruneCheckpointWriter,
RocksDBProviderFactory, StaticFileProviderFactory, StatsReader, StorageSettingsCache,
TransactionsProvider, TransactionsProviderExt,
make_rocksdb_batch_arg, make_rocksdb_provider, register_rocksdb_batch, BlockReader, DBProvider,
EitherWriter, PruneCheckpointReader, PruneCheckpointWriter, RocksDBProviderFactory,
StaticFileProviderFactory, StatsReader, StorageSettingsCache, TransactionsProvider,
TransactionsProviderExt,
};
use reth_prune_types::{PruneCheckpoint, PruneMode, PrunePurpose, PruneSegment};
use reth_stages_api::{
@@ -158,15 +159,11 @@ where
let append_only =
provider.count_entries::<tables::TransactionHashNumbers>()?.is_zero();
// Create RocksDB batch if feature is enabled
#[cfg(all(unix, feature = "rocksdb"))]
let rocksdb = provider.rocksdb_provider();
#[cfg(all(unix, feature = "rocksdb"))]
let rocksdb_batch = rocksdb.batch();
#[cfg(not(all(unix, feature = "rocksdb")))]
let rocksdb_batch = ();
// Create writer that routes to either MDBX or RocksDB based on settings
#[allow(clippy::let_unit_value)]
let rocksdb = make_rocksdb_provider(provider);
#[allow(clippy::let_unit_value)]
let rocksdb_batch = make_rocksdb_batch_arg(&rocksdb);
let mut writer =
EitherWriter::new_transaction_hash_numbers(provider, rocksdb_batch)?;
@@ -187,11 +184,8 @@ where
writer.put_transaction_hash_number(hash, tx_num, append_only)?;
}
// Extract and register RocksDB batch for commit at provider level
#[cfg(all(unix, feature = "rocksdb"))]
if let Some(batch) = writer.into_raw_rocksdb_batch() {
provider.set_pending_rocksdb_batch(batch);
}
// Register RocksDB batch for commit at provider level
register_rocksdb_batch(provider, writer);
trace!(target: "sync::stages::transaction_lookup",
total_hashes,
@@ -217,15 +211,11 @@ where
) -> Result<UnwindOutput, StageError> {
let (range, unwind_to, _) = input.unwind_block_range_with_threshold(self.chunk_size);
// Create RocksDB batch if feature is enabled
#[cfg(all(unix, feature = "rocksdb"))]
let rocksdb = provider.rocksdb_provider();
#[cfg(all(unix, feature = "rocksdb"))]
let rocksdb_batch = rocksdb.batch();
#[cfg(not(all(unix, feature = "rocksdb")))]
let rocksdb_batch = ();
// Create writer that routes to either MDBX or RocksDB based on settings
#[allow(clippy::let_unit_value)]
let rocksdb = make_rocksdb_provider(provider);
#[allow(clippy::let_unit_value)]
let rocksdb_batch = make_rocksdb_batch_arg(&rocksdb);
let mut writer = EitherWriter::new_transaction_hash_numbers(provider, rocksdb_batch)?;
let static_file_provider = provider.static_file_provider();
@@ -248,11 +238,8 @@ where
}
}
// Extract and register RocksDB batch for commit at provider level
#[cfg(all(unix, feature = "rocksdb"))]
if let Some(batch) = writer.into_raw_rocksdb_batch() {
provider.set_pending_rocksdb_batch(batch);
}
// Register RocksDB batch for commit at provider level
register_rocksdb_batch(provider, writer);
Ok(UnwindOutput {
checkpoint: StageCheckpoint::new(unwind_to)

View File

@@ -1,21 +1,28 @@
//! Utils for `stages`.
use alloy_primitives::{Address, BlockNumber, TxNumber};
use alloy_primitives::{Address, BlockNumber, TxNumber, B256};
use reth_config::config::EtlConfig;
use reth_db_api::{
cursor::{DbCursorRO, DbCursorRW},
models::{sharded_key::NUM_OF_INDICES_IN_SHARD, AccountBeforeTx, ShardedKey},
models::{
sharded_key::NUM_OF_INDICES_IN_SHARD, storage_sharded_key::StorageShardedKey,
AccountBeforeTx, ShardedKey,
},
table::{Decompress, Table},
tables,
transaction::{DbTx, DbTxMut},
BlockNumberList, DatabaseError,
};
use reth_etl::Collector;
use reth_primitives_traits::NodePrimitives;
use reth_provider::{
providers::StaticFileProvider, to_range, BlockReader, DBProvider, ProviderError,
StaticFileProviderFactory,
make_rocksdb_batch_arg, make_rocksdb_provider, providers::StaticFileProvider,
register_rocksdb_batch, to_range, BlockReader, DBProvider, EitherWriter, ProviderError,
RocksDBProviderFactory, StaticFileProviderFactory, StorageSettingsCache,
};
use reth_stages_api::StageError;
use reth_static_file_types::StaticFileSegment;
use reth_storage_api::ChangeSetReader;
use reth_storage_api::{ChangeSetReader, NodePrimitivesProvider};
use reth_storage_errors::provider::ProviderResult;
use std::{collections::HashMap, hash::Hash, ops::RangeBounds};
use tracing::info;
@@ -112,6 +119,40 @@ where
Ok::<(), StageError>(())
}
/// Generic shard-and-write helper used by both account and storage history loaders.
///
/// Chunks the list into shards, writes each shard via the provided write function,
/// and handles the last shard according to [`LoadMode`].
fn shard_and_write<F>(
list: &mut Vec<BlockNumber>,
mode: LoadMode,
mut write_fn: F,
) -> Result<(), StageError>
where
F: FnMut(Vec<u64>, BlockNumber) -> Result<(), StageError>,
{
if list.len() <= NUM_OF_INDICES_IN_SHARD && !mode.is_flush() {
return Ok(());
}
let chunks: Vec<_> = list.chunks(NUM_OF_INDICES_IN_SHARD).map(|c| c.to_vec()).collect();
let mut iter = chunks.into_iter().peekable();
while let Some(chunk) = iter.next() {
let highest = *chunk.last().expect("at least one index");
let is_last = iter.peek().is_none();
if !mode.is_flush() && is_last {
*list = chunk;
} else {
let highest = if is_last { u64::MAX } else { highest };
write_fn(chunk, highest)?;
}
}
Ok(())
}
/// Collects account history indices using a provider that implements `ChangeSetReader`.
pub(crate) fn collect_account_history_indices<Provider>(
provider: &Provider,
@@ -179,6 +220,7 @@ where
/// `Address.StorageKey`). It flushes indices to disk when reaching a shard's max length
/// (`NUM_OF_INDICES_IN_SHARD`) or when the partial key changes, ensuring the last previous partial
/// key shard is stored.
#[allow(dead_code)]
pub(crate) fn load_history_indices<Provider, H, P>(
provider: &Provider,
mut collector: Collector<H::Key, H::Value>,
@@ -263,6 +305,7 @@ where
}
/// Shard and insert the indices list according to [`LoadMode`] and its length.
#[allow(dead_code)]
pub(crate) fn load_indices<H, C, P>(
cursor: &mut C,
partial_key: P,
@@ -321,6 +364,289 @@ impl LoadMode {
}
}
/// Loads storage history indices from a collector into the database using `EitherWriter`.
///
/// This is a specialized version of [`load_history_indices`] for `tables::StoragesHistory`
/// that supports writing to either `MDBX` or `RocksDB` based on storage settings.
#[allow(dead_code)]
pub(crate) fn load_storages_history_indices<Provider, P>(
provider: &Provider,
mut collector: Collector<
<tables::StoragesHistory as Table>::Key,
<tables::StoragesHistory as Table>::Value,
>,
append_only: bool,
sharded_key_factory: impl Clone + Fn(P, u64) -> StorageShardedKey,
decode_key: impl Fn(Vec<u8>) -> Result<StorageShardedKey, DatabaseError>,
get_partial: impl Fn(StorageShardedKey) -> P,
) -> Result<(), StageError>
where
Provider: DBProvider<Tx: DbTxMut>
+ NodePrimitivesProvider
+ StorageSettingsCache
+ RocksDBProviderFactory,
P: Copy + Default + Eq,
{
// Create EitherWriter for storage history
#[allow(clippy::let_unit_value)]
let rocksdb = make_rocksdb_provider(provider);
#[allow(clippy::let_unit_value)]
let rocksdb_batch = make_rocksdb_batch_arg(&rocksdb);
let mut writer = EitherWriter::new_storages_history(provider, rocksdb_batch)?;
// Create read cursor for checking existing shards
let mut read_cursor = provider.tx_ref().cursor_read::<tables::StoragesHistory>()?;
let mut current_partial = P::default();
let mut current_list = Vec::<u64>::new();
// observability
let total_entries = collector.len();
let interval = (total_entries / 10).max(1);
for (index, element) in collector.iter()?.enumerate() {
let (k, v) = element?;
let sharded_key = decode_key(k)?;
let new_list = BlockNumberList::decompress_owned(v)?;
if index > 0 && index.is_multiple_of(interval) && total_entries > 10 {
info!(target: "sync::stages::index_history", progress = %format!("{:.2}%", (index as f64 / total_entries as f64) * 100.0), "Writing storage history indices");
}
let partial_key = get_partial(sharded_key);
if current_partial != partial_key {
// Flush last shard for previous partial key
load_storages_history_shard(
&mut writer,
current_partial,
&mut current_list,
&sharded_key_factory,
append_only,
LoadMode::Flush,
)?;
current_partial = partial_key;
current_list.clear();
// If not first sync, merge with existing shard
if !append_only &&
let Some((_, last_database_shard)) =
read_cursor.seek_exact(sharded_key_factory(current_partial, u64::MAX))?
{
current_list.extend(last_database_shard.iter());
}
}
current_list.extend(new_list.iter());
load_storages_history_shard(
&mut writer,
current_partial,
&mut current_list,
&sharded_key_factory,
append_only,
LoadMode::KeepLast,
)?;
}
// Flush remaining shard
load_storages_history_shard(
&mut writer,
current_partial,
&mut current_list,
&sharded_key_factory,
append_only,
LoadMode::Flush,
)?;
// Register RocksDB batch for commit
register_rocksdb_batch(provider, writer);
Ok(())
}
/// Shard and insert storage history indices according to [`LoadMode`] and list length.
#[allow(dead_code)]
fn load_storages_history_shard<P, CURSOR, N>(
writer: &mut EitherWriter<'_, CURSOR, N>,
partial_key: P,
list: &mut Vec<BlockNumber>,
sharded_key_factory: &impl Fn(P, BlockNumber) -> StorageShardedKey,
_append_only: bool,
mode: LoadMode,
) -> Result<(), StageError>
where
N: NodePrimitives,
CURSOR: DbCursorRW<tables::StoragesHistory> + DbCursorRO<tables::StoragesHistory>,
P: Copy,
{
shard_and_write(list, mode, |chunk, highest| {
let key = sharded_key_factory(partial_key, highest);
let value = BlockNumberList::new_pre_sorted(chunk);
Ok(writer.put_storage_history(key, &value)?)
})
}
/// Loads account history indices from a collector into the database using `EitherWriter`.
///
/// This is a specialized version of [`load_history_indices`] for `tables::AccountsHistory`
/// that supports writing to either `MDBX` or `RocksDB` based on storage settings.
#[allow(dead_code)]
pub(crate) fn load_accounts_history_indices<Provider, P>(
provider: &Provider,
mut collector: Collector<
<tables::AccountsHistory as Table>::Key,
<tables::AccountsHistory as Table>::Value,
>,
append_only: bool,
sharded_key_factory: impl Clone + Fn(P, u64) -> ShardedKey<Address>,
decode_key: impl Fn(Vec<u8>) -> Result<ShardedKey<Address>, DatabaseError>,
get_partial: impl Fn(ShardedKey<Address>) -> P,
) -> Result<(), StageError>
where
Provider: DBProvider<Tx: DbTxMut>
+ NodePrimitivesProvider
+ StorageSettingsCache
+ RocksDBProviderFactory,
P: Copy + Default + Eq,
{
// Create EitherWriter for account history
#[allow(clippy::let_unit_value)]
let rocksdb = make_rocksdb_provider(provider);
#[allow(clippy::let_unit_value)]
let rocksdb_batch = make_rocksdb_batch_arg(&rocksdb);
let mut writer = EitherWriter::new_accounts_history(provider, rocksdb_batch)?;
// Create read cursor for checking existing shards
let mut read_cursor = provider.tx_ref().cursor_read::<tables::AccountsHistory>()?;
let mut current_partial = P::default();
let mut current_list = Vec::<u64>::new();
// observability
let total_entries = collector.len();
let interval = (total_entries / 10).max(1);
for (index, element) in collector.iter()?.enumerate() {
let (k, v) = element?;
let sharded_key = decode_key(k)?;
let new_list = BlockNumberList::decompress_owned(v)?;
if index > 0 && index.is_multiple_of(interval) && total_entries > 10 {
info!(target: "sync::stages::index_history", progress = %format!("{:.2}%", (index as f64 / total_entries as f64) * 100.0), "Writing account history indices");
}
let partial_key = get_partial(sharded_key);
if current_partial != partial_key {
// Flush last shard for previous partial key
load_accounts_history_shard(
&mut writer,
current_partial,
&mut current_list,
&sharded_key_factory,
append_only,
LoadMode::Flush,
)?;
current_partial = partial_key;
current_list.clear();
// If not first sync, merge with existing shard
if !append_only &&
let Some((_, last_database_shard)) =
read_cursor.seek_exact(sharded_key_factory(current_partial, u64::MAX))?
{
current_list.extend(last_database_shard.iter());
}
}
current_list.extend(new_list.iter());
load_accounts_history_shard(
&mut writer,
current_partial,
&mut current_list,
&sharded_key_factory,
append_only,
LoadMode::KeepLast,
)?;
}
// Flush remaining shard
load_accounts_history_shard(
&mut writer,
current_partial,
&mut current_list,
&sharded_key_factory,
append_only,
LoadMode::Flush,
)?;
// Register RocksDB batch for commit
register_rocksdb_batch(provider, writer);
Ok(())
}
/// Shard and insert account history indices according to [`LoadMode`] and list length.
#[allow(dead_code)]
fn load_accounts_history_shard<P, CURSOR, N>(
writer: &mut EitherWriter<'_, CURSOR, N>,
partial_key: P,
list: &mut Vec<BlockNumber>,
sharded_key_factory: &impl Fn(P, BlockNumber) -> ShardedKey<Address>,
_append_only: bool,
mode: LoadMode,
) -> Result<(), StageError>
where
N: NodePrimitives,
CURSOR: DbCursorRW<tables::AccountsHistory> + DbCursorRO<tables::AccountsHistory>,
P: Copy,
{
shard_and_write(list, mode, |chunk, highest| {
let key = sharded_key_factory(partial_key, highest);
let value = BlockNumberList::new_pre_sorted(chunk);
Ok(writer.put_account_history(key, &value)?)
})
}
/// Unwinds storage history shards using `EitherWriter` for `RocksDB` support.
///
/// This reimplements the shard unwinding logic with support for both MDBX and `RocksDB`.
/// Walks through shards for a given key, deleting those >= unwind point and preserving
/// indices below the unwind point.
#[allow(dead_code)]
pub(crate) fn unwind_storages_history_shards<CURSOR, N>(
writer: &mut EitherWriter<'_, CURSOR, N>,
address: Address,
storage_key: B256,
block_number: BlockNumber,
) -> ProviderResult<()>
where
N: NodePrimitives,
CURSOR: DbCursorRW<tables::StoragesHistory> + DbCursorRO<tables::StoragesHistory>,
{
writer.unwind_storage_history_shards(address, storage_key, block_number)
}
/// Unwinds account history shards using `EitherWriter` for `RocksDB` support.
///
/// This reimplements the shard unwinding logic with support for both MDBX and `RocksDB`.
/// Walks through shards for a given key, deleting those >= unwind point and preserving
/// indices below the unwind point.
#[allow(dead_code)]
pub(crate) fn unwind_accounts_history_shards<CURSOR, N>(
writer: &mut EitherWriter<'_, CURSOR, N>,
address: Address,
block_number: BlockNumber,
) -> ProviderResult<()>
where
N: NodePrimitives,
CURSOR: DbCursorRW<tables::AccountsHistory> + DbCursorRO<tables::AccountsHistory>,
{
writer.unwind_account_history_shards(address, block_number)
}
/// Called when database is ahead of static files. Attempts to find the first block we are missing
/// transactions for.
pub(crate) fn missing_static_data_error<Provider>(

View File

@@ -44,9 +44,9 @@ impl StorageSettings {
receipts_in_static_files: true,
transaction_senders_in_static_files: true,
account_changesets_in_static_files: true,
storages_history_in_rocksdb: false,
transaction_hash_numbers_in_rocksdb: false,
account_history_in_rocksdb: false,
storages_history_in_rocksdb: true,
transaction_hash_numbers_in_rocksdb: true,
account_history_in_rocksdb: true,
}
}
@@ -101,4 +101,11 @@ impl StorageSettings {
self.account_changesets_in_static_files = value;
self
}
/// Returns `true` if any tables are configured to be stored in `RocksDB`.
pub const fn any_in_rocksdb(&self) -> bool {
self.transaction_hash_numbers_in_rocksdb ||
self.account_history_in_rocksdb ||
self.storages_history_in_rocksdb
}
}

View File

@@ -16,8 +16,8 @@ use reth_provider::{
errors::provider::ProviderResult, providers::StaticFileWriter, BlockHashReader, BlockNumReader,
BundleStateInit, ChainSpecProvider, DBProvider, DatabaseProviderFactory, ExecutionOutcome,
HashingWriter, HeaderProvider, HistoryWriter, MetadataWriter, OriginalValuesKnown,
ProviderError, RevertsInit, StageCheckpointReader, StageCheckpointWriter, StateWriter,
StaticFileProviderFactory, StorageSettings, StorageSettingsCache, TrieWriter,
ProviderError, RevertsInit, StageCheckpointReader, StageCheckpointWriter, StateWriteConfig,
StateWriter, StaticFileProviderFactory, StorageSettings, StorageSettingsCache, TrieWriter,
};
use reth_stages_types::{StageCheckpoint, StageId};
use reth_static_file_types::StaticFileSegment;
@@ -334,7 +334,11 @@ where
Vec::new(),
);
provider.write_state(&execution_outcome, OriginalValuesKnown::Yes)?;
provider.write_state(
&execution_outcome,
OriginalValuesKnown::Yes,
StateWriteConfig::default(),
)?;
trace!(target: "reth::cli", "Inserted state");
@@ -761,13 +765,9 @@ mod tests {
};
use alloy_genesis::Genesis;
use reth_chainspec::{Chain, ChainSpec, HOLESKY, MAINNET, SEPOLIA};
use reth_db::DatabaseEnv;
use reth_db_api::{
cursor::DbCursorRO,
models::{storage_sharded_key::StorageShardedKey, IntegerList, ShardedKey},
table::{Table, TableRow},
transaction::DbTx,
Database,
tables,
};
use reth_provider::{
test_utils::{create_test_provider_factory_with_chain_spec, MockNodeTypesWithDB},
@@ -775,6 +775,17 @@ mod tests {
};
use std::{collections::BTreeMap, sync::Arc};
#[cfg(not(feature = "edge"))]
use reth_db::DatabaseEnv;
#[cfg(not(feature = "edge"))]
use reth_db_api::{
cursor::DbCursorRO,
table::{Table, TableRow},
transaction::DbTx,
Database,
};
#[cfg(not(feature = "edge"))]
fn collect_table_entries<DB, T>(
tx: &<DB as Database>::TX,
) -> Result<Vec<TableRow<T>>, InitStorageError>
@@ -871,26 +882,74 @@ mod tests {
let factory = create_test_provider_factory_with_chain_spec(chain_spec);
init_genesis(&factory).unwrap();
let provider = factory.provider().unwrap();
// In edge mode, history indices are written to RocksDB instead of MDBX
#[cfg(feature = "edge")]
{
let rocksdb = factory.rocksdb_provider();
let tx = provider.tx_ref();
let account_history: Vec<_> = rocksdb
.iter::<tables::AccountsHistory>()
.expect("failed to iterate")
.collect::<Result<Vec<_>, _>>()
.expect("failed to collect");
assert_eq!(
collect_table_entries::<Arc<DatabaseEnv>, tables::AccountsHistory>(tx)
.expect("failed to collect"),
vec![
(ShardedKey::new(address_with_balance, u64::MAX), IntegerList::new([0]).unwrap()),
(ShardedKey::new(address_with_storage, u64::MAX), IntegerList::new([0]).unwrap())
],
);
assert_eq!(
account_history,
vec![
(
ShardedKey::new(address_with_balance, u64::MAX),
IntegerList::new([0]).unwrap()
),
(
ShardedKey::new(address_with_storage, u64::MAX),
IntegerList::new([0]).unwrap()
)
],
);
assert_eq!(
collect_table_entries::<Arc<DatabaseEnv>, tables::StoragesHistory>(tx)
.expect("failed to collect"),
vec![(
StorageShardedKey::new(address_with_storage, storage_key, u64::MAX),
IntegerList::new([0]).unwrap()
)],
);
let storage_history: Vec<_> = rocksdb
.iter::<tables::StoragesHistory>()
.expect("failed to iterate")
.collect::<Result<Vec<_>, _>>()
.expect("failed to collect");
assert_eq!(
storage_history,
vec![(
StorageShardedKey::new(address_with_storage, storage_key, u64::MAX),
IntegerList::new([0]).unwrap()
)],
);
}
#[cfg(not(feature = "edge"))]
{
let provider = factory.provider().unwrap();
let tx = provider.tx_ref();
assert_eq!(
collect_table_entries::<Arc<DatabaseEnv>, tables::AccountsHistory>(tx)
.expect("failed to collect"),
vec![
(
ShardedKey::new(address_with_balance, u64::MAX),
IntegerList::new([0]).unwrap()
),
(
ShardedKey::new(address_with_storage, u64::MAX),
IntegerList::new([0]).unwrap()
)
],
);
assert_eq!(
collect_table_entries::<Arc<DatabaseEnv>, tables::StoragesHistory>(tx)
.expect("failed to collect"),
vec![(
StorageShardedKey::new(address_with_storage, storage_key, u64::MAX),
IntegerList::new([0]).unwrap()
)],
);
}
}
}

View File

@@ -225,6 +225,9 @@ pub enum StaticFileWriterError {
/// Cannot call `sync_all` or `finalize` when prune is queued.
#[error("cannot call sync_all or finalize when prune is queued, use commit() instead")]
FinalizeWithPruneQueued,
/// Thread panicked during execution.
#[error("thread panicked: {_0}")]
ThreadPanic(&'static str),
/// Other error with message.
#[error("{0}")]
Other(String),

View File

@@ -58,6 +58,9 @@ impl TxnManager {
match rx.recv() {
Ok(msg) => match msg {
TxnManagerMessage::Begin { parent, flags, sender } => {
let _span =
tracing::debug_span!(target: "libmdbx::txn", "begin", flags)
.entered();
let mut txn: *mut ffi::MDBX_txn = ptr::null_mut();
let res = mdbx_result(unsafe {
ffi::mdbx_txn_begin_ex(
@@ -72,9 +75,13 @@ impl TxnManager {
sender.send(res).unwrap();
}
TxnManagerMessage::Abort { tx, sender } => {
let _span =
tracing::debug_span!(target: "libmdbx::txn", "abort").entered();
sender.send(mdbx_result(unsafe { ffi::mdbx_txn_abort(tx.0) })).unwrap();
}
TxnManagerMessage::Commit { tx, sender } => {
let _span =
tracing::debug_span!(target: "libmdbx::txn", "commit").entered();
sender
.send({
let mut latency = CommitLatency::new();

View File

@@ -13,7 +13,9 @@ use crate::{
providers::{StaticFileProvider, StaticFileProviderRWRefMut},
StaticFileProviderFactory,
};
use alloy_primitives::{map::HashMap, Address, BlockNumber, TxHash, TxNumber};
use alloy_primitives::{map::HashMap, Address, BlockNumber, TxHash, TxNumber, B256};
use crate::providers::{compute_history_rank, needs_prev_shard_check, HistoryInfo};
use rayon::slice::ParallelSliceMut;
use reth_db::{
cursor::{DbCursorRO, DbDupCursorRW},
@@ -36,6 +38,71 @@ use reth_storage_api::{ChangeSetReader, DBProvider, NodePrimitivesProvider, Stor
use reth_storage_errors::provider::ProviderResult;
use strum::{Display, EnumIs};
/// Collects shards to unwind from a `RocksDB` reverse iterator.
///
/// This is a generic helper for the `RocksDB` unwind logic used by both account and storage
/// history. It iterates through shards from highest to lowest block number, collecting shards to
/// delete and identifying any partial shard that needs to be preserved.
///
/// # Arguments
/// * `iter` - An iterator yielding `(K, BlockNumberList)` pairs in reverse order
/// * `belongs_to_target` - Predicate that returns `true` if the key belongs to the target being
/// unwound
/// * `highest_block_number` - Function to extract the highest block number from the key
/// * `block_number` - The unwind target block number
///
/// # Returns
/// A tuple of `(shards_to_delete, partial_shard_to_keep)` where:
/// - `shards_to_delete` contains all keys that should be deleted
/// - `partial_shard_to_keep` contains block numbers to preserve if a boundary shard was found
#[cfg(all(unix, feature = "rocksdb"))]
fn collect_shards_for_unwind<K, I, E>(
iter: I,
belongs_to_target: impl Fn(&K) -> bool,
highest_block_number: impl Fn(&K) -> BlockNumber,
block_number: BlockNumber,
) -> Result<(Vec<K>, Option<Vec<u64>>), E>
where
I: Iterator<Item = Result<(K, BlockNumberList), E>>,
{
let mut shards_to_delete = Vec::new();
let mut partial_shard_to_keep: Option<Vec<u64>> = None;
for result in iter {
let (key, list) = result?;
if !belongs_to_target(&key) {
break;
}
shards_to_delete.push(key);
let key = shards_to_delete.last().unwrap();
let first = list.iter().next().expect("List can't be empty");
// Case 1: Entire shard is at or above the unwinding point - keep it deleted
if first >= block_number {
continue;
}
// Case 2: Boundary shard - spans across the unwinding point
if block_number <= highest_block_number(key) {
let indices_to_keep: Vec<_> = list.iter().take_while(|i| *i < block_number).collect();
if !indices_to_keep.is_empty() {
partial_shard_to_keep = Some(indices_to_keep);
}
break;
}
// Case 3: Entire shard is below the unwinding point - keep all indices
let indices_to_keep: Vec<_> = list.iter().collect();
partial_shard_to_keep = Some(indices_to_keep);
break;
}
Ok((shards_to_delete, partial_shard_to_keep))
}
/// Type alias for [`EitherReader`] constructors.
type EitherReaderTy<'a, P, T> =
EitherReader<'a, CursorTy<<P as DBProvider>::Tx, T>, <P as NodePrimitivesProvider>::Primitives>;
@@ -106,6 +173,67 @@ pub enum EitherWriter<'a, CURSOR, N> {
RocksDB(RocksDBBatch<'a>),
}
/// Creates a `RocksDB` batch from the provider for use in [`EitherWriter`] constructors.
///
/// On `RocksDB`-enabled builds, returns a real batch.
/// On other builds, returns `()` to allow the same API without feature gates.
///
/// The `rocksdb` parameter should be obtained from [`make_rocksdb_provider`].
#[cfg(all(unix, feature = "rocksdb"))]
pub fn make_rocksdb_batch_arg(
rocksdb: &crate::providers::rocksdb::RocksDBProvider,
) -> RocksBatchArg<'_> {
rocksdb.batch()
}
/// Stub for non-`RocksDB` builds.
#[cfg(not(all(unix, feature = "rocksdb")))]
pub const fn make_rocksdb_batch_arg<T>(_rocksdb: &T) -> RocksBatchArg<'static> {}
/// Gets the `RocksDB` provider from a provider that implements [`RocksDBProviderFactory`].
///
/// On `RocksDB`-enabled builds, returns the real provider.
/// On other builds, returns `()` to allow the same API without feature gates.
///
/// This should be called first, and the result passed to [`make_rocksdb_batch_arg`].
/// The returned value must be kept alive for as long as the batch is used.
#[cfg(all(unix, feature = "rocksdb"))]
pub fn make_rocksdb_provider<P>(provider: &P) -> crate::providers::rocksdb::RocksDBProvider
where
P: crate::RocksDBProviderFactory,
{
provider.rocksdb_provider()
}
/// Stub for non-`RocksDB` builds.
#[cfg(not(all(unix, feature = "rocksdb")))]
pub const fn make_rocksdb_provider<P>(_provider: &P) {}
/// Registers a `RocksDB` batch extracted from an [`EitherWriter`] with the provider.
///
/// This should be called after operations on an [`EitherWriter`] that may use `RocksDB`,
/// to ensure the batch is committed when the provider commits.
///
/// On non-`RocksDB` builds, this is a no-op.
#[cfg(all(unix, feature = "rocksdb"))]
pub fn register_rocksdb_batch<P, CURSOR, N>(provider: &P, writer: EitherWriter<'_, CURSOR, N>)
where
P: crate::RocksDBProviderFactory,
N: NodePrimitives,
{
if let Some(batch) = writer.into_raw_rocksdb_batch() {
provider.set_pending_rocksdb_batch(batch);
}
}
/// Stub for non-`RocksDB` builds.
#[cfg(not(all(unix, feature = "rocksdb")))]
pub fn register_rocksdb_batch<P, CURSOR, N>(_provider: &P, _writer: EitherWriter<'_, CURSOR, N>)
where
N: NodePrimitives,
{
}
impl<'a> EitherWriter<'a, (), ()> {
/// Creates a new [`EitherWriter`] for receipts based on storage settings and prune modes.
pub fn new_receipts<P>(
@@ -273,7 +401,7 @@ impl<'a, CURSOR, N: NodePrimitives> EitherWriter<'a, CURSOR, N> {
#[cfg(all(unix, feature = "rocksdb"))]
pub fn into_raw_rocksdb_batch(self) -> Option<rocksdb::WriteBatchWithTransaction<true>> {
match self {
Self::Database(_) | Self::StaticFile(_) => None,
Self::Database(_) | Self::StaticFile(..) => None,
Self::RocksDB(batch) => Some(batch.into_inner()),
}
}
@@ -284,7 +412,7 @@ impl<'a, CURSOR, N: NodePrimitives> EitherWriter<'a, CURSOR, N> {
#[cfg(not(all(unix, feature = "rocksdb")))]
pub fn into_raw_rocksdb_batch(self) -> Option<RawRocksDBBatch> {
match self {
Self::Database(_) | Self::StaticFile(_) => None,
Self::Database(_) | Self::StaticFile(..) => None,
}
}
@@ -423,7 +551,7 @@ where
Ok(cursor.upsert(hash, &tx_num)?)
}
}
Self::StaticFile(_) => Err(ProviderError::UnsupportedProvider),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => batch.put::<tables::TransactionHashNumbers>(hash, &tx_num),
}
@@ -438,7 +566,7 @@ where
}
Ok(())
}
Self::StaticFile(_) => Err(ProviderError::UnsupportedProvider),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => batch.delete::<tables::TransactionHashNumbers>(hash),
}
@@ -457,7 +585,7 @@ where
) -> ProviderResult<()> {
match self {
Self::Database(cursor) => Ok(cursor.upsert(key, value)?),
Self::StaticFile(_) => Err(ProviderError::UnsupportedProvider),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => batch.put::<tables::StoragesHistory>(key, value),
}
@@ -472,11 +600,104 @@ where
}
Ok(())
}
Self::StaticFile(_) => Err(ProviderError::UnsupportedProvider),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => batch.delete::<tables::StoragesHistory>(key),
}
}
/// Unwinds storage history shards for a given address and storage key.
///
/// Walks through all shards for the given key, collecting indices below the unwind point,
/// then deletes all shards and reinserts the kept indices as a single sentinel shard.
pub fn unwind_storage_history_shards(
&mut self,
address: Address,
storage_key: B256,
block_number: BlockNumber,
) -> ProviderResult<()> {
let start_key = StorageShardedKey::last(address, storage_key);
match self {
Self::Database(cursor) => {
// Walk through shards from highest to lowest, following the same algorithm
// as unwind_history_shards in provider.rs
let mut item = cursor.seek_exact(start_key.clone())?;
while let Some((sharded_key, list)) = item {
// Check if shard belongs to this (address, storage_key)
if sharded_key.address != address || sharded_key.sharded_key.key != storage_key
{
break;
}
// Delete this shard
cursor.delete_current()?;
// Get the first (lowest) block number in this shard
let first = list.iter().next().expect("List can't be empty");
// Case 1: Entire shard is at or above the unwinding point
// Keep it deleted (already done above) and continue to next shard
if first >= block_number {
item = cursor.prev()?;
continue;
}
// Case 2: Boundary shard - spans across the unwinding point
// Reinsert only indices below unwind point, then STOP
if block_number <= sharded_key.sharded_key.highest_block_number {
let indices_to_keep: Vec<_> =
list.iter().take_while(|i| *i < block_number).collect();
if !indices_to_keep.is_empty() {
cursor.insert(
start_key,
&BlockNumberList::new_pre_sorted(indices_to_keep),
)?;
}
return Ok(());
}
// Case 3: Entire shard is below the unwinding point
// Reinsert all indices, then STOP (preserves earlier shards)
let indices_to_keep: Vec<_> = list.iter().collect();
cursor.insert(start_key, &BlockNumberList::new_pre_sorted(indices_to_keep))?;
return Ok(());
}
Ok(())
}
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => {
let provider = batch.provider();
let iter =
provider.iter_from_reverse::<tables::StoragesHistory>(start_key.clone())?;
let (shards_to_delete, partial_shard_to_keep) = collect_shards_for_unwind(
iter,
|k: &StorageShardedKey| {
k.address == address && k.sharded_key.key == storage_key
},
|k| k.sharded_key.highest_block_number,
block_number,
)?;
for key in shards_to_delete {
batch.delete::<tables::StoragesHistory>(key)?;
}
if let Some(indices) = partial_shard_to_keep {
batch.put::<tables::StoragesHistory>(
start_key,
&BlockNumberList::new_pre_sorted(indices),
)?;
}
Ok(())
}
}
}
}
impl<'a, CURSOR, N: NodePrimitives> EitherWriter<'a, CURSOR, N>
@@ -491,7 +712,7 @@ where
) -> ProviderResult<()> {
match self {
Self::Database(cursor) => Ok(cursor.upsert(key, value)?),
Self::StaticFile(_) => Err(ProviderError::UnsupportedProvider),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => batch.put::<tables::AccountsHistory>(key, value),
}
@@ -506,11 +727,97 @@ where
}
Ok(())
}
Self::StaticFile(_) => Err(ProviderError::UnsupportedProvider),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => batch.delete::<tables::AccountsHistory>(key),
}
}
/// Unwinds account history shards for a given address.
///
/// Walks through all shards for the given address, following the same algorithm
/// as `unwind_history_shards` in provider.rs: only delete/modify shards at or above
/// the unwind point, preserving earlier shards.
pub fn unwind_account_history_shards(
&mut self,
address: Address,
block_number: BlockNumber,
) -> ProviderResult<()> {
let start_key = ShardedKey::last(address);
match self {
Self::Database(cursor) => {
// Walk through shards from highest to lowest
let mut item = cursor.seek_exact(start_key.clone())?;
while let Some((sharded_key, list)) = item {
// Check if shard belongs to this address
if sharded_key.key != address {
break;
}
// Delete this shard
cursor.delete_current()?;
// Get the first (lowest) block number in this shard
let first = list.iter().next().expect("List can't be empty");
// Case 1: Entire shard is at or above the unwinding point
if first >= block_number {
item = cursor.prev()?;
continue;
}
// Case 2: Boundary shard - spans across the unwinding point
if block_number <= sharded_key.highest_block_number {
let indices_to_keep: Vec<_> =
list.iter().take_while(|i| *i < block_number).collect();
if !indices_to_keep.is_empty() {
cursor.insert(
start_key,
&BlockNumberList::new_pre_sorted(indices_to_keep),
)?;
}
return Ok(());
}
// Case 3: Entire shard is below the unwinding point
let indices_to_keep: Vec<_> = list.iter().collect();
cursor.insert(start_key, &BlockNumberList::new_pre_sorted(indices_to_keep))?;
return Ok(());
}
Ok(())
}
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(batch) => {
let provider = batch.provider();
let iter =
provider.iter_from_reverse::<tables::AccountsHistory>(start_key.clone())?;
let (shards_to_delete, partial_shard_to_keep) = collect_shards_for_unwind(
iter,
|k: &ShardedKey<Address>| k.key == address,
|k| k.highest_block_number,
block_number,
)?;
for key in shards_to_delete {
batch.delete::<tables::AccountsHistory>(key)?;
}
if let Some(indices) = partial_shard_to_keep {
batch.put::<tables::AccountsHistory>(
start_key,
&BlockNumberList::new_pre_sorted(indices),
)?;
}
Ok(())
}
}
}
}
impl<'a, CURSOR, N: NodePrimitives> EitherWriter<'a, CURSOR, N>
@@ -545,10 +852,13 @@ where
}
/// Represents a source for reading data, either from database, static files, or `RocksDB`.
///
/// Note: The `StaticFile` variant holds `PhantomData<&'a ()>` to ensure the lifetime `'a`
/// is used even when the `rocksdb` feature is disabled (where `RocksDB` variant is absent).
#[derive(Debug, Display)]
pub enum EitherReader<'a, CURSOR, N> {
/// Read from database table via cursor
Database(CURSOR, PhantomData<&'a ()>),
Database(CURSOR),
/// Read from static file
StaticFile(StaticFileProvider<N>, PhantomData<&'a ()>),
/// Read from `RocksDB` transaction
@@ -570,7 +880,6 @@ impl<'a> EitherReader<'a, (), ()> {
} else {
Ok(EitherReader::Database(
provider.tx_ref().cursor_read::<tables::TransactionSenders>()?,
PhantomData,
))
}
}
@@ -589,10 +898,7 @@ impl<'a> EitherReader<'a, (), ()> {
return Ok(EitherReader::RocksDB(_rocksdb_tx));
}
Ok(EitherReader::Database(
provider.tx_ref().cursor_read::<tables::StoragesHistory>()?,
PhantomData,
))
Ok(EitherReader::Database(provider.tx_ref().cursor_read::<tables::StoragesHistory>()?))
}
/// Creates a new [`EitherReader`] for transaction hash numbers based on storage settings.
@@ -611,7 +917,6 @@ impl<'a> EitherReader<'a, (), ()> {
Ok(EitherReader::Database(
provider.tx_ref().cursor_read::<tables::TransactionHashNumbers>()?,
PhantomData,
))
}
@@ -629,10 +934,7 @@ impl<'a> EitherReader<'a, (), ()> {
return Ok(EitherReader::RocksDB(_rocksdb_tx));
}
Ok(EitherReader::Database(
provider.tx_ref().cursor_read::<tables::AccountsHistory>()?,
PhantomData,
))
Ok(EitherReader::Database(provider.tx_ref().cursor_read::<tables::AccountsHistory>()?))
}
/// Creates a new [`EitherReader`] for account changesets based on storage settings.
@@ -648,7 +950,6 @@ impl<'a> EitherReader<'a, (), ()> {
} else {
Ok(EitherReader::Database(
provider.tx_ref().cursor_dup_read::<tables::AccountChangeSets>()?,
PhantomData,
))
}
}
@@ -664,7 +965,7 @@ where
range: Range<TxNumber>,
) -> ProviderResult<HashMap<TxNumber, Address>> {
match self {
Self::Database(cursor, _) => cursor
Self::Database(cursor) => cursor
.walk_range(range)?
.map(|result| result.map_err(ProviderError::from))
.collect::<ProviderResult<HashMap<_, _>>>(),
@@ -696,8 +997,8 @@ where
hash: TxHash,
) -> ProviderResult<Option<TxNumber>> {
match self {
Self::Database(cursor, _) => Ok(cursor.seek_exact(hash)?.map(|(_, v)| v)),
Self::StaticFile(_, _) => Err(ProviderError::UnsupportedProvider),
Self::Database(cursor) => Ok(cursor.seek_exact(hash)?.map(|(_, v)| v)),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(tx) => tx.get::<tables::TransactionHashNumbers>(hash),
}
@@ -714,12 +1015,68 @@ where
key: StorageShardedKey,
) -> ProviderResult<Option<BlockNumberList>> {
match self {
Self::Database(cursor, _) => Ok(cursor.seek_exact(key)?.map(|(_, v)| v)),
Self::StaticFile(_, _) => Err(ProviderError::UnsupportedProvider),
Self::Database(cursor) => Ok(cursor.seek_exact(key)?.map(|(_, v)| v)),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(tx) => tx.get::<tables::StoragesHistory>(key),
}
}
/// Lookup storage history and return [`HistoryInfo`] directly.
///
/// Uses the rank/select logic to efficiently find the first block >= target
/// where the storage slot was modified.
pub fn storage_history_info(
&mut self,
address: Address,
storage_key: B256,
block_number: BlockNumber,
lowest_available_block_number: Option<BlockNumber>,
) -> ProviderResult<HistoryInfo> {
match self {
Self::Database(cursor) => {
// Lookup the history chunk in the history index. If the key does not appear in the
// index, the first chunk for the next key will be returned so we filter out chunks
// that have a different key.
let key = StorageShardedKey::new(address, storage_key, block_number);
if let Some(chunk) = cursor
.seek(key)?
.filter(|(k, _)| k.address == address && k.sharded_key.key == storage_key)
.map(|x| x.1)
{
let (rank, found_block) = compute_history_rank(&chunk, block_number);
// Check if this is before the first write by looking at the previous shard.
let is_before_first_write =
needs_prev_shard_check(rank, found_block, block_number) &&
cursor.prev()?.is_none_or(|(k, _)| {
k.address != address || k.sharded_key.key != storage_key
});
Ok(HistoryInfo::from_lookup(
found_block,
is_before_first_write,
lowest_available_block_number,
))
} else if lowest_available_block_number.is_some() {
// The key may have been written, but due to pruning we may not have changesets
// and history, so we need to make a plain state lookup.
Ok(HistoryInfo::MaybeInPlainState)
} else {
// The key has not been written to at all.
Ok(HistoryInfo::NotYetWritten)
}
}
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(tx) => tx.storage_history_info(
address,
storage_key,
block_number,
lowest_available_block_number,
),
}
}
}
impl<CURSOR, N: NodePrimitives> EitherReader<'_, CURSOR, N>
@@ -732,12 +1089,60 @@ where
key: ShardedKey<Address>,
) -> ProviderResult<Option<BlockNumberList>> {
match self {
Self::Database(cursor, _) => Ok(cursor.seek_exact(key)?.map(|(_, v)| v)),
Self::StaticFile(_, _) => Err(ProviderError::UnsupportedProvider),
Self::Database(cursor) => Ok(cursor.seek_exact(key)?.map(|(_, v)| v)),
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(tx) => tx.get::<tables::AccountsHistory>(key),
}
}
/// Lookup account history and return [`HistoryInfo`] directly.
///
/// Uses the rank/select logic to efficiently find the first block >= target
/// where the account was modified.
pub fn account_history_info(
&mut self,
address: Address,
block_number: BlockNumber,
lowest_available_block_number: Option<BlockNumber>,
) -> ProviderResult<HistoryInfo> {
match self {
Self::Database(cursor) => {
// Lookup the history chunk in the history index. If the key does not appear in the
// index, the first chunk for the next key will be returned so we filter out chunks
// that have a different key.
let key = ShardedKey::new(address, block_number);
if let Some(chunk) =
cursor.seek(key)?.filter(|(k, _)| k.key == address).map(|x| x.1)
{
let (rank, found_block) = compute_history_rank(&chunk, block_number);
// Check if this is before the first write by looking at the previous shard.
let is_before_first_write =
needs_prev_shard_check(rank, found_block, block_number) &&
cursor.prev()?.is_none_or(|(k, _)| k.key != address);
Ok(HistoryInfo::from_lookup(
found_block,
is_before_first_write,
lowest_available_block_number,
))
} else if lowest_available_block_number.is_some() {
// The key may have been written, but due to pruning we may not have changesets
// and history, so we need to make a plain state lookup.
Ok(HistoryInfo::MaybeInPlainState)
} else {
// The key has not been written to at all.
Ok(HistoryInfo::NotYetWritten)
}
}
Self::StaticFile(..) => Err(ProviderError::UnsupportedProvider),
#[cfg(all(unix, feature = "rocksdb"))]
Self::RocksDB(tx) => {
tx.account_history_info(address, block_number, lowest_available_block_number)
}
}
}
}
impl<CURSOR, N: NodePrimitives> EitherReader<'_, CURSOR, N>
@@ -775,7 +1180,7 @@ where
Ok(changed_accounts)
}
Self::Database(provider, _) => provider
Self::Database(provider) => provider
.walk_range(range)?
.map(|entry| {
entry.map(|(_, account_before)| account_before.address).map_err(Into::into)
@@ -870,7 +1275,7 @@ mod tests {
if transaction_senders_in_static_files {
assert!(matches!(reader, EitherReader::StaticFile(_, _)));
} else {
assert!(matches!(reader, EitherReader::Database(_, _)));
assert!(matches!(reader, EitherReader::Database(_)));
}
assert_eq!(

View File

@@ -21,7 +21,8 @@ pub mod providers;
pub use providers::{
DatabaseProvider, DatabaseProviderRO, DatabaseProviderRW, HistoricalStateProvider,
HistoricalStateProviderRef, LatestStateProvider, LatestStateProviderRef, ProviderFactory,
StaticFileAccess, StaticFileProviderBuilder, StaticFileWriter,
SaveBlocksMode, StaticFileAccess, StaticFileProviderBuilder, StaticFileWriteCtx,
StaticFileWriter,
};
pub mod changeset_walker;
@@ -44,8 +45,8 @@ pub use revm_database::states::OriginalValuesKnown;
// reexport traits to avoid breaking changes
pub use reth_static_file_types as static_file;
pub use reth_storage_api::{
HistoryWriter, MetadataProvider, MetadataWriter, StatsReader, StorageSettings,
StorageSettingsCache,
HistoryWriter, MetadataProvider, MetadataWriter, StateWriteConfig, StatsReader,
StorageSettings, StorageSettingsCache,
};
/// Re-export provider error.
pub use reth_storage_errors::provider::{ProviderError, ProviderResult};

View File

@@ -789,7 +789,7 @@ mod tests {
create_test_provider_factory, create_test_provider_factory_with_chain_spec,
MockNodeTypesWithDB,
},
BlockWriter, CanonChainTracker, ProviderFactory,
BlockWriter, CanonChainTracker, ProviderFactory, SaveBlocksMode,
};
use alloy_eips::{BlockHashOrNumber, BlockNumHash, BlockNumberOrTag};
use alloy_primitives::{BlockNumber, TxNumber, B256};
@@ -808,8 +808,8 @@ mod tests {
use reth_storage_api::{
BlockBodyIndicesProvider, BlockHashReader, BlockIdReader, BlockNumReader, BlockReader,
BlockReaderIdExt, BlockSource, ChangeSetReader, DBProvider, DatabaseProviderFactory,
HeaderProvider, ReceiptProvider, ReceiptProviderIdExt, StateProviderFactory, StateWriter,
TransactionVariant, TransactionsProvider,
HeaderProvider, ReceiptProvider, ReceiptProviderIdExt, StateProviderFactory,
StateWriteConfig, StateWriter, TransactionVariant, TransactionsProvider,
};
use reth_testing_utils::generators::{
self, random_block, random_block_range, random_changeset_range, random_eoa_accounts,
@@ -907,6 +907,7 @@ mod tests {
..Default::default()
},
OriginalValuesKnown::No,
StateWriteConfig::default(),
)?;
}
@@ -997,7 +998,7 @@ mod tests {
// Push to disk
let provider_rw = hook_provider.database_provider_rw().unwrap();
provider_rw.save_blocks(vec![lowest_memory_block]).unwrap();
provider_rw.save_blocks(vec![lowest_memory_block], SaveBlocksMode::Full).unwrap();
provider_rw.commit().unwrap();
// Remove from memory

View File

@@ -40,16 +40,8 @@ pub(crate) enum Action {
InsertHeaderNumbers,
InsertBlockBodyIndices,
InsertTransactionBlocks,
GetNextTxNum,
InsertTransactionSenders,
InsertTransactionHashNumbers,
SaveBlocksInsertBlock,
SaveBlocksWriteState,
SaveBlocksWriteHashedState,
SaveBlocksWriteTrieChangesets,
SaveBlocksWriteTrieUpdates,
SaveBlocksUpdateHistoryIndices,
SaveBlocksUpdatePipelineStages,
}
/// Database provider metrics
@@ -66,19 +58,24 @@ pub(crate) struct DatabaseProviderMetrics {
insert_history_indices: Histogram,
/// Duration of update pipeline stages
update_pipeline_stages: Histogram,
/// Duration of insert canonical headers
/// Duration of insert header numbers
insert_header_numbers: Histogram,
/// Duration of insert block body indices
insert_block_body_indices: Histogram,
/// Duration of insert transaction blocks
insert_tx_blocks: Histogram,
/// Duration of get next tx num
get_next_tx_num: Histogram,
/// Duration of insert transaction senders
insert_transaction_senders: Histogram,
/// Duration of insert transaction hash numbers
insert_transaction_hash_numbers: Histogram,
/// Duration of `save_blocks`
save_blocks_total: Histogram,
/// Duration of MDBX work in `save_blocks`
save_blocks_mdbx: Histogram,
/// Duration of static file work in `save_blocks`
save_blocks_sf: Histogram,
/// Duration of `RocksDB` work in `save_blocks`
save_blocks_rocksdb: Histogram,
/// Duration of `insert_block` in `save_blocks`
save_blocks_insert_block: Histogram,
/// Duration of `write_state` in `save_blocks`
@@ -93,6 +90,39 @@ pub(crate) struct DatabaseProviderMetrics {
save_blocks_update_history_indices: Histogram,
/// Duration of `update_pipeline_stages` in `save_blocks`
save_blocks_update_pipeline_stages: Histogram,
/// Number of blocks per `save_blocks` call
save_blocks_block_count: Histogram,
/// Duration of MDBX commit in `save_blocks`
save_blocks_commit_mdbx: Histogram,
/// Duration of static file commit in `save_blocks`
save_blocks_commit_sf: Histogram,
/// Duration of `RocksDB` commit in `save_blocks`
save_blocks_commit_rocksdb: Histogram,
}
/// Timings collected during a `save_blocks` call.
#[derive(Debug, Default)]
pub(crate) struct SaveBlocksTimings {
pub total: Duration,
pub mdbx: Duration,
pub sf: Duration,
pub rocksdb: Duration,
pub insert_block: Duration,
pub write_state: Duration,
pub write_hashed_state: Duration,
pub write_trie_changesets: Duration,
pub write_trie_updates: Duration,
pub update_history_indices: Duration,
pub update_pipeline_stages: Duration,
pub block_count: u64,
}
/// Timings collected during a `commit` call.
#[derive(Debug, Default)]
pub(crate) struct CommitTimings {
pub mdbx: Duration,
pub sf: Duration,
pub rocksdb: Duration,
}
impl DatabaseProviderMetrics {
@@ -107,28 +137,33 @@ impl DatabaseProviderMetrics {
Action::InsertHeaderNumbers => self.insert_header_numbers.record(duration),
Action::InsertBlockBodyIndices => self.insert_block_body_indices.record(duration),
Action::InsertTransactionBlocks => self.insert_tx_blocks.record(duration),
Action::GetNextTxNum => self.get_next_tx_num.record(duration),
Action::InsertTransactionSenders => self.insert_transaction_senders.record(duration),
Action::InsertTransactionHashNumbers => {
self.insert_transaction_hash_numbers.record(duration)
}
Action::SaveBlocksInsertBlock => self.save_blocks_insert_block.record(duration),
Action::SaveBlocksWriteState => self.save_blocks_write_state.record(duration),
Action::SaveBlocksWriteHashedState => {
self.save_blocks_write_hashed_state.record(duration)
}
Action::SaveBlocksWriteTrieChangesets => {
self.save_blocks_write_trie_changesets.record(duration)
}
Action::SaveBlocksWriteTrieUpdates => {
self.save_blocks_write_trie_updates.record(duration)
}
Action::SaveBlocksUpdateHistoryIndices => {
self.save_blocks_update_history_indices.record(duration)
}
Action::SaveBlocksUpdatePipelineStages => {
self.save_blocks_update_pipeline_stages.record(duration)
}
}
}
/// Records all `save_blocks` timings.
pub(crate) fn record_save_blocks(&self, timings: &SaveBlocksTimings) {
self.save_blocks_total.record(timings.total);
self.save_blocks_mdbx.record(timings.mdbx);
self.save_blocks_sf.record(timings.sf);
self.save_blocks_rocksdb.record(timings.rocksdb);
self.save_blocks_insert_block.record(timings.insert_block);
self.save_blocks_write_state.record(timings.write_state);
self.save_blocks_write_hashed_state.record(timings.write_hashed_state);
self.save_blocks_write_trie_changesets.record(timings.write_trie_changesets);
self.save_blocks_write_trie_updates.record(timings.write_trie_updates);
self.save_blocks_update_history_indices.record(timings.update_history_indices);
self.save_blocks_update_pipeline_stages.record(timings.update_pipeline_stages);
self.save_blocks_block_count.record(timings.block_count as f64);
}
/// Records all commit timings.
pub(crate) fn record_commit(&self, timings: &CommitTimings) {
self.save_blocks_commit_mdbx.record(timings.mdbx);
self.save_blocks_commit_sf.record(timings.sf);
self.save_blocks_commit_rocksdb.record(timings.rocksdb);
}
}

View File

@@ -43,7 +43,7 @@ use std::{
use tracing::trace;
mod provider;
pub use provider::{DatabaseProvider, DatabaseProviderRO, DatabaseProviderRW};
pub use provider::{DatabaseProvider, DatabaseProviderRO, DatabaseProviderRW, SaveBlocksMode};
use super::ProviderNodeTypes;
use reth_trie::KeccakKeyHasher;
@@ -709,7 +709,7 @@ mod tests {
Arc::new(chain_spec),
DatabaseArguments::new(Default::default()),
StaticFileProvider::read_write(static_dir_path).unwrap(),
RocksDBProvider::builder(&rocksdb_path).build().unwrap(),
RocksDBProvider::builder(&rocksdb_path).with_default_tables().build().unwrap(),
)
.unwrap();
let provider = factory.provider().unwrap();

View File

@@ -4,8 +4,8 @@ use crate::{
},
providers::{
database::{chain::ChainStorage, metrics},
rocksdb::RocksDBProvider,
static_file::StaticFileWriter,
rocksdb::{PendingRocksDBBatches, RocksDBProvider, RocksDBWriteCtx},
static_file::{StaticFileWriteCtx, StaticFileWriter},
NodeTypesForProvider, StaticFileProvider,
},
to_range,
@@ -35,7 +35,7 @@ use alloy_primitives::{
use itertools::Itertools;
use parking_lot::RwLock;
use rayon::slice::ParallelSliceMut;
use reth_chain_state::ExecutedBlock;
use reth_chain_state::{ComputedTrieData, ExecutedBlock};
use reth_chainspec::{ChainInfo, ChainSpecProvider, EthChainSpec};
use reth_db_api::{
cursor::{DbCursorRO, DbCursorRW, DbDupCursorRO, DbDupCursorRW},
@@ -61,10 +61,10 @@ use reth_stages_types::{StageCheckpoint, StageId};
use reth_static_file_types::StaticFileSegment;
use reth_storage_api::{
BlockBodyIndicesProvider, BlockBodyReader, MetadataProvider, MetadataWriter,
NodePrimitivesProvider, StateProvider, StorageChangeSetReader, StorageSettingsCache,
TryIntoHistoricalStateProvider,
NodePrimitivesProvider, StateProvider, StateWriteConfig, StorageChangeSetReader,
StorageSettingsCache, TryIntoHistoricalStateProvider,
};
use reth_storage_errors::provider::ProviderResult;
use reth_storage_errors::provider::{ProviderResult, StaticFileWriterError};
use reth_trie::{
trie_cursor::{
InMemoryTrieCursor, InMemoryTrieCursorFactory, TrieCursor, TrieCursorFactory,
@@ -85,9 +85,10 @@ use std::{
fmt::Debug,
ops::{Deref, DerefMut, Range, RangeBounds, RangeFrom, RangeInclusive},
sync::Arc,
time::{Duration, Instant},
thread,
time::Instant,
};
use tracing::{debug, trace};
use tracing::{debug, instrument, trace};
/// A [`DatabaseProvider`] that holds a read-only database transaction.
pub type DatabaseProviderRO<DB, N> = DatabaseProvider<<DB as Database>::TX, N>;
@@ -150,6 +151,25 @@ impl<DB: Database, N: NodeTypes> From<DatabaseProviderRW<DB, N>>
}
}
/// Mode for [`DatabaseProvider::save_blocks`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SaveBlocksMode {
/// Full mode: write block structure + receipts + state + trie.
/// Used by engine/production code.
Full,
/// Blocks only: write block structure (headers, txs, senders, indices).
/// Receipts/state/trie are skipped - they may come later via separate calls.
/// Used by `insert_block`.
BlocksOnly,
}
impl SaveBlocksMode {
/// Returns `true` if this is [`SaveBlocksMode::Full`].
pub const fn with_state(self) -> bool {
matches!(self, Self::Full)
}
}
/// A provider struct that fetches data from the database.
/// Wrapper around [`DbTx`] and [`DbTxMut`]. Example: [`HeaderProvider`] [`BlockHashReader`]
pub struct DatabaseProvider<TX, N: NodeTypes> {
@@ -168,8 +188,7 @@ pub struct DatabaseProvider<TX, N: NodeTypes> {
/// `RocksDB` provider
rocksdb_provider: RocksDBProvider,
/// Pending `RocksDB` batches to be committed at provider commit time.
#[cfg(all(unix, feature = "rocksdb"))]
pending_rocksdb_batches: parking_lot::Mutex<Vec<rocksdb::WriteBatchWithTransaction<true>>>,
pending_rocksdb_batches: PendingRocksDBBatches,
/// Minimum distance from tip required for pruning
minimum_pruning_distance: u64,
/// Database provider metrics
@@ -185,10 +204,10 @@ impl<TX: Debug, N: NodeTypes> Debug for DatabaseProvider<TX, N> {
.field("prune_modes", &self.prune_modes)
.field("storage", &self.storage)
.field("storage_settings", &self.storage_settings)
.field("rocksdb_provider", &self.rocksdb_provider);
#[cfg(all(unix, feature = "rocksdb"))]
s.field("pending_rocksdb_batches", &"<pending batches>");
s.field("minimum_pruning_distance", &self.minimum_pruning_distance).finish()
.field("rocksdb_provider", &self.rocksdb_provider)
.field("pending_rocksdb_batches", &"<pending batches>")
.field("minimum_pruning_distance", &self.minimum_pruning_distance)
.finish()
}
}
@@ -316,8 +335,7 @@ impl<TX: DbTxMut, N: NodeTypes> DatabaseProvider<TX, N> {
storage,
storage_settings,
rocksdb_provider,
#[cfg(all(unix, feature = "rocksdb"))]
pending_rocksdb_batches: parking_lot::Mutex::new(Vec::new()),
pending_rocksdb_batches: Default::default(),
minimum_pruning_distance: MINIMUM_PRUNING_DISTANCE,
metrics: metrics::DatabaseProviderMetrics::default(),
}
@@ -356,98 +374,269 @@ impl<TX: DbTx + DbTxMut + 'static, N: NodeTypesForProvider> DatabaseProvider<TX,
Ok(result)
}
/// Creates the context for static file writes.
fn static_file_write_ctx(
&self,
save_mode: SaveBlocksMode,
first_block: BlockNumber,
last_block: BlockNumber,
) -> ProviderResult<StaticFileWriteCtx> {
let tip = self.last_block_number()?.max(last_block);
Ok(StaticFileWriteCtx {
write_senders: EitherWriterDestination::senders(self).is_static_file() &&
self.prune_modes.sender_recovery.is_none_or(|m| !m.is_full()),
write_receipts: save_mode.with_state() &&
EitherWriter::receipts_destination(self).is_static_file(),
write_account_changesets: save_mode.with_state() &&
EitherWriterDestination::account_changesets(self).is_static_file(),
tip,
receipts_prune_mode: self.prune_modes.receipts,
// Receipts are prunable if no receipts exist in SF yet and within pruning distance
receipts_prunable: self
.static_file_provider
.get_highest_static_file_tx(StaticFileSegment::Receipts)
.is_none() &&
PruneMode::Distance(self.minimum_pruning_distance)
.should_prune(first_block, tip),
})
}
/// Creates the context for `RocksDB` writes.
fn rocksdb_write_ctx(&self, first_block: BlockNumber) -> RocksDBWriteCtx {
RocksDBWriteCtx {
first_block_number: first_block,
prune_tx_lookup: self.prune_modes.transaction_lookup,
storage_settings: self.cached_storage_settings(),
pending_batches: self.pending_rocksdb_batches.clone(),
}
}
/// Writes executed blocks and state to storage.
pub fn save_blocks(&self, blocks: Vec<ExecutedBlock<N::Primitives>>) -> ProviderResult<()> {
///
/// This method parallelizes static file (SF) writes with MDBX writes.
/// The SF thread writes headers, transactions, senders (if SF), and receipts (if SF, Full mode
/// only). The main thread writes MDBX data (indices, state, trie - Full mode only).
///
/// Use [`SaveBlocksMode::Full`] for production (includes receipts, state, trie).
/// Use [`SaveBlocksMode::BlocksOnly`] for block structure only (used by `insert_block`).
#[instrument(level = "debug", target = "providers::db", skip_all, fields(block_count = blocks.len()))]
pub fn save_blocks(
&self,
blocks: Vec<ExecutedBlock<N::Primitives>>,
save_mode: SaveBlocksMode,
) -> ProviderResult<()> {
if blocks.is_empty() {
debug!(target: "providers::db", "Attempted to write empty block range");
return Ok(())
}
// NOTE: checked non-empty above
let first_block = blocks.first().unwrap().recovered_block();
let total_start = Instant::now();
let block_count = blocks.len() as u64;
let first_number = blocks.first().unwrap().recovered_block().number();
let last_block_number = blocks.last().unwrap().recovered_block().number();
let last_block = blocks.last().unwrap().recovered_block();
let first_number = first_block.number();
let last_block_number = last_block.number();
debug!(target: "providers::db", block_count, "Writing blocks and execution data to storage");
debug!(target: "providers::db", block_count = %blocks.len(), "Writing blocks and execution data to storage");
// Compute tx_nums upfront (both threads need these)
let first_tx_num = self
.tx
.cursor_read::<tables::TransactionBlocks>()?
.last()?
.map(|(n, _)| n + 1)
.unwrap_or_default();
// Accumulate durations for each step
let mut total_insert_block = Duration::ZERO;
let mut total_write_state = Duration::ZERO;
let mut total_write_hashed_state = Duration::ZERO;
let mut total_write_trie_changesets = Duration::ZERO;
let mut total_write_trie_updates = Duration::ZERO;
let tx_nums: Vec<TxNumber> = {
let mut nums = Vec::with_capacity(blocks.len());
let mut current = first_tx_num;
for block in &blocks {
nums.push(current);
current += block.recovered_block().body().transaction_count() as u64;
}
nums
};
// TODO: Do performant / batched writes for each type of object
// instead of a loop over all blocks,
// meaning:
// * blocks
// * state
// * hashed state
// * trie updates (cannot naively extend, need helper)
// * indices (already done basically)
// Insert the blocks
for block in blocks {
let trie_data = block.trie_data();
let ExecutedBlock { recovered_block, execution_output, .. } = block;
let block_number = recovered_block.number();
let mut timings = metrics::SaveBlocksTimings { block_count, ..Default::default() };
// avoid capturing &self.tx in scope below.
let sf_provider = &self.static_file_provider;
let sf_ctx = self.static_file_write_ctx(save_mode, first_number, last_block_number)?;
let rocksdb_provider = self.rocksdb_provider.clone();
let rocksdb_ctx = self.rocksdb_write_ctx(first_number);
thread::scope(|s| {
// SF writes
let sf_handle = s.spawn(|| {
let start = Instant::now();
sf_provider.write_blocks_data(&blocks, &tx_nums, sf_ctx)?;
Ok::<_, ProviderError>(start.elapsed())
});
// RocksDB writes (batches are pushed to pending_batches inside write_blocks_data)
let rocksdb_handle = rocksdb_ctx.storage_settings.any_in_rocksdb().then(|| {
s.spawn(|| {
let start = Instant::now();
rocksdb_provider.write_blocks_data(&blocks, &tx_nums, rocksdb_ctx)?;
Ok::<_, ProviderError>(start.elapsed())
})
});
// MDBX writes
let mdbx_start = Instant::now();
for (i, block) in blocks.iter().enumerate() {
let recovered_block = block.recovered_block();
let start = Instant::now();
self.insert_block_mdbx_only(recovered_block, tx_nums[i])?;
timings.insert_block += start.elapsed();
if save_mode.with_state() {
let execution_output = block.execution_outcome();
let block_number = recovered_block.number();
// Write state and changesets to the database.
// Must be written after blocks because of the receipt lookup.
// Skip receipts/account changesets if they're being written to static files.
let start = Instant::now();
self.write_state(
execution_output,
OriginalValuesKnown::No,
StateWriteConfig {
write_receipts: !sf_ctx.write_receipts,
write_account_changesets: !sf_ctx.write_account_changesets,
},
)?;
timings.write_state += start.elapsed();
let trie_data = block.trie_data();
// insert hashes and intermediate merkle nodes
let start = Instant::now();
self.write_hashed_state(&trie_data.hashed_state)?;
timings.write_hashed_state += start.elapsed();
let start = Instant::now();
self.write_trie_changesets(block_number, &trie_data.trie_updates, None)?;
timings.write_trie_changesets += start.elapsed();
let start = Instant::now();
self.write_trie_updates_sorted(&trie_data.trie_updates)?;
timings.write_trie_updates += start.elapsed();
}
}
// Full mode: update history indices
if save_mode.with_state() {
let start = Instant::now();
self.update_history_indices(first_number..=last_block_number)?;
timings.update_history_indices = start.elapsed();
}
// Update pipeline progress
let start = Instant::now();
self.insert_block(&recovered_block)?;
total_insert_block += start.elapsed();
self.update_pipeline_stages(last_block_number, false)?;
timings.update_pipeline_stages = start.elapsed();
// Write state and changesets to the database.
// Must be written after blocks because of the receipt lookup.
let start = Instant::now();
self.write_state(&execution_output, OriginalValuesKnown::No)?;
total_write_state += start.elapsed();
timings.mdbx = mdbx_start.elapsed();
// insert hashes and intermediate merkle nodes
let start = Instant::now();
self.write_hashed_state(&trie_data.hashed_state)?;
total_write_hashed_state += start.elapsed();
// Wait for SF thread
timings.sf = sf_handle
.join()
.map_err(|_| StaticFileWriterError::ThreadPanic("static file"))??;
let start = Instant::now();
self.write_trie_changesets(block_number, &trie_data.trie_updates, None)?;
total_write_trie_changesets += start.elapsed();
// Wait for RocksDB thread (batches already pushed to pending_batches)
#[cfg(all(unix, feature = "rocksdb"))]
if let Some(handle) = rocksdb_handle {
let elapsed = handle.join().expect("RocksDB thread panicked")?;
timings.rocksdb = elapsed;
}
#[cfg(not(all(unix, feature = "rocksdb")))]
let _ = rocksdb_handle;
timings.total = total_start.elapsed();
self.metrics.record_save_blocks(&timings);
debug!(target: "providers::db", range = ?first_number..=last_block_number, "Appended block data");
Ok(())
})
}
/// Writes MDBX-only data for a block (eg. indices, lookups, and senders).
///
/// SF data (headers, transactions, senders if SF, receipts if SF) must be written separately.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn insert_block_mdbx_only(
&self,
block: &RecoveredBlock<BlockTy<N>>,
first_tx_num: TxNumber,
) -> ProviderResult<StoredBlockBodyIndices> {
if self.prune_modes.sender_recovery.is_none_or(|m| !m.is_full()) &&
EitherWriterDestination::senders(self).is_database()
{
let start = Instant::now();
self.write_trie_updates_sorted(&trie_data.trie_updates)?;
total_write_trie_updates += start.elapsed();
let tx_nums_iter = std::iter::successors(Some(first_tx_num), |n| Some(n + 1));
let mut cursor = self.tx.cursor_write::<tables::TransactionSenders>()?;
for (tx_num, sender) in tx_nums_iter.zip(block.senders_iter().copied()) {
cursor.append(tx_num, &sender)?;
}
self.metrics
.record_duration(metrics::Action::InsertTransactionSenders, start.elapsed());
}
// update history indices
let block_number = block.number();
let tx_count = block.body().transaction_count() as u64;
let start = Instant::now();
self.update_history_indices(first_number..=last_block_number)?;
let duration_update_history_indices = start.elapsed();
self.tx.put::<tables::HeaderNumbers>(block.hash(), block_number)?;
self.metrics.record_duration(metrics::Action::InsertHeaderNumbers, start.elapsed());
// Update pipeline progress
// Write tx hash numbers to MDBX if not handled by RocksDB and not fully pruned
if !self.cached_storage_settings().transaction_hash_numbers_in_rocksdb &&
self.prune_modes.transaction_lookup.is_none_or(|m| !m.is_full())
{
let start = Instant::now();
let mut cursor = self.tx.cursor_write::<tables::TransactionHashNumbers>()?;
let mut tx_num = first_tx_num;
for transaction in block.body().transactions_iter() {
cursor.upsert(*transaction.tx_hash(), &tx_num)?;
tx_num += 1;
}
self.metrics
.record_duration(metrics::Action::InsertTransactionHashNumbers, start.elapsed());
}
self.write_block_body_indices(block_number, block.body(), first_tx_num, tx_count)?;
Ok(StoredBlockBodyIndices { first_tx_num, tx_count })
}
/// Writes MDBX block body indices (`BlockBodyIndices`, `TransactionBlocks`,
/// `Ommers`/`Withdrawals`).
fn write_block_body_indices(
&self,
block_number: BlockNumber,
body: &BodyTy<N>,
first_tx_num: TxNumber,
tx_count: u64,
) -> ProviderResult<()> {
// MDBX: BlockBodyIndices
let start = Instant::now();
self.update_pipeline_stages(last_block_number, false)?;
let duration_update_pipeline_stages = start.elapsed();
self.tx
.cursor_write::<tables::BlockBodyIndices>()?
.append(block_number, &StoredBlockBodyIndices { first_tx_num, tx_count })?;
self.metrics.record_duration(metrics::Action::InsertBlockBodyIndices, start.elapsed());
// Record all metrics at the end
self.metrics.record_duration(metrics::Action::SaveBlocksInsertBlock, total_insert_block);
self.metrics.record_duration(metrics::Action::SaveBlocksWriteState, total_write_state);
self.metrics
.record_duration(metrics::Action::SaveBlocksWriteHashedState, total_write_hashed_state);
self.metrics.record_duration(
metrics::Action::SaveBlocksWriteTrieChangesets,
total_write_trie_changesets,
);
self.metrics
.record_duration(metrics::Action::SaveBlocksWriteTrieUpdates, total_write_trie_updates);
self.metrics.record_duration(
metrics::Action::SaveBlocksUpdateHistoryIndices,
duration_update_history_indices,
);
self.metrics.record_duration(
metrics::Action::SaveBlocksUpdatePipelineStages,
duration_update_pipeline_stages,
);
// MDBX: TransactionBlocks (last tx -> block mapping)
if tx_count > 0 {
let start = Instant::now();
self.tx
.cursor_write::<tables::TransactionBlocks>()?
.append(first_tx_num + tx_count - 1, &block_number)?;
self.metrics.record_duration(metrics::Action::InsertTransactionBlocks, start.elapsed());
}
debug!(target: "providers::db", range = ?first_number..=last_block_number, "Appended block data");
// MDBX: Ommers/Withdrawals
self.storage.writer().write_block_bodies(self, vec![(block_number, Some(body))])?;
Ok(())
}
@@ -642,8 +831,7 @@ impl<TX: DbTx + 'static, N: NodeTypesForProvider> DatabaseProvider<TX, N> {
storage,
storage_settings,
rocksdb_provider,
#[cfg(all(unix, feature = "rocksdb"))]
pending_rocksdb_batches: parking_lot::Mutex::new(Vec::new()),
pending_rocksdb_batches: Default::default(),
minimum_pruning_distance: MINIMUM_PRUNING_DISTANCE,
metrics: metrics::DatabaseProviderMetrics::default(),
}
@@ -1727,6 +1915,7 @@ impl<TX: DbTxMut, N: NodeTypes> StageCheckpointWriter for DatabaseProvider<TX, N
Ok(self.tx.put::<tables::StageCheckpointProgresses>(id.to_string(), checkpoint)?)
}
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn update_pipeline_stages(
&self,
block_number: BlockNumber,
@@ -1817,24 +2006,31 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider> StateWriter
{
type Receipt = ReceiptTy<N>;
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_state(
&self,
execution_outcome: &ExecutionOutcome<Self::Receipt>,
is_value_known: OriginalValuesKnown,
config: StateWriteConfig,
) -> ProviderResult<()> {
let first_block = execution_outcome.first_block();
let (plain_state, reverts) =
execution_outcome.bundle.to_plain_state_and_reverts(is_value_known);
self.write_state_reverts(reverts, first_block, config)?;
self.write_state_changes(plain_state)?;
if !config.write_receipts {
return Ok(());
}
let block_count = execution_outcome.len() as u64;
let last_block = execution_outcome.last_block();
let block_range = first_block..=last_block;
let tip = self.last_block_number()?.max(last_block);
let (plain_state, reverts) =
execution_outcome.bundle.to_plain_state_and_reverts(is_value_known);
self.write_state_reverts(reverts, first_block)?;
self.write_state_changes(plain_state)?;
// Fetch the first transaction number for each block in the range
let block_indices: Vec<_> = self
.block_body_indices_range(block_range)?
@@ -1918,6 +2114,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider> StateWriter
&self,
reverts: PlainStateReverts,
first_block: BlockNumber,
config: StateWriteConfig,
) -> ProviderResult<()> {
// Write storage changes
tracing::trace!("Writing storage changes");
@@ -1965,7 +2162,11 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider> StateWriter
}
}
// Write account changes to static files
if !config.write_account_changesets {
return Ok(());
}
// Write account changes
tracing::debug!(target: "sync::stages::merkle_changesets", ?first_block, "Writing account changes");
for (block_index, account_block_reverts) in reverts.accounts.into_iter().enumerate() {
let block_number = first_block + block_index as BlockNumber;
@@ -2043,6 +2244,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider> StateWriter
Ok(())
}
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_hashed_state(&self, hashed_state: &HashedPostStateSorted) -> ProviderResult<()> {
// Write hashed account updates.
let mut hashed_accounts_cursor = self.tx_ref().cursor_write::<tables::HashedAccounts>()?;
@@ -2336,6 +2538,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypes> TrieWriter for DatabaseProvider
/// Writes trie updates to the database with already sorted updates.
///
/// Returns the number of entries modified.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_trie_updates_sorted(&self, trie_updates: &TrieUpdatesSorted) -> ProviderResult<usize> {
if trie_updates.is_empty() {
return Ok(0)
@@ -2379,6 +2582,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypes> TrieWriter for DatabaseProvider
/// the same `TrieUpdates`.
///
/// Returns the number of keys written.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_trie_changesets(
&self,
block_number: BlockNumber,
@@ -2970,15 +3174,15 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypes> HistoryWriter for DatabaseProvi
)
}
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn update_history_indices(&self, range: RangeInclusive<BlockNumber>) -> ProviderResult<()> {
// account history stage
{
let storage_settings = self.cached_storage_settings();
if !storage_settings.account_history_in_rocksdb {
let indices = self.changed_accounts_and_blocks_with_range(range.clone())?;
self.insert_account_history_index(indices)?;
}
// storage history stage
{
if !storage_settings.storages_history_in_rocksdb {
let indices = self.changed_storages_and_blocks_with_range(range)?;
self.insert_storage_history_index(indices)?;
}
@@ -2987,7 +3191,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypes> HistoryWriter for DatabaseProvi
}
}
impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockExecutionWriter
impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider> BlockExecutionWriter
for DatabaseProvider<TX, N>
{
fn take_block_and_execution_above(
@@ -3030,89 +3234,40 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockExecu
}
}
impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWriter
impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider> BlockWriter
for DatabaseProvider<TX, N>
{
type Block = BlockTy<N>;
type Receipt = ReceiptTy<N>;
/// Inserts the block into the database, always modifying the following static file segments and
/// tables:
/// * [`StaticFileSegment::Headers`]
/// * [`tables::HeaderNumbers`]
/// * [`tables::BlockBodyIndices`]
/// Inserts the block into the database, writing to both static files and MDBX.
///
/// If there are transactions in the block, the following static file segments and tables will
/// be modified:
/// * [`StaticFileSegment::Transactions`]
/// * [`tables::TransactionBlocks`]
///
/// If ommers are not empty, this will modify [`BlockOmmers`](tables::BlockOmmers).
/// If withdrawals are not empty, this will modify
/// [`BlockWithdrawals`](tables::BlockWithdrawals).
///
/// If the provider has __not__ configured full sender pruning, this will modify either:
/// * [`StaticFileSegment::TransactionSenders`] if senders are written to static files
/// * [`tables::TransactionSenders`] if senders are written to the database
///
/// If the provider has __not__ configured full transaction lookup pruning, this will modify
/// [`TransactionHashNumbers`](tables::TransactionHashNumbers).
/// This is a convenience method primarily used in tests. For production use,
/// prefer [`Self::save_blocks`] which handles execution output and trie data.
fn insert_block(
&self,
block: &RecoveredBlock<Self::Block>,
) -> ProviderResult<StoredBlockBodyIndices> {
let block_number = block.number();
let tx_count = block.body().transaction_count() as u64;
let mut durations_recorder = metrics::DurationsRecorder::new(&self.metrics);
self.static_file_provider
.get_writer(block_number, StaticFileSegment::Headers)?
.append_header(block.header(), &block.hash())?;
self.tx.put::<tables::HeaderNumbers>(block.hash(), block_number)?;
durations_recorder.record_relative(metrics::Action::InsertHeaderNumbers);
let first_tx_num = self
.tx
.cursor_read::<tables::TransactionBlocks>()?
.last()?
.map(|(n, _)| n + 1)
.unwrap_or_default();
durations_recorder.record_relative(metrics::Action::GetNextTxNum);
let tx_nums_iter = std::iter::successors(Some(first_tx_num), |n| Some(n + 1));
if self.prune_modes.sender_recovery.as_ref().is_none_or(|m| !m.is_full()) {
let mut senders_writer = EitherWriter::new_senders(self, block.number())?;
senders_writer.increment_block(block.number())?;
senders_writer
.append_senders(tx_nums_iter.clone().zip(block.senders_iter().copied()))?;
durations_recorder.record_relative(metrics::Action::InsertTransactionSenders);
}
if self.prune_modes.transaction_lookup.is_none_or(|m| !m.is_full()) {
self.with_rocksdb_batch(|batch| {
let mut writer = EitherWriter::new_transaction_hash_numbers(self, batch)?;
for (tx_num, transaction) in tx_nums_iter.zip(block.body().transactions_iter()) {
let hash = transaction.tx_hash();
writer.put_transaction_hash_number(*hash, tx_num, false)?;
}
Ok(((), writer.into_raw_rocksdb_batch()))
})?;
durations_recorder.record_relative(metrics::Action::InsertTransactionHashNumbers);
}
self.append_block_bodies(vec![(block_number, Some(block.body()))])?;
debug!(
target: "providers::db",
?block_number,
actions = ?durations_recorder.actions,
"Inserted block"
// Wrap block in ExecutedBlock with empty execution output (no receipts/state/trie)
let executed_block = ExecutedBlock::new(
Arc::new(block.clone()),
Arc::new(ExecutionOutcome::new(
Default::default(),
Vec::<Vec<ReceiptTy<N>>>::new(),
block_number,
vec![],
)),
ComputedTrieData::default(),
);
Ok(StoredBlockBodyIndices { first_tx_num, tx_count })
// Delegate to save_blocks with BlocksOnly mode (skips receipts/state/trie)
self.save_blocks(vec![executed_block], SaveBlocksMode::BlocksOnly)?;
// Return the body indices
self.block_body_indices(block_number)?
.ok_or(ProviderError::BlockBodyIndicesNotFound(block_number))
}
fn append_block_bodies(
@@ -3298,7 +3453,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
durations_recorder.record_relative(metrics::Action::InsertBlock);
}
self.write_state(execution_outcome, OriginalValuesKnown::No)?;
self.write_state(execution_outcome, OriginalValuesKnown::No, StateWriteConfig::default())?;
durations_recorder.record_relative(metrics::Action::InsertState);
// insert hashes and intermediate merkle nodes
@@ -3440,17 +3595,28 @@ impl<TX: DbTx + 'static, N: NodeTypes + 'static> DBProvider for DatabaseProvider
self.static_file_provider.commit()?;
} else {
self.static_file_provider.commit()?;
// Normal path: finalize() will call sync_all() if not already synced
let mut timings = metrics::CommitTimings::default();
let start = Instant::now();
self.static_file_provider.finalize()?;
timings.sf = start.elapsed();
#[cfg(all(unix, feature = "rocksdb"))]
{
let start = Instant::now();
let batches = std::mem::take(&mut *self.pending_rocksdb_batches.lock());
for batch in batches {
self.rocksdb_provider.commit_batch(batch)?;
}
timings.rocksdb = start.elapsed();
}
let start = Instant::now();
self.tx.commit()?;
timings.mdbx = start.elapsed();
self.metrics.record_commit(&timings);
}
Ok(true)
@@ -3523,10 +3689,17 @@ mod tests {
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
provider_rw.insert_block(&data.blocks[0].0).unwrap();
provider_rw.write_state(&data.blocks[0].1, crate::OriginalValuesKnown::No).unwrap();
provider_rw
.write_state(
&data.blocks[0].1,
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
provider_rw.commit().unwrap();
let provider = factory.provider().unwrap();
@@ -3549,11 +3722,18 @@ mod tests {
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
for i in 0..3 {
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
provider_rw
.write_state(
&data.blocks[i].1,
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
}
provider_rw.commit().unwrap();
@@ -3579,13 +3759,20 @@ mod tests {
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
// insert blocks 1-3 with receipts
for i in 0..3 {
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
provider_rw
.write_state(
&data.blocks[i].1,
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
}
provider_rw.commit().unwrap();
@@ -3610,11 +3797,18 @@ mod tests {
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
for i in 0..3 {
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
provider_rw
.write_state(
&data.blocks[i].1,
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
}
provider_rw.commit().unwrap();
@@ -3673,11 +3867,18 @@ mod tests {
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
for i in 0..3 {
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
provider_rw
.write_state(
&data.blocks[i].1,
crate::OriginalValuesKnown::No,
StateWriteConfig::default(),
)
.unwrap();
}
provider_rw.commit().unwrap();
@@ -4991,7 +5192,9 @@ mod tests {
}]],
..Default::default()
};
provider_rw.write_state(&outcome, crate::OriginalValuesKnown::No).unwrap();
provider_rw
.write_state(&outcome, crate::OriginalValuesKnown::No, StateWriteConfig::default())
.unwrap();
provider_rw.commit().unwrap();
};

View File

@@ -10,14 +10,14 @@ pub use database::*;
mod static_file;
pub use static_file::{
StaticFileAccess, StaticFileJarProvider, StaticFileProvider, StaticFileProviderBuilder,
StaticFileProviderRW, StaticFileProviderRWRefMut, StaticFileWriter,
StaticFileProviderRW, StaticFileProviderRWRefMut, StaticFileWriteCtx, StaticFileWriter,
};
mod state;
pub use state::{
historical::{
needs_prev_shard_check, HistoricalStateProvider, HistoricalStateProviderRef, HistoryInfo,
LowestAvailableBlocks,
compute_history_rank, needs_prev_shard_check, HistoricalStateProvider,
HistoricalStateProviderRef, HistoryInfo, LowestAvailableBlocks,
},
latest::{LatestStateProvider, LatestStateProviderRef},
overlay::{OverlayStateProvider, OverlayStateProviderFactory},
@@ -38,7 +38,7 @@ pub use consistent::ConsistentProvider;
#[cfg_attr(not(all(unix, feature = "rocksdb")), path = "rocksdb_stub.rs")]
pub(crate) mod rocksdb;
pub use rocksdb::{RocksDBBatch, RocksDBBuilder, RocksDBProvider, RocksTx};
pub use rocksdb::{RocksDBBuilder, RocksDBProvider};
/// Helper trait to bound [`NodeTypes`] so that combined with database they satisfy
/// [`ProviderNodeTypes`].

View File

@@ -164,15 +164,16 @@ impl RocksDBProvider {
self.prune_transaction_hash_numbers_in_range(provider, 0..=highest_tx)?;
}
(None, None) => {
// Both MDBX and static files are empty.
// If checkpoint says we should have data, that's an inconsistency.
// Both MDBX and static files are empty - this is expected on first run.
// Log a warning but don't require unwind to 0, as the pipeline will
// naturally populate the data during sync.
if checkpoint > 0 {
tracing::warn!(
target: "reth::providers::rocksdb",
checkpoint,
"Checkpoint set but no transaction data exists, unwind needed"
"TransactionHashNumbers: no transaction data exists but checkpoint is set. \
This is expected on first run with RocksDB enabled."
);
return Ok(Some(0));
}
}
}
@@ -263,16 +264,35 @@ impl RocksDBProvider {
}
// Find the max highest_block_number (excluding u64::MAX sentinel) across all
// entries
// entries. Also track if we found any non-sentinel entries.
let mut max_highest_block = 0u64;
let mut found_non_sentinel = false;
for result in self.iter::<tables::StoragesHistory>()? {
let (key, _) = result?;
let highest = key.sharded_key.highest_block_number;
if highest != u64::MAX && highest > max_highest_block {
max_highest_block = highest;
if highest != u64::MAX {
found_non_sentinel = true;
if highest > max_highest_block {
max_highest_block = highest;
}
}
}
// If all entries are sentinel entries (u64::MAX), treat as first-run scenario.
// Sentinel entries represent "open" shards that haven't been completed yet,
// so no actual history has been indexed.
if !found_non_sentinel {
if checkpoint > 0 {
tracing::warn!(
target: "reth::providers::rocksdb",
checkpoint,
"StoragesHistory has only sentinel entries but checkpoint is set. \
This is expected on first run with RocksDB enabled."
);
}
return Ok(None);
}
// If any entry has highest_block > checkpoint, prune excess
if max_highest_block > checkpoint {
tracing::info!(
@@ -296,10 +316,16 @@ impl RocksDBProvider {
Ok(None)
}
None => {
// Empty RocksDB table
// Empty RocksDB table - this is expected on first run / migration.
// Log a warning but don't require unwind to 0, as the pipeline will
// naturally populate the table during sync.
if checkpoint > 0 {
// Stage says we should have data but we don't
return Ok(Some(0));
tracing::warn!(
target: "reth::providers::rocksdb",
checkpoint,
"StoragesHistory is empty but checkpoint is set. \
This is expected on first run with RocksDB enabled."
);
}
Ok(None)
}
@@ -377,16 +403,35 @@ impl RocksDBProvider {
}
// Find the max highest_block_number (excluding u64::MAX sentinel) across all
// entries
// entries. Also track if we found any non-sentinel entries.
let mut max_highest_block = 0u64;
let mut found_non_sentinel = false;
for result in self.iter::<tables::AccountsHistory>()? {
let (key, _) = result?;
let highest = key.highest_block_number;
if highest != u64::MAX && highest > max_highest_block {
max_highest_block = highest;
if highest != u64::MAX {
found_non_sentinel = true;
if highest > max_highest_block {
max_highest_block = highest;
}
}
}
// If all entries are sentinel entries (u64::MAX), treat as first-run scenario.
// Sentinel entries represent "open" shards that haven't been completed yet,
// so no actual history has been indexed.
if !found_non_sentinel {
if checkpoint > 0 {
tracing::warn!(
target: "reth::providers::rocksdb",
checkpoint,
"AccountsHistory has only sentinel entries but checkpoint is set. \
This is expected on first run with RocksDB enabled."
);
}
return Ok(None);
}
// If any entry has highest_block > checkpoint, prune excess
if max_highest_block > checkpoint {
tracing::info!(
@@ -413,10 +458,16 @@ impl RocksDBProvider {
Ok(None)
}
None => {
// Empty RocksDB table
// Empty RocksDB table - this is expected on first run / migration.
// Log a warning but don't require unwind to 0, as the pipeline will
// naturally populate the table during sync.
if checkpoint > 0 {
// Stage says we should have data but we don't
return Ok(Some(0));
tracing::warn!(
target: "reth::providers::rocksdb",
checkpoint,
"AccountsHistory is empty but checkpoint is set. \
This is expected on first run with RocksDB enabled."
);
}
Ok(None)
}
@@ -542,7 +593,7 @@ mod tests {
}
#[test]
fn test_check_consistency_empty_rocksdb_with_checkpoint_needs_unwind() {
fn test_check_consistency_empty_rocksdb_with_checkpoint_is_first_run() {
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::TransactionHashNumbers>()
@@ -566,10 +617,10 @@ mod tests {
let provider = factory.database_provider_ro().unwrap();
// RocksDB is empty but checkpoint says block 100 was processed
// This means RocksDB is missing data and we need to unwind to rebuild
// RocksDB is empty but checkpoint says block 100 was processed.
// This is treated as a first-run/migration scenario - no unwind needed.
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(result, Some(0), "Should require unwind to block 0 to rebuild RocksDB");
assert_eq!(result, None, "Empty data with checkpoint is treated as first run");
}
#[test]
@@ -650,7 +701,7 @@ mod tests {
}
#[test]
fn test_check_consistency_storages_history_empty_with_checkpoint_needs_unwind() {
fn test_check_consistency_storages_history_empty_with_checkpoint_is_first_run() {
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::StoragesHistory>()
@@ -674,9 +725,10 @@ mod tests {
let provider = factory.database_provider_ro().unwrap();
// RocksDB is empty but checkpoint says block 100 was processed
// RocksDB is empty but checkpoint says block 100 was processed.
// This is treated as a first-run/migration scenario - no unwind needed.
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(result, Some(0), "Should require unwind to block 0 to rebuild StoragesHistory");
assert_eq!(result, None, "Empty RocksDB with checkpoint is treated as first run");
}
#[test]
@@ -978,6 +1030,97 @@ mod tests {
);
}
#[test]
fn test_check_consistency_storages_history_sentinel_only_with_checkpoint_is_first_run() {
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::StoragesHistory>()
.build()
.unwrap();
// Insert ONLY sentinel entries (highest_block_number = u64::MAX)
// This simulates a scenario where history tracking started but no shards were completed
let key_sentinel_1 = StorageShardedKey::new(Address::ZERO, B256::ZERO, u64::MAX);
let key_sentinel_2 = StorageShardedKey::new(Address::random(), B256::random(), u64::MAX);
let block_list = BlockNumberList::new_pre_sorted([10, 20, 30]);
rocksdb.put::<tables::StoragesHistory>(key_sentinel_1, &block_list).unwrap();
rocksdb.put::<tables::StoragesHistory>(key_sentinel_2, &block_list).unwrap();
// Verify entries exist (not empty table)
assert!(rocksdb.first::<tables::StoragesHistory>().unwrap().is_some());
// Create a test provider factory for MDBX
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_storages_history_in_rocksdb(true),
);
// Set a checkpoint indicating we should have processed up to block 100
{
let provider = factory.database_provider_rw().unwrap();
provider
.save_stage_checkpoint(StageId::IndexStorageHistory, StageCheckpoint::new(100))
.unwrap();
provider.commit().unwrap();
}
let provider = factory.database_provider_ro().unwrap();
// RocksDB has only sentinel entries (no completed shards) but checkpoint is set.
// This is treated as a first-run/migration scenario - no unwind needed.
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(
result, None,
"Sentinel-only entries with checkpoint should be treated as first run"
);
}
#[test]
fn test_check_consistency_accounts_history_sentinel_only_with_checkpoint_is_first_run() {
use reth_db_api::models::ShardedKey;
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::AccountsHistory>()
.build()
.unwrap();
// Insert ONLY sentinel entries (highest_block_number = u64::MAX)
let key_sentinel_1 = ShardedKey::new(Address::ZERO, u64::MAX);
let key_sentinel_2 = ShardedKey::new(Address::random(), u64::MAX);
let block_list = BlockNumberList::new_pre_sorted([10, 20, 30]);
rocksdb.put::<tables::AccountsHistory>(key_sentinel_1, &block_list).unwrap();
rocksdb.put::<tables::AccountsHistory>(key_sentinel_2, &block_list).unwrap();
// Verify entries exist (not empty table)
assert!(rocksdb.first::<tables::AccountsHistory>().unwrap().is_some());
// Create a test provider factory for MDBX
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
// Set a checkpoint indicating we should have processed up to block 100
{
let provider = factory.database_provider_rw().unwrap();
provider
.save_stage_checkpoint(StageId::IndexAccountHistory, StageCheckpoint::new(100))
.unwrap();
provider.commit().unwrap();
}
let provider = factory.database_provider_ro().unwrap();
// RocksDB has only sentinel entries (no completed shards) but checkpoint is set.
// This is treated as a first-run/migration scenario - no unwind needed.
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(
result, None,
"Sentinel-only entries with checkpoint should be treated as first run"
);
}
#[test]
fn test_check_consistency_storages_history_behind_checkpoint_single_entry() {
use reth_db_api::models::storage_sharded_key::StorageShardedKey;
@@ -1135,7 +1278,7 @@ mod tests {
}
#[test]
fn test_check_consistency_accounts_history_empty_with_checkpoint_needs_unwind() {
fn test_check_consistency_accounts_history_empty_with_checkpoint_is_first_run() {
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::AccountsHistory>()
@@ -1159,9 +1302,10 @@ mod tests {
let provider = factory.database_provider_ro().unwrap();
// RocksDB is empty but checkpoint says block 100 was processed
// RocksDB is empty but checkpoint says block 100 was processed.
// This is treated as a first-run/migration scenario - no unwind needed.
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(result, Some(0), "Should require unwind to block 0 to rebuild AccountsHistory");
assert_eq!(result, None, "Empty RocksDB with checkpoint is treated as first run");
}
#[test]

View File

@@ -6,7 +6,11 @@ use reth_db::Tables;
use reth_metrics::Metrics;
use strum::{EnumIter, IntoEnumIterator};
const ROCKSDB_TABLES: &[&str] = &[Tables::TransactionHashNumbers.name()];
const ROCKSDB_TABLES: &[&str] = &[
Tables::TransactionHashNumbers.name(),
Tables::AccountsHistory.name(),
Tables::StoragesHistory.name(),
];
/// Metrics for the `RocksDB` provider.
#[derive(Debug)]

View File

@@ -4,4 +4,5 @@ mod invariants;
mod metrics;
mod provider;
pub use provider::{RocksDBBatch, RocksDBBuilder, RocksDBProvider, RocksTx};
pub(crate) use provider::{PendingRocksDBBatches, RocksDBBatch, RocksDBWriteCtx, RocksTx};
pub use provider::{RocksDBBuilder, RocksDBProvider};

View File

@@ -1,11 +1,20 @@
use super::metrics::{RocksDBMetrics, RocksDBOperation};
use crate::providers::{needs_prev_shard_check, HistoryInfo};
use alloy_primitives::{Address, BlockNumber, B256};
use crate::providers::{compute_history_rank, needs_prev_shard_check, HistoryInfo};
use alloy_consensus::transaction::TxHashRef;
use alloy_primitives::{Address, BlockNumber, TxNumber, B256};
use itertools::Itertools;
use parking_lot::Mutex;
use reth_chain_state::ExecutedBlock;
use reth_db_api::{
models::{storage_sharded_key::StorageShardedKey, ShardedKey},
models::{
sharded_key::NUM_OF_INDICES_IN_SHARD, storage_sharded_key::StorageShardedKey, ShardedKey,
StorageSettings,
},
table::{Compress, Decode, Decompress, Encode, Table},
tables, BlockNumberList, DatabaseError,
};
use reth_primitives_traits::BlockBody as _;
use reth_prune_types::PruneMode;
use reth_storage_errors::{
db::{DatabaseErrorInfo, DatabaseWriteError, DatabaseWriteOperation, LogLevel},
provider::{ProviderError, ProviderResult},
@@ -16,11 +25,41 @@ use rocksdb::{
OptimisticTransactionOptions, Options, Transaction, WriteBatchWithTransaction, WriteOptions,
};
use std::{
collections::BTreeMap,
fmt,
path::{Path, PathBuf},
sync::Arc,
thread,
time::Instant,
};
use tracing::instrument;
/// Pending `RocksDB` batches type alias.
pub(crate) type PendingRocksDBBatches = Arc<Mutex<Vec<WriteBatchWithTransaction<true>>>>;
/// Context for `RocksDB` block writes.
#[derive(Clone)]
pub(crate) struct RocksDBWriteCtx {
/// The first block number being written.
pub first_block_number: BlockNumber,
/// The prune mode for transaction lookup, if any.
pub prune_tx_lookup: Option<PruneMode>,
/// Storage settings determining what goes to `RocksDB`.
pub storage_settings: StorageSettings,
/// Pending batches to push to after writing.
pub pending_batches: PendingRocksDBBatches,
}
impl fmt::Debug for RocksDBWriteCtx {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("RocksDBWriteCtx")
.field("first_block_number", &self.first_block_number)
.field("prune_tx_lookup", &self.prune_tx_lookup)
.field("storage_settings", &self.storage_settings)
.field("pending_batches", &"<pending batches>")
.finish()
}
}
/// Default cache size for `RocksDB` block cache (128 MB).
const DEFAULT_CACHE_SIZE: usize = 128 << 20;
@@ -391,6 +430,32 @@ impl RocksDBProvider {
})
}
/// Clears all entries from the specified table.
///
/// This iterates through all entries in the table and deletes them.
pub fn clear<T: Table>(&self) -> ProviderResult<()> {
self.execute_with_operation_metric(RocksDBOperation::Delete, T::NAME, |this| {
let cf = this.get_cf_handle::<T>()?;
let iter = this.0.db.iterator_cf(cf, IteratorMode::Start);
for result in iter {
let (key, _) = result.map_err(|e| {
ProviderError::Database(DatabaseError::Read(DatabaseErrorInfo {
message: e.to_string().into(),
code: -1,
}))
})?;
this.0.db.delete_cf(cf, &key).map_err(|e| {
ProviderError::Database(DatabaseError::Delete(DatabaseErrorInfo {
message: e.to_string().into(),
code: -1,
}))
})?;
}
Ok(())
})
}
/// Gets the first (smallest key) entry from the specified table.
pub fn first<T: Table>(&self) -> ProviderResult<Option<(T::Key, T::Value)>> {
self.execute_with_operation_metric(RocksDBOperation::Get, T::NAME, |this| {
@@ -474,6 +539,141 @@ impl RocksDBProvider {
}))
})
}
/// Creates a reverse iterator over a table starting from the given key.
///
/// Iterates from the key towards the beginning of the table.
pub fn iter_from_reverse<T: Table>(
&self,
key: T::Key,
) -> ProviderResult<RocksDBIterReverse<'_, T>> {
let cf = self.get_cf_handle::<T>()?;
let encoded_key = key.encode();
let iter = self
.0
.db
.iterator_cf(cf, IteratorMode::From(encoded_key.as_ref(), rocksdb::Direction::Reverse));
Ok(RocksDBIterReverse { inner: iter, _marker: std::marker::PhantomData })
}
/// Writes all `RocksDB` data for multiple blocks in parallel.
///
/// This handles transaction hash numbers, account history, and storage history based on
/// the provided storage settings. Each operation runs in parallel with its own batch,
/// pushing to `ctx.pending_batches` for later commit.
#[instrument(level = "debug", target = "providers::db", skip_all)]
pub(crate) fn write_blocks_data<N: reth_node_types::NodePrimitives>(
&self,
blocks: &[ExecutedBlock<N>],
tx_nums: &[TxNumber],
ctx: RocksDBWriteCtx,
) -> ProviderResult<()> {
if !ctx.storage_settings.any_in_rocksdb() {
return Ok(());
}
thread::scope(|s| {
let handles: Vec<_> = [
(ctx.storage_settings.transaction_hash_numbers_in_rocksdb &&
ctx.prune_tx_lookup.is_none_or(|m| !m.is_full()))
.then(|| s.spawn(|| self.write_tx_hash_numbers(blocks, tx_nums, &ctx))),
ctx.storage_settings
.account_history_in_rocksdb
.then(|| s.spawn(|| self.write_account_history(blocks, &ctx))),
ctx.storage_settings
.storages_history_in_rocksdb
.then(|| s.spawn(|| self.write_storage_history(blocks, &ctx))),
]
.into_iter()
.enumerate()
.filter_map(|(i, h)| h.map(|h| (i, h)))
.collect();
for (i, handle) in handles {
handle.join().map_err(|_| {
ProviderError::Database(DatabaseError::Other(format!(
"rocksdb write thread {i} panicked"
)))
})??;
}
Ok(())
})
}
/// Writes transaction hash to number mappings for the given blocks.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_tx_hash_numbers<N: reth_node_types::NodePrimitives>(
&self,
blocks: &[ExecutedBlock<N>],
tx_nums: &[TxNumber],
ctx: &RocksDBWriteCtx,
) -> ProviderResult<()> {
let mut batch = self.batch();
for (block, &first_tx_num) in blocks.iter().zip(tx_nums) {
let body = block.recovered_block().body();
let mut tx_num = first_tx_num;
for transaction in body.transactions_iter() {
batch.put::<tables::TransactionHashNumbers>(*transaction.tx_hash(), &tx_num)?;
tx_num += 1;
}
}
ctx.pending_batches.lock().push(batch.into_inner());
Ok(())
}
/// Writes account history indices for the given blocks.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_account_history<N: reth_node_types::NodePrimitives>(
&self,
blocks: &[ExecutedBlock<N>],
ctx: &RocksDBWriteCtx,
) -> ProviderResult<()> {
let mut batch = self.batch();
let mut account_history: BTreeMap<Address, Vec<u64>> = BTreeMap::new();
for (block_idx, block) in blocks.iter().enumerate() {
let block_number = ctx.first_block_number + block_idx as u64;
let bundle = &block.execution_outcome().bundle;
for &address in bundle.state().keys() {
account_history.entry(address).or_default().push(block_number);
}
}
// Write account history using proper shard append logic
for (address, indices) in account_history {
batch.append_account_history_shard(address, indices)?;
}
ctx.pending_batches.lock().push(batch.into_inner());
Ok(())
}
/// Writes storage history indices for the given blocks.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_storage_history<N: reth_node_types::NodePrimitives>(
&self,
blocks: &[ExecutedBlock<N>],
ctx: &RocksDBWriteCtx,
) -> ProviderResult<()> {
let mut batch = self.batch();
let mut storage_history: BTreeMap<(Address, B256), Vec<u64>> = BTreeMap::new();
for (block_idx, block) in blocks.iter().enumerate() {
let block_number = ctx.first_block_number + block_idx as u64;
let bundle = &block.execution_outcome().bundle;
for (&address, account) in bundle.state() {
for &slot in account.storage.keys() {
let key = B256::new(slot.to_be_bytes());
storage_history.entry((address, key)).or_default().push(block_number);
}
}
}
// Write storage history using proper shard append logic
for ((address, slot), indices) in storage_history {
batch.append_storage_history_shard(address, slot, indices)?;
}
ctx.pending_batches.lock().push(batch.into_inner());
Ok(())
}
}
/// Handle for building a batch of operations atomically.
@@ -560,6 +760,91 @@ impl<'a> RocksDBBatch<'a> {
pub fn into_inner(self) -> WriteBatchWithTransaction<true> {
self.inner
}
/// Appends indices to an account history shard with proper shard management.
///
/// Loads the existing shard (if any), appends new indices, and rechunks into
/// multiple shards if needed (respecting `NUM_OF_INDICES_IN_SHARD` limit).
pub fn append_account_history_shard(
&mut self,
address: Address,
indices: impl IntoIterator<Item = u64>,
) -> ProviderResult<()> {
let last_key = ShardedKey::new(address, u64::MAX);
let last_shard_opt = self.provider.get::<tables::AccountsHistory>(last_key.clone())?;
let mut last_shard = last_shard_opt.unwrap_or_else(BlockNumberList::empty);
last_shard.append(indices).map_err(ProviderError::other)?;
// Fast path: all indices fit in one shard
if last_shard.len() <= NUM_OF_INDICES_IN_SHARD as u64 {
self.put::<tables::AccountsHistory>(last_key, &last_shard)?;
return Ok(());
}
// Slow path: rechunk into multiple shards
let chunks = last_shard.iter().chunks(NUM_OF_INDICES_IN_SHARD);
let mut chunks_peekable = chunks.into_iter().peekable();
while let Some(chunk) = chunks_peekable.next() {
let shard = BlockNumberList::new_pre_sorted(chunk);
let highest_block_number = if chunks_peekable.peek().is_some() {
shard.iter().next_back().expect("`chunks` does not return empty list")
} else {
u64::MAX
};
self.put::<tables::AccountsHistory>(
ShardedKey::new(address, highest_block_number),
&shard,
)?;
}
Ok(())
}
/// Appends indices to a storage history shard with proper shard management.
///
/// Loads the existing shard (if any), appends new indices, and rechunks into
/// multiple shards if needed (respecting `NUM_OF_INDICES_IN_SHARD` limit).
pub fn append_storage_history_shard(
&mut self,
address: Address,
storage_key: B256,
indices: impl IntoIterator<Item = u64>,
) -> ProviderResult<()> {
let last_key = StorageShardedKey::last(address, storage_key);
let last_shard_opt = self.provider.get::<tables::StoragesHistory>(last_key.clone())?;
let mut last_shard = last_shard_opt.unwrap_or_else(BlockNumberList::empty);
last_shard.append(indices).map_err(ProviderError::other)?;
// Fast path: all indices fit in one shard
if last_shard.len() <= NUM_OF_INDICES_IN_SHARD as u64 {
self.put::<tables::StoragesHistory>(last_key, &last_shard)?;
return Ok(());
}
// Slow path: rechunk into multiple shards
let chunks = last_shard.iter().chunks(NUM_OF_INDICES_IN_SHARD);
let mut chunks_peekable = chunks.into_iter().peekable();
while let Some(chunk) = chunks_peekable.next() {
let shard = BlockNumberList::new_pre_sorted(chunk);
let highest_block_number = if chunks_peekable.peek().is_some() {
shard.iter().next_back().expect("`chunks` does not return empty list")
} else {
u64::MAX
};
self.put::<tables::StoragesHistory>(
StorageShardedKey::new(address, storage_key, highest_block_number),
&shard,
)?;
}
Ok(())
}
}
/// `RocksDB` transaction wrapper providing MDBX-like semantics.
@@ -800,21 +1085,9 @@ impl<'db> RocksTx<'db> {
};
};
let chunk = BlockNumberList::decompress(value_bytes)?;
let (rank, found_block) = compute_history_rank(&chunk, block_number);
// Get the rank of the first entry before or equal to our block.
let mut rank = chunk.rank(block_number);
// Adjust the rank, so that we have the rank of the first entry strictly before our
// block (not equal to it).
if rank.checked_sub(1).and_then(|r| chunk.select(r)) == Some(block_number) {
rank -= 1;
}
let found_block = chunk.select(rank);
// Lazy check for previous shard - only called when needed.
// If we can step to a previous shard for this same key, history already exists,
// so the target block is not before the first write.
// Check if this is before the first write by looking at the previous shard.
let is_before_first_write = if needs_prev_shard_check(rank, found_block, block_number) {
iter.prev();
Self::raw_iter_status_ok(&iter)?;
@@ -888,6 +1161,60 @@ impl<T: Table> Iterator for RocksDBIter<'_, T> {
}
}
/// Result type for raw iterator items.
type RocksDBRawIterResult = Result<(Box<[u8]>, Box<[u8]>), rocksdb::Error>;
/// Decodes an iterator result from `RocksDB` into a table key-value pair.
fn decode_iter_result<T: Table>(
result: RocksDBRawIterResult,
) -> Option<ProviderResult<(T::Key, T::Value)>> {
let (key_bytes, value_bytes) = match result {
Ok(kv) => kv,
Err(e) => {
return Some(Err(ProviderError::Database(DatabaseError::Read(DatabaseErrorInfo {
message: e.to_string().into(),
code: -1,
}))))
}
};
// Decode key
let key = match <T::Key as reth_db_api::table::Decode>::decode(&key_bytes) {
Ok(k) => k,
Err(_) => return Some(Err(ProviderError::Database(DatabaseError::Decode))),
};
// Decompress value
let value = match T::Value::decompress(&value_bytes) {
Ok(v) => v,
Err(_) => return Some(Err(ProviderError::Database(DatabaseError::Decode))),
};
Some(Ok((key, value)))
}
/// Reverse iterator over a `RocksDB` table (non-transactional).
///
/// Yields decoded `(Key, Value)` pairs in reverse key order.
pub struct RocksDBIterReverse<'db, T: Table> {
inner: rocksdb::DBIteratorWithThreadMode<'db, OptimisticTransactionDB>,
_marker: std::marker::PhantomData<T>,
}
impl<T: Table> fmt::Debug for RocksDBIterReverse<'_, T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("RocksDBIterReverse").field("table", &T::NAME).finish_non_exhaustive()
}
}
impl<T: Table> Iterator for RocksDBIterReverse<'_, T> {
type Item = ProviderResult<(T::Key, T::Value)>;
fn next(&mut self) -> Option<Self::Item> {
decode_iter_result::<T>(self.inner.next()?)
}
}
/// Iterator over a `RocksDB` table within a transaction.
///
/// Yields decoded `(Key, Value)` pairs. Sees uncommitted writes.

View File

@@ -4,12 +4,34 @@
//! available (either on non-Unix platforms or when the `rocksdb` feature is not enabled).
//! Operations will produce errors if actually attempted.
use reth_db_api::table::{Encode, Table};
use alloy_primitives::BlockNumber;
use parking_lot::Mutex;
use reth_db_api::{
models::StorageSettings,
table::{Encode, Table},
};
use reth_prune_types::PruneMode;
use reth_storage_errors::{
db::LogLevel,
provider::{ProviderError::UnsupportedProvider, ProviderResult},
};
use std::path::Path;
use std::{path::Path, sync::Arc};
/// Pending `RocksDB` batches type alias (stub - uses unit type).
pub(crate) type PendingRocksDBBatches = Arc<Mutex<Vec<()>>>;
/// Context for `RocksDB` block writes (stub).
#[derive(Debug, Clone)]
pub struct RocksDBWriteCtx {
/// The first block number being written.
pub first_block_number: BlockNumber,
/// The prune mode for transaction lookup, if any.
pub prune_tx_lookup: Option<PruneMode>,
/// Storage settings determining what goes to `RocksDB`.
pub storage_settings: StorageSettings,
/// Pending batches (stub - unused).
pub pending_batches: PendingRocksDBBatches,
}
/// A stub `RocksDB` provider.
///
@@ -65,6 +87,13 @@ impl RocksDBProvider {
Err(UnsupportedProvider)
}
/// Clear all entries from a table (stub implementation).
///
/// Returns `Ok(())` since the stub behaves as if the database is empty.
pub const fn clear<T: Table>(&self) -> ProviderResult<()> {
Ok(())
}
/// Write a batch of operations (stub implementation).
pub fn write_batch<F>(&self, _f: F) -> ProviderResult<()>
where
@@ -110,6 +139,21 @@ impl RocksDBProvider {
) -> ProviderResult<Option<alloy_primitives::BlockNumber>> {
Ok(None)
}
/// Writes all `RocksDB` data for multiple blocks (stub implementation).
///
/// No-op since `RocksDB` is not available on this platform.
pub fn write_blocks_data<N>(
&self,
_blocks: &[reth_chain_state::ExecutedBlock<N>],
_tx_nums: &[alloy_primitives::TxNumber],
_ctx: RocksDBWriteCtx,
) -> ProviderResult<()>
where
N: reth_node_types::NodePrimitives,
{
Ok(())
}
}
/// A stub batch writer for `RocksDB` on non-Unix platforms.

View File

@@ -1,20 +1,14 @@
use crate::{
AccountReader, BlockHashReader, ChangeSetReader, HashedPostStateProvider, ProviderError,
StateProvider, StateRootProvider,
AccountReader, BlockHashReader, ChangeSetReader, EitherReader, HashedPostStateProvider,
ProviderError, RocksDBProviderFactory, StateProvider, StateRootProvider,
};
use alloy_eips::merge::EPOCH_SLOTS;
use alloy_primitives::{Address, BlockNumber, Bytes, StorageKey, StorageValue, B256};
use reth_db_api::{
cursor::{DbCursorRO, DbDupCursorRO},
models::{storage_sharded_key::StorageShardedKey, ShardedKey},
table::Table,
tables,
transaction::DbTx,
BlockNumberList,
};
use reth_db_api::{cursor::DbDupCursorRO, tables, transaction::DbTx};
use reth_primitives_traits::{Account, Bytecode};
use reth_storage_api::{
BlockNumReader, BytecodeReader, DBProvider, StateProofProvider, StorageRootProvider,
BlockNumReader, BytecodeReader, DBProvider, NodePrimitivesProvider, StateProofProvider,
StorageRootProvider, StorageSettingsCache,
};
use reth_storage_errors::provider::ProviderResult;
use reth_trie::{
@@ -127,38 +121,47 @@ impl<'b, Provider: DBProvider + ChangeSetReader + BlockNumReader>
Self { provider, block_number, lowest_available_blocks }
}
/// Lookup an account in the `AccountsHistory` table
pub fn account_history_lookup(&self, address: Address) -> ProviderResult<HistoryInfo> {
/// Lookup an account in the `AccountsHistory` table using `EitherReader`.
pub fn account_history_lookup(&self, address: Address) -> ProviderResult<HistoryInfo>
where
Provider: StorageSettingsCache + RocksDBProviderFactory + NodePrimitivesProvider,
{
if !self.lowest_available_blocks.is_account_history_available(self.block_number) {
return Err(ProviderError::StateAtBlockPruned(self.block_number))
}
// history key to search IntegerList of block number changesets.
let history_key = ShardedKey::new(address, self.block_number);
self.history_info::<tables::AccountsHistory, _>(
history_key,
|key| key.key == address,
self.lowest_available_blocks.account_history_block_number,
)
self.provider.with_rocksdb_tx(|rocks_tx_ref| {
let mut reader = EitherReader::new_accounts_history(self.provider, rocks_tx_ref)?;
reader.account_history_info(
address,
self.block_number,
self.lowest_available_blocks.account_history_block_number,
)
})
}
/// Lookup a storage key in the `StoragesHistory` table
/// Lookup a storage key in the `StoragesHistory` table using `EitherReader`.
pub fn storage_history_lookup(
&self,
address: Address,
storage_key: StorageKey,
) -> ProviderResult<HistoryInfo> {
) -> ProviderResult<HistoryInfo>
where
Provider: StorageSettingsCache + RocksDBProviderFactory + NodePrimitivesProvider,
{
if !self.lowest_available_blocks.is_storage_history_available(self.block_number) {
return Err(ProviderError::StateAtBlockPruned(self.block_number))
}
// history key to search IntegerList of block number changesets.
let history_key = StorageShardedKey::new(address, storage_key, self.block_number);
self.history_info::<tables::StoragesHistory, _>(
history_key,
|key| key.address == address && key.sharded_key.key == storage_key,
self.lowest_available_blocks.storage_history_block_number,
)
self.provider.with_rocksdb_tx(|rocks_tx_ref| {
let mut reader = EitherReader::new_storages_history(self.provider, rocks_tx_ref)?;
reader.storage_history_info(
address,
storage_key,
self.block_number,
self.lowest_available_blocks.storage_history_block_number,
)
})
}
/// Checks and returns `true` if distance to historical block exceeds the provided limit.
@@ -204,57 +207,6 @@ impl<'b, Provider: DBProvider + ChangeSetReader + BlockNumReader>
Ok(HashedStorage::from_reverts(self.tx(), address, self.block_number)?)
}
fn history_info<T, K>(
&self,
key: K,
key_filter: impl Fn(&K) -> bool,
lowest_available_block_number: Option<BlockNumber>,
) -> ProviderResult<HistoryInfo>
where
T: Table<Key = K, Value = BlockNumberList>,
{
let mut cursor = self.tx().cursor_read::<T>()?;
// Lookup the history chunk in the history index. If the key does not appear in the
// index, the first chunk for the next key will be returned so we filter out chunks that
// have a different key.
if let Some(chunk) = cursor.seek(key)?.filter(|(key, _)| key_filter(key)).map(|x| x.1) {
// Get the rank of the first entry before or equal to our block.
let mut rank = chunk.rank(self.block_number);
// Adjust the rank, so that we have the rank of the first entry strictly before our
// block (not equal to it).
if rank.checked_sub(1).and_then(|r| chunk.select(r)) == Some(self.block_number) {
rank -= 1;
}
let found_block = chunk.select(rank);
// If our block is before the first entry in the index chunk and this first entry
// doesn't equal to our block, it might be before the first write ever. To check, we
// look at the previous entry and check if the key is the same.
// This check is worth it, the `cursor.prev()` check is rarely triggered (the if will
// short-circuit) and when it passes we save a full seek into the changeset/plain state
// table.
let is_before_first_write =
needs_prev_shard_check(rank, found_block, self.block_number) &&
!cursor.prev()?.is_some_and(|(key, _)| key_filter(&key));
Ok(HistoryInfo::from_lookup(
found_block,
is_before_first_write,
lowest_available_block_number,
))
} else if lowest_available_block_number.is_some() {
// The key may have been written, but due to pruning we may not have changesets and
// history, so we need to make a plain state lookup.
Ok(HistoryInfo::MaybeInPlainState)
} else {
// The key has not been written to at all.
Ok(HistoryInfo::NotYetWritten)
}
}
/// Set the lowest block number at which the account history is available.
pub const fn with_lowest_available_account_history_block_number(
mut self,
@@ -280,8 +232,14 @@ impl<Provider: DBProvider + BlockNumReader> HistoricalStateProviderRef<'_, Provi
}
}
impl<Provider: DBProvider + BlockNumReader + ChangeSetReader> AccountReader
for HistoricalStateProviderRef<'_, Provider>
impl<
Provider: DBProvider
+ BlockNumReader
+ ChangeSetReader
+ StorageSettingsCache
+ RocksDBProviderFactory
+ NodePrimitivesProvider,
> AccountReader for HistoricalStateProviderRef<'_, Provider>
{
/// Get basic account information.
fn basic_account(&self, address: &Address) -> ProviderResult<Option<Account>> {
@@ -436,8 +394,15 @@ impl<Provider> HashedPostStateProvider for HistoricalStateProviderRef<'_, Provid
}
}
impl<Provider: DBProvider + BlockNumReader + BlockHashReader + ChangeSetReader> StateProvider
for HistoricalStateProviderRef<'_, Provider>
impl<
Provider: DBProvider
+ BlockNumReader
+ BlockHashReader
+ ChangeSetReader
+ StorageSettingsCache
+ RocksDBProviderFactory
+ NodePrimitivesProvider,
> StateProvider for HistoricalStateProviderRef<'_, Provider>
{
/// Get storage.
fn storage(
@@ -527,7 +492,7 @@ impl<Provider: DBProvider + ChangeSetReader + BlockNumReader> HistoricalStatePro
}
// Delegates all provider impls to [HistoricalStateProviderRef]
reth_storage_api::macros::delegate_provider_impls!(HistoricalStateProvider<Provider> where [Provider: DBProvider + BlockNumReader + BlockHashReader + ChangeSetReader]);
reth_storage_api::macros::delegate_provider_impls!(HistoricalStateProvider<Provider> where [Provider: DBProvider + BlockNumReader + BlockHashReader + ChangeSetReader + StorageSettingsCache + RocksDBProviderFactory + NodePrimitivesProvider]);
/// Lowest blocks at which different parts of the state are available.
/// They may be [Some] if pruning is enabled.
@@ -557,6 +522,29 @@ impl LowestAvailableBlocks {
}
}
/// Computes the rank and selected block from a history shard chunk.
///
/// Given a `BlockNumberList` (history shard) and a target block number, this function:
/// 1. Finds the rank of the first entry at or before `block_number`
/// 2. Adjusts the rank if the found entry equals `block_number` (so we get strictly before)
/// 3. Returns `(rank, found_block)` for use with [`needs_prev_shard_check`] and
/// [`HistoryInfo::from_lookup`]
///
/// This logic is shared between MDBX cursor-based lookups and `RocksDB` iterator lookups.
#[inline]
pub fn compute_history_rank(
chunk: &reth_db_api::BlockNumberList,
block_number: BlockNumber,
) -> (u64, Option<u64>) {
let mut rank = chunk.rank(block_number);
// Adjust the rank, so that we have the rank of the first entry strictly before
// our block (not equal to it).
if rank.checked_sub(1).and_then(|r| chunk.select(r)) == Some(block_number) {
rank -= 1;
}
(rank, chunk.select(rank))
}
/// Checks if a previous shard lookup is needed to determine if we're before the first write.
///
/// Returns `true` when `rank == 0` (first entry in shard) and the found block doesn't match
@@ -576,7 +564,8 @@ mod tests {
use crate::{
providers::state::historical::{HistoryInfo, LowestAvailableBlocks},
test_utils::create_test_provider_factory,
AccountReader, HistoricalStateProvider, HistoricalStateProviderRef, StateProvider,
AccountReader, HistoricalStateProvider, HistoricalStateProviderRef, RocksDBProviderFactory,
StateProvider,
};
use alloy_primitives::{address, b256, Address, B256, U256};
use reth_db_api::{
@@ -588,6 +577,7 @@ mod tests {
use reth_primitives_traits::{Account, StorageEntry};
use reth_storage_api::{
BlockHashReader, BlockNumReader, ChangeSetReader, DBProvider, DatabaseProviderFactory,
NodePrimitivesProvider, StorageSettingsCache,
};
use reth_storage_errors::provider::ProviderError;
@@ -599,7 +589,13 @@ mod tests {
const fn assert_state_provider<T: StateProvider>() {}
#[expect(dead_code)]
const fn assert_historical_state_provider<
T: DBProvider + BlockNumReader + BlockHashReader + ChangeSetReader,
T: DBProvider
+ BlockNumReader
+ BlockHashReader
+ ChangeSetReader
+ StorageSettingsCache
+ RocksDBProviderFactory
+ NodePrimitivesProvider,
>() {
assert_state_provider::<HistoricalStateProvider<T>>();
}

View File

@@ -14,6 +14,7 @@ use alloy_primitives::{b256, keccak256, Address, BlockHash, BlockNumber, TxHash,
use dashmap::DashMap;
use notify::{RecommendedWatcher, RecursiveMode, Watcher};
use parking_lot::RwLock;
use reth_chain_state::ExecutedBlock;
use reth_chainspec::{ChainInfo, ChainSpecProvider, EthChainSpec, NamedChain};
use reth_db::{
lockfile::StorageLock,
@@ -24,7 +25,7 @@ use reth_db::{
};
use reth_db_api::{
cursor::DbCursorRO,
models::StoredBlockBodyIndices,
models::{AccountBeforeTx, StoredBlockBodyIndices},
table::{Decompress, Table, Value},
tables,
transaction::DbTx,
@@ -32,7 +33,9 @@ use reth_db_api::{
use reth_ethereum_primitives::{Receipt, TransactionSigned};
use reth_nippy_jar::{NippyJar, NippyJarChecker, CONFIG_FILE_EXTENSION};
use reth_node_types::NodePrimitives;
use reth_primitives_traits::{RecoveredBlock, SealedHeader, SignedTransaction};
use reth_primitives_traits::{
AlloyBlockHeader as _, BlockBody as _, RecoveredBlock, SealedHeader, SignedTransaction,
};
use reth_stages_types::{PipelineTarget, StageId};
use reth_static_file_types::{
find_fixed_range, HighestStaticFiles, SegmentHeader, SegmentRangeInclusive, StaticFileMap,
@@ -41,15 +44,16 @@ use reth_static_file_types::{
use reth_storage_api::{
BlockBodyIndicesProvider, ChangeSetReader, DBProvider, StorageSettingsCache,
};
use reth_storage_errors::provider::{ProviderError, ProviderResult};
use reth_storage_errors::provider::{ProviderError, ProviderResult, StaticFileWriterError};
use std::{
collections::BTreeMap,
fmt::Debug,
ops::{Deref, Range, RangeBounds, RangeInclusive},
path::{Path, PathBuf},
sync::{atomic::AtomicU64, mpsc, Arc},
thread,
};
use tracing::{debug, info, trace, warn};
use tracing::{debug, info, instrument, trace, warn};
/// Alias type for a map that can be queried for block or transaction ranges. It uses `u64` to
/// represent either a block or a transaction number end of a static file range.
@@ -77,6 +81,25 @@ impl StaticFileAccess {
}
}
/// Context for static file block writes.
///
/// Contains target segments and pruning configuration.
#[derive(Debug, Clone, Copy, Default)]
pub struct StaticFileWriteCtx {
/// Whether transaction senders should be written to static files.
pub write_senders: bool,
/// Whether receipts should be written to static files.
pub write_receipts: bool,
/// Whether account changesets should be written to static files.
pub write_account_changesets: bool,
/// The current chain tip block number (for pruning).
pub tip: BlockNumber,
/// The prune mode for receipts, if any.
pub receipts_prune_mode: Option<reth_prune_types::PruneMode>,
/// Whether receipts are prunable (based on storage settings and prune distance).
pub receipts_prunable: bool,
}
/// [`StaticFileProvider`] manages all existing [`StaticFileJarProvider`].
///
/// "Static files" contain immutable chain history data, such as:
@@ -504,6 +527,192 @@ impl<N: NodePrimitives> StaticFileProvider<N> {
Ok(())
}
/// Writes headers for all blocks to the static file segment.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_headers(
w: &mut StaticFileProviderRWRefMut<'_, N>,
blocks: &[ExecutedBlock<N>],
) -> ProviderResult<()> {
for block in blocks {
let b = block.recovered_block();
w.append_header(b.header(), &b.hash())?;
}
Ok(())
}
/// Writes transactions for all blocks to the static file segment.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_transactions(
w: &mut StaticFileProviderRWRefMut<'_, N>,
blocks: &[ExecutedBlock<N>],
tx_nums: &[TxNumber],
) -> ProviderResult<()> {
for (block, &first_tx) in blocks.iter().zip(tx_nums) {
let b = block.recovered_block();
w.increment_block(b.number())?;
for (i, tx) in b.body().transactions().iter().enumerate() {
w.append_transaction(first_tx + i as u64, tx)?;
}
}
Ok(())
}
/// Writes transaction senders for all blocks to the static file segment.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_transaction_senders(
w: &mut StaticFileProviderRWRefMut<'_, N>,
blocks: &[ExecutedBlock<N>],
tx_nums: &[TxNumber],
) -> ProviderResult<()> {
for (block, &first_tx) in blocks.iter().zip(tx_nums) {
let b = block.recovered_block();
w.increment_block(b.number())?;
for (i, sender) in b.senders_iter().enumerate() {
w.append_transaction_sender(first_tx + i as u64, sender)?;
}
}
Ok(())
}
/// Writes receipts for all blocks to the static file segment.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_receipts(
w: &mut StaticFileProviderRWRefMut<'_, N>,
blocks: &[ExecutedBlock<N>],
tx_nums: &[TxNumber],
ctx: &StaticFileWriteCtx,
) -> ProviderResult<()> {
for (block, &first_tx) in blocks.iter().zip(tx_nums) {
let block_number = block.recovered_block().number();
w.increment_block(block_number)?;
// skip writing receipts if pruning configuration requires us to.
if ctx.receipts_prunable &&
ctx.receipts_prune_mode
.is_some_and(|mode| mode.should_prune(block_number, ctx.tip))
{
continue
}
for (i, receipt) in block.execution_outcome().receipts.iter().flatten().enumerate() {
w.append_receipt(first_tx + i as u64, receipt)?;
}
}
Ok(())
}
/// Writes account changesets for all blocks to the static file segment.
#[instrument(level = "debug", target = "providers::db", skip_all)]
fn write_account_changesets(
w: &mut StaticFileProviderRWRefMut<'_, N>,
blocks: &[ExecutedBlock<N>],
) -> ProviderResult<()> {
for block in blocks {
let block_number = block.recovered_block().number();
let reverts = block.execution_outcome().bundle.reverts.to_plain_state_reverts();
for account_block_reverts in reverts.accounts {
let changeset = account_block_reverts
.into_iter()
.map(|(address, info)| AccountBeforeTx { address, info: info.map(Into::into) })
.collect::<Vec<_>>();
w.append_account_changeset(changeset, block_number)?;
}
}
Ok(())
}
/// Spawns a scoped thread that writes to a static file segment using the provided closure.
///
/// The closure receives a mutable reference to the segment writer. After the closure completes,
/// `sync_all()` is called to flush writes to disk.
fn spawn_segment_writer<'scope, 'env, F>(
&'env self,
scope: &'scope thread::Scope<'scope, 'env>,
segment: StaticFileSegment,
first_block_number: BlockNumber,
f: F,
) -> thread::ScopedJoinHandle<'scope, ProviderResult<()>>
where
F: FnOnce(&mut StaticFileProviderRWRefMut<'_, N>) -> ProviderResult<()> + Send + 'env,
{
scope.spawn(move || {
let mut w = self.get_writer(first_block_number, segment)?;
f(&mut w)?;
w.sync_all()
})
}
/// Writes all static file data for multiple blocks in parallel per-segment.
///
/// This spawns separate threads for each segment type and each thread calls `sync_all()` on its
/// writer when done.
#[instrument(level = "debug", target = "providers::db", skip_all)]
pub fn write_blocks_data(
&self,
blocks: &[ExecutedBlock<N>],
tx_nums: &[TxNumber],
ctx: StaticFileWriteCtx,
) -> ProviderResult<()> {
if blocks.is_empty() {
return Ok(());
}
let first_block_number = blocks[0].recovered_block().number();
thread::scope(|s| {
let h_headers =
self.spawn_segment_writer(s, StaticFileSegment::Headers, first_block_number, |w| {
Self::write_headers(w, blocks)
});
let h_txs = self.spawn_segment_writer(
s,
StaticFileSegment::Transactions,
first_block_number,
|w| Self::write_transactions(w, blocks, tx_nums),
);
let h_senders = ctx.write_senders.then(|| {
self.spawn_segment_writer(
s,
StaticFileSegment::TransactionSenders,
first_block_number,
|w| Self::write_transaction_senders(w, blocks, tx_nums),
)
});
let h_receipts = ctx.write_receipts.then(|| {
self.spawn_segment_writer(s, StaticFileSegment::Receipts, first_block_number, |w| {
Self::write_receipts(w, blocks, tx_nums, &ctx)
})
});
let h_account_changesets = ctx.write_account_changesets.then(|| {
self.spawn_segment_writer(
s,
StaticFileSegment::AccountChangeSets,
first_block_number,
|w| Self::write_account_changesets(w, blocks),
)
});
h_headers.join().map_err(|_| StaticFileWriterError::ThreadPanic("headers"))??;
h_txs.join().map_err(|_| StaticFileWriterError::ThreadPanic("transactions"))??;
if let Some(h) = h_senders {
h.join().map_err(|_| StaticFileWriterError::ThreadPanic("senders"))??;
}
if let Some(h) = h_receipts {
h.join().map_err(|_| StaticFileWriterError::ThreadPanic("receipts"))??;
}
if let Some(h) = h_account_changesets {
h.join()
.map_err(|_| StaticFileWriterError::ThreadPanic("account_changesets"))??;
}
Ok(())
})
}
/// Gets the [`StaticFileJarProvider`] of the requested segment and start index that can be
/// either block or transaction.
pub fn get_segment_provider(

View File

@@ -1,6 +1,7 @@
mod manager;
pub use manager::{
StaticFileAccess, StaticFileProvider, StaticFileProviderBuilder, StaticFileWriter,
StaticFileAccess, StaticFileProvider, StaticFileProviderBuilder, StaticFileWriteCtx,
StaticFileWriter,
};
mod jar;

View File

@@ -206,6 +206,8 @@ pub struct StaticFileProviderRW<N> {
metrics: Option<Arc<StaticFileProviderMetrics>>,
/// On commit, contains the pruning strategy to apply for the segment.
prune_on_commit: Option<PruneStrategy>,
/// Whether `sync_all()` has been called. Used by `finalize()` to avoid redundant syncs.
synced: bool,
}
impl<N: NodePrimitives> StaticFileProviderRW<N> {
@@ -227,6 +229,7 @@ impl<N: NodePrimitives> StaticFileProviderRW<N> {
reader,
metrics,
prune_on_commit: None,
synced: false,
};
writer.ensure_end_range_consistency()?;
@@ -335,12 +338,13 @@ impl<N: NodePrimitives> StaticFileProviderRW<N> {
if self.writer.is_dirty() {
self.writer.sync_all().map_err(ProviderError::other)?;
}
self.synced = true;
Ok(())
}
/// Commits configuration to disk and updates the reader index.
///
/// Must be called after [`Self::sync_all`] to complete the commit.
/// If `sync_all()` was not called, this will call it first to ensure data is persisted.
///
/// Returns an error if prune is queued (use [`Self::commit`] instead).
pub fn finalize(&mut self) -> ProviderResult<()> {
@@ -348,9 +352,14 @@ impl<N: NodePrimitives> StaticFileProviderRW<N> {
return Err(StaticFileWriterError::FinalizeWithPruneQueued.into());
}
if self.writer.is_dirty() {
if !self.synced {
self.writer.sync_all().map_err(ProviderError::other)?;
}
self.writer.finalize().map_err(ProviderError::other)?;
self.update_index()?;
}
self.synced = false;
Ok(())
}

View File

@@ -27,7 +27,7 @@ impl<C: Send + Sync, N: NodePrimitives> StaticFileProviderFactory for NoopProvid
impl<C: Send + Sync, N: NodePrimitives> RocksDBProviderFactory for NoopProvider<C, N> {
fn rocksdb_provider(&self) -> RocksDBProvider {
RocksDBProvider::builder(PathBuf::default()).build().unwrap()
RocksDBProvider::builder(PathBuf::default()).with_default_tables().build().unwrap()
}
#[cfg(all(unix, feature = "rocksdb"))]

View File

@@ -1,4 +1,5 @@
use crate::providers::RocksDBProvider;
use crate::{either_writer::RocksTxRefArg, providers::RocksDBProvider};
use reth_storage_errors::provider::ProviderResult;
/// `RocksDB` provider factory.
///
@@ -13,4 +14,21 @@ pub trait RocksDBProviderFactory {
/// commits, ensuring atomicity across all storage backends.
#[cfg(all(unix, feature = "rocksdb"))]
fn set_pending_rocksdb_batch(&self, batch: rocksdb::WriteBatchWithTransaction<true>);
/// Executes a closure with a `RocksDB` transaction for reading.
///
/// This helper encapsulates all the cfg-gated `RocksDB` transaction handling for reads.
fn with_rocksdb_tx<F, R>(&self, f: F) -> ProviderResult<R>
where
F: FnOnce(RocksTxRefArg<'_>) -> ProviderResult<R>,
{
#[cfg(all(unix, feature = "rocksdb"))]
{
let rocksdb = self.rocksdb_provider();
let tx = rocksdb.tx();
f(&tx)
}
#[cfg(not(all(unix, feature = "rocksdb")))]
f(())
}
}

View File

@@ -13,7 +13,9 @@ mod tests {
use reth_ethereum_primitives::Receipt;
use reth_execution_types::ExecutionOutcome;
use reth_primitives_traits::{Account, StorageEntry};
use reth_storage_api::{DatabaseProviderFactory, HashedPostStateProvider, StateWriter};
use reth_storage_api::{
DatabaseProviderFactory, HashedPostStateProvider, StateWriteConfig, StateWriter,
};
use reth_trie::{
test_utils::{state_root, storage_root_prehashed},
HashedPostState, HashedStorage, StateRoot, StorageRoot, StorageRootProgress,
@@ -135,7 +137,7 @@ mod tests {
provider.write_state_changes(plain_state).expect("Could not write plain state to DB");
assert_eq!(reverts.storage, [[]]);
provider.write_state_reverts(reverts, 1).expect("Could not write reverts to DB");
provider.write_state_reverts(reverts, 1, StateWriteConfig::default()).expect("Could not write reverts to DB");
let reth_account_a = account_a.into();
let reth_account_b = account_b.into();
@@ -201,7 +203,7 @@ mod tests {
reverts.storage,
[[PlainStorageRevert { address: address_b, wiped: true, storage_revert: vec![] }]]
);
provider.write_state_reverts(reverts, 2).expect("Could not write reverts to DB");
provider.write_state_reverts(reverts, 2, StateWriteConfig::default()).expect("Could not write reverts to DB");
// Check new plain state for account B
assert_eq!(
@@ -280,7 +282,7 @@ mod tests {
let outcome = ExecutionOutcome::new(state.take_bundle(), Default::default(), 1, Vec::new());
provider
.write_state(&outcome, OriginalValuesKnown::Yes)
.write_state(&outcome, OriginalValuesKnown::Yes, StateWriteConfig::default())
.expect("Could not write bundle state to DB");
// Check plain storage state
@@ -380,7 +382,7 @@ mod tests {
state.merge_transitions(BundleRetention::Reverts);
let outcome = ExecutionOutcome::new(state.take_bundle(), Default::default(), 2, Vec::new());
provider
.write_state(&outcome, OriginalValuesKnown::Yes)
.write_state(&outcome, OriginalValuesKnown::Yes, StateWriteConfig::default())
.expect("Could not write bundle state to DB");
assert_eq!(
@@ -448,7 +450,7 @@ mod tests {
let outcome =
ExecutionOutcome::new(init_state.take_bundle(), Default::default(), 0, Vec::new());
provider
.write_state(&outcome, OriginalValuesKnown::Yes)
.write_state(&outcome, OriginalValuesKnown::Yes, StateWriteConfig::default())
.expect("Could not write bundle state to DB");
let mut state = State::builder().with_bundle_update().build();
@@ -607,7 +609,7 @@ mod tests {
let outcome: ExecutionOutcome =
ExecutionOutcome::new(bundle, Default::default(), 1, Vec::new());
provider
.write_state(&outcome, OriginalValuesKnown::Yes)
.write_state(&outcome, OriginalValuesKnown::Yes, StateWriteConfig::default())
.expect("Could not write bundle state to DB");
let mut storage_changeset_cursor = provider
@@ -773,7 +775,7 @@ mod tests {
let outcome =
ExecutionOutcome::new(init_state.take_bundle(), Default::default(), 0, Vec::new());
provider
.write_state(&outcome, OriginalValuesKnown::Yes)
.write_state(&outcome, OriginalValuesKnown::Yes, StateWriteConfig::default())
.expect("Could not write bundle state to DB");
let mut state = State::builder().with_bundle_update().build();
@@ -822,7 +824,7 @@ mod tests {
state.merge_transitions(BundleRetention::Reverts);
let outcome = ExecutionOutcome::new(state.take_bundle(), Default::default(), 1, Vec::new());
provider
.write_state(&outcome, OriginalValuesKnown::Yes)
.write_state(&outcome, OriginalValuesKnown::Yes, StateWriteConfig::default())
.expect("Could not write bundle state to DB");
let mut storage_changeset_cursor = provider

View File

@@ -12,21 +12,26 @@ pub trait StateWriter {
/// Receipt type included into [`ExecutionOutcome`].
type Receipt;
/// Write the state and receipts to the database or static files if `static_file_producer` is
/// `Some`. It should be `None` if there is any kind of pruning/filtering over the receipts.
/// Write the state and optionally receipts to the database.
///
/// Use `config` to skip writing certain data types when they are written elsewhere.
fn write_state(
&self,
execution_outcome: &ExecutionOutcome<Self::Receipt>,
is_value_known: OriginalValuesKnown,
config: StateWriteConfig,
) -> ProviderResult<()>;
/// Write state reverts to the database.
///
/// NOTE: Reverts will delete all wiped storage from plain state.
///
/// Use `config` to skip writing certain data types when they are written elsewhere.
fn write_state_reverts(
&self,
reverts: PlainStateReverts,
first_block: BlockNumber,
config: StateWriteConfig,
) -> ProviderResult<()>;
/// Write state changes to the database.
@@ -46,3 +51,20 @@ pub trait StateWriter {
block: BlockNumber,
) -> ProviderResult<ExecutionOutcome<Self::Receipt>>;
}
/// Configuration for what to write when calling [`StateWriter::write_state`].
///
/// Used to skip writing certain data types, when they are being written separately.
#[derive(Debug, Clone, Copy)]
pub struct StateWriteConfig {
/// Whether to write receipts.
pub write_receipts: bool,
/// Whether to write account changesets.
pub write_account_changesets: bool,
}
impl Default for StateWriteConfig {
fn default() -> Self {
Self { write_receipts: true, write_account_changesets: true }
}
}

View File

@@ -30,6 +30,9 @@
- [`reth db settings set receipts`](./reth/db/settings/set/receipts.mdx)
- [`reth db settings set transaction_senders`](./reth/db/settings/set/transaction_senders.mdx)
- [`reth db settings set account_changesets`](./reth/db/settings/set/account_changesets.mdx)
- [`reth db settings set storages_history`](./reth/db/settings/set/storages_history.mdx)
- [`reth db settings set account_history`](./reth/db/settings/set/account_history.mdx)
- [`reth db settings set tx_hash_numbers`](./reth/db/settings/set/tx_hash_numbers.mdx)
- [`reth db account-storage`](./reth/db/account-storage.mdx)
- [`reth download`](./reth/download.mdx)
- [`reth stage`](./reth/stage.mdx)
@@ -83,6 +86,9 @@
- [`op-reth db settings set receipts`](./op-reth/db/settings/set/receipts.mdx)
- [`op-reth db settings set transaction_senders`](./op-reth/db/settings/set/transaction_senders.mdx)
- [`op-reth db settings set account_changesets`](./op-reth/db/settings/set/account_changesets.mdx)
- [`op-reth db settings set storages_history`](./op-reth/db/settings/set/storages_history.mdx)
- [`op-reth db settings set account_history`](./op-reth/db/settings/set/account_history.mdx)
- [`op-reth db settings set tx_hash_numbers`](./op-reth/db/settings/set/tx_hash_numbers.mdx)
- [`op-reth db account-storage`](./op-reth/db/account-storage.mdx)
- [`op-reth stage`](./op-reth/stage.mdx)
- [`op-reth stage run`](./op-reth/stage/run.mdx)

View File

@@ -145,6 +145,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -12,6 +12,9 @@ Commands:
receipts Store receipts in static files instead of the database
transaction_senders Store transaction senders in static files instead of the database
account_changesets Store account changesets in static files instead of the database
storages_history Store storages history in RocksDB instead of MDBX
account_history Store account history in RocksDB instead of MDBX
tx_hash_numbers Store transaction hash numbers in RocksDB instead of MDBX
help Print this message or the help of the given subcommand(s)
Options:

View File

@@ -0,0 +1,152 @@
# op-reth db settings set account_history
Store account history in RocksDB instead of MDBX
```bash
$ op-reth db settings set account_history --help
```
```txt
Usage: op-reth db settings set account_history [OPTIONS] <VALUE>
Arguments:
<VALUE>
[possible values: true, false]
Options:
-h, --help
Print help (see a summary with '-h')
Datadir:
--chain <CHAIN_OR_PATH>
The chain this node is running.
Possible values are either a built-in chain or the path to a chain specification file.
Built-in chains:
optimism, optimism_sepolia, optimism-sepolia, base, base_sepolia, base-sepolia, arena-z, arena-z-sepolia, automata, base-devnet-0-sepolia-dev-0, bob, boba-sepolia, boba, camp-sepolia, celo, creator-chain-testnet-sepolia, cyber, cyber-sepolia, ethernity, ethernity-sepolia, fraxtal, funki, funki-sepolia, hashkeychain, ink, ink-sepolia, lisk, lisk-sepolia, lyra, metal, metal-sepolia, mint, mode, mode-sepolia, oplabs-devnet-0-sepolia-dev-0, orderly, ozean-sepolia, pivotal-sepolia, polynomial, race, race-sepolia, radius_testnet-sepolia, redstone, rehearsal-0-bn-0-rehearsal-0-bn, rehearsal-0-bn-1-rehearsal-0-bn, settlus-mainnet, settlus-sepolia-sepolia, shape, shape-sepolia, silent-data-mainnet, snax, soneium, soneium-minato-sepolia, sseed, swan, swell, tbn, tbn-sepolia, unichain, unichain-sepolia, worldchain, worldchain-sepolia, xterio-eth, zora, zora-sepolia, dev
[default: optimism]
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.stdout.filter <FILTER>
The filter to use for logs written to stdout
[default: ]
--log.file.format <FORMAT>
The format to use for logs written to the log file
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.file.filter <FILTER>
The filter to use for logs written to the log file
[default: debug]
--log.file.directory <PATH>
The path to put log files in
[default: <CACHE_DIR>/logs]
--log.file.name <NAME>
The prefix name of the log files
[default: reth.log]
--log.file.max-size <SIZE>
The maximum size (in MB) of one log file
[default: 200]
--log.file.max-files <COUNT>
The maximum amount of log files that will be stored. If set to 0, background file logging is disabled
[default: 5]
--log.journald
Write logs to journald
--log.journald.filter <FILTER>
The filter to use for logs written to journald
[default: error]
--color <COLOR>
Sets whether or not the formatter emits ANSI terminal escape codes for colors and other text formatting
Possible values:
- always: Colors on
- auto: Auto-detect
- never: Colors off
[default: always]
Display:
-v, --verbosity...
Set the minimum log level.
-v Errors
-vv Warnings
-vvv Info
-vvvv Debug
-vvvvv Traces (warning: very verbose!)
-q, --quiet
Silence all log output
Tracing:
--tracing-otlp[=<URL>]
Enable `Opentelemetry` tracing export to an OTLP endpoint.
If no value provided, defaults based on protocol: - HTTP: `http://localhost:4318/v1/traces` - gRPC: `http://localhost:4317`
Example: --tracing-otlp=http://collector:4318/v1/traces
[env: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=]
--tracing-otlp-protocol <PROTOCOL>
OTLP transport protocol to use for exporting traces.
- `http`: expects endpoint path to end with `/v1/traces` - `grpc`: expects endpoint without a path
Defaults to HTTP if not specified.
Possible values:
- http: HTTP/Protobuf transport, port 4318, requires `/v1/traces` path
- grpc: gRPC transport, port 4317
[env: OTEL_EXPORTER_OTLP_PROTOCOL=]
[default: http]
--tracing-otlp.filter <FILTER>
Set a filter directive for the OTLP tracer. This controls the verbosity of spans and events sent to the OTLP endpoint. It follows the same syntax as the `RUST_LOG` environment variable.
Example: --tracing-otlp.filter=info,reth=debug,hyper_util=off
Defaults to TRACE if not specified.
[default: debug]
--tracing-otlp.sample-ratio <RATIO>
Trace sampling ratio to control the percentage of traces to export.
Valid range: 0.0 to 1.0 - 1.0, default: Sample all traces - 0.01: Sample 1% of traces - 0.0: Disable sampling
Example: --tracing-otlp.sample-ratio=0.0.
[env: OTEL_TRACES_SAMPLER_ARG=]
```

View File

@@ -0,0 +1,152 @@
# op-reth db settings set storages_history
Store storages history in RocksDB instead of MDBX
```bash
$ op-reth db settings set storages_history --help
```
```txt
Usage: op-reth db settings set storages_history [OPTIONS] <VALUE>
Arguments:
<VALUE>
[possible values: true, false]
Options:
-h, --help
Print help (see a summary with '-h')
Datadir:
--chain <CHAIN_OR_PATH>
The chain this node is running.
Possible values are either a built-in chain or the path to a chain specification file.
Built-in chains:
optimism, optimism_sepolia, optimism-sepolia, base, base_sepolia, base-sepolia, arena-z, arena-z-sepolia, automata, base-devnet-0-sepolia-dev-0, bob, boba-sepolia, boba, camp-sepolia, celo, creator-chain-testnet-sepolia, cyber, cyber-sepolia, ethernity, ethernity-sepolia, fraxtal, funki, funki-sepolia, hashkeychain, ink, ink-sepolia, lisk, lisk-sepolia, lyra, metal, metal-sepolia, mint, mode, mode-sepolia, oplabs-devnet-0-sepolia-dev-0, orderly, ozean-sepolia, pivotal-sepolia, polynomial, race, race-sepolia, radius_testnet-sepolia, redstone, rehearsal-0-bn-0-rehearsal-0-bn, rehearsal-0-bn-1-rehearsal-0-bn, settlus-mainnet, settlus-sepolia-sepolia, shape, shape-sepolia, silent-data-mainnet, snax, soneium, soneium-minato-sepolia, sseed, swan, swell, tbn, tbn-sepolia, unichain, unichain-sepolia, worldchain, worldchain-sepolia, xterio-eth, zora, zora-sepolia, dev
[default: optimism]
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.stdout.filter <FILTER>
The filter to use for logs written to stdout
[default: ]
--log.file.format <FORMAT>
The format to use for logs written to the log file
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.file.filter <FILTER>
The filter to use for logs written to the log file
[default: debug]
--log.file.directory <PATH>
The path to put log files in
[default: <CACHE_DIR>/logs]
--log.file.name <NAME>
The prefix name of the log files
[default: reth.log]
--log.file.max-size <SIZE>
The maximum size (in MB) of one log file
[default: 200]
--log.file.max-files <COUNT>
The maximum amount of log files that will be stored. If set to 0, background file logging is disabled
[default: 5]
--log.journald
Write logs to journald
--log.journald.filter <FILTER>
The filter to use for logs written to journald
[default: error]
--color <COLOR>
Sets whether or not the formatter emits ANSI terminal escape codes for colors and other text formatting
Possible values:
- always: Colors on
- auto: Auto-detect
- never: Colors off
[default: always]
Display:
-v, --verbosity...
Set the minimum log level.
-v Errors
-vv Warnings
-vvv Info
-vvvv Debug
-vvvvv Traces (warning: very verbose!)
-q, --quiet
Silence all log output
Tracing:
--tracing-otlp[=<URL>]
Enable `Opentelemetry` tracing export to an OTLP endpoint.
If no value provided, defaults based on protocol: - HTTP: `http://localhost:4318/v1/traces` - gRPC: `http://localhost:4317`
Example: --tracing-otlp=http://collector:4318/v1/traces
[env: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=]
--tracing-otlp-protocol <PROTOCOL>
OTLP transport protocol to use for exporting traces.
- `http`: expects endpoint path to end with `/v1/traces` - `grpc`: expects endpoint without a path
Defaults to HTTP if not specified.
Possible values:
- http: HTTP/Protobuf transport, port 4318, requires `/v1/traces` path
- grpc: gRPC transport, port 4317
[env: OTEL_EXPORTER_OTLP_PROTOCOL=]
[default: http]
--tracing-otlp.filter <FILTER>
Set a filter directive for the OTLP tracer. This controls the verbosity of spans and events sent to the OTLP endpoint. It follows the same syntax as the `RUST_LOG` environment variable.
Example: --tracing-otlp.filter=info,reth=debug,hyper_util=off
Defaults to TRACE if not specified.
[default: debug]
--tracing-otlp.sample-ratio <RATIO>
Trace sampling ratio to control the percentage of traces to export.
Valid range: 0.0 to 1.0 - 1.0, default: Sample all traces - 0.01: Sample 1% of traces - 0.0: Disable sampling
Example: --tracing-otlp.sample-ratio=0.0.
[env: OTEL_TRACES_SAMPLER_ARG=]
```

View File

@@ -0,0 +1,152 @@
# op-reth db settings set tx_hash_numbers
Store transaction hash numbers in RocksDB instead of MDBX
```bash
$ op-reth db settings set tx_hash_numbers --help
```
```txt
Usage: op-reth db settings set tx_hash_numbers [OPTIONS] <VALUE>
Arguments:
<VALUE>
[possible values: true, false]
Options:
-h, --help
Print help (see a summary with '-h')
Datadir:
--chain <CHAIN_OR_PATH>
The chain this node is running.
Possible values are either a built-in chain or the path to a chain specification file.
Built-in chains:
optimism, optimism_sepolia, optimism-sepolia, base, base_sepolia, base-sepolia, arena-z, arena-z-sepolia, automata, base-devnet-0-sepolia-dev-0, bob, boba-sepolia, boba, camp-sepolia, celo, creator-chain-testnet-sepolia, cyber, cyber-sepolia, ethernity, ethernity-sepolia, fraxtal, funki, funki-sepolia, hashkeychain, ink, ink-sepolia, lisk, lisk-sepolia, lyra, metal, metal-sepolia, mint, mode, mode-sepolia, oplabs-devnet-0-sepolia-dev-0, orderly, ozean-sepolia, pivotal-sepolia, polynomial, race, race-sepolia, radius_testnet-sepolia, redstone, rehearsal-0-bn-0-rehearsal-0-bn, rehearsal-0-bn-1-rehearsal-0-bn, settlus-mainnet, settlus-sepolia-sepolia, shape, shape-sepolia, silent-data-mainnet, snax, soneium, soneium-minato-sepolia, sseed, swan, swell, tbn, tbn-sepolia, unichain, unichain-sepolia, worldchain, worldchain-sepolia, xterio-eth, zora, zora-sepolia, dev
[default: optimism]
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.stdout.filter <FILTER>
The filter to use for logs written to stdout
[default: ]
--log.file.format <FORMAT>
The format to use for logs written to the log file
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.file.filter <FILTER>
The filter to use for logs written to the log file
[default: debug]
--log.file.directory <PATH>
The path to put log files in
[default: <CACHE_DIR>/logs]
--log.file.name <NAME>
The prefix name of the log files
[default: reth.log]
--log.file.max-size <SIZE>
The maximum size (in MB) of one log file
[default: 200]
--log.file.max-files <COUNT>
The maximum amount of log files that will be stored. If set to 0, background file logging is disabled
[default: 5]
--log.journald
Write logs to journald
--log.journald.filter <FILTER>
The filter to use for logs written to journald
[default: error]
--color <COLOR>
Sets whether or not the formatter emits ANSI terminal escape codes for colors and other text formatting
Possible values:
- always: Colors on
- auto: Auto-detect
- never: Colors off
[default: always]
Display:
-v, --verbosity...
Set the minimum log level.
-v Errors
-vv Warnings
-vvv Info
-vvvv Debug
-vvvvv Traces (warning: very verbose!)
-q, --quiet
Silence all log output
Tracing:
--tracing-otlp[=<URL>]
Enable `Opentelemetry` tracing export to an OTLP endpoint.
If no value provided, defaults based on protocol: - HTTP: `http://localhost:4318/v1/traces` - gRPC: `http://localhost:4317`
Example: --tracing-otlp=http://collector:4318/v1/traces
[env: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=]
--tracing-otlp-protocol <PROTOCOL>
OTLP transport protocol to use for exporting traces.
- `http`: expects endpoint path to end with `/v1/traces` - `grpc`: expects endpoint without a path
Defaults to HTTP if not specified.
Possible values:
- http: HTTP/Protobuf transport, port 4318, requires `/v1/traces` path
- grpc: gRPC transport, port 4317
[env: OTEL_EXPORTER_OTLP_PROTOCOL=]
[default: http]
--tracing-otlp.filter <FILTER>
Set a filter directive for the OTLP tracer. This controls the verbosity of spans and events sent to the OTLP endpoint. It follows the same syntax as the `RUST_LOG` environment variable.
Example: --tracing-otlp.filter=info,reth=debug,hyper_util=off
Defaults to TRACE if not specified.
[default: debug]
--tracing-otlp.sample-ratio <RATIO>
Trace sampling ratio to control the percentage of traces to export.
Valid range: 0.0 to 1.0 - 1.0, default: Sample all traces - 0.01: Sample 1% of traces - 0.0: Disable sampling
Example: --tracing-otlp.sample-ratio=0.0.
[env: OTEL_TRACES_SAMPLER_ARG=]
```

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--chunk-len <CHUNK_LEN>
Chunk byte length to read from file.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--chunk-len <CHUNK_LEN>
Chunk byte length to read from file.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--without-evm
Specifies whether to initialize the state without relying on EVM historical data.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -1019,6 +1019,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Rollup:
--rollup.sequencer <SEQUENCER>
Endpoint for the sequencer mempool (can be both HTTP and WS)

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--from <FROM>
The height to start at

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
<STAGE>
Possible values:
- headers: The headers stage within the pipeline

View File

@@ -136,6 +136,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--metrics <SOCKET>
Enable Prometheus metrics.

View File

@@ -134,6 +134,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--offline
If this is enabled, then all stages except headers, bodies, and sender recovery will be unwound

View File

@@ -145,6 +145,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -12,6 +12,9 @@ Commands:
receipts Store receipts in static files instead of the database
transaction_senders Store transaction senders in static files instead of the database
account_changesets Store account changesets in static files instead of the database
storages_history Store storages history in RocksDB instead of MDBX
account_history Store account history in RocksDB instead of MDBX
tx_hash_numbers Store transaction hash numbers in RocksDB instead of MDBX
help Print this message or the help of the given subcommand(s)
Options:

View File

@@ -0,0 +1,152 @@
# reth db settings set account_history
Store account history in RocksDB instead of MDBX
```bash
$ reth db settings set account_history --help
```
```txt
Usage: reth db settings set account_history [OPTIONS] <VALUE>
Arguments:
<VALUE>
[possible values: true, false]
Options:
-h, --help
Print help (see a summary with '-h')
Datadir:
--chain <CHAIN_OR_PATH>
The chain this node is running.
Possible values are either a built-in chain or the path to a chain specification file.
Built-in chains:
mainnet, sepolia, holesky, hoodi, dev
[default: mainnet]
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.stdout.filter <FILTER>
The filter to use for logs written to stdout
[default: ]
--log.file.format <FORMAT>
The format to use for logs written to the log file
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.file.filter <FILTER>
The filter to use for logs written to the log file
[default: debug]
--log.file.directory <PATH>
The path to put log files in
[default: <CACHE_DIR>/logs]
--log.file.name <NAME>
The prefix name of the log files
[default: reth.log]
--log.file.max-size <SIZE>
The maximum size (in MB) of one log file
[default: 200]
--log.file.max-files <COUNT>
The maximum amount of log files that will be stored. If set to 0, background file logging is disabled
[default: 5]
--log.journald
Write logs to journald
--log.journald.filter <FILTER>
The filter to use for logs written to journald
[default: error]
--color <COLOR>
Sets whether or not the formatter emits ANSI terminal escape codes for colors and other text formatting
Possible values:
- always: Colors on
- auto: Auto-detect
- never: Colors off
[default: always]
Display:
-v, --verbosity...
Set the minimum log level.
-v Errors
-vv Warnings
-vvv Info
-vvvv Debug
-vvvvv Traces (warning: very verbose!)
-q, --quiet
Silence all log output
Tracing:
--tracing-otlp[=<URL>]
Enable `Opentelemetry` tracing export to an OTLP endpoint.
If no value provided, defaults based on protocol: - HTTP: `http://localhost:4318/v1/traces` - gRPC: `http://localhost:4317`
Example: --tracing-otlp=http://collector:4318/v1/traces
[env: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=]
--tracing-otlp-protocol <PROTOCOL>
OTLP transport protocol to use for exporting traces.
- `http`: expects endpoint path to end with `/v1/traces` - `grpc`: expects endpoint without a path
Defaults to HTTP if not specified.
Possible values:
- http: HTTP/Protobuf transport, port 4318, requires `/v1/traces` path
- grpc: gRPC transport, port 4317
[env: OTEL_EXPORTER_OTLP_PROTOCOL=]
[default: http]
--tracing-otlp.filter <FILTER>
Set a filter directive for the OTLP tracer. This controls the verbosity of spans and events sent to the OTLP endpoint. It follows the same syntax as the `RUST_LOG` environment variable.
Example: --tracing-otlp.filter=info,reth=debug,hyper_util=off
Defaults to TRACE if not specified.
[default: debug]
--tracing-otlp.sample-ratio <RATIO>
Trace sampling ratio to control the percentage of traces to export.
Valid range: 0.0 to 1.0 - 1.0, default: Sample all traces - 0.01: Sample 1% of traces - 0.0: Disable sampling
Example: --tracing-otlp.sample-ratio=0.0.
[env: OTEL_TRACES_SAMPLER_ARG=]
```

View File

@@ -0,0 +1,152 @@
# reth db settings set storages_history
Store storages history in RocksDB instead of MDBX
```bash
$ reth db settings set storages_history --help
```
```txt
Usage: reth db settings set storages_history [OPTIONS] <VALUE>
Arguments:
<VALUE>
[possible values: true, false]
Options:
-h, --help
Print help (see a summary with '-h')
Datadir:
--chain <CHAIN_OR_PATH>
The chain this node is running.
Possible values are either a built-in chain or the path to a chain specification file.
Built-in chains:
mainnet, sepolia, holesky, hoodi, dev
[default: mainnet]
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.stdout.filter <FILTER>
The filter to use for logs written to stdout
[default: ]
--log.file.format <FORMAT>
The format to use for logs written to the log file
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.file.filter <FILTER>
The filter to use for logs written to the log file
[default: debug]
--log.file.directory <PATH>
The path to put log files in
[default: <CACHE_DIR>/logs]
--log.file.name <NAME>
The prefix name of the log files
[default: reth.log]
--log.file.max-size <SIZE>
The maximum size (in MB) of one log file
[default: 200]
--log.file.max-files <COUNT>
The maximum amount of log files that will be stored. If set to 0, background file logging is disabled
[default: 5]
--log.journald
Write logs to journald
--log.journald.filter <FILTER>
The filter to use for logs written to journald
[default: error]
--color <COLOR>
Sets whether or not the formatter emits ANSI terminal escape codes for colors and other text formatting
Possible values:
- always: Colors on
- auto: Auto-detect
- never: Colors off
[default: always]
Display:
-v, --verbosity...
Set the minimum log level.
-v Errors
-vv Warnings
-vvv Info
-vvvv Debug
-vvvvv Traces (warning: very verbose!)
-q, --quiet
Silence all log output
Tracing:
--tracing-otlp[=<URL>]
Enable `Opentelemetry` tracing export to an OTLP endpoint.
If no value provided, defaults based on protocol: - HTTP: `http://localhost:4318/v1/traces` - gRPC: `http://localhost:4317`
Example: --tracing-otlp=http://collector:4318/v1/traces
[env: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=]
--tracing-otlp-protocol <PROTOCOL>
OTLP transport protocol to use for exporting traces.
- `http`: expects endpoint path to end with `/v1/traces` - `grpc`: expects endpoint without a path
Defaults to HTTP if not specified.
Possible values:
- http: HTTP/Protobuf transport, port 4318, requires `/v1/traces` path
- grpc: gRPC transport, port 4317
[env: OTEL_EXPORTER_OTLP_PROTOCOL=]
[default: http]
--tracing-otlp.filter <FILTER>
Set a filter directive for the OTLP tracer. This controls the verbosity of spans and events sent to the OTLP endpoint. It follows the same syntax as the `RUST_LOG` environment variable.
Example: --tracing-otlp.filter=info,reth=debug,hyper_util=off
Defaults to TRACE if not specified.
[default: debug]
--tracing-otlp.sample-ratio <RATIO>
Trace sampling ratio to control the percentage of traces to export.
Valid range: 0.0 to 1.0 - 1.0, default: Sample all traces - 0.01: Sample 1% of traces - 0.0: Disable sampling
Example: --tracing-otlp.sample-ratio=0.0.
[env: OTEL_TRACES_SAMPLER_ARG=]
```

View File

@@ -0,0 +1,152 @@
# reth db settings set tx_hash_numbers
Store transaction hash numbers in RocksDB instead of MDBX
```bash
$ reth db settings set tx_hash_numbers --help
```
```txt
Usage: reth db settings set tx_hash_numbers [OPTIONS] <VALUE>
Arguments:
<VALUE>
[possible values: true, false]
Options:
-h, --help
Print help (see a summary with '-h')
Datadir:
--chain <CHAIN_OR_PATH>
The chain this node is running.
Possible values are either a built-in chain or the path to a chain specification file.
Built-in chains:
mainnet, sepolia, holesky, hoodi, dev
[default: mainnet]
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.stdout.filter <FILTER>
The filter to use for logs written to stdout
[default: ]
--log.file.format <FORMAT>
The format to use for logs written to the log file
Possible values:
- json: Represents JSON formatting for logs. This format outputs log records as JSON objects, making it suitable for structured logging
- log-fmt: Represents logfmt (key=value) formatting for logs. This format is concise and human-readable, typically used in command-line applications
- terminal: Represents terminal-friendly formatting for logs
[default: terminal]
--log.file.filter <FILTER>
The filter to use for logs written to the log file
[default: debug]
--log.file.directory <PATH>
The path to put log files in
[default: <CACHE_DIR>/logs]
--log.file.name <NAME>
The prefix name of the log files
[default: reth.log]
--log.file.max-size <SIZE>
The maximum size (in MB) of one log file
[default: 200]
--log.file.max-files <COUNT>
The maximum amount of log files that will be stored. If set to 0, background file logging is disabled
[default: 5]
--log.journald
Write logs to journald
--log.journald.filter <FILTER>
The filter to use for logs written to journald
[default: error]
--color <COLOR>
Sets whether or not the formatter emits ANSI terminal escape codes for colors and other text formatting
Possible values:
- always: Colors on
- auto: Auto-detect
- never: Colors off
[default: always]
Display:
-v, --verbosity...
Set the minimum log level.
-v Errors
-vv Warnings
-vvv Info
-vvvv Debug
-vvvvv Traces (warning: very verbose!)
-q, --quiet
Silence all log output
Tracing:
--tracing-otlp[=<URL>]
Enable `Opentelemetry` tracing export to an OTLP endpoint.
If no value provided, defaults based on protocol: - HTTP: `http://localhost:4318/v1/traces` - gRPC: `http://localhost:4317`
Example: --tracing-otlp=http://collector:4318/v1/traces
[env: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=]
--tracing-otlp-protocol <PROTOCOL>
OTLP transport protocol to use for exporting traces.
- `http`: expects endpoint path to end with `/v1/traces` - `grpc`: expects endpoint without a path
Defaults to HTTP if not specified.
Possible values:
- http: HTTP/Protobuf transport, port 4318, requires `/v1/traces` path
- grpc: gRPC transport, port 4317
[env: OTEL_EXPORTER_OTLP_PROTOCOL=]
[default: http]
--tracing-otlp.filter <FILTER>
Set a filter directive for the OTLP tracer. This controls the verbosity of spans and events sent to the OTLP endpoint. It follows the same syntax as the `RUST_LOG` environment variable.
Example: --tracing-otlp.filter=info,reth=debug,hyper_util=off
Defaults to TRACE if not specified.
[default: debug]
--tracing-otlp.sample-ratio <RATIO>
Trace sampling ratio to control the percentage of traces to export.
Valid range: 0.0 to 1.0 - 1.0, default: Sample all traces - 0.01: Sample 1% of traces - 0.0: Disable sampling
Example: --tracing-otlp.sample-ratio=0.0.
[env: OTEL_TRACES_SAMPLER_ARG=]
```

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
-u, --url <URL>
Specify a snapshot URL or let the command propose a default one.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--first-block-number <first-block-number>
Optional first block number to export from the db.
It is by default 0.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--path <IMPORT_ERA_PATH>
The path to a directory for import.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--no-state
Disables stages that require state.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--without-evm
Specifies whether to initialize the state without relying on EVM historical data.

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -1019,6 +1019,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Ress:
--ress.enable
Enable support for `ress` subprotocol

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--from <FROM>
The height to start at

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
<STAGE>
Possible values:
- headers: The headers stage within the pipeline

View File

@@ -136,6 +136,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
Logging:
--log.stdout.format <FORMAT>
The format to use for logs written to stdout

View File

@@ -129,6 +129,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--metrics <SOCKET>
Enable Prometheus metrics.

View File

@@ -134,6 +134,13 @@ Static Files:
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--storage.rocksdb
Use `RocksDB` for history indices instead of MDBX.
When enabled, `AccountsHistory`, `StoragesHistory`, and `TransactionHashNumbers` tables will be stored in `RocksDB` for better write performance.
Note: This setting can only be configured at genesis initialization. Once the node has been initialized, changing this flag requires re-syncing from scratch.
--offline
If this is enabled, then all stages except headers, bodies, and sender recovery will be unwound

View File

@@ -136,6 +136,18 @@ export const opRethCliSidebar: SidebarItem = {
{
text: "op-reth db settings set account_changesets",
link: "/cli/op-reth/db/settings/set/account_changesets"
},
{
text: "op-reth db settings set storages_history",
link: "/cli/op-reth/db/settings/set/storages_history"
},
{
text: "op-reth db settings set account_history",
link: "/cli/op-reth/db/settings/set/account_history"
},
{
text: "op-reth db settings set tx_hash_numbers",
link: "/cli/op-reth/db/settings/set/tx_hash_numbers"
}
]
}

View File

@@ -140,6 +140,18 @@ export const rethCliSidebar: SidebarItem = {
{
text: "reth db settings set account_changesets",
link: "/cli/reth/db/settings/set/account_changesets"
},
{
text: "reth db settings set storages_history",
link: "/cli/reth/db/settings/set/storages_history"
},
{
text: "reth db settings set account_history",
link: "/cli/reth/db/settings/set/account_history"
},
{
text: "reth db settings set tx_hash_numbers",
link: "/cli/reth/db/settings/set/tx_hash_numbers"
}
]
}

View File

@@ -17,7 +17,7 @@ use reth_primitives_traits::{Block as BlockTrait, RecoveredBlock, SealedBlock};
use reth_provider::{
test_utils::create_test_provider_factory_with_chain_spec, BlockWriter, DatabaseProviderFactory,
ExecutionOutcome, HeaderProvider, HistoryWriter, OriginalValuesKnown, StateProofProvider,
StateWriter, StaticFileProviderFactory, StaticFileSegment, StaticFileWriter,
StateWriteConfig, StateWriter, StaticFileProviderFactory, StaticFileSegment, StaticFileWriter,
};
use reth_revm::{database::StateProviderDatabase, witness::ExecutionWitnessRecord, State};
use reth_stateless::{
@@ -325,7 +325,11 @@ fn run_case(
// Commit the post state/state diff to the database
provider
.write_state(&ExecutionOutcome::single(block.number, output), OriginalValuesKnown::Yes)
.write_state(
&ExecutionOutcome::single(block.number, output),
OriginalValuesKnown::Yes,
StateWriteConfig::default(),
)
.map_err(|err| Error::block_failed(block_number, program_inputs.clone(), err))?;
provider