Compare commits

...

55 Commits

Author SHA1 Message Date
yongkangc
96673921e6 comment 2025-12-23 08:59:29 +00:00
yongkangc
5910807b94 reverted 2025-12-23 08:53:48 +00:00
yongkangc
ef678bfe37 fix: unwind when rocksdb history lags checkpoint 2025-12-23 08:47:16 +00:00
yongkangc
8eb4ba2f65 feat(provider): add check_file_consistency for proper ordering
Addresses Joshi's review comment about consistency check ordering.

The ordering is now:
1. check_file_consistency() - heals NippyJar files without pruning
2. rocksdb check_consistency() - needs static file tx data
3. static file check_consistency() - compares with MDBX, may prune

This ensures RocksDB can access transaction data before static files
potentially prune it during checkpoint comparison.
2025-12-23 08:19:11 +00:00
YK
9ff7a228b3 del 2025-12-23 16:06:48 +08:00
yongkangc
a2fec130f6 fix: adapt to #20508 API and add const to stub functions
- Remove non-existent check_file_consistency() call
- Simplify ordering: static file check first, then RocksDB check
- Add const to first() and last() stub functions
2025-12-23 04:20:34 +00:00
yongkangc
8a7a0f2f16 fix: make stub iter return empty iterator instead of error
This is consistent with first() and last() returning Ok(None) -
the stub behaves as if the database is empty rather than unavailable.
2025-12-23 04:17:07 +00:00
yongkangc
d3e812a985 fix: update consistency check ordering per Joshi's feedback
Reorder consistency checks as follows:
1. StaticFileProvider::check_file_consistency() - heals interrupted writes
2. RocksDB::check_consistency() - uses static file data for tx hash pruning
3. StaticFileProvider::check_consistency() - compares with MDBX checkpoints

This ensures RocksDB can access transaction data before static files prune it.

Amp-Thread-ID: https://ampcode.com/threads/T-019b3076-f7ed-700d-8916-779712facb8e
2025-12-23 04:17:07 +00:00
yongkangc
5dc4b5248e fix: address Joshi's review comments
- Revert 'allow RocksDB unwind to 0' - keep panic on unwind to 0 as it would
  leave MDBX with huge free list size
- Remove checked_add overflow protection (not needed in practice)
- Created issue #20506 for refactoring StaticFileProvider::check_consistency

Amp-Thread-ID: https://ampcode.com/threads/T-019b3076-f7ed-700d-8916-779712facb8e
2025-12-23 04:17:07 +00:00
yongkangc
d358d62353 fix: allow RocksDB unwind to block 0 for migration scenario
Address review feedback: nodes migrating to RocksDB will commonly trigger
unwind to block 0, which is expected. Only panic if static files also
require unwind to 0 (indicating corruption). Otherwise, warn and proceed
with full resync.
2025-12-23 04:17:07 +00:00
yongkangc
f741dd4792 fix(storage): address review findings for RocksDB consistency check
- Add missing methods to rocksdb_stub.rs (batch, first, last, iter, commit, RocksDBIter)
- Add checkpoint > 0 guard in (None, Some(highest_tx)) case for TransactionHashNumbers
- Use checked_add for mdbx_tx + 1 to prevent u64 overflow
- Update misleading docstring in stub's RocksDBProvider::new

Amp-Thread-ID: https://ampcode.com/threads/T-019b3076-f7ed-700d-8916-779712facb8e
2025-12-23 04:17:07 +00:00
yongkangc
3c776bbbc8 fix: add const to stub check_consistency fn 2025-12-23 04:17:07 +00:00
yongkangc
2a77d5f6de chore: format code 2025-12-23 04:17:07 +00:00
yongkangc
fb8eb0e3e6 feat(storage): add RocksDB consistency check on startup
Amp-Thread-ID: https://ampcode.com/threads/T-019b2fb1-ce5c-7251-b454-0d7472a0754a
2025-12-23 04:17:00 +00:00
DaniPopes
353c2a7f70 fix(cli): remove unnecessary bound from Cli::configure (#20583) 2025-12-23 03:52:04 +00:00
Matthias Seitz
21934d9946 fix: fuse shutdown (#20580) 2025-12-23 01:09:45 +00:00
cui
538de9e456 feat: update fork id in discv5[WIP] (#19139)
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
2025-12-23 00:30:36 +00:00
forkfury
b9d14d4a54 chore: delete redundant todo comment (#20571) 2025-12-23 00:14:05 +00:00
Matthew Vauxhall
529aa83777 chore: remove block_to_payload_v3 (#20540)
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
2025-12-23 00:10:38 +00:00
DaniPopes
da10201b88 chore: minor reth-bench cleanup (#20577) 2025-12-22 23:56:36 +00:00
Arsenii Kulikov
eec76a3faf perf: spawn prewarm workers in parallel (#20575) 2025-12-22 20:41:52 +00:00
Arsenii Kulikov
5e4a219182 perf: spawn prewarming before multiproof (#20572)
Co-authored-by: Brian Picciano <me@mediocregopher.com>
2025-12-22 17:56:14 +00:00
AJStonewee
ccb897f9a0 refactor(stages): cache hashed address in storage hashing loop (#20318)
Co-authored-by: Brian Picciano <me@mediocregopher.com>
2025-12-22 16:05:46 +00:00
radik878
f9d872e9cb fix(net): correct config builder doc comments (#20299) 2025-12-22 16:00:47 +00:00
Matthias Seitz
642bbea2a8 perf: make BlockState::parent_state_chain return iterator (#20496)
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-22 15:58:46 +00:00
fuder.eth
1c4233d1b4 chore: prevent false-positive log when peer not found in transaction propagation (#20523) 2025-12-22 15:55:41 +00:00
Lorsmirq Benton
eeb2d55f44 docs: add debug execution witness methods to pruning tables (#20561) 2025-12-22 15:53:58 +00:00
fig
96c77fd8b2 feat(storage): make insert_block() operate with references (#20504)
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
2025-12-22 15:13:43 +00:00
VolodymyrBg
ed7a5696b7 fix(engine): sync invalid header cache count gauge on hit eviction (#20567) 2025-12-22 14:59:18 +00:00
Brian Picciano
5a3cffa3e9 fix(stage): Don't clear merkle changesets in unwind near genesis (#20568) 2025-12-22 14:56:18 +00:00
YK
535d97f39e refactor(provider): extract heal_segment for NippyJar consistency (#20508) 2025-12-22 14:01:12 +00:00
DaniPopes
f3aea8dac0 chore: simplify size functions (#20560) 2025-12-22 11:14:50 +00:00
Matthias Seitz
807fac0409 chore: use clone_into_consensus (#20530) 2025-12-22 12:15:09 +01:00
Brian Picciano
7b2fbdcd51 chore(db): Remove Sync from DbTx (#20516) 2025-12-22 10:13:57 +00:00
Merkel Tranjes
3b8acd4b07 feat(payload): add transaction_count to ExecutionPayload trait (#20534) 2025-12-22 10:07:31 +01:00
YK
62abfdaeb5 feat(cli): add tracing-samply to profiling (#20546) 2025-12-21 11:52:26 +00:00
emmmm
256a9fdb79 docs: add missing trace methods to pruning tables (#20547) 2025-12-21 12:40:58 +01:00
github-actions[bot]
4d9aff99bf chore(deps): weekly cargo update (#20545)
Co-authored-by: github-merge-queue <118344674+github-merge-queue@users.noreply.github.com>
2025-12-21 12:40:14 +01:00
Vitalyr
28bb2891bb refactor(consensus): simplify verify_receipts return (#20517) 2025-12-20 19:05:50 +01:00
kurahin
1d8f265744 chore(net): remove stale ECIES rand TODO (#20531) 2025-12-20 19:05:37 +01:00
Matthias Seitz
c754caf8c7 fix: remove stale blobs (#20528) 2025-12-20 15:35:22 +00:00
cui
e1b0046329 chore: remove todo after jovian fork (#20535)
Co-authored-by: weixie.cui <weixie.cui@okg.com>
2025-12-20 15:31:08 +00:00
cui
ddfe177578 chore: remove todo (#20533)
Co-authored-by: weixie.cui <weixie.cui@okg.com>
2025-12-20 15:19:53 +00:00
Gigi
178558c6d7 fix(tree): correct block buffer eviction policy comment (#20512) 2025-12-20 09:44:51 +00:00
Emilia Hane
f4d3a9701f chore(trie): Rm redundant clone of propagated error (#20466)
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
2025-12-20 08:42:20 +00:00
Gigi
42e41a9370 docs: add reth JSON-RPC namespace documentation (#20522) 2025-12-20 08:03:06 +00:00
pepes
a66dcce834 chore(evm): remove deprecated state_change compatibility alias (#20518) 2025-12-20 07:50:12 +00:00
Arsenii Kulikov
21d835cf2b perf: use LRU eviction policy for precompile cache (#20527) 2025-12-20 02:12:42 +00:00
Alexey Shekhirin
29438631be fix: propagate keccak-cache-global feature to reth-node-core (#20524) 2025-12-19 17:11:41 +00:00
Brian Picciano
0eb4e0ce29 fix(stages): Fix two bugs related to stage checkpoints and pipeline syncs (#20521) 2025-12-19 16:09:57 +00:00
gustavo
9147f9aafe perf(trie): remove more unnecessary channels (#20489) 2025-12-19 15:34:42 +00:00
Snezhkko
13b111e058 refactor: remove dead storage multiproof path (#20485) 2025-12-19 15:11:31 +00:00
leniram159
25c247b14c refactor(engine): simplify fork detection in insert_block (#20441) 2025-12-19 14:49:33 +00:00
Matthias Seitz
72bea44d8c chore: remove redundant num hash (#20501) 2025-12-19 14:48:42 +00:00
alex017
63b9d5fe57 refactor(db-api): remove redundant clone and unused import in unwind (#20499)
Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
2025-12-19 14:47:11 +00:00
118 changed files with 1450 additions and 887 deletions

357
Cargo.lock generated
View File

@@ -2,6 +2,15 @@
# It is not intended for manual editing.
version = 4
[[package]]
name = "addr2line"
version = "0.25.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1b5d307320b3181d6d7954e663bd7c774a838b8220fe0593c86d9fb09f498b4b"
dependencies = [
"gimli",
]
[[package]]
name = "adler2"
version = "2.0.1"
@@ -177,9 +186,9 @@ dependencies = [
[[package]]
name = "alloy-dyn-abi"
version = "1.4.1"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fdff496dd4e98a81f4861e66f7eaf5f2488971848bb42d9c892f871730245c8"
checksum = "0d48a9101f4a67c22fae57489f1ddf3057b8ab4a368d8eac3be088b6e9d9c9d9"
dependencies = [
"alloy-json-abi",
"alloy-primitives",
@@ -329,9 +338,9 @@ dependencies = [
[[package]]
name = "alloy-json-abi"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6bfca3dbbcb7498f0f60e67aff2ad6aff57032e22eb2fd03189854be11a22c03"
checksum = "9914c147bb9b25f440eca68a31dc29f5c22298bfa7754aa802965695384122b0"
dependencies = [
"alloy-primitives",
"alloy-sol-type-parser",
@@ -426,9 +435,9 @@ dependencies = [
[[package]]
name = "alloy-primitives"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c850e6ccbd34b8a463a1e934ffc8fc00e1efc5e5489f2ad82d7797949f3bd4e"
checksum = "7db950a29746be9e2f2c6288c8bd7a6202a81f999ce109a2933d2379970ec0fa"
dependencies = [
"alloy-rlp",
"arbitrary",
@@ -436,6 +445,7 @@ dependencies = [
"cfg-if",
"const-hex",
"derive_more",
"fixed-cache",
"foldhash 0.2.0",
"getrandom 0.3.4",
"hashbrown 0.16.1",
@@ -782,9 +792,9 @@ dependencies = [
[[package]]
name = "alloy-sol-macro"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2218e3aeb3ee665d117fdf188db0d5acfdc3f7b7502c827421cb78f26a2aec0"
checksum = "a3b96d5f5890605ba9907ce1e2158e2701587631dc005bfa582cf92dd6f21147"
dependencies = [
"alloy-sol-macro-expander",
"alloy-sol-macro-input",
@@ -796,9 +806,9 @@ dependencies = [
[[package]]
name = "alloy-sol-macro-expander"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b231cb8cc48e66dd1c6e11a1402f3ac86c3667cbc13a6969a0ac030ba7bb8c88"
checksum = "b8247b7cca5cde556e93f8b3882b01dbd272f527836049083d240c57bf7b4c15"
dependencies = [
"alloy-sol-macro-input",
"const-hex",
@@ -814,9 +824,9 @@ dependencies = [
[[package]]
name = "alloy-sol-macro-input"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "49a522d79929c1bf0152b07567a38f7eaed3ab149e53e7528afa78ff11994668"
checksum = "3cd54f38512ac7bae10bbc38480eefb1b9b398ca2ce25db9cc0c048c6411c4f1"
dependencies = [
"const-hex",
"dunce",
@@ -830,9 +840,9 @@ dependencies = [
[[package]]
name = "alloy-sol-type-parser"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0475c459859c8d9428af6ff3736614655a57efda8cc435a3b8b4796fa5ac1dd0"
checksum = "444b09815b44899564566d4d56613d14fa9a274b1043a021f00468568752f449"
dependencies = [
"serde",
"winnow",
@@ -840,9 +850,9 @@ dependencies = [
[[package]]
name = "alloy-sol-types"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "35287d9d821d5f26011bcd8d9101340898f761c9933cf50fca689bb7ed62fdeb"
checksum = "dc1038284171df8bfd48befc0c7b78f667a7e2be162f45f07bd1c378078ebe58"
dependencies = [
"alloy-json-abi",
"alloy-primitives",
@@ -1500,6 +1510,21 @@ dependencies = [
"tokio",
]
[[package]]
name = "backtrace"
version = "0.3.76"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb531853791a215d7c62a30daf0dde835f381ab5de4589cfe7c649d2cbe92bd6"
dependencies = [
"addr2line",
"cfg-if",
"libc",
"miniz_oxide",
"object",
"rustc-demangle",
"windows-link",
]
[[package]]
name = "base-x"
version = "0.2.11"
@@ -1939,9 +1964,9 @@ dependencies = [
[[package]]
name = "bumpalo"
version = "3.19.0"
version = "3.19.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "46c5e41b57b8bba42a04676d81cb89e9ee8e859a1a66f80a5a72e1cb76b34d43"
checksum = "5dd9dc738b7a8311c7ade152424974d8115f2cdad61e8dab8dac9f2362298510"
[[package]]
name = "byte-slice-cast"
@@ -2018,9 +2043,9 @@ dependencies = [
[[package]]
name = "camino"
version = "1.2.1"
version = "1.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "276a59bf2b2c967788139340c9f0c5b12d7fd6630315c15c217e559de85d2609"
checksum = "e629a66d692cb9ff1a1c664e41771b3dcaf961985a9774c0eb0bd1b51cf60a48"
dependencies = [
"serde_core",
]
@@ -2131,7 +2156,7 @@ dependencies = [
"num-traits",
"serde",
"wasm-bindgen",
"windows-link 0.2.1",
"windows-link",
]
[[package]]
@@ -2337,6 +2362,33 @@ dependencies = [
"thiserror 1.0.69",
]
[[package]]
name = "color-eyre"
version = "0.6.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5920befb47832a6d61ee3a3a846565cfa39b331331e68a3b1d1116630f2f26d"
dependencies = [
"backtrace",
"color-spantrace",
"eyre",
"indenter",
"once_cell",
"owo-colors",
"tracing-error",
]
[[package]]
name = "color-spantrace"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8b88ea9df13354b55bc7234ebcce36e6ef896aca2e42a15de9e10edce01b427"
dependencies = [
"once_cell",
"owo-colors",
"tracing-core",
"tracing-error",
]
[[package]]
name = "colorchoice"
version = "1.0.4"
@@ -3986,6 +4038,15 @@ dependencies = [
"windows-sys 0.60.2",
]
[[package]]
name = "fixed-cache"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba59b6c98ba422a13f17ee1305c995cb5742bba7997f5b4d9af61b2ff0ffb213"
dependencies = [
"equivalent",
]
[[package]]
name = "fixed-hash"
version = "0.8.0"
@@ -4229,16 +4290,17 @@ checksum = "42012b0f064e01aa58b545fe3727f90f7dd4020f4a3ea735b50344965f5a57e9"
[[package]]
name = "generator"
version = "0.8.7"
version = "0.8.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "605183a538e3e2a9c1038635cc5c2d194e2ee8fd0d1b66b8349fad7dbacce5a2"
checksum = "52f04ae4152da20c76fe800fa48659201d5cf627c5149ca0b707b69d7eef6cf9"
dependencies = [
"cc",
"cfg-if",
"libc",
"log",
"rustversion",
"windows 0.61.3",
"windows-link",
"windows-result 0.4.1",
]
[[package]]
@@ -4300,6 +4362,12 @@ dependencies = [
"polyval",
]
[[package]]
name = "gimli"
version = "0.32.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7"
[[package]]
name = "git2"
version = "0.20.3"
@@ -5017,13 +5085,14 @@ dependencies = [
[[package]]
name = "insta"
version = "1.44.3"
version = "1.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5c943d4415edd8153251b6f197de5eb1640e56d84e8d9159bea190421c73698"
checksum = "b76866be74d68b1595eb8060cb9191dca9c021db2316558e52ddc5d55d41b66c"
dependencies = [
"console",
"once_cell",
"similar",
"tempfile",
]
[[package]]
@@ -5146,9 +5215,9 @@ dependencies = [
[[package]]
name = "itoa"
version = "1.0.15"
version = "1.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c"
checksum = "7ee5b5339afb4c41626dde77b7a611bd4f2c202b897852b4bcf5d03eddc61010"
[[package]]
name = "jni"
@@ -5464,7 +5533,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
dependencies = [
"cfg-if",
"windows-link 0.2.1",
"windows-link",
]
[[package]]
@@ -5505,13 +5574,13 @@ dependencies = [
[[package]]
name = "libredox"
version = "0.1.10"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "416f7e718bdb06000964960ffa43b4335ad4012ae8b99060261aa4a8088d5ccb"
checksum = "df15f6eac291ed1cf25865b1ee60399f57e7c227e7f51bdbd4c5270396a9ed50"
dependencies = [
"bitflags 2.10.0",
"libc",
"redox_syscall",
"redox_syscall 0.6.0",
]
[[package]]
@@ -6020,9 +6089,9 @@ checksum = "5e0826a989adedc2a244799e823aece04662b66609d96af8dff7ac6df9a8925d"
[[package]]
name = "ntapi"
version = "0.4.1"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e8a3895c6391c39d7fe7ebc444a87eb2991b2a0bc718fdabd071eec617fc68e4"
checksum = "c70f219e21142367c70c0b30c6a9e3a14d55b4d12a204d897fbec83a0363f081"
dependencies = [
"winapi",
]
@@ -6188,6 +6257,15 @@ version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33"
[[package]]
name = "object"
version = "0.37.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe"
dependencies = [
"memchr",
]
[[package]]
name = "once_cell"
version = "1.21.3"
@@ -6469,6 +6547,12 @@ dependencies = [
"num-traits",
]
[[package]]
name = "owo-colors"
version = "4.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c6901729fa79e91a0913333229e9ca5dc725089d1c363b2f4b4760709dc4a52"
[[package]]
name = "p256"
version = "0.13.2"
@@ -6545,9 +6629,9 @@ checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1"
dependencies = [
"cfg-if",
"libc",
"redox_syscall",
"redox_syscall 0.5.18",
"smallvec",
"windows-link 0.2.1",
"windows-link",
]
[[package]]
@@ -6744,9 +6828,9 @@ dependencies = [
[[package]]
name = "portable-atomic"
version = "1.11.1"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f84267b20a16ea918e43c6a88433c2d54fa145c92a811b5b047ccbe153674483"
checksum = "f59e70c4aef1e55797c2e8fd94a4f2a973fc972cfde0e0b05f683667b0cd39dd"
[[package]]
name = "potential_utf"
@@ -6818,7 +6902,7 @@ version = "3.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983"
dependencies = [
"toml_edit 0.23.9",
"toml_edit 0.23.10+spec-1.0.0",
]
[[package]]
@@ -7212,9 +7296,9 @@ dependencies = [
[[package]]
name = "rapidhash"
version = "4.1.1"
version = "4.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d8e65c75143ce5d47c55b510297eeb1182f3c739b6043c537670e9fc18612dae"
checksum = "2988730ee014541157f48ce4dcc603940e00915edc3c7f9a8d78092256bb2493"
dependencies = [
"rand 0.9.2",
"rustversion",
@@ -7285,6 +7369,15 @@ dependencies = [
"bitflags 2.10.0",
]
[[package]]
name = "redox_syscall"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec96166dafa0886eb81fe1c0a388bece180fbef2135f97c1e2cf8302e74b43b5"
dependencies = [
"bitflags 2.10.0",
]
[[package]]
name = "redox_users"
version = "0.4.6"
@@ -7374,9 +7467,9 @@ checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2"
[[package]]
name = "reqwest"
version = "0.12.25"
version = "0.12.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6eff9328d40131d43bd911d42d79eb6a47312002a4daefc9e37f17e74a7701a"
checksum = "3b4c14b2d9afca6a60277086b0cc6a6ae0b568f6f7916c943a8cdc79f8be240f"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -7508,6 +7601,7 @@ dependencies = [
"alloy-transport-ws",
"async-trait",
"clap",
"color-eyre",
"csv",
"eyre",
"futures",
@@ -9613,6 +9707,7 @@ dependencies = [
"alloy-consensus",
"alloy-genesis",
"alloy-network",
"alloy-op-hardforks",
"alloy-primitives",
"alloy-rpc-types-engine",
"alloy-rpc-types-eth",
@@ -10829,6 +10924,7 @@ dependencies = [
"tracing-appender",
"tracing-journald",
"tracing-logfmt",
"tracing-samply",
"tracing-subscriber 0.3.22",
]
@@ -11592,9 +11688,9 @@ dependencies = [
[[package]]
name = "rustls-pki-types"
version = "1.13.1"
version = "1.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "708c0f9d5f54ba0272468c1d306a52c495b31fa155e91bc25371e6df7996908c"
checksum = "21e6f2ab2928ca4291b86736a8bd920a277a399bba1589409d72154ff87c1282"
dependencies = [
"web-time",
"zeroize",
@@ -11658,9 +11754,9 @@ dependencies = [
[[package]]
name = "ryu"
version = "1.0.20"
version = "1.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "28d3b2b1366ec20994f1fd18c3c594f05c5dd4bc44d8bb0c1c632c8d6829481f"
checksum = "62049b2877bf12821e8f9ad256ee38fdc31db7387ec2d3b3f403024de2034aea"
[[package]]
name = "ryu-js"
@@ -12358,9 +12454,9 @@ dependencies = [
[[package]]
name = "syn-solidity"
version = "1.5.0"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60ceeb7c95a4536de0c0e1649bd98d1a72a4bb9590b1f3e45a8a0bfdb7c188c0"
checksum = "f6b1d2e2059056b66fec4a6bb2b79511d5e8d76196ef49c38996f4b48db7662f"
dependencies = [
"paste",
"proc-macro2",
@@ -12818,9 +12914,9 @@ dependencies = [
[[package]]
name = "toml_datetime"
version = "0.7.3"
version = "0.7.5+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2cdb639ebbc97961c51720f858597f7f24c4fc295327923af55b74c3c724533"
checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347"
dependencies = [
"serde_core",
]
@@ -12841,21 +12937,21 @@ dependencies = [
[[package]]
name = "toml_edit"
version = "0.23.9"
version = "0.23.10+spec-1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d7cbc3b4b49633d57a0509303158ca50de80ae32c265093b24c414705807832"
checksum = "84c8b9f757e028cee9fa244aea147aab2a9ec09d5325a9b01e0a49730c2b5269"
dependencies = [
"indexmap 2.12.1",
"toml_datetime 0.7.3",
"toml_datetime 0.7.5+spec-1.1.0",
"toml_parser",
"winnow",
]
[[package]]
name = "toml_parser"
version = "1.0.4"
version = "1.0.6+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0cbe268d35bdb4bb5a56a2de88d0ad0eb70af5384a99d648cd4b3d04039800e"
checksum = "a3198b4b0a8e11f09dd03e133c0280504d0801269e9afa46362ffde1cbeebf44"
dependencies = [
"winnow",
]
@@ -12968,9 +13064,9 @@ checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
[[package]]
name = "tracing"
version = "0.1.43"
version = "0.1.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d15d90a0b5c19378952d479dc858407149d7bb45a14de0142f6c534b16fc647"
checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
dependencies = [
"log",
"pin-project-lite",
@@ -13003,14 +13099,24 @@ dependencies = [
[[package]]
name = "tracing-core"
version = "0.1.35"
version = "0.1.36"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a04e24fab5c89c6a36eb8558c9656f30d81de51dfa4d3b45f26b21d61fa0a6c"
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
dependencies = [
"once_cell",
"valuable",
]
[[package]]
name = "tracing-error"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b1581020d7a273442f5b45074a6a57d5757ad0a47dac0e9f0bd57b81936f3db"
dependencies = [
"tracing",
"tracing-subscriber 0.3.22",
]
[[package]]
name = "tracing-futures"
version = "0.2.5"
@@ -13074,6 +13180,22 @@ dependencies = [
"web-time",
]
[[package]]
name = "tracing-samply"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c175f7ecc002b6ef04776a39f440503e4e788790ddbdbfac8259b7a069526334"
dependencies = [
"cfg-if",
"itoa",
"libc",
"mach2",
"memmap2",
"smallvec",
"tracing-core",
"tracing-subscriber 0.3.22",
]
[[package]]
name = "tracing-serde"
version = "0.2.0"
@@ -13685,38 +13807,16 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows"
version = "0.61.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893"
dependencies = [
"windows-collections 0.2.0",
"windows-core 0.61.2",
"windows-future 0.2.1",
"windows-link 0.1.3",
"windows-numerics 0.2.0",
]
[[package]]
name = "windows"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580"
dependencies = [
"windows-collections 0.3.2",
"windows-collections",
"windows-core 0.62.2",
"windows-future 0.3.2",
"windows-numerics 0.3.1",
]
[[package]]
name = "windows-collections"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3beeceb5e5cfd9eb1d76b381630e82c4241ccd0d27f1a39ed41b2760b255c5e8"
dependencies = [
"windows-core 0.61.2",
"windows-future",
"windows-numerics",
]
[[package]]
@@ -13740,19 +13840,6 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-core"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0fdd3ddb90610c7638aa2b3a3ab2904fb9e5cdbecc643ddb3647212781c4ae3"
dependencies = [
"windows-implement 0.60.2",
"windows-interface 0.59.3",
"windows-link 0.1.3",
"windows-result 0.3.4",
"windows-strings 0.4.2",
]
[[package]]
name = "windows-core"
version = "0.62.2"
@@ -13761,20 +13848,9 @@ checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [
"windows-implement 0.60.2",
"windows-interface 0.59.3",
"windows-link 0.2.1",
"windows-link",
"windows-result 0.4.1",
"windows-strings 0.5.1",
]
[[package]]
name = "windows-future"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e"
dependencies = [
"windows-core 0.61.2",
"windows-link 0.1.3",
"windows-threading 0.1.0",
"windows-strings",
]
[[package]]
@@ -13784,8 +13860,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb"
dependencies = [
"windows-core 0.62.2",
"windows-link 0.2.1",
"windows-threading 0.2.1",
"windows-link",
"windows-threading",
]
[[package]]
@@ -13832,28 +13908,12 @@ dependencies = [
"syn 2.0.111",
]
[[package]]
name = "windows-link"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e6ad25900d524eaabdbbb96d20b4311e1e7ae1699af4fb28c17ae66c80d798a"
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-numerics"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9150af68066c4c5c07ddc0ce30421554771e528bde427614c61038bc2c92c2b1"
dependencies = [
"windows-core 0.61.2",
"windows-link 0.1.3",
]
[[package]]
name = "windows-numerics"
version = "0.3.1"
@@ -13861,7 +13921,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26"
dependencies = [
"windows-core 0.62.2",
"windows-link 0.2.1",
"windows-link",
]
[[package]]
@@ -13873,31 +13933,13 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-result"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56f42bd332cc6c8eac5af113fc0c1fd6a8fd2aa08a0119358686e5160d0586c6"
dependencies = [
"windows-link 0.1.3",
]
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link 0.2.1",
]
[[package]]
name = "windows-strings"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56e6c93f3a0c3b36176cb1327a4958a0353d5d166c2a35cb268ace15e91d3b57"
dependencies = [
"windows-link 0.1.3",
"windows-link",
]
[[package]]
@@ -13906,7 +13948,7 @@ version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [
"windows-link 0.2.1",
"windows-link",
]
[[package]]
@@ -13960,7 +14002,7 @@ version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link 0.2.1",
"windows-link",
]
[[package]]
@@ -14015,7 +14057,7 @@ version = "0.53.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"
dependencies = [
"windows-link 0.2.1",
"windows-link",
"windows_aarch64_gnullvm 0.53.1",
"windows_aarch64_msvc 0.53.1",
"windows_i686_gnu 0.53.1",
@@ -14026,22 +14068,13 @@ dependencies = [
"windows_x86_64_msvc 0.53.1",
]
[[package]]
name = "windows-threading"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b66463ad2e0ea3bbf808b7f1d371311c80e115c0b71d60efc142cafbcfb057a6"
dependencies = [
"windows-link 0.1.3",
]
[[package]]
name = "windows-threading"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37"
dependencies = [
"windows-link 0.2.1",
"windows-link",
]
[[package]]

View File

@@ -548,6 +548,7 @@ bytes = { version = "1.5", default-features = false }
brotli = "8"
cfg-if = "1.0"
clap = "4"
color-eyre = "0.6"
dashmap = "6.0"
derive_more = { version = "2", default-features = false, features = ["full"] }
dirs-next = "2.0.0"
@@ -730,6 +731,7 @@ socket2 = { version = "0.5", default-features = false }
sysinfo = { version = "0.33", default-features = false }
tracing-journald = "0.3"
tracing-logfmt = "0.3.3"
tracing-samply = "0.1"
tracing-subscriber = { version = "0.3", default-features = false }
triehash = "0.8"
typenum = "1.15.0"

View File

@@ -329,6 +329,7 @@ pub(crate) async fn run_comparison(args: Args, _ctx: CliContext) -> Result<()> {
output_dir.clone(),
git_manager.clone(),
args.features.clone(),
args.profile,
)?;
// Initialize node manager
let mut node_manager = NodeManager::new(&args);

View File

@@ -14,6 +14,7 @@ pub(crate) struct CompilationManager {
output_dir: PathBuf,
git_manager: GitManager,
features: String,
enable_profiling: bool,
}
impl CompilationManager {
@@ -23,8 +24,9 @@ impl CompilationManager {
output_dir: PathBuf,
git_manager: GitManager,
features: String,
enable_profiling: bool,
) -> Result<Self> {
Ok(Self { repo_root, output_dir, git_manager, features })
Ok(Self { repo_root, output_dir, git_manager, features, enable_profiling })
}
/// Detect if the RPC endpoint is an Optimism chain
@@ -100,9 +102,18 @@ impl CompilationManager {
let mut cmd = Command::new("cargo");
cmd.arg("build").arg("--profile").arg("profiling");
// Add features
cmd.arg("--features").arg(&self.features);
info!("Using features: {}", self.features);
// Append samply feature when profiling to enable tracing span markers.
// NOTE: The `samply` feature must exist in the branch being compiled. If comparing
// against an older branch that predates the samply integration, compilation will fail
// or markers won't appear. In that case, omit --profile or ensure both branches
// include the samply feature support.
let features = if self.enable_profiling && !self.features.contains("samply") {
format!("{},samply", self.features)
} else {
self.features.clone()
};
cmd.arg("--features").arg(&features);
info!("Using features: {}", features);
// Add bin-specific arguments for optimism
if is_optimism {

View File

@@ -58,6 +58,7 @@ tokio = { workspace = true, features = ["sync", "macros", "time", "rt-multi-thre
# misc
clap = { workspace = true, features = ["derive", "env"] }
eyre.workspace = true
color-eyre.workspace = true
thiserror.workspace = true
humantime.workspace = true

View File

@@ -23,7 +23,7 @@ use bench::BenchmarkCommand;
use clap::Parser;
use reth_cli_runner::CliRunner;
fn main() {
fn main() -> eyre::Result<()> {
// Enable backtraces unless a RUST_BACKTRACE value has already been explicitly provided.
if std::env::var_os("RUST_BACKTRACE").is_none() {
unsafe {
@@ -31,12 +31,11 @@ fn main() {
}
}
color_eyre::install()?;
// Run until either exit or sigint or sigterm
let runner = CliRunner::try_default_runtime().unwrap();
runner
.run_command_until_exit(|ctx| {
let command = BenchmarkCommand::parse();
command.execute(ctx)
})
.unwrap();
let runner = CliRunner::try_default_runtime()?;
runner.run_command_until_exit(|ctx| BenchmarkCommand::parse().execute(ctx))?;
Ok(())
}

View File

@@ -87,6 +87,10 @@ otlp = [
"reth-ethereum-cli/otlp",
"reth-node-core/otlp",
]
samply = [
"reth-ethereum-cli/samply",
"reth-node-core/samply",
]
js-tracer = [
"reth-node-builder/js-tracer",
"reth-node-ethereum/js-tracer",
@@ -103,6 +107,7 @@ asm-keccak = [
"reth-node-ethereum/asm-keccak",
]
keccak-cache-global = [
"reth-node-core/keccak-cache-global",
"reth-node-ethereum/keccak-cache-global",
]
jemalloc = [

View File

@@ -664,22 +664,14 @@ impl<N: NodePrimitives> BlockState<N> {
receipts.first().map(|receipts| receipts.deref()).unwrap_or_default()
}
/// Returns a vector of __parent__ `BlockStates`.
/// Returns an iterator over __parent__ `BlockStates`.
///
/// The block state order in the output vector is newest to oldest (highest to lowest):
/// The block state order is newest to oldest (highest to lowest):
/// `[5,4,3,2,1]`
///
/// Note: This does not include self.
pub fn parent_state_chain(&self) -> Vec<&Self> {
let mut parents = Vec::new();
let mut current = self.parent.as_deref();
while let Some(parent) = current {
parents.push(parent);
current = parent.parent.as_deref();
}
parents
pub fn parent_state_chain(&self) -> impl Iterator<Item = &Self> + '_ {
std::iter::successors(self.parent.as_deref(), |state| state.parent.as_deref())
}
/// Returns a vector of `BlockStates` representing the entire in memory chain.
@@ -690,6 +682,11 @@ impl<N: NodePrimitives> BlockState<N> {
}
/// Appends the parent chain of this [`BlockState`] to the given vector.
///
/// Parents are appended in order from newest to oldest (highest to lowest).
/// This does not include self, only the parent states.
///
/// This is a convenience method equivalent to `chain.extend(self.parent_state_chain())`.
pub fn append_parent_chain<'a>(&'a self, chain: &mut Vec<&'a Self>) {
chain.extend(self.parent_state_chain());
}
@@ -1453,18 +1450,18 @@ mod tests {
let mut test_block_builder: TestBlockBuilder = TestBlockBuilder::default();
let chain = create_mock_state_chain(&mut test_block_builder, 4);
let parents = chain[3].parent_state_chain();
let parents: Vec<_> = chain[3].parent_state_chain().collect();
assert_eq!(parents.len(), 3);
assert_eq!(parents[0].block().recovered_block().number, 3);
assert_eq!(parents[1].block().recovered_block().number, 2);
assert_eq!(parents[2].block().recovered_block().number, 1);
let parents = chain[2].parent_state_chain();
let parents: Vec<_> = chain[2].parent_state_chain().collect();
assert_eq!(parents.len(), 2);
assert_eq!(parents[0].block().recovered_block().number, 2);
assert_eq!(parents[1].block().recovered_block().number, 1);
let parents = chain[0].parent_state_chain();
let parents: Vec<_> = chain[0].parent_state_chain().collect();
assert_eq!(parents.len(), 0);
}
@@ -1476,7 +1473,7 @@ mod tests {
create_mock_state(&mut test_block_builder, single_block_number, B256::random());
let single_block_hash = single_block.block().recovered_block().hash();
let parents = single_block.parent_state_chain();
let parents: Vec<_> = single_block.parent_state_chain().collect();
assert_eq!(parents.len(), 0);
let block_state_chain = single_block.chain().collect::<Vec<_>>();

View File

@@ -5,14 +5,14 @@ use reth_errors::ProviderResult;
use reth_primitives_traits::{Account, Bytecode, NodePrimitives};
use reth_storage_api::{
AccountReader, BlockHashReader, BytecodeReader, HashedPostStateProvider, StateProofProvider,
StateProvider, StateRootProvider, StorageRootProvider,
StateProvider, StateProviderBox, StateRootProvider, StorageRootProvider,
};
use reth_trie::{
updates::TrieUpdates, AccountProof, HashedPostState, HashedStorage, MultiProof,
MultiProofTargets, StorageMultiProof, TrieInput,
};
use revm_database::BundleState;
use std::sync::OnceLock;
use std::{borrow::Cow, sync::OnceLock};
/// A state provider that stores references to in-memory blocks along with their state as well as a
/// reference of the historical state provider for fallback lookups.
@@ -24,15 +24,11 @@ pub struct MemoryOverlayStateProviderRef<
/// Historical state provider for state lookups that are not found in memory blocks.
pub(crate) historical: Box<dyn StateProvider + 'a>,
/// The collection of executed parent blocks. Expected order is newest to oldest.
pub(crate) in_memory: Vec<ExecutedBlock<N>>,
pub(crate) in_memory: Cow<'a, [ExecutedBlock<N>]>,
/// Lazy-loaded in-memory trie data.
pub(crate) trie_input: OnceLock<TrieInput>,
}
/// A state provider that stores references to in-memory blocks along with their state as well as
/// the historical state provider for fallback lookups.
pub type MemoryOverlayStateProvider<N> = MemoryOverlayStateProviderRef<'static, N>;
impl<'a, N: NodePrimitives> MemoryOverlayStateProviderRef<'a, N> {
/// Create new memory overlay state provider.
///
@@ -42,7 +38,7 @@ impl<'a, N: NodePrimitives> MemoryOverlayStateProviderRef<'a, N> {
/// - `historical` - a historical state provider for the latest ancestor block stored in the
/// database.
pub fn new(historical: Box<dyn StateProvider + 'a>, in_memory: Vec<ExecutedBlock<N>>) -> Self {
Self { historical, in_memory, trie_input: OnceLock::new() }
Self { historical, in_memory: Cow::Owned(in_memory), trie_input: OnceLock::new() }
}
/// Turn this state provider into a state provider
@@ -71,7 +67,7 @@ impl<'a, N: NodePrimitives> MemoryOverlayStateProviderRef<'a, N> {
impl<N: NodePrimitives> BlockHashReader for MemoryOverlayStateProviderRef<'_, N> {
fn block_hash(&self, number: BlockNumber) -> ProviderResult<Option<B256>> {
for block in &self.in_memory {
for block in self.in_memory.iter() {
if block.recovered_block().number() == number {
return Ok(Some(block.recovered_block().hash()));
}
@@ -90,7 +86,7 @@ impl<N: NodePrimitives> BlockHashReader for MemoryOverlayStateProviderRef<'_, N>
let mut in_memory_hashes = Vec::with_capacity(range.size_hint().0);
// iterate in ascending order (oldest to newest = low to high)
for block in &self.in_memory {
for block in self.in_memory.iter() {
let block_num = block.recovered_block().number();
if range.contains(&block_num) {
in_memory_hashes.push(block.recovered_block().hash());
@@ -112,7 +108,7 @@ impl<N: NodePrimitives> BlockHashReader for MemoryOverlayStateProviderRef<'_, N>
impl<N: NodePrimitives> AccountReader for MemoryOverlayStateProviderRef<'_, N> {
fn basic_account(&self, address: &Address) -> ProviderResult<Option<Account>> {
for block in &self.in_memory {
for block in self.in_memory.iter() {
if let Some(account) = block.execution_output.account(address) {
return Ok(account);
}
@@ -216,7 +212,7 @@ impl<N: NodePrimitives> StateProvider for MemoryOverlayStateProviderRef<'_, N> {
address: Address,
storage_key: StorageKey,
) -> ProviderResult<Option<StorageValue>> {
for block in &self.in_memory {
for block in self.in_memory.iter() {
if let Some(value) = block.execution_output.storage(&address, storage_key.into()) {
return Ok(Some(value));
}
@@ -228,7 +224,7 @@ impl<N: NodePrimitives> StateProvider for MemoryOverlayStateProviderRef<'_, N> {
impl<N: NodePrimitives> BytecodeReader for MemoryOverlayStateProviderRef<'_, N> {
fn bytecode_by_hash(&self, code_hash: &B256) -> ProviderResult<Option<Bytecode>> {
for block in &self.in_memory {
for block in self.in_memory.iter() {
if let Some(contract) = block.execution_output.bytecode(code_hash) {
return Ok(Some(contract));
}
@@ -237,3 +233,46 @@ impl<N: NodePrimitives> BytecodeReader for MemoryOverlayStateProviderRef<'_, N>
self.historical.bytecode_by_hash(code_hash)
}
}
/// An owned state provider that stores references to in-memory blocks along with their state as
/// well as a reference of the historical state provider for fallback lookups.
#[expect(missing_debug_implementations)]
pub struct MemoryOverlayStateProvider<N: NodePrimitives = reth_ethereum_primitives::EthPrimitives> {
/// Historical state provider for state lookups that are not found in memory blocks.
pub(crate) historical: StateProviderBox,
/// The collection of executed parent blocks. Expected order is newest to oldest.
pub(crate) in_memory: Vec<ExecutedBlock<N>>,
/// Lazy-loaded in-memory trie data.
pub(crate) trie_input: OnceLock<TrieInput>,
}
impl<N: NodePrimitives> MemoryOverlayStateProvider<N> {
/// Create new memory overlay state provider.
///
/// ## Arguments
///
/// - `in_memory` - the collection of executed ancestor blocks in reverse.
/// - `historical` - a historical state provider for the latest ancestor block stored in the
/// database.
pub fn new(historical: StateProviderBox, in_memory: Vec<ExecutedBlock<N>>) -> Self {
Self { historical, in_memory, trie_input: OnceLock::new() }
}
/// Returns a new provider that takes the `TX` as reference
#[inline(always)]
fn as_ref(&self) -> MemoryOverlayStateProviderRef<'_, N> {
MemoryOverlayStateProviderRef {
historical: Box::new(self.historical.as_ref()),
in_memory: Cow::Borrowed(&self.in_memory),
trie_input: self.trie_input.clone(),
}
}
/// Wraps the [`Self`] in a `Box`.
pub fn boxed(self) -> StateProviderBox {
Box::new(self)
}
}
// Delegates all provider impls to [`MemoryOverlayStateProviderRef`]
reth_storage_api::macros::delegate_provider_impls!(MemoryOverlayStateProvider<N> where [N: NodePrimitives]);

View File

@@ -970,7 +970,7 @@ impl<H: BlockHeader> EthereumHardforks for ChainSpec<H> {
/// A trait for reading the current chainspec.
#[auto_impl::auto_impl(&, Arc)]
pub trait ChainSpecProvider: Debug + Send + Sync {
pub trait ChainSpecProvider: Debug + Send {
/// The chain spec type.
type ChainSpec: EthChainSpec + 'static;

View File

@@ -79,7 +79,7 @@ where
+ StaticFileProviderFactory<Primitives: NodePrimitives<BlockHeader: Compact>>,
{
provider_rw.insert_block(
SealedBlock::<<Provider::Primitives as NodePrimitives>::Block>::from_sealed_parts(
&SealedBlock::<<Provider::Primitives as NodePrimitives>::Block>::from_sealed_parts(
header.clone(),
Default::default(),
)

View File

@@ -22,7 +22,6 @@ pub const DEFAULT_BLOCK_INTERVAL: usize = 5;
#[cfg_attr(feature = "serde", serde(default))]
pub struct Config {
/// Configuration for each stage in the pipeline.
// TODO(onbjerg): Can we make this easier to maintain when we add/remove stages?
pub stages: StageConfig,
/// Configuration for pruning.
#[cfg_attr(feature = "serde", serde(default))]

View File

@@ -2,9 +2,7 @@
use crate::testsuite::{Action, Environment};
use alloy_primitives::B256;
use alloy_rpc_types_engine::{
ExecutionPayloadV1, ExecutionPayloadV2, ExecutionPayloadV3, PayloadStatusEnum,
};
use alloy_rpc_types_engine::{ExecutionPayloadV3, PayloadStatusEnum};
use alloy_rpc_types_eth::{Block, Header, Receipt, Transaction, TransactionRequest};
use eyre::Result;
use futures_util::future::BoxFuture;
@@ -131,7 +129,10 @@ where
})?;
// Convert block to ExecutionPayloadV3
let payload = block_to_payload_v3(block.clone());
let payload = ExecutionPayloadV3::from_block_unchecked(
block.hash(),
&block.map_transactions(|tx| tx.inner).into_consensus(),
);
// Send the payload to the target node
let target_engine = env.node_clients[self.node_idx].engine.http_client();
@@ -327,32 +328,3 @@ where
})
}
}
/// Helper function to convert a block to `ExecutionPayloadV3`
fn block_to_payload_v3(block: Block) -> ExecutionPayloadV3 {
use alloy_primitives::U256;
ExecutionPayloadV3 {
payload_inner: ExecutionPayloadV2 {
payload_inner: ExecutionPayloadV1 {
parent_hash: block.header.inner.parent_hash,
fee_recipient: block.header.inner.beneficiary,
state_root: block.header.inner.state_root,
receipts_root: block.header.inner.receipts_root,
logs_bloom: block.header.inner.logs_bloom,
prev_randao: block.header.inner.mix_hash,
block_number: block.header.inner.number,
gas_limit: block.header.inner.gas_limit,
gas_used: block.header.inner.gas_used,
timestamp: block.header.inner.timestamp,
extra_data: block.header.inner.extra_data.clone(),
base_fee_per_gas: U256::from(block.header.inner.base_fee_per_gas.unwrap_or(0)),
block_hash: block.header.hash,
transactions: vec![], // No transactions needed for buffering tests
},
withdrawals: block.withdrawals.unwrap_or_default().to_vec(),
},
blob_gas_used: block.header.inner.blob_gas_used.unwrap_or(0),
excess_blob_gas: block.header.inner.excess_blob_gas.unwrap_or(0),
}
}

View File

@@ -5,7 +5,7 @@ use pretty_assertions::Comparison;
use reth_engine_primitives::InvalidBlockHook;
use reth_evm::{execute::Executor, ConfigureEvm};
use reth_primitives_traits::{NodePrimitives, RecoveredBlock, SealedHeader};
use reth_provider::{BlockExecutionOutput, StateProvider, StateProviderFactory};
use reth_provider::{BlockExecutionOutput, StateProvider, StateProviderBox, StateProviderFactory};
use reth_revm::{
database::StateProviderDatabase,
db::{BundleState, State},
@@ -114,7 +114,7 @@ fn sort_bundle_state_for_comparison(bundle_state: &BundleState) -> BundleStateSo
/// Extracts execution data including codes, preimages, and hashed state from database
fn collect_execution_data(
mut db: State<StateProviderDatabase<Box<dyn StateProvider>>>,
mut db: State<StateProviderDatabase<StateProviderBox>>,
) -> eyre::Result<CollectionResult> {
let bundle_state = db.take_bundle();
let mut codes = BTreeMap::new();
@@ -530,9 +530,7 @@ mod tests {
// Create a State with StateProviderTest
let state_provider = StateProviderTest::default();
let mut state = State::builder()
.with_database(StateProviderDatabase::new(
Box::new(state_provider) as Box<dyn StateProvider>
))
.with_database(StateProviderDatabase::new(Box::new(state_provider) as StateProviderBox))
.with_bundle_update()
.build();

View File

@@ -47,7 +47,7 @@ impl BackfillSyncState {
}
/// Backfill sync mode functionality.
pub trait BackfillSync: Send + Sync {
pub trait BackfillSync: Send {
/// Performs a backfill action.
fn on_action(&mut self, action: BackfillAction);

View File

@@ -1,5 +1,4 @@
use crate::metrics::PersistenceMetrics;
use alloy_consensus::BlockHeader;
use alloy_eips::BlockNumHash;
use reth_chain_state::ExecutedBlock;
use reth_errors::ProviderError;
@@ -142,27 +141,23 @@ where
&self,
blocks: Vec<ExecutedBlock<N::Primitives>>,
) -> Result<Option<BlockNumHash>, PersistenceError> {
let first_block_hash = blocks.first().map(|b| b.recovered_block.num_hash());
let last_block_hash = blocks.last().map(|b| b.recovered_block.num_hash());
debug!(target: "engine::persistence", first=?first_block_hash, last=?last_block_hash, "Saving range of blocks");
let first_block = blocks.first().map(|b| b.recovered_block.num_hash());
let last_block = blocks.last().map(|b| b.recovered_block.num_hash());
debug!(target: "engine::persistence", first=?first_block, last=?last_block, "Saving range of blocks");
let start_time = Instant::now();
let last_block_hash_num = blocks.last().map(|block| BlockNumHash {
hash: block.recovered_block().hash(),
number: block.recovered_block().header().number(),
});
if last_block_hash_num.is_some() {
if last_block.is_some() {
let provider_rw = self.provider.database_provider_rw()?;
provider_rw.save_blocks(blocks)?;
provider_rw.commit()?;
}
debug!(target: "engine::persistence", first=?first_block_hash, last=?last_block_hash, "Saved range of blocks");
debug!(target: "engine::persistence", first=?first_block, last=?last_block, "Saved range of blocks");
self.metrics.save_blocks_duration_seconds.record(start_time.elapsed());
Ok(last_block_hash_num)
Ok(last_block)
}
}

View File

@@ -14,7 +14,7 @@ use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
/// * [`BlockBuffer::remove_old_blocks`] to remove old blocks that precede the finalized number.
///
/// Note: Buffer is limited by number of blocks that it can contain and eviction of the block
/// is done by last recently used block.
/// is done in FIFO order (oldest inserted block is evicted first).
#[derive(Debug)]
pub struct BlockBuffer<B: Block> {
/// All blocks in the buffer stored by their block hash.

View File

@@ -48,6 +48,7 @@ impl InvalidHeaderCache {
// if we get here, the entry has been hit too many times, so we evict it
self.headers.remove(hash);
self.metrics.hit_evictions.increment(1);
self.metrics.count.set(self.headers.len() as f64);
None
}

View File

@@ -931,48 +931,6 @@ where
Ok(())
}
/// Determines if the given block is part of a fork by checking that these
/// conditions are true:
/// * walking back from the target hash to verify that the target hash is not part of an
/// extension of the canonical chain.
/// * walking back from the current head to verify that the target hash is not already part of
/// the canonical chain.
///
/// The header is required as an arg, because we might be checking that the header is a fork
/// block before it's in the tree state and before it's in the database.
fn is_fork(&self, target: BlockWithParent) -> ProviderResult<bool> {
let target_hash = target.block.hash;
// verify that the given hash is not part of an extension of the canon chain.
let canonical_head = self.state.tree_state.canonical_head();
let mut current_hash;
let mut current_block = target;
loop {
if current_block.block.hash == canonical_head.hash {
return Ok(false)
}
// We already passed the canonical head
if current_block.block.number <= canonical_head.number {
break
}
current_hash = current_block.parent;
let Some(next_block) = self.sealed_header_by_hash(current_hash)? else { break };
current_block = next_block.block_with_parent();
}
// verify that the given hash is not already part of canonical chain stored in memory
if self.canonical_in_memory_state.header_by_hash(target_hash).is_some() {
return Ok(false)
}
// verify that the given hash is not already part of persisted canonical chain
if self.provider.block_number(target_hash)?.is_some() {
return Ok(false)
}
Ok(true)
}
/// Invoked when we receive a new forkchoice update message. Calls into the blockchain tree
/// to resolve chain forks and ensure that the Execution Layer is working with the latest valid
/// chain.
@@ -2569,14 +2527,11 @@ where
Ok(Some(_)) => {}
}
// determine whether we are on a fork chain
let is_fork = match self.is_fork(block_id) {
Err(err) => {
let block = convert_to_block(self, input)?;
return Err(InsertBlockError::new(block, err.into()).into());
}
Ok(is_fork) => is_fork,
};
// determine whether we are on a fork chain by comparing the block number with the
// canonical head. This is a simple check that is sufficient for the event emission below.
// A block is considered a fork if its number is less than or equal to the canonical head,
// as this indicates there's already a canonical block at that height.
let is_fork = block_id.block.number <= self.state.tree_state.current_canonical_head.number;
let ctx = TreeCtx::new(&mut self.state, &self.canonical_in_memory_state);

View File

@@ -51,7 +51,7 @@ use std::{
},
time::Instant,
};
use tracing::{debug, debug_span, error, instrument, warn, Span};
use tracing::{debug, debug_span, instrument, warn, Span};
pub mod bal;
mod configured_sparse_trie;
@@ -236,31 +236,7 @@ where
let span = Span::current();
let (to_sparse_trie, sparse_trie_rx) = channel();
// We rely on the cursor factory to provide whatever DB overlay is necessary to see a
// consistent view of the database, including the trie tables. Because of this there is no
// need for an overarching prefix set to invalidate any section of the trie tables, and so
// we use an empty prefix set.
// Create and spawn the storage proof task
let task_ctx = ProofTaskCtx::new(multiproof_provider_factory);
let storage_worker_count = config.storage_worker_count();
let account_worker_count = config.account_worker_count();
let proof_handle = ProofWorkerHandle::new(
self.executor.handle().clone(),
task_ctx,
storage_worker_count,
account_worker_count,
);
let multi_proof_task = MultiProofTask::new(
proof_handle.clone(),
to_sparse_trie,
config.multiproof_chunking_enabled().then_some(config.multiproof_chunk_size()),
);
// wire the multiproof task to the prewarm task
let to_multi_proof = Some(multi_proof_task.state_root_message_sender());
let (to_multi_proof, from_multi_proof) = crossbeam_channel::unbounded();
// Handle BAL-based optimization if available
let prewarm_handle = if let Some(bal) = bal {
@@ -268,12 +244,7 @@ where
debug!(target: "engine::tree::payload_processor", "BAL present, skipping prewarm tasks");
// Send BAL message immediately to MultiProofTask
if let Some(ref sender) = to_multi_proof &&
let Err(err) = sender.send(MultiProofMessage::BlockAccessList(bal))
{
// In this case state root validation will simply fail
error!(target: "engine::tree::payload_processor", ?err, "Failed to send BAL to MultiProofTask");
}
let _ = to_multi_proof.send(MultiProofMessage::BlockAccessList(bal));
// Spawn minimal cache-only task without prewarming
self.spawn_caching_with(
@@ -290,10 +261,32 @@ where
prewarm_rx,
transaction_count_hint,
provider_builder.clone(),
to_multi_proof.clone(),
Some(to_multi_proof.clone()),
)
};
// Create and spawn the storage proof task
let task_ctx = ProofTaskCtx::new(multiproof_provider_factory);
let storage_worker_count = config.storage_worker_count();
let account_worker_count = config.account_worker_count();
let proof_handle = ProofWorkerHandle::new(
self.executor.handle().clone(),
task_ctx,
storage_worker_count,
account_worker_count,
);
let multi_proof_task = MultiProofTask::new(
proof_handle.clone(),
to_sparse_trie,
config.multiproof_chunking_enabled().then_some(config.multiproof_chunk_size()),
to_multi_proof,
from_multi_proof,
);
// wire the multiproof task to the prewarm task
let to_multi_proof = Some(multi_proof_task.state_root_message_sender());
// spawn multi-proof task
let parent_span = span.clone();
self.executor.spawn_blocking(move || {

View File

@@ -3,11 +3,7 @@
use crate::tree::payload_processor::bal::bal_to_hashed_post_state;
use alloy_eip7928::BlockAccessList;
use alloy_evm::block::StateChangeSource;
use alloy_primitives::{
keccak256,
map::{B256Set, HashSet},
B256,
};
use alloy_primitives::{keccak256, map::HashSet, B256};
use crossbeam_channel::{unbounded, Receiver as CrossbeamReceiver, Sender as CrossbeamSender};
use dashmap::DashMap;
use derive_more::derive::Deref;
@@ -23,7 +19,6 @@ use reth_trie_parallel::{
proof::ParallelProof,
proof_task::{
AccountMultiproofInput, ProofResultContext, ProofResultMessage, ProofWorkerHandle,
StorageProofInput,
},
};
use std::{collections::BTreeMap, mem, ops::DerefMut, sync::Arc, time::Instant};
@@ -236,74 +231,6 @@ pub(crate) fn evm_state_to_hashed_post_state(update: EvmState) -> HashedPostStat
hashed_state
}
/// A pending multiproof task, either [`StorageMultiproofInput`] or [`MultiproofInput`].
#[derive(Debug)]
enum PendingMultiproofTask {
/// A storage multiproof task input.
Storage(StorageMultiproofInput),
/// A regular multiproof task input.
Regular(MultiproofInput),
}
impl PendingMultiproofTask {
/// Returns the proof sequence number of the task.
const fn proof_sequence_number(&self) -> u64 {
match self {
Self::Storage(input) => input.proof_sequence_number,
Self::Regular(input) => input.proof_sequence_number,
}
}
/// Returns whether or not the proof targets are empty.
fn proof_targets_is_empty(&self) -> bool {
match self {
Self::Storage(input) => input.proof_targets.is_empty(),
Self::Regular(input) => input.proof_targets.is_empty(),
}
}
/// Destroys the input and sends a [`MultiProofMessage::EmptyProof`] message to the sender.
fn send_empty_proof(self) {
match self {
Self::Storage(input) => input.send_empty_proof(),
Self::Regular(input) => input.send_empty_proof(),
}
}
}
impl From<StorageMultiproofInput> for PendingMultiproofTask {
fn from(input: StorageMultiproofInput) -> Self {
Self::Storage(input)
}
}
impl From<MultiproofInput> for PendingMultiproofTask {
fn from(input: MultiproofInput) -> Self {
Self::Regular(input)
}
}
/// Input parameters for dispatching a dedicated storage multiproof calculation.
#[derive(Debug)]
struct StorageMultiproofInput {
hashed_state_update: HashedPostState,
hashed_address: B256,
proof_targets: B256Set,
proof_sequence_number: u64,
state_root_message_sender: CrossbeamSender<MultiProofMessage>,
multi_added_removed_keys: Arc<MultiAddedRemovedKeys>,
}
impl StorageMultiproofInput {
/// Destroys the input and sends a [`MultiProofMessage::EmptyProof`] message to the sender.
fn send_empty_proof(self) {
let _ = self.state_root_message_sender.send(MultiProofMessage::EmptyProof {
sequence_number: self.proof_sequence_number,
state: self.hashed_state_update,
});
}
}
/// Input parameters for dispatching a multiproof calculation.
#[derive(Debug)]
struct MultiproofInput {
@@ -378,91 +305,18 @@ impl MultiproofManager {
}
/// Dispatches a new multiproof calculation to worker pools.
fn dispatch(&self, input: PendingMultiproofTask) {
fn dispatch(&self, input: MultiproofInput) {
// If there are no proof targets, we can just send an empty multiproof back immediately
if input.proof_targets_is_empty() {
if input.proof_targets.is_empty() {
trace!(
sequence_number = input.proof_sequence_number(),
sequence_number = input.proof_sequence_number,
"No proof targets, sending empty multiproof back immediately"
);
input.send_empty_proof();
return;
}
match input {
PendingMultiproofTask::Storage(storage_input) => {
self.dispatch_storage_proof(storage_input);
}
PendingMultiproofTask::Regular(multiproof_input) => {
self.dispatch_multiproof(multiproof_input);
}
}
}
/// Dispatches a single storage proof calculation to worker pool.
fn dispatch_storage_proof(&self, storage_multiproof_input: StorageMultiproofInput) {
let StorageMultiproofInput {
hashed_state_update,
hashed_address,
proof_targets,
proof_sequence_number,
multi_added_removed_keys,
state_root_message_sender: _,
} = storage_multiproof_input;
let storage_targets = proof_targets.len();
trace!(
target: "engine::tree::payload_processor::multiproof",
proof_sequence_number,
?proof_targets,
storage_targets,
"Dispatching storage proof to workers"
);
let start = Instant::now();
// Create prefix set from targets
let prefix_set = reth_trie::prefix_set::PrefixSetMut::from(
proof_targets.iter().map(reth_trie::Nibbles::unpack),
);
let prefix_set = prefix_set.freeze();
// Build computation input (data only)
let input = StorageProofInput::new(
hashed_address,
prefix_set,
proof_targets,
true, // with_branch_node_masks
Some(multi_added_removed_keys),
);
// Dispatch to storage worker
if let Err(e) = self.proof_worker_handle.dispatch_storage_proof(
input,
ProofResultContext::new(
self.proof_result_tx.clone(),
proof_sequence_number,
hashed_state_update,
start,
),
) {
error!(target: "engine::tree::payload_processor::multiproof", ?e, "Failed to dispatch storage proof");
return;
}
self.metrics
.active_storage_workers_histogram
.record(self.proof_worker_handle.active_storage_workers() as f64);
self.metrics
.active_account_workers_histogram
.record(self.proof_worker_handle.active_account_workers() as f64);
self.metrics
.pending_storage_multiproofs_histogram
.record(self.proof_worker_handle.pending_storage_tasks() as f64);
self.metrics
.pending_account_multiproofs_histogram
.record(self.proof_worker_handle.pending_account_tasks() as f64);
self.dispatch_multiproof(input);
}
/// Signals that a multiproof calculation has finished.
@@ -740,8 +594,9 @@ impl MultiProofTask {
proof_worker_handle: ProofWorkerHandle,
to_sparse_trie: std::sync::mpsc::Sender<SparseTrieUpdate>,
chunk_size: Option<usize>,
tx: CrossbeamSender<MultiProofMessage>,
rx: CrossbeamReceiver<MultiProofMessage>,
) -> Self {
let (tx, rx) = unbounded();
let (proof_result_tx, proof_result_rx) = unbounded();
let metrics = MultiProofTaskMetrics::default();
@@ -809,17 +664,14 @@ impl MultiProofTask {
available_storage_workers,
MultiProofTargets::chunks,
|proof_targets| {
self.multiproof_manager.dispatch(
MultiproofInput {
source: None,
hashed_state_update: Default::default(),
proof_targets,
proof_sequence_number: self.proof_sequencer.next_sequence(),
state_root_message_sender: self.tx.clone(),
multi_added_removed_keys: Some(multi_added_removed_keys.clone()),
}
.into(),
);
self.multiproof_manager.dispatch(MultiproofInput {
source: None,
hashed_state_update: Default::default(),
proof_targets,
proof_sequence_number: self.proof_sequencer.next_sequence(),
state_root_message_sender: self.tx.clone(),
multi_added_removed_keys: Some(multi_added_removed_keys.clone()),
});
},
);
self.metrics.prefetch_proof_chunks_histogram.record(num_chunks as f64);
@@ -967,17 +819,14 @@ impl MultiProofTask {
);
spawned_proof_targets.extend_ref(&proof_targets);
self.multiproof_manager.dispatch(
MultiproofInput {
source: Some(source),
hashed_state_update,
proof_targets,
proof_sequence_number: self.proof_sequencer.next_sequence(),
state_root_message_sender: self.tx.clone(),
multi_added_removed_keys: Some(multi_added_removed_keys.clone()),
}
.into(),
);
self.multiproof_manager.dispatch(MultiproofInput {
source: Some(source),
hashed_state_update,
proof_targets,
proof_sequence_number: self.proof_sequencer.next_sequence(),
state_root_message_sender: self.tx.clone(),
multi_added_removed_keys: Some(multi_added_removed_keys.clone()),
});
},
);
self.metrics
@@ -1683,8 +1532,9 @@ mod tests {
let task_ctx = ProofTaskCtx::new(overlay_factory);
let proof_handle = ProofWorkerHandle::new(rt_handle, task_ctx, 1, 1);
let (to_sparse_trie, _receiver) = std::sync::mpsc::channel();
let (tx, rx) = crossbeam_channel::unbounded();
MultiProofTask::new(proof_handle, to_sparse_trie, Some(1))
MultiProofTask::new(proof_handle, to_sparse_trie, Some(1), tx, rx)
}
#[test]

View File

@@ -164,12 +164,7 @@ where
};
// Initialize worker handles container
let mut handles = Vec::with_capacity(workers_needed);
// Only spawn initial workers as needed
for i in 0..workers_needed {
handles.push(ctx.spawn_worker(i, &executor, actions_tx.clone(), done_tx.clone()));
}
let handles = ctx.clone().spawn_workers(workers_needed, &executor, actions_tx.clone(), done_tx.clone());
// Distribute transactions to workers
let mut tx_index = 0usize;
@@ -536,27 +531,43 @@ where
}
/// Spawns a worker task for transaction execution and returns its sender channel.
fn spawn_worker<Tx>(
&self,
idx: usize,
executor: &WorkloadExecutor,
fn spawn_workers<Tx>(
self,
workers_needed: usize,
task_executor: &WorkloadExecutor,
actions_tx: Sender<PrewarmTaskEvent<N::Receipt>>,
done_tx: Sender<()>,
) -> mpsc::Sender<IndexedTransaction<Tx>>
) -> Vec<mpsc::Sender<IndexedTransaction<Tx>>>
where
Tx: ExecutableTxFor<Evm> + Send + 'static,
{
let (tx, rx) = mpsc::channel();
let ctx = self.clone();
let span =
debug_span!(target: "engine::tree::payload_processor::prewarm", "prewarm worker", idx);
let mut handles = Vec::with_capacity(workers_needed);
let mut receivers = Vec::with_capacity(workers_needed);
executor.spawn_blocking(move || {
for _ in 0..workers_needed {
let (tx, rx) = mpsc::channel();
handles.push(tx);
receivers.push(rx);
}
// Spawn a separate task spawning workers in parallel.
let executor = task_executor.clone();
let span = Span::current();
task_executor.spawn_blocking(move || {
let _enter = span.entered();
ctx.transact_batch(rx, actions_tx, done_tx);
for (idx, rx) in receivers.into_iter().enumerate() {
let ctx = self.clone();
let actions_tx = actions_tx.clone();
let done_tx = done_tx.clone();
let span = debug_span!(target: "engine::tree::payload_processor::prewarm", "prewarm worker", idx);
executor.spawn_blocking(move || {
let _enter = span.entered();
ctx.transact_batch(rx, actions_tx, done_tx);
});
}
});
tx
handles
}
}

View File

@@ -166,8 +166,7 @@ where
// Update storage slots with new values and calculate storage roots.
let span = tracing::Span::current();
let (tx, rx) = mpsc::channel();
state
let results: Vec<_> = state
.storages
.into_iter()
.map(|(address, storage)| (address, storage, trie.take_storage_trie(&address)))
@@ -217,13 +216,7 @@ where
SparseStateTrieResult::Ok((address, storage_trie))
})
.for_each_init(
|| tx.clone(),
|tx, result| {
let _ = tx.send(result);
},
);
drop(tx);
.collect();
// Defer leaf removals until after updates/additions, so that we don't delete an intermediate
// branch node during a removal and then re-add that branch back during a later leaf addition.
@@ -235,7 +228,7 @@ where
let _enter =
tracing::debug_span!(target: "engine::tree::payload_processor::sparse_trie", "account trie")
.entered();
for result in rx {
for result in results {
let (address, storage_trie) = result?;
trie.insert_storage_trie(address, storage_trie);

View File

@@ -435,8 +435,7 @@ where
}
// Execute the block and handle any execution errors
let (output, senders) = match self.execute_block(&state_provider, env, &input, &mut handle)
{
let (output, senders) = match self.execute_block(state_provider, env, &input, &mut handle) {
Ok(output) => output,
Err(err) => return self.handle_execution_error(input, err, &parent_block),
};
@@ -603,7 +602,7 @@ where
handle: &mut PayloadHandle<impl ExecutableTxFor<Evm>, Err, N::Receipt>,
) -> Result<(BlockExecutionOutput<N::Receipt>, Vec<Address>), InsertBlockErrorKind>
where
S: StateProvider,
S: StateProvider + Send,
Err: core::error::Error + Send + Sync + 'static,
V: PayloadValidator<T, Block = N::Block>,
T: PayloadTypes<BuiltPayload: BuiltPayload<Primitives = N>>,

View File

@@ -2,6 +2,7 @@
use alloy_primitives::Bytes;
use dashmap::DashMap;
use moka::policy::EvictionPolicy;
use reth_evm::precompiles::{DynPrecompile, Precompile, PrecompileInput};
use revm::precompile::{PrecompileId, PrecompileOutput, PrecompileResult};
use revm_primitives::Address;
@@ -49,6 +50,7 @@ where
Self(
moka::sync::CacheBuilder::new(MAX_CACHE_SIZE as u64)
.initial_capacity(MAX_CACHE_SIZE as usize)
.eviction_policy(EvictionPolicy::lru())
.build_with_hasher(Default::default()),
)
}

View File

@@ -116,7 +116,7 @@ where
/// these stages that this work has already been done. Otherwise, there might be some conflict with
/// database integrity.
pub fn save_stage_checkpoints<P>(
provider: &P,
provider: P,
from: BlockNumber,
to: BlockNumber,
processed: u64,
@@ -309,7 +309,7 @@ where
writer.append_header(&header, &hash)?;
// Write bodies to database.
provider.append_block_bodies(vec![(header.number(), Some(body))])?;
provider.append_block_bodies(vec![(header.number(), Some(&body))])?;
hash_collector.insert(hash, number)?;
}

View File

@@ -38,6 +38,7 @@ tempfile.workspace = true
default = []
otlp = ["reth-tracing/otlp", "reth-node-core/otlp"]
samply = ["reth-tracing/samply", "reth-node-core/samply"]
dev = ["reth-cli-commands/arbitrary"]

View File

@@ -76,10 +76,7 @@ impl<C: ChainSpecParser, Ext: clap::Args + fmt::Debug, Rpc: RpcModuleValidator>
///
/// This method is used to prepare the CLI for execution by wrapping it in a
/// [`CliApp`] that can be further configured before running.
pub fn configure(self) -> CliApp<C, Ext, Rpc>
where
C: ChainSpecParser<ChainSpec = ChainSpec>,
{
pub fn configure(self) -> CliApp<C, Ext, Rpc> {
CliApp::new(self)
}

View File

@@ -87,9 +87,7 @@ fn verify_receipts<R: Receipt>(
logs_bloom,
expected_receipts_root,
expected_logs_bloom,
)?;
Ok(())
)
}
/// Compare the calculated receipts root with the expected receipts root, also compare

View File

@@ -91,6 +91,7 @@ asm-keccak = [
]
keccak-cache-global = [
"alloy-primitives/keccak-cache-global",
"reth-node-core/keccak-cache-global",
]
js-tracer = [
"reth-node-builder/js-tracer",

View File

@@ -153,9 +153,9 @@ where
let PayloadConfig { parent_header, attributes } = config;
let state_provider = client.state_by_block_hash(parent_header.hash())?;
let state = StateProviderDatabase::new(&state_provider);
let state = StateProviderDatabase::new(state_provider.as_ref());
let mut db =
State::builder().with_database(cached_reads.as_db_mut(state)).with_bundle_update().build();
State::builder().with_database_ref(cached_reads.as_db(state)).with_bundle_update().build();
let mut builder = evm_config
.builder_for_next_block(
@@ -358,7 +358,8 @@ where
return Ok(BuildOutcome::Aborted { fees: total_fees, cached_reads })
}
let BlockBuilderOutcome { execution_result, block, .. } = builder.finish(&state_provider)?;
let BlockBuilderOutcome { execution_result, block, .. } =
builder.finish(state_provider.as_ref())?;
let requests = chain_spec
.is_prague_active_at_timestamp(attributes.timestamp)

View File

@@ -258,10 +258,7 @@ impl<T: TxTy> IsTyped2718 for Receipt<T> {
impl<T: TxTy> InMemorySize for Receipt<T> {
fn size(&self) -> usize {
self.tx_type.size() +
core::mem::size_of::<bool>() +
core::mem::size_of::<u64>() +
self.logs.iter().map(|log| log.size()).sum::<usize>()
size_of::<Self>() + self.logs.iter().map(|log| log.size()).sum::<usize>()
}
}

View File

@@ -81,6 +81,7 @@ arbitrary = [
]
keccak-cache-global = [
"reth-node-ethereum?/keccak-cache-global",
"reth-node-core?/keccak-cache-global",
]
test-utils = [
"reth-chainspec/test-utils",

View File

@@ -61,8 +61,6 @@ pub use alloy_evm::{
*,
};
pub use alloy_evm::block::state_changes as state_change;
/// A complete configuration of EVM for Reth.
///
/// This trait encapsulates complete configuration required for transaction execution and block

View File

@@ -13,7 +13,7 @@ use reth_evm_ethereum::EthEvmConfig;
use reth_node_api::NodePrimitives;
use reth_primitives_traits::{Block as _, RecoveredBlock};
use reth_provider::{
providers::ProviderNodeTypes, BlockWriter as _, ExecutionOutcome, LatestStateProviderRef,
providers::ProviderNodeTypes, BlockWriter as _, ExecutionOutcome, LatestStateProvider,
ProviderFactory,
};
use reth_revm::database::StateProviderDatabase;
@@ -69,7 +69,7 @@ where
// Execute the block to produce a block execution output
let mut block_execution_output = EthEvmConfig::ethereum(chain_spec)
.batch_executor(StateProviderDatabase::new(LatestStateProviderRef::new(&provider)))
.batch_executor(StateProviderDatabase::new(LatestStateProvider::new(provider)))
.execute(block)?;
block_execution_output.state.reverts.sort();
@@ -203,8 +203,8 @@ where
let provider = provider_factory.provider()?;
let evm_config = EthEvmConfig::new(chain_spec);
let executor = evm_config
.batch_executor(StateProviderDatabase::new(LatestStateProviderRef::new(&provider)));
let executor =
evm_config.batch_executor(StateProviderDatabase::new(LatestStateProvider::new(provider)));
let mut execution_outcome = executor.execute_batch(vec![&block1, &block2])?;
execution_outcome.state_mut().reverts.sort();

View File

@@ -1303,7 +1303,7 @@ mod tests {
.try_recover()
.unwrap();
let provider_rw = provider_factory.database_provider_rw().unwrap();
provider_rw.insert_block(block.clone()).unwrap();
provider_rw.insert_block(&block).unwrap();
provider_rw.commit().unwrap();
let provider = BlockchainProvider::new(provider_factory).unwrap();

View File

@@ -481,12 +481,12 @@ mod tests {
&mut rng,
genesis_block.number + 1,
BlockParams { parent: Some(genesis_hash), tx_count: Some(0), ..Default::default() },
);
let provider_rw = provider_factory.provider_rw()?;
provider_rw.insert_block(node_head_block.clone().try_recover()?)?;
provider_rw.commit()?;
)
.try_recover()?;
let node_head = node_head_block.num_hash();
let provider_rw = provider_factory.provider_rw()?;
provider_rw.insert_block(&node_head_block)?;
provider_rw.commit()?;
let exex_head =
ExExHead { block: BlockNumHash { number: genesis_block.number, hash: genesis_hash } };
@@ -613,7 +613,7 @@ mod tests {
.try_recover()?;
let node_head = node_head_block.num_hash();
let provider_rw = provider.database_provider_rw()?;
provider_rw.insert_block(node_head_block)?;
provider_rw.insert_block(&node_head_block)?;
provider_rw.commit()?;
let node_head_notification = ExExNotification::ChainCommitted {
new: Arc::new(

View File

@@ -212,13 +212,13 @@ impl Discv4ConfigBuilder {
self
}
/// Whether to enforce expiration timestamps in messages.
/// Whether to enable EIP-868
pub const fn enable_eip868(&mut self, enable_eip868: bool) -> &mut Self {
self.config.enable_eip868 = enable_eip868;
self
}
/// Whether to enable EIP-868
/// Whether to enforce expiration timestamps in messages.
pub const fn enforce_expiration_timestamps(
&mut self,
enforce_expiration_timestamps: bool,

View File

@@ -312,7 +312,6 @@ impl ECIES {
/// Create a new ECIES client with the given static secret key and remote peer ID.
pub fn new_client(secret_key: SecretKey, remote_id: PeerId) -> Result<Self, ECIESError> {
// TODO(rand): use rng for nonce
let mut rng = rng();
let nonce = B256::random();
let ephemeral_secret_key = SecretKey::new(&mut rng);

View File

@@ -151,13 +151,16 @@ impl Discovery {
self.discovery_listeners.retain_mut(|listener| listener.send(event.clone()).is_ok());
}
/// Updates the `eth:ForkId` field in discv4.
/// Updates the `eth:ForkId` field in discv4/discv5.
pub(crate) fn update_fork_id(&self, fork_id: ForkId) {
if let Some(discv4) = &self.discv4 {
// use forward-compatible forkid entry
discv4.set_eip868_rlp(b"eth".to_vec(), EnrForkIdEntry::from(fork_id))
}
// todo: update discv5 enr
if let Some(discv5) = &self.discv5 {
discv5
.encode_and_set_eip868_in_local_enr(b"eth".to_vec(), EnrForkIdEntry::from(fork_id))
}
}
/// Bans the [`IpAddr`] in the discovery service.

View File

@@ -860,9 +860,8 @@ where
peer_id: PeerId,
propagation_mode: PropagationMode,
) -> Option<PropagatedTransactions> {
trace!(target: "net::tx", ?peer_id, "Propagating transactions to peer");
let peer = self.peers.get_mut(&peer_id)?;
trace!(target: "net::tx", ?peer_id, "Propagating transactions to peer");
let mut propagated = PropagatedTransactions::default();
// filter all transactions unknown to the peer

View File

@@ -13,9 +13,7 @@ pub type BodyDownloaderResult<B> = DownloadResult<Vec<BlockResponse<B>>>;
/// A downloader represents a distinct strategy for submitting requests to download block bodies,
/// while a [`BodiesClient`][crate::bodies::client::BodiesClient] represents a client capable of
/// fulfilling these requests.
pub trait BodyDownloader:
Send + Sync + Stream<Item = BodyDownloaderResult<Self::Block>> + Unpin
{
pub trait BodyDownloader: Send + Stream<Item = BodyDownloaderResult<Self::Block>> + Unpin {
/// The Block type this downloader supports
type Block: Block + 'static;

View File

@@ -37,6 +37,14 @@ where
Self::Empty(_) => None,
}
}
/// Return the reference to the response body
pub const fn body(&self) -> Option<&B::Body> {
match self {
Self::Full(block) => Some(block.body()),
Self::Empty(_) => None,
}
}
}
impl<B: Block> InMemorySize for BlockResponse<B> {

View File

@@ -66,8 +66,9 @@ use reth_node_metrics::{
};
use reth_provider::{
providers::{NodeTypesForProvider, ProviderNodeTypes, RocksDBProvider, StaticFileProvider},
BlockHashReader, BlockNumReader, ProviderError, ProviderFactory, ProviderResult,
StageCheckpointReader, StaticFileProviderBuilder, StaticFileProviderFactory,
BlockHashReader, BlockNumReader, DatabaseProviderFactory, ProviderError, ProviderFactory,
ProviderResult, RocksDBProviderFactory, StageCheckpointReader, StaticFileProviderBuilder,
StaticFileProviderFactory,
};
use reth_prune::{PruneModes, PrunerBuilder};
use reth_rpc_builder::config::RethRpcServerConfig;
@@ -501,20 +502,61 @@ where
)?
.with_prune_modes(self.prune_modes());
// Check for consistency between database and static files. If it fails, it unwinds to
// the first block that's consistent between database and static files.
if let Some(unwind_target) =
factory.static_file_provider().check_consistency(&factory.provider()?)?
{
// Check for consistency between database, static files, and RocksDB. If any
// inconsistencies are found, unwind to the first block that's consistent across all
// storage layers.
//
// The ordering is critical:
// 1. File healing - heals NippyJar inconsistencies without pruning datae
// 2. RocksDB check - needs static file tx data for hash lookups
// 3. Static file checkpoint check - compares with MDBX, may prune data
//
// We compute a combined unwind target from all checks and run a single unwind pass.
// Step 1: Heal file-level inconsistencies (no pruning)
let file_unwind = factory.static_file_provider().check_file_consistency()?;
// Step 2: RocksDB consistency check (needs static files tx data)
let rocksdb_unwind =
factory.rocksdb_provider().check_consistency(&factory.database_provider_ro()?)?;
// Step 3: Static file checkpoint consistency (may prune)
let static_file_unwind = factory
.static_file_provider()
.check_consistency(&factory.provider()?)?
.map(|target| match target {
PipelineTarget::Unwind(block) => block,
PipelineTarget::Sync(_) => unreachable!("check_consistency returns Unwind"),
});
// Combine all unwind targets - take the minimum (most conservative)
let unwind_target =
[file_unwind, rocksdb_unwind, static_file_unwind].into_iter().flatten().min();
if let Some(unwind_block) = unwind_target {
// Highly unlikely to happen, and given its destructive nature, it's better to panic
// instead.
// instead. Unwinding to 0 would leave MDBX with a huge free list size.
let inconsistency_source = match (file_unwind, rocksdb_unwind, static_file_unwind) {
(Some(_), Some(_), Some(_)) => {
"static file healing, RocksDB <> database, and static file <> database"
}
(Some(_), Some(_), None) => "static file healing and RocksDB <> database",
(Some(_), None, Some(_)) => "static file healing and static file <> database",
(None, Some(_), Some(_)) => "RocksDB <> database and static file <> database",
(Some(_), None, None) => "static file healing",
(None, Some(_), None) => "RocksDB <> database",
(None, None, Some(_)) => "static file <> database",
(None, None, None) => unreachable!(),
};
assert_ne!(
unwind_target,
PipelineTarget::Unwind(0),
"A static file <> database inconsistency was found that would trigger an unwind to block 0"
unwind_block,
0,
"A {inconsistency_source} inconsistency was found that would trigger an unwind to block 0"
);
info!(target: "reth::cli", unwind_target = %unwind_target, "Executing an unwind after a failed storage consistency check.");
let unwind_target = PipelineTarget::Unwind(unwind_block);
info!(target: "reth::cli", %unwind_target, %inconsistency_source, "Executing unwind after consistency check.");
let (_tip_tx, tip_rx) = watch::channel(B256::ZERO);
@@ -548,7 +590,7 @@ where
}),
);
rx.await?.inspect_err(|err| {
error!(target: "reth::cli", unwind_target = %unwind_target, %err, "failed to run unwind")
error!(target: "reth::cli", unwind_target=%unwind_target, %err, "failed to run unwind")
})?;
}
@@ -938,9 +980,13 @@ where
///
/// A target block hash if the pipeline is inconsistent, otherwise `None`.
pub fn check_pipeline_consistency(&self) -> ProviderResult<Option<B256>> {
// We skip the era stage if it's not enabled
let era_enabled = self.era_import_source().is_some();
let mut all_stages =
StageId::ALL.into_iter().filter(|id| era_enabled || id != &StageId::Era);
// Get the expected first stage based on config.
let first_stage =
if self.era_import_source().is_some() { StageId::Era } else { StageId::Headers };
let first_stage = all_stages.next().expect("there must be at least one stage");
// If no target was provided, check if the stages are congruent - check if the
// checkpoint of the last stage matches the checkpoint of the first.
@@ -950,20 +996,28 @@ where
.unwrap_or_default()
.block_number;
// Skip the first stage as we've already retrieved it and comparing all other checkpoints
// against it.
for stage_id in StageId::ALL.iter().skip(1) {
// Compare all other stages against the first
for stage_id in all_stages {
let stage_checkpoint = self
.blockchain_db()
.get_stage_checkpoint(*stage_id)?
.get_stage_checkpoint(stage_id)?
.unwrap_or_default()
.block_number;
// If the checkpoint of any stage is less than the checkpoint of the first stage,
// retrieve and return the block hash of the latest header and use it as the target.
debug!(
target: "consensus::engine",
first_stage_id = %first_stage,
first_stage_checkpoint,
stage_id = %stage_id,
stage_checkpoint = stage_checkpoint,
"Checking stage against first stage",
);
if stage_checkpoint < first_stage_checkpoint {
debug!(
target: "consensus::engine",
first_stage_id = %first_stage,
first_stage_checkpoint,
inconsistent_stage_id = %stage_id,
inconsistent_stage_checkpoint = stage_checkpoint,

View File

@@ -9,7 +9,7 @@ use crate::{
NodeBuilderWithComponents, NodeComponents, NodeComponentsBuilder, NodeHandle, NodeTypesAdapter,
};
use alloy_consensus::BlockHeader;
use futures::{stream_select, StreamExt};
use futures::{stream_select, FutureExt, StreamExt};
use reth_chainspec::{EthChainSpec, EthereumHardforks};
use reth_engine_service::service::{ChainEvent, EngineService};
use reth_engine_tree::{
@@ -270,7 +270,7 @@ impl EngineNodeLauncher {
} = add_ons.launch_add_ons(add_ons_ctx).await?;
// Create engine shutdown handle
let (engine_shutdown, mut shutdown_rx) = EngineShutdown::new();
let (engine_shutdown, shutdown_rx) = EngineShutdown::new();
// Run consensus engine to completion
let initial_target = ctx.initial_backfill_target()?;
@@ -300,6 +300,7 @@ impl EngineNodeLauncher {
}
let mut res = Ok(());
let mut shutdown_rx = shutdown_rx.fuse();
// advance the chain and await payloads built locally to add into the engine api tree handler to prevent re-execution if that block is received as payload from the CL
loop {

View File

@@ -80,8 +80,9 @@ tokio.workspace = true
# Features for vergen to generate correct env vars
jemalloc = ["reth-cli-util/jemalloc"]
asm-keccak = ["alloy-primitives/asm-keccak"]
# Feature to enable opentelemetry export
keccak-cache-global = ["alloy-primitives/keccak-cache-global"]
otlp = ["reth-tracing/otlp"]
samply = ["reth-tracing/samply"]
min-error-logs = ["tracing/release_max_level_error"]
min-warn-logs = ["tracing/release_max_level_warn"]

View File

@@ -30,6 +30,7 @@ workspace = true
default = ["jemalloc", "otlp", "reth-optimism-evm/portable", "js-tracer", "keccak-cache-global", "asm-keccak"]
otlp = ["reth-optimism-cli/otlp"]
samply = ["reth-optimism-cli/samply"]
js-tracer = [
"reth-optimism-node/js-tracer",

View File

@@ -78,6 +78,7 @@ default = []
# Opentelemtry feature to activate metrics export
otlp = ["reth-tracing/otlp", "reth-node-core/otlp"]
samply = ["reth-tracing/samply", "reth-node-core/samply"]
asm-keccak = [
"alloy-primitives/asm-keccak",

View File

@@ -83,6 +83,7 @@ reth-rpc-eth-types.workspace = true
reth-stages-types.workspace = true
alloy-network.workspace = true
alloy-op-hardforks.workspace = true
futures.workspace = true
op-alloy-network.workspace = true
@@ -96,6 +97,7 @@ asm-keccak = [
]
keccak-cache-global = [
"alloy-primitives/keccak-cache-global",
"reth-node-core/keccak-cache-global",
"reth-optimism-node/keccak-cache-global",
]
js-tracer = [

View File

@@ -299,23 +299,16 @@ mod test {
use super::*;
use crate::engine;
use alloy_op_hardforks::BASE_SEPOLIA_JOVIAN_TIMESTAMP;
use alloy_primitives::{b64, Address, B256, B64};
use alloy_rpc_types_engine::PayloadAttributes;
use reth_chainspec::{ChainSpec, ForkCondition, Hardfork};
use reth_chainspec::ChainSpec;
use reth_optimism_chainspec::{OpChainSpec, BASE_SEPOLIA};
use reth_optimism_forks::OpHardfork;
use reth_provider::noop::NoopProvider;
use reth_trie_common::KeccakKeyHasher;
const JOVIAN_TIMESTAMP: u64 = 1744909000;
fn get_chainspec() -> Arc<OpChainSpec> {
let mut base_sepolia_spec = BASE_SEPOLIA.inner.clone();
// TODO: Remove this once we know the Jovian timestamp
base_sepolia_spec
.hardforks
.insert(OpHardfork::Jovian.boxed(), ForkCondition::Timestamp(JOVIAN_TIMESTAMP));
let base_sepolia_spec = BASE_SEPOLIA.inner.clone();
Arc::new(OpChainSpec {
inner: ChainSpec {
@@ -427,7 +420,8 @@ mod test {
fn test_well_formed_attributes_jovian_valid() {
let validator =
OpEngineValidator::new::<KeccakKeyHasher>(get_chainspec(), NoopProvider::default());
let attributes = get_attributes(Some(b64!("0000000000000000")), Some(1), JOVIAN_TIMESTAMP);
let attributes =
get_attributes(Some(b64!("0000000000000000")), Some(1), BASE_SEPOLIA_JOVIAN_TIMESTAMP);
let result = <engine::OpEngineValidator<_, _, _> as EngineApiValidator<
OpEngineTypes,
@@ -442,7 +436,7 @@ mod test {
fn test_malformed_attributes_jovian_with_eip_1559_params_none() {
let validator =
OpEngineValidator::new::<KeccakKeyHasher>(get_chainspec(), NoopProvider::default());
let attributes = get_attributes(None, Some(1), JOVIAN_TIMESTAMP);
let attributes = get_attributes(None, Some(1), BASE_SEPOLIA_JOVIAN_TIMESTAMP);
let result = <engine::OpEngineValidator<_, _, _> as EngineApiValidator<
OpEngineTypes,
@@ -472,7 +466,8 @@ mod test {
fn test_malformed_attributes_post_jovian_with_min_base_fee_none() {
let validator =
OpEngineValidator::new::<KeccakKeyHasher>(get_chainspec(), NoopProvider::default());
let attributes = get_attributes(Some(b64!("0000000000000000")), None, JOVIAN_TIMESTAMP);
let attributes =
get_attributes(Some(b64!("0000000000000000")), None, BASE_SEPOLIA_JOVIAN_TIMESTAMP);
let result = <engine::OpEngineValidator<_, _, _> as EngineApiValidator<
OpEngineTypes,

View File

@@ -76,6 +76,7 @@ arbitrary = [
]
keccak-cache-global = [
"reth-optimism-node?/keccak-cache-global",
"reth-node-core?/keccak-cache-global",
]
test-utils = [
"reth-chainspec/test-utils",

View File

@@ -88,7 +88,8 @@ impl<Client, Tx> OpTransactionValidator<Client, Tx> {
impl<Client, Tx> OpTransactionValidator<Client, Tx>
where
Client: ChainSpecProvider<ChainSpec: OpHardforks> + StateProviderFactory + BlockReaderIdExt,
Client:
ChainSpecProvider<ChainSpec: OpHardforks> + StateProviderFactory + BlockReaderIdExt + Sync,
Tx: EthPoolTransaction + OpPooledTx,
{
/// Create a new [`OpTransactionValidator`].
@@ -177,7 +178,7 @@ where
&self,
origin: TransactionOrigin,
transaction: Tx,
state: &mut Option<Box<dyn AccountInfoReader>>,
state: &mut Option<Box<dyn AccountInfoReader + Send>>,
) -> TransactionValidationOutcome<Tx> {
if transaction.is_eip4844() {
return TransactionValidationOutcome::Invalid(
@@ -289,7 +290,8 @@ where
impl<Client, Tx> TransactionValidator for OpTransactionValidator<Client, Tx>
where
Client: ChainSpecProvider<ChainSpec: OpHardforks> + StateProviderFactory + BlockReaderIdExt,
Client:
ChainSpecProvider<ChainSpec: OpHardforks> + StateProviderFactory + BlockReaderIdExt + Sync,
Tx: EthPoolTransaction + OpPooledTx,
{
type Transaction = Tx;

View File

@@ -55,6 +55,9 @@ pub trait ExecutionPayload:
/// Returns the total gas consumed by all transactions in this block.
fn gas_used(&self) -> u64;
/// Returns the number of transactions in the payload.
fn transaction_count(&self) -> usize;
}
impl ExecutionPayload for ExecutionData {
@@ -89,6 +92,10 @@ impl ExecutionPayload for ExecutionData {
fn gas_used(&self) -> u64 {
self.payload.as_v1().gas_used
}
fn transaction_count(&self) -> usize {
self.payload.as_v1().transactions.len()
}
}
/// A unified type for handling both execution payloads and payload attributes.
@@ -196,6 +203,10 @@ impl ExecutionPayload for op_alloy_rpc_types_engine::OpExecutionData {
fn gas_used(&self) -> u64 {
self.payload.as_v1().gas_used
}
fn transaction_count(&self) -> usize {
self.payload.as_v1().transactions.len()
}
}
/// Extended functionality for Ethereum execution payloads

View File

@@ -472,7 +472,7 @@ impl<B: Block + Default> Default for RecoveredBlock<B> {
impl<B: Block> InMemorySize for RecoveredBlock<B> {
#[inline]
fn size(&self) -> usize {
self.block.size() + self.senders.len() * core::mem::size_of::<Address>()
self.block.size() + self.senders.capacity() * core::mem::size_of::<Address>()
}
}

View File

@@ -235,7 +235,7 @@ mod tests {
for block in &blocks {
provider_rw
.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)
.expect("failed to insert block");
}
@@ -273,7 +273,7 @@ mod tests {
for block in &blocks {
provider_rw
.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)
.expect("failed to insert block");
}
@@ -319,7 +319,7 @@ mod tests {
for block in &blocks {
provider_rw
.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)
.expect("failed to insert block");
}
@@ -355,7 +355,7 @@ mod tests {
for block in &blocks {
provider_rw
.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)
.expect("failed to insert block");
}

View File

@@ -9,7 +9,7 @@ use revm::{bytecode::Bytecode, state::AccountInfo, Database, DatabaseRef};
/// A helper trait responsible for providing state necessary for EVM execution.
///
/// This serves as the data layer for [`Database`].
pub trait EvmStateProvider: Send + Sync {
pub trait EvmStateProvider {
/// Get basic account information.
///
/// Returns [`None`] if the account doesn't exist.

View File

@@ -33,7 +33,7 @@ use reth_rpc_eth_types::{
simulate::{self, EthSimulateError},
EthApiError, StateCacheDb,
};
use reth_storage_api::{BlockIdReader, ProviderTx, StateProvider};
use reth_storage_api::{BlockIdReader, ProviderTx, StateProviderBox};
use revm::{
context::Block,
context_interface::{result::ResultAndState, Transaction},
@@ -491,11 +491,11 @@ pub trait Call:
) -> impl Future<Output = Result<R, Self::Error>> + Send
where
R: Send + 'static,
F: FnOnce(Self, &dyn StateProvider) -> Result<R, Self::Error> + Send + 'static,
F: FnOnce(Self, StateProviderBox) -> Result<R, Self::Error> + Send + 'static,
{
self.spawn_blocking_io_fut(move |this| async move {
let state = this.state_at_block_id(at).await?;
f(this, &state)
f(this, state)
})
}

View File

@@ -235,7 +235,7 @@ pub trait LoadPendingBlock:
.provider()
.history_by_block_hash(parent.hash())
.map_err(Self::Error::from_eth_err)?;
let state = StateProviderDatabase::new(&state_provider);
let state = StateProviderDatabase::new(state_provider);
let mut db = State::builder().with_database(state).with_bundle_update().build();
let mut builder = self

View File

@@ -238,7 +238,7 @@ pub struct UnwindOutput {
///
/// Stages receive [`DBProvider`](reth_provider::DBProvider).
#[auto_impl::auto_impl(Box)]
pub trait Stage<Provider>: Send + Sync {
pub trait Stage<Provider>: Send {
/// Get the ID of the stage.
///
/// Stage IDs must be unique.

View File

@@ -202,10 +202,7 @@ where
// Write bodies to database.
provider.append_block_bodies(
buffer
.into_iter()
.map(|response| (response.block_number(), response.into_body()))
.collect(),
buffer.iter().map(|response| (response.block_number(), response.body())).collect(),
)?;
// The stage is "done" if:

View File

@@ -195,7 +195,7 @@ where
}
era::save_stage_checkpoints(
&provider,
provider,
input.checkpoint().block_number,
height,
height,

View File

@@ -748,8 +748,8 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f90262f901f9a075c371ba45999d87f4542326910a11af515897aebce5265d3f6acd1f1161f82fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa098f2dcd87c8ae4083e7017a05456c14eea4b1db2032126e27b3b1563d57d7cc0a08151d548273f6683169524b66ca9fe338b9ce42bc3540046c828fd939ae23bcba03f4e5c2ec5b2170b711d97ee755c160457bb58d8daa338e835ec02ae6860bbabb901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083020000018502540be40082a8798203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f863f861800a8405f5e10094100000000000000000000000000000000000000080801ba07e09e26678ed4fac08a249ebe8ed680bf9051a5e14ad223e4b2b9d26e0208f37a05f6e3f188e3e6eab7d7d3b6568f5eac7d687b08d307d3154ccd8c87b4630509bc0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider.insert_block(genesis.try_recover().unwrap()).unwrap();
provider.insert_block(block.clone().try_recover().unwrap()).unwrap();
provider.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider.insert_block(&block.clone().try_recover().unwrap()).unwrap();
provider
.static_file_provider()
.latest_writer(StaticFileSegment::Headers)
@@ -789,8 +789,8 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f90262f901f9a075c371ba45999d87f4542326910a11af515897aebce5265d3f6acd1f1161f82fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa098f2dcd87c8ae4083e7017a05456c14eea4b1db2032126e27b3b1563d57d7cc0a08151d548273f6683169524b66ca9fe338b9ce42bc3540046c828fd939ae23bcba03f4e5c2ec5b2170b711d97ee755c160457bb58d8daa338e835ec02ae6860bbabb901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083020000018502540be40082a8798203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f863f861800a8405f5e10094100000000000000000000000000000000000000080801ba07e09e26678ed4fac08a249ebe8ed680bf9051a5e14ad223e4b2b9d26e0208f37a05f6e3f188e3e6eab7d7d3b6568f5eac7d687b08d307d3154ccd8c87b4630509bc0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider.insert_block(genesis.try_recover().unwrap()).unwrap();
provider.insert_block(block.clone().try_recover().unwrap()).unwrap();
provider.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider.insert_block(&block.clone().try_recover().unwrap()).unwrap();
provider
.static_file_provider()
.latest_writer(StaticFileSegment::Headers)
@@ -830,8 +830,8 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f90262f901f9a075c371ba45999d87f4542326910a11af515897aebce5265d3f6acd1f1161f82fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa098f2dcd87c8ae4083e7017a05456c14eea4b1db2032126e27b3b1563d57d7cc0a08151d548273f6683169524b66ca9fe338b9ce42bc3540046c828fd939ae23bcba03f4e5c2ec5b2170b711d97ee755c160457bb58d8daa338e835ec02ae6860bbabb901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083020000018502540be40082a8798203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f863f861800a8405f5e10094100000000000000000000000000000000000000080801ba07e09e26678ed4fac08a249ebe8ed680bf9051a5e14ad223e4b2b9d26e0208f37a05f6e3f188e3e6eab7d7d3b6568f5eac7d687b08d307d3154ccd8c87b4630509bc0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider.insert_block(genesis.try_recover().unwrap()).unwrap();
provider.insert_block(block.clone().try_recover().unwrap()).unwrap();
provider.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider.insert_block(&block.clone().try_recover().unwrap()).unwrap();
provider
.static_file_provider()
.latest_writer(StaticFileSegment::Headers)
@@ -863,8 +863,8 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f90262f901f9a075c371ba45999d87f4542326910a11af515897aebce5265d3f6acd1f1161f82fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa098f2dcd87c8ae4083e7017a05456c14eea4b1db2032126e27b3b1563d57d7cc0a08151d548273f6683169524b66ca9fe338b9ce42bc3540046c828fd939ae23bcba03f4e5c2ec5b2170b711d97ee755c160457bb58d8daa338e835ec02ae6860bbabb901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083020000018502540be40082a8798203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f863f861800a8405f5e10094100000000000000000000000000000000000000080801ba07e09e26678ed4fac08a249ebe8ed680bf9051a5e14ad223e4b2b9d26e0208f37a05f6e3f188e3e6eab7d7d3b6568f5eac7d687b08d307d3154ccd8c87b4630509bc0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider.insert_block(genesis.try_recover().unwrap()).unwrap();
provider.insert_block(block.clone().try_recover().unwrap()).unwrap();
provider.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider.insert_block(&block.clone().try_recover().unwrap()).unwrap();
provider
.static_file_provider()
.latest_writer(StaticFileSegment::Headers)
@@ -1005,8 +1005,8 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f90262f901f9a075c371ba45999d87f4542326910a11af515897aebce5265d3f6acd1f1161f82fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa098f2dcd87c8ae4083e7017a05456c14eea4b1db2032126e27b3b1563d57d7cc0a08151d548273f6683169524b66ca9fe338b9ce42bc3540046c828fd939ae23bcba03f4e5c2ec5b2170b711d97ee755c160457bb58d8daa338e835ec02ae6860bbabb901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083020000018502540be40082a8798203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f863f861800a8405f5e10094100000000000000000000000000000000000000080801ba07e09e26678ed4fac08a249ebe8ed680bf9051a5e14ad223e4b2b9d26e0208f37a05f6e3f188e3e6eab7d7d3b6568f5eac7d687b08d307d3154ccd8c87b4630509bc0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider.insert_block(genesis.try_recover().unwrap()).unwrap();
provider.insert_block(block.clone().try_recover().unwrap()).unwrap();
provider.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider.insert_block(&block.clone().try_recover().unwrap()).unwrap();
provider
.static_file_provider()
.latest_writer(StaticFileSegment::Headers)
@@ -1115,8 +1115,8 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f9025ff901f7a0c86e8cc0310ae7c531c758678ddbfd16fc51c8cef8cec650b032de9869e8b94fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa050554882fbbda2c2fd93fdc466db9946ea262a67f7a76cc169e714f105ab583da00967f09ef1dfed20c0eacfaa94d5cd4002eda3242ac47eae68972d07b106d192a0e3c8b47fbfc94667ef4cceb17e5cc21e3b1eebd442cebb27f07562b33836290db90100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008302000001830f42408238108203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f862f860800a83061a8094095e7baea6a6c7c4c2dfeb977efac326af552d8780801ba072ed817487b84ba367d15d2f039b5fc5f087d0a8882fbdf73e8cb49357e1ce30a0403d800545b8fc544f92ce8124e2255f8c3c6af93f28243a120585d4c4c6a2a3c0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider.insert_block(genesis.try_recover().unwrap()).unwrap();
provider.insert_block(block.clone().try_recover().unwrap()).unwrap();
provider.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider.insert_block(&block.clone().try_recover().unwrap()).unwrap();
provider
.static_file_provider()
.latest_writer(StaticFileSegment::Headers)
@@ -1265,13 +1265,15 @@ mod tests {
let provider_rw = factory.database_provider_rw().unwrap();
let mut rng = generators::rng();
let genesis = generators::random_block(&mut rng, 0, Default::default());
provider_rw.insert_block(genesis.try_recover().unwrap()).expect("failed to insert genesis");
provider_rw
.insert_block(&genesis.try_recover().unwrap())
.expect("failed to insert genesis");
let block = generators::random_block(
&mut rng,
1,
generators::BlockParams { tx_count: Some(2), ..Default::default() },
);
provider_rw.insert_block(block.try_recover().unwrap()).expect("failed to insert block");
provider_rw.insert_block(&block.try_recover().unwrap()).expect("failed to insert block");
let static_file_provider = provider_rw.static_file_provider();
static_file_provider.latest_writer(StaticFileSegment::Headers).unwrap().commit().unwrap();

View File

@@ -86,7 +86,7 @@ impl AccountHashingStage {
);
for block in blocks {
provider.insert_block(block.try_recover().unwrap()).unwrap();
provider.insert_block(&block.try_recover().unwrap()).unwrap();
}
provider
.static_file_provider()

View File

@@ -1,4 +1,4 @@
use alloy_primitives::{bytes::BufMut, keccak256, B256};
use alloy_primitives::{bytes::BufMut, keccak256, Address, B256};
use itertools::Itertools;
use reth_config::config::{EtlConfig, HashingConfig};
use reth_db_api::{
@@ -101,9 +101,16 @@ where
let chunk = chunk.collect::<Result<Vec<_>, _>>()?;
// Spawn the hashing task onto the global rayon pool
rayon::spawn(move || {
// Cache hashed address since PlainStorageState is sorted by address
let (mut last_addr, mut hashed_addr) =
(Address::ZERO, keccak256(Address::ZERO));
for (address, slot) in chunk {
if address != last_addr {
last_addr = address;
hashed_addr = keccak256(address);
}
let mut addr_key = Vec::with_capacity(64);
addr_key.put_slice(keccak256(address).as_slice());
addr_key.put_slice(hashed_addr.as_slice());
addr_key.put_slice(keccak256(slot.key).as_slice());
let _ = tx.send((addr_key, CompactU256::from(slot.value)));
}

View File

@@ -5,7 +5,7 @@ use reth_consensus::ConsensusError;
use reth_primitives_traits::{GotExpected, SealedHeader};
use reth_provider::{
ChainStateBlockReader, DBProvider, HeaderProvider, ProviderError, PruneCheckpointReader,
PruneCheckpointWriter, StageCheckpointReader, TrieWriter,
PruneCheckpointWriter, StageCheckpointReader, StageCheckpointWriter, TrieWriter,
};
use reth_prune_types::{
PruneCheckpoint, PruneMode, PruneSegment, MERKLE_CHANGESETS_RETENTION_BLOCKS,
@@ -300,6 +300,7 @@ where
+ DBProvider
+ HeaderProvider
+ ChainStateBlockReader
+ StageCheckpointWriter
+ PruneCheckpointReader
+ PruneCheckpointWriter,
{
@@ -404,6 +405,34 @@ where
computed_range.start = computed_range.end;
}
// If we've unwound so far that there are no longer enough trie changesets available then
// simply clear them and the checkpoints, so that on next pipeline startup they will be
// regenerated.
//
// We don't do this check if the target block is not greater than the retention threshold
// (which happens near genesis), as in that case would could still have all possible
// changesets even if the total count doesn't meet the threshold.
debug!(
target: "sync::stages::merkle_changesets",
?computed_range,
retention_blocks=?self.retention_blocks,
"Checking if computed range is over retention threshold",
);
if input.unwind_to > self.retention_blocks &&
computed_range.end - computed_range.start < self.retention_blocks
{
debug!(
target: "sync::stages::merkle_changesets",
?computed_range,
retention_blocks=?self.retention_blocks,
"Clearing checkpoints completely",
);
provider.clear_trie_changesets()?;
provider
.save_stage_checkpoint(StageId::MerkleChangeSets, StageCheckpoint::default())?;
return Ok(UnwindOutput { checkpoint: StageCheckpoint::default() })
}
// `computed_range.end` is exclusive
let checkpoint = StageCheckpoint::new(computed_range.end.saturating_sub(1));

View File

@@ -96,11 +96,11 @@ mod tests {
let genesis = SealedBlock::<Block>::decode(&mut genesis_rlp).unwrap();
let mut block_rlp = hex!("f90262f901f9a075c371ba45999d87f4542326910a11af515897aebce5265d3f6acd1f1161f82fa01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942adc25665018aa1fe0e6bc666dac8fc2697ff9baa098f2dcd87c8ae4083e7017a05456c14eea4b1db2032126e27b3b1563d57d7cc0a08151d548273f6683169524b66ca9fe338b9ce42bc3540046c828fd939ae23bcba03f4e5c2ec5b2170b711d97ee755c160457bb58d8daa338e835ec02ae6860bbabb901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000083020000018502540be40082a8798203e800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f863f861800a8405f5e10094100000000000000000000000000000000000000080801ba07e09e26678ed4fac08a249ebe8ed680bf9051a5e14ad223e4b2b9d26e0208f37a05f6e3f188e3e6eab7d7d3b6568f5eac7d687b08d307d3154ccd8c87b4630509bc0").as_slice();
let block = SealedBlock::<Block>::decode(&mut block_rlp).unwrap();
provider_rw.insert_block(genesis.try_recover().unwrap()).unwrap();
provider_rw.insert_block(block.clone().try_recover().unwrap()).unwrap();
let mut head = block.hash();
provider_rw.insert_block(&genesis.try_recover().unwrap()).unwrap();
provider_rw.insert_block(&block.try_recover().unwrap()).unwrap();
// Fill with bogus blocks to respect PruneMode distance.
let mut head = block.hash();
let mut rng = generators::rng();
for block_number in 2..=tip {
let nblock = random_block(
@@ -109,7 +109,7 @@ mod tests {
generators::BlockParams { parent: Some(head), ..Default::default() },
);
head = nblock.hash();
provider_rw.insert_block(nblock.try_recover().unwrap()).unwrap();
provider_rw.insert_block(&nblock.try_recover().unwrap()).unwrap();
}
provider_rw
.static_file_provider()

View File

@@ -12,7 +12,7 @@ pub type CursorTy<TX, T> = <TX as DbTx>::Cursor<T>;
pub type CursorMutTy<TX, T> = <TX as DbTxMut>::CursorMut<T>;
/// Read only transaction
pub trait DbTx: Debug + Send + Sync {
pub trait DbTx: Debug + Send {
/// Cursor type for this read-only transaction
type Cursor<T: Table>: DbCursorRO<T> + Send + Sync;
/// `DupCursor` type for this read-only transaction
@@ -43,7 +43,7 @@ pub trait DbTx: Debug + Send + Sync {
}
/// Read write transaction that allows writing to database
pub trait DbTxMut: Send + Sync {
pub trait DbTxMut: Send {
/// Read-Write Cursor type
type CursorMut<T: Table>: DbCursorRW<T> + DbCursorRO<T> + Send + Sync;
/// Read-Write `DupCursor` type

View File

@@ -30,7 +30,7 @@ pub trait DbTxUnwindExt: DbTxMut {
let mut deleted = 0;
while let Some(Ok((entry_key, _))) = reverse_walker.next() {
if selector(entry_key.clone()) <= key {
if selector(entry_key) <= key {
break
}
reverse_walker.delete_current()?;

View File

@@ -875,7 +875,7 @@ mod tests {
// Insert blocks into the database
for block in &database_blocks {
provider_rw.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)?;
}
@@ -1005,9 +1005,10 @@ mod tests {
let provider_rw = factory.provider_rw()?;
for block in database_blocks {
provider_rw.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)?;
}
provider_rw.commit()?;
// Create a new provider
@@ -1103,7 +1104,7 @@ mod tests {
let provider_rw = factory.provider_rw()?;
for block in database_blocks {
provider_rw.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)?;
}
provider_rw.commit()?;
@@ -1320,7 +1321,7 @@ mod tests {
// Insert and commit the block.
let provider_rw = factory.provider_rw()?;
provider_rw.insert_block(block_1)?;
provider_rw.insert_block(&block_1)?;
provider_rw.commit()?;
let provider = BlockchainProvider::new(factory)?;

View File

@@ -26,7 +26,7 @@ use reth_stages_types::{StageCheckpoint, StageId};
use reth_static_file_types::StaticFileSegment;
use reth_storage_api::{
BlockBodyIndicesProvider, DatabaseProviderFactory, NodePrimitivesProvider, StateProvider,
StorageChangeSetReader, TryIntoHistoricalStateProvider,
StateProviderBox, StorageChangeSetReader, TryIntoHistoricalStateProvider,
};
use reth_storage_errors::provider::ProviderResult;
use reth_trie::updates::TrieUpdatesSorted;
@@ -596,9 +596,9 @@ impl<N: ProviderNodeTypes> ConsistentProvider<N> {
pub(crate) fn into_state_provider_at_block_hash(
self,
block_hash: BlockHash,
) -> ProviderResult<Box<dyn StateProvider>> {
) -> ProviderResult<StateProviderBox> {
let Self { storage_provider, head_block, .. } = self;
let into_history_at_block_hash = |block_hash| -> ProviderResult<Box<dyn StateProvider>> {
let into_history_at_block_hash = |block_hash| -> ProviderResult<StateProviderBox> {
let block_number = storage_provider
.block_number(block_hash)?
.ok_or(ProviderError::BlockHashNotFound(block_hash))?;
@@ -1566,7 +1566,7 @@ mod tests {
let provider_rw = factory.provider_rw()?;
for block in database_blocks {
provider_rw.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)?;
}
provider_rw.commit()?;
@@ -1677,7 +1677,7 @@ mod tests {
let provider_rw = factory.provider_rw()?;
for block in database_blocks {
provider_rw.insert_block(
block.clone().try_recover().expect("failed to seal block with senders"),
&block.clone().try_recover().expect("failed to seal block with senders"),
)?;
}
provider_rw.commit()?;

View File

@@ -103,7 +103,7 @@ mod tests {
// insert the block
let provider_rw = provider_factory.provider_rw().unwrap();
provider_rw.insert_block(genesis_block).unwrap();
provider_rw.insert_block(&genesis_block).unwrap();
provider_rw.commit().unwrap();
// create a consistent view provider and check that a ro provider can be made
@@ -121,7 +121,7 @@ mod tests {
// insert the block
let provider_rw = provider_factory.provider_rw().unwrap();
provider_rw.insert_block(recovered_block).unwrap();
provider_rw.insert_block(&recovered_block).unwrap();
provider_rw.commit().unwrap();
// ensure successful creation of a read-only provider, based on this new db state.
@@ -136,7 +136,7 @@ mod tests {
// insert the block
let provider_rw = provider_factory.provider_rw().unwrap();
provider_rw.insert_block(recovered_block).unwrap();
provider_rw.insert_block(&recovered_block).unwrap();
provider_rw.commit().unwrap();
// check that creation of a read-only provider still works
@@ -156,7 +156,7 @@ mod tests {
// insert the block
let provider_rw = provider_factory.provider_rw().unwrap();
provider_rw.insert_block(genesis_block).unwrap();
provider_rw.insert_block(&genesis_block).unwrap();
provider_rw.commit().unwrap();
// create a consistent view provider and check that a ro provider can be made
@@ -174,7 +174,7 @@ mod tests {
// insert the block
let provider_rw = provider_factory.provider_rw().unwrap();
provider_rw.insert_block(recovered_block).unwrap();
provider_rw.insert_block(&recovered_block).unwrap();
provider_rw.commit().unwrap();
// create a second consistent view provider and check that a ro provider can be made
@@ -208,7 +208,7 @@ mod tests {
// reinsert the block at the same height, but with a different hash
let provider_rw = provider_factory.provider_rw().unwrap();
provider_rw.insert_block(recovered_block).unwrap();
provider_rw.insert_block(&recovered_block).unwrap();
provider_rw.commit().unwrap();
// ensure unsuccessful creation of a read-only provider, based on this new db state.

View File

@@ -726,7 +726,7 @@ mod tests {
{
let factory = create_test_provider_factory();
let provider = factory.provider_rw().unwrap();
assert_matches!(provider.insert_block(block.clone().try_recover().unwrap()), Ok(_));
assert_matches!(provider.insert_block(&block.clone().try_recover().unwrap()), Ok(_));
assert_matches!(
provider.transaction_sender(0), Ok(Some(sender))
if sender == block.body().transactions[0].recover_signer().unwrap()
@@ -745,7 +745,7 @@ mod tests {
};
let factory = create_test_provider_factory();
let provider = factory.with_prune_modes(prune_modes).provider_rw().unwrap();
assert_matches!(provider.insert_block(block.clone().try_recover().unwrap()), Ok(_));
assert_matches!(provider.insert_block(&block.clone().try_recover().unwrap()), Ok(_));
assert_matches!(provider.transaction_sender(0), Ok(None));
assert_matches!(
provider.transaction_id(*block.body().transactions[0].tx_hash()),
@@ -765,7 +765,7 @@ mod tests {
let factory = create_test_provider_factory();
let provider = factory.provider_rw().unwrap();
assert_matches!(provider.insert_block(block.clone().try_recover().unwrap()), Ok(_));
assert_matches!(provider.insert_block(&block.clone().try_recover().unwrap()), Ok(_));
let senders = provider.take::<tables::TransactionSenders>(range.clone());
assert_eq!(

View File

@@ -284,7 +284,7 @@ impl<TX, N: NodeTypes> RocksDBProviderFactory for DatabaseProvider<TX, N> {
}
}
impl<TX: Debug + Send + Sync, N: NodeTypes<ChainSpec: EthChainSpec + 'static>> ChainSpecProvider
impl<TX: Debug + Send, N: NodeTypes<ChainSpec: EthChainSpec + 'static>> ChainSpecProvider
for DatabaseProvider<TX, N>
{
type ChainSpec = N::ChainSpec;
@@ -356,7 +356,7 @@ impl<TX: DbTx + DbTxMut + 'static, N: NodeTypesForProvider> DatabaseProvider<TX,
let trie_data = block.trie_data();
let ExecutedBlock { recovered_block, execution_output, .. } = block;
let block_number = recovered_block.number();
self.insert_block(Arc::unwrap_or_clone(recovered_block))?;
self.insert_block(&recovered_block)?;
// Write state and changesets to the database.
// Must be written after blocks because of the receipt lookup.
@@ -2875,7 +2875,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
/// [`TransactionHashNumbers`](tables::TransactionHashNumbers).
fn insert_block(
&self,
block: RecoveredBlock<Self::Block>,
block: &RecoveredBlock<Self::Block>,
) -> ProviderResult<StoredBlockBodyIndices> {
let block_number = block.number();
let tx_count = block.body().transaction_count() as u64;
@@ -2915,7 +2915,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
durations_recorder.record_relative(metrics::Action::InsertTransactionHashNumbers);
}
self.append_block_bodies(vec![(block_number, Some(block.into_body()))])?;
self.append_block_bodies(vec![(block_number, Some(block.body()))])?;
debug!(
target: "providers::db",
@@ -2929,7 +2929,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
fn append_block_bodies(
&self,
bodies: Vec<(BlockNumber, Option<BodyTy<N>>)>,
bodies: Vec<(BlockNumber, Option<&BodyTy<N>>)>,
) -> ProviderResult<()> {
let Some(from_block) = bodies.first().map(|(block, _)| *block) else { return Ok(()) };
@@ -3081,7 +3081,7 @@ impl<TX: DbTxMut + DbTx + 'static, N: NodeTypesForProvider + 'static> BlockWrite
// Insert the blocks
for block in blocks {
self.insert_block(block)?;
self.insert_block(&block)?;
durations_recorder.record_relative(metrics::Action::InsertBlock);
}
@@ -3253,7 +3253,7 @@ impl<TX: DbTxMut, N: NodeTypes> MetadataWriter for DatabaseProvider<TX, N> {
}
}
impl<TX: Send + Sync, N: NodeTypes> StorageSettingsCache for DatabaseProvider<TX, N> {
impl<TX: Send, N: NodeTypes> StorageSettingsCache for DatabaseProvider<TX, N> {
fn cached_storage_settings(&self) -> StorageSettings {
*self.storage_settings.read()
}
@@ -3302,14 +3302,14 @@ mod tests {
let data = BlockchainTestData::default();
let provider_rw = factory.provider_rw().unwrap();
provider_rw.insert_block(data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw.insert_block(&data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
crate::OriginalValuesKnown::No,
)
.unwrap();
provider_rw.insert_block(data.blocks[0].0.clone()).unwrap();
provider_rw.insert_block(&data.blocks[0].0).unwrap();
provider_rw.write_state(&data.blocks[0].1, crate::OriginalValuesKnown::No).unwrap();
provider_rw.commit().unwrap();
@@ -3328,7 +3328,7 @@ mod tests {
let data = BlockchainTestData::default();
let provider_rw = factory.provider_rw().unwrap();
provider_rw.insert_block(data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw.insert_block(&data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
@@ -3336,7 +3336,7 @@ mod tests {
)
.unwrap();
for i in 0..3 {
provider_rw.insert_block(data.blocks[i].0.clone()).unwrap();
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
}
provider_rw.commit().unwrap();
@@ -3358,7 +3358,7 @@ mod tests {
let data = BlockchainTestData::default();
let provider_rw = factory.provider_rw().unwrap();
provider_rw.insert_block(data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw.insert_block(&data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
@@ -3368,7 +3368,7 @@ mod tests {
// insert blocks 1-3 with receipts
for i in 0..3 {
provider_rw.insert_block(data.blocks[i].0.clone()).unwrap();
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
}
provider_rw.commit().unwrap();
@@ -3389,7 +3389,7 @@ mod tests {
let data = BlockchainTestData::default();
let provider_rw = factory.provider_rw().unwrap();
provider_rw.insert_block(data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw.insert_block(&data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
@@ -3397,7 +3397,7 @@ mod tests {
)
.unwrap();
for i in 0..3 {
provider_rw.insert_block(data.blocks[i].0.clone()).unwrap();
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
}
provider_rw.commit().unwrap();
@@ -3433,7 +3433,7 @@ mod tests {
let provider_rw = factory.provider_rw().unwrap();
for block in blocks {
provider_rw.insert_block(block.try_recover().unwrap()).unwrap();
provider_rw.insert_block(&block.try_recover().unwrap()).unwrap();
}
provider_rw.commit().unwrap();
@@ -3452,7 +3452,7 @@ mod tests {
let data = BlockchainTestData::default();
let provider_rw = factory.provider_rw().unwrap();
provider_rw.insert_block(data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw.insert_block(&data.genesis.clone().try_recover().unwrap()).unwrap();
provider_rw
.write_state(
&ExecutionOutcome { first_block: 0, receipts: vec![vec![]], ..Default::default() },
@@ -3460,7 +3460,7 @@ mod tests {
)
.unwrap();
for i in 0..3 {
provider_rw.insert_block(data.blocks[i].0.clone()).unwrap();
provider_rw.insert_block(&data.blocks[i].0).unwrap();
provider_rw.write_state(&data.blocks[i].1, crate::OriginalValuesKnown::No).unwrap();
}
provider_rw.commit().unwrap();
@@ -4760,7 +4760,7 @@ mod tests {
block_num,
BlockParams { tx_count: Some(tx_count), ..Default::default() },
);
provider_rw.insert_block(block.try_recover().unwrap()).unwrap();
provider_rw.insert_block(&block.try_recover().unwrap()).unwrap();
}
};

View File

@@ -69,6 +69,13 @@ impl RocksDBProvider {
unwind_target = Some(unwind_target.map_or(target, |t| t.min(target)));
}
// Check AccountsHistory if stored in RocksDB
if provider.cached_storage_settings().account_history_in_rocksdb &&
let Some(target) = self.check_accounts_history(provider)?
{
unwind_target = Some(unwind_target.map_or(target, |t| t.min(target)));
}
Ok(unwind_target)
}
@@ -155,6 +162,16 @@ impl RocksDBProvider {
"MDBX empty but static files have data, pruning all TransactionHashNumbers"
);
self.prune_transaction_hash_numbers_in_range(provider, 0..=highest_tx)?;
// If checkpoint claims progress but MDBX is empty, that's an inconsistency
if checkpoint > 0 {
tracing::warn!(
target: "reth::providers::rocksdb",
checkpoint,
"Checkpoint set but MDBX has no transactions, unwind needed"
);
return Ok(Some(0));
}
}
(None, None) => {
// Both MDBX and static files are empty.
@@ -275,6 +292,18 @@ impl RocksDBProvider {
"StoragesHistory ahead of checkpoint, pruning excess data"
);
self.prune_storages_history_above(checkpoint)?;
return Ok(None);
}
// If RocksDB is behind the checkpoint, request an unwind to rebuild.
if max_highest_block < checkpoint {
tracing::warn!(
target: "reth::providers::rocksdb",
rocks_highest = max_highest_block,
checkpoint,
"StoragesHistory behind checkpoint, unwind needed"
);
return Ok(Some(max_highest_block));
}
Ok(None)
@@ -327,6 +356,125 @@ impl RocksDBProvider {
Ok(())
}
/// Checks invariants for the `AccountsHistory` table.
///
/// Returns a block number to unwind to if `RocksDB` is behind the checkpoint.
/// If `RocksDB` is ahead of the checkpoint, excess entries are pruned (healed).
fn check_accounts_history<Provider>(
&self,
provider: &Provider,
) -> ProviderResult<Option<BlockNumber>>
where
Provider: DBProvider + StageCheckpointReader,
{
// Get the IndexAccountHistory stage checkpoint
let checkpoint = provider
.get_stage_checkpoint(StageId::IndexAccountHistory)?
.map(|cp| cp.block_number)
.unwrap_or(0);
// Check if RocksDB has any data
let rocks_first = self.first::<tables::AccountsHistory>()?;
match rocks_first {
Some(_) => {
// If checkpoint is 0 but we have data, clear everything
if checkpoint == 0 {
tracing::info!(
target: "reth::providers::rocksdb",
"AccountsHistory has data but checkpoint is 0, clearing all"
);
self.prune_accounts_history_above(0)?;
return Ok(None);
}
// Find the max highest_block_number (excluding u64::MAX sentinel) across all
// entries
let mut max_highest_block = 0u64;
for result in self.iter::<tables::AccountsHistory>()? {
let (key, _) = result?;
let highest = key.highest_block_number;
if highest != u64::MAX && highest > max_highest_block {
max_highest_block = highest;
}
}
// If any entry has highest_block > checkpoint, prune excess
if max_highest_block > checkpoint {
tracing::info!(
target: "reth::providers::rocksdb",
rocks_highest = max_highest_block,
checkpoint,
"AccountsHistory ahead of checkpoint, pruning excess data"
);
self.prune_accounts_history_above(checkpoint)?;
return Ok(None);
}
// If RocksDB is behind the checkpoint, request an unwind to rebuild.
if max_highest_block < checkpoint {
tracing::warn!(
target: "reth::providers::rocksdb",
rocks_highest = max_highest_block,
checkpoint,
"AccountsHistory behind checkpoint, unwind needed"
);
return Ok(Some(max_highest_block));
}
Ok(None)
}
None => {
// Empty RocksDB table
if checkpoint > 0 {
// Stage says we should have data but we don't
return Ok(Some(0));
}
Ok(None)
}
}
}
/// Prunes `AccountsHistory` entries where `highest_block_number` > `max_block`.
///
/// For `AccountsHistory`, the key is `ShardedKey<Address>` which contains
/// `highest_block_number`, so we can iterate and delete entries where
/// `key.highest_block_number > max_block`.
///
/// TODO(<https://github.com/paradigmxyz/reth/issues/20417>): this iterates the whole table,
/// which is inefficient. Use changeset-based pruning instead.
fn prune_accounts_history_above(&self, max_block: BlockNumber) -> ProviderResult<()> {
use alloy_primitives::Address;
use reth_db_api::models::ShardedKey;
let mut to_delete: Vec<ShardedKey<Address>> = Vec::new();
for result in self.iter::<tables::AccountsHistory>()? {
let (key, _) = result?;
let highest_block = key.highest_block_number;
if max_block == 0 || (highest_block != u64::MAX && highest_block > max_block) {
to_delete.push(key);
}
}
let deleted = to_delete.len();
if deleted > 0 {
tracing::info!(
target: "reth::providers::rocksdb",
deleted_count = deleted,
max_block,
"Pruning AccountsHistory entries"
);
let mut batch = self.batch();
for key in to_delete {
batch.delete::<tables::AccountsHistory>(key)?;
}
batch.commit()?;
}
Ok(())
}
}
#[cfg(test)]
@@ -463,7 +611,9 @@ mod tests {
let provider = factory.database_provider_rw().unwrap();
let mut tx_count = 0u64;
for block in &blocks {
provider.insert_block(block.clone().try_recover().expect("recover block")).unwrap();
provider
.insert_block(&block.clone().try_recover().expect("recover block"))
.unwrap();
for tx in &block.body().transactions {
let hash = tx.trie_hash();
tx_hashes.push(hash);
@@ -578,6 +728,44 @@ mod tests {
);
}
#[test]
fn test_check_consistency_storages_history_behind_checkpoint_needs_unwind() {
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::StoragesHistory>()
.build()
.unwrap();
// Insert data into RocksDB with highest_block_number below checkpoint
let key_block_50 = StorageShardedKey::new(Address::ZERO, B256::ZERO, 50);
let block_list = BlockNumberList::new_pre_sorted([10, 20, 30, 50]);
rocksdb.put::<tables::StoragesHistory>(key_block_50, &block_list).unwrap();
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_storages_history_in_rocksdb(true),
);
// Set checkpoint to block 100
{
let provider = factory.database_provider_rw().unwrap();
provider
.save_stage_checkpoint(StageId::IndexStorageHistory, StageCheckpoint::new(100))
.unwrap();
provider.commit().unwrap();
}
let provider = factory.database_provider_ro().unwrap();
// RocksDB only has data up to block 50, but checkpoint says block 100 was processed
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(
result,
Some(50),
"Should require unwind to block 50 to rebuild StoragesHistory"
);
}
#[test]
fn test_check_consistency_mdbx_behind_checkpoint_needs_unwind() {
let temp_dir = TempDir::new().unwrap();
@@ -603,7 +791,9 @@ mod tests {
let provider = factory.database_provider_rw().unwrap();
let mut tx_count = 0u64;
for block in &blocks {
provider.insert_block(block.clone().try_recover().expect("recover block")).unwrap();
provider
.insert_block(&block.clone().try_recover().expect("recover block"))
.unwrap();
for tx in &block.body().transactions {
let hash = tx.trie_hash();
rocksdb.put::<tables::TransactionHashNumbers>(hash, &tx_count).unwrap();
@@ -667,7 +857,9 @@ mod tests {
let provider = factory.database_provider_rw().unwrap();
// Insert ALL blocks (0-5) to write transactions to static files
for block in &blocks {
provider.insert_block(block.clone().try_recover().expect("recover block")).unwrap();
provider
.insert_block(&block.clone().try_recover().expect("recover block"))
.unwrap();
for tx in &block.body().transactions {
let hash = tx.trie_hash();
tx_hashes.push(hash);
@@ -797,6 +989,177 @@ mod tests {
);
}
#[test]
fn test_check_consistency_accounts_history_empty_with_checkpoint_needs_unwind() {
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::AccountsHistory>()
.build()
.unwrap();
// Create a test provider factory for MDBX
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
// Set a checkpoint indicating we should have processed up to block 100
{
let provider = factory.database_provider_rw().unwrap();
provider
.save_stage_checkpoint(StageId::IndexAccountHistory, StageCheckpoint::new(100))
.unwrap();
provider.commit().unwrap();
}
let provider = factory.database_provider_ro().unwrap();
// RocksDB is empty but checkpoint says block 100 was processed
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(result, Some(0), "Should require unwind to block 0 to rebuild AccountsHistory");
}
#[test]
fn test_check_consistency_accounts_history_has_data_no_checkpoint_prunes_data() {
use reth_db_api::models::ShardedKey;
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::AccountsHistory>()
.build()
.unwrap();
// Insert data into RocksDB
let key = ShardedKey::new(Address::ZERO, 50);
let block_list = BlockNumberList::new_pre_sorted([10, 20, 30, 50]);
rocksdb.put::<tables::AccountsHistory>(key, &block_list).unwrap();
// Verify data exists
assert!(rocksdb.last::<tables::AccountsHistory>().unwrap().is_some());
// Create a test provider factory for MDBX with NO checkpoint
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
let provider = factory.database_provider_ro().unwrap();
// RocksDB has data but checkpoint is 0
// This means RocksDB has stale data that should be pruned (healed)
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(result, None, "Should heal by pruning, no unwind needed");
// Verify data was pruned
assert!(
rocksdb.last::<tables::AccountsHistory>().unwrap().is_none(),
"RocksDB should be empty after pruning"
);
}
#[test]
fn test_check_consistency_accounts_history_ahead_of_checkpoint_prunes_excess() {
use reth_db_api::models::ShardedKey;
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::AccountsHistory>()
.build()
.unwrap();
// Insert data into RocksDB with different highest_block_numbers
let key_block_50 = ShardedKey::new(Address::ZERO, 50);
let key_block_100 = ShardedKey::new(Address::random(), 100);
let key_block_150 = ShardedKey::new(Address::random(), 150);
let key_block_max = ShardedKey::new(Address::random(), u64::MAX);
let block_list = BlockNumberList::new_pre_sorted([10, 20, 30]);
rocksdb.put::<tables::AccountsHistory>(key_block_50.clone(), &block_list).unwrap();
rocksdb.put::<tables::AccountsHistory>(key_block_100.clone(), &block_list).unwrap();
rocksdb.put::<tables::AccountsHistory>(key_block_150.clone(), &block_list).unwrap();
rocksdb.put::<tables::AccountsHistory>(key_block_max.clone(), &block_list).unwrap();
// Create a test provider factory for MDBX
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
// Set checkpoint to block 100
{
let provider = factory.database_provider_rw().unwrap();
provider
.save_stage_checkpoint(StageId::IndexAccountHistory, StageCheckpoint::new(100))
.unwrap();
provider.commit().unwrap();
}
let provider = factory.database_provider_ro().unwrap();
// RocksDB has entries with highest_block = 150 which exceeds checkpoint (100)
// Should prune entries where highest_block > 100 (but not u64::MAX sentinel)
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(result, None, "Should heal by pruning, no unwind needed");
// Verify key_block_150 was pruned, but others remain
assert!(
rocksdb.get::<tables::AccountsHistory>(key_block_50).unwrap().is_some(),
"Entry with highest_block=50 should remain"
);
assert!(
rocksdb.get::<tables::AccountsHistory>(key_block_100).unwrap().is_some(),
"Entry with highest_block=100 should remain"
);
assert!(
rocksdb.get::<tables::AccountsHistory>(key_block_150).unwrap().is_none(),
"Entry with highest_block=150 should be pruned"
);
assert!(
rocksdb.get::<tables::AccountsHistory>(key_block_max).unwrap().is_some(),
"Entry with highest_block=u64::MAX (sentinel) should remain"
);
}
#[test]
fn test_check_consistency_accounts_history_behind_checkpoint_needs_unwind() {
use reth_db_api::models::ShardedKey;
let temp_dir = TempDir::new().unwrap();
let rocksdb = RocksDBBuilder::new(temp_dir.path())
.with_table::<tables::AccountsHistory>()
.build()
.unwrap();
// Insert data into RocksDB with highest_block_number below checkpoint
let key_block_50 = ShardedKey::new(Address::ZERO, 50);
let block_list = BlockNumberList::new_pre_sorted([10, 20, 30, 50]);
rocksdb.put::<tables::AccountsHistory>(key_block_50, &block_list).unwrap();
let factory = create_test_provider_factory();
factory.set_storage_settings_cache(
StorageSettings::legacy().with_account_history_in_rocksdb(true),
);
// Set checkpoint to block 100
{
let provider = factory.database_provider_rw().unwrap();
provider
.save_stage_checkpoint(StageId::IndexAccountHistory, StageCheckpoint::new(100))
.unwrap();
provider.commit().unwrap();
}
let provider = factory.database_provider_ro().unwrap();
// RocksDB only has data up to block 50, but checkpoint says block 100 was processed
let result = rocksdb.check_consistency(&provider).unwrap();
assert_eq!(
result,
Some(50),
"Should require unwind to block 50 to rebuild AccountsHistory"
);
}
/// Test that pruning works by fetching transactions and computing their hashes,
/// rather than iterating all rows. This test uses random blocks with unique
/// transactions so we can verify the correct entries are pruned.
@@ -831,7 +1194,9 @@ mod tests {
let provider = factory.database_provider_rw().unwrap();
for block in &blocks {
provider.insert_block(block.clone().try_recover().expect("recover block")).unwrap();
provider
.insert_block(&block.clone().try_recover().expect("recover block"))
.unwrap();
// Store transaction hash -> tx_number mappings in RocksDB
for tx in &block.body().transactions {

View File

@@ -23,7 +23,9 @@ pub struct RocksDBProvider;
impl RocksDBProvider {
/// Creates a new stub `RocksDB` provider.
///
/// On non-Unix platforms, this returns an error indicating `RocksDB` is not supported.
/// This stub always succeeds but any actual operations will return `UnsupportedProvider`.
/// When using this stub, all `*_in_rocksdb` flags should be set to `false` to ensure
/// operations route to MDBX instead.
pub fn new(_path: impl AsRef<Path>) -> ProviderResult<Self> {
Ok(Self)
}
@@ -77,6 +79,39 @@ impl RocksDBProvider {
pub const fn tx(&self) -> RocksTx {
RocksTx
}
/// Creates a new batch for atomic writes (stub implementation).
pub const fn batch(&self) -> RocksDBBatch {
RocksDBBatch
}
/// Gets the first key-value pair from a table (stub implementation).
pub const fn first<T: Table>(&self) -> ProviderResult<Option<(T::Key, T::Value)>> {
Ok(None)
}
/// Gets the last key-value pair from a table (stub implementation).
pub const fn last<T: Table>(&self) -> ProviderResult<Option<(T::Key, T::Value)>> {
Ok(None)
}
/// Creates an iterator for the specified table (stub implementation).
///
/// Returns an empty iterator. This is consistent with `first()` and `last()` returning
/// `Ok(None)` - the stub behaves as if the database is empty rather than unavailable.
pub const fn iter<T: Table>(&self) -> ProviderResult<RocksDBIter<'_, T>> {
Ok(RocksDBIter { _marker: std::marker::PhantomData })
}
/// Check consistency of `RocksDB` tables (stub implementation).
///
/// Returns `None` since there is no `RocksDB` data to check when the feature is disabled.
pub const fn check_consistency<Provider>(
&self,
_provider: &Provider,
) -> ProviderResult<Option<alloy_primitives::BlockNumber>> {
Ok(None)
}
}
/// A stub batch writer for `RocksDB` on non-Unix platforms.
@@ -102,6 +137,25 @@ impl RocksDBBatch {
pub fn delete<T: Table>(&self, _key: T::Key) -> ProviderResult<()> {
Err(UnsupportedProvider)
}
/// Commits the batch (stub implementation).
pub const fn commit(self) -> ProviderResult<()> {
Err(UnsupportedProvider)
}
}
/// A stub iterator for `RocksDB` (non-transactional).
#[derive(Debug)]
pub struct RocksDBIter<'a, T> {
_marker: std::marker::PhantomData<(&'a (), T)>,
}
impl<T: Table> Iterator for RocksDBIter<'_, T> {
type Item = ProviderResult<(T::Key, T::Value)>;
fn next(&mut self) -> Option<Self::Item> {
None
}
}
/// A stub builder for `RocksDB` on non-Unix platforms.
@@ -213,3 +267,11 @@ impl RocksTx {
pub struct RocksTxIter<'a, T> {
_marker: std::marker::PhantomData<(&'a (), T)>,
}
impl<T: Table> Iterator for RocksTxIter<'_, T> {
type Item = ProviderResult<(T::Key, T::Value)>;
fn next(&mut self) -> Option<Self::Item> {
None
}
}

View File

@@ -1,6 +1,6 @@
use crate::{
providers::state::macros::delegate_provider_impls, AccountReader, BlockHashReader,
ChangeSetReader, HashedPostStateProvider, ProviderError, StateProvider, StateRootProvider,
AccountReader, BlockHashReader, ChangeSetReader, HashedPostStateProvider, ProviderError,
StateProvider, StateRootProvider,
};
use alloy_eips::merge::EPOCH_SLOTS;
use alloy_primitives::{Address, BlockNumber, Bytes, StorageKey, StorageValue, B256};
@@ -397,7 +397,7 @@ impl<Provider: DBProvider + BlockNumReader> StateProofProvider
}
}
impl<Provider: Sync> HashedPostStateProvider for HistoricalStateProviderRef<'_, Provider> {
impl<Provider> HashedPostStateProvider for HistoricalStateProviderRef<'_, Provider> {
fn hashed_post_state(&self, bundle_state: &revm_database::BundleState) -> HashedPostState {
HashedPostState::from_bundle_state::<KeccakKeyHasher>(bundle_state.state())
}
@@ -494,7 +494,7 @@ impl<Provider: DBProvider + BlockNumReader> HistoricalStateProvider<Provider> {
}
// Delegates all provider impls to [HistoricalStateProviderRef]
delegate_provider_impls!(HistoricalStateProvider<Provider> where [Provider: DBProvider + BlockNumReader + BlockHashReader + ChangeSetReader]);
reth_storage_api::macros::delegate_provider_impls!(HistoricalStateProvider<Provider> where [Provider: DBProvider + BlockNumReader + BlockHashReader + ChangeSetReader]);
/// Lowest blocks at which different parts of the state are available.
/// They may be [Some] if pruning is enabled.

View File

@@ -1,6 +1,5 @@
use crate::{
providers::state::macros::delegate_provider_impls, AccountReader, BlockHashReader,
HashedPostStateProvider, StateProvider, StateRootProvider,
AccountReader, BlockHashReader, HashedPostStateProvider, StateProvider, StateRootProvider,
};
use alloy_primitives::{Address, BlockNumber, Bytes, StorageKey, StorageValue, B256};
use reth_db_api::{cursor::DbDupCursorRO, tables, transaction::DbTx};
@@ -58,7 +57,7 @@ impl<Provider: BlockHashReader> BlockHashReader for LatestStateProviderRef<'_, P
}
}
impl<Provider: DBProvider + Sync> StateRootProvider for LatestStateProviderRef<'_, Provider> {
impl<Provider: DBProvider> StateRootProvider for LatestStateProviderRef<'_, Provider> {
fn state_root(&self, hashed_state: HashedPostState) -> ProviderResult<B256> {
StateRoot::overlay_root(self.tx(), &hashed_state.into_sorted())
.map_err(|err| ProviderError::Database(err.into()))
@@ -89,7 +88,7 @@ impl<Provider: DBProvider + Sync> StateRootProvider for LatestStateProviderRef<'
}
}
impl<Provider: DBProvider + Sync> StorageRootProvider for LatestStateProviderRef<'_, Provider> {
impl<Provider: DBProvider> StorageRootProvider for LatestStateProviderRef<'_, Provider> {
fn storage_root(
&self,
address: Address,
@@ -120,7 +119,7 @@ impl<Provider: DBProvider + Sync> StorageRootProvider for LatestStateProviderRef
}
}
impl<Provider: DBProvider + Sync> StateProofProvider for LatestStateProviderRef<'_, Provider> {
impl<Provider: DBProvider> StateProofProvider for LatestStateProviderRef<'_, Provider> {
fn proof(
&self,
input: TrieInput,
@@ -147,7 +146,7 @@ impl<Provider: DBProvider + Sync> StateProofProvider for LatestStateProviderRef<
}
}
impl<Provider: DBProvider + Sync> HashedPostStateProvider for LatestStateProviderRef<'_, Provider> {
impl<Provider: DBProvider> HashedPostStateProvider for LatestStateProviderRef<'_, Provider> {
fn hashed_post_state(&self, bundle_state: &revm_database::BundleState) -> HashedPostState {
HashedPostState::from_bundle_state::<KeccakKeyHasher>(bundle_state.state())
}
@@ -199,7 +198,7 @@ impl<Provider: DBProvider> LatestStateProvider<Provider> {
}
// Delegates all provider impls to [LatestStateProviderRef]
delegate_provider_impls!(LatestStateProvider<Provider> where [Provider: DBProvider + BlockHashReader ]);
reth_storage_api::macros::delegate_provider_impls!(LatestStateProvider<Provider> where [Provider: DBProvider + BlockHashReader ]);
#[cfg(test)]
mod tests {

View File

@@ -1,5 +1,4 @@
//! [`StateProvider`](crate::StateProvider) implementations
pub(crate) mod historical;
pub(crate) mod latest;
pub(crate) mod macros;
pub(crate) mod overlay;

View File

@@ -1084,31 +1084,13 @@ impl<N: NodePrimitives> StaticFileProvider<N> {
}
}
let initial_highest_block = self.get_highest_static_file_block(segment);
debug!(target: "reth::providers::static_file", ?segment, ?initial_highest_block, "Initial highest block for segment");
// File consistency is broken if:
//
// * appending data was interrupted before a config commit, then data file will be
// truncated according to the config.
//
// * pruning data was interrupted before a config commit, then we have deleted data that
// we are expected to still have. We need to check the Database and unwind everything
// accordingly.
if self.access.is_read_only() {
debug!(target: "reth::providers::static_file", ?segment, "Checking segment consistency (read-only)");
self.check_segment_consistency(segment)?;
} else {
debug!(target: "reth::providers::static_file", ?segment, "Fetching latest writer which might heal any potential inconsistency");
// Fetching the writer will attempt to heal any file level inconsistency.
self.latest_writer(segment)?;
}
// Heal file-level inconsistencies and get before/after highest block
let (initial_highest_block, mut highest_block) = self.maybe_heal_segment(segment)?;
// Only applies to block-based static files. (Headers)
//
// The updated `highest_block` may have decreased if we healed from a pruning
// interruption.
let mut highest_block = self.get_highest_static_file_block(segment);
if initial_highest_block != highest_block {
info!(
target: "reth::providers::static_file",
@@ -1202,6 +1184,44 @@ impl<N: NodePrimitives> StaticFileProvider<N> {
Ok(unwind_target.map(PipelineTarget::Unwind))
}
/// Heals file-level (`NippyJar`) inconsistencies for all static file segments.
///
/// This should be called BEFORE any checks that depend on static file data (e.g., `RocksDB`
/// consistency checks that need transaction data), as it ensures files are in a consistent
/// state without pruning any data.
///
/// Unlike [`Self::check_consistency`], this method:
/// - Does NOT compare with database checkpoints
/// - Does NOT prune data
/// - Applies to ALL segments unconditionally
///
/// Returns an unwind target if file healing detected a decrease in the highest block
/// (indicating a pruning interruption that needs a database unwind).
pub fn check_file_consistency(&self) -> ProviderResult<Option<BlockNumber>> {
info!(target: "reth::cli", "Healing static file inconsistencies.");
let mut unwind_target: Option<BlockNumber> = None;
for segment in StaticFileSegment::iter() {
let (initial_highest_block, highest_block) = self.maybe_heal_segment(segment)?;
if initial_highest_block != highest_block {
// Healing decreased highest block - need unwind
info!(
target: "reth::providers::static_file",
?segment,
?initial_highest_block,
?highest_block,
"File healing changed highest block, unwind needed"
);
let target = highest_block.unwrap_or_default();
unwind_target = Some(unwind_target.map_or(target, |t| t.min(target)));
}
}
Ok(unwind_target)
}
/// Checks consistency of the latest static file segment and throws an error if at fault.
/// Read-only.
pub fn check_segment_consistency(&self, segment: StaticFileSegment) -> ProviderResult<()> {
@@ -1223,6 +1243,47 @@ impl<N: NodePrimitives> StaticFileProvider<N> {
Ok(())
}
/// Attempts to heal file-level (`NippyJar`) inconsistencies for a single static file segment.
///
/// Returns the highest block before and after healing, which can be used to detect
/// if healing from a pruning interruption decreased the highest block.
///
/// File consistency is broken if:
///
/// * appending data was interrupted before a config commit, then data file will be truncated
/// according to the config.
///
/// * pruning data was interrupted before a config commit, then we have deleted data that we are
/// expected to still have. We need to check the Database and unwind everything accordingly.
///
/// **Note:** In read-only mode, this will return an error if a consistency issue is detected,
/// since healing requires write access.
fn maybe_heal_segment(
&self,
segment: StaticFileSegment,
) -> ProviderResult<(Option<BlockNumber>, Option<BlockNumber>)> {
let initial_highest_block = self.get_highest_static_file_block(segment);
debug!(target: "reth::providers::static_file", ?segment, ?initial_highest_block, "Initial highest block for segment");
if self.access.is_read_only() {
// Read-only mode: cannot modify files, so just validate consistency and error if
// broken.
debug!(target: "reth::providers::static_file", ?segment, "Checking segment consistency (read-only)");
self.check_segment_consistency(segment)?;
} else {
// Writable mode: fetching the writer will automatically heal any file-level
// inconsistency by truncating data to match the last committed config.
debug!(target: "reth::providers::static_file", ?segment, "Fetching latest writer which might heal any potential inconsistency");
self.latest_writer(segment)?;
}
// The updated `highest_block` may have decreased if we healed from a pruning
// interruption.
let highest_block = self.get_highest_static_file_block(segment);
Ok((initial_highest_block, highest_block))
}
/// Check invariants for each corresponding table and static file segment:
///
/// * the corresponding database table should overlap or have continuity in their keys

View File

@@ -54,7 +54,6 @@ pub trait BlockReader:
+ TransactionsProvider
+ ReceiptProvider
+ Send
+ Sync
{
/// The block type this provider reads.
type Block: reth_primitives_traits::Block<
@@ -149,7 +148,7 @@ pub trait BlockReader:
fn block_by_transaction_id(&self, id: TxNumber) -> ProviderResult<Option<BlockNumber>>;
}
impl<T: BlockReader> BlockReader for Arc<T> {
impl<T: BlockReader + Send + Sync> BlockReader for Arc<T> {
type Block = T::Block;
fn find_block_by_hash(
@@ -210,7 +209,7 @@ impl<T: BlockReader> BlockReader for Arc<T> {
}
}
impl<T: BlockReader> BlockReader for &T {
impl<T: BlockReader + Send + Sync> BlockReader for &T {
type Block = T::Block;
fn find_block_by_hash(
@@ -382,7 +381,7 @@ pub trait BlockReaderIdExt: BlockReader + ReceiptProviderIdExt {
}
/// Functionality to read the last known chain blocks from the database.
pub trait ChainStateBlockReader: Send + Sync {
pub trait ChainStateBlockReader: Send {
/// Returns the last finalized block number.
///
/// If no finalized block has been written yet, this returns `None`.
@@ -394,7 +393,7 @@ pub trait ChainStateBlockReader: Send + Sync {
}
/// Functionality to write the last known chain blocks to the database.
pub trait ChainStateBlockWriter: Send + Sync {
pub trait ChainStateBlockWriter: Send {
/// Saves the given finalized block number in the DB.
fn save_finalized_block_number(&self, block_number: BlockNumber) -> ProviderResult<()>;

View File

@@ -4,8 +4,8 @@ use alloy_primitives::{BlockNumber, B256};
use reth_storage_errors::provider::ProviderResult;
/// Client trait for fetching block hashes by number.
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait BlockHashReader: Send + Sync {
#[auto_impl::auto_impl(&, Box, Arc)]
pub trait BlockHashReader {
/// Get the hash of the block with the given number. Returns `None` if no block with this number
/// exists.
fn block_hash(&self, number: BlockNumber) -> ProviderResult<Option<B256>>;

View File

@@ -9,7 +9,7 @@ use reth_storage_errors::provider::{ProviderError, ProviderResult};
///
/// This trait also supports fetching block hashes and block numbers from a [`BlockHashOrNumber`].
#[auto_impl::auto_impl(&, Arc)]
pub trait BlockNumReader: BlockHashReader + Send + Sync {
pub trait BlockNumReader: BlockHashReader + Send {
/// Returns the current info for the chain.
fn chain_info(&self) -> ProviderResult<ChainInfo>;

View File

@@ -6,7 +6,7 @@ use reth_storage_errors::provider::ProviderResult;
/// Client trait for fetching block body indices related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait BlockBodyIndicesProvider: Send + Sync {
pub trait BlockBodyIndicesProvider: Send {
/// Returns the block body indices with matching number from database.
///
/// Returns `None` if block is not found.

View File

@@ -9,7 +9,7 @@ use reth_trie_common::HashedPostStateSorted;
/// `BlockExecution` Writer
pub trait BlockExecutionWriter:
NodePrimitivesProvider<Primitives: NodePrimitives<Block = Self::Block>> + BlockWriter + Send + Sync
NodePrimitivesProvider<Primitives: NodePrimitives<Block = Self::Block>> + BlockWriter
{
/// Take all of the blocks above the provided number and their execution result
///
@@ -39,8 +39,8 @@ impl<T: BlockExecutionWriter> BlockExecutionWriter for &T {
}
/// Block Writer
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait BlockWriter: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait BlockWriter {
/// The body this writer can write.
type Block: Block;
/// The receipt type for [`ExecutionOutcome`].
@@ -53,7 +53,7 @@ pub trait BlockWriter: Send + Sync {
/// and transition in the block.
fn insert_block(
&self,
block: RecoveredBlock<Self::Block>,
block: &RecoveredBlock<Self::Block>,
) -> ProviderResult<StoredBlockBodyIndices>;
/// Appends a batch of block bodies extending the canonical chain. This is invoked during
@@ -63,7 +63,7 @@ pub trait BlockWriter: Send + Sync {
/// Bodies are passed as [`Option`]s, if body is `None` the corresponding block is empty.
fn append_block_bodies(
&self,
bodies: Vec<(BlockNumber, Option<<Self::Block as Block>::Body>)>,
bodies: Vec<(BlockNumber, Option<&<Self::Block as Block>::Body>)>,
) -> ProviderResult<()>;
/// Removes all blocks above the given block number from the database.

View File

@@ -28,7 +28,7 @@ pub trait BlockBodyWriter<Provider, Body: BlockBody> {
fn write_block_bodies(
&self,
provider: &Provider,
bodies: Vec<(BlockNumber, Option<Body>)>,
bodies: Vec<(BlockNumber, Option<&Body>)>,
) -> ProviderResult<()>;
/// Removes all block bodies above the given block number from the database.
@@ -102,7 +102,7 @@ where
fn write_block_bodies(
&self,
provider: &Provider,
bodies: Vec<(u64, Option<alloy_consensus::BlockBody<T, H>>)>,
bodies: Vec<(u64, Option<&alloy_consensus::BlockBody<T, H>>)>,
) -> ProviderResult<()> {
let mut ommers_cursor = provider.tx_ref().cursor_write::<tables::BlockOmmers<H>>()?;
let mut withdrawals_cursor =
@@ -113,11 +113,12 @@ where
// Write ommers if any
if !body.ommers.is_empty() {
ommers_cursor.append(block_number, &StoredBlockOmmers { ommers: body.ommers })?;
ommers_cursor
.append(block_number, &StoredBlockOmmers { ommers: body.ommers.clone() })?;
}
// Write withdrawals if any
if let Some(withdrawals) = body.withdrawals &&
if let Some(withdrawals) = body.withdrawals.clone() &&
!withdrawals.is_empty()
{
withdrawals_cursor.append(block_number, &StoredBlockWithdrawals { withdrawals })?;
@@ -212,7 +213,7 @@ where
fn write_block_bodies(
&self,
_provider: &Provider,
_bodies: Vec<(u64, Option<alloy_consensus::BlockBody<T, H>>)>,
_bodies: Vec<(u64, Option<&alloy_consensus::BlockBody<T, H>>)>,
) -> ProviderResult<()> {
// noop
Ok(())

View File

@@ -8,8 +8,8 @@ use reth_primitives_traits::{Account, StorageEntry};
use reth_storage_errors::provider::ProviderResult;
/// Hashing Writer
#[auto_impl(&, Arc, Box)]
pub trait HashingWriter: Send + Sync {
#[auto_impl(&, Box)]
pub trait HashingWriter: Send {
/// Unwind and clear account hashing.
///
/// # Returns

View File

@@ -10,7 +10,7 @@ pub type ProviderHeader<P> = <P as HeaderProvider>::Header;
/// Client trait for fetching `Header` related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait HeaderProvider: Send + Sync {
pub trait HeaderProvider: Send {
/// The header type this provider supports.
type Header: BlockHeader;

View File

@@ -3,7 +3,7 @@ use reth_primitives_traits::{BlockHeader, SealedHeader};
use reth_storage_errors::provider::ProviderResult;
/// Provider for getting the local tip header for sync gap calculation.
pub trait HeaderSyncGapProvider: Send + Sync {
pub trait HeaderSyncGapProvider: Send {
/// The header type.
type Header: BlockHeader;

View File

@@ -7,8 +7,8 @@ use reth_primitives_traits::StorageEntry;
use reth_storage_errors::provider::ProviderResult;
/// History Writer
#[auto_impl(&, Arc, Box)]
pub trait HistoryWriter: Send + Sync {
#[auto_impl(&, Box)]
pub trait HistoryWriter: Send {
/// Unwind and clear account history indices.
///
/// Returns number of changesets walked.

View File

@@ -103,3 +103,5 @@ pub use reth_db_api::models::StorageSettings;
mod full;
pub use full::*;
pub mod macros;

View File

@@ -1,9 +1,10 @@
//! Helper macros for implementing traits for various [`StateProvider`](crate::StateProvider)
//! Helper macros for implementing traits for various `StateProvider`
//! implementations
/// A macro that delegates trait implementations to the `as_ref` function of the type.
///
/// Used to implement provider traits.
#[macro_export]
macro_rules! delegate_impls_to_as_ref {
(for $target:ty => $($trait:ident $(where [$($generics:tt)*])? { $(fn $func:ident$(<$($generic_arg:ident: $generic_arg_ty:path),*>)?(&self, $($arg:ident: $argty:ty),*) -> $ret:path;)* })* ) => {
@@ -19,45 +20,46 @@ macro_rules! delegate_impls_to_as_ref {
};
}
pub(crate) use delegate_impls_to_as_ref;
pub use delegate_impls_to_as_ref;
/// Delegates the provider trait implementations to the `as_ref` function of the type:
///
/// [`AccountReader`](crate::AccountReader)
/// [`BlockHashReader`](crate::BlockHashReader)
/// [`StateProvider`](crate::StateProvider)
#[macro_export]
macro_rules! delegate_provider_impls {
($target:ty $(where [$($generics:tt)*])?) => {
$crate::providers::state::macros::delegate_impls_to_as_ref!(
$crate::macros::delegate_impls_to_as_ref!(
for $target =>
AccountReader $(where [$($generics)*])? {
fn basic_account(&self, address: &alloy_primitives::Address) -> reth_storage_errors::provider::ProviderResult<Option<reth_primitives_traits::Account>>;
fn basic_account(&self, address: &alloy_primitives::Address) -> reth_storage_api::errors::provider::ProviderResult<Option<reth_primitives_traits::Account>>;
}
BlockHashReader $(where [$($generics)*])? {
fn block_hash(&self, number: u64) -> reth_storage_errors::provider::ProviderResult<Option<alloy_primitives::B256>>;
fn canonical_hashes_range(&self, start: alloy_primitives::BlockNumber, end: alloy_primitives::BlockNumber) -> reth_storage_errors::provider::ProviderResult<Vec<alloy_primitives::B256>>;
fn block_hash(&self, number: u64) -> reth_storage_api::errors::provider::ProviderResult<Option<alloy_primitives::B256>>;
fn canonical_hashes_range(&self, start: alloy_primitives::BlockNumber, end: alloy_primitives::BlockNumber) -> reth_storage_api::errors::provider::ProviderResult<Vec<alloy_primitives::B256>>;
}
StateProvider $(where [$($generics)*])? {
fn storage(&self, account: alloy_primitives::Address, storage_key: alloy_primitives::StorageKey) -> reth_storage_errors::provider::ProviderResult<Option<alloy_primitives::StorageValue>>;
fn storage(&self, account: alloy_primitives::Address, storage_key: alloy_primitives::StorageKey) -> reth_storage_api::errors::provider::ProviderResult<Option<alloy_primitives::StorageValue>>;
}
BytecodeReader $(where [$($generics)*])? {
fn bytecode_by_hash(&self, code_hash: &alloy_primitives::B256) -> reth_storage_errors::provider::ProviderResult<Option<reth_primitives_traits::Bytecode>>;
fn bytecode_by_hash(&self, code_hash: &alloy_primitives::B256) -> reth_storage_api::errors::provider::ProviderResult<Option<reth_primitives_traits::Bytecode>>;
}
StateRootProvider $(where [$($generics)*])? {
fn state_root(&self, state: reth_trie::HashedPostState) -> reth_storage_errors::provider::ProviderResult<alloy_primitives::B256>;
fn state_root_from_nodes(&self, input: reth_trie::TrieInput) -> reth_storage_errors::provider::ProviderResult<alloy_primitives::B256>;
fn state_root_with_updates(&self, state: reth_trie::HashedPostState) -> reth_storage_errors::provider::ProviderResult<(alloy_primitives::B256, reth_trie::updates::TrieUpdates)>;
fn state_root_from_nodes_with_updates(&self, input: reth_trie::TrieInput) -> reth_storage_errors::provider::ProviderResult<(alloy_primitives::B256, reth_trie::updates::TrieUpdates)>;
fn state_root(&self, state: reth_trie::HashedPostState) -> reth_storage_api::errors::provider::ProviderResult<alloy_primitives::B256>;
fn state_root_from_nodes(&self, input: reth_trie::TrieInput) -> reth_storage_api::errors::provider::ProviderResult<alloy_primitives::B256>;
fn state_root_with_updates(&self, state: reth_trie::HashedPostState) -> reth_storage_api::errors::provider::ProviderResult<(alloy_primitives::B256, reth_trie::updates::TrieUpdates)>;
fn state_root_from_nodes_with_updates(&self, input: reth_trie::TrieInput) -> reth_storage_api::errors::provider::ProviderResult<(alloy_primitives::B256, reth_trie::updates::TrieUpdates)>;
}
StorageRootProvider $(where [$($generics)*])? {
fn storage_root(&self, address: alloy_primitives::Address, storage: reth_trie::HashedStorage) -> reth_storage_errors::provider::ProviderResult<alloy_primitives::B256>;
fn storage_proof(&self, address: alloy_primitives::Address, slot: alloy_primitives::B256, storage: reth_trie::HashedStorage) -> reth_storage_errors::provider::ProviderResult<reth_trie::StorageProof>;
fn storage_multiproof(&self, address: alloy_primitives::Address, slots: &[alloy_primitives::B256], storage: reth_trie::HashedStorage) -> reth_storage_errors::provider::ProviderResult<reth_trie::StorageMultiProof>;
fn storage_root(&self, address: alloy_primitives::Address, storage: reth_trie::HashedStorage) -> reth_storage_api::errors::provider::ProviderResult<alloy_primitives::B256>;
fn storage_proof(&self, address: alloy_primitives::Address, slot: alloy_primitives::B256, storage: reth_trie::HashedStorage) -> reth_storage_api::errors::provider::ProviderResult<reth_trie::StorageProof>;
fn storage_multiproof(&self, address: alloy_primitives::Address, slots: &[alloy_primitives::B256], storage: reth_trie::HashedStorage) -> reth_storage_api::errors::provider::ProviderResult<reth_trie::StorageMultiProof>;
}
StateProofProvider $(where [$($generics)*])? {
fn proof(&self, input: reth_trie::TrieInput, address: alloy_primitives::Address, slots: &[alloy_primitives::B256]) -> reth_storage_errors::provider::ProviderResult<reth_trie::AccountProof>;
fn multiproof(&self, input: reth_trie::TrieInput, targets: reth_trie::MultiProofTargets) -> reth_storage_errors::provider::ProviderResult<reth_trie::MultiProof>;
fn witness(&self, input: reth_trie::TrieInput, target: reth_trie::HashedPostState) -> reth_storage_errors::provider::ProviderResult<Vec<alloy_primitives::Bytes>>;
fn proof(&self, input: reth_trie::TrieInput, address: alloy_primitives::Address, slots: &[alloy_primitives::B256]) -> reth_storage_api::errors::provider::ProviderResult<reth_trie::AccountProof>;
fn multiproof(&self, input: reth_trie::TrieInput, targets: reth_trie::MultiProofTargets) -> reth_storage_api::errors::provider::ProviderResult<reth_trie::MultiProof>;
fn witness(&self, input: reth_trie::TrieInput, target: reth_trie::HashedPostState) -> reth_storage_api::errors::provider::ProviderResult<Vec<alloy_primitives::Bytes>>;
}
HashedPostStateProvider $(where [$($generics)*])? {
fn hashed_post_state(&self, bundle_state: &revm_database::BundleState) -> reth_trie::HashedPostState;
@@ -66,4 +68,4 @@ macro_rules! delegate_provider_impls {
}
}
pub(crate) use delegate_provider_impls;
pub use delegate_provider_impls;

View File

@@ -10,8 +10,8 @@ pub mod keys {
}
/// Client trait for reading node metadata from the database.
#[auto_impl::auto_impl(&, Arc)]
pub trait MetadataProvider: Send + Sync {
#[auto_impl::auto_impl(&)]
pub trait MetadataProvider: Send {
/// Get a metadata value by key
fn get_metadata(&self, key: &str) -> ProviderResult<Option<Vec<u8>>>;
@@ -24,7 +24,7 @@ pub trait MetadataProvider: Send + Sync {
}
/// Client trait for writing node metadata to the database.
pub trait MetadataWriter: Send + Sync {
pub trait MetadataWriter: Send {
/// Write a metadata value
fn write_metadata(&self, key: &str, value: Vec<u8>) -> ProviderResult<()>;
@@ -41,7 +41,7 @@ pub trait MetadataWriter: Send + Sync {
}
/// Trait for caching storage settings on a provider factory.
pub trait StorageSettingsCache: Send + Sync {
pub trait StorageSettingsCache: Send {
/// Gets the cached storage settings.
fn cached_storage_settings(&self) -> StorageSettings;

View File

@@ -3,8 +3,8 @@ use reth_prune_types::{PruneCheckpoint, PruneSegment};
use reth_storage_errors::provider::ProviderResult;
/// The trait for fetching prune checkpoint related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait PruneCheckpointReader: Send + Sync {
#[auto_impl::auto_impl(&)]
pub trait PruneCheckpointReader: Send {
/// Fetch the prune checkpoint for the given segment.
fn get_prune_checkpoint(
&self,
@@ -16,8 +16,8 @@ pub trait PruneCheckpointReader: Send + Sync {
}
/// The trait for updating prune checkpoint related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait PruneCheckpointWriter: Send + Sync {
#[auto_impl::auto_impl(&)]
pub trait PruneCheckpointWriter {
/// Save prune checkpoint.
fn save_prune_checkpoint(
&self,

View File

@@ -11,7 +11,7 @@ pub type ProviderReceipt<P> = <P as ReceiptProvider>::Receipt;
/// Client trait for fetching receipt data.
#[auto_impl::auto_impl(&, Arc)]
pub trait ReceiptProvider: Send + Sync {
pub trait ReceiptProvider {
/// The receipt type.
type Receipt: Receipt;

View File

@@ -4,8 +4,8 @@ use reth_stages_types::{StageCheckpoint, StageId};
use reth_storage_errors::provider::ProviderResult;
/// The trait for fetching stage checkpoint related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait StageCheckpointReader: Send + Sync {
#[auto_impl::auto_impl(&)]
pub trait StageCheckpointReader: Send {
/// Fetch the checkpoint for the given stage.
fn get_stage_checkpoint(&self, id: StageId) -> ProviderResult<Option<StageCheckpoint>>;
@@ -18,8 +18,8 @@ pub trait StageCheckpointReader: Send + Sync {
}
/// The trait for updating stage checkpoint related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait StageCheckpointWriter: Send + Sync {
#[auto_impl::auto_impl(&)]
pub trait StageCheckpointWriter {
/// Save stage checkpoint.
fn save_stage_checkpoint(&self, id: StageId, checkpoint: StageCheckpoint)
-> ProviderResult<()>;

View File

@@ -14,8 +14,8 @@ use reth_trie_common::HashedPostState;
use revm_database::BundleState;
/// This just receives state, or [`ExecutionOutcome`], from the provider
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait StateReader: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait StateReader: Send {
/// Receipt type in [`ExecutionOutcome`].
type Receipt: Send + Sync;
@@ -27,10 +27,10 @@ pub trait StateReader: Send + Sync {
}
/// Type alias of boxed [`StateProvider`].
pub type StateProviderBox = Box<dyn StateProvider>;
pub type StateProviderBox = Box<dyn StateProvider + Send + 'static>;
/// An abstraction for a type that provides state data.
#[auto_impl(&, Arc, Box)]
#[auto_impl(&, Box)]
pub trait StateProvider:
BlockHashReader
+ AccountReader
@@ -39,8 +39,6 @@ pub trait StateProvider:
+ StorageRootProvider
+ StateProofProvider
+ HashedPostStateProvider
+ Send
+ Sync
{
/// Get storage of given account.
fn storage(
@@ -97,15 +95,15 @@ pub trait AccountInfoReader: AccountReader + BytecodeReader {}
impl<T: AccountReader + BytecodeReader> AccountInfoReader for T {}
/// Trait that provides the hashed state from various sources.
#[auto_impl(&, Arc, Box)]
pub trait HashedPostStateProvider: Send + Sync {
#[auto_impl(&, Box)]
pub trait HashedPostStateProvider {
/// Returns the `HashedPostState` of the provided [`BundleState`].
fn hashed_post_state(&self, bundle_state: &BundleState) -> HashedPostState;
}
/// Trait for reading bytecode associated with a given code hash.
#[auto_impl(&, Arc, Box)]
pub trait BytecodeReader: Send + Sync {
#[auto_impl(&, Box)]
pub trait BytecodeReader {
/// Get account code by its hash
fn bytecode_by_hash(&self, code_hash: &B256) -> ProviderResult<Option<Bytecode>>;
}
@@ -142,8 +140,8 @@ pub trait TryIntoHistoricalStateProvider {
/// This affects tracing, or replaying blocks, which will need to be executed on top of the state of
/// the parent block. For example, in order to trace block `n`, the state after block `n - 1` needs
/// to be used, since block `n` was executed on its parent block's state.
#[auto_impl(&, Arc, Box)]
pub trait StateProviderFactory: BlockIdReader + Send + Sync {
#[auto_impl(&, Box, Arc)]
pub trait StateProviderFactory: BlockIdReader + Send {
/// Storage provider for latest block.
fn latest(&self) -> ProviderResult<StateProviderBox>;

View File

@@ -1,8 +1,8 @@
use reth_db_api::table::Table;
/// The trait for fetching provider statistics.
#[auto_impl::auto_impl(&, Arc)]
pub trait StatsReader: Send + Sync {
#[auto_impl::auto_impl(&)]
pub trait StatsReader {
/// Fetch the number of entries in the corresponding [Table]. Depending on the provider, it may
/// route to different data sources other than [Table].
fn count_entries<T: Table>(&self) -> reth_storage_errors::provider::ProviderResult<usize>;

View File

@@ -8,8 +8,8 @@ use reth_primitives_traits::StorageEntry;
use reth_storage_errors::provider::ProviderResult;
/// Storage reader
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait StorageReader: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait StorageReader: Send {
/// Get plainstate storages for addresses and storage keys.
fn plain_state_storages(
&self,
@@ -34,8 +34,8 @@ pub trait StorageReader: Send + Sync {
/// Storage `ChangeSet` reader
#[cfg(feature = "db-api")]
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait StorageChangeSetReader: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait StorageChangeSetReader: Send {
/// Iterate over storage changesets and return the storage state from before this block.
fn storage_changeset(
&self,

View File

@@ -22,7 +22,7 @@ pub enum TransactionVariant {
/// Client trait for fetching transactions related data.
#[auto_impl::auto_impl(&, Arc)]
pub trait TransactionsProvider: BlockNumReader + Send + Sync {
pub trait TransactionsProvider: BlockNumReader + Send {
/// The transaction type this provider reads.
type Transaction: Send + Sync + SignedTransaction;

View File

@@ -9,7 +9,7 @@ use reth_trie_common::{
/// A type that can compute the state root of a given post state.
#[auto_impl::auto_impl(&, Box, Arc)]
pub trait StateRootProvider: Send + Sync {
pub trait StateRootProvider {
/// Returns the state root of the `BundleState` on top of the current state.
///
/// # Note
@@ -40,8 +40,8 @@ pub trait StateRootProvider: Send + Sync {
}
/// A type that can compute the storage root for a given account.
#[auto_impl::auto_impl(&, Box, Arc)]
pub trait StorageRootProvider: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait StorageRootProvider {
/// Returns the storage root of the `HashedStorage` for target address on top of the current
/// state.
fn storage_root(&self, address: Address, hashed_storage: HashedStorage)
@@ -66,8 +66,8 @@ pub trait StorageRootProvider: Send + Sync {
}
/// A type that can generate state proof on top of a given post state.
#[auto_impl::auto_impl(&, Box, Arc)]
pub trait StateProofProvider: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait StateProofProvider {
/// Get account and storage proofs of target keys in the `HashedPostState`
/// on top of the current state.
fn proof(
@@ -90,8 +90,8 @@ pub trait StateProofProvider: Send + Sync {
}
/// Trie Reader
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait TrieReader: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait TrieReader: Send {
/// Returns the [`TrieUpdatesSorted`] for reverting the trie database to its state prior to the
/// given block and onwards having been processed.
fn trie_reverts(&self, from: BlockNumber) -> ProviderResult<TrieUpdatesSorted>;
@@ -104,8 +104,8 @@ pub trait TrieReader: Send + Sync {
}
/// Trie Writer
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait TrieWriter: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait TrieWriter: Send {
/// Writes trie updates to the database.
///
/// Returns the number of entries modified.
@@ -146,8 +146,8 @@ pub trait TrieWriter: Send + Sync {
}
/// Storage Trie Writer
#[auto_impl::auto_impl(&, Arc, Box)]
pub trait StorageTrieWriter: Send + Sync {
#[auto_impl::auto_impl(&, Box)]
pub trait StorageTrieWriter: Send {
/// Writes storage trie updates from the given storage trie map with already sorted updates.
///
/// Expects the storage trie updates to already be sorted by the hashed address key.

Some files were not shown because too many files have changed in this diff Show More