Compare commits

..

41 Commits

Author SHA1 Message Date
Brian Picciano
c07e228412 fix(provider): clamp partial trie unwind during reorgs
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dd7d9-2cea-745d-a1ea-e8965237cf20
Co-authored-by: Amp <amp@ampcode.com>
2026-04-29 07:14:10 +00:00
Brian Picciano
4b4a1b80d8 fix(trie): mask storage entries in disjoint merges
Only drop an entire storage entry when the masking batch wipes or deletes it.
Otherwise, filter overlapping storage slots and storage trie nodes individually to preserve the rest of the account state.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dd592-8e45-756e-9e7c-7c9c98f8687c
Co-authored-by: Amp <amp@ampcode.com>
2026-04-28 19:41:52 +00:00
Brian Picciano
5ab335d04e refactor(provider): drive save_blocks from plan steps
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dd4f6-c7f3-7649-884e-55217d141526
Co-authored-by: Amp <amp@ampcode.com>
2026-04-28 16:57:41 +00:00
Brian Picciano
baf6ef9778 Revert "fix(provider): preserve masked persistence frontier state"
This reverts commit e71cf3040b.
2026-04-28 16:55:26 +00:00
Brian Picciano
e71cf3040b fix(provider): preserve masked persistence frontier state
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dd4f6-c7f3-7649-884e-55217d141526
Co-authored-by: Amp <amp@ampcode.com>
2026-04-28 16:50:08 +00:00
Brian Picciano
adf2930e84 refactor(engine): split partial persistence frontiers
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dd442-503a-7710-90ee-60264449117e
Co-authored-by: Amp <amp@ampcode.com>
2026-04-28 14:25:06 +00:00
Brian Picciano
d9d3f69557 fix(engine): wire deferred trie persistence config
Expose --engine.deferred-trie-blocks and make threshold-driven persistence advance the full persisted region before leaving the deferred trie tail. This keeps the trie and non-trie frontiers aligned with the configured in-memory buffer and deferred-trie window.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dcf65-4035-724f-b1ce-ebd1250af9f1
Co-authored-by: Amp <amp@ampcode.com>
2026-04-27 15:29:10 +00:00
Brian Picciano
8248aa29d1 chore: merge origin/main
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dcefd-7813-7693-9eca-9c8ad3da5c5b
Co-authored-by: Amp <amp@ampcode.com>
2026-04-27 14:25:47 +00:00
Brian Picciano
0d19b17bf3 fix(provider): preserve partial trie frontier across overlay and unwind
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dcefd-7813-7693-9eca-9c8ad3da5c5b
Co-authored-by: Amp <amp@ampcode.com>
2026-04-27 14:03:31 +00:00
Brian Picciano
5d4019049a fix(provider): stop masking trie writes with in-memory suffix
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dce00-c0f1-7665-a244-00f02292fc3c
Co-authored-by: Amp <amp@ampcode.com>
2026-04-27 08:52:28 +00:00
Brian Picciano
743d42ff6d fix(provider): anchor overlay state providers to trie frontier
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dcdca-2c60-711c-b1b8-7a6001a950fd
Co-authored-by: Amp <amp@ampcode.com>
2026-04-27 08:04:06 +00:00
Brian Picciano
843b5a826a merge(engine): bring lazy overlay refactor into partial persistence
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dcdca-2c60-711c-b1b8-7a6001a950fd
Co-authored-by: Amp <amp@ampcode.com>
2026-04-27 08:03:56 +00:00
Brian Picciano
b6eec2e684 refactor(provider): require overlay builder anchor hash
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dbf08-3c9d-736b-9b47-3070f2bf2a54
Co-authored-by: Amp <amp@ampcode.com>
2026-04-24 15:28:57 +00:00
Brian Picciano
31d0c7852d fix(ci): clean bench checkouts and lock cargo builds
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dbee7-f576-769d-923c-dfdfe0a40858
Co-authored-by: Amp <amp@ampcode.com>
2026-04-24 10:06:10 +00:00
Brian Picciano
b60758ef73 fix(trie): remove unused parallel test dependency
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dbc30-68d0-76b8-a32c-e1173122ca48
Co-authored-by: Amp <amp@ampcode.com>
2026-04-24 09:32:47 +00:00
Brian Picciano
6e8dbe34a4 Merge branch 'main' into mediocregopher/lazyoverlay-refactor
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dbc30-68d0-76b8-a32c-e1173122ca48
Co-authored-by: Amp <amp@ampcode.com>
2026-04-23 21:29:57 +00:00
Brian Picciano
c4d0949a23 style(engine): format sparse trie overlay test
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db57a-7f08-7761-9a50-27a2a5c8f917
Co-authored-by: Amp <amp@ampcode.com>
2026-04-22 14:14:11 +00:00
Brian Picciano
d5169eda88 fix(engine): update sparse trie overlay factory test
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db57a-7f08-7761-9a50-27a2a5c8f917
Co-authored-by: Amp <amp@ampcode.com>
2026-04-22 14:11:03 +00:00
Brian Picciano
87b5240ec1 Merge branch 'main' into mediocregopher/lazyoverlay-refactor
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019db57a-7f08-7761-9a50-27a2a5c8f917
Co-authored-by: Amp <amp@ampcode.com>
2026-04-22 13:59:13 +00:00
Brian Picciano
ffb0587b19 fix(provider): satisfy overlay lint checks
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafbb-3571-73ef-ae67-43c242e2bf23
Co-authored-by: Amp <amp@ampcode.com>
2026-04-21 11:53:34 +00:00
Brian Picciano
45db5e0b5d fix(trie): initialize test overlay anchors
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafbb-3571-73ef-ae67-43c242e2bf23
Co-authored-by: Amp <amp@ampcode.com>
2026-04-21 11:36:34 +00:00
Brian Picciano
7db14d095d fix(engine): anchor state-root test overlay factory
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafbb-3571-73ef-ae67-43c242e2bf23
Co-authored-by: Amp <amp@ampcode.com>
2026-04-21 11:22:09 +00:00
Brian Picciano
134a7f364b fix(provider): pass overlay anchors via constructor
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
2026-04-21 11:08:27 +00:00
Brian Picciano
d92ad5aa34 fix(provider): anchor overlay state providers by hash
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dafa5-05ee-739e-a0b6-037cc30e4a0e
Co-authored-by: Amp <amp@ampcode.com>
2026-04-21 10:53:23 +00:00
Brian Picciano
b5ad0018a3 refactor(provider): thread explicit requested anchors
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 16:36:09 +00:00
Brian Picciano
5036eb59fb refactor(provider): infer overlay anchors from sources
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 16:23:57 +00:00
Brian Picciano
812e479b69 refactor(provider): separate overlay anchors from revert state
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019daba4-3598-758d-8771-cb5db784db81
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 16:10:42 +00:00
Brian Picciano
5041d55bc3 refactor(provider): resolve lazy overlay anchors at use time
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab6f-7cee-755e-9f9c-309ae0b8517c
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 15:46:58 +00:00
Brian Picciano
ebfaa6f4c5 refactor(chain-state): cache lazy overlays by anchor
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab6f-7cee-755e-9f9c-309ae0b8517c
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 15:17:35 +00:00
Brian Picciano
cacb69aca9 refactor(chain-state): address lazy overlay review feedback
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab40-7d65-709b-90d6-98965c5c6a65
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 15:01:54 +00:00
Brian Picciano
be4e8cd017 refactor(chain-state): derive lazy overlay anchor from blocks
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dab40-7d65-709b-90d6-98965c5c6a65
Co-authored-by: Amp <amp@ampcode.com>
2026-04-20 14:51:24 +00:00
Brian Picciano
c757a310e1 feat(engine): add dual-frontier persistence planning
Co-Authored-By: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
2026-04-17 12:04:12 +00:00
Brian Picciano
ca20cc13ef fix(engine): retain partial trie suffix after persistence
Return finish-stage partial trie progress from the persistence thread and keep blocks above that trie boundary resident in memory after persistence completes.

This preserves the old prune-through-tip behavior when the partial trie matches the persisted tip or is absent.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9ab4-1eab-713f-9d5d-71903b7bc724
Co-authored-by: Amp <amp@ampcode.com>
2026-04-17 10:02:17 +00:00
Brian Picciano
9e38dde3e0 fix(provider): persist partial trie finish checkpoint
Make masked save_blocks persistence track partial trie progress in the Finish checkpoint and switch the trie-masking API to a masked-suffix start index so the final block is always covered by the mask when partial trie persistence is used.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9a2b-716e-774d-8db7-b0308fa96a23
Co-authored-by: Amp <amp@ampcode.com>
2026-04-17 09:05:04 +00:00
Brian Picciano
ca352832b8 fix(engine): raise persistence defaults
Raise the default persistence threshold to 10 and derive the default backpressure threshold from the persistence and in-memory thresholds. Enable alloy getrandom for reth-engine-primitives tests so the existing B256::random() test coverage keeps compiling.

Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9a2b-716e-774d-8db7-b0308fa96a23
Co-authored-by: Amp <amp@ampcode.com>
2026-04-17 08:32:16 +00:00
Brian Picciano
84b520560b docs(provider): explain save_blocks trie masking range
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9728-846b-7418-8a69-ddbba65ce656
Co-authored-by: Amp <amp@ampcode.com>
2026-04-16 21:26:24 +00:00
Brian Picciano
e2bd518097 refactor(provider): simplify save_blocks trie masking API
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9728-846b-7418-8a69-ddbba65ce656
Co-authored-by: Amp <amp@ampcode.com>
2026-04-16 21:25:21 +00:00
Brian Picciano
761acad803 fix(node): unwind startup to partial trie checkpoint
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d968b-970d-735e-844c-b83ff245ce05
Co-authored-by: Amp <amp@ampcode.com>
2026-04-16 16:33:52 +00:00
Brian Picciano
b97544a05e feat(stages): move partial trie progress into finish checkpoint
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d968b-970d-735e-844c-b83ff245ce05
Co-authored-by: Amp <amp@ampcode.com>
2026-04-16 14:10:51 +00:00
Brian Picciano
037828f6aa feat(stages): add partial finish checkpoint field
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9639-935f-73a4-ba50-7682a7b9aca0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-16 12:55:43 +00:00
Brian Picciano
9883b7140e feat(trie): add disjoint_by_keys for sorted overlays
Co-authored-by: Brian Picciano <933154+mediocregopher@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d9639-935f-73a4-ba50-7682a7b9aca0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-16 12:38:41 +00:00
136 changed files with 3996 additions and 4192 deletions

View File

@@ -4,14 +4,10 @@ updates:
directory: "/"
schedule:
interval: "weekly"
cooldown:
default-days: 7
- package-ecosystem: "cargo"
directory: "/"
schedule:
interval: "weekly"
cooldown:
default-days: 7
labels:
- "A-dependencies"
commit-message:

View File

@@ -323,18 +323,13 @@ if [ "$BIG_BLOCKS" = "true" ]; then
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
else
# Standard mode: warmup + new-payload-fcu
WARMUP="${BENCH_WARMUP_BLOCKS:-50}"
if [ "$WARMUP" -gt 0 ] 2>/dev/null; then
# Warm up the node before measuring the benchmark window.
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
--rpc-url "$BENCH_RPC_URL" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "$WARMUP" \
"${EXTRA_BENCH_ARGS[@]}" 2>&1 | sed -u "s/^/[bench] /"
else
echo "Skipping warmup (0 blocks)..."
fi
# Warmup
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
--rpc-url "$BENCH_RPC_URL" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "${BENCH_WARMUP_BLOCKS:-50}" \
"${EXTRA_BENCH_ARGS[@]}" 2>&1 | sed -u "s/^/[bench] /"
# Start tracy-capture after warmup so profile only covers the benchmark
if [ "${BENCH_TRACY:-off}" != "off" ]; then

View File

@@ -5,8 +5,8 @@ fixture_variant="${1:-osaka}"
case "${fixture_variant}" in
amsterdam)
eels_fixtures="https://github.com/ethereum/execution-spec-tests/releases/download/snobal-devnet-5@v8037.0.0/fixtures_snobal-devnet-5.tar.gz"
eels_branch="devnets/snobal/5"
eels_fixtures="https://github.com/ethereum/execution-spec-tests/releases/download/bal@v5.7.0/fixtures_bal.tar.gz"
eels_branch="devnets/bal/4"
;;
osaka)
eels_fixtures="https://github.com/ethereum/execution-spec-tests/releases/download/v5.3.0/fixtures_develop.tar.gz"

View File

@@ -13,15 +13,15 @@ rpc-compat:
# syncing mode, the test expects syncing to be false on start
- eth_syncing/check-syncing (reth)
engine-withdrawals: []
engine-withdrawals: [ ]
engine-api: []
engine-api: [ ]
engine-cancun: []
engine-cancun: [ ]
sync: []
sync: [ ]
engine-auth: []
engine-auth: [ ]
# EIP-7610 related tests (Revert creation in case of non-empty storage):
#
@@ -99,40 +99,6 @@ eels/consume-engine:
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_2-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Shanghai-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE2-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Amsterdam-blockchain_test_engine_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE2-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_engine_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_engine_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_engine_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Prague-blockchain_test_engine_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Shanghai-blockchain_test_engine_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Paris-blockchain_test_engine_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_engine_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Cancun-blockchain_test_engine_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_engine_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_engine_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_engine_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_engine_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Osaka-blockchain_test_engine_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_engine_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_engine_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_engine_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_engine_from_state_test-empty-initcode]-reth
# Blob limit tests:
#
@@ -227,37 +193,3 @@ eels/consume-rlp:
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_1-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_2-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Shanghai-tx_type_0-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Amsterdam-blockchain_test_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE2-non-empty-balance-correct-initcode]-reth
- tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE2-non-empty-balance-revert-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Prague-blockchain_test_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Shanghai-blockchain_test_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Paris-blockchain_test_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Cancun-blockchain_test_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_from_state_test-initcode-with-deploy]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Osaka-blockchain_test_from_state_test]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_from_state_test-sstore-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_from_state_test-empty-initcode]-reth
- tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_from_state_test-initcode-with-deploy]-reth

View File

@@ -54,7 +54,9 @@ env:
name: bench-scheduled
permissions: {}
permissions:
contents: read
actions: read
jobs:
# ---------------------------------------------------------------------------
@@ -63,9 +65,6 @@ jobs:
resolve-refs:
name: resolve-refs
runs-on: ubuntu-latest
permissions:
contents: read
actions: read
outputs:
mode: ${{ steps.mode.outputs.mode }}
baseline-ref: ${{ steps.refs.outputs.baseline-ref }}
@@ -77,26 +76,21 @@ jobs:
long-running: ${{ steps.refs.outputs.long-running }}
release-tag: ${{ steps.refs.outputs.release-tag }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
sparse-checkout: .github/scripts
sparse-checkout-cone-mode: true
fetch-depth: 2
- name: Detect mode
id: mode
env:
EVENT_NAME: ${{ github.event_name }}
INPUT_MODE: ${{ inputs.mode }}
SCHEDULE: ${{ github.event.schedule }}
run: |
# Maps cron schedules to modes (must match the schedule entries above)
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
MODE="${INPUT_MODE:-nightly}"
elif [ "$SCHEDULE" = "30 5 * * *" ]; then
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
MODE="${{ inputs.mode || 'nightly' }}"
elif [ "${{ github.event.schedule }}" = "30 5 * * *" ]; then
MODE="nightly"
elif [ "$SCHEDULE" = "0 9 * * *" ]; then
elif [ "${{ github.event.schedule }}" = "0 9 * * *" ]; then
MODE="release"
else
MODE="hourly"
@@ -111,15 +105,14 @@ jobs:
DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_RUN_ID: ${{ github.run_id }}
INPUT_FORCE: ${{ inputs.force || 'false' }}
run: |
FORCE="${INPUT_FORCE:-false}"
FORCE="${{ inputs.force || 'false' }}"
MODE="${{ steps.mode.outputs.mode }}"
.github/scripts/bench-scheduled-refs.sh "$FORCE" "$MODE"
- name: Alert on long-running hourly
if: steps.mode.outputs.mode == 'hourly' && steps.refs.outputs.long-running == 'true' && !(github.event_name == 'workflow_dispatch' && inputs.slack == 'never')
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
@@ -161,7 +154,7 @@ jobs:
- name: Alert on stale nightly
if: steps.mode.outputs.mode == 'nightly' && steps.refs.outputs.is-stale == 'true' && !(github.event_name == 'workflow_dispatch' && inputs.slack == 'never')
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
@@ -249,9 +242,6 @@ jobs:
needs.resolve-refs.outputs.is-stale != 'true'
name: bench-scheduled
runs-on: [self-hosted, Linux, X64, available]
permissions:
contents: read
actions: read
timeout-minutes: 120
env:
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
@@ -280,16 +270,15 @@ jobs:
- name: Clean up previous bench-work
run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
submodules: true
fetch-depth: 0
ref: ${{ needs.resolve-refs.outputs.feature-ref }}
- name: Resolve job URL
id: job-url
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
const { data: jobs } = await github.rest.actions.listJobsForWorkflowRun({
@@ -302,9 +291,8 @@ jobs:
core.exportVariable('BENCH_JOB_URL', jobUrl);
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: mozilla-actions/sccache-action@v0.0.9
continue-on-error: true
- uses: astral-sh/setup-uv@bd01e18f51369d5a26f1651c3cb451d3417e3bba # v6.3.1
- name: Install dependencies
env:
@@ -640,7 +628,7 @@ jobs:
- name: Upload results
if: "!cancelled()"
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: bench-scheduled-results
path: ${{ env.BENCH_WORK_DIR }}
@@ -648,12 +636,10 @@ jobs:
- name: Push charts
id: push-charts
if: success() && env.BENCH_MODE != 'hourly'
env:
DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
RUN_ID: ${{ github.run_id }}
run: |
RUN_ID=${{ github.run_id }}
CHART_DIR="${BENCH_MODE}/${RUN_ID}"
CHARTS_REPO="https://x-access-token:${DEREK_TOKEN}@github.com/decofe/reth-bench-charts.git"
CHARTS_REPO="https://x-access-token:${{ secrets.DEREK_TOKEN }}@github.com/decofe/reth-bench-charts.git"
TMP_DIR=$(mktemp -d)
if git clone --depth 1 "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
@@ -674,7 +660,7 @@ jobs:
- name: Write job summary
if: success()
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
const fs = require('fs');
@@ -753,7 +739,7 @@ jobs:
- name: Send Slack notification (success)
if: success() && (env.BENCH_SLACK == 'always' || env.BENCH_SLACK == 'on-win')
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
@@ -908,7 +894,7 @@ jobs:
- name: Send Slack notification (failure)
if: failure() && env.BENCH_SLACK != 'never' && env.BENCH_SLACK != 'on-win'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}

View File

@@ -82,7 +82,7 @@ on:
- on-error
- never
abba:
description: "Run ABBA (FBBF) interleaved order; false = single FB pass"
description: "Run ABBA (BFFB) interleaved order; false = single AB pass"
required: false
default: "true"
type: boolean
@@ -99,7 +99,9 @@ env:
name: bench
permissions: {}
permissions:
contents: read
pull-requests: write
jobs:
reth-bench-ack:
@@ -108,9 +110,6 @@ jobs:
github.event_name == 'workflow_dispatch'
name: reth-bench-ack
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
outputs:
pr: ${{ steps.args.outputs.pr }}
actor: ${{ steps.args.outputs.actor }}
@@ -134,7 +133,7 @@ jobs:
steps:
- name: Check org membership
if: github.event_name == 'issue_comment'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
@@ -153,7 +152,7 @@ jobs:
- name: Parse arguments
id: args
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
@@ -360,7 +359,7 @@ jobs:
- name: Acknowledge request
id: ack
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
@@ -446,7 +445,7 @@ jobs:
- name: Poll queue position
if: steps.ack.outputs.comment-id && steps.ack.outputs.queue-position != '0'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
@@ -530,9 +529,6 @@ jobs:
needs: reth-bench-ack
name: reth-bench
runs-on: [self-hosted, Linux, X64, available]
permissions:
contents: read
pull-requests: write
timeout-minutes: 120
env:
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
@@ -564,7 +560,7 @@ jobs:
- name: Resolve checkout ref
id: checkout-ref
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
if (!process.env.BENCH_PR) {
@@ -582,16 +578,15 @@ jobs:
core.info(`PR #${process.env.BENCH_PR} (${pr.state}), using head SHA ${pr.head.sha}`);
core.setOutput('ref', pr.head.sha);
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
submodules: true
fetch-depth: 0
ref: ${{ steps.checkout-ref.outputs.ref }}
- name: Resolve job URL and update status
if: env.BENCH_COMMENT_ID
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
@@ -639,9 +634,8 @@ jobs:
});
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: mozilla-actions/sccache-action@v0.0.9
continue-on-error: true
- uses: astral-sh/setup-uv@bd01e18f51369d5a26f1651c3cb451d3417e3bba # v6.3.1
- name: Install dependencies
env:
@@ -702,7 +696,7 @@ jobs:
# Build binaries
- name: Resolve PR head branch
id: pr-info
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
if (process.env.BENCH_PR) {
@@ -720,7 +714,7 @@ jobs:
- name: Resolve refs
id: refs
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
const { execSync } = require('child_process');
@@ -943,28 +937,15 @@ jobs:
- name: Update status (running benchmarks)
if: success() && env.BENCH_COMMENT_ID
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
const s = require('./.github/scripts/bench-update-status.js');
await s({github, context, status: 'Running benchmarks...'});
# Interleaved run order (F-B-B-F) to reduce systematic bias from
# Interleaved run order (B-F-F-B) to reduce systematic bias from
# thermal drift and cache warming.
- name: "Run benchmark: feature (1/2)"
id: run-feature-1
env:
FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-1,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
run: |
LAST_RUN_START=$(date +%s)
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh feature "../reth-feature/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/feature-1"
- name: "Run benchmark: baseline (1/2)"
id: run-baseline-1
env:
@@ -978,19 +959,18 @@ jobs:
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline "../reth-baseline/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/baseline-1"
- name: "Run benchmark: baseline (2/2)"
if: env.BENCH_ABBA != 'false'
id: run-baseline-2
- name: "Run benchmark: feature (1/2)"
id: run-feature-1
env:
BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-2,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-1,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
run: |
LAST_RUN_START=$(date +%s)
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
{"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline "../reth-baseline/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/baseline-2"
taskset -c 0 .github/scripts/bench-reth-run.sh feature "../reth-feature/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/feature-1"
- name: "Run benchmark: feature (2/2)"
if: env.BENCH_ABBA != 'false'
@@ -1006,6 +986,20 @@ jobs:
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh feature "../reth-feature/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/feature-2"
- name: "Run benchmark: baseline (2/2)"
if: env.BENCH_ABBA != 'false'
id: run-baseline-2
env:
BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-2,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
run: |
LAST_RUN_START=$(date +%s)
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline "../reth-baseline/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/baseline-2"
- name: Stop metrics proxy & generate Grafana URL
id: metrics
if: "!cancelled()"
@@ -1172,7 +1166,7 @@ jobs:
- name: Upload results
if: "!cancelled()"
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: bench-reth-results
path: ${{ env.BENCH_WORK_DIR }}
@@ -1180,13 +1174,11 @@ jobs:
- name: Push charts
id: push-charts
if: success()
env:
DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
RUN_ID: ${{ github.run_id }}
run: |
PR_NUMBER="${BENCH_PR:-0}"
RUN_ID=${{ github.run_id }}
CHART_DIR="pr/${PR_NUMBER}/${RUN_ID}"
CHARTS_REPO="https://x-access-token:${DEREK_TOKEN}@github.com/decofe/reth-bench-charts.git"
CHARTS_REPO="https://x-access-token:${{ secrets.DEREK_TOKEN }}@github.com/decofe/reth-bench-charts.git"
TMP_DIR=$(mktemp -d)
if git clone --depth 1 "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
@@ -1199,35 +1191,15 @@ jobs:
mkdir -p "${TMP_DIR}/${CHART_DIR}"
cp "$BENCH_WORK_DIR"/charts/*.png "${TMP_DIR}/${CHART_DIR}/"
git -C "${TMP_DIR}" add "${CHART_DIR}"
if git -C "${TMP_DIR}" diff --cached --quiet; then
echo "Charts for ${CHART_DIR} are already present, skipping push"
echo "sha=$(git -C "${TMP_DIR}" rev-parse HEAD)" >> "$GITHUB_OUTPUT"
rm -rf "${TMP_DIR}"
exit 0
fi
git -C "${TMP_DIR}" -c user.name="github-actions" -c user.email="github-actions@github.com" \
commit -m "bench charts for PR #${PR_NUMBER} run ${RUN_ID}"
for attempt in 1 2 3 4 5; do
if git -C "${TMP_DIR}" push origin HEAD:main; then
break
fi
if [ "$attempt" -eq 5 ]; then
echo "::error::Failed to push charts after ${attempt} attempts"
rm -rf "${TMP_DIR}"
exit 1
fi
sleep "$attempt"
git -C "${TMP_DIR}" fetch origin main
git -C "${TMP_DIR}" rebase origin/main
done
git -C "${TMP_DIR}" push origin HEAD:main
echo "sha=$(git -C "${TMP_DIR}" rev-parse HEAD)" >> "$GITHUB_OUTPUT"
rm -rf "${TMP_DIR}"
- name: Compare & comment
if: success() && env.BENCH_COMMENT_ID
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
@@ -1263,7 +1235,7 @@ jobs:
// Samply profile links (URLs point directly to Firefox Profiler)
if (process.env.BENCH_SAMPLY === 'true') {
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
const runs = abba ? ['feature-1', 'baseline-1', 'baseline-2', 'feature-2'] : ['feature-1', 'baseline-1'];
const runs = abba ? ['baseline-1', 'feature-1', 'feature-2', 'baseline-2'] : ['baseline-1', 'feature-1'];
const links = [];
for (const run of runs) {
try {
@@ -1307,7 +1279,7 @@ jobs:
- name: Write job summary
if: success()
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
const jobSummary = require('./.github/scripts/bench-job-summary.js');
@@ -1321,7 +1293,7 @@ jobs:
- name: Send Slack notification (success)
if: success() && (env.BENCH_SLACK == 'always' || env.BENCH_SLACK == 'on-win')
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
@@ -1332,7 +1304,7 @@ jobs:
- name: Update status (failed)
if: failure() && env.BENCH_COMMENT_ID
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
@@ -1342,10 +1314,10 @@ jobs:
...(bigBlocks ? [['validating local big-block data', '${{ steps.big-blocks-check.outcome }}']] : []),
['validating local snapshot', '${{ steps.snapshot-check.outcome }}'],
['building binaries', '${{ steps.build.outcome }}'],
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
];
const failed = steps_status.find(([, o]) => o === 'failure');
const failedStep = failed ? failed[0] : 'unknown step';
@@ -1368,7 +1340,7 @@ jobs:
- name: Send Slack notification (failure)
if: failure() && env.BENCH_SLACK != 'never' && env.BENCH_SLACK != 'on-win'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
@@ -1380,10 +1352,10 @@ jobs:
...(bigBlocks ? [['validating local big-block data', '${{ steps.big-blocks-check.outcome }}']] : []),
['validating local snapshot', '${{ steps.snapshot-check.outcome }}'],
['building binaries', '${{ steps.build.outcome }}'],
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
];
const failed = steps_status.find(([, o]) => o === 'failure');
const failedStep = failed ? failed[0] : 'unknown step';
@@ -1392,7 +1364,7 @@ jobs:
- name: Update status (cancelled)
if: cancelled() && env.BENCH_COMMENT_ID
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |

View File

@@ -10,22 +10,19 @@ on:
types: [opened, reopened, synchronize, closed]
merge_group:
permissions: {}
env:
RUSTC_WRAPPER: "sccache"
jobs:
build:
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 90
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
uses: actions/checkout@v6
- name: Install bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
uses: oven-sh/setup-bun@v2
with:
bun-version: v1.2.23
@@ -39,6 +36,8 @@ jobs:
- name: Install Rust nightly
uses: dtolnay/rust-toolchain@nightly
- uses: mozilla-actions/sccache-action@v0.0.9
- name: Build docs
run: cd docs/vocs && bash scripts/build-cargo-docs.sh
@@ -48,10 +47,10 @@ jobs:
echo "Vocs Build Complete"
- name: Setup Pages
uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
uses: actions/configure-pages@v6
- name: Upload artifact
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
uses: actions/upload-pages-artifact@v5
with:
path: "./docs/vocs/docs/dist"
@@ -75,4 +74,4 @@ jobs:
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0
uses: actions/deploy-pages@v5

View File

@@ -22,41 +22,31 @@ on:
env:
CARGO_TERM_COLOR: always
permissions: {}
jobs:
check:
name: Check compilation with patched alloy
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-16' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 60
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Apply alloy patches
env:
ALLOY_BRANCH: ${{ inputs.alloy_branch }}
ALLOY_EVM_BRANCH: ${{ inputs.alloy_evm_branch }}
OP_ALLOY_BRANCH: ${{ inputs.op_alloy_branch }}
run: |
ARGS=""
if [ -n "$ALLOY_BRANCH" ]; then
ARGS="$ARGS --alloy $ALLOY_BRANCH"
if [ -n "${{ inputs.alloy_branch }}" ]; then
ARGS="$ARGS --alloy ${{ inputs.alloy_branch }}"
fi
if [ -n "$ALLOY_EVM_BRANCH" ]; then
ARGS="$ARGS --evm $ALLOY_EVM_BRANCH"
if [ -n "${{ inputs.alloy_evm_branch }}" ]; then
ARGS="$ARGS --evm ${{ inputs.alloy_evm_branch }}"
fi
if [ -n "$OP_ALLOY_BRANCH" ]; then
ARGS="$ARGS --op $OP_ALLOY_BRANCH"
if [ -n "${{ inputs.op_alloy_branch }}" ]; then
ARGS="$ARGS --op ${{ inputs.op_alloy_branch }}"
fi
if [ -z "$ARGS" ]; then

View File

@@ -16,38 +16,32 @@ env:
RUSTC_WRAPPER: "sccache"
name: compact-codec
permissions: {}
jobs:
compact-codec:
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
strategy:
matrix:
bin:
- cargo run --bin reth --features "dev"
steps:
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Checkout base
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v6
with:
ref: ${{ github.base_ref || 'main' }}
persist-credentials: false
# On `main` branch, generates test vectors and serializes them to disk using `Compact`.
- name: Generate compact vectors
run: |
${{ matrix.bin }} -- test-vectors compact --write
- name: Checkout PR
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v6
with:
clean: false
persist-credentials: false
# On incoming merge try to read and decode previously generated vectors with `Compact`
- name: Read vectors
run: ${{ matrix.bin }} -- test-vectors compact --read

View File

@@ -9,14 +9,13 @@ on:
workflow_dispatch:
# Needed so we can run it manually
permissions: {}
permissions:
contents: write
pull-requests: write
jobs:
update:
if: github.repository == 'paradigmxyz/reth'
permissions:
contents: write
pull-requests: write
uses: tempoxyz/ci/.github/workflows/cargo-update-pr.yml@main
secrets:
token: ${{ secrets.GITHUB_TOKEN }}
token: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -17,8 +17,6 @@ on:
env:
DOCKER_USERNAME: ${{ github.actor }}
permissions: {}
jobs:
tag-reth-latest:
name: Tag reth as latest
@@ -29,22 +27,16 @@ jobs:
contents: read
steps:
- name: Log in to Docker
env:
DOCKER_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
run: |
echo "$DOCKER_PASSWORD" | docker login ghcr.io --username "${DOCKER_USERNAME}" --password-stdin
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io --username ${DOCKER_USERNAME} --password-stdin
- name: Pull reth release image
env:
VERSION: ${{ inputs.version }}
run: |
docker pull ghcr.io/${{ github.repository_owner }}/reth:${VERSION}
docker pull ghcr.io/${{ github.repository_owner }}/reth:${{ inputs.version }}
- name: Tag reth as latest
env:
VERSION: ${{ inputs.version }}
run: |
docker tag ghcr.io/${{ github.repository_owner }}/reth:${VERSION} ghcr.io/${{ github.repository_owner }}/reth:latest
docker tag ghcr.io/${{ github.repository_owner }}/reth:${{ inputs.version }} ghcr.io/${{ github.repository_owner }}/reth:latest
- name: Push reth latest tag
run: |

View File

@@ -13,8 +13,6 @@ on:
default: "artifacts"
description: "Name for the uploaded artifact"
permissions: {}
jobs:
build:
timeout-minutes: 45
@@ -23,9 +21,7 @@ jobs:
id-token: write
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- run: mkdir -p artifacts
- name: Get git info
@@ -36,12 +32,8 @@ jobs:
- name: Detect fork
id: fork
env:
EVENT_NAME: ${{ github.event_name }}
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
REPO: ${{ github.repository }}
run: |
if [ "$EVENT_NAME" = "pull_request" ] && [ "$HEAD_REPO" != "$REPO" ]; then
if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.repository }}" ]; then
echo "is_fork=true" >> "$GITHUB_OUTPUT"
else
echo "is_fork=false" >> "$GITHUB_OUTPUT"
@@ -50,11 +42,11 @@ jobs:
# Depot build (upstream only)
- name: Set up Depot CLI
if: steps.fork.outputs.is_fork == 'false'
uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1.7.1
uses: depot/setup-action@v1
- name: Build reth image (Depot)
if: steps.fork.outputs.is_fork == 'false'
uses: depot/bake-action@1d58c2668346981089b088b7ef36755b206b20e9 # v1.13.0
uses: depot/bake-action@v1
env:
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
@@ -68,11 +60,11 @@ jobs:
# Docker build (forks)
- name: Set up Docker Buildx
if: steps.fork.outputs.is_fork == 'true'
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
uses: docker/setup-buildx-action@v3
- name: Build reth image (Docker)
if: steps.fork.outputs.is_fork == 'true'
uses: docker/bake-action@82490499d2e5613fcead7e128237ef0b0ea210f7 # v7.0.0
uses: docker/bake-action@v6
env:
VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
VERGEN_GIT_DESCRIBE: ${{ steps.git.outputs.describe }}
@@ -84,7 +76,7 @@ jobs:
*.dockerfile=Dockerfile
- name: Upload reth image
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.artifact_name }}
path: ./artifacts

View File

@@ -29,8 +29,6 @@ on:
type: boolean
default: false
permissions: {}
jobs:
build:
if: github.repository == 'paradigmxyz/reth'
@@ -41,15 +39,13 @@ jobs:
contents: read
id-token: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- name: Set up Depot CLI
uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1.7.1
uses: depot/setup-action@v1
- name: Log in to GHCR
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -64,13 +60,10 @@ jobs:
- name: Determine build parameters
id: params
env:
EVENT_NAME: ${{ github.event_name }}
BUILD_TYPE: ${{ inputs.build_type }}
run: |
REGISTRY="ghcr.io/${{ github.repository_owner }}"
if [[ "${EVENT_NAME}" == "push" ]]; then
if [[ "${{ github.event_name }}" == "push" ]]; then
VERSION="${GITHUB_REF#refs/tags/}"
echo "targets=ethereum" >> "$GITHUB_OUTPUT"
@@ -88,7 +81,7 @@ jobs:
echo "ethereum_set=ethereum.tags=${REGISTRY}/reth:${VERSION}" >> "$GITHUB_OUTPUT"
fi
elif [[ "${EVENT_NAME}" == "schedule" ]] || [[ "${BUILD_TYPE}" == "nightly" ]]; then
elif [[ "${{ github.event_name }}" == "schedule" ]] || [[ "${{ inputs.build_type }}" == "nightly" ]]; then
echo "targets=nightly" >> "$GITHUB_OUTPUT"
echo "ethereum_tags=${REGISTRY}/reth:nightly" >> "$GITHUB_OUTPUT"
echo "ethereum_set=ethereum.tags=${REGISTRY}/reth:nightly" >> "$GITHUB_OUTPUT"
@@ -101,7 +94,7 @@ jobs:
fi
- name: Build and push images
uses: depot/bake-action@1d58c2668346981089b088b7ef36755b206b20e9 # v1.13.0
uses: depot/bake-action@v1
env:
VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
VERGEN_GIT_DESCRIBE: ${{ steps.git.outputs.describe }}
@@ -112,8 +105,6 @@ jobs:
files: docker-bake.hcl
targets: ${{ steps.params.outputs.targets }}
push: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }}
save: false
load: false
set: |
${{ steps.params.outputs.ethereum_set }}
@@ -133,7 +124,7 @@ jobs:
if: failure() && github.event_name == 'schedule'
steps:
- name: Slack Webhook Action
uses: rtCamp/action-slack-notify@e31e87e03dd19038e411e38ae27cbad084a90661 # v2.3.3
uses: rtCamp/action-slack-notify@v2
env:
SLACK_COLOR: danger
SLACK_ICON_EMOJI: ":rotating_light:"

View File

@@ -17,27 +17,19 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
test:
name: e2e-testsuite
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_BACKTRACE: 1
timeout-minutes: 90
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: nextest
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: taiki-e/install-action@nextest
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Run e2e tests
@@ -56,21 +48,15 @@ jobs:
rocksdb:
name: e2e-rocksdb
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_BACKTRACE: 1
timeout-minutes: 60
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: nextest
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: taiki-e/install-action@nextest
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Run RocksDB e2e tests

View File

@@ -12,8 +12,6 @@ on:
required: true
default: "etc/grafana/dashboards/overview.json"
permissions: {}
jobs:
fetch:
runs-on: ubuntu-latest
@@ -21,11 +19,9 @@ jobs:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- uses: actions/setup-python@v6
with:
python-version: "3.12"
@@ -33,18 +29,14 @@ jobs:
env:
FETCH_GRAFANA_DASHBOARD_URL: ${{ secrets.FETCH_GRAFANA_DASHBOARD_URL }}
FETCH_GRAFANA_DASHBOARD_TOKEN: ${{ secrets.FETCH_GRAFANA_DASHBOARD_TOKEN }}
DASHBOARD_UID: ${{ inputs.dashboard_uid }}
TARGET_PATH: ${{ inputs.target_path }}
run: |
python3 .github/scripts/fetch-grafana-dashboard.py "${DASHBOARD_UID}" \
> "${TARGET_PATH}"
python3 .github/scripts/fetch-grafana-dashboard.py "${{ inputs.dashboard_uid }}" \
> "${{ inputs.target_path }}"
- name: Check for changes
id: diff
env:
TARGET_PATH: ${{ inputs.target_path }}
run: |
if git diff --quiet "${TARGET_PATH}"; then
if git diff --quiet "${{ inputs.target_path }}"; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "No changes detected."
else
@@ -55,10 +47,8 @@ jobs:
if: steps.diff.outputs.changed == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DASHBOARD_UID: ${{ inputs.dashboard_uid }}
TARGET_PATH: ${{ inputs.target_path }}
run: |
TARGET="${TARGET_PATH}"
TARGET="${{ inputs.target_path }}"
FILENAME="$(basename "$TARGET")"
BRANCH="chore/sync-grafana-${FILENAME%.*}-$(date +%Y%m%d-%H%M%S)"
git config user.name "github-actions[bot]"
@@ -69,4 +59,4 @@ jobs:
git push origin "$BRANCH"
gh pr create \
--title "chore: update Grafana dashboard ${FILENAME}" \
--body "Automated export from Grafana (dashboard UID: \`${DASHBOARD_UID}\`, target: \`${TARGET}\`)."
--body "Automated export from Grafana (dashboard UID: \`${{ inputs.dashboard_uid }}\`, target: \`${TARGET}\`)."

View File

@@ -6,17 +6,11 @@ on:
push:
branches: [main]
permissions: {}
jobs:
check-dashboard:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- name: Validate dashboard format
run: |
python3 -c "

View File

@@ -14,13 +14,8 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
build-reth:
permissions:
contents: read
id-token: write
uses: ./.github/workflows/docker-test.yml
with:
hive_target: hive
@@ -31,8 +26,6 @@ jobs:
if: github.repository == 'paradigmxyz/reth'
timeout-minutes: 45
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-16' || 'ubuntu-latest' }}
permissions:
contents: read
strategy:
fail-fast: false
matrix:
@@ -41,28 +34,25 @@ jobs:
- osaka
name: Prepare Hive - ${{ matrix.variant == 'amsterdam' && 'Amsterdam' || 'Osaka' }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- name: Checkout hive tests
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v6
with:
repository: ethereum/hive
path: hivetests
persist-credentials: false
- name: Get hive commit hash
id: hive-commit
run: echo "hash=$(cd hivetests && git rev-parse HEAD)" >> $GITHUB_OUTPUT
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
- uses: actions/setup-go@v6
with:
go-version: "^1.13.1"
- run: go version
- name: Restore hive assets cache
id: cache-hive
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
uses: actions/cache@v5
with:
path: ./hive_assets
key: hive-assets-${{ matrix.variant }}-${{ steps.hive-commit.outputs.hash }}-${{ hashFiles('.github/scripts/hive/build_simulators.sh') }}
@@ -85,7 +75,7 @@ jobs:
chmod +x hive
- name: Upload hive assets
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: hive_assets_${{ matrix.variant }}
path: ./hive_assets
@@ -203,22 +193,20 @@ jobs:
# Use larger runners for eels tests to avoid OOM runner crashes
runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Download hive assets
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: hive_assets_amsterdam
path: /tmp
- name: Download reth image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: reth
path: /tmp
@@ -232,21 +220,16 @@ jobs:
chmod +x /usr/local/bin/hive
- name: Checkout hive tests
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v6
with:
repository: ethereum/hive
ref: master
path: hivetests
persist-credentials: false
- name: Run simulator
env:
SCENARIO_SIM: ${{ matrix.scenario.sim }}
SCENARIO_LIMIT: ${{ matrix.scenario.limit }}
SCENARIO_TESTS: ${{ join(matrix.scenario.include, '|') }}
run: |
LIMIT="$SCENARIO_LIMIT"
TESTS="$SCENARIO_TESTS"
LIMIT="${{ matrix.scenario.limit }}"
TESTS="${{ join(matrix.scenario.include, '|') }}"
if [ -n "$LIMIT" ] && [ -n "$TESTS" ]; then
FILTER="$LIMIT/$TESTS"
elif [ -n "$LIMIT" ]; then
@@ -257,7 +240,7 @@ jobs:
FILTER="/"
fi
echo "filter: $FILTER"
.github/scripts/hive/run_simulator.sh "$SCENARIO_SIM" "$FILTER" "amsterdam"
.github/scripts/hive/run_simulator.sh "${{ matrix.scenario.sim }}" "$FILTER" "amsterdam"
- name: Parse hive output
run: |
@@ -383,22 +366,20 @@ jobs:
# Use larger runners for eels tests to avoid OOM runner crashes
runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Download hive assets
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: hive_assets_osaka
path: /tmp
- name: Download reth image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: reth
path: /tmp
@@ -412,21 +393,16 @@ jobs:
chmod +x /usr/local/bin/hive
- name: Checkout hive tests
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v6
with:
repository: ethereum/hive
ref: master
path: hivetests
persist-credentials: false
- name: Run simulator
env:
SCENARIO_SIM: ${{ matrix.scenario.sim }}
SCENARIO_LIMIT: ${{ matrix.scenario.limit }}
SCENARIO_TESTS: ${{ join(matrix.scenario.include, '|') }}
run: |
LIMIT="$SCENARIO_LIMIT"
TESTS="$SCENARIO_TESTS"
LIMIT="${{ matrix.scenario.limit }}"
TESTS="${{ join(matrix.scenario.include, '|') }}"
if [ -n "$LIMIT" ] && [ -n "$TESTS" ]; then
FILTER="$LIMIT/$TESTS"
elif [ -n "$LIMIT" ]; then
@@ -437,7 +413,7 @@ jobs:
FILTER="/"
fi
echo "filter: $FILTER"
.github/scripts/hive/run_simulator.sh "$SCENARIO_SIM" "$FILTER" "osaka"
.github/scripts/hive/run_simulator.sh "${{ matrix.scenario.sim }}" "$FILTER" "osaka"
- name: Parse hive output
run: |
@@ -460,7 +436,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Slack Webhook Action
uses: rtCamp/action-slack-notify@e31e87e03dd19038e411e38ae27cbad084a90661 # v2.3.3
uses: rtCamp/action-slack-notify@v2
env:
SLACK_COLOR: ${{ job.status }}
SLACK_MESSAGE: "Failed run: https://github.com/paradigmxyz/reth/actions/runs/${{ github.run_id }}"

View File

@@ -20,15 +20,11 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
test:
name: test / ${{ matrix.network }}
if: github.event_name != 'schedule'
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_BACKTRACE: 1
strategy:
@@ -36,18 +32,14 @@ jobs:
network: ["ethereum"]
timeout-minutes: 60
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- name: Install Geth
run: .github/scripts/install_geth.sh
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: nextest
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: taiki-e/install-action@nextest
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Run tests
@@ -66,7 +58,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe # release/v1
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}
@@ -74,19 +66,13 @@ jobs:
name: era1 file integration tests once a day
if: github.event_name == 'schedule' && github.repository == 'paradigmxyz/reth'
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: nextest
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: taiki-e/install-action@nextest
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: run era1 files integration tests

View File

@@ -18,14 +18,9 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
build-reth:
if: github.repository == 'paradigmxyz/reth'
permissions:
contents: read
id-token: write
uses: ./.github/workflows/docker-test.yml
with:
hive_target: kurtosis
@@ -37,18 +32,15 @@ jobs:
fail-fast: false
name: run kurtosis
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
needs:
- build-reth
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Download reth image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: artifacts
path: /tmp
@@ -60,7 +52,7 @@ jobs:
docker image ls -a
- name: Run kurtosis
uses: ethpandaops/kurtosis-assertoor-github-action@f64942cbc780df731a731ea9f45765b161d2c8df # v1.0.1
uses: ethpandaops/kurtosis-assertoor-github-action@v1
with:
ethereum_package_args: ".github/assets/kurtosis_network_params.yaml"
@@ -70,7 +62,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Slack Webhook Action
uses: rtCamp/action-slack-notify@e31e87e03dd19038e411e38ae27cbad084a90661 # v2.3.3
uses: rtCamp/action-slack-notify@v2
env:
SLACK_COLOR: ${{ job.status }}
SLACK_MESSAGE: "Failed run: https://github.com/paradigmxyz/reth/actions/runs/${{ github.run_id }}"

View File

@@ -4,23 +4,19 @@ on:
pull_request:
types: [opened]
permissions: {}
jobs:
label_prs:
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
pull-requests: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Label PRs
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
uses: actions/github-script@v9
with:
script: |
const label_pr = require('./.github/scripts/label_pr.js')

View File

@@ -8,17 +8,11 @@ on:
paths:
- '.github/**'
permissions: {}
jobs:
actionlint:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- name: Download actionlint
id: get_actionlint
run: bash <(curl https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash)

View File

@@ -10,14 +10,10 @@ env:
CARGO_TERM_COLOR: always
RUSTC_WRAPPER: "sccache"
permissions: {}
jobs:
clippy-binaries:
name: clippy binaries / ${{ matrix.type }}
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
strategy:
matrix:
@@ -26,19 +22,17 @@ jobs:
args: --workspace --lib --examples --tests --benches --locked
features: "ethereum asm-keccak jemalloc jemalloc-prof min-error-logs min-warn-logs min-info-logs min-debug-logs min-trace-logs"
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@clippy
with:
components: clippy
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- if: "${{ matrix.type == 'book' }}"
uses: arduino/setup-protoc@c65c819552d16ad3c9b72d9dfd5ba5237b9c906b # v3.0.0
uses: arduino/setup-protoc@v3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Run clippy on binaries
@@ -49,19 +43,15 @@ jobs:
clippy:
name: clippy
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@nightly
with:
components: clippy
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- run: cargo clippy --workspace --lib --examples --tests --benches --all-features --locked
@@ -70,25 +60,19 @@ jobs:
wasm:
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
with:
target: wasm32-wasip1
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: cargo-hack
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: taiki-e/install-action@cargo-hack
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- uses: dcarbone/install-jq-action@b7ef57d46ece78760b4019dbc4080a1ba2a40b45 # v3.2.0
- uses: dcarbone/install-jq-action@v3
- name: Run Wasm checks
run: |
sudo apt update && sudo apt install gcc-multilib
@@ -96,49 +80,37 @@ jobs:
riscv:
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 60
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
with:
target: riscv32imac-unknown-none-elf
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: cargo-hack
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: taiki-e/install-action@cargo-hack
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- uses: dcarbone/install-jq-action@b7ef57d46ece78760b4019dbc4080a1ba2a40b45 # v3.2.0
- uses: dcarbone/install-jq-action@v3
- name: Run RISC-V checks
run: .github/scripts/check_rv32imac.sh
crate-checks:
name: crate-checks (${{ matrix.partition }}/${{ matrix.total_partitions }})
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
permissions:
contents: read
strategy:
matrix:
partition: [1, 2, 3]
total_partitions: [3]
timeout-minutes: 60
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: cargo-hack
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: taiki-e/install-action@cargo-hack
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- run: cargo hack check --workspace --partition ${{ matrix.partition }}/${{ matrix.total_partitions }}
@@ -146,19 +118,15 @@ jobs:
msrv:
name: MSRV
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@master
with:
toolchain: "1.93" # MSRV
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- run: cargo build --bin reth --workspace
@@ -168,17 +136,13 @@ jobs:
docs:
name: docs
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@nightly
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- run: cargo docs --document-private-items
@@ -190,56 +154,42 @@ jobs:
fmt:
name: fmt
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@nightly
with:
components: rustfmt
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: mozilla-actions/sccache-action@v0.0.9
- name: Run fmt
run: cargo fmt --all --check
udeps:
name: udeps
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@nightly
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: cargo-udeps
- uses: taiki-e/install-action@cargo-udeps
- run: cargo udeps --workspace --lib --examples --tests --benches --all-features --locked
book:
name: book
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@nightly
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- run: cargo build --bin reth --workspace
@@ -251,54 +201,38 @@ jobs:
typos:
runs-on: ubuntu-latest
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: crate-ci/typos@02ea592e44b3a53c302f697cddca7641cd051c3d # v1.45.0
- uses: actions/checkout@v6
- uses: crate-ci/typos@v1
check-toml:
runs-on: ubuntu-latest
permissions:
contents: read
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
uses: actions/checkout@v6
- name: Run dprint
uses: dprint/check@9cb3a2b17a8e606d37aae341e49df3654933fc23 # v2.3
uses: dprint/check@v2.3
with:
config-path: dprint.json
grafana:
runs-on: ubuntu-latest
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- name: Check dashboard JSON with jq
uses: sergeysova/jq-action@a3f0d4ff59cc1dddf023fc0b325dd75b10deec58 # v2.3.0
uses: sergeysova/jq-action@v2
with:
cmd: jq empty etc/grafana/dashboards/overview.json
no-test-deps:
runs-on: ubuntu-latest
permissions:
contents: read
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- name: Ensure no arbitrary or proptest dependency on default build
run: cargo tree --package reth -e=features,no-dev | grep -Eq "arbitrary|proptest" && exit 1 || exit 0
@@ -306,17 +240,13 @@ jobs:
# Check crates correctly propagate features
feature-propagation:
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
timeout-minutes: 20
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: taiki-e/cache-cargo-install-action@a8b9ecf8e0c0ea09d7481cfc583a5203ecd585b5 # v3.0.5
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: rui314/setup-mold@v1
- uses: taiki-e/cache-cargo-install-action@v3
with:
tool: zepter
- name: Eagerly pull dependencies
@@ -324,8 +254,6 @@ jobs:
- run: zepter run check
deny:
permissions:
contents: read
uses: tempoxyz/ci/.github/workflows/deny.yml@main
lint-success:
@@ -349,6 +277,6 @@ jobs:
timeout-minutes: 30
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe # release/v1
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}

View File

@@ -4,36 +4,27 @@ on:
pull_request:
types: [labeled]
permissions: {}
jobs:
publish:
runs-on: ubuntu-latest
if: github.event.label.name == 'cyclops'
permissions: {}
steps:
- name: Publish event
env:
EVENTS_KEY: ${{ secrets.EVENTS_KEY }}
EVENTS_CERT: ${{ secrets.EVENTS_CERT }}
EVENTS_ARGS: ${{ secrets.EVENTS_ARGS }}
PR_NUMBER: ${{ github.event.pull_request.number }}
PR_SHA: ${{ github.event.pull_request.head.sha }}
run: |
set -euo pipefail
echo "$EVENTS_KEY" > "${{ runner.temp }}/key"
echo "$EVENTS_CERT" > "${{ runner.temp }}/cert"
echo "${{ secrets.EVENTS_KEY }}" > ${{ runner.temp }}/key
echo "${{ secrets.EVENTS_CERT }}" > ${{ runner.temp }}/cert
curl -sf -o /dev/null -X POST $EVENTS_ARGS \
curl -sf -o /dev/null -X POST ${{ secrets.EVENTS_ARGS }} \
-H "Content-Type: application/json" \
--key "${{ runner.temp }}/key" \
--cert "${{ runner.temp }}/cert" \
--key ${{ runner.temp }}/key \
--cert ${{ runner.temp }}/cert \
-d '{
"repository": "${{ github.repository }}",
"event": "pr_audit",
"data": {
"pr_number": '"$PR_NUMBER"',
"sha": "'"$PR_SHA"'"
"pr_number": ${{ github.event.pull_request.number }},
"sha": "${{ github.event.pull_request.head.sha }}"
}
}'

View File

@@ -8,19 +8,20 @@ on:
- edited
- synchronize
permissions: {}
permissions:
pull-requests: read
contents: read
jobs:
conventional-title:
name: Validate PR title is Conventional Commit
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- name: Check title
id: lint_pr_title
uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
uses: amannn/action-semantic-pull-request@v6
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
@@ -39,7 +40,7 @@ jobs:
continue-on-error: true
- name: Add PR Comment for Invalid Title
if: steps.lint_pr_title.outcome == 'failure'
uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff # v3.0.3
uses: marocchino/sticky-pull-request-comment@v2
with:
header: pr-title-lint-error
message: |
@@ -75,7 +76,7 @@ jobs:
- name: Remove Comment for Valid Title
if: steps.lint_pr_title.outcome == 'success'
uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff # v3.0.3
uses: marocchino/sticky-pull-request-comment@v2
with:
header: pr-title-lint-error
delete: true

View File

@@ -7,15 +7,12 @@ on:
release:
types: [published]
permissions: {}
jobs:
release-homebrew:
runs-on: ubuntu-latest
permissions: {}
steps:
- name: Update Homebrew formula
uses: dawidd6/action-homebrew-bump-formula@1446dca236b0440c6f02723a3f14f13be2c04ab0 # v7
uses: dawidd6/action-homebrew-bump-formula@v7
with:
token: ${{ secrets.HOMEBREW }}
no_fork: true

View File

@@ -2,8 +2,6 @@
name: release-reproducible
permissions: {}
on:
workflow_run:
workflows: [release]
@@ -17,23 +15,20 @@ jobs:
name: extract version
if: ${{ github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Extract version from triggering tag
id: extract_version
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
run: |
# Get the tag that points to the head SHA of the triggering workflow
TAG=$(gh api /repos/${{ github.repository }}/git/refs/tags \
--jq ".[] | select(.object.sha == \"${HEAD_SHA}\") | .ref" \
--jq '.[] | select(.object.sha == "${{ github.event.workflow_run.head_sha }}") | .ref' \
| head -1 \
| sed 's|refs/tags/||')
if [ -z "$TAG" ]; then
echo "No tag found for SHA ${HEAD_SHA}"
echo "No tag found for SHA ${{ github.event.workflow_run.head_sha }}"
exit 1
fi
@@ -49,16 +44,15 @@ jobs:
packages: write
contents: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
ref: ${{ needs.extract-version.outputs.VERSION }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
uses: docker/setup-buildx-action@v3
- name: Log in to GitHub Container Registry
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -71,7 +65,7 @@ jobs:
echo "RUST_TOOLCHAIN=$RUST_TOOLCHAIN" >> $GITHUB_OUTPUT
- name: Build reproducible artifacts
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
uses: docker/build-push-action@v6
id: docker_build
with:
context: .
@@ -81,11 +75,13 @@ jobs:
VERSION=${{ needs.extract-version.outputs.VERSION }}
target: artifacts
outputs: type=local,dest=./docker-artifacts
cache-from: type=gha
cache-to: type=gha,mode=max
env:
DOCKER_BUILD_RECORD_UPLOAD: false
- name: Build and push final image
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile.reproducible
@@ -96,6 +92,8 @@ jobs:
tags: |
${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }}:${{ needs.extract-version.outputs.VERSION }}
${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }}:latest
cache-from: type=gha
cache-to: type=gha,mode=max
provenance: false
env:
DOCKER_BUILD_RECORD_UPLOAD: false

View File

@@ -3,8 +3,6 @@
name: release
permissions: {}
on:
push:
tags:
@@ -22,24 +20,21 @@ env:
REPRODUCIBLE_IMAGE_NAME: ${{ github.repository_owner }}/reth-reproducible
CARGO_TERM_COLOR: always
DOCKER_IMAGE_NAME_URL: https://ghcr.io/${{ github.repository_owner }}/reth
RUSTC_WRAPPER: "sccache"
jobs:
dry-run:
name: check dry run
runs-on: ubuntu-latest
permissions: {}
steps:
- env:
DRY_RUN: ${{ github.event.inputs.dry_run }}
run: |
echo "Dry run value: ${DRY_RUN}"
echo "Dry run enabled: $( [ "${DRY_RUN}" = 'true' ] && echo true || echo false )"
echo "Dry run disabled: $( [ "${DRY_RUN}" != 'true' ] && echo true || echo false )"
- run: |
echo "Dry run value: ${{ github.event.inputs.dry_run }}"
echo "Dry run enabled: ${{ github.event.inputs.dry_run == 'true'}}"
echo "Dry run disabled: ${{ github.event.inputs.dry_run != 'true'}}"
extract-version:
name: extract version
runs-on: ubuntu-latest
permissions: {}
steps:
- name: Extract version
run: echo "VERSION=${GITHUB_REF_NAME//\//-}" >> $GITHUB_OUTPUT
@@ -50,15 +45,12 @@ jobs:
check-version:
name: check version
runs-on: ubuntu-latest
permissions:
contents: read
needs: extract-version
if: ${{ github.event.inputs.dry_run != 'true' }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@v0.0.9
- name: Verify crate version matches tag
# Check that the Cargo version starts with the tag,
# so that Cargo version 1.4.8 can be matched against both v1.4.8 and v1.4.8-rc.1
@@ -71,8 +63,6 @@ jobs:
build:
name: build release
runs-on: ${{ matrix.configs.os }}
permissions:
contents: read
needs: extract-version
continue-on-error: ${{ matrix.configs.allow_fail }}
strategy:
@@ -105,20 +95,20 @@ jobs:
- command: build
binary: reth
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
with:
target: ${{ matrix.configs.target }}
- uses: mozilla-actions/sccache-action@v0.0.9
- name: Install cross main
if: ${{ !matrix.configs.native }}
id: cross_main
run: |
cargo install cross --locked \
--git https://github.com/cross-rs/cross \
--rev 65fe72b0cdb1e7e0cc0652517498d4389cc8f5cf
cargo install cross --locked --git https://github.com/cross-rs/cross
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Apple M1 setup
if: matrix.configs.target == 'aarch64-apple-darwin'
@@ -155,14 +145,14 @@ jobs:
- name: Upload artifact
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
- name: Upload signature
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
@@ -181,12 +171,11 @@ jobs:
steps:
# This is necessary for generating the changelog.
# It has to come before "Download Artifacts" or else it deletes the artifacts.
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Download artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
- name: Generate full changelog
id: changelog
run: |
@@ -272,7 +261,6 @@ jobs:
dry-run-summary:
name: dry run summary
runs-on: ubuntu-latest
permissions: {}
needs: [build, extract-version]
if: ${{ github.event.inputs.dry_run == 'true' }}
env:

View File

@@ -5,15 +5,11 @@ on:
schedule:
- cron: "0 1 */2 * *"
permissions: {}
jobs:
build:
if: github.repository == 'paradigmxyz/reth'
name: build reproducible binaries
runs-on: ${{ matrix.runner }}
permissions:
contents: read
strategy:
matrix:
include:
@@ -22,16 +18,14 @@ jobs:
- runner: ubuntu-22.04
machine: machine-2
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
with:
target: x86_64-unknown-linux-gnu
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
uses: docker/setup-buildx-action@v3
- name: Build reproducible binary with Docker
run: |
@@ -49,7 +43,7 @@ jobs:
echo "Binaries SHA256 on ${{ matrix.machine }}: $(cat checksum.sha256)"
- name: Upload the hash
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
uses: actions/upload-artifact@v7
with:
name: checksum-${{ matrix.machine }}
path: |
@@ -62,12 +56,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Download artifacts from machine-1
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: checksum-machine-1
path: machine-1/
- name: Download artifacts from machine-2
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@v8
with:
name: checksum-machine-2
path: machine-2/

View File

@@ -18,28 +18,22 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
stage:
name: stage-run-test
# Only run stage commands test in merge groups
if: github.event_name == 'merge_group'
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_LOG: info,sync=error
RUST_BACKTRACE: 1
timeout-minutes: 60
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Build reth

View File

@@ -7,8 +7,6 @@ on:
schedule:
- cron: "30 1 * * *"
permissions: {}
jobs:
close-issues:
if: github.repository == 'paradigmxyz/reth'
@@ -17,7 +15,7 @@ jobs:
issues: write
pull-requests: write
steps:
- uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0
- uses: actions/stale@v10
with:
days-before-stale: 21
days-before-close: 7

View File

@@ -15,15 +15,11 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
sync:
if: github.repository == 'paradigmxyz/reth'
name: sync (${{ matrix.chain.bin }})
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_LOG: info,sync=error
RUST_BACKTRACE: 1
@@ -38,13 +34,11 @@ jobs:
block: 100000
unwind-target: "0x52e0509d33a988ef807058e2980099ee3070187f7333aae12b64d4d675f34c5a"
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Build ${{ matrix.chain.bin }}

View File

@@ -15,15 +15,11 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
sync:
if: github.repository == 'paradigmxyz/reth'
name: sync (${{ matrix.chain.bin }})
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_LOG: info,sync=error
RUST_BACKTRACE: 1
@@ -38,13 +34,11 @@ jobs:
block: 100000
unwind-target: "0x52e0509d33a988ef807058e2980099ee3070187f7333aae12b64d4d675f34c5a"
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Build ${{ matrix.chain.bin }}

View File

@@ -17,14 +17,10 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions: {}
jobs:
test:
name: test / ${{ matrix.type }}
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_BACKTRACE: 1
strategy:
@@ -36,20 +32,16 @@ jobs:
exclude_args: ""
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: nextest
- uses: taiki-e/install-action@nextest
- if: "${{ matrix.type == 'book' }}"
uses: arduino/setup-protoc@c65c819552d16ad3c9b72d9dfd5ba5237b9c906b # v3.0.0
uses: arduino/setup-protoc@v3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Run tests
@@ -64,25 +56,20 @@ jobs:
state:
name: Ethereum state tests
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_LOG: info,sync=error
RUST_BACKTRACE: 1
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/checkout@v6
- name: Checkout ethereum/tests
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v6
with:
repository: ethereum/tests
ref: 81862e4848585a438d64f911a19b3825f0f4cd95
path: testing/ef-tests/ethereum-tests
submodules: recursive
fetch-depth: 1
persist-credentials: false
- name: Download & extract EEST fixtures (public)
shell: bash
env:
@@ -92,13 +79,11 @@ jobs:
mkdir -p testing/ef-tests/execution-spec-tests
URL="https://github.com/ethereum/execution-spec-tests/releases/download/${EEST_TESTS_TAG}/fixtures_stable.tar.gz"
curl -L "$URL" | tar -xz --strip-components=1 -C testing/ef-tests/execution-spec-tests
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
with:
tool: nextest
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: taiki-e/install-action@nextest
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- run: cargo nextest run --no-fail-fast --cargo-profile hivetests -p ef-tests --features "asm-keccak ef-tests"
@@ -106,19 +91,15 @@ jobs:
doc:
name: doc tests
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
permissions:
contents: read
env:
RUST_BACKTRACE: 1
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: actions/checkout@v6
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Run doctests
@@ -132,6 +113,6 @@ jobs:
timeout-minutes: 30
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe # release/v1
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}

544
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,5 @@
[workspace.package]
version = "2.2.0"
version = "2.1.0"
edition = "2024"
rust-version = "1.93"
license = "MIT OR Apache-2.0"
@@ -450,39 +450,39 @@ alloy-sol-types = { version = "1.5.6", default-features = false }
alloy-chains = { version = "0.2.33", default-features = false }
alloy-eip2124 = { version = "0.2.0", default-features = false }
alloy-eip7928 = { version = "0.3.4", default-features = false }
alloy-evm = { version = "0.34.0", default-features = false }
alloy-evm = { version = "0.33.0", default-features = false }
alloy-rlp = { version = "0.3.13", default-features = false, features = ["core-net"] }
alloy-trie = { version = "0.9.4", default-features = false }
alloy-hardforks = "0.4.7"
alloy-consensus = { version = "2.0.4", default-features = false }
alloy-contract = { version = "2.0.4", default-features = false }
alloy-eips = { version = "2.0.4", default-features = false }
alloy-genesis = { version = "2.0.4", default-features = false }
alloy-json-rpc = { version = "2.0.4", default-features = false }
alloy-network = { version = "2.0.4", default-features = false }
alloy-network-primitives = { version = "2.0.4", default-features = false }
alloy-provider = { version = "2.0.4", features = ["reqwest", "debug-api"], default-features = false }
alloy-pubsub = { version = "2.0.4", default-features = false }
alloy-rpc-client = { version = "2.0.4", default-features = false }
alloy-rpc-types = { version = "2.0.4", features = ["eth"], default-features = false }
alloy-rpc-types-admin = { version = "2.0.4", default-features = false }
alloy-rpc-types-anvil = { version = "2.0.4", default-features = false }
alloy-rpc-types-beacon = { version = "2.0.4", default-features = false }
alloy-rpc-types-debug = { version = "2.0.4", default-features = false }
alloy-rpc-types-engine = { version = "2.0.4", default-features = false }
alloy-rpc-types-eth = { version = "2.0.4", default-features = false }
alloy-rpc-types-mev = { version = "2.0.4", default-features = false }
alloy-rpc-types-trace = { version = "2.0.4", default-features = false }
alloy-rpc-types-txpool = { version = "2.0.4", default-features = false }
alloy-serde = { version = "2.0.4", default-features = false }
alloy-signer = { version = "2.0.4", default-features = false }
alloy-signer-local = { version = "2.0.4", default-features = false }
alloy-transport = { version = "2.0.4" }
alloy-transport-http = { version = "2.0.4", features = ["reqwest-rustls-tls"], default-features = false }
alloy-transport-ipc = { version = "2.0.4", default-features = false }
alloy-transport-ws = { version = "2.0.4", default-features = false }
alloy-consensus = { version = "2.0.1", default-features = false }
alloy-contract = { version = "2.0.1", default-features = false }
alloy-eips = { version = "2.0.1", default-features = false }
alloy-genesis = { version = "2.0.1", default-features = false }
alloy-json-rpc = { version = "2.0.1", default-features = false }
alloy-network = { version = "2.0.1", default-features = false }
alloy-network-primitives = { version = "2.0.1", default-features = false }
alloy-provider = { version = "2.0.1", features = ["reqwest", "debug-api"], default-features = false }
alloy-pubsub = { version = "2.0.1", default-features = false }
alloy-rpc-client = { version = "2.0.1", default-features = false }
alloy-rpc-types = { version = "2.0.1", features = ["eth"], default-features = false }
alloy-rpc-types-admin = { version = "2.0.1", default-features = false }
alloy-rpc-types-anvil = { version = "2.0.1", default-features = false }
alloy-rpc-types-beacon = { version = "2.0.1", default-features = false }
alloy-rpc-types-debug = { version = "2.0.1", default-features = false }
alloy-rpc-types-engine = { version = "2.0.1", default-features = false }
alloy-rpc-types-eth = { version = "2.0.1", default-features = false }
alloy-rpc-types-mev = { version = "2.0.1", default-features = false }
alloy-rpc-types-trace = { version = "2.0.1", default-features = false }
alloy-rpc-types-txpool = { version = "2.0.1", default-features = false }
alloy-serde = { version = "2.0.1", default-features = false }
alloy-signer = { version = "2.0.1", default-features = false }
alloy-signer-local = { version = "2.0.1", default-features = false }
alloy-transport = { version = "2.0.1" }
alloy-transport-http = { version = "2.0.1", features = ["reqwest-rustls-tls"], default-features = false }
alloy-transport-ipc = { version = "2.0.1", default-features = false }
alloy-transport-ws = { version = "2.0.1", default-features = false }
# misc
either = { version = "1.15.0", default-features = false }
@@ -581,7 +581,7 @@ tower = "0.5"
tower-http = "0.6"
# p2p
discv5 = { git = "https://github.com/sigp/discv5", rev = "7663c00" }
discv5 = "0.10"
if-addrs = "0.14"
# rpc

View File

@@ -50,7 +50,7 @@ RUN if [ -n "$RUSTFLAGS" ]; then \
RUN cp /app/target/$BUILD_PROFILE/reth /app/reth
# Use Ubuntu as the release image
FROM ubuntu:24.04 AS runtime
FROM ubuntu AS runtime
WORKDIR /app
# Copy reth over from the build stage

View File

@@ -7,16 +7,15 @@
//! `execute_transaction` to apply segment-boundary changes.
use crate::evm_config::BigBlockSegment;
use alloy_consensus::TransactionEnvelope;
use alloy_eips::eip7685::Requests;
use alloy_evm::{
block::{
BlockExecutionError, BlockExecutionResult, BlockExecutor, BlockExecutorFactory,
ExecutableTx, GasOutput, OnStateHook, StateChangeSource, StateDB,
BlockExecutorFor, ExecutableTx, GasOutput, OnStateHook, StateChangeSource, StateDB,
},
eth::{EthBlockExecutionCtx, EthBlockExecutor, EthEvmContext, EthTxResult},
precompiles::PrecompilesMap,
Database, EthEvm, EthEvmFactory, Evm, EvmFactory, FromRecoveredTx, FromTxWithEncoded,
Database, EthEvm, EthEvmFactory, Evm, FromRecoveredTx, FromTxWithEncoded,
};
use alloy_primitives::B256;
use reth_ethereum_primitives::{Receipt, TransactionSigned};
@@ -37,7 +36,6 @@ use tracing::{debug, trace};
// ---------------------------------------------------------------------------
/// Runtime state for segment boundary tracking.
#[derive(Clone)]
pub(crate) struct BbEvmPlan {
/// The segment boundaries and environments.
pub(crate) segments: Vec<BigBlockSegment>,
@@ -75,10 +73,6 @@ impl BbEvmPlan {
.filter(|(n, _)| *n >= min && *n < block_number)
.collect()
}
pub(crate) fn segment_index_for_tx(&self, tx_index: usize) -> usize {
self.segments.partition_point(|segment| segment.start_tx <= tx_index).saturating_sub(1)
}
}
impl std::fmt::Debug for BbEvmPlan {
@@ -103,9 +97,6 @@ impl std::fmt::Debug for BbEvmPlan {
/// segment boundaries without requiring additional trait bounds on `DB`.
pub(crate) type BlockHashSeeder<DB> = fn(&mut DB, &[(u64, B256)]);
/// Function pointer that reads the BAL index from the DB.
pub(crate) type BalIndexReader<DB> = fn(&DB) -> u64;
/// Block executor that wraps [`EthBlockExecutor`] and handles segment-boundary
/// changes for big-block execution.
///
@@ -117,8 +108,7 @@ pub(crate) type BalIndexReader<DB> = fn(&DB) -> u64;
/// Gas counters reset at each boundary so that each segment's real gas limit
/// is used (preserving correct GASLIMIT opcode behavior). Accumulated offsets
/// are applied to receipts and totals in `finish()`.
#[expect(missing_debug_implementations)]
pub struct BbBlockExecutor<'a, DB, I, P, Spec>
pub(crate) struct BbBlockExecutor<'a, DB, I, P, Spec>
where
DB: Database,
{
@@ -141,10 +131,6 @@ where
/// Callback to reseed block hashes into the DB's cache at segment
/// boundaries. See [`BlockHashSeeder`].
block_hash_seeder: Option<BlockHashSeeder<DB>>,
/// Callback to read the BAL index from the DB.
bal_index_reader: Option<BalIndexReader<DB>>,
/// Whether the executor has selected its starting segment.
initialized: bool,
}
impl<'a, DB, I, P, Spec> BbBlockExecutor<'a, DB, I, P, Spec>
@@ -170,7 +156,6 @@ where
receipt_builder: RethReceiptBuilder,
plan: Option<BbEvmPlan>,
block_hash_seeder: Option<BlockHashSeeder<DB>>,
bal_index_reader: Option<BalIndexReader<DB>>,
) -> Self {
let inner = EthBlockExecutor::new(evm, ctx, spec, receipt_builder);
Self {
@@ -181,63 +166,9 @@ where
blob_gas_used_offset: 0,
shared_hook: Arc::new(Mutex::new(None)),
block_hash_seeder,
bal_index_reader,
initialized: false,
}
}
fn initialize(&mut self) -> Result<(), BlockExecutionError> {
if self.initialized {
return Ok(());
}
let plan = match &self.plan {
Some(plan) => plan,
None => return Ok(()),
};
self.initialized = true;
let bal_index =
self.bal_index_reader.map(|reader| reader(self.inner().evm().db())).unwrap_or(0);
let segment_idx =
if bal_index == 0 { 0 } else { plan.segment_index_for_tx((bal_index - 1) as usize) };
let segment = &plan.segments[segment_idx];
// Swap the EVM's block_env and executor ctx to the selected segment's
// values so that EIP-2935/EIP-4788 system calls use the correct block
// number and parent hash. Without this, the outer big block header's
// block_number (which is synthetic) would be used, writing to wrong
// EIP-2935 slots and corrupting state.
let block_env = segment.evm_env.block_env.clone();
let block_number = block_env.number.saturating_to::<u64>();
let mut cfg_env = segment.evm_env.cfg_env.clone();
cfg_env.disable_base_fee = true;
let ctx = EthBlockExecutionCtx {
parent_hash: segment.ctx.parent_hash,
parent_beacon_block_root: segment.ctx.parent_beacon_block_root,
ommers: segment.ctx.ommers,
withdrawals: segment.ctx.withdrawals.clone(),
extra_data: segment.ctx.extra_data.clone(),
tx_count_hint: segment.ctx.tx_count_hint,
slot_number: segment.ctx.slot_number,
};
let inner = self.inner_mut();
let evm_ctx = inner.evm.ctx_mut();
evm_ctx.block = block_env;
evm_ctx.cfg = cfg_env;
inner.ctx = ctx;
self.reseed_block_hashes_for(block_number);
if bal_index > 0 {
self.plan = None;
}
Ok(())
}
/// Creates a forwarding `OnStateHook` that delegates to the shared hook.
fn forwarding_hook(&self) -> Option<Box<dyn OnStateHook>> {
let shared = self.shared_hook.clone();
@@ -418,9 +349,35 @@ where
type Result = EthTxResult<HaltReason, alloy_consensus::TxType>;
fn apply_pre_execution_changes(&mut self) -> Result<(), BlockExecutionError> {
// The outer big-block header uses a synthetic block number, so start
// system calls must run against the selected real segment env.
self.initialize()?;
// Swap the EVM's block_env and executor ctx to the first segment's
// values so that the initial EIP-2935/EIP-4788 system calls use the
// correct block number and parent hash. Without this, the outer big
// block header's block_number (which is synthetic) would be used,
// writing to wrong EIP-2935 slots and corrupting state.
if let Some(seg0) = self.plan.as_ref().map(|p| &p.segments[0]) {
let block_env = seg0.evm_env.block_env.clone();
let block_number = block_env.number.saturating_to::<u64>();
let mut cfg_env = seg0.evm_env.cfg_env.clone();
cfg_env.disable_base_fee = true;
let seg0_ctx = EthBlockExecutionCtx {
parent_hash: seg0.ctx.parent_hash,
parent_beacon_block_root: seg0.ctx.parent_beacon_block_root,
ommers: seg0.ctx.ommers,
withdrawals: seg0.ctx.withdrawals.clone(),
extra_data: seg0.ctx.extra_data.clone(),
tx_count_hint: seg0.ctx.tx_count_hint,
slot_number: seg0.ctx.slot_number,
};
let inner = self.inner_mut();
let evm_ctx = inner.evm.ctx_mut();
evm_ctx.block = block_env;
evm_ctx.cfg = cfg_env;
inner.ctx = seg0_ctx;
self.reseed_block_hashes_for(block_number);
}
self.inner_mut().apply_pre_execution_changes()
}
@@ -428,16 +385,15 @@ where
&mut self,
tx: impl ExecutableTx<Self>,
) -> Result<Self::Result, BlockExecutionError> {
self.initialize()?;
self.maybe_apply_boundary()?;
self.inner_mut().execute_transaction_without_commit(tx)
}
fn commit_transaction(&mut self, output: Self::Result) -> GasOutput {
self.maybe_apply_boundary()
.expect("segment boundary application must succeed before committing transaction");
let gas_used = self.inner_mut().commit_transaction(output);
fn commit_transaction(
&mut self,
output: Self::Result,
) -> Result<GasOutput, BlockExecutionError> {
let gas_used = self.inner_mut().commit_transaction(output)?;
// Fix up cumulative_gas_used on the just-committed receipt so that
// the receipt root task (which reads receipts incrementally) sees
@@ -452,7 +408,7 @@ where
if let Some(plan) = &mut self.plan {
plan.tx_counter += 1;
}
gas_used
Ok(gas_used)
}
fn finish(
@@ -543,7 +499,7 @@ pub struct BbBlockExecutorFactory<Spec> {
receipt_builder: RethReceiptBuilder,
spec: Spec,
evm_factory: EthEvmFactory,
/// Staged plan cloned into each [`BbBlockExecutor`].
/// Staged plan consumed by the next [`BbBlockExecutor`].
pub(crate) staged_plan: Arc<Mutex<Option<BbEvmPlan>>>,
}
@@ -572,12 +528,8 @@ impl<Spec> BbBlockExecutorFactory<Spec> {
*self.staged_plan.lock().unwrap() = Some(plan);
}
pub(crate) fn clear_staged_plan(&self) {
*self.staged_plan.lock().unwrap() = None;
}
fn peek_plan(&self) -> Option<BbEvmPlan> {
self.staged_plan.lock().unwrap().clone()
fn take_plan(&self) -> Option<BbEvmPlan> {
self.staged_plan.lock().unwrap().take()
}
pub(crate) fn create_executor_with_seeder<'a, DB, I>(
@@ -585,23 +537,14 @@ impl<Spec> BbBlockExecutorFactory<Spec> {
evm: EthEvm<DB, I, PrecompilesMap>,
ctx: EthBlockExecutionCtx<'a>,
block_hash_seeder: Option<BlockHashSeeder<DB>>,
bal_index_reader: Option<BalIndexReader<DB>>,
) -> BbBlockExecutor<'a, DB, I, PrecompilesMap, &'a Spec>
where
Spec: alloy_evm::eth::spec::EthExecutorSpec,
DB: StateDB + 'a,
I: Inspector<EthEvmContext<DB>> + 'a,
{
let plan = self.peek_plan();
BbBlockExecutor::new(
evm,
ctx,
&self.spec,
self.receipt_builder,
plan,
block_hash_seeder,
bal_index_reader,
)
let plan = self.take_plan();
BbBlockExecutor::new(evm, ctx, &self.spec, self.receipt_builder, plan, block_hash_seeder)
}
}
@@ -614,12 +557,6 @@ where
type ExecutionCtx<'a> = EthBlockExecutionCtx<'a>;
type Transaction = TransactionSigned;
type Receipt = Receipt;
type TxExecutionResult = EthTxResult<
<EthEvmFactory as EvmFactory>::HaltReason,
<TransactionSigned as TransactionEnvelope>::TxType,
>;
type Executor<'a, DB: StateDB, I: Inspector<EthEvmContext<DB>>> =
BbBlockExecutor<'a, DB, I, PrecompilesMap, &'a Spec>;
fn evm_factory(&self) -> &Self::EvmFactory {
&self.evm_factory
@@ -629,12 +566,12 @@ where
&'a self,
evm: EthEvm<DB, I, PrecompilesMap>,
ctx: EthBlockExecutionCtx<'a>,
) -> Self::Executor<'a, DB, I>
) -> impl BlockExecutorFor<'a, Self, DB, I>
where
DB: StateDB,
I: Inspector<EthEvmContext<DB>>,
DB: StateDB + 'a,
I: Inspector<EthEvmContext<DB>> + 'a,
{
let plan = self.peek_plan();
BbBlockExecutor::new(evm, ctx, &self.spec, self.receipt_builder, plan, None, None)
let plan = self.take_plan();
BbBlockExecutor::new(evm, ctx, &self.spec, self.receipt_builder, plan, None)
}
}

View File

@@ -8,14 +8,11 @@
pub(crate) use reth_engine_primitives::BigBlockData;
use crate::{
evm::{BalIndexReader, BbBlockExecutorFactory, BbEvmPlan},
evm::{BbBlockExecutorFactory, BbEvmPlan},
BigBlockMap,
};
use alloy_consensus::Header;
use alloy_evm::{
eth::{spec::EthExecutorSpec, EthBlockExecutionCtx},
EthEvmFactory,
};
use alloy_evm::eth::EthBlockExecutionCtx;
use alloy_primitives::B256;
use alloy_rpc_types::engine::ExecutionData;
use core::convert::Infallible;
@@ -23,8 +20,8 @@ use reth_chainspec::{ChainSpec, EthChainSpec};
use reth_ethereum_forks::Hardforks;
use reth_ethereum_primitives::EthPrimitives;
use reth_evm::{
ConfigureEngineEvm, ConfigureEvm, Database, EvmEnv, EvmEnvFor, ExecutableTxIterator,
ExecutionCtxFor, NextBlockEnvAttributes,
ConfigureEngineEvm, ConfigureEvm, Database, EvmEnv, ExecutableTxIterator,
NextBlockEnvAttributes,
};
use reth_evm_ethereum::{EthBlockAssembler, EthEvmConfig, RethReceiptBuilder};
use reth_primitives_traits::{SealedBlock, SealedHeader};
@@ -32,6 +29,9 @@ use revm::primitives::hardfork::SpecId;
use std::sync::Arc;
use tracing::debug;
use alloy_evm::{eth::spec::EthExecutorSpec, EthEvmFactory};
use reth_evm::{EvmEnvFor, ExecutionCtxFor};
// ---------------------------------------------------------------------------
// Execution plan types
// ---------------------------------------------------------------------------
@@ -55,7 +55,7 @@ pub(crate) struct BigBlockSegment {
///
/// Wraps [`EthEvmConfig`] and a shared [`BigBlockMap`]. When a big-block
/// payload is received, the plan is staged on the [`BbBlockExecutorFactory`]
/// and cloned when executors are created. Block hashes for inter-segment
/// and consumed when the executor is created. Block hashes for inter-segment
/// BLOCKHASH resolution are reseeded into `State::block_hashes` at each
/// segment boundary via a [`BlockHashSeeder`](crate::evm::BlockHashSeeder)
/// callback injected in [`ConfigureEvm::create_executor`].
@@ -106,10 +106,6 @@ fn seed_state_block_hashes<DB>(state: &mut &mut revm::database::State<DB>, hashe
}
}
fn read_bal_index<DB>(state: &&mut revm::database::State<DB>) -> u64 {
state.bal_state.bal_index()
}
// ---------------------------------------------------------------------------
// ConfigureEvm
// ---------------------------------------------------------------------------
@@ -148,12 +144,6 @@ where
&self,
block: &'a SealedBlock<reth_ethereum_primitives::Block>,
) -> Result<EthBlockExecutionCtx<'a>, Self::Error> {
if let Some(plan) = self.plan_for_payload_hash(&block.hash()) {
self.executor_factory.stage_plan(plan);
} else {
self.executor_factory.clear_staged_plan();
}
self.inner.context_for_block(block)
}
@@ -169,7 +159,7 @@ where
&'a self,
evm: reth_evm::EvmFor<Self, &'a mut revm::database::State<DB>, I>,
ctx: EthBlockExecutionCtx<'a>,
) -> alloy_evm::block::BlockExecutorFor<
) -> impl alloy_evm::block::BlockExecutorFor<
'a,
Self::BlockExecutorFactory,
&'a mut revm::database::State<DB>,
@@ -179,16 +169,15 @@ where
DB: Database,
I: reth_evm::InspectorFor<Self, &'a mut revm::database::State<DB>> + 'a,
{
let bal_index_reader: Option<BalIndexReader<&'a mut revm::database::State<DB>>> =
Some(read_bal_index::<DB>);
// Inject concrete function pointers that know the `State<DB>` type so
// the generic executor can reseed block hashes and read `bal_index`.
// Use create_executor_with_seeder to inject a concrete seeder that
// can reseed State::block_hashes at segment boundaries. The seeder
// is a function pointer that knows the concrete State<DB> type,
// allowing the generic BbBlockExecutor to reseed without additional
// trait bounds on DB.
self.executor_factory.create_executor_with_seeder(
evm,
ctx,
Some(seed_state_block_hashes::<DB>),
bal_index_reader,
)
}
}
@@ -225,7 +214,6 @@ where
Ok(env)
} else {
self.executor_factory.clear_staged_plan();
self.inner.evm_env_for_payload(payload)
}
}
@@ -260,12 +248,10 @@ where
/// In practice, this is called from `evm_env_for_payload` in the
/// engine pipeline.
pub fn stage_plan_for_payload(&self, payload_hash: &B256) {
let Some(plan) = self.plan_for_payload_hash(payload_hash) else { return };
self.executor_factory.stage_plan(plan);
}
fn plan_for_payload_hash(&self, payload_hash: &B256) -> Option<BbEvmPlan> {
let bb = self.pending.lock().unwrap().remove(payload_hash)?;
let bb = match self.pending.lock().unwrap().remove(payload_hash) {
Some(bb) => bb,
None => return,
};
let segments: Vec<_> = bb
.env_switches
@@ -301,6 +287,6 @@ where
plan.block_hashes_to_seed.sort_unstable_by_key(|(n, _)| *n);
Some(plan)
self.executor_factory.stage_plan(plan);
}
}

View File

@@ -39,7 +39,7 @@ Both `new-payload-fcu` and `new-payload-only` support `--rpc-block-fetch-retries
to control how many times block fetches are retried after an RPC failure. The default is `10`.
Use `--rpc-block-fetch-retries forever` to keep retrying indefinitely.
When using `--wait-for-persistence`, the benchmark waits after every `(threshold + 1)` blocks, where the threshold defaults to the engine's persistence threshold (2). This can be customized with `--persistence-threshold <N>`.
When using `--wait-for-persistence`, the benchmark waits after every `(threshold + 1)` blocks, where the threshold defaults to the engine's persistence threshold. This can be customized with `--persistence-threshold <N>`.
By default, the WebSocket URL for persistence subscriptions is derived from `--engine-rpc-url` (converting to ws:// on port 8546). Use `--ws-rpc-url` to override this.

View File

@@ -21,7 +21,6 @@ use alloy_rpc_types_engine::{
};
use clap::Parser;
use eyre::Context;
use futures::{stream, StreamExt};
use reth_chainspec::EthChainSpec;
use reth_cli::chainspec::ChainSpecParser;
use reth_cli_runner::CliContext;
@@ -30,10 +29,7 @@ use reth_ethereum_cli::chainspec::EthereumChainSpecParser;
use reth_ethereum_primitives::Receipt;
use reth_primitives_traits::proofs;
use serde::{Deserialize, Serialize};
use std::{
collections::{HashMap, HashSet},
future::Future,
};
use std::{collections::HashMap, future::Future};
use tracing::{info, warn};
use crate::bench::helpers::fetch_block_access_list;
@@ -271,15 +267,6 @@ pub struct Command {
/// the flattened BAL on the stored payload.
#[arg(long, default_value_t = false)]
bal: bool,
/// Maximum number of in-flight RPC fetches to keep buffered ahead of the merger.
///
/// Each entry is one full per-block fetch (block + receipts, plus BAL when `--bal` is
/// set). Larger values absorb RPC latency at the cost of more concurrent connections
/// and memory; the buffer persists across `--num-big-blocks` so prefetching continues
/// across big-block boundaries.
#[arg(long, value_name = "PREFETCH_BUFFER", default_value_t = 32)]
prefetch_buffer: usize,
}
impl Command {
@@ -332,27 +319,13 @@ impl Command {
}
let mut prev_big_block_header: Option<PrevBigBlockHeader> = None;
// Persistent prefetch stream: keeps `prefetch_buffer` per-block fetches in flight
// ahead of the merger across all big blocks. Each item is a fully materialized
// `FetchedBlock` (or `None` once the chain tip is reached on this fetch).
let prefetch_buffer = self.prefetch_buffer.max(1);
let bal_enabled = self.bal;
let block_stream = stream::iter(self.from_block..)
.map(|block_number| {
let provider = provider.clone();
async move { fetch_one_block(provider, block_number, bal_enabled).await }
})
.buffered(prefetch_buffer);
let mut block_stream = Box::pin(block_stream);
// Track the next block number we expect from the stream (purely for logging /
// big-block range bookkeeping; the stream produces blocks in `from_block..` order).
// Track the next block to fetch across big blocks so they don't overlap.
let mut next_block = self.from_block;
for big_block_idx in 0..self.num_big_blocks {
let range_start = next_block;
// Drain the prefetch stream until the gas target is reached for this big block.
// Fetch consecutive blocks until the gas target is reached.
let mut blocks = Vec::new();
let mut block_receipts: Vec<Vec<Receipt>> = Vec::new();
let mut block_access_lists: Vec<Option<BlockAccessList>> = Vec::new();
@@ -361,11 +334,16 @@ impl Command {
let mut reached_chain_tip = false;
while accumulated_block_gas < self.target_gas {
let block_number = next_block;
info!(target: "reth-bench", block_number, big_block = big_block_idx, "Awaiting prefetched block");
info!(target: "reth-bench", block_number, big_block = big_block_idx, "Fetching block");
let fetched = match block_stream.next().await {
Some(Ok(Some(fetched))) => fetched,
Some(Ok(None)) => {
let fetch_result = tokio::try_join!(
provider.get_block_by_number(block_number.into()).full(),
provider.get_block_receipts(block_number.into()),
);
let (rpc_block, receipts) = match fetch_result {
Ok((Some(block), Some(receipts))) => (block, receipts),
Ok((None, _) | (_, None)) => {
warn!(
target: "reth-bench",
block_number,
@@ -374,16 +352,52 @@ impl Command {
reached_chain_tip = true;
break;
}
Some(Err(e)) => return Err(e),
// The block-number stream is open-ended; this only fires if the
// upstream `iter(from..)` is somehow exhausted.
None => {
reached_chain_tip = true;
break;
}
Err(e) => return Err(e.into()),
};
let FetchedBlock { execution_data, consensus_receipts, block_access_list } =
fetched;
let block_access_list = if self.bal {
Some(fetch_block_access_list(&provider, block_number).await.wrap_err_with(
|| format!("Failed to fetch BAL for block {block_number}"),
)?)
} else {
None
};
// Convert RPC receipts to consensus receipts
let consensus_receipts: Vec<Receipt> = receipts
.iter()
.map(|r| {
let inner = &r.inner.inner.inner;
let tx_type = r.inner.inner.r#type.try_into().unwrap_or_default();
Receipt {
tx_type,
success: inner.receipt.status.coerce_status(),
cumulative_gas_used: inner.receipt.cumulative_gas_used,
logs: inner
.receipt
.logs
.iter()
.map(|log| alloy_primitives::Log {
address: log.inner.address,
data: log.inner.data.clone(),
})
.collect(),
}
})
.collect();
// Convert to consensus block
let block = rpc_block
.into_inner()
.map_header(|header| header.map(|h| h.into_header_with_defaults()))
.try_map_transactions(|tx| -> eyre::Result<TxEnvelope> {
tx.try_into().map_err(|_| eyre::eyre!("unsupported tx type"))
})?
.into_consensus();
// Convert to ExecutionData
let (payload, sidecar) = ExecutionPayload::from_block_slow(&block);
let execution_data = ExecutionData { payload, sidecar };
let block_gas = execution_data.payload.as_v1().gas_used;
let block_blob_gas =
@@ -657,79 +671,6 @@ impl Command {
}
}
/// One fully-materialized block fetched by the prefetcher.
struct FetchedBlock {
/// Execution payload with sidecar derived from the RPC block.
execution_data: ExecutionData,
/// Consensus-format receipts (`cumulative_gas_used` is still per-block, callers offset
/// it when merging).
consensus_receipts: Vec<Receipt>,
/// `eth_getBlockAccessListByBlockNumber` result when `--bal` is enabled.
block_access_list: Option<BlockAccessList>,
}
/// Fetches one block + receipts (and optionally its BAL) from the RPC. Returns `Ok(None)`
/// when the block doesn't exist yet (chain-tip reached).
async fn fetch_one_block(
provider: RootProvider<AnyNetwork>,
block_number: u64,
bal_enabled: bool,
) -> eyre::Result<Option<FetchedBlock>> {
let (rpc_block, receipts) = tokio::try_join!(
provider.get_block_by_number(block_number.into()).full(),
provider.get_block_receipts(block_number.into()),
)?;
let (rpc_block, receipts) = match (rpc_block, receipts) {
(Some(b), Some(r)) => (b, r),
_ => return Ok(None),
};
let block_access_list = if bal_enabled {
Some(
fetch_block_access_list(&provider, block_number)
.await
.wrap_err_with(|| format!("Failed to fetch BAL for block {block_number}"))?,
)
} else {
None
};
let consensus_receipts: Vec<Receipt> = receipts
.iter()
.map(|r| {
let inner = &r.inner.inner.inner;
let tx_type = r.inner.inner.r#type.try_into().unwrap_or_default();
Receipt {
tx_type,
success: inner.receipt.status.coerce_status(),
cumulative_gas_used: inner.receipt.cumulative_gas_used,
logs: inner
.receipt
.logs
.iter()
.map(|log| alloy_primitives::Log {
address: log.inner.address,
data: log.inner.data.clone(),
})
.collect(),
}
})
.collect();
let block = rpc_block
.into_inner()
.map_header(|header| header.map(|h| h.into_header_with_defaults()))
.try_map_transactions(|tx| -> eyre::Result<TxEnvelope> {
tx.try_into().map_err(|_| eyre::eyre!("unsupported tx type"))
})?
.into_consensus();
let (payload, sidecar) = ExecutionPayload::from_block_slow(&block);
let execution_data = ExecutionData { payload, sidecar };
Ok(Some(FetchedBlock { execution_data, consensus_receipts, block_access_list }))
}
fn merge_block_access_list(
merged: &mut BlockAccessList,
incoming: BlockAccessList,
@@ -776,17 +717,6 @@ fn merge_account_changes(existing: &mut AccountChanges, incoming: AccountChanges
existing.balance_changes.extend(incoming.balance_changes);
existing.nonce_changes.extend(incoming.nonce_changes);
existing.code_changes.extend(incoming.code_changes);
// EIP-7928 invariant: a slot must appear in either storage_changes or storage_reads,
// not both. Per-block BALs respect this, but merging blocks can produce a slot
// that is read in one block and changed in another. Without this normalization,
// an empty read entry can shadow the real writes during BAL deserialization,
// making reads of that slot fall through to stale snapshot state.
let written: HashSet<_> =
existing.storage_changes.iter().map(|slot_changes| slot_changes.slot).collect();
existing.storage_reads.retain(|slot| !written.contains(slot));
let mut seen = HashSet::with_capacity(existing.storage_reads.len());
existing.storage_reads.retain(|slot| seen.insert(*slot));
}
fn merge_slot_changes(existing: &mut Vec<SlotChanges>, incoming: Vec<SlotChanges>) {
@@ -906,54 +836,4 @@ mod tests {
assert_eq!(other.address, Address::repeat_byte(0x22));
assert_eq!(other.storage_changes[0].changes[0].block_access_index, 3);
}
#[test]
fn merge_account_changes_normalizes_storage_reads_after_cross_block_merge() {
let address = Address::repeat_byte(0x33);
const A: U256 = U256::from_limbs([1, 0, 0, 0]);
const B: U256 = U256::from_limbs([2, 0, 0, 0]);
const C: U256 = U256::from_limbs([3, 0, 0, 0]);
const D: U256 = U256::from_limbs([4, 0, 0, 0]);
// Each AccountChanges value is valid on its own: storage slots only appear in
// either reads or changes. The invalid read/change overlap is introduced when
// these per-block BAL entries are merged for a standalone big block.
let mut existing = AccountChanges {
address,
storage_changes: vec![SlotChanges::new(A, vec![StorageChange::new(0, U256::from(10))])],
storage_reads: vec![B, C],
balance_changes: vec![],
nonce_changes: vec![],
code_changes: vec![],
};
// B is read before it is written by the incoming block, and A is written before
// it appears as a read in the incoming block. C is read in both blocks, so the
// merge should also dedupe it. D remains read-only.
let incoming = AccountChanges {
address,
storage_changes: vec![SlotChanges::new(B, vec![StorageChange::new(1, U256::from(20))])],
storage_reads: vec![A, C, D],
balance_changes: vec![],
nonce_changes: vec![],
code_changes: vec![],
};
merge_account_changes(&mut existing, incoming);
// Written slots remain represented by storage_changes, while storage_reads only
// keeps unique read-only slots in first-seen order.
assert_eq!(
existing
.storage_changes
.iter()
.map(|slot_changes| slot_changes.slot)
.collect::<Vec<_>>(),
vec![A, B]
);
assert_eq!(existing.storage_reads, vec![C, D]);
assert!(existing.storage_reads.iter().all(|read_slot| {
!existing.storage_changes.iter().any(|slot_changes| slot_changes.slot == *read_slot)
}));
}
}

View File

@@ -67,9 +67,8 @@ pub struct Command {
/// Engine persistence threshold used for deciding when to wait for persistence.
///
/// The benchmark waits after every `(threshold + 1)` blocks. By default this
/// matches the engine's `DEFAULT_PERSISTENCE_THRESHOLD` (2), so waits occur
/// at blocks 3, 6, 9, etc.
/// The benchmark waits after every `(threshold + 1)` blocks.
/// By default this matches the engine's `DEFAULT_PERSISTENCE_THRESHOLD`.
#[arg(
long = "persistence-threshold",
value_name = "PERSISTENCE_THRESHOLD",

View File

@@ -320,6 +320,19 @@ impl<N: NodePrimitives> CanonicalInMemoryState<N> {
/// This will update the links between blocks and remove all blocks that are [..
/// `persisted_height`].
pub fn remove_persisted_blocks(&self, persisted_num_hash: BlockNumHash) {
self.remove_persisted_blocks_until(persisted_num_hash, persisted_num_hash.number);
}
/// Removes blocks from the in-memory state through `remove_until` while still reporting the
/// provided block as the persisted tip.
///
/// This is used when block bodies/plain state have been persisted further than trie data, so a
/// suffix still needs to remain in memory for trie-backed operations.
pub fn remove_persisted_blocks_until(
&self,
persisted_num_hash: BlockNumHash,
remove_until: BlockNumber,
) {
self.set_persisted(persisted_num_hash);
// if the persisted hash is not in the canonical in memory state, do nothing, because it
// means canonical blocks were not actually persisted.
@@ -337,16 +350,15 @@ impl<N: NodePrimitives> CanonicalInMemoryState<N> {
let mut numbers = self.inner.in_memory_state.numbers.write();
let mut blocks = self.inner.in_memory_state.blocks.write();
let BlockNumHash { number: persisted_height, hash: _ } = persisted_num_hash;
let remove_until = remove_until.min(persisted_num_hash.number);
// clear all numbers
numbers.clear();
// drain all blocks and only keep the ones that are not persisted (below the persisted
// height)
// Drain all blocks and keep only the suffix that still has to stay in memory.
let mut old_blocks = blocks
.drain()
.filter(|(_, b)| b.block_ref().recovered_block().number() > persisted_height)
.filter(|(_, b)| b.block_ref().recovered_block().number() > remove_until)
.map(|(_, b)| b.block.clone())
.collect::<Vec<_>>();

View File

@@ -20,10 +20,7 @@ use reth_provider::{
};
use reth_revm::{
database::StateProviderDatabase,
db::{
states::reverts::{AccountInfoRevert, RevertToSlot},
BundleState,
},
db::{states::reverts::AccountInfoRevert, BundleState},
};
use reth_stages::stages::calculate_gas_used_from_headers;
use reth_storage_api::{ChangeSetReader, DBProvider, StorageChangeSetReader};
@@ -428,19 +425,14 @@ where
let mut cs_slots = cs_storage.get_mut(addr);
for (slot_key, revert_slot) in &revert.storage {
let b256_key = B256::from(*slot_key);
let cs_value = cs_slots.as_mut().and_then(|s| s.remove(&b256_key));
match (revert_slot, cs_value) {
// When a contract is selfdestructed and re-created at the same address
// within the same block, revm marks slots touched by the new contract
// as `Destroyed` and never reads the original DB value, so
// `to_previous_value()` would resolve to zero, which might be wrong.
(RevertToSlot::Destroyed, _) => {}
(RevertToSlot::Some(prev), Some(cs_value)) => eyre::ensure!(
*prev == cs_value,
match cs_slots.as_mut().and_then(|s| s.remove(&b256_key)) {
Some(cs_value) => eyre::ensure!(
revert_slot.to_previous_value() == cs_value,
"Block {block_number}: {addr} slot {b256_key} mismatch: \
revert={prev} cs={cs_value}",
revert={} cs={cs_value}",
revert_slot.to_previous_value(),
),
(RevertToSlot::Some(_), None) => eyre::ensure!(
None => eyre::ensure!(
revert.wipe_storage,
"Block {block_number}: {addr} slot {b256_key} in reverts but not in changeset",
),

View File

@@ -1,8 +1,8 @@
use futures_util::StreamExt;
use reth_node_api::{PayloadAttributes, PayloadKind};
use reth_node_api::{BlockBody, PayloadAttributes, PayloadKind};
use reth_payload_builder::{PayloadBuilderHandle, PayloadId};
use reth_payload_builder_primitives::Events;
use reth_payload_primitives::PayloadTypes;
use reth_payload_primitives::{BuiltPayload, PayloadTypes};
use tokio_stream::wrappers::BroadcastStream;
/// Helper for payload operations
@@ -53,11 +53,27 @@ impl<T: PayloadTypes> PayloadTestContext<T> {
///
/// Panics if the payload builder does not produce a non-empty payload within 30 seconds.
pub async fn wait_for_built_payload(&self, payload_id: PayloadId) {
self.payload_builder
.resolve_kind(payload_id, PayloadKind::WaitForPending)
.await
.unwrap()
.unwrap();
let start = std::time::Instant::now();
loop {
let payload =
self.payload_builder.best_payload(payload_id).await.transpose().ok().flatten();
if payload.is_none_or(|p| p.block().body().transactions().is_empty()) {
assert!(
start.elapsed() < std::time::Duration::from_secs(30),
"timed out waiting for a non-empty payload for {payload_id} — \
check that the chain spec supports all generated tx types"
);
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
continue
}
// Resolve payload once its built
self.payload_builder
.resolve_kind(payload_id, PayloadKind::Earliest)
.await
.unwrap()
.unwrap();
break;
}
}
/// Expects the next event to be a built payload event or panics

View File

@@ -374,7 +374,7 @@ async fn test_setup_builder_with_custom_tree_config() -> Result<()> {
PayloadAttributes::default()
})
.with_tree_config_modifier(|config| {
config.with_persistence_threshold(0).with_memory_block_buffer_target(5)
config.with_persistence_threshold(6).with_memory_block_buffer_target(5)
})
.build()
.await?;

View File

@@ -189,7 +189,7 @@ async fn test_rocksdb_transaction_queries() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.build()
.await?;
@@ -200,7 +200,7 @@ async fn test_rocksdb_transaction_queries() -> Result<()> {
let signer = wallets[0].clone();
let client = nodes[0].rpc_client().expect("RPC client should be available");
let raw_tx = TransactionTestContext::transfer_tx_bytes(chain_id, signer).await;
let raw_tx = TransactionTestContext::transfer_tx_bytes(chain_id, signer.clone()).await;
let tx_hash = nodes[0].rpc.inject_tx(raw_tx).await?;
// Wait for tx to enter pending pool before mining
@@ -209,6 +209,14 @@ async fn test_rocksdb_transaction_queries() -> Result<()> {
let payload = nodes[0].advance_block().await?;
assert_eq!(payload.block().number(), 1);
let flush_tx =
TransactionTestContext::transfer_tx_bytes_with_nonce(chain_id, signer.clone(), 1).await;
let flush_tx_hash = nodes[0].rpc.inject_tx(flush_tx).await?;
wait_for_pending_tx(&client, flush_tx_hash).await;
let flush_payload = nodes[0].advance_block().await?;
assert_eq!(flush_payload.block().number(), 2);
// Query each transaction by hash
let tx: Option<Transaction> = client.request("eth_getTransactionByHash", [tx_hash]).await?;
let tx = tx.expect("Transaction should be found");
@@ -256,7 +264,7 @@ async fn test_rocksdb_multi_tx_same_block() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.build()
.await?;
@@ -283,6 +291,14 @@ async fn test_rocksdb_multi_tx_same_block() -> Result<()> {
let payload = nodes[0].advance_block().await?;
assert_eq!(payload.block().number(), 1);
let flush_tx =
TransactionTestContext::transfer_tx_bytes_with_nonce(chain_id, signer.clone(), 3).await;
let flush_tx_hash = nodes[0].rpc.inject_tx(flush_tx).await?;
wait_for_pending_tx(&client, flush_tx_hash).await;
let flush_payload = nodes[0].advance_block().await?;
assert_eq!(flush_payload.block().number(), 2);
// Verify block contains all 3 txs
let block: Option<alloy_rpc_types_eth::Block> =
client.request("eth_getBlockByNumber", ("0x1", true)).await?;
@@ -324,7 +340,7 @@ async fn test_rocksdb_txs_across_blocks() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.build()
.await?;
@@ -409,7 +425,7 @@ async fn test_rocksdb_pending_tx_not_in_storage() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.build()
.await?;
@@ -417,7 +433,7 @@ async fn test_rocksdb_pending_tx_not_in_storage() -> Result<()> {
let signer = wallets[0].clone();
// Inject tx but do NOT mine
let raw_tx = TransactionTestContext::transfer_tx_bytes(chain_id, signer).await;
let raw_tx = TransactionTestContext::transfer_tx_bytes(chain_id, signer.clone()).await;
let tx_hash = nodes[0].rpc.inject_tx(raw_tx).await?;
// Verify tx is in pending pool via RPC
@@ -442,6 +458,14 @@ async fn test_rocksdb_pending_tx_not_in_storage() -> Result<()> {
let payload = nodes[0].advance_block().await?;
assert_eq!(payload.block().number(), 1);
let flush_tx =
TransactionTestContext::transfer_tx_bytes_with_nonce(chain_id, signer.clone(), 1).await;
let flush_tx_hash = nodes[0].rpc.inject_tx(flush_tx).await?;
wait_for_pending_tx(&client, flush_tx_hash).await;
let flush_payload = nodes[0].advance_block().await?;
assert_eq!(flush_payload.block().number(), 2);
// Poll until tx appears in RocksDB
let tx_number = poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash).await;
assert_eq!(tx_number, 0, "First tx should have tx_number 0");
@@ -473,7 +497,7 @@ async fn test_rocksdb_reorg_unwind() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.build()
.await?;
@@ -495,10 +519,6 @@ async fn test_rocksdb_reorg_unwind() -> Result<()> {
let block1_hash = payload1.block().hash();
assert_eq!(payload1.block().number(), 1);
// Poll until tx1 appears in RocksDB (ensures persistence happened)
let tx_number1 = poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash1).await;
assert_eq!(tx_number1, 0, "First tx should have tx_number 0");
// Mine block 2 with transaction from signer1 (nonce 1)
let raw_tx2 =
TransactionTestContext::transfer_tx_bytes_with_nonce(chain_id, signer1.clone(), 1).await;
@@ -508,6 +528,10 @@ async fn test_rocksdb_reorg_unwind() -> Result<()> {
let payload2 = nodes[0].advance_block().await?;
assert_eq!(payload2.block().number(), 2);
// The second block triggers the first persistence cycle, which flushes both block 1 and 2.
let tx_number1 = poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash1).await;
assert_eq!(tx_number1, 0, "First tx should have tx_number 0");
// Poll until tx2 appears in RocksDB
let tx_number2 = poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash2).await;
assert_eq!(tx_number2, 1, "Second tx should have tx_number 1");
@@ -521,6 +545,14 @@ async fn test_rocksdb_reorg_unwind() -> Result<()> {
let payload3 = nodes[0].advance_block().await?;
assert_eq!(payload3.block().number(), 3);
let flush_tx =
TransactionTestContext::transfer_tx_bytes_with_nonce(chain_id, signer1.clone(), 3).await;
let flush_tx_hash = nodes[0].rpc.inject_tx(flush_tx).await?;
wait_for_pending_tx(&client, flush_tx_hash).await;
let flush_payload = nodes[0].advance_block().await?;
assert_eq!(flush_payload.block().number(), 4);
// Poll until tx3 appears in RocksDB
let tx_number3 = poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash3).await;
assert_eq!(tx_number3, 2, "Third tx should have tx_number 2");
@@ -532,7 +564,7 @@ async fn test_rocksdb_reorg_unwind() -> Result<()> {
let alt_tx_hash = nodes[0].rpc.inject_tx(raw_alt_tx).await?;
wait_for_pending_tx(&client, alt_tx_hash).await;
// Build an alternate payload (this builds on top of the current head, i.e., block 3)
// Build an alternate payload on top of the current flushed head.
// But we want to reorg back to block 1, so we'll use the payload and then FCU to it
let alt_payload = nodes[0].new_payload().await?;
let alt_block_hash = nodes[0].submit_payload(alt_payload.clone()).await?;
@@ -550,8 +582,8 @@ async fn test_rocksdb_reorg_unwind() -> Result<()> {
let latest: Option<alloy_rpc_types_eth::Block> =
client.request("eth_getBlockByNumber", ("latest", false)).await?;
let latest = latest.expect("Latest block should exist");
// The alt block is at height 4 (on top of block 3)
assert!(latest.header.number >= 3, "Should be at height >= 3 after operation");
// The alt block is built on top of the flushed canonical head.
assert!(latest.header.number >= 4, "Should be at height >= 4 after operation");
// tx1 from block 1 should still be there
let tx1: Option<Transaction> = client.request("eth_getTransactionByHash", [tx_hash1]).await?;
@@ -596,7 +628,7 @@ async fn test_rocksdb_historical_account_queries() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.build()
.await?;
@@ -621,8 +653,6 @@ async fn test_rocksdb_historical_account_queries() -> Result<()> {
let payload1 = nodes[0].advance_block().await?;
assert_eq!(payload1.block().number(), 1);
poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash1).await;
// Record state after block 1
let balance_at_1: U256 = client.request("eth_getBalance", (sender, "0x1")).await?;
let nonce_at_1: U256 = client.request("eth_getTransactionCount", (sender, "0x1")).await?;
@@ -637,8 +667,6 @@ async fn test_rocksdb_historical_account_queries() -> Result<()> {
let payload2 = nodes[0].advance_block().await?;
assert_eq!(payload2.block().number(), 2);
poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash2).await;
let balance_at_2: U256 = client.request("eth_getBalance", (sender, "0x2")).await?;
let nonce_at_2: U256 = client.request("eth_getTransactionCount", (sender, "0x2")).await?;
assert!(balance_at_2 < balance_at_1, "Balance should decrease further after second tx");
@@ -652,18 +680,14 @@ async fn test_rocksdb_historical_account_queries() -> Result<()> {
let payload3 = nodes[0].advance_block().await?;
assert_eq!(payload3.block().number(), 3);
poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash3).await;
let balance_at_3: U256 = client.request("eth_getBalance", (sender, "0x3")).await?;
let nonce_at_3: U256 = client.request("eth_getTransactionCount", (sender, "0x3")).await?;
assert!(balance_at_3 < balance_at_2, "Balance should decrease further after third tx");
assert_eq!(nonce_at_3, U256::from(3), "Nonce should be 3 after third tx");
// Mine additional blocks to push blocks 1-3 out of the in-memory overlay.
// With persistence_threshold=0 and memory_block_buffer_target=0, each new block
// triggers persistence up to `head` followed by in-memory eviction. Mining several
// more blocks ensures the engine loop has completed at least one full
// persist-then-evict cycle covering blocks 1-3.
// With a persistence threshold of 1, every second block triggers a flush, so a few extra
// blocks are enough to durably persist and evict the earlier history we want to query.
// Each block needs a transaction because the payload builder requires non-empty payloads.
for nonce in 3..8u64 {
let raw_tx =
@@ -673,6 +697,7 @@ async fn test_rocksdb_historical_account_queries() -> Result<()> {
wait_for_pending_tx(&client, tx_hash).await;
nodes[0].advance_block().await?;
}
poll_tx_in_rocksdb(&nodes[0].inner.provider, tx_hash3).await;
// Allow the engine loop to process the persistence completions
tokio::time::sleep(Duration::from_millis(500)).await;
@@ -743,7 +768,7 @@ async fn test_rocksdb_account_history_pruning() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.with_node_config_modifier(|mut config| {
config.pruning.account_history_distance = Some(PRUNE_DISTANCE);
config.pruning.minimum_distance = Some(PRUNE_DISTANCE);
@@ -840,7 +865,7 @@ async fn test_rocksdb_storage_history_pruning() -> Result<()> {
test_attributes_generator,
)
.with_storage_v2()
.with_tree_config_modifier(|config| config.with_persistence_threshold(0))
.with_tree_config_modifier(|config| config.with_persistence_threshold(1))
.with_node_config_modifier(|mut config| {
config.pruning.storage_history_distance = Some(PRUNE_DISTANCE);
config.pruning.minimum_distance = Some(PRUNE_DISTANCE);
@@ -912,10 +937,6 @@ async fn test_rocksdb_storage_history_pruning() -> Result<()> {
let payload1 = nodes[0].advance_block().await?;
assert_eq!(payload1.block().number(), 1);
poll_tx_in_rocksdb(&nodes[0].inner.provider, deploy_hash).await;
// Let the persistence cycle complete before the next block (same cadence as the loop below)
tokio::time::sleep(Duration::from_millis(300)).await;
// Get the deployed contract address from the receipt
let receipt: Option<TransactionReceipt> =
@@ -965,6 +986,10 @@ async fn test_rocksdb_storage_history_pruning() -> Result<()> {
assert_eq!(payload.block().number(), block_num);
last_tx_hash = tx_hash;
if nonce == 1 {
poll_tx_in_rocksdb(&nodes[0].inner.provider, deploy_hash).await;
}
// Let the persistence cycle complete before the next block
tokio::time::sleep(Duration::from_millis(300)).await;
}

View File

@@ -37,6 +37,9 @@ auto_impl.workspace = true
serde.workspace = true
thiserror.workspace = true
[dev-dependencies]
alloy-primitives = { workspace = true, features = ["getrandom"] }
[features]
default = ["std"]
trie-debug = []

View File

@@ -6,12 +6,33 @@ use core::time::Duration;
/// Triggers persistence when the number of canonical blocks in memory exceeds this threshold.
pub const DEFAULT_PERSISTENCE_THRESHOLD: u64 = 2;
/// Maximum canonical-minus-persisted gap before engine API processing is stalled.
pub const DEFAULT_PERSISTENCE_BACKPRESSURE_THRESHOLD: u64 = 16;
/// Maximum number of consecutive canonical blocks whose non-trie outputs may be persisted ahead
/// of trie persistence.
pub const DEFAULT_DEFERRED_TRIE_BLOCKS: u64 = 0;
/// How close to the canonical head we persist blocks.
pub const DEFAULT_MEMORY_BLOCK_BUFFER_TARGET: u64 = 0;
/// Derives the default canonical-minus-persisted gap that triggers backpressure.
pub const fn default_persistence_backpressure_threshold(
persistence_threshold: u64,
memory_block_buffer_target: u64,
) -> u64 {
let threshold = 2 * (persistence_threshold + memory_block_buffer_target);
if threshold < 16 {
16
} else {
threshold
}
}
/// Maximum canonical-minus-persisted gap before engine API processing is stalled.
pub const DEFAULT_PERSISTENCE_BACKPRESSURE_THRESHOLD: u64 =
default_persistence_backpressure_threshold(
DEFAULT_PERSISTENCE_THRESHOLD,
DEFAULT_MEMORY_BLOCK_BUFFER_TARGET,
);
/// The size of proof targets chunk to spawn in one multiproof calculation.
pub const DEFAULT_MULTIPROOF_TASK_CHUNK_SIZE: usize = 5;
@@ -60,6 +81,17 @@ const fn assert_backpressure_threshold_invariant(
);
}
const fn assert_state_masking_invariant(
persistence_threshold: u64,
num_state_masking_blocks: u64,
memory_block_buffer_target: u64,
) {
debug_assert!(
num_state_masking_blocks + memory_block_buffer_target < persistence_threshold,
"num_state_masking_blocks + memory_block_buffer_target must be less than persistence_threshold",
);
}
const fn default_cross_block_cache_size() -> usize {
if cfg!(test) {
1024 * 1024 // 1 MB in tests
@@ -93,6 +125,9 @@ pub struct TreeConfig {
/// Maximum number of blocks to be kept only in memory without triggering
/// persistence.
persistence_threshold: u64,
/// Number of persisted blocks whose state/trie writes are masked instead of being durably
/// written in the current cycle.
num_state_masking_blocks: u64,
/// How close to the canonical head we persist blocks. Represents the ideal
/// number of most recent blocks to keep in memory for quick access and reorgs.
///
@@ -191,9 +226,9 @@ pub struct TreeConfig {
/// When disabled, the BAL hashed post state is not sent to the multiproof task for
/// early parallel state root computation.
disable_bal_parallel_state_root: bool,
/// Whether to disable BAL (Block Access List) storage prefetch IO during prewarming.
/// When set, BAL storage slots are not read into the execution cache. BAL hashed-state
/// streaming for parallel state-root computation is controlled separately.
/// Whether to disable BAL (Block Access List) batched IO during prewarming.
/// When disabled, falls back to individual per-slot storage reads instead of
/// batched cursor reads via `storage_range`.
disable_bal_batch_io: bool,
/// Maximum random jitter applied before each proof computation (trie-debug only).
/// When set, each proof worker sleeps for a random duration up to this value
@@ -204,14 +239,24 @@ pub struct TreeConfig {
impl Default for TreeConfig {
fn default() -> Self {
let persistence_backpressure_threshold = default_persistence_backpressure_threshold(
DEFAULT_PERSISTENCE_THRESHOLD,
DEFAULT_MEMORY_BLOCK_BUFFER_TARGET,
);
assert_backpressure_threshold_invariant(
DEFAULT_PERSISTENCE_THRESHOLD,
DEFAULT_PERSISTENCE_BACKPRESSURE_THRESHOLD,
persistence_backpressure_threshold,
);
assert_state_masking_invariant(
DEFAULT_PERSISTENCE_THRESHOLD,
DEFAULT_DEFERRED_TRIE_BLOCKS,
DEFAULT_MEMORY_BLOCK_BUFFER_TARGET,
);
Self {
persistence_threshold: DEFAULT_PERSISTENCE_THRESHOLD,
num_state_masking_blocks: DEFAULT_DEFERRED_TRIE_BLOCKS,
memory_block_buffer_target: DEFAULT_MEMORY_BLOCK_BUFFER_TARGET,
persistence_backpressure_threshold: DEFAULT_PERSISTENCE_BACKPRESSURE_THRESHOLD,
persistence_backpressure_threshold,
block_buffer_limit: DEFAULT_BLOCK_BUFFER_LIMIT,
max_invalid_header_cache_length: DEFAULT_MAX_INVALID_HEADER_CACHE_LENGTH,
invalid_header_hit_eviction_threshold: DEFAULT_INVALID_HEADER_HIT_EVICTION_THRESHOLD,
@@ -239,7 +284,7 @@ impl Default for TreeConfig {
share_execution_cache_with_payload_builder: false,
share_sparse_trie_with_payload_builder: false,
suppress_persistence_during_build: false,
disable_bal_parallel_execution: true,
disable_bal_parallel_execution: false,
disable_bal_parallel_state_root: false,
disable_bal_batch_io: false,
#[cfg(feature = "trie-debug")]
@@ -253,6 +298,7 @@ impl TreeConfig {
#[expect(clippy::too_many_arguments)]
pub const fn new(
persistence_threshold: u64,
num_state_masking_blocks: u64,
memory_block_buffer_target: u64,
persistence_backpressure_threshold: u64,
block_buffer_limit: u32,
@@ -285,8 +331,14 @@ impl TreeConfig {
persistence_threshold,
persistence_backpressure_threshold,
);
assert_state_masking_invariant(
persistence_threshold,
num_state_masking_blocks,
memory_block_buffer_target,
);
Self {
persistence_threshold,
num_state_masking_blocks,
memory_block_buffer_target,
persistence_backpressure_threshold,
block_buffer_limit,
@@ -316,7 +368,7 @@ impl TreeConfig {
share_execution_cache_with_payload_builder,
share_sparse_trie_with_payload_builder,
suppress_persistence_during_build: false,
disable_bal_parallel_execution: true,
disable_bal_parallel_execution: false,
disable_bal_parallel_state_root: false,
disable_bal_batch_io: false,
#[cfg(feature = "trie-debug")]
@@ -329,6 +381,11 @@ impl TreeConfig {
self.persistence_threshold
}
/// Return the number of persisted blocks whose state/trie writes are masked.
pub const fn num_state_masking_blocks(&self) -> u64 {
self.num_state_masking_blocks
}
/// Return the memory block buffer target.
pub const fn memory_block_buffer_target(&self) -> u64 {
self.memory_block_buffer_target
@@ -447,6 +504,22 @@ impl TreeConfig {
self.persistence_threshold,
self.persistence_backpressure_threshold,
);
assert_state_masking_invariant(
self.persistence_threshold,
self.num_state_masking_blocks,
self.memory_block_buffer_target,
);
self
}
/// Setter for the number of persisted blocks whose state/trie writes are masked.
pub const fn with_num_state_masking_blocks(mut self, num_state_masking_blocks: u64) -> Self {
self.num_state_masking_blocks = num_state_masking_blocks;
assert_state_masking_invariant(
self.persistence_threshold,
self.num_state_masking_blocks,
self.memory_block_buffer_target,
);
self
}
@@ -456,6 +529,11 @@ impl TreeConfig {
memory_block_buffer_target: u64,
) -> Self {
self.memory_block_buffer_target = memory_block_buffer_target;
assert_state_masking_invariant(
self.persistence_threshold,
self.num_state_masking_blocks,
self.memory_block_buffer_target,
);
self
}
@@ -765,7 +843,26 @@ impl TreeConfig {
#[cfg(test)]
mod tests {
use super::TreeConfig;
use super::{
default_persistence_backpressure_threshold, TreeConfig, DEFAULT_DEFERRED_TRIE_BLOCKS,
DEFAULT_MEMORY_BLOCK_BUFFER_TARGET, DEFAULT_PERSISTENCE_THRESHOLD,
};
#[test]
fn default_thresholds_use_derived_backpressure_threshold() {
let config = TreeConfig::default();
assert_eq!(config.persistence_threshold(), DEFAULT_PERSISTENCE_THRESHOLD);
assert_eq!(config.num_state_masking_blocks(), DEFAULT_DEFERRED_TRIE_BLOCKS);
assert_eq!(config.memory_block_buffer_target(), DEFAULT_MEMORY_BLOCK_BUFFER_TARGET);
assert_eq!(
config.persistence_backpressure_threshold(),
default_persistence_backpressure_threshold(
DEFAULT_PERSISTENCE_THRESHOLD,
DEFAULT_MEMORY_BLOCK_BUFFER_TARGET,
)
);
}
#[test]
#[should_panic(
@@ -776,4 +873,15 @@ mod tests {
.with_persistence_threshold(4)
.with_persistence_backpressure_threshold(4);
}
#[test]
#[should_panic(
expected = "num_state_masking_blocks + memory_block_buffer_target must be less than persistence_threshold"
)]
fn rejects_state_masking_window_at_or_above_persistence_threshold() {
let _ = TreeConfig::default()
.with_persistence_threshold(4)
.with_num_state_masking_blocks(2)
.with_memory_block_buffer_target(2);
}
}

View File

@@ -21,8 +21,7 @@ impl ForkchoiceStateTracker {
/// `sync_target` to `None`, since we're now fully synced.
pub const fn set_latest(&mut self, state: ForkchoiceState, status: ForkchoiceStatus) {
if status.is_valid() {
self.last_syncing = None;
self.last_valid = Some(state);
self.set_valid(state);
} else if status.is_syncing() {
self.last_syncing = Some(state);
}
@@ -31,24 +30,11 @@ impl ForkchoiceStateTracker {
self.latest = Some(received);
}
/// Promotes a previously tracked syncing forkchoice state to valid, without overwriting a
/// newer `latest` state.
///
/// This is used when a `Syncing` FCU's head finally becomes canonical via the downloaded-block
/// flow, so the safe/finalized anchors of that FCU can be applied. Unlike
/// [`Self::set_latest`], this preserves a newer `latest` (e.g. an `Invalid` FCU received
/// after the syncing one) and only flips `latest` to `Valid` when it still refers to the same
/// syncing FCU being promoted.
pub fn promote_sync_target_to_valid(&mut self, state: ForkchoiceState) {
const fn set_valid(&mut self, state: ForkchoiceState) {
// we no longer need to sync to this state.
self.last_syncing = None;
self.last_valid = Some(state);
if let Some(received) = self.latest.as_mut() &&
received.state == state &&
received.status.is_syncing()
{
received.status = ForkchoiceStatus::Valid;
}
self.last_valid = Some(state);
}
/// Returns the [`ForkchoiceStatus`] of the latest received FCU.

View File

@@ -1,16 +1,16 @@
use crate::metrics::PersistenceMetrics;
use alloy_eips::BlockNumHash;
use crossbeam_channel::Sender as CrossbeamSender;
use reth_chain_state::ExecutedBlock;
use reth_errors::ProviderError;
use reth_ethereum_primitives::EthPrimitives;
use reth_primitives_traits::{FastInstant as Instant, NodePrimitives};
use reth_provider::{
providers::ProviderNodeTypes, BlockExecutionWriter, BlockHashReader, ChainStateBlockWriter,
DBProvider, DatabaseProviderFactory, ProviderFactory, SaveBlocksMode,
DBProvider, DatabaseProviderFactory, ProviderFactory, SaveBlocksMode, SaveBlocksPlan,
StageCheckpointReader,
};
use reth_prune::{PrunerError, PrunerWithFactory};
use reth_stages_api::{MetricEvent, MetricEventsSender};
use reth_stages_api::{MetricEvent, MetricEventsSender, StageId};
use reth_tasks::spawn_os_thread;
use std::{
sync::{
@@ -26,8 +26,13 @@ use tracing::{debug, error, instrument};
/// Unified result of any persistence operation.
#[derive(Debug)]
pub struct PersistenceResult {
/// The last block that was persisted, if any.
/// The highest block whose non-state/trie outputs are persisted, if any.
pub last_block: Option<BlockNumHash>,
/// The highest block whose state/trie data is fully persisted, if known.
///
/// When this lags behind [`Self::last_block`], callers must retain the suffix
/// above it in memory so trie-backed operations can still unwind from that point.
pub last_state_trie_block: Option<u64>,
/// The commit duration, only available for save-blocks operations.
pub commit_duration: Option<Duration>,
}
@@ -96,14 +101,14 @@ where
while let Ok(action) = self.incoming.recv() {
match action {
PersistenceAction::RemoveBlocksAbove(new_tip_num, sender) => {
let last_block = self.on_remove_blocks_above(new_tip_num)?;
let result = self.on_remove_blocks_above(new_tip_num)?;
// send new sync metrics based on removed blocks
let _ =
self.sync_metrics_tx.send(MetricEvent::SyncHeight { height: new_tip_num });
let _ = sender.send(PersistenceResult { last_block, commit_duration: None });
let _ = sender.send(result);
}
PersistenceAction::SaveBlocks(blocks, sender) => {
let result = self.on_save_blocks(blocks)?;
PersistenceAction::SaveBlocks(plan, sender) => {
let result = self.on_save_blocks(plan)?;
let result_number = result.last_block.map(|b| b.number);
let _ = sender.send(result);
@@ -130,28 +135,40 @@ where
fn on_remove_blocks_above(
&self,
new_tip_num: u64,
) -> Result<Option<BlockNumHash>, PersistenceError> {
) -> Result<PersistenceResult, PersistenceError> {
debug!(target: "engine::persistence", ?new_tip_num, "Removing blocks");
let start_time = Instant::now();
let provider_rw = self.provider.database_provider_rw()?;
let new_tip_hash = provider_rw.block_hash(new_tip_num)?;
provider_rw.remove_block_and_execution_above(new_tip_num)?;
let last_state_trie_block =
provider_rw.get_stage_checkpoint(StageId::Finish)?.map(|checkpoint| {
checkpoint
.finish_stage_checkpoint()
.and_then(|finish| finish.partial_state_trie)
.unwrap_or(checkpoint.block_number)
});
provider_rw.commit()?;
debug!(target: "engine::persistence", ?new_tip_num, ?new_tip_hash, "Removed blocks from disk");
self.metrics.remove_blocks_above_duration_seconds.record(start_time.elapsed());
Ok(new_tip_hash.map(|hash| BlockNumHash { hash, number: new_tip_num }))
Ok(PersistenceResult {
last_block: new_tip_hash.map(|hash| BlockNumHash { hash, number: new_tip_num }),
last_state_trie_block,
commit_duration: None,
})
}
#[instrument(level = "debug", target = "engine::persistence", skip_all, fields(block_count = blocks.len()))]
#[instrument(level = "debug", target = "engine::persistence", skip_all, fields(block_count = plan.blocks.len()))]
fn on_save_blocks(
&mut self,
blocks: Vec<ExecutedBlock<N::Primitives>>,
plan: SaveBlocksPlan<N::Primitives>,
) -> Result<PersistenceResult, PersistenceError> {
let first_block = blocks.first().map(|b| b.recovered_block.num_hash());
let last_block = blocks.last().map(|b| b.recovered_block.num_hash());
let block_count = blocks.len();
let first_block = plan.blocks.first().map(|block| block.recovered_block().num_hash());
let last_block = plan.last_block();
let block_count = plan.blocks.len();
let mut last_state_trie_block = None;
let pending_finalized = self.pending_finalized_block.take();
let pending_safe = self.pending_safe_block.take();
@@ -160,19 +177,27 @@ where
let start_time = Instant::now();
if let Some(last) = last_block {
if let Some(last_block) = last_block {
let provider_rw = self.provider.database_provider_rw()?;
provider_rw.save_blocks(blocks, SaveBlocksMode::Full)?;
provider_rw.save_blocks(&plan, SaveBlocksMode::Full)?;
last_state_trie_block = provider_rw
.get_stage_checkpoint(StageId::Finish)?
.and_then(|checkpoint| {
checkpoint
.finish_stage_checkpoint()
.and_then(|finish| finish.partial_state_trie)
})
.or(Some(last_block.number));
if let Some(finalized) = pending_finalized {
provider_rw.save_finalized_block_number(finalized.min(last.number))?;
if finalized > last.number {
provider_rw.save_finalized_block_number(finalized.min(last_block.number))?;
if finalized > last_block.number {
self.pending_finalized_block = Some(finalized);
}
}
if let Some(safe) = pending_safe {
provider_rw.save_safe_block_number(safe.min(last.number))?;
if safe > last.number {
provider_rw.save_safe_block_number(safe.min(last_block.number))?;
if safe > last_block.number {
self.pending_safe_block = Some(safe);
}
}
@@ -185,13 +210,13 @@ where
//
// The pruner reads the indices from rocksdb, filters it, and writes to indices, so it
// must be able to read anything written by save_blocks.
if self.pruner.is_pruning_needed(last.number) {
debug!(target: "engine::persistence", block_num=?last.number, "Running pruner");
if self.pruner.is_pruning_needed(last_block.number) {
debug!(target: "engine::persistence", block_num=?last_block.number, "Running pruner");
let prune_start = Instant::now();
let provider_rw = self.provider.database_provider_rw()?;
let _ = self.pruner.run_with_provider(&provider_rw, last.number)?;
let _ = self.pruner.run_with_provider(&provider_rw, last_block.number)?;
provider_rw.commit()?;
debug!(target: "engine::persistence", tip=?last.number, "Finished pruning after saving blocks");
debug!(target: "engine::persistence", tip=?last_block.number, "Finished pruning after saving blocks");
self.metrics.prune_before_duration_seconds.record(prune_start.elapsed());
}
}
@@ -200,7 +225,7 @@ where
self.metrics.save_blocks_batch_size.record(block_count as f64);
self.metrics.save_blocks_duration_seconds.record(elapsed);
Ok(PersistenceResult { last_block, commit_duration: Some(elapsed) })
Ok(PersistenceResult { last_block, last_state_trie_block, commit_duration: Some(elapsed) })
}
}
@@ -222,9 +247,10 @@ pub enum PersistenceAction<N: NodePrimitives = EthPrimitives> {
/// The section of tree state that should be persisted. These blocks are expected in order of
/// increasing block number.
///
/// First, header, transaction, and receipt-related data should be written to static files.
/// Then the execution history-related data will be written to the database.
SaveBlocks(Vec<ExecutedBlock<N>>, CrossbeamSender<PersistenceResult>),
/// First, header, transaction, and receipt-related data should be written to static files for
/// the deferred trie region. Then the execution history-related data will be written to the
/// database, while trie catchup is persisted for the prefix.
SaveBlocks(SaveBlocksPlan<N>, CrossbeamSender<PersistenceResult>),
/// Removes block data above the given block number from the database.
///
@@ -308,10 +334,10 @@ impl<T: NodePrimitives> PersistenceHandle<T> {
/// If there are no blocks to persist, then `None` is sent in the sender.
pub fn save_blocks(
&self,
blocks: Vec<ExecutedBlock<T>>,
plan: SaveBlocksPlan<T>,
tx: CrossbeamSender<PersistenceResult>,
) -> Result<(), SendError<PersistenceAction<T>>> {
self.send_action(PersistenceAction::SaveBlocks(blocks, tx))
self.send_action(PersistenceAction::SaveBlocks(plan, tx))
}
/// Queues the finalized block number to be persisted on disk.
@@ -375,12 +401,12 @@ impl Drop for ServiceGuard {
mod tests {
use super::*;
use alloy_primitives::{B256, U256};
use reth_chain_state::test_utils::TestBlockBuilder;
use reth_chain_state::{test_utils::TestBlockBuilder, ExecutedBlock};
use reth_exex_types::FinishedExExHeight;
use reth_provider::{
providers::{ProviderFactoryBuilder, ReadOnlyConfig},
test_utils::{create_test_provider_factory, MockNodeTypes},
AccountReader, ChainSpecProvider, HeaderProvider, StorageSettingsCache,
AccountReader, ChainSpecProvider, HeaderProvider, SaveBlocksPlanStep, StorageSettingsCache,
TryIntoHistoricalStateProvider,
};
use reth_prune::Pruner;
@@ -389,6 +415,13 @@ mod tests {
fn default_persistence_handle() -> PersistenceHandle<EthPrimitives> {
let provider = create_test_provider_factory();
persistence_handle(provider)
}
fn persistence_handle<N>(provider: ProviderFactory<N>) -> PersistenceHandle<EthPrimitives>
where
N: ProviderNodeTypes<Primitives = EthPrimitives>,
{
let (_finished_exex_height_tx, finished_exex_height_rx) =
tokio::sync::watch::channel(FinishedExExHeight::NoExExs);
@@ -399,18 +432,31 @@ mod tests {
PersistenceHandle::<EthPrimitives>::spawn_service(provider, pruner, sync_metrics_tx)
}
fn full_save_plan(blocks: Vec<ExecutedBlock<EthPrimitives>>) -> SaveBlocksPlan<EthPrimitives> {
let full_range = 0..blocks.len();
SaveBlocksPlan::new(
blocks,
vec![SaveBlocksPlanStep::new(
full_range.clone(),
Some(full_range.end..full_range.end),
true,
)],
)
}
#[test]
fn test_save_blocks_empty() {
reth_tracing::init_test_tracing();
let handle = default_persistence_handle();
let blocks = vec![];
let blocks = full_save_plan(vec![]);
let (tx, rx) = crossbeam_channel::bounded(1);
handle.save_blocks(blocks, tx).unwrap();
let result = rx.recv().unwrap();
assert!(result.last_block.is_none());
assert!(result.last_state_trie_block.is_none());
}
#[test]
@@ -423,14 +469,16 @@ mod tests {
test_block_builder.get_executed_block_with_number(block_number, B256::random());
let block_hash = executed.recovered_block().hash();
let blocks = vec![executed];
let blocks = full_save_plan(vec![executed]);
let (tx, rx) = crossbeam_channel::bounded(1);
handle.save_blocks(blocks, tx).unwrap();
let result = rx.recv_timeout(std::time::Duration::from_secs(10)).expect("test timed out");
assert_eq!(block_hash, result.last_block.unwrap().hash);
let last_block = result.last_block.unwrap();
assert_eq!(block_hash, last_block.hash);
assert_eq!(result.last_state_trie_block, Some(last_block.number));
}
#[test]
@@ -443,9 +491,11 @@ mod tests {
let last_hash = blocks.last().unwrap().recovered_block().hash();
let (tx, rx) = crossbeam_channel::bounded(1);
handle.save_blocks(blocks, tx).unwrap();
handle.save_blocks(full_save_plan(blocks), tx).unwrap();
let result = rx.recv().unwrap();
assert_eq!(last_hash, result.last_block.unwrap().hash);
let last_block = result.last_block.unwrap();
assert_eq!(last_hash, last_block.hash);
assert_eq!(result.last_state_trie_block, Some(last_block.number));
}
#[test]
@@ -460,13 +510,57 @@ mod tests {
let last_hash = blocks.last().unwrap().recovered_block().hash();
let (tx, rx) = crossbeam_channel::bounded(1);
handle.save_blocks(blocks, tx).unwrap();
handle.save_blocks(full_save_plan(blocks), tx).unwrap();
let result = rx.recv().unwrap();
assert_eq!(last_hash, result.last_block.unwrap().hash);
let last_block = result.last_block.unwrap();
assert_eq!(last_hash, last_block.hash);
assert_eq!(result.last_state_trie_block, Some(last_block.number));
}
}
#[test]
fn test_remove_blocks_above_preserves_partial_state_trie() {
reth_tracing::init_test_tracing();
let provider = create_test_provider_factory();
let mut test_block_builder = TestBlockBuilder::eth().with_state();
let blocks = test_block_builder.get_executed_blocks(0..4).collect::<Vec<_>>();
let provider_rw = provider.database_provider_rw().unwrap();
provider_rw
.save_blocks(
&SaveBlocksPlan::new(
blocks,
vec![
SaveBlocksPlanStep::new(0..2, Some(2..4), true),
SaveBlocksPlanStep::new(2..4, None, true),
],
),
SaveBlocksMode::Full,
)
.unwrap();
provider_rw.commit().unwrap();
let handle = persistence_handle(provider.clone());
let (tx, rx) = crossbeam_channel::bounded(1);
handle.remove_blocks_above(2, tx).unwrap();
let result = rx.recv_timeout(std::time::Duration::from_secs(10)).expect("test timed out");
let last_block = result.last_block.unwrap();
assert_eq!(last_block.number, 2);
assert_eq!(result.last_state_trie_block, Some(1));
let finish_checkpoint =
provider.provider().unwrap().get_stage_checkpoint(StageId::Finish).unwrap().unwrap();
assert_eq!(finish_checkpoint.block_number, 2);
assert_eq!(
finish_checkpoint.finish_stage_checkpoint().unwrap().partial_state_trie,
Some(1)
);
}
/// Verifies that committing `save_blocks` history before running the pruner
/// prevents the pruner from overwriting new entries.
///
@@ -555,7 +649,7 @@ mod tests {
{
let provider_rw = provider_factory.database_provider_rw().unwrap();
provider_rw.save_blocks(blocks_a, SaveBlocksMode::Full).unwrap();
provider_rw.save_blocks(&full_save_plan(blocks_a), SaveBlocksMode::Full).unwrap();
provider_rw.commit().unwrap();
}
@@ -612,7 +706,12 @@ mod tests {
provider_rw.commit().unwrap();
let provider_rw = pf.database_provider_rw().unwrap();
provider_rw.save_blocks(vec![block_b2], SaveBlocksMode::Full).unwrap();
provider_rw
.save_blocks(
&full_save_plan(std::slice::from_ref(&block_b2).to_vec()),
SaveBlocksMode::Full,
)
.unwrap();
provider_rw.commit().unwrap();
});

View File

@@ -30,9 +30,9 @@ use reth_primitives_traits::{
};
use reth_provider::{
BlockExecutionOutput, BlockExecutionResult, BlockReader, ChangeSetReader,
DatabaseProviderFactory, HashedPostStateProvider, ProviderError, StageCheckpointReader,
StateProviderBox, StateProviderFactory, StateReader, StorageChangeSetReader,
StorageSettingsCache, TransactionVariant,
DatabaseProviderFactory, HashedPostStateProvider, ProviderError, SaveBlocksPlan,
SaveBlocksPlanStep, StageCheckpointReader, StateProviderBox, StateProviderFactory, StateReader,
StorageChangeSetReader, StorageSettingsCache, TransactionVariant,
};
use reth_revm::database::StateProviderDatabase;
use reth_stages_api::ControlFlow;
@@ -433,6 +433,7 @@ where
let persistence_state = PersistenceState {
last_persisted_block: BlockNumHash::new(best_block_number, header.hash()),
last_state_trie_persisted_block: BlockNumHash::new(best_block_number, header.hash()),
rx: None,
};
@@ -1350,7 +1351,7 @@ where
/// Helper method to remove blocks and set the persistence state. This ensures we keep track of
/// the current persistence action while we're removing blocks.
fn remove_blocks(&mut self, new_tip_num: u64) {
debug!(target: "engine::tree", ?new_tip_num, last_persisted_block_number=?self.persistence_state.last_persisted_block.number, "Removing blocks using persistence task");
debug!(target: "engine::tree", ?new_tip_num, last_persisted_block=?self.persistence_state.last_persisted_block.number, "Removing blocks using persistence task");
if new_tip_num < self.persistence_state.last_persisted_block.number {
debug!(target: "engine::tree", ?new_tip_num, "Starting remove blocks job");
let (tx, rx) = crossbeam_channel::bounded(1);
@@ -1361,24 +1362,25 @@ where
/// Helper method to save blocks and set the persistence state. This ensures we keep track of
/// the current persistence action while we're saving blocks.
fn persist_blocks(&mut self, blocks_to_persist: Vec<ExecutedBlock<N>>) {
if blocks_to_persist.is_empty() {
fn persist_blocks(&mut self, plan: SaveBlocksPlan<N>) {
if plan.is_empty() {
debug!(target: "engine::tree", "Returned empty set of blocks to persist");
return
}
// NOTE: checked non-empty above
let highest_num_hash = blocks_to_persist
.iter()
.max_by_key(|block| block.recovered_block().number())
.map(|b| b.recovered_block().num_hash())
.expect("Checked non-empty persisting blocks");
let last_block = plan.last_block().expect("checked non-empty persisting blocks");
debug!(target: "engine::tree", count=blocks_to_persist.len(), blocks = ?blocks_to_persist.iter().map(|block| block.recovered_block().num_hash()).collect::<Vec<_>>(), "Persisting blocks");
debug!(
target: "engine::tree",
count = plan.blocks.len(),
steps = ?plan.steps,
blocks = ?plan.blocks.iter().map(|block| block.recovered_block().num_hash()).collect::<Vec<_>>(),
"Persisting blocks"
);
let (tx, rx) = crossbeam_channel::bounded(1);
let _ = self.persistence.save_blocks(blocks_to_persist, tx);
let _ = self.persistence.save_blocks(plan, tx);
self.persistence_state.start_save(highest_num_hash, rx);
self.persistence_state.start_save(last_block, rx);
}
/// Triggers new persistence actions if no persistence task is currently in progress.
@@ -1390,9 +1392,8 @@ where
if let Some(new_tip_num) = self.find_disk_reorg()? {
self.remove_blocks(new_tip_num)
} else if self.should_persist() {
let blocks_to_persist =
self.get_canonical_blocks_to_persist(PersistTarget::Threshold)?;
self.persist_blocks(blocks_to_persist);
let plan = self.get_save_blocks_plan(PersistTarget::Threshold)?;
self.persist_blocks(plan);
}
}
@@ -1423,15 +1424,15 @@ where
self.on_persistence_complete(result, start_time)?;
}
let blocks_to_persist = self.get_canonical_blocks_to_persist(PersistTarget::Head)?;
let plan = self.get_save_blocks_plan(PersistTarget::Head)?;
if blocks_to_persist.is_empty() {
if plan.is_empty() {
debug!(target: "engine::tree", "persistence complete, signaling termination");
return Ok(())
}
debug!(target: "engine::tree", count = blocks_to_persist.len(), "persisting remaining blocks before shutdown");
self.persist_blocks(blocks_to_persist);
debug!(target: "engine::tree", count = plan.blocks.len(), "persisting remaining blocks before shutdown");
self.persist_blocks(plan);
}
}
@@ -1467,25 +1468,25 @@ where
) -> Result<(), AdvancePersistenceError> {
self.metrics.engine.persistence_duration.record(start_time.elapsed());
let commit_duration = result.commit_duration;
let Some(BlockNumHash {
hash: last_persisted_block_hash,
number: last_persisted_block_number,
}) = result.last_block
let PersistenceResult { last_block, last_state_trie_block, commit_duration } = result;
let Some(BlockNumHash { hash: last_block_hash, number: last_block_number }) = last_block
else {
// if this happened, then we persisted no blocks because we sent an empty vec of blocks
warn!(target: "engine::tree", "Persistence task completed but did not persist any blocks");
return Ok(())
};
debug!(target: "engine::tree", ?last_persisted_block_hash, ?last_persisted_block_number, elapsed=?start_time.elapsed(), "Finished persisting, calling finish");
self.persistence_state.finish(last_persisted_block_hash, last_persisted_block_number);
let last_block = BlockNumHash::new(last_block_number, last_block_hash);
let last_state_trie_persisted_block =
self.last_state_trie_persisted_block(last_block, last_state_trie_block)?;
debug!(target: "engine::tree", ?last_block_hash, ?last_block_number, last_state_trie_persisted_block = last_state_trie_persisted_block.number, elapsed=?start_time.elapsed(), "Finished persisting, calling finish");
self.persistence_state.finish(last_block, last_state_trie_persisted_block);
// Evict trie changesets for blocks below the eviction threshold.
// Keep at least CHANGESET_CACHE_RETENTION_BLOCKS from the persisted tip, and also respect
// the finalized block if set.
let min_threshold =
last_persisted_block_number.saturating_sub(CHANGESET_CACHE_RETENTION_BLOCKS);
let min_threshold = last_block_number.saturating_sub(CHANGESET_CACHE_RETENTION_BLOCKS);
let eviction_threshold =
if let Some(finalized) = self.canonical_in_memory_state.get_finalized_num_hash() {
// Use the minimum of finalized block and retention threshold to be conservative
@@ -1496,7 +1497,7 @@ where
};
debug!(
target: "engine::tree",
last_persisted = last_persisted_block_number,
last_persisted_block = last_block_number,
finalized_number = ?self.canonical_in_memory_state.get_finalized_num_hash().map(|f| f.number),
eviction_threshold,
"Evicting changesets below threshold"
@@ -1506,7 +1507,7 @@ where
// Invalidate cached overlay since the anchor has changed
self.state.tree_state.invalidate_cached_overlay();
self.on_new_persisted_block()?;
self.on_new_persisted_block(last_state_trie_persisted_block)?;
// Re-prepare overlay for the current canonical head with the new anchor.
// Spawn a background task to trigger computation so it's ready when the next payload
@@ -1517,11 +1518,39 @@ where
});
}
self.purge_timing_stats(last_persisted_block_number, commit_duration);
self.purge_timing_stats(last_block_number, commit_duration);
Ok(())
}
/// Returns the highest block that can be dropped from memory after persistence completes.
fn last_state_trie_persisted_block(
&self,
last_block: BlockNumHash,
last_state_trie_block: Option<u64>,
) -> ProviderResult<BlockNumHash> {
let Some(last_state_trie_block) = last_state_trie_block else { return Ok(last_block) };
debug_assert!(
last_state_trie_block <= last_block.number,
"state/trie frontier cannot exceed the last persisted block"
);
if last_state_trie_block >= last_block.number {
return Ok(last_block)
}
let hash = self
.canonical_in_memory_state
.hash_by_number(last_state_trie_block)
.map(Ok)
.unwrap_or_else(|| {
self.provider
.block_hash(last_state_trie_block)?
.ok_or_else(|| ProviderError::HeaderNotFound(last_state_trie_block.into()))
})?;
Ok(BlockNumHash::new(last_state_trie_block, hash))
}
/// Handles a message from the engine.
///
/// Returns `ControlFlow::Break(())` if the engine should terminate.
@@ -1825,7 +1854,7 @@ where
// update the tracked chain height, after backfill sync both the canonical height and
// persisted height are the same
self.state.tree_state.set_canonical_head(new_head.num_hash());
self.persistence_state.finish(new_head.hash(), new_head.number());
self.persistence_state.finish(new_head.num_hash(), new_head.num_hash());
// update the tracked canonical head
self.canonical_in_memory_state.set_canonical_head(new_head);
@@ -1917,37 +1946,9 @@ where
self.on_canonical_chain_update(chain_update);
}
self.on_canonicalized_sync_target(target);
Ok(())
}
/// Applies the tracked forkchoice state once its sync target head becomes canonical.
fn on_canonicalized_sync_target(&mut self, target: B256) {
let Some(sync_target_state) = self
.state
.forkchoice_state_tracker
.sync_target_state()
.filter(|state| state.head_block_hash == target)
else {
return;
};
if let Err(outcome) = self.ensure_consistent_forkchoice_state(sync_target_state) {
debug!(
target: "engine::tree",
head = %sync_target_state.head_block_hash,
safe = %sync_target_state.safe_block_hash,
finalized = %sync_target_state.finalized_block_hash,
?outcome,
"Canonicalized sync target head before safe/finalized could be applied"
);
return;
}
self.state.forkchoice_state_tracker.promote_sync_target_to_valid(sync_target_state);
}
/// Convenience function to handle an optional tree event.
fn on_maybe_tree_event(&mut self, event: Option<TreeEvent>) -> ProviderResult<()> {
if let Some(event) = event {
@@ -2061,62 +2062,96 @@ where
self.config.persistence_threshold()
}
/// Returns a batch of consecutive canonical blocks to persist in the range
/// `(last_persisted_number .. target]`. The expected order is oldest -> newest.
fn get_canonical_blocks_to_persist(
/// Returns the save plan for the next persistence cycle.
fn get_save_blocks_plan(
&self,
target: PersistTarget,
) -> Result<Vec<ExecutedBlock<N>>, AdvancePersistenceError> {
) -> Result<SaveBlocksPlan<N>, AdvancePersistenceError> {
// We will calculate the state root using the database, so we need to be sure there are no
// changes
debug_assert!(!self.persistence_state.in_progress());
let mut blocks_to_persist = Vec::new();
let mut blocks = Vec::new();
let mut current_hash = self.state.tree_state.canonical_block_hash();
let last_persisted_number = self.persistence_state.last_persisted_block.number;
let last_state_trie_persisted_block_number =
self.persistence_state.last_state_trie_persisted_block.number;
let last_persisted_block_number = self.persistence_state.last_persisted_block.number;
let canonical_head_number = self.state.tree_state.canonical_block_number();
let target_number = match target {
PersistTarget::Head => canonical_head_number,
let last_block_target_number = match target {
PersistTarget::Threshold => {
canonical_head_number.saturating_sub(self.config.memory_block_buffer_target())
}
PersistTarget::Head => canonical_head_number,
};
debug!(
target: "engine::tree",
?current_hash,
?last_persisted_number,
?last_state_trie_persisted_block_number,
?last_persisted_block_number,
?canonical_head_number,
?target_number,
"Returning canonical blocks to persist"
target = ?target,
"Returning save plan"
);
while let Some(block) = self.state.tree_state.blocks_by_hash.get(&current_hash) {
if block.recovered_block().number() <= last_persisted_number {
if block.recovered_block().number() <= last_state_trie_persisted_block_number {
break;
}
if block.recovered_block().number() <= target_number {
blocks_to_persist.push(block.clone());
if block.recovered_block().number() <= last_block_target_number {
blocks.push(block.clone());
}
current_hash = block.recovered_block().parent_hash();
}
// Reverse the order so that the oldest block comes first
blocks_to_persist.reverse();
blocks.reverse();
Ok(blocks_to_persist)
let trie_catchup_block_count = last_persisted_block_number
.saturating_sub(last_state_trie_persisted_block_number)
.min(blocks.len() as u64) as usize;
let persist_rest_block_count = blocks.len().saturating_sub(trie_catchup_block_count);
let state_masking_block_count =
persist_rest_block_count.min(self.config.num_state_masking_blocks() as usize);
let full_persist_block_count = persist_rest_block_count - state_masking_block_count;
let full_persist_start = trie_catchup_block_count;
let state_masking_start = full_persist_start + full_persist_block_count;
let state_masking_range = state_masking_start..blocks.len();
let mut steps = Vec::new();
if trie_catchup_block_count > 0 {
steps.push(SaveBlocksPlanStep::new(
0..trie_catchup_block_count,
Some(state_masking_range.clone()),
false,
));
}
if full_persist_block_count > 0 {
steps.push(SaveBlocksPlanStep::new(
full_persist_start..state_masking_start,
Some(state_masking_range.clone()),
true,
));
}
if state_masking_block_count > 0 {
steps.push(SaveBlocksPlanStep::new(state_masking_range, None, true));
}
Ok(SaveBlocksPlan::new(blocks, steps))
}
/// This clears the blocks from the in-memory tree state that have been persisted to the
/// database.
/// This clears the blocks from the in-memory tree state that no longer need to stay resident
/// after persistence completes.
///
/// This also updates the canonical in-memory state to reflect the newest persisted block
/// height.
/// This also updates the canonical in-memory state to reflect the newest persisted block tip,
/// even if trie persistence only advanced through an earlier block.
///
/// Assumes that `finish` has been called on the `persistence_state` at least once
fn on_new_persisted_block(&mut self) -> ProviderResult<()> {
fn on_new_persisted_block(
&mut self,
in_memory_persisted_block: BlockNumHash,
) -> ProviderResult<()> {
// If we have an on-disk reorg, we need to handle it first before touching the in-memory
// state.
if let Some(remove_above) = self.find_disk_reorg()? {
@@ -2125,11 +2160,11 @@ where
}
let finalized = self.state.forkchoice_state_tracker.last_valid_finalized();
self.remove_before(self.persistence_state.last_persisted_block, finalized)?;
self.canonical_in_memory_state.remove_persisted_blocks(BlockNumHash {
number: self.persistence_state.last_persisted_block.number,
hash: self.persistence_state.last_persisted_block.hash,
});
self.remove_before(in_memory_persisted_block, finalized)?;
self.canonical_in_memory_state.remove_persisted_blocks_until(
self.persistence_state.last_persisted_block,
in_memory_persisted_block.number,
);
Ok(())
}

View File

@@ -361,44 +361,8 @@ where
let (prefetch_tx, prefetch_rx) = oneshot::channel();
let (stream_tx, stream_rx) = oneshot::channel();
if let Some(to_sparse_trie_task) = to_sparse_trie_task {
let stream_ctx = ctx.clone();
executor.bal_streaming_pool().spawn(move || {
let branch_span = debug_span!(
target: "engine::tree::payload_processor::prewarm",
parent: &stream_parent_span,
"bal_hashed_state_stream",
bal_accounts = stream_bal.as_bal().len(),
);
let provider_parent_span = branch_span.clone();
let _span = branch_span.entered();
stream_bal.as_bal().par_iter().for_each_init(
|| {
(
stream_ctx.clone(),
None::<Box<dyn AccountReader>>,
provider_parent_span.clone(),
)
},
|(ctx, provider, parent_span), account_changes| {
ctx.send_bal_hashed_state(
parent_span,
provider,
account_changes,
&to_sparse_trie_task,
);
},
);
let _ = to_sparse_trie_task.send(StateRootMessage::FinishedStateUpdates);
let _ = stream_tx.send(());
});
} else {
let _ = stream_tx.send(());
}
if ctx.saved_cache.is_some() {
let prefetch_ctx = ctx.clone();
executor.prewarming_pool().spawn(move || {
let branch_span = debug_span!(
target: "engine::tree::payload_processor::prewarm",
@@ -412,7 +376,7 @@ where
prefetch_bal.as_bal().par_iter().for_each_init(
|| {
(
ctx.clone(),
prefetch_ctx.clone(),
None::<CachedStateProvider<reth_provider::StateProviderBox, true>>,
provider_parent_span.clone(),
)
@@ -431,6 +395,36 @@ where
let _ = prefetch_tx.send(());
}
if let Some(to_sparse_trie_task) = to_sparse_trie_task {
executor.bal_streaming_pool().spawn(move || {
let branch_span = debug_span!(
target: "engine::tree::payload_processor::prewarm",
parent: &stream_parent_span,
"bal_hashed_state_stream",
bal_accounts = stream_bal.as_bal().len(),
);
let provider_parent_span = branch_span.clone();
let _span = branch_span.entered();
stream_bal.as_bal().par_iter().for_each_init(
|| (ctx.clone(), None::<Box<dyn AccountReader>>, provider_parent_span.clone()),
|(ctx, provider, parent_span), account_changes| {
ctx.send_bal_hashed_state(
parent_span,
provider,
account_changes,
&to_sparse_trie_task,
);
},
);
let _ = to_sparse_trie_task.send(StateRootMessage::FinishedStateUpdates);
let _ = stream_tx.send(());
});
} else {
let _ = stream_tx.send(());
}
prefetch_rx
.blocking_recv()
.expect("BAL prefetch task dropped without signaling completion");
@@ -757,9 +751,7 @@ where
provider: &mut Option<CachedStateProvider<reth_provider::StateProviderBox, true>>,
account: &alloy_eip7928::AccountChanges,
) {
if self.disable_bal_batch_io ||
(account.storage_changes.is_empty() && account.storage_reads.is_empty())
{
if account.storage_changes.is_empty() && account.storage_reads.is_empty() {
return;
}

View File

@@ -22,7 +22,6 @@
use crate::persistence::PersistenceResult;
use alloy_eips::BlockNumHash;
use alloy_primitives::B256;
use crossbeam_channel::Receiver as CrossbeamReceiver;
use reth_primitives_traits::FastInstant as Instant;
use tracing::trace;
@@ -30,10 +29,12 @@ use tracing::trace;
/// The state of the persistence task.
#[derive(Debug)]
pub struct PersistenceState {
/// Hash and number of the last block persisted.
/// Hash and number of the highest block whose non-state/trie outputs are persisted.
///
/// This tracks the chain height that is persisted on disk
/// This tracks the highest canonical block with durable block/static-file/plain-state data.
pub(crate) last_persisted_block: BlockNumHash,
/// Hash and number of the highest block whose state/trie outputs are persisted.
pub(crate) last_state_trie_persisted_block: BlockNumHash,
/// Receiver end of channel where the result of the persistence task will be
/// sent when done. A None value means there's no persistence task in progress.
pub(crate) rx:
@@ -76,13 +77,18 @@ impl PersistenceState {
/// Sets state for a finished persistence task.
pub(crate) fn finish(
&mut self,
last_persisted_block_hash: B256,
last_persisted_block_number: u64,
last_persisted_block: BlockNumHash,
last_state_trie_persisted_block: BlockNumHash,
) {
trace!(target: "engine::tree", block= %last_persisted_block_number, hash=%last_persisted_block_hash, "updating persistence state");
trace!(
target: "engine::tree",
last_persisted_block = %last_persisted_block.number,
last_state_trie_persisted_block = %last_state_trie_persisted_block.number,
"updating persistence state"
);
self.rx = None;
self.last_persisted_block =
BlockNumHash::new(last_persisted_block_number, last_persisted_block_hash);
self.last_persisted_block = last_persisted_block;
self.last_state_trie_persisted_block = last_state_trie_persisted_block;
}
}

View File

@@ -13,7 +13,7 @@ use std::{hash::Hash, sync::Arc};
use tracing::error;
/// Default max cache size for [`PrecompileCache`]
const MAX_CACHE_SIZE: u32 = 1024 * 1024;
const MAX_CACHE_SIZE: u32 = 10_000;
/// Stores caches for each precompile.
#[derive(Debug, Clone, Default)]
@@ -54,9 +54,6 @@ where
moka::sync::CacheBuilder::new(MAX_CACHE_SIZE as u64)
.initial_capacity(MAX_CACHE_SIZE as usize)
.eviction_policy(EvictionPolicy::lru())
.weigher(|key: &Bytes, value: &CacheEntry<S>| {
(key.len() + value.output.bytes.len()) as u32
})
.build_with_hasher(Default::default()),
)
}

View File

@@ -222,7 +222,11 @@ impl TestHarness {
engine_api_tree_state,
canonical_in_memory_state,
persistence_handle,
PersistenceState { last_persisted_block: BlockNumHash::default(), rx: None },
PersistenceState {
last_persisted_block: BlockNumHash::default(),
last_state_trie_persisted_block: BlockNumHash::default(),
rx: None,
},
payload_builder,
tree_config,
EngineApiKind::Ethereum,
@@ -360,6 +364,17 @@ impl TestHarness {
}
}
type ExpectedPlanStep = (std::ops::Range<usize>, Option<std::ops::Range<usize>>, bool);
fn assert_plan_steps(plan: &SaveBlocksPlan<EthPrimitives>, expected: &[ExpectedPlanStep]) {
assert_eq!(plan.steps.len(), expected.len());
for (step, (block_range, masking_range, persist_rest)) in plan.steps.iter().zip(expected) {
assert_eq!(&step.block_range, block_range);
assert_eq!(&step.state_trie_masking_range, masking_range);
assert_eq!(step.persist_rest, *persist_rest);
}
}
/// Simplified test metrics for validation calls
#[derive(Debug, Default)]
struct TestMetrics {
@@ -554,12 +569,16 @@ async fn test_tree_persist_blocks() {
let received_action =
test_harness.action_rx.recv().expect("Failed to receive save blocks action");
if let PersistenceAction::SaveBlocks(saved_blocks, _) = received_action {
if let PersistenceAction::SaveBlocks(plan, _) = received_action {
// only blocks.len() - tree_config.memory_block_buffer_target() will be
// persisted
let expected_persist_len = blocks.len() - tree_config.memory_block_buffer_target() as usize;
assert_eq!(saved_blocks.len(), expected_persist_len);
assert_eq!(saved_blocks, blocks[..expected_persist_len]);
assert_eq!(plan.blocks.len(), expected_persist_len);
assert_eq!(plan.blocks, blocks[..expected_persist_len]);
assert_plan_steps(
&plan,
&[(0..expected_persist_len, Some(expected_persist_len..expected_persist_len), true)],
);
} else {
panic!("unexpected action received {received_action:?}");
}
@@ -704,8 +723,8 @@ fn test_backpressure_waits_for_persistence_before_reading_incoming() {
test_harness.tree.config = test_harness
.tree
.config
.with_persistence_threshold(0)
.with_persistence_backpressure_threshold(1);
.with_persistence_threshold(1)
.with_persistence_backpressure_threshold(2);
let (persist_tx, persist_rx) = crossbeam_channel::bounded(1);
let persisted = blocks.last().unwrap().recovered_block().num_hash();
@@ -736,6 +755,7 @@ fn test_backpressure_waits_for_persistence_before_reading_incoming() {
persist_tx
.send(PersistenceResult {
last_block: Some(persisted),
last_state_trie_block: Some(persisted.number),
commit_duration: Some(Duration::ZERO),
})
.unwrap();
@@ -770,10 +790,10 @@ async fn test_tree_state_on_new_head_reorg() {
reth_tracing::init_test_tracing();
let chain_spec = MAINNET.clone();
// Set persistence_threshold to 1
// Keep a single block in memory while still leaving room for the persistence threshold.
let mut test_harness = TestHarness::new(chain_spec);
test_harness.tree.config =
test_harness.tree.config.with_persistence_threshold(1).with_memory_block_buffer_target(1);
test_harness.tree.config.with_persistence_threshold(2).with_memory_block_buffer_target(1);
let mut test_block_builder = TestBlockBuilder::eth();
let blocks: Vec<_> = test_block_builder.get_executed_blocks(1..6).collect();
@@ -824,15 +844,16 @@ async fn test_tree_state_on_new_head_reorg() {
// get rid of the prev action
let received_action = test_harness.action_rx.recv().unwrap();
let PersistenceAction::SaveBlocks(saved_blocks, sender) = received_action else {
let PersistenceAction::SaveBlocks(plan, sender) = received_action else {
panic!("received wrong action");
};
assert_eq!(saved_blocks, vec![blocks[0].clone(), blocks[1].clone()]);
assert_eq!(plan.blocks, vec![blocks[0].clone(), blocks[1].clone()]);
// send the response so we can advance again
sender
.send(PersistenceResult {
last_block: Some(blocks[1].recovered_block().num_hash()),
last_state_trie_block: Some(blocks[1].recovered_block().number()),
commit_duration: Some(Duration::ZERO),
})
.unwrap();
@@ -968,8 +989,10 @@ async fn test_get_canonical_blocks_to_persist() {
test_harness = test_harness.with_blocks(blocks.clone());
let last_persisted_block_number = 3;
test_harness.tree.persistence_state.last_persisted_block =
let last_persisted_block =
blocks[last_persisted_block_number as usize].recovered_block.num_hash();
test_harness.tree.persistence_state.last_persisted_block = last_persisted_block;
test_harness.tree.persistence_state.last_state_trie_persisted_block = last_persisted_block;
let persistence_threshold = 4;
let memory_block_buffer_target = 3;
@@ -977,16 +1000,15 @@ async fn test_get_canonical_blocks_to_persist() {
.with_persistence_threshold(persistence_threshold)
.with_memory_block_buffer_target(memory_block_buffer_target);
let blocks_to_persist =
test_harness.tree.get_canonical_blocks_to_persist(PersistTarget::Threshold).unwrap();
let plan = test_harness.tree.get_save_blocks_plan(PersistTarget::Threshold).unwrap();
let expected_blocks_to_persist_length: usize =
(canonical_head_number - memory_block_buffer_target - last_persisted_block_number)
.try_into()
.unwrap();
assert_eq!(blocks_to_persist.len(), expected_blocks_to_persist_length);
for (i, item) in blocks_to_persist.iter().enumerate().take(expected_blocks_to_persist_length) {
assert_eq!(plan.blocks.len(), expected_blocks_to_persist_length);
for (i, item) in plan.blocks.iter().enumerate().take(expected_blocks_to_persist_length) {
assert_eq!(item.recovered_block().number, last_persisted_block_number + i as u64 + 1);
}
@@ -997,15 +1019,14 @@ async fn test_get_canonical_blocks_to_persist() {
assert!(test_harness.tree.state.tree_state.sealed_header_by_hash(&fork_block_hash).is_some());
let blocks_to_persist =
test_harness.tree.get_canonical_blocks_to_persist(PersistTarget::Threshold).unwrap();
assert_eq!(blocks_to_persist.len(), expected_blocks_to_persist_length);
let plan = test_harness.tree.get_save_blocks_plan(PersistTarget::Threshold).unwrap();
assert_eq!(plan.blocks.len(), expected_blocks_to_persist_length);
// check that the fork block is not included in the blocks to persist
assert!(!blocks_to_persist.iter().any(|b| b.recovered_block().hash() == fork_block_hash));
assert!(!plan.blocks.iter().any(|b| b.recovered_block().hash() == fork_block_hash));
// check that the original block 4 is still included
assert!(blocks_to_persist.iter().any(|b| b.recovered_block().number == 4 &&
assert!(plan.blocks.iter().any(|b| b.recovered_block().number == 4 &&
b.recovered_block().hash() == blocks[4].recovered_block().hash()));
// check that if we advance persistence, the persistence action is the correct value
@@ -1013,11 +1034,193 @@ async fn test_get_canonical_blocks_to_persist() {
assert_eq!(
test_harness.tree.persistence_state.current_action().cloned(),
Some(CurrentPersistenceAction::SavingBlocks {
highest: blocks_to_persist.last().unwrap().recovered_block().num_hash()
highest: plan.blocks.last().unwrap().recovered_block().num_hash()
})
);
}
#[test]
fn test_get_save_blocks_plan_with_deferred_trie_blocks() {
let chain_spec = MAINNET.clone();
let mut test_harness = TestHarness::new(chain_spec);
let mut test_block_builder = TestBlockBuilder::eth();
let blocks: Vec<_> = test_block_builder.get_executed_blocks(0..7).collect();
test_harness = test_harness.with_blocks(blocks.clone());
test_harness.tree.persistence_state.last_state_trie_persisted_block =
blocks[1].recovered_block().num_hash();
test_harness.tree.persistence_state.last_persisted_block =
blocks[3].recovered_block().num_hash();
test_harness.tree.config = TreeConfig::default()
.with_persistence_threshold(4)
.with_memory_block_buffer_target(1)
.with_num_state_masking_blocks(2);
let plan = test_harness.tree.get_save_blocks_plan(PersistTarget::Threshold).unwrap();
assert_plan_steps(&plan, &[(0..2, Some(2..4), false), (2..4, None, true)]);
assert_eq!(plan.blocks.len(), 4);
assert_eq!(
plan.blocks.iter().map(|block| block.recovered_block().number()).collect::<Vec<_>>(),
vec![2, 3, 4, 5]
);
assert_eq!(plan.last_block(), Some(blocks[5].recovered_block().num_hash()));
}
#[test]
fn test_get_save_blocks_plan_persists_full_region_before_deferred_tail() {
let chain_spec = MAINNET.clone();
let mut test_harness = TestHarness::new(chain_spec);
let mut test_block_builder = TestBlockBuilder::eth();
let blocks: Vec<_> = test_block_builder.get_executed_blocks(0..31).collect();
test_harness = test_harness.with_blocks(blocks.clone());
test_harness.tree.persistence_state.last_state_trie_persisted_block =
blocks[12].recovered_block().num_hash();
test_harness.tree.persistence_state.last_persisted_block =
blocks[15].recovered_block().num_hash();
test_harness.tree.config = TreeConfig::default()
.with_persistence_threshold(5)
.with_memory_block_buffer_target(2)
.with_num_state_masking_blocks(2);
let plan = test_harness.tree.get_save_blocks_plan(PersistTarget::Threshold).unwrap();
assert_plan_steps(
&plan,
&[(0..3, Some(14..16), false), (3..14, Some(14..16), true), (14..16, None, true)],
);
assert_eq!(plan.blocks.len(), 16);
assert_eq!(
plan.blocks.iter().map(|block| block.recovered_block().number()).collect::<Vec<_>>(),
(13..=28).collect::<Vec<_>>()
);
assert_eq!(plan.last_block(), Some(blocks[28].recovered_block().num_hash()));
}
#[test]
fn test_on_persistence_complete_retains_blocks_above_partial_state_trie() {
let chain_spec = MAINNET.clone();
let mut test_harness = TestHarness::new(chain_spec);
let mut test_block_builder = TestBlockBuilder::eth();
let blocks: Vec<_> = test_block_builder.get_executed_blocks(0..7).collect();
test_harness = test_harness.with_blocks(blocks.clone());
test_harness.tree.persistence_state.last_persisted_block =
blocks[1].recovered_block().num_hash();
test_harness.tree.persistence_state.last_state_trie_persisted_block =
blocks[1].recovered_block().num_hash();
let persisted_tip = blocks[5].recovered_block().num_hash();
let last_state_trie_block = blocks[3].recovered_block().number();
test_harness
.tree
.on_persistence_complete(
PersistenceResult {
last_block: Some(persisted_tip),
last_state_trie_block: Some(last_state_trie_block),
commit_duration: Some(Duration::ZERO),
},
Instant::now(),
)
.unwrap();
assert_eq!(test_harness.tree.persistence_state.last_persisted_block, persisted_tip);
assert_eq!(
test_harness.tree.persistence_state.last_state_trie_persisted_block,
blocks[3].recovered_block().num_hash()
);
assert_eq!(
test_harness.tree.canonical_in_memory_state.get_persisted_num_hash(),
Some(persisted_tip)
);
for block in &blocks[..=last_state_trie_block as usize] {
assert!(test_harness
.tree
.state
.tree_state
.executed_block_by_hash(block.recovered_block().hash())
.is_none());
assert!(test_harness
.tree
.canonical_in_memory_state
.state_by_number(block.recovered_block().number())
.is_none());
}
for block in &blocks[last_state_trie_block as usize + 1..] {
assert!(test_harness
.tree
.state
.tree_state
.executed_block_by_hash(block.recovered_block().hash())
.is_some());
assert!(test_harness
.tree
.canonical_in_memory_state
.state_by_number(block.recovered_block().number())
.is_some());
}
}
#[test]
fn test_on_persistence_complete_without_partial_state_trie_prunes_through_tip() {
let chain_spec = MAINNET.clone();
let mut test_harness = TestHarness::new(chain_spec);
let mut test_block_builder = TestBlockBuilder::eth();
let blocks: Vec<_> = test_block_builder.get_executed_blocks(0..7).collect();
test_harness = test_harness.with_blocks(blocks.clone());
test_harness.tree.persistence_state.last_persisted_block =
blocks[1].recovered_block().num_hash();
test_harness.tree.persistence_state.last_state_trie_persisted_block =
blocks[1].recovered_block().num_hash();
let persisted_tip = blocks[5].recovered_block().num_hash();
test_harness
.tree
.on_persistence_complete(
PersistenceResult {
last_block: Some(persisted_tip),
last_state_trie_block: None,
commit_duration: Some(Duration::ZERO),
},
Instant::now(),
)
.unwrap();
for block in &blocks[..=persisted_tip.number as usize] {
assert!(test_harness
.tree
.state
.tree_state
.executed_block_by_hash(block.recovered_block().hash())
.is_none());
assert!(test_harness
.tree
.canonical_in_memory_state
.state_by_number(block.recovered_block().number())
.is_none());
}
for block in &blocks[persisted_tip.number as usize + 1..] {
assert!(test_harness
.tree
.state
.tree_state
.executed_block_by_hash(block.recovered_block().hash())
.is_some());
assert!(test_harness
.tree
.canonical_in_memory_state
.state_by_number(block.recovered_block().number())
.is_some());
}
}
#[tokio::test]
async fn test_engine_tree_fcu_missing_head() {
let chain_spec = MAINNET.clone();
@@ -2112,15 +2315,18 @@ mod forkchoice_updated_tests {
break;
}
if let Ok(PersistenceAction::SaveBlocks(saved_blocks, sender)) =
if let Ok(PersistenceAction::SaveBlocks(plan, sender)) =
action_rx.recv_timeout(std::time::Duration::from_millis(100))
{
if let Some(last) = saved_blocks.last() {
if let Some(last) = plan.last_block() {
last_persisted_number = last.number;
} else if let Some(last) = plan.blocks.last() {
last_persisted_number = last.recovered_block().number;
}
sender
.send(PersistenceResult {
last_block: saved_blocks.last().map(|b| b.recovered_block().num_hash()),
last_block: plan.last_block(),
last_state_trie_block: plan.last_block().map(|tip| tip.number),
commit_duration: Some(Duration::ZERO),
})
.unwrap();
@@ -2254,65 +2460,3 @@ fn test_on_valid_downloaded_head_sync_target_returns_make_canonical() {
other => panic!("Expected MakeCanonical for head block, got: {other:?}"),
}
}
/// Tests that canonicalizing a downloaded sync target head also applies the tracked finalized
/// block from the original `SYNCING` forkchoice state.
#[test]
fn test_canonicalizing_downloaded_sync_target_head_updates_finalized() {
reth_tracing::init_test_tracing();
let chain_spec = MAINNET.clone();
let mut test_harness = TestHarness::new(chain_spec);
let blocks: Vec<_> = test_harness.block_builder.get_executed_blocks(0..3).collect();
let genesis = &blocks[0];
let finalized_block = &blocks[1];
let head_block = &blocks[2];
test_harness = test_harness.with_blocks(vec![
genesis.clone(),
finalized_block.clone(),
head_block.clone(),
]);
let finalized_num_hash = finalized_block.recovered_block().num_hash();
let head_num_hash = head_block.recovered_block().num_hash();
test_harness.tree.state.tree_state.set_canonical_head(genesis.recovered_block().num_hash());
let fcu_state = ForkchoiceState {
head_block_hash: head_num_hash.hash,
safe_block_hash: head_num_hash.hash,
finalized_block_hash: finalized_num_hash.hash,
};
test_harness
.tree
.state
.forkchoice_state_tracker
.set_latest(fcu_state, ForkchoiceStatus::Syncing);
let event = test_harness
.tree
.on_valid_downloaded_block(head_num_hash)
.unwrap()
.expect("expected canonicalization event for sync target head");
test_harness.tree.on_tree_event(event).unwrap();
assert_eq!(test_harness.tree.state.tree_state.canonical_block_hash(), head_num_hash.hash);
assert_eq!(
test_harness.tree.canonical_in_memory_state.get_finalized_num_hash(),
Some(finalized_num_hash),
"Finalized block from the syncing FCU should be applied once the head becomes canonical"
);
assert_eq!(
test_harness.tree.canonical_in_memory_state.get_safe_num_hash(),
Some(head_num_hash),
"Safe block from the syncing FCU should be applied once the head becomes canonical"
);
assert_eq!(
test_harness.tree.state.forkchoice_state_tracker.last_valid_state(),
Some(fcu_state)
);
assert!(test_harness.tree.state.forkchoice_state_tracker.sync_target_state().is_none());
}

View File

@@ -12,7 +12,7 @@ use alloy_rpc_types_eth::TransactionRequest;
use rand::{rngs::StdRng, Rng, SeedableRng};
use reth_chainspec::{ChainSpecBuilder, EthChainSpec, MAINNET};
use reth_e2e_test_utils::setup_engine;
use reth_network::{types::NatResolver, PeersInfo};
use reth_network::types::NatResolver;
use reth_node_builder::{NodeBuilder, NodeHandle};
use reth_node_core::{
args::{NetworkArgs, RpcServerArgs},
@@ -375,47 +375,3 @@ async fn test_admin_external_ip() -> eyre::Result<()> {
Ok(())
}
#[tokio::test]
async fn test_admin_node_info_uses_discv5_port_when_discv4_is_disabled() -> eyre::Result<()> {
reth_tracing::init_test_tracing();
let runtime = Runtime::test();
let genesis: Genesis = serde_json::from_str(include_str!("../assets/genesis.json")).unwrap();
let chain_spec =
Arc::new(ChainSpecBuilder::default().chain(MAINNET.chain).genesis(genesis).build());
let mut network = NetworkArgs::default().with_unused_ports();
network.bootnodes = Some(Vec::new());
network.discovery.disable_dns_discovery = true;
network.discovery.disable_discv4_discovery = true;
network = network.with_nat_resolver(NatResolver::ExternalIp("127.0.0.1".parse().unwrap()));
let node_config = NodeConfig::test()
.with_chain(chain_spec)
.with_network(network)
.with_rpc(RpcServerArgs::default().with_unused_ports().with_http());
let NodeHandle { node, node_exit_future: _ } = NodeBuilder::new(node_config)
.testing_node(runtime)
.node(EthereumNode::default())
.launch()
.await?;
assert!(node.network.discv4().is_none());
let discv5_port = node.network.discv5().expect("discv5 should be enabled").local_port();
let local_record = node.network.local_node_record();
let local_enr = node.network.local_enr();
let info = node.add_ons_handle.admin_api().node_info().await.unwrap();
assert_eq!(local_record.udp_port, discv5_port);
assert_eq!(local_enr.udp4(), Some(discv5_port));
assert_eq!(info.ports.discovery, discv5_port);
assert_eq!(info.ports.listener, local_record.tcp_port);
assert_eq!(info.enode, local_record.to_string());
assert!(info.enode.contains(&format!("?discport={discv5_port}")));
Ok(())
}

View File

@@ -21,7 +21,6 @@ use reth_consensus_common::validation::MAX_RLP_BLOCK_SIZE;
use reth_errors::{BlockExecutionError, BlockValidationError, ConsensusError};
use reth_ethereum_primitives::{EthPrimitives, TransactionSigned};
use reth_evm::{
block::TxResult,
execute::{BlockBuilder, BlockBuilderOutcome, BlockExecutor},
ConfigureEvm, Evm, NextBlockEnvAttributes,
};
@@ -38,7 +37,7 @@ use reth_transaction_pool::{
BestTransactions, BestTransactionsAttributes, PoolTransaction, TransactionPool,
ValidPoolTransaction,
};
use revm::context_interface::{Block as _, Cfg as _};
use revm::context_interface::Block as _;
use std::sync::Arc;
use tracing::{debug, trace, warn};
@@ -205,11 +204,8 @@ where
.map_err(PayloadBuilderError::other)?;
debug!(target: "payload_builder", id=%payload_id, parent_header = ?parent_header.hash(), parent_number = parent_header.number, "building new payload");
let mut cumulative_tx_gas_used = 0;
let mut block_regular_gas_used = 0;
let mut block_state_gas_used = 0;
let mut cumulative_gas_used = 0;
let block_gas_limit: u64 = builder.evm_mut().block().gas_limit();
let tx_gas_limit_cap = builder.evm_mut().cfg_env().tx_gas_limit_cap();
let base_fee = builder.evm_mut().block().basefee();
let mut best_txs = best_txs(BestTransactionsAttributes::new(
@@ -254,34 +250,13 @@ where
while let Some(pool_tx) = best_txs.next() {
// ensure we still have capacity for this transaction
let exceeds_gas_limit = if is_amsterdam {
let regular_available_gas = block_gas_limit.saturating_sub(block_regular_gas_used);
let state_available_gas = block_gas_limit.saturating_sub(block_state_gas_used);
let regular_tx_gas_limit = pool_tx.gas_limit().min(tx_gas_limit_cap);
if regular_tx_gas_limit > regular_available_gas {
Some((regular_tx_gas_limit, regular_available_gas))
} else if pool_tx.gas_limit() > state_available_gas {
Some((pool_tx.gas_limit(), state_available_gas))
} else {
None
}
} else {
let block_available_gas = block_gas_limit.saturating_sub(cumulative_tx_gas_used);
(pool_tx.gas_limit() > block_available_gas)
.then_some((pool_tx.gas_limit(), block_available_gas))
};
if let Some((transaction_gas_limit, block_available_gas)) = exceeds_gas_limit {
if cumulative_gas_used + pool_tx.gas_limit() > block_gas_limit {
// we can't fit this transaction into the block, so we need to mark it as invalid
// which also removes all dependent transaction from the iterator before we can
// continue
best_txs.mark_invalid(
&pool_tx,
&InvalidPoolTransactionError::ExceedsGasLimit(
transaction_gas_limit,
block_available_gas,
),
&InvalidPoolTransactionError::ExceedsGasLimit(pool_tx.gas_limit(), block_gas_limit),
);
continue
}
@@ -366,11 +341,8 @@ where
let miner_fee = tx.effective_tip_per_gas(base_fee);
let tx_hash = *tx.tx_hash();
let mut tx_regular_gas_used = 0;
let gas_output = match builder.execute_transaction_with_result_closure(tx, |result| {
tx_regular_gas_used = result.result().result.gas().block_regular_gas_used();
}) {
Ok(gas_output) => gas_output,
let gas_used = match builder.execute_transaction(tx) {
Ok(gas_used) => gas_used.tx_gas_used(),
Err(BlockExecutionError::Validation(BlockValidationError::InvalidTx {
error, ..
})) => {
@@ -390,8 +362,9 @@ where
}
continue
}
// The executor is the source of truth for block gas availability. Keep this
// non-fatal in case local builder accounting diverges from executor rules.
// EIP-7778: the executor tracks gas_before_refund while the payload builder's
// pre-check uses gas_after_refund. Near-full blocks can pass the pre-check but
// fail the executor's check. Skip the tx and continue building.
Err(BlockExecutionError::Validation(
BlockValidationError::TransactionGasLimitMoreThanAvailableBlockGas {
transaction_gas_limit,
@@ -425,12 +398,9 @@ where
block_transactions_rlp_length += tx_rlp_len;
// update and add to total fees
let gas_used = gas_output.tx_gas_used();
let miner_fee = miner_fee.expect("fee is always valid; execution succeeded");
total_fees += U256::from(miner_fee) * U256::from(gas_used);
cumulative_tx_gas_used += gas_used;
block_regular_gas_used += tx_regular_gas_used;
block_state_gas_used += gas_output.state_gas_used();
cumulative_gas_used += gas_used;
// Add blob tx sidecar to the payload.
if let Some(sidecar) = blob_tx_sidecar {

View File

@@ -1,11 +1,8 @@
//! Helper aliases when working with [`ConfigureEvm`] and the traits in this crate.
use crate::ConfigureEvm;
use alloy_evm::{
block::{BlockExecutorFactory, BlockExecutorFor},
Database, EvmEnv, EvmFactory,
};
use revm::{database::State, inspector::NoOpInspector, Inspector};
use alloy_evm::{block::BlockExecutorFactory, Database, EvmEnv, EvmFactory};
use revm::{inspector::NoOpInspector, Inspector};
/// Helper to access [`EvmFactory`] for a given [`ConfigureEvm`].
pub type EvmFactoryFor<Evm> =
@@ -36,10 +33,6 @@ pub type TxEnvFor<Evm> = <EvmFactoryFor<Evm> as EvmFactory>::Tx;
pub type ExecutionCtxFor<'a, Evm> =
<<Evm as ConfigureEvm>::BlockExecutorFactory as BlockExecutorFactory>::ExecutionCtx<'a>;
/// Helper to access [`alloy_evm::block::BlockExecutor`] for a given [`ConfigureEvm`].
pub type BlockExecutorForEvm<'a, Evm, DB, I = NoOpInspector> =
BlockExecutorFor<'a, <Evm as ConfigureEvm>::BlockExecutorFactory, &'a mut State<DB>, I>;
/// Type alias for [`EvmEnv`] for a given [`ConfigureEvm`].
pub type EvmEnvFor<Evm> = EvmEnv<SpecFor<Evm>, BlockEnvFor<Evm>>;

View File

@@ -20,7 +20,10 @@ extern crate alloc;
use crate::execute::{BasicBlockBuilder, Executor};
use alloc::vec::Vec;
use alloy_eips::eip4895::Withdrawals;
use alloy_evm::{block::BlockExecutorFactory, precompiles::PrecompilesMap};
use alloy_evm::{
block::{BlockExecutorFactory, BlockExecutorFor},
precompiles::PrecompilesMap,
};
use alloy_primitives::{Address, Bytes, B256};
use core::{error::Error, fmt::Debug};
use execute::{BasicBlockExecutor, BlockAssembler, BlockBuilder};
@@ -309,7 +312,7 @@ pub trait ConfigureEvm: Clone + Debug + Send + Sync + Unpin {
&'a self,
evm: EvmFor<Self, &'a mut State<DB>, I>,
ctx: <Self::BlockExecutorFactory as BlockExecutorFactory>::ExecutionCtx<'a>,
) -> BlockExecutorForEvm<'a, Self, DB, I>
) -> impl BlockExecutorFor<'a, Self::BlockExecutorFactory, &'a mut State<DB>, I>
where
DB: Database,
I: InspectorFor<Self, &'a mut State<DB>> + 'a,
@@ -322,7 +325,8 @@ pub trait ConfigureEvm: Clone + Debug + Send + Sync + Unpin {
&'a self,
db: &'a mut State<DB>,
block: &'a SealedBlock<<Self::Primitives as NodePrimitives>::Block>,
) -> Result<BlockExecutorForEvm<'a, Self, DB>, Self::Error> {
) -> Result<impl BlockExecutorFor<'a, Self::BlockExecutorFactory, &'a mut State<DB>>, Self::Error>
{
let evm = self.evm_for_block(db, block.header())?;
let ctx = self.context_for_block(block)?;
Ok(self.create_executor(evm, ctx))
@@ -348,7 +352,10 @@ pub trait ConfigureEvm: Clone + Debug + Send + Sync + Unpin {
evm: EvmFor<Self, &'a mut State<DB>, I>,
parent: &'a SealedHeader<HeaderTy<Self::Primitives>>,
ctx: <Self::BlockExecutorFactory as BlockExecutorFactory>::ExecutionCtx<'a>,
) -> impl BlockBuilder<Primitives = Self::Primitives, Executor = BlockExecutorForEvm<'a, Self, DB, I>>
) -> impl BlockBuilder<
Primitives = Self::Primitives,
Executor: BlockExecutorFor<'a, Self::BlockExecutorFactory, &'a mut State<DB>, I>,
>
where
DB: Database,
I: InspectorFor<Self, &'a mut State<DB>> + 'a,
@@ -397,7 +404,10 @@ pub trait ConfigureEvm: Clone + Debug + Send + Sync + Unpin {
parent: &'a SealedHeader<<Self::Primitives as NodePrimitives>::BlockHeader>,
attributes: Self::NextBlockEnvCtx,
) -> Result<
impl BlockBuilder<Primitives = Self::Primitives, Executor = BlockExecutorForEvm<'a, Self, DB>>,
impl BlockBuilder<
Primitives = Self::Primitives,
Executor: BlockExecutorFor<'a, Self::BlockExecutorFactory, &'a mut State<DB>>,
>,
Self::Error,
> {
let evm_env = self.next_evm_env(parent, &attributes)?;

View File

@@ -16,13 +16,10 @@ workspace = true
metrics.workspace = true
metrics-derive.workspace = true
# reth
reth-primitives-traits = { workspace = true, optional = true }
# async
tokio = { workspace = true, features = ["full"], optional = true }
futures = { workspace = true, optional = true }
tokio-util = { workspace = true, optional = true }
[features]
common = ["tokio", "futures", "tokio-util", "reth-primitives-traits"]
common = ["tokio", "futures", "tokio-util"]

View File

@@ -4,13 +4,8 @@
use crate::Metrics;
use futures::Stream;
use metrics::Counter;
use reth_primitives_traits::InMemorySize;
use std::{
pin::Pin,
sync::{
atomic::{AtomicUsize, Ordering},
Arc,
},
task::{ready, Context, Poll},
};
use tokio::sync::mpsc::{
@@ -404,147 +399,3 @@ struct MeteredPollSenderMetrics {
/// Number of delayed message deliveries caused by a full channel
back_pressure_total: Counter,
}
/// Shared state for tracking memory budget across sender and receiver.
///
/// `used` is a pure accounting counter — it does not gate access to any other
/// shared memory, so all operations on it use [`Ordering::Relaxed`]. Cross-thread
/// publication of message contents is handled by the underlying mpsc channel.
#[derive(Debug)]
struct MemoryBudget {
/// Current number of bytes used by buffered messages.
used: AtomicUsize,
/// Maximum allowed bytes.
max_bytes: usize,
}
/// Guard that releases memory budget when dropped.
///
/// Holds the size of the message and a reference to the shared budget counter.
/// When dropped, it atomically decreases the used counter.
#[derive(Debug)]
struct BudgetGuard {
size: usize,
budget: Arc<MemoryBudget>,
}
impl Drop for BudgetGuard {
fn drop(&mut self) {
self.budget.used.fetch_sub(self.size, Ordering::Relaxed);
}
}
/// Message envelope that holds the memory budget while the message sits in the channel.
///
/// The guard is dropped (releasing the budget) as soon as the receiver dequeues
/// the message via [`MemoryBoundedReceiver::recv`] / [`MemoryBoundedReceiver::poll_recv`],
/// so the budget tracks bytes *currently in the channel queue*, not bytes in flight
/// downstream of the receiver.
#[derive(Debug)]
struct Budgeted<T> {
msg: T,
_guard: BudgetGuard,
}
/// A sender that enforces a byte budget before enqueueing messages.
///
/// Uses a shared atomic counter to track memory usage. Each message's size is added
/// to the counter on send and subtracted when the message is dequeued by the receiver.
///
/// The current call sites (specifically [`crate::common::mpsc::MemoryBoundedSender`] used
/// for the `NetworkManager → TransactionsManager` channel) have a single producer driven
/// from a single `poll`, so the `fetch_add → check → fetch_sub-on-overflow` reservation
/// pattern can never race with itself. The atomic is still used so the receiver can
/// release budget from a different task.
#[derive(Debug, Clone)]
pub struct MemoryBoundedSender<T: InMemorySize> {
/// The underlying unbounded metered sender
inner: UnboundedMeteredSender<Budgeted<T>>,
/// Shared memory budget tracker
budget: Arc<MemoryBudget>,
}
impl<T: InMemorySize> MemoryBoundedSender<T> {
/// Tries to send a message if there is sufficient budget.
///
/// Returns `TrySendError::Full` if insufficient budget is available.
pub fn try_send(&self, msg: T) -> Result<(), TrySendError<T>> {
let size = msg.size();
// Reserve budget: add first, check after
let prev = self.budget.used.fetch_add(size, Ordering::Relaxed);
if prev.saturating_add(size) > self.budget.max_bytes {
// Over budget, undo
self.budget.used.fetch_sub(size, Ordering::Relaxed);
return Err(TrySendError::Full(msg));
}
let guard = BudgetGuard { size, budget: Arc::clone(&self.budget) };
let budgeted = Budgeted { msg, _guard: guard };
self.inner.send(budgeted).map_err(|e| {
// Guard will be dropped here, releasing the budget
TrySendError::Closed(e.0.msg)
})
}
}
/// A receiver for memory-bounded messages.
///
/// On receive, the budget reserved for the message is released immediately and the
/// inner `T` is yielded — callers do not need to opt into any wrapper type.
#[derive(Debug)]
pub struct MemoryBoundedReceiver<T> {
/// The underlying unbounded metered receiver
inner: UnboundedMeteredReceiver<Budgeted<T>>,
}
impl<T> MemoryBoundedReceiver<T> {
/// Receives the next message, returning `None` if the channel is closed.
///
/// Releases the message's reserved budget before returning.
pub async fn recv(&mut self) -> Option<T> {
self.inner.recv().await.map(unwrap_budgeted)
}
/// Polls to receive the next message on this channel.
///
/// Releases the message's reserved budget before returning.
pub fn poll_recv(&mut self, cx: &mut Context<'_>) -> Poll<Option<T>> {
self.inner.poll_recv(cx).map(|opt| opt.map(unwrap_budgeted))
}
}
/// Releases the budget guard and returns the inner message.
fn unwrap_budgeted<T>(b: Budgeted<T>) -> T {
// Destructuring binds `_guard` so it is dropped when this function returns,
// which runs `BudgetGuard::drop` and releases the reserved bytes.
let Budgeted { msg, _guard } = b;
msg
}
impl<T> Stream for MemoryBoundedReceiver<T> {
type Item = T;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
self.poll_recv(cx)
}
}
/// Creates a new memory-bounded channel with the given byte budget.
///
/// The budget tracks bytes currently buffered in the channel; it is reserved on
/// [`MemoryBoundedSender::try_send`] and released as soon as the receiver dequeues
/// the message.
pub fn memory_bounded_channel<T: InMemorySize>(
max_bytes: usize,
scope: &'static str,
) -> (MemoryBoundedSender<T>, MemoryBoundedReceiver<T>) {
let (tx, rx) = metered_unbounded_channel(scope);
let budget = Arc::new(MemoryBudget { used: AtomicUsize::new(0), max_bytes });
let sender = MemoryBoundedSender { inner: tx, budget };
let receiver = MemoryBoundedReceiver { inner: rx };
(sender, receiver)
}

View File

@@ -47,7 +47,9 @@ use secp256k1::SecretKey;
use std::{
cell::RefCell,
collections::{btree_map, hash_map::Entry, BTreeMap, HashMap, VecDeque},
fmt, io,
fmt,
future::poll_fn,
io,
net::{IpAddr, Ipv4Addr, SocketAddr, SocketAddrV4},
pin::Pin,
rc::Rc,
@@ -241,56 +243,17 @@ impl Discv4 {
/// ```
pub async fn bind(
local_address: SocketAddr,
local_node_record: NodeRecord,
secret_key: SecretKey,
config: Discv4Config,
) -> io::Result<(Self, Discv4Service)> {
let socket = Arc::new(UdpSocket::bind(local_address).await?);
trace!(target: "discv4", local_addr=?socket.local_addr(), "opened UDP socket");
let (tx, rx) = mpsc::channel(config.udp_ingress_message_buffer);
Self::bind_with_socket(socket, Some(tx), rx, local_node_record, secret_key, config)
}
/// Creates a new `Discv4` instance using a pre-bound shared socket. No receive loop is
/// spawned; instead returns an [`IngressHandler`] that should be used to forward raw packets
/// received by the socket owner (e.g. discv5 unrecognized frames).
pub fn bind_shared(
socket: Arc<UdpSocket>,
local_node_record: NodeRecord,
secret_key: SecretKey,
config: Discv4Config,
) -> io::Result<(Self, Discv4Service, IngressHandler)> {
let (tx, rx) = mpsc::channel(config.udp_ingress_message_buffer);
let local_id = local_node_record.id;
let (discv4, service) =
Self::bind_with_socket(socket, None, rx, local_node_record, secret_key, config)?;
let handler = IngressHandler::new(tx, local_id);
Ok((discv4, service, handler))
}
fn bind_with_socket(
socket: Arc<UdpSocket>,
ingress_tx: Option<IngressSender>,
ingress_rx: IngressReceiver,
mut local_node_record: NodeRecord,
secret_key: SecretKey,
config: Discv4Config,
) -> io::Result<(Self, Discv4Service)> {
let socket = UdpSocket::bind(local_address).await?;
let local_addr = socket.local_addr()?;
local_node_record.udp_port = local_addr.port();
trace!(target: "discv4", ?local_addr,"opened UDP socket");
let mut service = Discv4Service::new(
socket,
ingress_tx,
ingress_rx,
local_addr,
local_node_record,
secret_key,
config,
);
let mut service =
Discv4Service::new(socket, local_addr, local_node_record, secret_key, config);
// resolve the external address immediately
service.resolve_external_ip();
@@ -557,25 +520,20 @@ pub struct Discv4Service {
impl Discv4Service {
/// Create a new instance for a bound [`UdpSocket`].
///
/// If `ingress_tx` is `Some`, the receive loop is spawned to read from the socket. If `None`,
/// the caller feeds packets into `ingress_rx` externally (shared socket mode).
pub(crate) fn new(
socket: Arc<UdpSocket>,
ingress_tx: Option<IngressSender>,
ingress_rx: IngressReceiver,
socket: UdpSocket,
local_address: SocketAddr,
local_node_record: NodeRecord,
secret_key: SecretKey,
config: Discv4Config,
) -> Self {
let socket = Arc::new(socket);
let (ingress_tx, ingress_rx) = mpsc::channel(config.udp_ingress_message_buffer);
let (egress_tx, egress_rx) = mpsc::channel(config.udp_egress_message_buffer);
let mut tasks = JoinSet::<()>::new();
if let Some(ingress_tx) = ingress_tx {
let udp = Arc::clone(&socket);
tasks.spawn(receive_loop(udp, ingress_tx, local_node_record.id));
}
let udp = Arc::clone(&socket);
tasks.spawn(receive_loop(udp, ingress_tx, local_node_record.id));
let udp = Arc::clone(&socket);
tasks.spawn(send_loop(udp, egress_rx));
@@ -989,7 +947,7 @@ impl Discv4Service {
let key = kad_key(peer_id);
match self.kbuckets.entry(&key) {
BucketEntry::Present(entry, _) => Some(f(entry.value())),
BucketEntry::Pending(entry, _) => Some(f(entry.value())),
BucketEntry::Pending(mut entry, _) => Some(f(entry.value())),
_ => None,
}
}
@@ -1015,9 +973,7 @@ impl Discv4Service {
kbucket::Entry::Present(mut entry, _) => {
entry.value_mut().update_with_enr(last_enr_seq)
}
kbucket::Entry::Pending(mut entry, _) => {
entry.value_mut().update_with_enr(last_enr_seq)
}
kbucket::Entry::Pending(mut entry, _) => entry.value().update_with_enr(last_enr_seq),
_ => return,
};
@@ -1069,8 +1025,8 @@ impl Discv4Service {
}
kbucket::Entry::Pending(mut entry, mut status) => {
// endpoint is now proven
entry.value_mut().establish_proof();
entry.value_mut().update_with_enr(last_enr_seq);
entry.value().establish_proof();
entry.value().update_with_enr(last_enr_seq);
if !status.is_connected() {
status.state = ConnectionState::Connected;
@@ -1202,7 +1158,7 @@ impl Discv4Service {
} else {
is_proven = entry.value().has_endpoint_proof;
}
entry.value_mut().update_with_enr(ping.enr_sq)
entry.value().update_with_enr(ping.enr_sq)
}
kbucket::Entry::Absent(entry) => {
let mut node = NodeEntry::new(record);
@@ -1432,7 +1388,7 @@ impl Discv4Service {
(entry.value().record, id)
}
kbucket::Entry::Pending(mut entry, _) => {
let id = entry.value_mut().update_with_fork_id(fork_id);
let id = entry.value().update_with_fork_id(fork_id);
(entry.value().record, id)
}
_ => return,
@@ -1582,7 +1538,7 @@ impl Discv4Service {
}
}
}
BucketEntry::Pending(entry, _) => {
BucketEntry::Pending(mut entry, _) => {
if entry.value().has_endpoint_proof {
if entry
.value()
@@ -1686,7 +1642,7 @@ impl Discv4Service {
entry.value().find_node_failures
}
kbucket::Entry::Pending(mut entry, _) => {
entry.value_mut().inc_failed_request();
entry.value().inc_failed_request();
entry.value().find_node_failures
}
_ => continue,
@@ -2006,100 +1962,80 @@ const MAX_INCOMING_PACKETS_PER_MINUTE_BY_IP: usize = 60usize;
/// Continuously awaits new incoming messages and sends them back through the channel.
///
/// The receive loop enforces primitive rate limiting for IPs to prevent message spams from
/// individual IPs.
/// The receive loop enforce primitive rate limiting for ips to prevent message spams from
/// individual IPs
pub(crate) async fn receive_loop(udp: Arc<UdpSocket>, tx: IngressSender, local_id: PeerId) {
let mut handler = IngressHandler::new(tx, local_id);
let send = |event: IngressEvent| async {
let _ = tx.send(event).await.map_err(|err| {
debug!(
target: "discv4",
%err,
"failed send incoming packet",
)
});
};
let mut cache = ReceiveCache::default();
// tick at half the rate of the limit
let tick = MAX_INCOMING_PACKETS_PER_MINUTE_BY_IP / 2;
let mut interval = tokio::time::interval(Duration::from_secs(tick as u64));
let mut buf = [0; MAX_PACKET_SIZE];
loop {
let res = udp.recv_from(&mut buf).await;
match res {
Err(err) => {
debug!(target: "discv4", %err, "Failed to read datagram.");
handler.send(IngressEvent::RecvError(err)).await;
send(IngressEvent::RecvError(err)).await;
}
Ok((read, remote_addr)) => {
handler.handle_packet(&buf[..read], remote_addr).await;
}
}
}
}
/// Handles decoding, rate-limiting, and deduplication of incoming discv4 packets.
///
/// Used by both the standalone receive loop and the shared-port mode via
/// [`Discv4::bind_shared`].
#[derive(Debug)]
pub struct IngressHandler {
tx: IngressSender,
local_id: PeerId,
tick: usize,
tick_interval: Duration,
cache: ReceiveCache,
last_tick: Instant,
}
impl IngressHandler {
fn new(tx: IngressSender, local_id: PeerId) -> Self {
let tick = MAX_INCOMING_PACKETS_PER_MINUTE_BY_IP / 2;
Self {
tx,
local_id,
tick,
tick_interval: Duration::from_secs(tick as u64),
cache: ReceiveCache::default(),
last_tick: Instant::now(),
}
}
async fn send(&self, event: IngressEvent) {
let _ = self.tx.send(event).await.map_err(|err| {
debug!(target: "discv4", %err, "failed send incoming packet");
});
}
/// Handles an incoming raw packet: decodes, rate-limits, deduplicates, and forwards to the
/// discv4 service. Used in shared-port mode to process unrecognized frames from discv5.
pub async fn handle_packet(&mut self, data: &[u8], src: SocketAddr) {
if self.last_tick.elapsed() >= self.tick_interval {
self.cache.tick_ips(self.tick);
self.last_tick = Instant::now();
}
// rate limit incoming packets by IP
if self.cache.inc_ip(src.ip()) > MAX_INCOMING_PACKETS_PER_MINUTE_BY_IP {
trace!(target: "discv4", ?src, "Too many incoming packets from IP.");
return
}
let event = match Message::decode(data) {
Ok(packet) => {
if packet.node_id == self.local_id {
debug!(target: "discv4", ?src, "Received own packet.");
return
// rate limit incoming packets by IP
if cache.inc_ip(remote_addr.ip()) > MAX_INCOMING_PACKETS_PER_MINUTE_BY_IP {
trace!(target: "discv4", ?remote_addr, "Too many incoming packets from IP.");
continue
}
if self.cache.contains_packet(packet.hash) {
debug!(target: "discv4", ?src, "Received duplicate packet.");
return
let packet = &buf[..read];
match Message::decode(packet) {
Ok(packet) => {
if packet.node_id == local_id {
// received our own message
debug!(target: "discv4", ?remote_addr, "Received own packet.");
continue
}
// skip if we've already received the same packet
if cache.contains_packet(packet.hash) {
debug!(target: "discv4", ?remote_addr, "Received duplicate packet.");
continue
}
send(IngressEvent::Packet(remote_addr, packet)).await;
}
Err(err) => {
trace!(target: "discv4", %err,"Failed to decode packet");
send(IngressEvent::BadPacket(remote_addr, err, packet.to_vec())).await
}
}
IngressEvent::Packet(src, packet)
}
Err(err) => {
trace!(target: "discv4", %err, "Failed to decode packet");
IngressEvent::BadPacket(src, err, data.to_vec())
}
};
}
self.send(event).await;
// reset the tracked ips if the interval has passed
if poll_fn(|cx| match interval.poll_tick(cx) {
Poll::Ready(_) => Poll::Ready(true),
Poll::Pending => Poll::Ready(false),
})
.await
{
cache.tick_ips(tick);
}
}
}
/// A cache for received packets and their source address.
///
/// This is used to discard duplicated packets and rate limit messages from the same source.
#[derive(Debug)]
struct ReceiveCache {
/// keeps track of how many messages we've received from a given IP address since the last
/// tick.

View File

@@ -308,18 +308,6 @@ impl Config {
}
}
/// Returns a mutable reference to the inner [`discv5::Config`]. This allows overriding
/// the listen config after the config has been built.
pub const fn discv5_config_mut(&mut self) -> &mut discv5::Config {
&mut self.discv5_config
}
/// Returns `true` if any socket in the discv5 listen config matches the given address.
pub fn has_matching_socket(&self, addr: SocketAddr) -> bool {
ipv4(&self.discv5_config.listen_config).is_some_and(|v4| SocketAddr::V4(v4) == addr) ||
ipv6(&self.discv5_config.listen_config).is_some_and(|v6| SocketAddr::V6(v6) == addr)
}
/// Inserts a new boot node to the list of boot nodes.
pub fn insert_boot_node(&mut self, boot_node: BootNode) {
self.bootstrap_nodes.insert(boot_node);
@@ -345,11 +333,11 @@ impl Config {
/// socket, if both IPv4 and v6 are configured. This socket will be advertised to peers in the
/// local [`Enr`](discv5::enr::Enr).
pub fn discovery_socket(&self) -> SocketAddr {
// Prefer v6 when both are configured (matches original `DualStack` behavior).
ipv6(&self.discv5_config.listen_config)
.map(SocketAddr::V6)
.or_else(|| ipv4(&self.discv5_config.listen_config).map(SocketAddr::V4))
.unwrap_or_else(|| SocketAddr::from((std::net::Ipv4Addr::UNSPECIFIED, 0)))
match self.discv5_config.listen_config {
ListenConfig::Ipv4 { ip, port } => (ip, port).into(),
ListenConfig::Ipv6 { ip, port } => (ip, port).into(),
ListenConfig::DualStack { ipv6, ipv6_port, .. } => (ipv6, ipv6_port).into(),
}
}
/// Returns the `RLPx` (TCP) socket contained in the [`discv5::Config`]. This socket will be
@@ -360,32 +348,24 @@ impl Config {
}
/// Returns the IPv4 discovery socket if one is configured.
pub fn ipv4(listen_config: &ListenConfig) -> Option<SocketAddrV4> {
pub const fn ipv4(listen_config: &ListenConfig) -> Option<SocketAddrV4> {
match listen_config {
ListenConfig::Ipv4 { ip, port } |
ListenConfig::DualStack { ipv4: ip, ipv4_port: port, .. } => {
Some(SocketAddrV4::new(*ip, *port))
}
ListenConfig::FromSockets { ipv4: Some(s), .. } => match s.local_addr().ok()? {
SocketAddr::V4(addr) => Some(addr),
SocketAddr::V6(_) => None,
},
_ => None,
ListenConfig::Ipv6 { .. } => None,
}
}
/// Returns the IPv6 discovery socket if one is configured.
pub fn ipv6(listen_config: &ListenConfig) -> Option<SocketAddrV6> {
pub const fn ipv6(listen_config: &ListenConfig) -> Option<SocketAddrV6> {
match listen_config {
ListenConfig::Ipv4 { .. } => None,
ListenConfig::Ipv6 { ip, port } |
ListenConfig::DualStack { ipv6: ip, ipv6_port: port, .. } => {
Some(SocketAddrV6::new(*ip, *port, 0, 0))
}
ListenConfig::FromSockets { ipv6: Some(s), .. } => match s.local_addr().ok()? {
SocketAddr::V6(addr) => Some(addr),
SocketAddr::V4(_) => None,
},
_ => None,
}
}

View File

@@ -18,6 +18,7 @@ use std::{
use ::enr::Enr;
use alloy_primitives::bytes::Bytes;
use discv5::ListenConfig;
use enr::{discv4_id_to_discv5_id, EnrCombinedKeyWrapper};
use futures::future::join_all;
use itertools::Itertools;
@@ -246,9 +247,7 @@ impl Discv5 {
match update {
discv5::Event::SocketUpdated(_) | discv5::Event::TalkRequest(_) |
// `Discovered` not unique discovered peers
discv5::Event::Discovered(_) |
// Unrecognized frames are handled separately by the discovery layer
discv5::Event::UnrecognizedFrame(_) => None,
discv5::Event::Discovered(_) => None,
discv5::Event::NodeInserted { .. } => {
// node has been inserted into kbuckets
@@ -473,33 +472,39 @@ pub fn build_local_enr(
let Config { discv5_config, fork, tcp_socket, other_enr_kv_pairs, .. } = config;
let socket = {
let v4 = crate::config::ipv4(&discv5_config.listen_config);
let v6 = crate::config::ipv6(&discv5_config.listen_config);
if let Some(addr) = v4 {
if *addr.ip() != Ipv4Addr::UNSPECIFIED {
builder.ip4(*addr.ip());
let socket = match discv5_config.listen_config {
ListenConfig::Ipv4 { ip, port } => {
if ip != Ipv4Addr::UNSPECIFIED {
builder.ip4(ip);
}
builder.udp4(addr.port());
}
if let Some(addr) = v6 {
if *addr.ip() != Ipv6Addr::UNSPECIFIED {
builder.ip6(*addr.ip());
}
builder.udp6(addr.port());
}
// Advertise tcp4 when v4 is configured, else tcp6.
if v4.is_some() {
builder.udp4(port);
builder.tcp4(tcp_socket.port());
} else if v6.is_some() {
builder.tcp6(tcp_socket.port());
}
// Prefer v6 when both are configured
v6.map(SocketAddr::V6)
.or_else(|| v4.map(SocketAddr::V4))
.unwrap_or_else(|| SocketAddr::from((Ipv4Addr::UNSPECIFIED, 0)))
(ip, port).into()
}
ListenConfig::Ipv6 { ip, port } => {
if ip != Ipv6Addr::UNSPECIFIED {
builder.ip6(ip);
}
builder.udp6(port);
builder.tcp6(tcp_socket.port());
(ip, port).into()
}
ListenConfig::DualStack { ipv4, ipv4_port, ipv6, ipv6_port } => {
if ipv4 != Ipv4Addr::UNSPECIFIED {
builder.ip4(ipv4);
}
builder.udp4(ipv4_port);
builder.tcp4(tcp_socket.port());
if ipv6 != Ipv6Addr::UNSPECIFIED {
builder.ip6(ipv6);
}
builder.udp6(ipv6_port);
(ipv6, ipv6_port).into()
}
};
let rlpx_ip_mode = if tcp_socket.is_ipv4() { IpMode::Ip4 } else { IpMode::Ip6 };
@@ -706,7 +711,6 @@ mod test {
#![allow(deprecated)]
use super::*;
use ::enr::{CombinedKey, EnrKey};
use discv5::ListenConfig;
use rand_08::thread_rng;
use reth_chainspec::MAINNET;
use std::{

View File

@@ -7,14 +7,13 @@ use alloy_primitives::{
Bytes, TxHash, B256, U128,
};
use alloy_rlp::{
Decodable, Encodable, Header, RlpDecodable, RlpDecodableWrapper, RlpEncodable,
RlpEncodableWrapper,
Decodable, Encodable, RlpDecodable, RlpDecodableWrapper, RlpEncodable, RlpEncodableWrapper,
};
use core::{fmt::Debug, mem};
use derive_more::{Constructor, Deref, DerefMut, From, IntoIterator};
use reth_codecs_derive::{add_arbitrary_tests, generate_tests};
use reth_ethereum_primitives::TransactionSigned;
use reth_primitives_traits::{Block, InMemorySize, SignedTransaction};
use reth_primitives_traits::{Block, SignedTransaction};
/// This informs peers of new blocks that have appeared on the network.
#[derive(
@@ -144,53 +143,6 @@ impl<T> From<Transactions<T>> for Vec<T> {
}
}
impl<T: Decodable + InMemorySize> Transactions<T> {
/// Decodes the RLP list of transactions, stopping once the cumulative
/// [`InMemorySize`] of decoded transactions exceeds `memory_budget` bytes.
/// Any remaining transactions in the payload are skipped.
pub fn decode_with_memory_budget(
buf: &mut &[u8],
memory_budget: usize,
) -> alloy_rlp::Result<Self> {
decode_list_with_memory_budget(buf, memory_budget).map(Self)
}
}
/// Decodes an RLP list, stopping once the cumulative [`InMemorySize`] of decoded items exceeds
/// `memory_budget` bytes. Any remaining items in the payload are skipped.
pub fn decode_list_with_memory_budget<T: Decodable + InMemorySize>(
buf: &mut &[u8],
memory_budget: usize,
) -> alloy_rlp::Result<Vec<T>> {
let header = Header::decode(buf)?;
if !header.list {
return Err(alloy_rlp::Error::UnexpectedString);
}
if buf.len() < header.payload_length {
return Err(alloy_rlp::Error::InputTooShort);
}
let (payload, rest) = buf.split_at(header.payload_length);
let mut payload = payload;
let mut txs = Vec::new();
let mut total_size = 0usize;
while !payload.is_empty() {
let item = T::decode(&mut payload)?;
total_size = total_size.saturating_add(item.size());
if total_size > memory_budget {
break;
}
txs.push(item);
}
*buf = rest;
Ok(txs)
}
/// Same as [`Transactions`] but this is intended as egress message send from local to _many_ peers.
///
/// The list of transactions is constructed on per-peers basis, but the underlying transaction
@@ -885,19 +837,6 @@ pub struct BlockRangeUpdate {
pub latest_hash: B256,
}
impl InMemorySize for NewPooledTransactionHashes {
fn size(&self) -> usize {
match self {
Self::Eth66(msg) => msg.0.len() * core::mem::size_of::<B256>(),
Self::Eth68(msg) => {
msg.types.len() * core::mem::size_of::<u8>() +
msg.sizes.len() * core::mem::size_of::<usize>() +
msg.hashes.len() * core::mem::size_of::<B256>()
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;

View File

@@ -28,15 +28,6 @@ use core::fmt::Debug;
// https://github.com/ethereum/go-ethereum/blob/30602163d5d8321fbc68afdcbbaf2362b2641bde/eth/protocols/eth/protocol.go#L50
pub const MAX_MESSAGE_SIZE: usize = 10 * 1024 * 1024;
/// Multiplier applied to `max_message_size` to derive the in-memory budget for decoding
/// `Transactions` and `PooledTransactions` messages.
///
/// Decoded transactions expand relative to their RLP encoding due to struct overhead and heap
/// allocations. With many peers in flight this can cause significant memory pressure, so we
/// stop decoding once the cumulative in-memory size of decoded transactions exceeds
/// `max_message_size * TX_MEMORY_BUDGET_MULTIPLIER`. Remaining transactions are silently dropped.
pub const TX_MEMORY_BUDGET_MULTIPLIER: usize = 2;
/// Error when sending/receiving a message
#[derive(thiserror::Error, Debug)]
pub enum MessageError {
@@ -96,19 +87,6 @@ impl<N: NetworkPrimitives> ProtocolMessage<N> {
///
/// This will enforce decoding according to the given [`EthVersion`] of the connection.
pub fn decode_message(version: EthVersion, buf: &mut &[u8]) -> Result<Self, MessageError> {
Self::decode_message_with_tx_memory_budget(version, buf, usize::MAX)
}
/// Like [`Self::decode_message`], but caps the cumulative in-memory size of decoded
/// transactions in `Transactions` and `PooledTransactions` messages. Once exceeded,
/// remaining transactions are silently dropped.
///
/// Use [`TX_MEMORY_BUDGET_MULTIPLIER`] to derive a reasonable default.
pub fn decode_message_with_tx_memory_budget(
version: EthVersion,
buf: &mut &[u8],
tx_memory_budget: usize,
) -> Result<Self, MessageError> {
let message_type = EthMessageID::decode(buf)?;
// For EIP-7642 (https://github.com/ethereum/EIPs/blob/master/EIPS/eip-7642.md):
@@ -125,9 +103,7 @@ impl<N: NetworkPrimitives> ProtocolMessage<N> {
EthMessageID::NewBlock => {
EthMessage::NewBlock(Box::new(N::NewBlockPayload::decode(buf)?))
}
EthMessageID::Transactions => EthMessage::Transactions(
Transactions::decode_with_memory_budget(buf, tx_memory_budget)?,
),
EthMessageID::Transactions => EthMessage::Transactions(Transactions::decode(buf)?),
EthMessageID::NewPooledTransactionHashes => {
if version >= EthVersion::Eth68 {
EthMessage::NewPooledTransactionHashes68(NewPooledTransactionHashes68::decode(
@@ -147,9 +123,7 @@ impl<N: NetworkPrimitives> ProtocolMessage<N> {
EthMessage::GetPooledTransactions(RequestPair::decode(buf)?)
}
EthMessageID::PooledTransactions => {
EthMessage::PooledTransactions(RequestPair::decode_with(buf, |buf| {
PooledTransactions::decode_with_memory_budget(buf, tx_memory_budget)
})?)
EthMessage::PooledTransactions(RequestPair::decode(buf)?)
}
EthMessageID::GetNodeData => {
if version >= EthVersion::Eth67 {
@@ -758,25 +732,6 @@ impl<T> RequestPair<T> {
let Self { request_id, message } = self;
RequestPair { request_id, message: f(message) }
}
/// Decodes the request id and then decodes the message payload using `decode_msg`.
pub fn decode_with<F>(buf: &mut &[u8], decode_msg: F) -> alloy_rlp::Result<Self>
where
F: FnOnce(&mut &[u8]) -> alloy_rlp::Result<T>,
{
let header = Header::decode(buf)?;
let initial_length = buf.len();
let request_id = u64::decode(buf)?;
let message = decode_msg(buf)?;
let consumed_len = initial_length - buf.len();
if consumed_len != header.payload_length {
return Err(alloy_rlp::Error::UnexpectedLength)
}
Ok(Self { request_id, message })
}
}
/// Allows messages with request ids to be serialized into RLP bytes.

View File

@@ -1,14 +1,12 @@
//! Implements the `GetPooledTransactions` and `PooledTransactions` message types.
use crate::broadcast::decode_list_with_memory_budget;
use alloc::vec::Vec;
use alloy_consensus::transaction::PooledTransaction;
use alloy_eips::eip2718::Encodable2718;
use alloy_primitives::B256;
use alloy_rlp::{Decodable, RlpDecodableWrapper, RlpEncodableWrapper};
use alloy_rlp::{RlpDecodableWrapper, RlpEncodableWrapper};
use derive_more::{Constructor, Deref, IntoIterator};
use reth_codecs_derive::add_arbitrary_tests;
use reth_primitives_traits::InMemorySize;
/// A list of transaction hashes that the peer would like transaction bodies for.
#[derive(
@@ -39,12 +37,6 @@ where
}
}
impl InMemorySize for GetPooledTransactions {
fn size(&self) -> usize {
self.0.len() * core::mem::size_of::<B256>()
}
}
/// The response to [`GetPooledTransactions`], containing the transaction bodies associated with
/// the requested hashes.
///
@@ -70,18 +62,6 @@ pub struct PooledTransactions<T = PooledTransaction>(
pub Vec<T>,
);
impl<T: Decodable + InMemorySize> PooledTransactions<T> {
/// Decodes the RLP list of transactions, stopping once the cumulative
/// [`InMemorySize`] of decoded transactions exceeds `memory_budget` bytes.
/// Any remaining transactions in the payload are skipped.
pub fn decode_with_memory_budget(
buf: &mut &[u8],
memory_budget: usize,
) -> alloy_rlp::Result<Self> {
decode_list_with_memory_budget(buf, memory_budget).map(Self)
}
}
impl<T: Encodable2718> PooledTransactions<T> {
/// Returns an iterator over the transaction hashes in this response.
pub fn hashes(&self) -> impl Iterator<Item = B256> + '_ {

View File

@@ -5,7 +5,7 @@
use super::message::MAX_MESSAGE_SIZE;
use crate::{
message::{EthBroadcastMessage, ProtocolBroadcastMessage, TX_MEMORY_BUDGET_MULTIPLIER},
message::{EthBroadcastMessage, ProtocolBroadcastMessage},
EthMessage, EthMessageID, EthNetworkPrimitives, EthVersion, NetworkPrimitives, ProtocolMessage,
RawCapabilityMessage, SnapProtocolMessage, SnapVersion,
};
@@ -298,11 +298,7 @@ where
// See also <https://github.com/paradigmxyz/reth/blob/main/crates/net/eth-wire/src/capability.rs#L272-L283>.
if message_id <= EthMessageID::max(self.eth_version) {
let mut buf = bytes.as_ref();
match ProtocolMessage::decode_message_with_tx_memory_budget(
self.eth_version,
&mut buf,
self.max_message_size * TX_MEMORY_BUDGET_MULTIPLIER,
) {
match ProtocolMessage::decode_message(self.eth_version, &mut buf) {
Ok(protocol_msg) => {
if matches!(protocol_msg.message, EthMessage::Status(_)) {
return Err(EthSnapStreamError::StatusNotInHandshake);

View File

@@ -7,10 +7,7 @@
use crate::{
errors::{EthHandshakeError, EthStreamError},
handshake::EthereumEthHandshake,
message::{
EthBroadcastMessage, ProtocolBroadcastMessage, MAX_MESSAGE_SIZE,
TX_MEMORY_BUDGET_MULTIPLIER,
},
message::{EthBroadcastMessage, EthMessageID, ProtocolBroadcastMessage, MAX_MESSAGE_SIZE},
p2pstream::HANDSHAKE_TIMEOUT,
CanDisconnect, DisconnectReason, EthMessage, EthNetworkPrimitives, EthVersion, ProtocolMessage,
UnifiedStatus,
@@ -19,7 +16,7 @@ use alloy_primitives::bytes::{Bytes, BytesMut};
use alloy_rlp::Encodable;
use futures::{ready, Sink, SinkExt};
use pin_project::pin_project;
use reth_eth_wire_types::{EthMessageID, NetworkPrimitives, RawCapabilityMessage};
use reth_eth_wire_types::{NetworkPrimitives, RawCapabilityMessage};
use reth_ethereum_forks::ForkFilter;
use std::{
future::Future,
@@ -161,11 +158,7 @@ where
return Err(EthStreamError::UnsupportedMessage { message_id: id });
}
let msg = match ProtocolMessage::decode_message_with_tx_memory_budget(
self.version,
&mut bytes.as_ref(),
self.max_message_size * TX_MEMORY_BUDGET_MULTIPLIER,
) {
let msg = match ProtocolMessage::decode_message(self.version, &mut bytes.as_ref()) {
Ok(m) => m,
Err(err) => {
let msg = if bytes.len() > 50 {

View File

@@ -50,7 +50,6 @@ reth-ethereum-primitives.workspace = true
futures.workspace = true
pin-project.workspace = true
tokio = { workspace = true, features = ["io-util", "net", "macros", "rt-multi-thread", "time"] }
socket2 = { workspace = true, features = ["all"] }
tokio-stream.workspace = true
tokio-util = { workspace = true, features = ["codec"] }

View File

@@ -2,7 +2,6 @@
use crate::{
eth_requests::EthRequestHandler,
metrics::NETWORK_POOL_TRANSACTIONS_SCOPE,
transactions::{
config::{
AnnouncementFilteringPolicy, StrictEthAnnouncementFilter, TransactionPropagationKind,
@@ -13,7 +12,6 @@ use crate::{
NetworkHandle, NetworkManager,
};
use reth_eth_wire::{EthNetworkPrimitives, NetworkPrimitives};
use reth_metrics::common::mpsc::memory_bounded_channel;
use reth_network_api::test_utils::PeersHandleProvider;
use reth_storage_api::BalProvider;
use reth_transaction_pool::TransactionPool;
@@ -124,10 +122,7 @@ impl<Tx, Eth, N: NetworkPrimitives> NetworkBuilder<Tx, Eth, N> {
announcement_policy: A,
) -> NetworkBuilder<TransactionsManager<Pool, N>, Eth, N> {
let Self { mut network, request_handler, .. } = self;
let (tx, rx) = memory_bounded_channel(
transactions_manager_config.tx_channel_memory_limit_bytes,
NETWORK_POOL_TRANSACTIONS_SCOPE,
);
let (tx, rx) = mpsc::unbounded_channel();
network.set_transactions(tx);
let handle = network.handle().clone();
let policies = NetworkPolicies::new(propagation_policy, announcement_policy);

View File

@@ -23,7 +23,7 @@ use std::{
sync::Arc,
task::{ready, Context, Poll},
};
use tokio::{net::UdpSocket, sync::mpsc, task::JoinHandle};
use tokio::{sync::mpsc, task::JoinHandle};
use tokio_stream::{wrappers::ReceiverStream, Stream};
use tracing::{debug, trace};
@@ -54,9 +54,6 @@ pub struct Discovery {
discv5: Option<Discv5>,
/// All KAD table updates from the discv5 service.
discv5_updates: Option<ReceiverStream<discv5::Event>>,
/// Background task that, in shared-port mode, drains `UnrecognizedFrame`s from discv5 and
/// feeds them into the discv4 ingress so packets advance without polling `Discovery`.
_discv5_forwarder: Option<JoinHandle<()>>,
/// Handler to interact with the DNS discovery service
_dns_discovery: Option<DnsDiscoveryHandle>,
/// Updates from the DNS discovery service.
@@ -79,138 +76,39 @@ impl Discovery {
discovery_v4_addr: SocketAddr,
sk: SecretKey,
discv4_config: Option<Discv4Config>,
mut discv5_config: Option<reth_discv5::Config>, // contains discv5 listen address
discv5_config: Option<reth_discv5::Config>, // contains discv5 listen address
dns_discovery_config: Option<DnsDiscoveryConfig>,
) -> Result<Self, NetworkError> {
// setup discv4 with the discovery address and tcp port
let local_enr =
NodeRecord::from_secret_key(discovery_v4_addr, &sk).with_tcp_port(tcp_addr.port());
// For IPv6 we set IPV6_V6ONLY=true so an IPv4 sibling socket on the same port doesn't
// clash with the IPv6 one (Linux's default of V6ONLY=0 has IPv6 also claim the IPv4
// port via mapped addresses), matching how discv5 binds its `DualStack` sockets.
let bind_socket = async |addr: SocketAddr| {
let result = match addr {
SocketAddr::V4(_) => UdpSocket::bind(addr).await,
SocketAddr::V6(_) => {
use socket2::{Domain, Protocol, Socket, Type};
(|| {
let socket = Socket::new(Domain::IPV6, Type::DGRAM, Some(Protocol::UDP))?;
socket.set_only_v6(true)?;
socket.set_nonblocking(true)?;
socket.bind(&addr.into())?;
UdpSocket::from_std(socket.into())
})()
}
};
result
.map(Arc::new)
.map_err(|err| NetworkError::from_io_error(err, ServiceKind::Discovery(addr)))
};
// In shared-port mode, bind the shared socket and start discv4 without its own receive
// loop. Unrecognized frames from discv5 will be forwarded to the ingress handler.
let (discv4, discv4_updates, _discv4_service, discv4_ingress, shared_socket) =
if let Some(config) = discv4_config {
if let Some(discv5_config) = &mut discv5_config &&
discv5_config.has_matching_socket(discovery_v4_addr)
{
let socket = bind_socket(discovery_v4_addr).await?;
let (discv4, mut discv4_service, ingress) = Discv4::bind_shared(
socket.clone(),
local_enr,
sk,
config,
)
.map_err(|err| {
let discv4_future = async {
let Some(disc_config) = discv4_config else { return Ok((None, None, None)) };
let (discv4, mut discv4_service) =
Discv4::bind(discovery_v4_addr, local_enr, sk, disc_config).await.map_err(
|err| {
NetworkError::from_io_error(err, ServiceKind::Discovery(discovery_v4_addr))
})?;
},
)?;
let discv4_updates = discv4_service.update_stream();
// spawn the service
let discv4_service = discv4_service.spawn();
let discv4_updates = discv4_service.update_stream();
let discv4_service = discv4_service.spawn();
debug!(target:"net", ?discovery_v4_addr, "started discovery v4 (shared port)");
(
Some(discv4),
Some(discv4_updates),
Some(discv4_service),
Some(ingress),
Some(socket),
)
} else {
let (discv4, mut discv4_service) =
Discv4::bind(discovery_v4_addr, local_enr, sk, config).await.map_err(
|err| {
NetworkError::from_io_error(
err,
ServiceKind::Discovery(discovery_v4_addr),
)
},
)?;
let discv4_updates = discv4_service.update_stream();
// spawn the service
let discv4_service = discv4_service.spawn();
debug!(target:"net", ?discovery_v4_addr, "started discovery v4");
debug!(target:"net", ?discovery_v4_addr, "started discovery v4");
(Some(discv4), Some(discv4_updates), Some(discv4_service), None, None)
}
} else {
(None, None, None, None, None)
};
// Start discv5, wiring in the shared socket if in shared-port mode.
let (discv5, discv5_updates) = if let Some(mut config) = discv5_config {
if let Some(socket) = shared_socket {
let discv5_cfg = config.discv5_config_mut();
// The shared socket covers discv4's address family; bind the opposite family
// only if discv5 was configured for dual-stack.
let (mut ipv4, mut ipv6) = (None, None);
if discovery_v4_addr.is_ipv4() {
ipv4 = Some(socket);
if let Some(addr) = reth_discv5::config::ipv6(&discv5_cfg.listen_config) {
ipv6 = Some(bind_socket(SocketAddr::V6(addr)).await?);
}
} else {
ipv6 = Some(socket);
if let Some(addr) = reth_discv5::config::ipv4(&discv5_cfg.listen_config) {
ipv4 = Some(bind_socket(SocketAddr::V4(addr)).await?);
}
}
discv5_cfg.listen_config = discv5::ListenConfig::FromSockets { ipv4, ipv6 };
}
Ok((Some(discv4), Some(discv4_updates), Some(discv4_service)))
};
let discv5_future = async {
let Some(config) = discv5_config else { return Ok::<_, NetworkError>((None, None)) };
let (discv5, discv5_updates) = Discv5::start(&sk, config).await?;
debug!(target:"net", discovery_v5_enr=?discv5.local_enr(), "started discovery v5");
(Some(discv5), Some(discv5_updates))
} else {
(None, None)
debug!(target:"net", discovery_v5_enr=? discv5.local_enr(), "started discovery v5");
Ok((Some(discv5), Some(discv5_updates.into())))
};
// In shared-port mode, spawn a task that peels `UnrecognizedFrame` events off the discv5
// update stream and feeds them into discv4's ingress. Other events are forwarded through
// a new channel that `Discovery::poll` reads. This keeps both protocols moving without
// requiring the main `Discovery::poll` loop to be driven for packets to be routed.
let (discv5_updates, _discv5_forwarder) = match (discv4_ingress, discv5_updates) {
(Some(mut ingress), Some(mut updates)) => {
let (tx, rx) = mpsc::channel(updates.max_capacity());
let handle = tokio::spawn(async move {
while let Some(event) = updates.recv().await {
if let discv5::Event::UnrecognizedFrame(frame) = &event {
ingress.handle_packet(&frame.packet, frame.src_address).await;
continue;
}
if tx.send(event).await.is_err() {
break;
}
}
});
(Some(ReceiverStream::new(rx)), Some(handle))
}
(_, updates) => (updates.map(ReceiverStream::new), None),
};
let ((discv4, discv4_updates, _discv4_service), (discv5, discv5_updates)) =
tokio::try_join!(discv4_future, discv5_future)?;
// setup DNS discovery
let (_dns_discovery, dns_discovery_updates, _dns_disc_service) =
@@ -234,7 +132,6 @@ impl Discovery {
_discv4_service,
discv5,
discv5_updates,
_discv5_forwarder,
discovered_nodes: LruMap::new(DEFAULT_MAX_CAPACITY_DISCOVERED_PEERS_CACHE),
queued_events: Default::default(),
_dns_disc_service,
@@ -412,9 +309,6 @@ impl Drop for Discovery {
if let Some(handle) = self._discv4_service.take() {
handle.abort();
}
if let Some(handle) = self._discv5_forwarder.take() {
handle.abort();
}
if let Some(handle) = self._dns_disc_service.take() {
handle.abort();
}
@@ -448,11 +342,10 @@ impl Discovery {
},
discv4: Default::default(),
discv4_updates: Default::default(),
_discv4_service: Default::default(),
_discv5_forwarder: None,
discv5: None,
discv5_updates: None,
queued_events: Default::default(),
_discv4_service: Default::default(),
_dns_discovery: None,
dns_discovery_updates: None,
_dns_disc_service: None,
@@ -594,179 +487,4 @@ mod tests {
assert_eq!(1, node_1.discovered_nodes.len());
assert_eq!(1, node_2.discovered_nodes.len());
}
/// Starts a discovery node with discv4 and discv5 sharing the same UDP port.
async fn start_shared_port_node(port: u16) -> Discovery {
let secret_key = SecretKey::new(&mut rand_08::thread_rng());
let disc_addr: SocketAddr = format!("127.0.0.1:{port}").parse().unwrap();
// Use a non-zero TCP port so the node record isn't filtered out by
// `on_node_record_update` (which drops peers with tcp port == 0).
let tcp_addr: SocketAddr = "127.0.0.1:30303".parse().unwrap();
let discv4_config = Discv4ConfigBuilder::default().external_ip_resolver(None).build();
let discv5_listen_config = discv5::ListenConfig::from(disc_addr);
let discv5_config = reth_discv5::Config::builder(tcp_addr)
.discv5_config(discv5::ConfigBuilder::new(discv5_listen_config).build())
.build();
// Both protocols use the same address, triggering shared-port mode
Discovery::new(
tcp_addr,
disc_addr,
secret_key,
Some(discv4_config),
Some(discv5_config),
None,
)
.await
.expect("should start with shared port")
}
#[tokio::test(flavor = "multi_thread")]
async fn test_shared_port_setup() {
reth_tracing::init_test_tracing();
// Use port 0 so the OS picks a free port
let node = start_shared_port_node(0).await;
// Both protocols should be active
assert!(node.discv4.is_some(), "discv4 should be running");
assert!(node.discv5.is_some(), "discv5 should be running");
}
#[tokio::test(flavor = "multi_thread")]
async fn test_shared_port_discv5_discovery() {
reth_tracing::init_test_tracing();
let mut node_1 = start_shared_port_node(0).await;
let mut node_2 = start_shared_port_node(0).await;
let discv5_enr_1 = node_1.discv5.as_ref().unwrap().with_discv5(|discv5| discv5.local_enr());
let discv5_enr_2 = node_2.discv5.as_ref().unwrap().with_discv5(|discv5| discv5.local_enr());
let peer_id_1 = enr_to_discv4_id(&discv5_enr_1).unwrap();
let peer_id_2 = enr_to_discv4_id(&discv5_enr_2).unwrap();
// Add node_2's ENR to node_1's discv5 kbuckets and trigger a ping to establish a session.
// send_ping awaits the PONG, so the handshake completes before we poll the Discovery
// stream. The discv5 service runs its own background task.
node_1.add_discv5_node(EnrCombinedKeyWrapper(discv5_enr_2.clone()).into()).unwrap();
node_1
.discv5
.as_ref()
.unwrap()
.with_discv5(|discv5| discv5.send_ping(discv5_enr_2))
.await
.unwrap();
// Both SessionEstablished events should now be buffered in the update channels.
// Drive both nodes concurrently to collect them.
let mut event_1 = None;
let mut event_2 = None;
let timeout = tokio::time::sleep(std::time::Duration::from_secs(5));
tokio::pin!(timeout);
loop {
tokio::select! {
ev = node_1.next(), if event_1.is_none() => {
event_1 = ev;
}
ev = node_2.next(), if event_2.is_none() => {
event_2 = ev;
}
_ = &mut timeout => {
panic!("timed out waiting for discv5 discovery events");
}
}
if event_1.is_some() && event_2.is_some() {
break;
}
}
assert!(matches!(
event_1.unwrap(),
DiscoveryEvent::NewNode(DiscoveredEvent::EventQueued { peer_id, .. })
if peer_id == peer_id_2
));
assert!(matches!(
event_2.unwrap(),
DiscoveryEvent::NewNode(DiscoveredEvent::EventQueued { peer_id, .. })
if peer_id == peer_id_1
));
}
#[tokio::test(flavor = "multi_thread")]
async fn test_shared_port_discv4_discovery() {
reth_tracing::init_test_tracing();
let mut node_1 = start_shared_port_node(0).await;
let mut node_2 = start_shared_port_node(0).await;
let enr_1 = node_1.discv4.as_ref().unwrap().node_record();
let enr_2 = node_2.discv4.as_ref().unwrap().node_record();
// Introduce node_2 to node_1 via discv4
node_1.add_discv4_node(enr_2);
// Both nodes should discover each other via discv4 ping/pong
let event_1 = node_1.next().await.unwrap();
let event_2 = node_2.next().await.unwrap();
assert_eq!(
DiscoveryEvent::NewNode(DiscoveredEvent::EventQueued {
peer_id: enr_2.id,
addr: PeerAddr::new(enr_2.tcp_addr(), Some(enr_2.udp_addr())),
fork_id: None
}),
event_1
);
assert_eq!(
DiscoveryEvent::NewNode(DiscoveredEvent::EventQueued {
peer_id: enr_1.id,
addr: PeerAddr::new(enr_1.tcp_addr(), Some(enr_1.udp_addr())),
fork_id: None
}),
event_2
);
}
/// Verifies that shared-port mode binds correctly when discv5 is configured for dual-stack.
/// On Linux this exercises the IPv6 V6ONLY path: without it, the IPv4 sibling would clash
/// with the IPv6 socket bound to the same port.
#[tokio::test(flavor = "multi_thread")]
async fn test_shared_port_dual_stack() {
reth_tracing::init_test_tracing();
// Find a port that's free on the v4 wildcard so we can use it for both v4 and v6.
let probe = UdpSocket::bind("0.0.0.0:0").await.expect("probe bind");
let port = probe.local_addr().unwrap().port();
drop(probe);
let secret_key = SecretKey::new(&mut rand_08::thread_rng());
let v4_addr: SocketAddr = format!("0.0.0.0:{port}").parse().unwrap();
let tcp_addr: SocketAddr = "0.0.0.0:30303".parse().unwrap();
let discv4_config = Discv4ConfigBuilder::default().external_ip_resolver(None).build();
let discv5_listen_config = discv5::ListenConfig::DualStack {
ipv4: std::net::Ipv4Addr::UNSPECIFIED,
ipv4_port: port,
ipv6: std::net::Ipv6Addr::UNSPECIFIED,
ipv6_port: port,
};
let discv5_config = reth_discv5::Config::builder(tcp_addr)
.discv5_config(discv5::ConfigBuilder::new(discv5_listen_config).build())
.build();
Discovery::new(
tcp_addr,
v4_addr,
secret_key,
Some(discv4_config),
Some(discv5_config),
None,
)
.await
.expect("discovery should start with shared port + dual-stack");
}
}

View File

@@ -46,11 +46,6 @@ pub const MAX_HEADERS_SERVE: usize = 1024;
/// `SOFT_RESPONSE_LIMIT`.
pub const MAX_BODIES_SERVE: usize = 1024;
/// Maximum number of block access lists to serve.
///
/// Used to limit lookups.
pub const MAX_BLOCK_ACCESS_LISTS_SERVE: usize = 1024;
/// Maximum size of replies to data retrievals: 2MB
pub const SOFT_RESPONSE_LIMIT: usize = 2 * 1024 * 1024;
@@ -328,11 +323,9 @@ where
fn on_block_access_lists_request(
&self,
_peer_id: PeerId,
mut request: GetBlockAccessLists,
request: GetBlockAccessLists,
response: oneshot::Sender<RequestResult<BlockAccessLists>>,
) {
request.0.truncate(MAX_BLOCK_ACCESS_LISTS_SERVE);
let limit = GetBlockAccessListLimit::ResponseSizeSoftLimit(SOFT_RESPONSE_LIMIT);
let access_lists = self
.client

View File

@@ -26,7 +26,7 @@ use crate::{
message::{NewBlockMessage, PeerMessage},
metrics::{
BackedOffPeersMetrics, ClosedSessionsMetrics, DirectionalDisconnectMetrics, NetworkMetrics,
PendingSessionFailureMetrics,
PendingSessionFailureMetrics, NETWORK_POOL_TRANSACTIONS_SCOPE,
},
network::{NetworkHandle, NetworkHandleMessage},
peers::{BackoffReason, PeersManager},
@@ -44,7 +44,7 @@ use parking_lot::Mutex;
use reth_chainspec::EnrForkIdEntry;
use reth_eth_wire::{DisconnectReason, EthNetworkPrimitives, NetworkPrimitives};
use reth_fs_util::{self as fs, FsPathError};
use reth_metrics::common::mpsc::MemoryBoundedSender;
use reth_metrics::common::mpsc::UnboundedMeteredSender;
use reth_network_api::{
events::{PeerEvent, SessionInfo},
test_utils::PeersHandle,
@@ -118,7 +118,7 @@ pub struct NetworkManager<N: NetworkPrimitives = EthNetworkPrimitives> {
event_sender: EventSender<NetworkEvent<PeerRequest<N>>>,
/// Sender half to send events to the
/// [`TransactionsManager`](crate::transactions::TransactionsManager) task, if configured.
to_transactions_manager: Option<MemoryBoundedSender<NetworkTransactionEvent<N>>>,
to_transactions_manager: Option<UnboundedMeteredSender<NetworkTransactionEvent<N>>>,
/// Sender half to send events to the
/// [`EthRequestHandler`](crate::eth_requests::EthRequestHandler) task, if configured.
///
@@ -175,7 +175,7 @@ impl<N: NetworkPrimitives> NetworkManager<N> {
/// [`TransactionsManager`](crate::transactions::TransactionsManager).
pub fn with_transactions(
mut self,
tx: MemoryBoundedSender<NetworkTransactionEvent<N>>,
tx: mpsc::UnboundedSender<NetworkTransactionEvent<N>>,
) -> Self {
self.set_transactions(tx);
self
@@ -183,8 +183,9 @@ impl<N: NetworkPrimitives> NetworkManager<N> {
/// Sets the dedicated channel for events intended for the
/// [`TransactionsManager`](crate::transactions::TransactionsManager).
pub fn set_transactions(&mut self, tx: MemoryBoundedSender<NetworkTransactionEvent<N>>) {
self.to_transactions_manager = Some(tx);
pub fn set_transactions(&mut self, tx: mpsc::UnboundedSender<NetworkTransactionEvent<N>>) {
self.to_transactions_manager =
Some(UnboundedMeteredSender::new(tx, NETWORK_POOL_TRANSACTIONS_SCOPE));
}
/// Sets the dedicated channel for events intended for the
@@ -495,16 +496,8 @@ impl<N: NetworkPrimitives> NetworkManager<N> {
/// Sends an event to the [`TransactionsManager`](crate::transactions::TransactionsManager) if
/// configured.
fn notify_tx_manager(&self, event: NetworkTransactionEvent<N>) {
if let Some(ref tx) = self.to_transactions_manager &&
let Err(e) = tx.try_send(event)
{
match e {
TrySendError::Full(_) => {
trace!(target: "net", "Transaction events channel at capacity, dropping event");
self.metrics.total_dropped_tx_events_at_full_capacity.increment(1);
}
TrySendError::Closed(_) => {}
}
if let Some(ref tx) = self.to_transactions_manager {
let _ = tx.send(event);
}
}
@@ -772,7 +765,7 @@ impl<N: NetworkPrimitives> NetworkManager<N> {
NetworkHandleMessage::AddRlpxSubProtocol(proto) => self.add_rlpx_sub_protocol(proto),
NetworkHandleMessage::GetTransactionsHandle(tx) => {
if let Some(ref tx_inner) = self.to_transactions_manager {
let _ = tx_inner.try_send(NetworkTransactionEvent::GetTransactionsHandle(tx));
let _ = tx_inner.send(NetworkTransactionEvent::GetTransactionsHandle(tx));
} else {
let _ = tx.send(None);
}

View File

@@ -46,9 +46,6 @@ pub struct NetworkMetrics {
/// Number of Eth Requests dropped due to channel being at full capacity
pub(crate) total_dropped_eth_requests_at_full_capacity: Counter,
/// Number of transaction events dropped due to the tx manager channel being at full capacity
pub(crate) total_dropped_tx_events_at_full_capacity: Counter,
/* ================ POLL DURATION ================ */
/* -- Total poll duration of `NetworksManager` future -- */

View File

@@ -20,7 +20,6 @@ use reth_eth_wire::{
};
use reth_ethereum_primitives::{PooledTransactionVariant, TransactionSigned};
use reth_evm_ethereum::EthEvmConfig;
use reth_metrics::common::mpsc::memory_bounded_channel;
use reth_network_api::{
events::{PeerEvent, SessionInfo},
test_utils::{PeersHandle, PeersHandleProvider},
@@ -47,12 +46,13 @@ use std::{
task::{Context, Poll},
};
use tokio::{
sync::{mpsc::channel, oneshot},
sync::{
mpsc::{channel, unbounded_channel},
oneshot,
},
task::JoinHandle,
};
use crate::transactions::constants::tx_manager::DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES;
/// A test network consisting of multiple peers.
pub struct Testnet<C, Pool> {
/// All running peers in the network.
@@ -478,10 +478,7 @@ where
/// Set a new transactions manager that's connected to the peer's network
pub fn install_transactions_manager(&mut self, pool: Pool) {
let (tx, rx) = memory_bounded_channel(
DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
"test_tx_channel",
);
let (tx, rx) = unbounded_channel();
self.network.set_transactions(tx);
let transactions_manager = TransactionsManager::new(
self.handle(),
@@ -499,10 +496,7 @@ where
P: TransactionPool,
{
let Self { mut network, request_handler, client, secret_key, .. } = self;
let (tx, rx) = memory_bounded_channel(
DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
"test_tx_channel",
);
let (tx, rx) = unbounded_channel();
network.set_transactions(tx);
let transactions_manager = TransactionsManager::new(
network.handle().clone(),
@@ -543,10 +537,7 @@ where
P: TransactionPool,
{
let Self { mut network, request_handler, client, secret_key, .. } = self;
let (tx, rx) = memory_bounded_channel(
DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
"test_tx_channel",
);
let (tx, rx) = unbounded_channel();
network.set_transactions(tx);
let announcement_policy = StrictEthAnnouncementFilter::default();

View File

@@ -6,12 +6,9 @@ use super::{
DEFAULT_SOFT_LIMIT_BYTE_SIZE_POOLED_TRANSACTIONS_RESP_ON_PACK_GET_POOLED_TRANSACTIONS_REQ,
SOFT_LIMIT_BYTE_SIZE_POOLED_TRANSACTIONS_RESPONSE,
};
use crate::transactions::constants::{
tx_fetcher::{
DEFAULT_MAX_CAPACITY_CACHE_PENDING_FETCH, DEFAULT_MAX_COUNT_CONCURRENT_REQUESTS,
DEFAULT_MAX_COUNT_CONCURRENT_REQUESTS_PER_PEER,
},
tx_manager::DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
use crate::transactions::constants::tx_fetcher::{
DEFAULT_MAX_CAPACITY_CACHE_PENDING_FETCH, DEFAULT_MAX_COUNT_CONCURRENT_REQUESTS,
DEFAULT_MAX_COUNT_CONCURRENT_REQUESTS_PER_PEER,
};
use alloy_eips::eip2718::IsTyped2718;
use alloy_primitives::B256;
@@ -33,17 +30,6 @@ pub struct TransactionsManagerConfig {
/// Which peers we accept incoming transactions or announcements from.
#[cfg_attr(feature = "serde", serde(default))]
pub ingress_policy: TransactionIngressPolicy,
/// Memory limit (in bytes) for the channel that carries
/// `NetworkTransactionEvent`s from the `NetworkManager` to the `TransactionsManager`.
///
/// When the budget is exhausted, new events are dropped.
#[cfg_attr(feature = "serde", serde(default = "default_tx_channel_memory_limit_bytes"))]
pub tx_channel_memory_limit_bytes: usize,
}
#[cfg(feature = "serde")]
const fn default_tx_channel_memory_limit_bytes() -> usize {
DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES
}
impl Default for TransactionsManagerConfig {
@@ -53,7 +39,6 @@ impl Default for TransactionsManagerConfig {
max_transactions_seen_by_peer_history: DEFAULT_MAX_COUNT_TRANSACTIONS_SEEN_BY_PEER,
propagation_mode: TransactionPropagationMode::default(),
ingress_policy: TransactionIngressPolicy::default(),
tx_channel_memory_limit_bytes: DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
}
}
}

View File

@@ -53,15 +53,6 @@ pub mod tx_manager {
///
/// Default is 100 KiB, i.e. 3 200 transaction hashes.
pub const DEFAULT_MAX_COUNT_BAD_IMPORTS: u32 = 100 * 1024 / 32;
/// Default memory limit (in bytes) for the channel between
/// [`NetworkManager`](crate::NetworkManager) and
/// [`TransactionsManager`](crate::transactions::TransactionsManager).
///
/// Caps the total in-flight bytes of `NetworkTransactionEvent`s buffered between the two
/// tasks. When the budget is exhausted, new events are dropped (see metric
/// `total_dropped_tx_events_at_full_capacity`).
pub const DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES: usize = 1024 * 1024 * 1024;
}
/// Constants used by [`TransactionFetcher`](super::TransactionFetcher).

View File

@@ -33,7 +33,9 @@ use crate::{
},
cache::LruCache,
duration_metered_exec, metered_poll_nested_stream_with_budget,
metrics::{AnnouncedTxTypesMetrics, TransactionsManagerMetrics},
metrics::{
AnnouncedTxTypesMetrics, TransactionsManagerMetrics, NETWORK_POOL_TRANSACTIONS_SCOPE,
},
transactions::config::{StrictEthAnnouncementFilter, TransactionPropagationKind},
NetworkHandle, TxTypesCounter,
};
@@ -47,7 +49,7 @@ use reth_eth_wire::{
RequestTxHashes, Transactions, ValidAnnouncementData,
};
use reth_ethereum_primitives::{TransactionSigned, TxType};
use reth_metrics::common::mpsc::MemoryBoundedReceiver;
use reth_metrics::common::mpsc::UnboundedMeteredReceiver;
use reth_network_api::{
events::{PeerEvent, SessionInfo},
NetworkEvent, NetworkEventListenerProvider, PeerKind, PeerRequest, PeerRequestSender, Peers,
@@ -58,7 +60,7 @@ use reth_network_p2p::{
};
use reth_network_peers::PeerId;
use reth_network_types::ReputationChangeKind;
use reth_primitives_traits::{InMemorySize, SignedTransaction};
use reth_primitives_traits::SignedTransaction;
use reth_tokio_util::EventStream;
use reth_transaction_pool::{
error::{PoolError, PoolResult},
@@ -331,7 +333,7 @@ pub struct TransactionsManager<Pool, N: NetworkPrimitives = EthNetworkPrimitives
/// - account has enough balance to cover the transaction's gas
pending_transactions: mpsc::Receiver<TxHash>,
/// Incoming events from the [`NetworkManager`](crate::NetworkManager).
transaction_events: MemoryBoundedReceiver<NetworkTransactionEvent<N>>,
transaction_events: UnboundedMeteredReceiver<NetworkTransactionEvent<N>>,
/// How the `TransactionsManager` is configured.
config: TransactionsManagerConfig,
/// Network Policies
@@ -349,7 +351,7 @@ impl<Pool: TransactionPool, N: NetworkPrimitives> TransactionsManager<Pool, N> {
pub fn new(
network: NetworkHandle<N>,
pool: Pool,
from_network: MemoryBoundedReceiver<NetworkTransactionEvent<N>>,
from_network: mpsc::UnboundedReceiver<NetworkTransactionEvent<N>>,
transactions_manager_config: TransactionsManagerConfig,
) -> Self {
Self::with_policy(
@@ -372,7 +374,7 @@ impl<Pool: TransactionPool, N: NetworkPrimitives> TransactionsManager<Pool, N> {
pub fn with_policy(
network: NetworkHandle<N>,
pool: Pool,
from_network: MemoryBoundedReceiver<NetworkTransactionEvent<N>>,
from_network: mpsc::UnboundedReceiver<NetworkTransactionEvent<N>>,
transactions_manager_config: TransactionsManagerConfig,
policies: NetworkPolicies<N>,
) -> Self {
@@ -407,7 +409,10 @@ impl<Pool: TransactionPool, N: NetworkPrimitives> TransactionsManager<Pool, N> {
command_tx,
command_rx: UnboundedReceiverStream::new(command_rx),
pending_transactions: pending,
transaction_events: from_network,
transaction_events: UnboundedMeteredReceiver::new(
from_network,
NETWORK_POOL_TRANSACTIONS_SCOPE,
),
config: transactions_manager_config,
policies,
metrics,
@@ -1621,7 +1626,7 @@ where
"Network transaction events stream",
DEFAULT_BUDGET_TRY_DRAIN_NETWORK_TRANSACTION_EVENTS,
this.transaction_events.poll_next_unpin(cx),
|event: NetworkTransactionEvent<N>| this.on_network_tx_event(event),
|event| this.on_network_tx_event(event),
);
// Advance inflight fetch requests (flush transaction fetcher and queue for
@@ -2169,28 +2174,6 @@ struct TxManagerPollDurations {
acc_cmds: Duration,
}
impl<N: NetworkPrimitives> InMemorySize for NetworkTransactionEvent<N> {
// `N::BroadcastedTransaction` and `N::PooledTransaction` already implement
// `InMemorySize` via `SignedTransaction: InMemorySize`, so no extra bound is needed.
fn size(&self) -> usize {
match self {
Self::IncomingTransactions { peer_id, msg } => {
core::mem::size_of_val(peer_id) +
msg.0.iter().map(InMemorySize::size).sum::<usize>()
}
Self::IncomingPooledTransactionHashes { peer_id, msg } => {
core::mem::size_of_val(peer_id) + msg.size()
}
Self::GetPooledTransactions { peer_id, request, response } => {
core::mem::size_of_val(peer_id) +
request.0.len() * core::mem::size_of::<TxHash>() +
core::mem::size_of_val(response)
}
Self::GetTransactionsHandle(_) => 0,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -3123,12 +3106,7 @@ mod tests {
let mut network_manager = NetworkManager::new(network_config).await.unwrap();
let (to_tx_manager_tx, from_network_rx) =
reth_metrics::common::mpsc::memory_bounded_channel::<
NetworkTransactionEvent<EthNetworkPrimitives>,
>(
crate::transactions::constants::tx_manager::DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
"test_tx_channel",
);
mpsc::unbounded_channel::<NetworkTransactionEvent<EthNetworkPrimitives>>();
network_manager.set_transactions(to_tx_manager_tx);
let network_handle = network_manager.handle().clone();
let network_service_handle = tokio::spawn(network_manager);

View File

@@ -7,7 +7,7 @@ use rand::Rng;
use reth_eth_wire::{BlockAccessLists, EthVersion, GetBlockAccessLists, HeadersDirection};
use reth_ethereum_primitives::Block;
use reth_network::{
eth_requests::{MAX_BLOCK_ACCESS_LISTS_SERVE, SOFT_RESPONSE_LIMIT},
eth_requests::SOFT_RESPONSE_LIMIT,
test_utils::{NetworkEventStream, PeerConfig, Testnet, TestnetHandle},
BlockDownloaderProvider, NetworkEventListenerProvider,
};
@@ -607,26 +607,6 @@ async fn test_eth71_get_block_access_lists_empty_request() {
assert_eq!(response, BlockAccessLists(Vec::new()));
}
// Ensures BAL responses are capped at MAX_BLOCK_ACCESS_LISTS_SERVE entries.
#[tokio::test(flavor = "multi_thread")]
async fn test_eth71_get_block_access_lists_caps_count() {
reth_tracing::init_test_tracing();
let (net, bal_store) = spawn_eth71_bal_testnet().await;
// Request more hashes than the count cap.
let request_count = MAX_BLOCK_ACCESS_LISTS_SERVE + 100;
let hashes: Vec<B256> = (0..request_count).map(|_| B256::random()).collect();
// Insert one BAL so the store isn't entirely empty (not strictly needed,
// but keeps the test path closer to real usage).
let bal = Bytes::from_static(&[0xc1, 0x01]);
bal_store.insert(hashes[0], 1, bal).unwrap();
let response = request_block_access_lists(&net, hashes).await;
assert_eq!(response.0.len(), MAX_BLOCK_ACCESS_LISTS_SERVE);
}
// Ensures the fetch client can request BALs through an eth/71 peer.
#[tokio::test(flavor = "multi_thread")]
async fn test_eth71_fetch_client_get_block_access_lists() {

View File

@@ -4,7 +4,7 @@ use std::{
};
use reth_chainspec::MAINNET;
use reth_discv4::{Discv4Config, NatResolver, DEFAULT_DISCOVERY_ADDR};
use reth_discv4::{Discv4Config, NatResolver, DEFAULT_DISCOVERY_ADDR, DEFAULT_DISCOVERY_PORT};
use reth_network::{
error::{NetworkError, ServiceKind},
Discovery, NetworkConfigBuilder, NetworkManager,
@@ -73,31 +73,27 @@ async fn test_discovery_addr_in_use() {
}
#[tokio::test(flavor = "multi_thread")]
async fn test_discv5_and_discv4_same_socket_ok() {
// Pick a free port for the shared UDP discovery socket and TCP RLPx listener.
let test_port: u16 = TcpListener::bind("127.0.0.1:0")
.await
.expect("Failed to bind to a port")
.local_addr()
.unwrap()
.port();
async fn test_discv5_and_discv4_same_socket_fails() {
let secret_key = SecretKey::new(&mut rand_08::thread_rng());
let config = NetworkConfigBuilder::eth(secret_key, Runtime::test())
.listener_port(test_port)
.discovery_port(test_port)
.listener_port(DEFAULT_DISCOVERY_PORT)
.discovery_v5(
reth_discv5::Config::builder((DEFAULT_DISCOVERY_ADDR, test_port).into()).discv5_config(
discv5::ConfigBuilder::new(discv5::ListenConfig::from_ip(
DEFAULT_DISCOVERY_ADDR,
test_port,
))
.build(),
),
reth_discv5::Config::builder((DEFAULT_DISCOVERY_ADDR, DEFAULT_DISCOVERY_PORT).into())
.discv5_config(
discv5::ConfigBuilder::new(discv5::ListenConfig::from_ip(
DEFAULT_DISCOVERY_ADDR,
DEFAULT_DISCOVERY_PORT,
))
.build(),
),
)
.disable_dns_discovery()
.build(NoopProvider::default());
let _network = NetworkManager::new(config).await.expect("shared port discovery should start");
let addr = config.listener_addr;
let result = NetworkManager::new(config).await;
let err = result.err().unwrap();
assert!(is_addr_in_use_kind(&err, ServiceKind::Listener(addr)), "{err:?}")
}
#[tokio::test(flavor = "multi_thread")]

View File

@@ -66,8 +66,8 @@ use reth_node_metrics::{
};
use reth_provider::{
providers::{NodeTypesForProvider, ProviderNodeTypes, RocksDBProvider, StaticFileProvider},
BlockHashReader, BlockNumReader, ProviderError, ProviderFactory, ProviderResult,
RocksDBProviderFactory, StageCheckpointReader, StaticFileProviderBuilder,
BlockHashReader, BlockNumReader, DatabaseProviderFactory, ProviderError, ProviderFactory,
ProviderResult, RocksDBProviderFactory, StageCheckpointReader, StaticFileProviderBuilder,
StaticFileProviderFactory,
};
use reth_prune::{PruneModes, PrunerBuilder};
@@ -75,7 +75,7 @@ use reth_rpc_builder::config::RethRpcServerConfig;
use reth_rpc_layer::JwtSecret;
use reth_stages::{
sets::DefaultStages, stages::EraImportSource, MetricEvent, PipelineBuilder, PipelineTarget,
StageId,
StageCheckpoint, StageId,
};
use reth_static_file::StaticFileProducer;
use reth_tasks::TaskExecutor;
@@ -518,19 +518,26 @@ where
// the unwind targets for each storage layer if inconsistencies are
// found.
let (rocksdb_unwind, static_file_unwind) = factory.check_consistency()?;
let partial_trie_unwind = partial_trie_unwind_target(
factory.database_provider_ro()?.get_stage_checkpoint(StageId::Finish)?,
);
// Take the minimum block number to ensure all storage layers are consistent.
let unwind_target = [rocksdb_unwind, static_file_unwind].into_iter().flatten().min();
let unwind_target =
[rocksdb_unwind, static_file_unwind, partial_trie_unwind].into_iter().flatten().min();
if let Some(unwind_block) = unwind_target {
let inconsistency_source = [
rocksdb_unwind.map(|_| "RocksDB"),
static_file_unwind.map(|_| "static file"),
partial_trie_unwind.map(|_| "partial state trie"),
]
.into_iter()
.flatten()
.collect::<Vec<_>>()
.join(" and ");
// Highly unlikely to happen, and given its destructive nature, it's better to panic
// instead. Unwinding to 0 would leave MDBX with a huge free list size.
let inconsistency_source = match (rocksdb_unwind, static_file_unwind) {
(Some(_), Some(_)) => "RocksDB and static file",
(Some(_), None) => "RocksDB",
(None, Some(_)) => "static file",
(None, None) => unreachable!(),
};
assert_ne!(
unwind_block, 0,
"A {} inconsistency was found that would trigger an unwind to block 0",
@@ -1269,11 +1276,19 @@ pub fn metrics_hooks<N: NodeTypesWithDB>(provider_factory: &ProviderFactory<N>)
.build()
}
fn partial_trie_unwind_target(finish_checkpoint: Option<StageCheckpoint>) -> Option<BlockNumber> {
let finish_checkpoint = finish_checkpoint?;
let partial_state_trie = finish_checkpoint.finish_stage_checkpoint()?.partial_state_trie?;
(partial_state_trie != finish_checkpoint.block_number).then_some(partial_state_trie)
}
#[cfg(test)]
mod tests {
use super::{LaunchContext, NodeConfig};
use super::{partial_trie_unwind_target, LaunchContext, NodeConfig};
use reth_config::Config;
use reth_node_core::args::PruningArgs;
use reth_stages::{FinishCheckpoint, StageCheckpoint};
const EXTENSION: &str = "toml";
@@ -1325,4 +1340,24 @@ mod tests {
assert_eq!(reth_config, loaded_config);
})
}
#[test]
fn partial_trie_unwind_target_uses_partial_finish_checkpoint() {
let finish_checkpoint = StageCheckpoint::new(42)
.with_finish_stage_checkpoint(FinishCheckpoint { partial_state_trie: Some(21) });
assert_eq!(partial_trie_unwind_target(Some(finish_checkpoint)), Some(21));
}
#[test]
fn partial_trie_unwind_target_ignores_matching_or_missing_partial_checkpoint() {
let matching_finish_checkpoint = StageCheckpoint::new(42)
.with_finish_stage_checkpoint(FinishCheckpoint { partial_state_trie: Some(42) });
let missing_partial_finish_checkpoint = StageCheckpoint::new(42)
.with_finish_stage_checkpoint(FinishCheckpoint { partial_state_trie: None });
assert_eq!(partial_trie_unwind_target(Some(matching_finish_checkpoint)), None);
assert_eq!(partial_trie_unwind_target(Some(missing_partial_finish_checkpoint)), None);
assert_eq!(partial_trie_unwind_target(None), None);
}
}

View File

@@ -4,9 +4,9 @@ use clap::{builder::Resettable, Args};
use eyre::ensure;
use reth_cli_util::{parse_duration_from_secs_or_ms, parsers::format_duration_as_secs_or_ms};
use reth_engine_primitives::{
TreeConfig, DEFAULT_INVALID_HEADER_HIT_EVICTION_THRESHOLD, DEFAULT_MULTIPROOF_TASK_CHUNK_SIZE,
DEFAULT_PERSISTENCE_BACKPRESSURE_THRESHOLD, DEFAULT_SPARSE_TRIE_MAX_HOT_ACCOUNTS,
DEFAULT_SPARSE_TRIE_MAX_HOT_SLOTS,
default_persistence_backpressure_threshold, TreeConfig, DEFAULT_DEFERRED_TRIE_BLOCKS,
DEFAULT_INVALID_HEADER_HIT_EVICTION_THRESHOLD, DEFAULT_MULTIPROOF_TASK_CHUNK_SIZE,
DEFAULT_SPARSE_TRIE_MAX_HOT_ACCOUNTS, DEFAULT_SPARSE_TRIE_MAX_HOT_SLOTS,
};
use std::{sync::OnceLock, time::Duration};
@@ -24,7 +24,8 @@ static ENGINE_DEFAULTS: OnceLock<DefaultEngineValues> = OnceLock::new();
#[derive(Debug, Clone)]
pub struct DefaultEngineValues {
persistence_threshold: u64,
persistence_backpressure_threshold: u64,
persistence_backpressure_threshold: Option<u64>,
deferred_trie_blocks: u64,
memory_block_buffer_target: u64,
invalid_header_hit_eviction_threshold: u8,
legacy_state_root_task_enabled: bool,
@@ -73,9 +74,26 @@ impl DefaultEngineValues {
self
}
/// Get the default persistence backpressure threshold.
pub const fn persistence_backpressure_threshold(&self) -> u64 {
match self.persistence_backpressure_threshold {
Some(v) => v,
None => default_persistence_backpressure_threshold(
self.persistence_threshold,
self.memory_block_buffer_target,
),
}
}
/// Set the default persistence backpressure threshold
pub const fn with_persistence_backpressure_threshold(mut self, v: u64) -> Self {
self.persistence_backpressure_threshold = v;
self.persistence_backpressure_threshold = Some(v);
self
}
/// Set the default deferred trie block target
pub const fn with_deferred_trie_blocks(mut self, v: u64) -> Self {
self.deferred_trie_blocks = v;
self
}
@@ -261,7 +279,8 @@ impl Default for DefaultEngineValues {
fn default() -> Self {
Self {
persistence_threshold: DEFAULT_PERSISTENCE_THRESHOLD,
persistence_backpressure_threshold: DEFAULT_PERSISTENCE_BACKPRESSURE_THRESHOLD,
persistence_backpressure_threshold: None,
deferred_trie_blocks: DEFAULT_DEFERRED_TRIE_BLOCKS,
memory_block_buffer_target: DEFAULT_MEMORY_BLOCK_BUFFER_TARGET,
invalid_header_hit_eviction_threshold: DEFAULT_INVALID_HEADER_HIT_EVICTION_THRESHOLD,
legacy_state_root_task_enabled: false,
@@ -289,7 +308,7 @@ impl Default for DefaultEngineValues {
share_execution_cache_with_payload_builder: false,
share_sparse_trie_with_payload_builder: false,
suppress_persistence_during_build: false,
bal_parallel_execution_disabled: true,
bal_parallel_execution_disabled: false,
bal_parallel_state_root_disabled: false,
}
}
@@ -311,9 +330,14 @@ pub struct EngineArgs {
/// Configure the maximum canonical-minus-persisted gap before engine API processing stalls.
///
/// This value must be greater than `--engine.persistence-threshold`.
#[arg(long = "engine.persistence-backpressure-threshold", default_value_t = DefaultEngineValues::get_global().persistence_backpressure_threshold)]
#[arg(long = "engine.persistence-backpressure-threshold", default_value_t = DefaultEngineValues::get_global().persistence_backpressure_threshold())]
pub persistence_backpressure_threshold: u64,
/// Configure how many of the blocks being persisted should only mask state/trie writes instead
/// of durably persisting their state/trie updates in the current cycle.
#[arg(long = "engine.deferred-trie-blocks", default_value_t = DefaultEngineValues::get_global().deferred_trie_blocks)]
pub deferred_trie_blocks: u64,
/// Configure the target number of blocks to keep in memory.
#[arg(long = "engine.memory-block-buffer-target", default_value_t = DefaultEngineValues::get_global().memory_block_buffer_target)]
pub memory_block_buffer_target: u64,
@@ -511,8 +535,8 @@ pub struct EngineArgs {
)]
pub suppress_persistence_during_build: bool,
/// Disable BAL (Block Access List, EIP-7928) based parallel execution. Defaults to disabled,
/// falling back to transaction-based prewarming even when a BAL is available.
/// Disable BAL (Block Access List, EIP-7928) based parallel execution. When set, falls back
/// to transaction-based prewarming even when a BAL is available.
#[arg(long = "engine.disable-bal-parallel-execution", default_value_t = DefaultEngineValues::get_global().bal_parallel_execution_disabled)]
pub bal_parallel_execution_disabled: bool,
@@ -521,8 +545,8 @@ pub struct EngineArgs {
#[arg(long = "engine.disable-bal-parallel-state-root", default_value_t = DefaultEngineValues::get_global().bal_parallel_state_root_disabled)]
pub bal_parallel_state_root_disabled: bool,
/// Disable BAL (Block Access List) storage prefetch IO during prewarming. When set, BAL
/// storage slots are not read into the execution cache.
/// Disable BAL (Block Access List) batched IO during prewarming. When set, falls back
/// to individual per-slot storage reads instead of batched cursor reads.
#[arg(long = "engine.disable-bal-batch-io", default_value_t = false)]
pub disable_bal_batch_io: bool,
@@ -546,6 +570,7 @@ impl Default for EngineArgs {
let DefaultEngineValues {
persistence_threshold,
persistence_backpressure_threshold,
deferred_trie_blocks,
memory_block_buffer_target,
invalid_header_hit_eviction_threshold,
legacy_state_root_task_enabled,
@@ -578,7 +603,15 @@ impl Default for EngineArgs {
} = DefaultEngineValues::get_global().clone();
Self {
persistence_threshold,
persistence_backpressure_threshold,
persistence_backpressure_threshold: persistence_backpressure_threshold.unwrap_or_else(
|| {
default_persistence_backpressure_threshold(
persistence_threshold,
memory_block_buffer_target,
)
},
),
deferred_trie_blocks,
memory_block_buffer_target,
invalid_header_hit_eviction_threshold,
legacy_state_root_task_enabled,
@@ -630,6 +663,13 @@ impl EngineArgs {
self.persistence_backpressure_threshold,
self.persistence_threshold
);
ensure!(
self.deferred_trie_blocks + self.memory_block_buffer_target < self.persistence_threshold,
"--engine.deferred-trie-blocks ({}) + --engine.memory-block-buffer-target ({}) must be less than --engine.persistence-threshold ({})",
self.deferred_trie_blocks,
self.memory_block_buffer_target,
self.persistence_threshold,
);
Ok(())
}
@@ -638,6 +678,7 @@ impl EngineArgs {
let config = TreeConfig::default()
.with_persistence_threshold(self.persistence_threshold)
.with_persistence_backpressure_threshold(self.persistence_backpressure_threshold)
.with_num_state_masking_blocks(self.deferred_trie_blocks)
.with_memory_block_buffer_target(self.memory_block_buffer_target)
.with_invalid_header_hit_eviction_threshold(self.invalid_header_hit_eviction_threshold)
.with_legacy_state_root(self.legacy_state_root_task_enabled)
@@ -695,12 +736,48 @@ mod tests {
assert_eq!(args, default_args);
}
#[test]
fn default_engine_values_derive_backpressure_threshold() {
let defaults = DefaultEngineValues::default()
.with_persistence_threshold(10)
.with_memory_block_buffer_target(3);
assert_eq!(defaults.persistence_backpressure_threshold(), 26);
}
#[test]
fn explicit_backpressure_default_override_is_preserved() {
let defaults = DefaultEngineValues::default()
.with_persistence_backpressure_threshold(99)
.with_persistence_threshold(10)
.with_memory_block_buffer_target(3);
assert_eq!(defaults.persistence_backpressure_threshold(), 99);
}
#[test]
fn engine_args_default_thresholds_match_expected_defaults() {
let args = EngineArgs::default();
assert_eq!(args.persistence_threshold, DEFAULT_PERSISTENCE_THRESHOLD);
assert_eq!(args.deferred_trie_blocks, DEFAULT_DEFERRED_TRIE_BLOCKS);
assert_eq!(args.memory_block_buffer_target, DEFAULT_MEMORY_BLOCK_BUFFER_TARGET);
assert_eq!(
args.persistence_backpressure_threshold,
default_persistence_backpressure_threshold(
args.persistence_threshold,
args.memory_block_buffer_target,
)
);
}
#[test]
#[allow(deprecated)]
fn engine_args() {
let args = EngineArgs {
persistence_threshold: 100,
persistence_backpressure_threshold: 101,
deferred_trie_blocks: 25,
memory_block_buffer_target: 50,
invalid_header_hit_eviction_threshold: 7,
legacy_state_root_task_enabled: true,
@@ -745,6 +822,8 @@ mod tests {
"100",
"--engine.persistence-backpressure-threshold",
"101",
"--engine.deferred-trie-blocks",
"25",
"--engine.memory-block-buffer-target",
"50",
"--engine.invalid-header-cache-hit-eviction-threshold",
@@ -788,6 +867,21 @@ mod tests {
assert_eq!(parsed_args, args);
}
#[test]
fn test_parse_deferred_trie_blocks() {
let args = CommandParser::<EngineArgs>::parse_from([
"reth",
"--engine.persistence-threshold",
"8",
"--engine.deferred-trie-blocks",
"7",
])
.args;
assert_eq!(args.deferred_trie_blocks, 7);
assert_eq!(args.tree_config().num_state_masking_blocks(), 7);
}
#[test]
fn validate_rejects_invalid_backpressure_threshold() {
let args = EngineArgs {
@@ -801,6 +895,21 @@ mod tests {
assert!(err.contains("engine.persistence-threshold"));
}
#[test]
fn validate_rejects_state_masking_window_at_or_above_threshold() {
let args = EngineArgs {
persistence_threshold: 4,
deferred_trie_blocks: 2,
memory_block_buffer_target: 2,
..EngineArgs::default()
};
let err = args.validate().unwrap_err().to_string();
assert!(err.contains("engine.deferred-trie-blocks"));
assert!(err.contains("engine.memory-block-buffer-target"));
assert!(err.contains("engine.persistence-threshold"));
}
#[test]
fn test_parse_slow_block_threshold() {
// Test default value (None - disabled)

View File

@@ -2,7 +2,7 @@
/// NetworkArg struct for configuring the network
mod network;
pub use network::{DefaultDiscoveryArgs, DefaultNetworkArgs, DiscoveryArgs, NetworkArgs};
pub use network::{DefaultNetworkArgs, DiscoveryArgs, NetworkArgs};
/// RpcServerArg struct for configuring the RPC
mod rpc_server;

View File

@@ -11,10 +11,7 @@ use std::{
};
use crate::version::version_metadata;
use clap::{
builder::{OsStr, Resettable},
Args,
};
use clap::Args;
use reth_chainspec::EthChainSpec;
use reth_cli_util::{get_secret_key, load_secret_key::SecretKeyError};
use reth_config::Config;
@@ -34,9 +31,7 @@ use reth_network::{
DEFAULT_MAX_COUNT_CONCURRENT_REQUESTS_PER_PEER,
},
tx_manager::{
DEFAULT_MAX_COUNT_PENDING_POOL_IMPORTS,
DEFAULT_MAX_COUNT_TRANSACTIONS_SEEN_BY_PEER,
DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
DEFAULT_MAX_COUNT_PENDING_POOL_IMPORTS, DEFAULT_MAX_COUNT_TRANSACTIONS_SEEN_BY_PEER,
},
},
TransactionFetcherConfig, TransactionPropagationMode, TransactionsManagerConfig,
@@ -81,8 +76,6 @@ pub struct DefaultNetworkArgs {
pub soft_limit_byte_size_pooled_transactions_response_on_pack_request: usize,
/// Default max capacity of cache of hashes for transactions pending fetch.
pub max_capacity_cache_txns_pending_fetch: u32,
/// Default memory limit (in bytes) for the network manager → transactions manager channel.
pub tx_channel_memory_limit_bytes: usize,
/// Default transaction propagation policy.
pub tx_propagation_policy: TransactionPropagationKind,
/// Default transaction ingress policy.
@@ -176,13 +169,6 @@ impl DefaultNetworkArgs {
self
}
/// Set the default memory limit (in bytes) for the network manager → transactions
/// manager channel.
pub const fn with_tx_channel_memory_limit_bytes(mut self, v: usize) -> Self {
self.tx_channel_memory_limit_bytes = v;
self
}
/// Set the default transaction propagation policy.
pub const fn with_tx_propagation_policy(mut self, v: TransactionPropagationKind) -> Self {
self.tx_propagation_policy = v;
@@ -224,7 +210,6 @@ impl Default for DefaultNetworkArgs {
soft_limit_byte_size_pooled_transactions_response_on_pack_request:
DEFAULT_SOFT_LIMIT_BYTE_SIZE_POOLED_TRANSACTIONS_RESP_ON_PACK_GET_POOLED_TRANSACTIONS_REQ,
max_capacity_cache_txns_pending_fetch: DEFAULT_MAX_CAPACITY_CACHE_PENDING_FETCH,
tx_channel_memory_limit_bytes: DEFAULT_TX_MANAGER_CHANNEL_MEMORY_LIMIT_BYTES,
tx_propagation_policy: TransactionPropagationKind::default(),
tx_ingress_policy: TransactionIngressPolicy::default(),
propagation_mode: TransactionPropagationMode::Sqrt,
@@ -363,15 +348,6 @@ pub struct NetworkArgs {
#[arg(long = "max-tx-pending-fetch", value_name = "COUNT", default_value_t = DefaultNetworkArgs::get_global().max_capacity_cache_txns_pending_fetch, verbatim_doc_comment)]
pub max_capacity_cache_txns_pending_fetch: u32,
/// Memory limit (in bytes) for the channel that buffers transaction events flowing
/// from the network manager to the transactions manager.
///
/// When the budget is exhausted, new events are dropped (see metric
/// `total_dropped_tx_events_at_full_capacity`). Acts as a backstop against unbounded
/// memory growth under sustained P2P transaction flooding.
#[arg(long = "tx-channel-memory-limit", value_name = "BYTES", default_value_t = DefaultNetworkArgs::get_global().tx_channel_memory_limit_bytes, verbatim_doc_comment)]
pub tx_channel_memory_limit_bytes: usize,
/// Name of network interface used to communicate with peers.
///
/// If flag is set, but no value is passed, the default interface for docker `eth0` is tried.
@@ -509,7 +485,6 @@ impl NetworkArgs {
max_transactions_seen_by_peer_history: self.max_seen_tx_history,
propagation_mode: self.propagation_mode,
ingress_policy: self.tx_ingress_policy,
tx_channel_memory_limit_bytes: self.tx_channel_memory_limit_bytes,
}
}
@@ -572,8 +547,15 @@ impl NetworkArgs {
let rlpx_socket = (addr, self.port).into();
self.discovery.apply_to_builder(builder, rlpx_socket, chain_bootnodes)
})
.listener_addr(SocketAddr::new(addr, self.port))
.discovery_addr(SocketAddr::new(self.discovery.addr, self.discovery.port))
.listener_addr(SocketAddr::new(
addr, // set discovery port based on instance number
self.port,
))
.discovery_addr(SocketAddr::new(
self.discovery.addr,
// set discovery port based on instance number
self.discovery.port,
))
.disable_tx_gossip(self.disable_tx_gossip)
.required_block_hashes(self.required_block_hashes.clone())
.eth_max_message_size_opt(self.eth_max_message_size.map(NonZeroUsize::get))
@@ -678,7 +660,6 @@ impl Default for NetworkArgs {
soft_limit_byte_size_pooled_transactions_response,
soft_limit_byte_size_pooled_transactions_response_on_pack_request,
max_capacity_cache_txns_pending_fetch,
tx_channel_memory_limit_bytes,
tx_propagation_policy,
tx_ingress_policy,
propagation_mode,
@@ -708,7 +689,6 @@ impl Default for NetworkArgs {
max_pending_pool_imports,
max_seen_tx_history,
max_capacity_cache_txns_pending_fetch,
tx_channel_memory_limit_bytes,
net_if: None,
tx_propagation_policy,
tx_ingress_policy,
@@ -723,172 +703,19 @@ impl Default for NetworkArgs {
}
}
/// Global static discovery defaults
static DISCOVERY_DEFAULTS: OnceLock<DefaultDiscoveryArgs> = OnceLock::new();
/// Default values for discovery CLI arguments that can be customized.
#[derive(Debug, Clone, Copy)]
pub struct DefaultDiscoveryArgs {
/// Default for `--disable-discovery`.
pub disable_discovery: bool,
/// Default for `--disable-dns-discovery`.
pub disable_dns_discovery: bool,
/// Default for `--disable-discv4-discovery`.
pub disable_discv4_discovery: bool,
/// Default for `--disable-discv5-discovery`.
pub disable_discv5_discovery: bool,
/// Default for `--disable-nat`.
pub disable_nat: bool,
/// Default UDP address for devp2p discovery v4.
pub addr: IpAddr,
/// Default UDP port for devp2p discovery v4.
pub port: u16,
/// Default UDP IPv4 address for devp2p discovery v5.
pub discv5_addr: Option<Ipv4Addr>,
/// Default UDP IPv6 address for devp2p discovery v5.
pub discv5_addr_ipv6: Option<Ipv6Addr>,
/// Default UDP IPv4 port for devp2p discovery v5.
pub discv5_port: Option<u16>,
/// Default UDP IPv6 port for devp2p discovery v5.
pub discv5_port_ipv6: Option<u16>,
/// Default discv5 periodic lookup interval (seconds).
pub discv5_lookup_interval: u64,
/// Default discv5 bootstrap lookup interval (seconds).
pub discv5_bootstrap_lookup_interval: u64,
/// Default discv5 bootstrap lookup countdown.
pub discv5_bootstrap_lookup_countdown: u64,
}
impl DefaultDiscoveryArgs {
/// Initialize the global discovery defaults with this configuration.
pub fn try_init(self) -> Result<(), Self> {
DISCOVERY_DEFAULTS.set(self)
}
/// Get a reference to the global discovery defaults.
pub fn get_global() -> &'static Self {
DISCOVERY_DEFAULTS.get_or_init(Self::default)
}
/// Set the default for `--disable-discovery`.
pub const fn with_disable_discovery(mut self, disable: bool) -> Self {
self.disable_discovery = disable;
self
}
/// Set the default for `--disable-dns-discovery`.
pub const fn with_disable_dns_discovery(mut self, disable: bool) -> Self {
self.disable_dns_discovery = disable;
self
}
/// Set the default for `--disable-discv4-discovery`.
pub const fn with_disable_discv4_discovery(mut self, disable: bool) -> Self {
self.disable_discv4_discovery = disable;
self
}
/// Set the default for `--disable-discv5-discovery`.
pub const fn with_disable_discv5_discovery(mut self, disable: bool) -> Self {
self.disable_discv5_discovery = disable;
self
}
/// Set the default for `--disable-nat`.
pub const fn with_disable_nat(mut self, disable: bool) -> Self {
self.disable_nat = disable;
self
}
/// Set the default discovery v4 address.
pub const fn with_addr(mut self, addr: IpAddr) -> Self {
self.addr = addr;
self
}
/// Set the default discovery v4 port.
pub const fn with_port(mut self, port: u16) -> Self {
self.port = port;
self
}
/// Set the default discovery v5 IPv4 address.
pub fn with_discv5_addr(mut self, addr: impl Into<Option<Ipv4Addr>>) -> Self {
self.discv5_addr = addr.into();
self
}
/// Set the default discovery v5 IPv6 address.
pub fn with_discv5_addr_ipv6(mut self, addr: impl Into<Option<Ipv6Addr>>) -> Self {
self.discv5_addr_ipv6 = addr.into();
self
}
/// Set the default discovery V5 port.
pub fn with_discv5_port(mut self, port: impl Into<Option<u16>>) -> Self {
self.discv5_port = port.into();
self
}
/// Set the default discovery v5 IPv6 port.
pub fn with_discv5_port_ipv6(mut self, port: impl Into<Option<u16>>) -> Self {
self.discv5_port_ipv6 = port.into();
self
}
/// Set the default discv5 periodic lookup interval (seconds).
pub const fn with_discv5_lookup_interval(mut self, interval: u64) -> Self {
self.discv5_lookup_interval = interval;
self
}
/// Set the default discv5 bootstrap lookup interval (seconds).
pub const fn with_discv5_bootstrap_lookup_interval(mut self, interval: u64) -> Self {
self.discv5_bootstrap_lookup_interval = interval;
self
}
/// Set the default discv5 bootstrap lookup countdown.
pub const fn with_discv5_bootstrap_lookup_countdown(mut self, countdown: u64) -> Self {
self.discv5_bootstrap_lookup_countdown = countdown;
self
}
}
impl Default for DefaultDiscoveryArgs {
fn default() -> Self {
Self {
disable_discovery: false,
disable_dns_discovery: false,
disable_discv4_discovery: false,
disable_discv5_discovery: false,
disable_nat: false,
addr: DEFAULT_DISCOVERY_ADDR,
port: DEFAULT_DISCOVERY_PORT,
discv5_addr: None,
discv5_addr_ipv6: None,
discv5_port: Some(DEFAULT_DISCOVERY_V5_PORT),
discv5_port_ipv6: Some(DEFAULT_DISCOVERY_V5_PORT),
discv5_lookup_interval: DEFAULT_SECONDS_LOOKUP_INTERVAL,
discv5_bootstrap_lookup_interval: DEFAULT_SECONDS_BOOTSTRAP_LOOKUP_INTERVAL,
discv5_bootstrap_lookup_countdown: DEFAULT_COUNT_BOOTSTRAP_LOOKUPS,
}
}
}
/// Arguments to setup discovery
#[derive(Debug, Clone, Args, PartialEq, Eq)]
pub struct DiscoveryArgs {
/// Disable the discovery service.
#[arg(short, long, default_value_if("dev", "true", "true"), default_value_t = DefaultDiscoveryArgs::get_global().disable_discovery)]
#[arg(short, long, default_value_if("dev", "true", "true"))]
pub disable_discovery: bool,
/// Disable the DNS discovery.
#[arg(long, conflicts_with = "disable_discovery", default_value_t = DefaultDiscoveryArgs::get_global().disable_dns_discovery)]
#[arg(long, conflicts_with = "disable_discovery")]
pub disable_dns_discovery: bool,
/// Disable Discv4 discovery.
#[arg(long, conflicts_with = "disable_discovery", default_value_t = DefaultDiscoveryArgs::get_global().disable_discv4_discovery)]
#[arg(long, conflicts_with = "disable_discovery")]
pub disable_discv4_discovery: bool,
/// Enable Discv5 discovery.
@@ -899,57 +726,57 @@ pub struct DiscoveryArgs {
pub enable_discv5_discovery: bool,
/// Disable Discv5 discovery.
#[arg(long, conflicts_with = "disable_discovery", default_value_t = DefaultDiscoveryArgs::get_global().disable_discv5_discovery)]
#[arg(long, conflicts_with = "disable_discovery")]
pub disable_discv5_discovery: bool,
/// Disable Nat discovery.
#[arg(long, conflicts_with = "disable_discovery", default_value_t = DefaultDiscoveryArgs::get_global().disable_nat)]
#[arg(long, conflicts_with = "disable_discovery")]
pub disable_nat: bool,
/// The UDP address to use for devp2p peer discovery version 4.
#[arg(id = "discovery.addr", long = "discovery.addr", value_name = "DISCOVERY_ADDR", default_value_t = DefaultDiscoveryArgs::get_global().addr)]
#[arg(id = "discovery.addr", long = "discovery.addr", value_name = "DISCOVERY_ADDR", default_value_t = DEFAULT_DISCOVERY_ADDR)]
pub addr: IpAddr,
/// The UDP port to use for devp2p peer discovery version 4.
#[arg(id = "discovery.port", long = "discovery.port", value_name = "DISCOVERY_PORT", default_value_t = DefaultDiscoveryArgs::get_global().port)]
#[arg(id = "discovery.port", long = "discovery.port", value_name = "DISCOVERY_PORT", default_value_t = DEFAULT_DISCOVERY_PORT)]
pub port: u16,
/// The UDP IPv4 address to use for devp2p peer discovery version 5. Overwritten by `RLPx`
/// address, if it's also IPv4.
#[arg(id = "discovery.v5.addr", long = "discovery.v5.addr", value_name = "DISCOVERY_V5_ADDR", default_value = Resettable::from(DefaultDiscoveryArgs::get_global().discv5_addr.map(|a| OsStr::from(a.to_string()))))]
#[arg(id = "discovery.v5.addr", long = "discovery.v5.addr", value_name = "DISCOVERY_V5_ADDR", default_value = None)]
pub discv5_addr: Option<Ipv4Addr>,
/// The UDP IPv6 address to use for devp2p peer discovery version 5. Overwritten by `RLPx`
/// address, if it's also IPv6.
#[arg(id = "discovery.v5.addr.ipv6", long = "discovery.v5.addr.ipv6", value_name = "DISCOVERY_V5_ADDR_IPV6", default_value = Resettable::from(DefaultDiscoveryArgs::get_global().discv5_addr_ipv6.map(|a| OsStr::from(a.to_string()))))]
#[arg(id = "discovery.v5.addr.ipv6", long = "discovery.v5.addr.ipv6", value_name = "DISCOVERY_V5_ADDR_IPV6", default_value = None)]
pub discv5_addr_ipv6: Option<Ipv6Addr>,
/// The UDP IPv4 port to use for devp2p peer discovery version 5. Not used unless `--addr` is
/// IPv4, or `--discovery.v5.addr` is set.
#[arg(id = "discovery.v5.port", long = "discovery.v5.port", value_name = "DISCOVERY_V5_PORT", default_value = Resettable::from(DefaultDiscoveryArgs::get_global().discv5_port.map(|p| OsStr::from(p.to_string()))))]
pub discv5_port: Option<u16>,
#[arg(id = "discovery.v5.port", long = "discovery.v5.port", value_name = "DISCOVERY_V5_PORT",
default_value_t = DEFAULT_DISCOVERY_V5_PORT)]
pub discv5_port: u16,
/// The UDP IPv6 port to use for devp2p peer discovery version 5. Not used unless `--addr` is
/// IPv6, or `--discovery.addr.ipv6` is set.
///
/// If not provided, discovery V5 defaults to same port as discovery V4 (--discovery.port).
#[arg(id = "discovery.v5.port.ipv6", long = "discovery.v5.port.ipv6", value_name = "DISCOVERY_V5_PORT_IPV6", default_value = Resettable::from(DefaultDiscoveryArgs::get_global().discv5_port_ipv6.map(|p| OsStr::from(p.to_string()))))]
pub discv5_port_ipv6: Option<u16>,
#[arg(id = "discovery.v5.port.ipv6", long = "discovery.v5.port.ipv6", value_name = "DISCOVERY_V5_PORT_IPV6",
default_value_t = DEFAULT_DISCOVERY_V5_PORT)]
pub discv5_port_ipv6: u16,
/// The interval in seconds at which to carry out periodic lookup queries, for the whole
/// run of the program.
#[arg(id = "discovery.v5.lookup-interval", long = "discovery.v5.lookup-interval", value_name = "DISCOVERY_V5_LOOKUP_INTERVAL", default_value_t = DefaultDiscoveryArgs::get_global().discv5_lookup_interval)]
#[arg(id = "discovery.v5.lookup-interval", long = "discovery.v5.lookup-interval", value_name = "DISCOVERY_V5_LOOKUP_INTERVAL", default_value_t = DEFAULT_SECONDS_LOOKUP_INTERVAL)]
pub discv5_lookup_interval: u64,
/// The interval in seconds at which to carry out boost lookup queries, for a fixed number of
/// times, at bootstrap.
#[arg(id = "discovery.v5.bootstrap.lookup-interval", long = "discovery.v5.bootstrap.lookup-interval", value_name = "DISCOVERY_V5_BOOTSTRAP_LOOKUP_INTERVAL",
default_value_t = DefaultDiscoveryArgs::get_global().discv5_bootstrap_lookup_interval)]
default_value_t = DEFAULT_SECONDS_BOOTSTRAP_LOOKUP_INTERVAL)]
pub discv5_bootstrap_lookup_interval: u64,
/// The number of times to carry out boost lookup queries at bootstrap.
#[arg(id = "discovery.v5.bootstrap.lookup-countdown", long = "discovery.v5.bootstrap.lookup-countdown", value_name = "DISCOVERY_V5_BOOTSTRAP_LOOKUP_COUNTDOWN",
default_value_t = DefaultDiscoveryArgs::get_global().discv5_bootstrap_lookup_countdown)]
default_value_t = DEFAULT_COUNT_BOOTSTRAP_LOOKUPS)]
pub discv5_bootstrap_lookup_countdown: u64,
}
@@ -999,7 +826,6 @@ impl DiscoveryArgs {
discv5_lookup_interval,
discv5_bootstrap_lookup_interval,
discv5_bootstrap_lookup_countdown,
port,
..
} = self;
@@ -1017,9 +843,8 @@ impl DiscoveryArgs {
let mut discv5_config_builder =
reth_discv5::discv5::ConfigBuilder::new(ListenConfig::from_two_sockets(
discv5_addr_ipv4.map(|addr| SocketAddrV4::new(addr, discv5_port.unwrap_or(*port))),
discv5_addr_ipv6
.map(|addr| SocketAddrV6::new(addr, discv5_port_ipv6.unwrap_or(*port), 0, 0)),
discv5_addr_ipv4.map(|addr| SocketAddrV4::new(addr, *discv5_port)),
discv5_addr_ipv6.map(|addr| SocketAddrV6::new(addr, *discv5_port_ipv6, 0, 0)),
));
if has_discv5_addr_args || self.disable_nat {
@@ -1049,14 +874,14 @@ impl DiscoveryArgs {
/// discovery binds to the sockets.
pub const fn with_unused_discovery_port(mut self) -> Self {
self.port = 0;
self.discv5_port = Some(0);
self.discv5_port_ipv6 = Some(0);
self.discv5_port = 0;
self.discv5_port_ipv6 = 0;
self
}
/// Set the discovery V5 port
pub fn with_discv5_port(mut self, port: impl Into<Option<u16>>) -> Self {
self.discv5_port = port.into();
pub const fn with_discv5_port(mut self, port: u16) -> Self {
self.discv5_port = port;
self
}
@@ -1068,45 +893,29 @@ impl DiscoveryArgs {
pub fn adjust_instance_ports(&mut self, instance: u16) {
debug_assert_ne!(instance, 0, "instance must be non-zero");
self.port += instance - 1;
self.discv5_port = self.discv5_port.map(|port| port + instance - 1);
self.discv5_port_ipv6 = self.discv5_port_ipv6.map(|port| port + instance - 1);
self.discv5_port += instance - 1;
self.discv5_port_ipv6 += instance - 1;
}
}
impl Default for DiscoveryArgs {
fn default() -> Self {
let DefaultDiscoveryArgs {
disable_discovery,
disable_dns_discovery,
disable_discv4_discovery,
disable_discv5_discovery,
disable_nat,
addr,
port,
discv5_addr,
discv5_addr_ipv6,
discv5_port,
discv5_port_ipv6,
discv5_lookup_interval,
discv5_bootstrap_lookup_interval,
discv5_bootstrap_lookup_countdown,
} = *DefaultDiscoveryArgs::get_global();
Self {
disable_discovery,
disable_dns_discovery,
disable_discv4_discovery,
disable_discovery: false,
disable_dns_discovery: false,
disable_discv4_discovery: false,
enable_discv5_discovery: false,
disable_discv5_discovery,
disable_nat,
addr,
port,
discv5_addr,
discv5_addr_ipv6,
discv5_port,
discv5_port_ipv6,
discv5_lookup_interval,
discv5_bootstrap_lookup_interval,
discv5_bootstrap_lookup_countdown,
disable_discv5_discovery: false,
disable_nat: false,
addr: DEFAULT_DISCOVERY_ADDR,
port: DEFAULT_DISCOVERY_PORT,
discv5_addr: None,
discv5_addr_ipv6: None,
discv5_port: DEFAULT_DISCOVERY_V5_PORT,
discv5_port_ipv6: DEFAULT_DISCOVERY_V5_PORT,
discv5_lookup_interval: DEFAULT_SECONDS_LOOKUP_INTERVAL,
discv5_bootstrap_lookup_interval: DEFAULT_SECONDS_BOOTSTRAP_LOOKUP_INTERVAL,
discv5_bootstrap_lookup_countdown: DEFAULT_COUNT_BOOTSTRAP_LOOKUPS,
}
}
}

View File

@@ -4,12 +4,12 @@
//! the consensus client.
use alloy_eips::{
eip4844::{BlobAndProofV1, BlobAndProofV2, BlobCellsAndProofsV1},
eip4844::{BlobAndProofV1, BlobAndProofV2},
eip7685::RequestsOrHash,
BlockId, BlockNumberOrTag,
};
use alloy_json_rpc::RpcObject;
use alloy_primitives::{Address, BlockHash, Bytes, B128, B256, U256, U64};
use alloy_primitives::{Address, BlockHash, Bytes, B256, U256, U64};
use alloy_rpc_types_engine::{
ClientVersionV1, ExecutionPayloadBodiesV1, ExecutionPayloadBodiesV2, ExecutionPayloadInputV2,
ExecutionPayloadV1, ExecutionPayloadV3, ExecutionPayloadV4, ForkchoiceState, ForkchoiceUpdated,
@@ -324,20 +324,6 @@ pub trait EngineApi<Engine: EngineTypes> {
&self,
versioned_hashes: Vec<B256>,
) -> RpcResult<Option<Vec<Option<BlobAndProofV2>>>>;
/// Fetch blob cells for the consensus layer from the blob store.
///
/// Returns a response of the same length as the request. Missing blobs are returned as `null`
/// elements; missing requested cells within an available blob are returned as `null` cell and
/// proof entries.
///
/// Returns `null` if syncing.
#[method(name = "getBlobsV4")]
async fn get_blobs_v4(
&self,
versioned_hashes: Vec<B256>,
indices_bitarray: B128,
) -> RpcResult<Option<Vec<Option<BlobCellsAndProofsV1>>>>;
}
/// A subset of the ETH rpc interface: <https://ethereum.github.io/execution-apis/api-documentation>

View File

@@ -37,7 +37,6 @@ pub const CAPABILITIES: &[&str] = &[
"engine_getBlobsV1",
"engine_getBlobsV2",
"engine_getBlobsV3",
"engine_getBlobsV4",
];
/// Engine API capabilities set.
@@ -219,7 +218,6 @@ mod tests {
assert!(!is_critical_method("engine_getBlobsV1"));
assert!(!is_critical_method("engine_getBlobsV3"));
assert!(!is_critical_method("engine_getBlobsV4"));
assert!(!is_critical_method("engine_getPayloadBodiesByHashV1"));
assert!(!is_critical_method("engine_getPayloadBodiesByRangeV1"));
assert!(!is_critical_method("engine_getClientVersionV1"));

View File

@@ -3,11 +3,11 @@ use crate::{
};
use alloy_eips::{
eip1898::BlockHashOrNumber,
eip4844::{BlobAndProofV1, BlobAndProofV2, BlobCellsAndProofsV1},
eip4844::{BlobAndProofV1, BlobAndProofV2},
eip4895::Withdrawals,
eip7685::RequestsOrHash,
};
use alloy_primitives::{BlockHash, BlockNumber, B128, B256, U64};
use alloy_primitives::{BlockHash, BlockNumber, B256, U64};
use alloy_rpc_types_engine::{
CancunPayloadFields, ClientVersionV1, ExecutionData, ExecutionPayloadBodiesV1,
ExecutionPayloadBodiesV2, ExecutionPayloadBodyV1, ExecutionPayloadBodyV2,
@@ -953,35 +953,6 @@ where
.map_err(|err| EngineApiError::Internal(Box::new(err)))
}
fn get_blobs_v4(
&self,
versioned_hashes: Vec<B256>,
indices_bitarray: B128,
) -> EngineApiResult<Option<Vec<Option<BlobCellsAndProofsV1>>>> {
let current_timestamp =
SystemTime::now().duration_since(SystemTime::UNIX_EPOCH).unwrap_or_default().as_secs();
if !self.inner.chain_spec.is_amsterdam_active_at_timestamp(current_timestamp) {
return Err(EngineApiError::EngineObjectValidationError(
reth_payload_primitives::EngineObjectValidationError::UnsupportedFork,
));
}
if versioned_hashes.len() > MAX_BLOB_LIMIT {
return Err(EngineApiError::BlobRequestTooLarge { len: versioned_hashes.len() })
}
// Spec requires returning `null` if syncing.
if (*self.inner.is_syncing)() {
return Ok(None)
}
self.inner
.tx_pool
.get_blobs_for_versioned_hashes_v4(&versioned_hashes, indices_bitarray)
.map(Some)
.map_err(|err| EngineApiError::Internal(Box::new(err)))
}
/// Metered version of `get_blobs_v2`.
pub fn get_blobs_v2_metered(
&self,
@@ -1038,28 +1009,6 @@ where
res
}
/// Metered version of `get_blobs_v4`.
pub fn get_blobs_v4_metered(
&self,
versioned_hashes: Vec<B256>,
indices_bitarray: B128,
) -> EngineApiResult<Option<Vec<Option<BlobCellsAndProofsV1>>>> {
let hashes_len = versioned_hashes.len();
let start = Instant::now();
let res = Self::get_blobs_v4(self, versioned_hashes, indices_bitarray);
self.inner.metrics.latency.get_blobs_v4.record(start.elapsed());
if let Ok(Some(blobs)) = &res {
let blobs_found = blobs.iter().flatten().count();
let blobs_missed = hashes_len - blobs_found;
self.inner.metrics.blob_metrics.blob_count.increment(blobs_found as u64);
self.inner.metrics.blob_metrics.blob_misses.increment(blobs_missed as u64);
}
res
}
}
// This is the concrete ethereum engine API implementation.
@@ -1426,15 +1375,6 @@ where
trace!(target: "rpc::engine", "Serving engine_getBlobsV3");
Ok(self.get_blobs_v3_metered(versioned_hashes)?)
}
async fn get_blobs_v4(
&self,
versioned_hashes: Vec<B256>,
indices_bitarray: B128,
) -> RpcResult<Option<Vec<Option<BlobCellsAndProofsV1>>>> {
trace!(target: "rpc::engine", "Serving engine_getBlobsV4");
Ok(self.get_blobs_v4_metered(versioned_hashes, indices_bitarray)?)
}
}
impl<Provider, EngineT, Pool, Validator, ChainSpec> IntoEngineApiRpcModule
@@ -1720,37 +1660,6 @@ mod tests {
assert_matches!(res, Ok(None));
}
#[tokio::test]
async fn get_blobs_v4_returns_null_when_syncing() {
let chain_spec: Arc<ChainSpec> =
Arc::new(ChainSpecBuilder::mainnet().amsterdam_activated().build());
let provider = Arc::new(MockEthProvider::default());
let payload_store = spawn_test_payload_service::<EthEngineTypes>();
let (to_engine, _engine_rx) = unbounded_channel::<BeaconEngineMessage<EthEngineTypes>>();
let api = EngineApi::new(
provider,
chain_spec.clone(),
ConsensusEngineHandle::new(to_engine),
payload_store.into(),
NoopTransactionPool::default(),
Runtime::test(),
ClientVersionV1 {
code: ClientCode::RH,
name: "Reth".to_string(),
version: "v0.0.0-test".to_string(),
commit: "test".to_string(),
},
EngineCapabilities::default(),
EthereumEngineValidator::new(chain_spec),
false,
TestNetworkInfo { syncing: true },
);
let res = api.get_blobs_v4_metered(vec![B256::ZERO], B128::from(1u128));
assert_matches!(res, Ok(None));
}
#[tokio::test]
async fn fcu_v3_syncing_precedes_invalid_payload_attributes_validation() {
let (mut handle, api) = setup_engine_api();

View File

@@ -58,8 +58,6 @@ pub(crate) struct EngineApiLatencyMetrics {
pub(crate) get_blobs_v2: Histogram,
/// Latency for `engine_getBlobsV3`
pub(crate) get_blobs_v3: Histogram,
/// Latency for `engine_getBlobsV4`
pub(crate) get_blobs_v4: Histogram,
}
#[derive(Metrics)]

View File

@@ -9,10 +9,9 @@ use alloy_primitives::{B256, U256};
use alloy_rpc_types_eth::BlockNumberOrTag;
use futures::Future;
use reth_chain_state::{BlockState, ComputedTrieData, ExecutedBlock};
use reth_chainspec::{ChainSpecProvider, EthChainSpec, EthereumHardforks};
use reth_chainspec::{ChainSpecProvider, EthChainSpec};
use reth_errors::{BlockExecutionError, BlockValidationError, ProviderError, RethError};
use reth_evm::{
block::TxResult,
execute::{BlockBuilder, BlockBuilderOutcome, BlockExecutionOutput},
ConfigureEvm, Evm, EvmEnvFor, NextBlockEnvAttributes,
};
@@ -31,7 +30,7 @@ use reth_transaction_pool::{
error::InvalidPoolTransactionError, BestTransactions, BestTransactionsAttributes,
PoolTransaction, TransactionPool,
};
use revm::context_interface::{Block, Cfg as _};
use revm::context_interface::Block;
use std::{
sync::Arc,
time::{Duration, Instant},
@@ -264,10 +263,6 @@ pub trait LoadPendingBlock:
builder.apply_pre_execution_changes().map_err(Self::Error::from_eth_err)?;
let block_gas_limit: u64 = builder.evm().block().gas_limit();
let is_amsterdam = self
.provider()
.chain_spec()
.is_amsterdam_active_at_timestamp(builder.evm().block().timestamp().saturating_to());
let basefee = builder.evm().block().basefee();
let blob_gasprice = builder.evm().block().blob_gasprice().map(|p| p as u64);
@@ -276,11 +271,8 @@ pub trait LoadPendingBlock:
.chain_spec()
.blob_params_at_timestamp(parent.timestamp())
.unwrap_or_else(BlobParams::cancun);
let mut cumulative_tx_gas_used = 0;
let mut block_regular_gas_used = 0;
let mut block_state_gas_used = 0;
let mut cumulative_gas_used = 0;
let mut sum_blob_gas_used = 0;
let tx_gas_limit_cap = builder.evm().cfg_env().tx_gas_limit_cap();
// Only include transactions if not configured as Empty
if !self.pending_block_kind().is_empty() {
@@ -295,35 +287,15 @@ pub trait LoadPendingBlock:
while let Some(pool_tx) = best_txs.next() {
// ensure we still have capacity for this transaction
let exceeds_gas_limit = if is_amsterdam {
let regular_available_gas =
block_gas_limit.saturating_sub(block_regular_gas_used);
let state_available_gas = block_gas_limit.saturating_sub(block_state_gas_used);
let regular_tx_gas_limit = pool_tx.gas_limit().min(tx_gas_limit_cap);
if regular_tx_gas_limit > regular_available_gas {
Some((regular_tx_gas_limit, regular_available_gas))
} else if pool_tx.gas_limit() > state_available_gas {
Some((pool_tx.gas_limit(), state_available_gas))
} else {
None
}
} else {
let block_available_gas =
block_gas_limit.saturating_sub(cumulative_tx_gas_used);
(pool_tx.gas_limit() > block_available_gas)
.then_some((pool_tx.gas_limit(), block_available_gas))
};
if let Some((transaction_gas_limit, block_available_gas)) = exceeds_gas_limit {
if cumulative_gas_used + pool_tx.gas_limit() > block_gas_limit {
// we can't fit this transaction into the block, so we need to mark it as
// invalid which also removes all dependent transaction from
// the iterator before we can continue
best_txs.mark_invalid(
&pool_tx,
&InvalidPoolTransactionError::ExceedsGasLimit(
transaction_gas_limit,
block_available_gas,
pool_tx.gas_limit(),
block_gas_limit,
),
);
continue
@@ -365,48 +337,29 @@ pub trait LoadPendingBlock:
continue
}
let mut tx_regular_gas_used = 0;
let gas_output =
match builder.execute_transaction_with_result_closure(tx, |result| {
tx_regular_gas_used = result.result().result.gas().block_regular_gas_used();
}) {
Ok(gas_output) => gas_output,
Err(BlockExecutionError::Validation(BlockValidationError::InvalidTx {
error,
..
})) => {
if error.is_nonce_too_low() {
// if the nonce is too low, we can skip this transaction
} else {
// if the transaction is invalid, we can skip it and all of its
// descendants
best_txs.mark_invalid(
&pool_tx,
&InvalidPoolTransactionError::Consensus(
InvalidTransactionError::TxTypeNotSupported,
),
);
}
continue
}
Err(BlockExecutionError::Validation(
BlockValidationError::TransactionGasLimitMoreThanAvailableBlockGas {
transaction_gas_limit,
block_available_gas,
},
)) => {
let gas_used = match builder.execute_transaction(tx) {
Ok(gas_used) => gas_used.tx_gas_used(),
Err(BlockExecutionError::Validation(BlockValidationError::InvalidTx {
error,
..
})) => {
if error.is_nonce_too_low() {
// if the nonce is too low, we can skip this transaction
} else {
// if the transaction is invalid, we can skip it and all of its
// descendants
best_txs.mark_invalid(
&pool_tx,
&InvalidPoolTransactionError::ExceedsGasLimit(
transaction_gas_limit,
block_available_gas,
&InvalidPoolTransactionError::Consensus(
InvalidTransactionError::TxTypeNotSupported,
),
);
continue
}
// this is an error that we should treat as fatal for this attempt
Err(err) => return Err(Self::Error::from_eth_err(err)),
};
continue
}
// this is an error that we should treat as fatal for this attempt
Err(err) => return Err(Self::Error::from_eth_err(err)),
};
// add to the total blob gas used if the transaction successfully executed
if let Some(tx_blob_gas) = tx_blob_gas {
@@ -418,11 +371,9 @@ pub trait LoadPendingBlock:
}
}
// Track receipt gas and the Amsterdam block-capacity counter separately.
let gas_used = gas_output.tx_gas_used();
cumulative_tx_gas_used += gas_used;
block_regular_gas_used += tx_regular_gas_used;
block_state_gas_used += gas_output.state_gas_used();
// add gas used by the transaction to cumulative gas used, before creating the
// receipt
cumulative_gas_used += gas_used;
}
}

View File

@@ -737,21 +737,18 @@ where
is_multi_block_range &&
all_logs.len() > max_logs_per_response
{
let retry_to_block =
if num_hash.number == from_block { from_block } else { num_hash.number - 1 };
debug!(
target: "rpc::eth::filter",
logs_found = all_logs.len(),
max_logs_per_response,
from_block,
to_block = retry_to_block,
to_block = num_hash.number,
"Query exceeded max logs per response limit"
);
return Err(EthFilterError::QueryExceedsMaxResults {
max_logs: max_logs_per_response,
from_block,
to_block: retry_to_block,
to_block: num_hash.number,
});
}
}
@@ -1819,87 +1816,6 @@ mod tests {
assert!(result.is_none());
}
#[tokio::test]
async fn test_log_limit_retry_range_excludes_overflow_block() {
let provider = MockEthProvider::default();
use alloy_consensus::TxLegacy;
use reth_db_api::models::StoredBlockBodyIndices;
use reth_ethereum_primitives::{TransactionSigned, TxType};
let tx_inner = TxLegacy {
chain_id: Some(1),
nonce: 0,
gas_price: 21_000,
gas_limit: 21_000,
to: alloy_primitives::TxKind::Call(alloy_primitives::Address::ZERO),
value: alloy_primitives::U256::ZERO,
input: alloy_primitives::Bytes::new(),
};
let signature = alloy_primitives::Signature::test_signature();
let tx = TransactionSigned::new_unhashed(tx_inner.into(), signature);
let mock_log = alloy_primitives::Log {
address: alloy_primitives::Address::ZERO,
data: alloy_primitives::LogData::new_unchecked(vec![], alloy_primitives::Bytes::new()),
};
let receipt = reth_ethereum_primitives::Receipt {
tx_type: TxType::Legacy,
cumulative_gas_used: 21_000,
logs: vec![mock_log],
success: true,
};
let mut prev_hash = alloy_primitives::B256::default();
for (idx, block_number) in (100u64..=102).enumerate() {
let header = alloy_consensus::Header {
number: block_number,
parent_hash: prev_hash,
logs_bloom: alloy_primitives::Bloom::from([1u8; 256]),
..Default::default()
};
let hash = header.hash_slow();
prev_hash = hash;
let block = reth_ethereum_primitives::Block {
header,
body: reth_ethereum_primitives::BlockBody {
transactions: vec![tx.clone()],
..Default::default()
},
};
provider.add_block(hash, block);
provider.add_receipts(block_number, vec![receipt.clone()]);
provider.add_block_body_indices(
block_number,
StoredBlockBodyIndices { first_tx_num: idx as u64, tx_count: 1 },
);
}
let eth_api = build_test_eth_api(provider);
let eth_filter = EthFilter::new(eth_api, EthFilterConfig::default(), Runtime::test());
let err = eth_filter
.inner
.clone()
.get_logs_in_block_range(
Filter::default(),
100,
102,
QueryLimits { max_blocks_per_filter: None, max_logs_per_response: Some(2) },
)
.await
.expect_err("range should exceed max logs");
let EthFilterError::QueryExceedsMaxResults { max_logs, from_block, to_block } = err else {
panic!("unexpected error: {err:?}");
};
assert_eq!(max_logs, 2);
assert_eq!(from_block, 100);
assert_eq!(to_block, 101);
}
#[tokio::test]
async fn test_non_consecutive_headers_after_bloom_filter() {
let provider = MockEthProvider::default();

View File

@@ -9,7 +9,7 @@ use reth_primitives_traits::constants::BEACON_CONSENSUS_REORG_UNWIND_DEPTH;
use reth_provider::{
providers::ProviderNodeTypes, BlockHashReader, BlockNumReader, ChainStateBlockReader,
ChainStateBlockWriter, DBProvider, DatabaseProviderFactory, ProviderFactory,
PruneCheckpointReader, StageCheckpointReader, StageCheckpointWriter, StorageSettingsCache,
PruneCheckpointReader, StageCheckpointReader, StageCheckpointWriter,
};
use reth_prune::PrunerBuilder;
use reth_static_file::StaticFileProducer;
@@ -269,16 +269,9 @@ impl<N: ProviderNodeTypes> Pipeline<N> {
/// - [`StaticFileSegment::Transactions`](reth_static_file_types::StaticFileSegment::Transactions)
/// -> [`StageId::Bodies`]
///
/// This is a legacy storage.v1 backfill step. Storage.v2 writes directly to static files and
/// `RocksDB`, so there is no MDBX -> static-file migration to perform.
///
/// CAUTION: This method locks the static file producer Mutex, hence can block the thread if the
/// lock is occupied.
pub fn move_to_static_files(&self) -> RethResult<()> {
if self.provider_factory.cached_storage_settings().is_v2() {
return Ok(())
}
// Copies data from database to static files
let lowest_static_file_height =
self.static_file_producer.lock().copy_to_static_files()?.min_block_num();

View File

@@ -295,7 +295,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed, // 1 seeded block body + batch size
total // seeded headers
}))
})),
..
}, done: false }) if block_number < 200 &&
processed == batch_size + 1 && total == previous_stage + 1
);
@@ -333,7 +334,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
},
done: true
}) if processed + 1 == total && total == previous_stage + 1
@@ -370,7 +372,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
}, done: false }) if block_number >= 10 &&
processed - 1 == batch_size && total == previous_stage + 1
);
@@ -391,7 +394,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
}, done: true }) if block_number > first_run_checkpoint.block_number &&
processed + 1 == total && total == previous_stage + 1
);
@@ -432,7 +436,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
}, done: true }) if block_number == previous_stage &&
processed + 1 == total && total == previous_stage + 1
);
@@ -460,7 +465,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed: 1,
total
}))
})),
..
}}) if total == previous_stage + 1
);

View File

@@ -298,7 +298,7 @@ mod tests {
assert_matches!(
output,
Ok(ExecOutput {
checkpoint: StageCheckpoint { block_number, stage_checkpoint: None },
checkpoint: StageCheckpoint { block_number, stage_checkpoint: None, .. },
done: false
}) if block_number == era_cap
);
@@ -318,7 +318,7 @@ mod tests {
assert_matches!(
output,
Ok(ExecOutput {
checkpoint: StageCheckpoint { block_number, stage_checkpoint: None },
checkpoint: StageCheckpoint { block_number, stage_checkpoint: None, .. },
done: true
}) if block_number == target
);

View File

@@ -1015,7 +1015,8 @@ mod tests {
processed,
total
}
}))
})),
..
},
done: true
} if processed == total && total == block.gas_used);
@@ -1170,7 +1171,8 @@ mod tests {
processed: 0,
total
}
}))
})),
..
}
} if total == block.gas_used);

View File

@@ -397,6 +397,7 @@ mod tests {
},
..
})),
..
},
done: true,
}) if block_number == previous_stage &&

View File

@@ -594,7 +594,8 @@ mod tests {
processed,
total,
}
}))
})),
..
}, done: true }) if block_number == tip.number &&
from == checkpoint && to == previous_stage &&
// -1 because we don't need to download the local head
@@ -666,7 +667,8 @@ mod tests {
processed,
total,
}
}))
})),
..
}, done: true }) if block_number == tip.number &&
from == checkpoint && to == previous_stage &&
// -1 because we don't need to download the local head

View File

@@ -502,7 +502,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
},
done: true
}) if block_number == previous_stage && processed == total &&
@@ -542,7 +543,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
},
done: true
}) if block_number == previous_stage && processed == total &&
@@ -584,7 +586,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
})),
..
},
done: true
}) if block_number == previous_stage && processed == total &&

View File

@@ -527,7 +527,8 @@ mod tests {
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed: 1,
total: 1
}))
})),
..
}, done: true }) if block_number == previous_stage
);

View File

@@ -337,12 +337,12 @@ mod tests {
result,
Ok(ExecOutput {
checkpoint: StageCheckpoint {
block_number,
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
}, done: true }) if block_number == previous_stage && processed == total &&
block_number,
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
}, done: true }) if block_number == previous_stage && processed == total &&
total == runner.db.count_entries::<tables::Transactions>().unwrap() as u64
);
@@ -383,12 +383,12 @@ mod tests {
result,
Ok(ExecOutput {
checkpoint: StageCheckpoint {
block_number,
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
}, done: true }) if block_number == previous_stage && processed == total &&
block_number,
stage_checkpoint: Some(StageUnitCheckpoint::Entities(EntitiesCheckpoint {
processed,
total
}))
}, done: true }) if block_number == previous_stage && processed == total &&
total == runner.db.count_entries::<tables::Transactions>().unwrap() as u64
);

Some files were not shown because too many files have changed in this diff Show More