wip

Co-authored-by: Amp <amp@ampcode.com>

.changelog/big-mules-cook.md

@@ -1,4 +0,0 @@
----
----
-
-Added site-level meta description for SEO.

.changelog/bold-frogs-run.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: patch
----
-
-Renamed and documented validation methods for clarity. `validate_one_no_state` and `validate_one_against_state` are now public methods `validate_stateless` and `validate_stateful` with improved documentation explaining their respective validation phases.

.changelog/brave-dogs-fly.md

@@ -1,10 +0,0 @@
----
-reth-engine-primitives: patch
-reth-engine-tree: patch
-reth-node-core: patch
-reth-trie-parallel: minor
----
-
-Removed legacy proof calculation system and V2-specific configuration flags.
-
-Removed the legacy (non-V2) proof calculation code paths, simplified multiproof task architecture by removing the dual-mode system, and cleaned up V2-specific CLI flags (`--engine.disable-proof-v2`, `--engine.disable-trie-cache`) that are no longer needed. The codebase now exclusively uses V2 proofs with the sparse trie cache.

.changelog/calm-clams-buzz.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Refactored sparse trie node state tracking to use RLP nodes instead of hashes. Replaced `Option<B256>` hash fields with `SparseNodeState` enum that tracks either dirty nodes or cached RLP nodes with optional database storage flags. Added debug assertions to validate leaf path lengths and improved pruning logic to use node paths directly instead of path-hash tuples.

.changelog/config.toml

@@ -1,20 +0,0 @@
-# Changelogs configuration for reth
-# https://github.com/wevm/changelogs
-
-# How to bump packages that depend on changed packages
-dependent_bump = "patch"
-
-[changelog]
-# Generate per-crate changelogs (vs single root changelog)
-format = "per-crate"
-
-# Fixed groups: all always share the same version
-# reth binaries share version
-[[fixed]]
-members = ["reth"]
-
-# Packages to ignore (internal/test-only crates)
-ignore = [
-    "reth-testing-utils",
-    "reth-bench",
-]

.changelog/cool-owls-rest.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Fixed a bug in `merge_subtrie_updates` where source insertions did not cancel destination removals (and vice versa), causing inconsistent trie updates accumulated across multiple `root()` calls without intermediate `take_updates()`. Added a test covering the cross-cancellation behavior.

.changelog/cool-suns-rest.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: minor
----
-
-Added support for optional custom stateless and stateful validation hooks in `EthTransactionValidator` via `set_additional_stateless_validation` and `set_additional_stateful_validation` methods. Also implemented a manual `Debug` impl to handle the non-`Debug` function pointer fields.

.changelog/cute-bears-paint.md

@@ -1,17 +0,0 @@
----
-reth: minor
-reth-cli-commands: minor
-reth-e2e-test-utils: minor
-reth-ethereum-cli: minor
-reth-node-core: minor
-reth-optimism-bin: minor
-reth-optimism-cli: minor
-reth-prune: patch
-reth-stages: patch
-reth-storage-api: minor
-reth-storage-db-api: minor
-reth-storage-db-common: patch
-reth-storage-provider: patch
----
-
-Introduced `--storage.v2` flag to control storage mode defaults, replacing the `edge` feature flag with `rocksdb` feature. The new flag enables v2 storage settings (static files + RocksDB routing) while individual `--static-files.*` and `--rocksdb.*` flags can still override defaults. Updated feature gates from `edge` to `rocksdb` across all affected crates.

.changelog/dark-ants-write.md

@@ -1,5 +0,0 @@
----
-reth-tasks: patch
----
-
-Added panic handler to all rayon thread pools that logs panics via `tracing::error` instead of aborting the process.

.changelog/dry-ants-bow.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Removed Windows platform support from the codebase, including the Windows cross-compilation Dockerfile, build targets in Cross.toml and Makefile, and Windows-specific options in the bug report template.

.changelog/dry-ducks-write.md

@@ -1,5 +0,0 @@
----
-reth-network: minor
----
-
-Added reason label to backed_off_peers metric. The metric now tracks backed off peers by reason (too_many_peers, graceful_close, connection_error) to improve observability.

.changelog/dry-koalas-wink.md

@@ -1,5 +0,0 @@
----
-reth-network-types: patch
----
-
-Increased default maximum concurrent outbound dials from 15 to 30.

.changelog/dull-clams-pack.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Refactored arena trie internals by adding a `BranchChildIdx::sibling()` helper, deduplicating `Index`/`NodeArena` type aliases, and replacing `is_empty()` with a `drop_root()` method. Fixed a bug where `cursor.pop()` was called before checking if the leaf was the root node, which could cause incorrect dirty-state propagation.

.changelog/dull-koalas-play.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Added recording of `SetRoot` operation in `ParallelSparseTrie::set_root` when the `trie-debug` feature is enabled.

.changelog/eager-mules-fold.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse-parallel: patch
----
-
-Fixed parallel sparse trie to skip revealing disconnected leaves by checking parent branch reachability before inserting leaf nodes.

.changelog/easy-clouds-meow.md

@@ -1,5 +0,0 @@
----
-ef-tests: patch
----
-
-Removed reth-stateless crate and stateless validation from ef-tests.

.changelog/evil-fish-smile.md

@@ -1,6 +0,0 @@
----
-reth-rpc-convert: minor
-reth-storage-rpc-provider: minor
----
-
-Replaced the separate `TryFromBlockResponse`, `TryFromReceiptResponse`, and `TryFromTransactionResponse` traits with a unified `RpcResponseConverter` trait and default `EthRpcConverter` implementation. Removed the `op-alloy-network` dependency and refactored `RpcBlockchainProvider` to store a dynamic converter instance instead of relying on per-type trait bounds.

.changelog/evil-pigs-cry.md

@@ -1,6 +0,0 @@
----
-reth-engine-tree: patch
-reth-trie-sparse-parallel: patch
----
-
-Added tracing spans and debug logs to sparse trie operations for better observability during parallel state root computation.

.changelog/fair-winds-growl.md

@@ -1,6 +0,0 @@
----
-reth-exex: patch
-reth-exex-types: patch
----
-
-Added configurable backfill thresholds to ExEx notifications stream and added regression tests for state provider parity between pipeline and backfill execution paths.

.changelog/fair-wolves-smile.md

@@ -1,5 +0,0 @@
----
-reth-payload-builder: minor
----
-
-Added observability metrics for payload resolve latency and new payload job creation latency to the payload builder service.

.changelog/fast-fish-cry.md

@@ -1,4 +0,0 @@
----
----
-
-Added WebSocket subscription integration tests for eth_subscribe.

.changelog/fast-seals-play.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: minor
----
-
-Added `consensus_ref` method to `PoolTransaction` trait for borrowing consensus transactions without cloning.

.changelog/fast-waves-smile.md

@@ -1,4 +0,0 @@
----
----
-
-Improved nightly Docker build failure Slack notification with more detailed formatting and context.

.changelog/fine-koalas-shout.md

@@ -1,10 +0,0 @@
----
-reth-chain-state: minor
-reth-engine-primitives: minor
-reth-engine-tree: minor
-reth-node-core: minor
-reth-node-events: minor
-reth: patch
----
-
-Added configurable slow block logging (`--engine.slow-block-threshold`) that emits a structured `warn!` log with detailed timing, state-operation counts, and cache hit-rate metrics for blocks whose total processing time exceeds the threshold. Introduced `ExecutionTimingStats`, `CacheStats`, `StateProviderStats`, and `SlowBlockInfo` types to carry execution statistics from block validation through persistence, and refactored `PersistenceResult` to carry commit duration alongside the last persisted block.

.changelog/fix-simulate-revert-code.md

@@ -1,5 +0,0 @@
----
-reth-rpc-eth-types: patch
----
-
-Updated `eth_simulateV1` revert error code from `-32000` to `3` to be consistent with `eth_call`, per [execution-apis#748](https://github.com/ethereum/execution-apis/pull/748).

.changelog/gentle-moons-cry.md

@@ -1,5 +0,0 @@
----
-reth-engine-tree: patch
----
-
-Reordered cache size calculations in `ExecutionCache::new` to group related operations together.

.changelog/icy-lions-slide.md

@@ -1,7 +0,0 @@
----
-reth: patch
-reth-cli-commands: patch
-reth-node-core: patch
----
-
-Removed experimental ress protocol support for stateless Ethereum nodes.

.changelog/icy-tigers-buzz.md

@@ -1,6 +0,0 @@
----
-reth-trie-db: minor
-reth-engine-tree: minor
----
-
-Added `PendingChangeset` and `PendingChangesetGuard` to `ChangesetCache` so concurrent readers wait for an in-progress computation instead of falling back to the expensive DB-based path. The guard automatically cancels the pending entry on drop (e.g. task panic), ensuring waiters always make progress.

.changelog/keen-geese-bake.md

@@ -1,5 +0,0 @@
----
-reth-engine-tree: patch
----
-
-Added sub-phase timing histograms to the sparse trie event loop, tracking channel wait, proof coalescing, multiproof reveal, and trie update durations separately.

.changelog/lazy-lakes-shout.md

@@ -1,5 +0,0 @@
----
-reth-node-builder: patch
----
-
-Removed biased select in engine service loop to allow fair scheduling of shutdown requests alongside event processing.

.changelog/lively-clams-drink.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: patch
----
-
-Fixed swapped arguments in `blob_tx_priority` function calls, correcting the parameter order to match the function signature.

.changelog/lively-clouds-bake.md

@@ -1,4 +0,0 @@
----
----
-
-Improved documentation overview page with better structure and clarity.

.changelog/lively-deer-shout.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Re-enabled changelog workflow to run automatically on pull requests.

.changelog/lively-foxes-play.md

@@ -1,5 +0,0 @@
----
-reth-node-events: patch
----
-
-Updated consensus engine log message to be more accurate about received updates.

.changelog/merry-bees-break.md

@@ -1,8 +0,0 @@
----
-reth-engine-primitives: minor
-reth-engine-tree: minor
-reth-node-core: minor
-reth-trie-parallel: minor
----
-
-Added `--engine.proof-jitter` CLI option behind the `trie-debug` feature flag. When set, each proof worker sleeps for a random duration up to the specified value before starting proof computation, useful for stress-testing timing-sensitive proof logic.

.changelog/merry-koalas-nod.md

@@ -1,6 +0,0 @@
----
-reth-rpc-eth-api: minor
-reth-rpc-server-types: minor
----
-
-Added `eth_getStorageValues` RPC method for batch storage slot retrieval across multiple addresses.

.changelog/mild-foxes-bark.md

@@ -1,6 +0,0 @@
----
-reth-chainspec: minor
-reth-network-peers: minor
----
-
-Removed OP stack bootnodes from default chain configurations and network peers module.

.changelog/mild-foxes-cook.md

@@ -1,5 +0,0 @@
----
-reth-rpc-convert: patch
----
-
-Updated `alloy-evm` dependency to git revision `9bc2dba` and adapted `TxEnvConverter` impl to the updated `TryIntoTxEnv` trait signature that now includes a `Spec` generic parameter.

.changelog/neat-ducks-whisper.md

@@ -1,6 +0,0 @@
----
-reth-trie: patch
-reth-trie-sparse: patch
----
-
-Refactored test harness for sparse trie tests by extracting `TrieTestHarness` into a shared `reth-trie` test utility, replacing duplicated inline harness code across multiple test modules. Updated `proof_v2` return type to include an optional root hash, and converted `original_root` and `storage` from public fields to accessor methods.

.changelog/nice-bears-cook.md

@@ -1,5 +0,0 @@
----
-reth-trie: patch
----
-
-Removed the local `increment_and_strip_trailing_zeros` function and `PATH_ALL_ZEROS` static in `proof_v2`, replacing them with the equivalent `Nibbles::next_without_prefix` and `Nibbles::is_zeroes` builtins. Also replaced manual `.get()` calls on `state_mask`/`hash_mask` with direct field access and switched to `Nibbles::unpack_array` over the unsafe `unpack_unchecked`.

.changelog/nice-trees-drink.md

@@ -1,9 +0,0 @@
----
-reth-network-api: minor
-reth-network-types: minor
-reth-network: minor
-reth-node-core: minor
-reth: minor
----
-
-Added optional ENR fork ID enforcement to filter out peers from incompatible networks during peer discovery, controlled by the `--enforce-enr-fork-id` CLI flag.

.changelog/nice-waves-bow.md

@@ -1,5 +0,0 @@
----
-reth-primitives: patch
----
-
-Moved feature-referenced dependencies from dev-dependencies to optional dependencies to ensure they are available when their corresponding features are enabled.

.changelog/odd-donkeys-chirp.md

@@ -1,5 +0,0 @@
----
-reth-engine-tree: patch
----
-
-Fixed `compare_trie_updates` to return `bool` indicating whether differences were found, and updated the caller to properly use the return value instead of treating all successful comparisons as having no differences.

.changelog/odd-frogs-break.md

@@ -1,7 +0,0 @@
----
-reth-cli-commands: minor
-reth-node-core: minor
-reth: patch
----
-
-Made v2 storage the default for all new databases, deprecating the `--storage.v2` flag to a hidden no-op kept for backwards compatibility. Updated CLI reference docs to remove the now-hidden flag from all command help pages.

.changelog/odd-snails-play.md

@@ -1,5 +0,0 @@
----
-reth-node-core: minor
----
-
-Added `with_dev_block_time` helper method to `NodeConfig` for configuring dev miner block production interval.

.changelog/old-dogs-shake.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: minor
----
-
-Added `IntoIter: Send` bounds to `validate_transactions` and `validate_transactions_with_origin` in the `TransactionValidator` trait, avoiding unnecessary `Vec` collects. Simplified default `validate_transactions_with_origin` to delegate to `validate_transactions`.

.changelog/proud-crabs-bark.md

@@ -1,5 +0,0 @@
----
-reth-db-api: patch
----
-
-Changed `StoredNibblesSubKey` encoding to use a stack-allocated `[u8; 65]` array instead of a heap-allocated `Vec<u8>`, avoiding unnecessary heap allocation.

.changelog/proud-slugs-fly.md

@@ -1,7 +0,0 @@
----
-reth-engine-tree: patch
-reth-trie-sparse: patch
-reth-tasks: patch
----
-
-Offloaded deallocation of expensive proof node buffers to a persistent background thread (`Runtime::spawn_drop`) to avoid blocking state root computation or lock-holding code.

.changelog/proud-wolves-spin.md

@@ -1,5 +0,0 @@
----
-reth-storage-api: patch
----
-
-Added `Arc` to `auto_impl` derive for storage-api traits to support automatic `Arc` wrapper implementations.

.changelog/quick-mice-jump.md

@@ -1,8 +0,0 @@
----
-reth: patch
-reth-engine-tree: patch
-reth-node-builder: patch
-reth-trie-sparse: minor
----
-
-Added `trie-debug` feature for recording sparse trie mutations to aid in debugging state root mismatches.

.changelog/quiet-foxes-dance.md

@@ -1,6 +0,0 @@
----
-reth-static-file-types: patch
-reth-provider: patch
----
-
-Move changeset offsets from segment header to external `.csoff` sidecar file for incremental writes and crash recovery.

.changelog/quiet-frogs-whisper.md

@@ -1,5 +0,0 @@
----
-reth-provider: patch
----
-
-Removed unused staging types from ProviderFactoryBuilder.

.changelog/rare-tigers-nod.md

@@ -1,6 +0,0 @@
----
-reth-trie-sparse: patch
-reth-engine-tree: patch
----
-
-Removed the `skip_proof_node_filtering` flag, `revealed_account_paths`/`revealed_paths` tracking, and the `filter_revealed_v2_proof_nodes` function from the sparse trie implementation. Also removed the corresponding skipped-nodes metrics, simplifying the proof node reveal path to always pass nodes directly to the sparse trie without pre-filtering.

.changelog/rare-whales-pack.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: minor
----
-
-Added a comprehensive generic `SparseTrie` test suite covering `set_root`, `reveal_nodes`, `update_leaves`, `root`, `take_updates`, `commit_updates`, `prune`, `wipe`/`clear`, `get_leaf_value`, `find_leaf`, `size_hint`, and integration lifecycle scenarios. Tests are stamped out for all concrete `SparseTrie` implementations via a macro.

.changelog/rich-lakes-bark.md

@@ -1,5 +0,0 @@
----
-reth-provider: patch
----
-
-Fixed sender pruning during block reorg to skip when sender_recovery is fully pruned, preventing a fatal crash when no sender data exists in static files.

.changelog/safe-waves-read.md

@@ -1,7 +0,0 @@
----
-reth-network-types: minor
-reth-network: minor
-reth-node-core: patch
----
-
-Added `PersistedPeerInfo` struct to persist richer peer metadata (kind, fork ID, reputation) to disk. Updated `PeersConfig::with_basic_nodes_from_file` to support both the new `PersistedPeerInfo` format and the legacy `Vec<NodeRecord>` format with automatic conversion, and updated `write_peers_to_file` to exclude backed-off and banned peers.

.changelog/shy-tigers-dry.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Added automated changelog generation infrastructure using wevm/changelogs-rs with Claude Code integration. Configured per-crate changelog format with fixed version groups for reth binaries and exclusions for internal test utilities.

.changelog/swift-owls-fly.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: minor
----
-
-Removed `SerialSparseTrie` from the workspace, consolidating on `ParallelSparseTrie` as the single sparse trie implementation in `reth-trie-sparse`.

.changelog/tall-fish-climb.md

@@ -1,5 +0,0 @@
----
-reth-cli-commands: minor
----
-
-Added `reth_version` field to `SnapshotManifest` to record the Reth version that produced a snapshot. The field is optional and populated automatically during manifest generation.

.changelog/tall-stars-shout.md

@@ -1,5 +0,0 @@
----
-reth-network: minor
----
-
-Added `fork_id` as a tiebreaker in peer selection when reputations are equal, preferring peers with a discovered `fork_id` as it indicates fork compatibility. Added a test to verify the tiebreaker behavior.

.changelog/tidy-stars-cry.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Fixed a bug where trie nodes could appear in both `updated_nodes` and `removed_nodes` simultaneously by removing entries from `removed_nodes` when a node is inserted as updated.

.changelog/vain-bats-bark.md

@@ -1,8 +0,0 @@
----
-reth-trie-common: minor
-reth-trie: minor
-reth-trie-parallel: minor
-reth-engine-tree: patch
----
-
-Moved `ProofV2Target`, `MultiProofTargetsV2`, and `ChunkedMultiProofTargetsV2` from `reth-trie-parallel::targets_v2` into a new `reth-trie-common::target_v2` module, making these types available at a lower level without pulling in the full parallel trie crate. Added a `multiproof_v2` method to `Proof` in `reth-trie` that generates a state multiproof using the V2 proof calculator with synchronous account value encoding.

.changelog/vain-birds-laugh.md

@@ -1,14 +0,0 @@
----
-reth-engine-tree: patch
-reth-evm-ethereum: patch
-reth-evm: patch
-reth-primitives-traits: patch
-reth-revm: patch
-reth-rpc-eth-api: patch
-reth-rpc-eth-types: patch
-reth-provider: patch
-example-custom-evm: patch
-example-precompile-cache: patch
----
-
-Bumped revm to v35.0.0, revm-inspectors to 0.35.0, and alloy-evm to 0.29.0. Updated call sites throughout the codebase to align with the new APIs, including `ExecutionResult` field changes (`gas_used` → `gas.used()`/`gas.final_refunded()`), removal of `.without_state_clear()`, updated `EthPrecompiles::new(spec)` constructor, and updated `block_hashes.lowest()` access.

.changelog/vain-goats-cook.md

@@ -1,9 +0,0 @@
----
-reth-trie-sparse: minor
-reth-engine-primitives: minor
-reth-engine-tree: minor
-reth-node-core: minor
-reth-trie-common: patch
----
-
-Added an arena-based sparse trie implementation (`ArenaParallelSparseTrie`) using `slotmap` arena allocation for node storage, enabling parallel subtrie mutation without per-node hashing overhead. Added `ConfigurableSparseTrie` enum to switch between the arena and hash-map implementations, and a `--engine.enable-arena-sparse-trie` CLI flag to opt in at runtime.

.changelog/vain-lakes-cry.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Updated Alloy dependencies from 1.5.2 to 1.6.1.

.changelog/vast-waves-fold.md

@@ -1,4 +0,0 @@
----
----
-
-Expanded CLI integration tests with subcommand help coverage, config TOML validation, genesis JSON validation, and send transaction round-trip test for dev mode.

.changelog/warm-donkeys-dry.md

@@ -1,6 +0,0 @@
----
-reth-engine-local: minor
-reth-node-builder: minor
----
-
-Added trigger-based `MiningMode` variant that allows blocks to be built on-demand via custom streams, and exposed `with_mining_mode` method on `DebugNodeLauncherFuture` to override default mining configuration.

.changelog/warm-foxes-glow.md

@@ -1,5 +0,0 @@
----
-reth-network: minor
----
-
-Added direction labels to `closed_sessions` and `pending_session_failures` metrics. Operators can now distinguish session closures and failures by direction (`active`, `incoming_pending`, `outgoing_pending` for closed sessions; `inbound`, `outbound` for pending session failures).

.changelog/warm-foxes-jump.md

@@ -1,4 +0,0 @@
----
----
-
-Moved Kurtosis CI failure notifications to the hive Slack channel.

.changelog/warm-geese-build.md

@@ -1,7 +0,0 @@
----
-reth-rpc-api: minor
-reth-rpc-builder: patch
-reth-rpc: minor
----
-
-Added `subscribeFinalizedChainNotifications` RPC endpoint that buffers committed chain notifications and emits them once a new finalized block is received.

.changelog/warm-pandas-cook.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: patch
----
-
-Fixed a bug where transactions from the same sender were added to the pending subpool out of nonce order. Ensured `process_updates` runs before `add_new_transaction` so that lower-nonce promotions are enqueued before the newly inserted higher-nonce transaction, preserving correct ordering for live `BestTransactions` iterators.

.changelog/young-horses-play.md

@@ -1,7 +0,0 @@
----
-reth-trie: major
-reth-trie-db: major
-reth-provider: minor
----
-
-Added `MaskedTrieCursorFactory` and `MaskedTrieCursor` to handle prefix-set-based hash invalidation at the cursor layer, replacing the `DatabaseTrieWitness` trait abstraction. Removed `with_prefix_sets_mut` from `TrieWitness` and deleted `DatabaseTrieWitness` — callers should now wrap their cursor factory with `MaskedTrieCursorFactory` to apply prefix sets during witness/proof computation.

.changelog/zesty-birds-smile.md

@@ -1,6 +0,0 @@
----
-reth-trie: minor
-reth-trie-parallel: minor
----
-
-Added `root_node` and `storage_root_node` methods to proof calculators for efficient root-only calculations. These methods directly return the root node without requiring dummy targets, replacing the previous workaround of passing fake targets to proof generation.

.changelog/zesty-clouds-wave.md

@@ -1,5 +0,0 @@
----
-reth-trie: patch
----
-
-Fixed a potential panic in `ProofCalculator` by clearing internal computation state (`branch_stack`, `child_stack`, `branch_path`, etc.) after errors, preventing stale state from causing `usize` underflow panics when the calculator is reused. Added a test verifying correct behavior after simulated mid-computation errors.

.config/zepter.yaml

@@ -12,7 +12,7 @@ workflows:
         # Check that `A` activates the features of `B`.
         "propagate-feature",
         # These are the features to check:
-        "--features=std,op,dev,asm-keccak,jemalloc,jemalloc-prof,tracy-allocator,tracy,serde-bincode-compat,serde,test-utils,arbitrary,bench,alloy-compat,min-error-logs,min-warn-logs,min-info-logs,min-debug-logs,min-trace-logs,otlp,otlp-logs,js-tracer,portable,keccak-cache-global",
+        "--features=std,op,dev,asm-keccak,jemalloc,jemalloc-prof,tracy-allocator,tracy,serde-bincode-compat,serde,test-utils,arbitrary,bench,alloy-compat,min-error-logs,min-warn-logs,min-info-logs,min-debug-logs,min-trace-logs,otlp,otlp-logs,js-tracer,portable,keccak-cache-global,trie-debug",
         # Do not try to add a new section to `[features]` of `A` only because `B` exposes that feature. There are edge-cases where this is still needed, but we can add them manually.
         "--left-side-feature-missing=ignore",
         # Ignore the case that `A` it outside of the workspace. Otherwise it will report errors in external dependencies that we have no influence on.

.dockerignore

@@ -20,11 +20,6 @@
 # include dist directory, where the reth binary is located after compilation
 !/dist
 
-# include PGO build helper used by Dockerfile.depot
-!/.github
-!/.github/scripts
-!/.github/scripts/build_pgo_bolt.sh
-
 # include licenses
 !LICENSE-*
 

.github/scripts/bench-job-summary.js

@@ -0,0 +1,106 @@
+// Generates a rich GitHub Actions job summary for reth-bench results.
+//
+// Reads from environment:
+//   BENCH_WORK_DIR       – Directory containing summary.json
+//   BENCH_PR             – PR number (may be empty)
+//   BENCH_ACTOR          – GitHub user who triggered the bench
+//   BENCH_CORES          – CPU core limit (0 = all)
+//   BENCH_WARMUP_BLOCKS  – Number of warmup blocks
+//   BENCH_SAMPLY         – 'true' if samply profiling was enabled
+//   BENCH_ABBA           – 'true' if ABBA interleaved order was used
+//
+// Usage from actions/github-script:
+//   const jobSummary = require('./.github/scripts/bench-job-summary.js');
+//   await jobSummary({ core, context, chartSha, grafanaUrl, runId });
+
+const fs = require('fs');
+const { verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
+
+module.exports = async function ({ core, context, chartSha, grafanaUrl, runId }) {
+  let summary;
+  try {
+    summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+  } catch (e) {
+    await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
+    return;
+  }
+
+  const repo = `${context.repo.owner}/${context.repo.repo}`;
+  const prNumber = process.env.BENCH_PR;
+  const actor = process.env.BENCH_ACTOR;
+  const commitUrl = `https://github.com/${repo}/commit`;
+
+  const { emoji, label } = verdict(summary.changes);
+  const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
+  const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
+  const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+
+  // Header & metadata
+  const metaParts = [];
+  if (prNumber) metaParts.push(`**[PR #${prNumber}](https://github.com/${repo}/pull/${prNumber})**`);
+  metaParts.push(`triggered by @${actor}`);
+
+  let md = `# ${emoji} ${label}\n\n`;
+  md += metaParts.join(' · ') + '\n\n';
+  md += `**Baseline:** ${baselineLink}\n`;
+  md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
+  md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
+
+  // Main comparison table
+  const rows = metricRows(summary);
+  md += `| Metric | Baseline | Feature | Change |\n`;
+  md += `|--------|----------|---------|--------|\n`;
+  for (const r of rows) {
+    md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
+  }
+  md += '\n';
+
+  // Wait time breakdown
+  const wtRows = waitTimeRows(summary);
+  if (wtRows.length > 0) {
+    md += `### Wait Time Breakdown\n\n`;
+    md += `| Metric | Baseline | Feature |\n`;
+    md += `|--------|----------|--------|\n`;
+    for (const r of wtRows) {
+      md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
+    }
+    md += '\n';
+  }
+
+  // Charts
+  if (chartSha) {
+    const prNum = prNumber || '0';
+    const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/pr/${prNum}/${runId}`;
+    const charts = [
+      { file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
+      { file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
+      { file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
+    ];
+    md += `### Charts\n\n`;
+    for (const chart of charts) {
+      md += `<details><summary>${chart.label}</summary>\n\n`;
+      md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
+      md += `</details>\n\n`;
+    }
+  }
+
+  // Samply profiles
+  const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
+  const samplyLinks = Object.entries(samplyUrls).map(([run, url]) => `- **${run}**: [Firefox Profiler](${url})`);
+  if (samplyLinks.length > 0) {
+    md += `### Samply Profiles\n\n${samplyLinks.join('\n')}\n\n`;
+  }
+
+  // Grafana
+  if (grafanaUrl) {
+    md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
+  }
+
+  // Node errors
+  try {
+    const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
+    if (errors.trim()) md += '\n' + errors + '\n';
+  } catch {}
+
+  await core.summary.addRaw(md).write();
+};

.github/scripts/bench-reth-build.sh

@@ -1,27 +1,35 @@
 #!/usr/bin/env bash
 #
-# Builds (or fetches from cache) reth binaries for benchmarking.
+# Builds reth binaries for benchmarking from local source only.
 #
-# Usage: bench-reth-build.sh <baseline|feature> <source-dir> <commit> [branch-sha]
+# Usage: bench-reth-build.sh <baseline|feature> <source-dir> <commit>
 #
-#   baseline — build/fetch the baseline binary at <commit> (merge-base)
+#   baseline — build the baseline binary at <commit> (merge-base)
 #              source-dir must be checked out at <commit>
-#   feature  — build/fetch the candidate binary + reth-bench at <commit>
+#   feature  — build the candidate binary + reth-bench at <commit>
 #              source-dir must be checked out at <commit>
-#              optional branch-sha is the PR head commit for cache key
 #
 # Outputs:
-#   baseline: <source-dir>/target/profiling/reth
-#   feature:  <source-dir>/target/profiling/reth, reth-bench installed to cargo bin
+#   baseline: <source-dir>/target/profiling/reth (or reth-bb if BENCH_BIG_BLOCKS=true)
+#   feature:  <source-dir>/target/profiling/reth (or reth-bb), reth-bench installed to cargo bin
 #
-# Required: mc (MinIO client) with a configured alias
-set -euo pipefail
+# Optional env: BENCH_BIG_BLOCKS (true/false) — build reth-bb instead of reth
+set -euxo pipefail
 
-MC="mc"
 MODE="$1"
 SOURCE_DIR="$2"
 COMMIT="$3"
 
+BIG_BLOCKS="${BENCH_BIG_BLOCKS:-false}"
+# The node binary to build: reth-bb for big blocks, reth otherwise
+if [ "$BIG_BLOCKS" = "true" ]; then
+  NODE_BIN="reth-bb"
+  NODE_PKG="-p reth-bb"
+else
+  NODE_BIN="reth"
+  NODE_PKG="--bin reth"
+fi
+
 # Tracy support: when BENCH_TRACY is "on" or "full", add Tracy cargo features
 # and frame pointers for accurate stack traces.
 EXTRA_FEATURES=""
@@ -31,101 +39,38 @@ if [ "${BENCH_TRACY:-off}" != "off" ]; then
   EXTRA_RUSTFLAGS=" -C force-frame-pointers=yes"
 fi
 
-# Cache suffix: hash of features+rustflags so different build configs get separate cache entries
-if [ -n "$EXTRA_FEATURES" ] || [ -n "$EXTRA_RUSTFLAGS" ]; then
-  BUILD_SUFFIX="-$(echo "${EXTRA_FEATURES}${EXTRA_RUSTFLAGS}" | sha256sum | cut -c1-12)"
-else
-  BUILD_SUFFIX=""
-fi
+# Build the requested node binary with the benchmark profile.
+build_node_binary() {
+  local features_arg=""
+  local workspace_arg=""
 
-# Verify a cached reth binary was built from the expected commit.
-# `reth --version` outputs "Commit SHA: <full-sha>" on its own line.
-verify_binary() {
-  local binary="$1" expected_commit="$2"
-  local version binary_sha
-  version=$("$binary" --version 2>/dev/null) || return 1
-  binary_sha=$(echo "$version" | sed -n 's/^Commit SHA: *//p')
-  if [ -z "$binary_sha" ]; then
-    echo "Warning: could not extract commit SHA from version output"
-    return 1
+  cd "$SOURCE_DIR"
+  if [ -n "$EXTRA_FEATURES" ]; then
+    # --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
+    features_arg="--features ${EXTRA_FEATURES}"
+    workspace_arg="--workspace"
   fi
-  if [ "$binary_sha" = "$expected_commit" ]; then
-    return 0
-  fi
-  echo "Cache mismatch: binary built from ${binary_sha} but expected ${expected_commit}"
-  return 1
+
+  # shellcheck disable=SC2086
+  RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
+    cargo build --profile profiling $NODE_PKG $workspace_arg $features_arg
 }
 
 case "$MODE" in
   baseline|main)
-    BUCKET="minio/reth-binaries/${COMMIT}${BUILD_SUFFIX}"
-    mkdir -p "${SOURCE_DIR}/target/profiling"
-
-    CACHE_VALID=false
-    if $MC stat "${BUCKET}/reth" &>/dev/null; then
-      echo "Cache hit for baseline (${COMMIT}), downloading binary..."
-      $MC cp "${BUCKET}/reth" "${SOURCE_DIR}/target/profiling/reth"
-      chmod +x "${SOURCE_DIR}/target/profiling/reth"
-      if verify_binary "${SOURCE_DIR}/target/profiling/reth" "${COMMIT}"; then
-        CACHE_VALID=true
-      else
-        echo "Cached baseline binary is stale, rebuilding..."
-      fi
-    fi
-    if [ "$CACHE_VALID" = false ]; then
-      echo "Building baseline (${COMMIT}) from source..."
-      cd "${SOURCE_DIR}"
-      FEATURES_ARG=""
-      WORKSPACE_ARG=""
-      if [ -n "$EXTRA_FEATURES" ]; then
-        # --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
-        FEATURES_ARG="--features ${EXTRA_FEATURES}"
-        WORKSPACE_ARG="--workspace"
-      fi
-      # shellcheck disable=SC2086
-      RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
-        cargo build --profile profiling --bin reth $WORKSPACE_ARG $FEATURES_ARG
-      $MC cp target/profiling/reth "${BUCKET}/reth"
-    fi
+    echo "Building baseline ${NODE_BIN} (${COMMIT}) from source..."
+    build_node_binary
     ;;
 
   feature|branch)
-    BRANCH_SHA="${4:-$COMMIT}"
-    BUCKET="minio/reth-binaries/${BRANCH_SHA}${BUILD_SUFFIX}"
-
-    CACHE_VALID=false
-    if $MC stat "${BUCKET}/reth" &>/dev/null && $MC stat "${BUCKET}/reth-bench" &>/dev/null; then
-      echo "Cache hit for ${BRANCH_SHA}, downloading binaries..."
-      mkdir -p "${SOURCE_DIR}/target/profiling"
-      $MC cp "${BUCKET}/reth" "${SOURCE_DIR}/target/profiling/reth"
-      $MC cp "${BUCKET}/reth-bench" /home/ubuntu/.cargo/bin/reth-bench
-      chmod +x "${SOURCE_DIR}/target/profiling/reth" /home/ubuntu/.cargo/bin/reth-bench
-      if verify_binary "${SOURCE_DIR}/target/profiling/reth" "${COMMIT}"; then
-        CACHE_VALID=true
-      else
-        echo "Cached feature binary is stale, rebuilding..."
-      fi
-    fi
-    if [ "$CACHE_VALID" = false ]; then
-      echo "Building feature (${COMMIT}) from source..."
-      cd "${SOURCE_DIR}"
-      rustup show active-toolchain || rustup default stable
-      if [ -n "$EXTRA_FEATURES" ]; then
-        # Can't use `make profiling` when adding features; build explicitly
-        # --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
-        RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
-          cargo build --profile profiling --workspace --bin reth --features "${EXTRA_FEATURES}"
-      else
-        make profiling
-      fi
-      make install-reth-bench
-      $MC cp target/profiling/reth "${BUCKET}/reth"
-      $MC cp "$(which reth-bench)" "${BUCKET}/reth-bench"
-    fi
+    echo "Building feature ${NODE_BIN} (${COMMIT}) from source..."
+    rustup show active-toolchain || rustup default stable
+    build_node_binary
+    make -C "$SOURCE_DIR" install-reth-bench
     ;;
 
   *)
-    echo "Usage: $0 <baseline|feature> <source-dir> <commit> [branch-sha]"
+    echo "Usage: $0 <baseline|feature> <source-dir> <commit>"
     exit 1
     ;;
 esac

.github/scripts/bench-reth-local.sh

@@ -2,7 +2,7 @@
 #
 # local-reth-bench.sh — Run the reth Engine API benchmark locally.
 #
-# Replicates the CI bench.yml workflow (build, snapshot, system tuning,
+# Replicates the CI bench.yml workflow (build, local snapshot validation, system tuning,
 # interleaved B-F-F-B execution, summary, charts) without any GitHub
 # Actions glue (no PR comments, no artifact upload, no Slack).
 #
@@ -21,15 +21,17 @@
 # Requires: the reth repo at RETH_REPO (default: ~/reth)
 #
 # Dependencies (install before first run):
-#   mc (MinIO client), schelk, cpupower, taskset, stdbuf, python3, curl,
-#   make, uv, pzstd, jq, Rust toolchain (cargo/rustup)
+#   schelk, cpupower, taskset, stdbuf, python3, curl,
+#   make, uv, jq, Rust toolchain (cargo/rustup)
+# Optional:
+#   mc for Tracy profile upload
 #
 # The script delegates to the existing bench-reth-*.sh scripts in the reth
 # repo for the actual build, snapshot, and run steps.
-set -euo pipefail
+set -euxo pipefail
 
 # ── PATH ──────────────────────────────────────────────────────────────
-# Ensure cargo and user-local bins (mc, uv) are visible
+# Ensure cargo and user-local bins (uv) are visible
 export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
 
 # ── Defaults ──────────────────────────────────────────────────────────
@@ -106,7 +108,7 @@ fi
 
 # ── Check dependencies ───────────────────────────────────────────────
 missing=()
-for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq cargo; do
+for cmd in schelk cpupower taskset stdbuf python3 curl make uv jq cargo; do
   command -v "$cmd" &>/dev/null || missing+=("$cmd")
 done
 if [ ${#missing[@]} -gt 0 ]; then
@@ -238,19 +240,14 @@ echo "  Baseline src : $BASELINE_SRC"
 echo "  Feature src  : $FEATURE_SRC"
 echo
 
-# ── Step 3: Check / download snapshot ────────────────────────────────
-echo "▸ Checking snapshot..."
+# ── Step 3: Validate local snapshot ──────────────────────────────────
+echo "▸ Validating local snapshot..."
 cd "$RETH_REPO"
-SNAPSHOT_NEEDED=false
-if ! "${SCRIPTS_DIR}/bench-reth-snapshot.sh" --check; then
-  SNAPSHOT_NEEDED=true
-  echo "  Snapshot needs update."
-else
-  echo "  Snapshot is up-to-date."
-fi
+"${SCRIPTS_DIR}/bench-reth-snapshot.sh"
+echo "  Snapshot is ready."
 echo
 
-# ── Step 4: Build binaries (+ snapshot download) in parallel ─────────
+# ── Step 4: Build binaries in parallel ───────────────────────────────
 echo "▸ Building binaries (parallel)..."
 cd "$RETH_REPO"
 
@@ -262,19 +259,11 @@ PID_BASELINE=$!
 "${SCRIPTS_DIR}/bench-reth-build.sh" feature "$FEATURE_SRC" "$FEATURE_SHA" &
 PID_FEATURE=$!
 
-PID_SNAPSHOT=
-if [ "$SNAPSHOT_NEEDED" = "true" ]; then
-  echo "  Also downloading snapshot in parallel..."
-  "${SCRIPTS_DIR}/bench-reth-snapshot.sh" &
-  PID_SNAPSHOT=$!
-fi
-
 wait $PID_BASELINE || FAIL=1
 wait $PID_FEATURE  || FAIL=1
-[ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
 
 if [ $FAIL -ne 0 ]; then
-  echo "Error: one or more parallel tasks failed (builds / snapshot)"
+  echo "Error: one or more build tasks failed"
   exit 1
 fi
 echo "  Binaries built successfully."
@@ -399,12 +388,9 @@ upload_tracy() {
 # ── Step 6: Pre-flight cleanup ───────────────────────────────────────
 echo "▸ Pre-flight cleanup..."
 pkill -f bench-metrics-proxy 2>/dev/null || true
-sudo pkill -9 reth 2>/dev/null || true
-sleep 1
-if mountpoint -q "$SCHELK_MOUNT" 2>/dev/null; then
-  sudo umount -l "$SCHELK_MOUNT" 2>/dev/null || true
-  sudo schelk recover -y 2>/dev/null || true
-fi
+sudo systemctl stop "${RETH_SCOPE:-reth-bench.scope}" 2>/dev/null || true
+sudo systemctl reset-failed "${RETH_SCOPE:-reth-bench.scope}" 2>/dev/null || true
+sudo schelk recover -y --kill || sudo schelk full-recover -y || true
 echo
 
 # ── Step 7: Interleaved benchmark runs (B-F-F-B) ────────────────────

.github/scripts/bench-reth-run.sh

@@ -7,21 +7,28 @@
 #
 # Required env: SCHELK_MOUNT, BENCH_RPC_URL, BENCH_BLOCKS, BENCH_WARMUP_BLOCKS
 # Optional env: BENCH_BIG_BLOCKS (true/false), BENCH_WORK_DIR (for big blocks path)
-#               BENCH_RETH_NEW_PAYLOAD (true/false, default true)
+#               BENCH_BAL (false/true/feature/baseline; only used with big blocks)
 #               BENCH_WAIT_TIME (duration like 500ms, default empty)
 #               BENCH_BASELINE_ARGS (extra reth node args for baseline runs)
 #               BENCH_FEATURE_ARGS (extra reth node args for feature runs)
 #               BENCH_OTLP_TRACES_ENDPOINT (OTLP HTTP endpoint for traces, e.g. https://host/insert/opentelemetry/v1/traces)
 #               BENCH_OTLP_LOGS_ENDPOINT (OTLP HTTP endpoint for logs, e.g. https://host/insert/opentelemetry/v1/logs)
-set -euo pipefail
+#               BENCH_OTLP_DISABLED (true to skip OTLP export even if endpoints are set)
+set -euxo pipefail
 
 LABEL="$1"
 BINARY="$2"
 OUTPUT_DIR="$3"
-DATADIR="$SCHELK_MOUNT/datadir"
+DATADIR_NAME="datadir"
+if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
+  DATADIR_NAME="datadir-big-blocks"
+fi
+DATADIR="$SCHELK_MOUNT/$DATADIR_NAME"
 mkdir -p "$OUTPUT_DIR"
 LOG="${OUTPUT_DIR}/node.log"
 
+RETH_SCOPE="${RETH_SCOPE:-reth-bench.scope}"
+
 cleanup() {
   kill "$TAIL_PID" 2>/dev/null || true
   # Stop tracy-capture first (SIGINT makes it disconnect and flush to disk)
@@ -42,7 +49,7 @@ cleanup() {
     fi
     wait "$TRACY_PID" 2>/dev/null || true
   fi
-  if [ -n "${RETH_PID:-}" ] && sudo kill -0 "$RETH_PID" 2>/dev/null; then
+  if sudo systemctl is-active "$RETH_SCOPE" >/dev/null 2>&1; then
     if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
       # Send SIGINT to the inner reth process by exact name (not -f which
       # would also match samply's cmdline containing "reth"). Samply will
@@ -60,34 +67,37 @@ cleanup() {
         echo "Samply still running after 120s, sending SIGTERM..."
         sudo pkill -x samply 2>/dev/null || true
       fi
-    else
-      sudo kill "$RETH_PID"
-      for i in $(seq 1 30); do
-        sudo kill -0 "$RETH_PID" 2>/dev/null || break
-        sleep 1
-      done
     fi
-    sudo kill -9 "$RETH_PID" 2>/dev/null || true
+    # Stop the entire systemd scope — kills all processes in the cgroup.
+    # This is reliable regardless of process reparenting or PID wrapper issues.
+    sudo systemctl stop "$RETH_SCOPE" 2>/dev/null || true
     sleep 1
   fi
+  sudo systemctl reset-failed "$RETH_SCOPE" 2>/dev/null || true
   # Fix ownership of reth-created files (reth runs as root)
   sudo chown -R "$(id -un):$(id -gn)" "$OUTPUT_DIR" 2>/dev/null || true
-  if mountpoint -q "$SCHELK_MOUNT"; then
-    sudo umount -l "$SCHELK_MOUNT" || true
-    sudo schelk recover -y || true
-  fi
+  # Let schelk recover the mounted volume in place so dm-era can restore only
+  # the changed blocks and clean up its own state.
+  sudo schelk recover -y --kill || true
 }
 TAIL_PID=
 TRACY_PID=
 trap cleanup EXIT
 
-# Clean up stale schelk state from a previous cancelled run.
-# If schelk thinks it's still mounted (e.g. a cancelled run skipped cleanup),
-# recover first to reset state.
-sudo schelk recover -y -k || true
+# Clean up stale state from a previous cancelled run.
+# Stop any leftover reth process in the scope, then recover schelk state.
+sudo systemctl stop "$RETH_SCOPE" 2>/dev/null || true
+sudo systemctl reset-failed "$RETH_SCOPE" 2>/dev/null || true
+sudo schelk recover -y --kill || sudo schelk full-recover -y || true
 
 # Mount
-sudo schelk mount -y
+sudo schelk mount -y || true
+if [ ! -d "$DATADIR/db" ] || [ ! -d "$DATADIR/static_files" ]; then
+  echo "::error::Failed to mount benchmark datadir at ${DATADIR}"
+  ls -la "$SCHELK_MOUNT" || true
+  ls -la "$DATADIR" || true
+  exit 1
+fi
 sync
 sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
 echo "=== Cache state after drop ==="
@@ -120,9 +130,12 @@ RETH_ARGS=(
   --no-persist-peers
 )
 
-# Big blocks mode requires the testing API and skip-invalid-transactions
-if [ "$BIG_BLOCKS" = "true" ]; then
-  RETH_ARGS+=(--http.api eth,net,web3,reth,testing --testing.skip-invalid-transactions)
+# Gate flag on binary support (older baselines may not have it).
+# Uses --help which exits immediately via clap without node init.
+SYNC_STATE_IDLE=false
+if "$BINARY" node --help 2>/dev/null | grep -qF -- '--debug.startup-sync-state-idle'; then
+  RETH_ARGS+=(--debug.startup-sync-state-idle)
+  SYNC_STATE_IDLE=true
 fi
 
 # Append per-label extra node args (baseline or feature)
@@ -142,11 +155,13 @@ if [ -n "${BENCH_METRICS_ADDR:-}" ]; then
 fi
 
 # OTLP traces and logs export
-if [ -n "${BENCH_OTLP_TRACES_ENDPOINT:-}" ]; then
-  RETH_ARGS+=(--tracing-otlp="${BENCH_OTLP_TRACES_ENDPOINT}" --tracing-otlp.service-name=reth-bench)
-fi
-if [ -n "${BENCH_OTLP_LOGS_ENDPOINT:-}" ]; then
-  RETH_ARGS+=(--logs-otlp="${BENCH_OTLP_LOGS_ENDPOINT}" --logs-otlp.filter=debug)
+if [ "${BENCH_OTLP_DISABLED:-false}" != "true" ]; then
+  if [ -n "${BENCH_OTLP_TRACES_ENDPOINT:-}" ]; then
+    RETH_ARGS+=(--tracing-otlp="${BENCH_OTLP_TRACES_ENDPOINT}" --tracing-otlp.service-name=reth-bench)
+  fi
+  if [ -n "${BENCH_OTLP_LOGS_ENDPOINT:-}" ]; then
+    RETH_ARGS+=(--logs-otlp="${BENCH_OTLP_LOGS_ENDPOINT}" --logs-otlp.filter=debug)
+  fi
 fi
 
 # Tracy profiling: add --log.tracy flags and set environment
@@ -173,19 +188,19 @@ echo "Memory limit: $(( MEM_LIMIT / 1024 / 1024 ))MB (95% of $(( TOTAL_MEM_KB /
 if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
   RETH_ARGS+=(--log.samply)
   SAMPLY="$(which samply)"
-  sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
+  sudo systemd-run --quiet --scope --collect --unit="$RETH_SCOPE" \
+    -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
     env "${SUDO_ENV[@]}" nice -n -20 \
     "$SAMPLY" record --save-only --presymbolicate --rate 10000 \
     --output "$OUTPUT_DIR/samply-profile.json.gz" \
     -- "$BINARY" "${RETH_ARGS[@]}" \
     > "$LOG" 2>&1 &
 else
-  sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
+  sudo systemd-run --quiet --scope --collect --unit="$RETH_SCOPE" \
+    -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
     env "${SUDO_ENV[@]}" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
     > "$LOG" 2>&1 &
 fi
-
-RETH_PID=$!
 stdbuf -oL tail -f "$LOG" | sed -u "s/^/[reth] /" &
 TAIL_PID=$!
 
@@ -194,7 +209,7 @@ for i in $(seq 1 60); do
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
     > /dev/null 2>&1; then
-    echo "reth (${LABEL}) is ready after ${i}s"
+    echo "reth (${LABEL}) RPC is up after ${i}s"
     break
   fi
   if [ "$i" -eq 60 ]; then
@@ -205,28 +220,83 @@ for i in $(seq 1 60); do
   sleep 1
 done
 
+# Wait for the pipeline to finish (eth_syncing returns false) so the
+# engine is in live mode and can accept newPayload calls.
+# Only possible when --debug.startup-sync-state-idle is supported.
+if [ "$SYNC_STATE_IDLE" = "true" ]; then
+  for i in $(seq 1 300); do
+    SYNC_RESULT=$(curl -sf http://127.0.0.1:8545 -X POST \
+      -H 'Content-Type: application/json' \
+      -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' 2>/dev/null || true)
+    if [ -n "$SYNC_RESULT" ] && jq -e '.result == false' <<< "$SYNC_RESULT" > /dev/null 2>&1; then
+      echo "reth (${LABEL}) pipeline finished after ${i}s, engine is live"
+      break
+    fi
+    if [ "$i" -eq 300 ]; then
+      echo "::error::reth (${LABEL}) pipeline did not finish within 300s"
+      cat "$LOG"
+      exit 1
+    fi
+    sleep 1
+  done
+else
+  echo "reth (${LABEL}) binary does not support --debug.startup-sync-state-idle, skipping sync wait"
+fi
+
 # Run reth-bench with high priority but as the current user so output
 # files are not root-owned (avoids EACCES on next checkout).
 BENCH_NICE="sudo nice -n -20 sudo -u $(id -un)"
 
 # Build optional flags
-EXTRA_BENCH_ARGS=()
-if [ "${BENCH_RETH_NEW_PAYLOAD:-true}" != "false" ]; then
-  EXTRA_BENCH_ARGS+=(--reth-new-payload)
-fi
+EXTRA_BENCH_ARGS=(--reth-new-payload)
 if [ -n "${BENCH_WAIT_TIME:-}" ]; then
   EXTRA_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
 fi
 
 if [ "$BIG_BLOCKS" = "true" ]; then
-  # Big blocks mode: replay pre-generated payloads with gas ramp
-  BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
-  # Count gas ramp blocks for reporting
-  GAS_RAMP_COUNT=$(find "$BIG_BLOCKS_DIR/gas-ramp-dir" -name '*.json' | wc -l)
-  echo "$GAS_RAMP_COUNT" > "$OUTPUT_DIR/gas_ramp_blocks.txt"
-  echo "Gas ramp blocks: $GAS_RAMP_COUNT"
+  # Big blocks mode: replay pre-generated payloads
+  BIG_BLOCKS_DIR="${BENCH_BIG_BLOCKS_DIR:-${BENCH_WORK_DIR}/big-blocks}"
+  BENCH_BAL_MODE="${BENCH_BAL:-false}"
 
-  # Start tracy-capture so profile only covers the benchmark
+  BB_BENCH_ARGS=(--reth-new-payload)
+  if [ -n "${BENCH_WAIT_TIME:-}" ]; then
+    BB_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
+  fi
+  case "$BENCH_BAL_MODE" in
+    false)
+      ;;
+    true)
+      BB_BENCH_ARGS+=(--bal)
+      ;;
+    baseline)
+      if [[ "$LABEL" == baseline* ]]; then
+        BB_BENCH_ARGS+=(--bal)
+      fi
+      ;;
+    feature)
+      if [[ "$LABEL" == feature* ]]; then
+        BB_BENCH_ARGS+=(--bal)
+      fi
+      ;;
+    *)
+      echo "::error::Unknown BENCH_BAL value: $BENCH_BAL_MODE"
+      exit 1
+      ;;
+  esac
+
+  # Warmup
+  WARMUP="${BENCH_WARMUP_BLOCKS:-50}"
+  if [ "$WARMUP" -gt 0 ] 2>/dev/null; then
+    echo "Running big blocks warmup (${WARMUP} payloads)..."
+    $BENCH_NICE "$RETH_BENCH" replay-payloads \
+      "${BB_BENCH_ARGS[@]}" \
+      --count "$WARMUP" \
+      --payload-dir "$BIG_BLOCKS_DIR/payloads" \
+      --engine-rpc-url http://127.0.0.1:8551 \
+      --jwt-secret "$DATADIR/jwt.hex" 2>&1 | sed -u "s/^/[bench] /"
+  fi
+
+  # Start tracy-capture after warmup so profile only covers the benchmark
   if [ "${BENCH_TRACY:-off}" != "off" ]; then
     echo "Starting tracy-capture..."
     tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
@@ -234,10 +304,19 @@ if [ "$BIG_BLOCKS" = "true" ]; then
     sleep 0.5  # give tracy-capture time to connect
   fi
 
-  echo "Running big blocks benchmark (replay-payloads)..."
+  # Benchmark — skip warmup payloads so they aren't measured
+  BB_SKIP=0
+  if [ "$WARMUP" -gt 0 ] 2>/dev/null; then
+    BB_SKIP="$WARMUP"
+  fi
+  if [ "${BENCH_BLOCKS:-0}" -gt 0 ] 2>/dev/null; then
+    BB_BENCH_ARGS+=(--count "$BENCH_BLOCKS")
+  fi
+
+  echo "Running big blocks benchmark (replay-payloads, skip=${BB_SKIP})..."
   $BENCH_NICE "$RETH_BENCH" replay-payloads \
-    "${EXTRA_BENCH_ARGS[@]}" \
-    --gas-ramp-dir "$BIG_BLOCKS_DIR/gas-ramp-dir" \
+    "${BB_BENCH_ARGS[@]}" \
+    --skip "$BB_SKIP" \
     --payload-dir "$BIG_BLOCKS_DIR/payloads" \
     --engine-rpc-url http://127.0.0.1:8551 \
     --jwt-secret "$DATADIR/jwt.hex" \

.github/scripts/bench-reth-snapshot.sh

@@ -1,127 +1,56 @@
 #!/usr/bin/env bash
 #
-# Downloads the latest nightly snapshot into the schelk volume with
-# progress reporting to the GitHub PR comment.
-#
-# Skips the download if the local ETag marker matches the remote one.
+# Validates that the benchmark snapshot has already been populated into the
+# local schelk volume.
 #
 # Usage: bench-reth-snapshot.sh [--check]
-#   --check   Only check if a download is needed; exits 0 if up-to-date, 1 if not.
+#   --check   Exit 0 if the local snapshot is ready, 10 if it is missing.
 #
 # Required env:
-#   SCHELK_MOUNT       – schelk mount point (e.g. /reth-bench)
-#   GITHUB_TOKEN       – token for GitHub API calls (only for download)
-#   BENCH_COMMENT_ID   – PR comment ID to update (optional)
-#   BENCH_REPO         – owner/repo (e.g. paradigmxyz/reth)
-#   BENCH_JOB_URL      – link to the Actions job
-#   BENCH_ACTOR        – user who triggered the benchmark
-#   BENCH_CONFIG       – config summary line
-set -euo pipefail
+#   SCHELK_MOUNT – schelk mount point (e.g. /reth-bench)
+# Optional env:
+#   BENCH_BIG_BLOCKS   – true when validating the big-blocks snapshot datadir
+#   BENCH_SNAPSHOT_NAME – expected snapshot label for log/error output
+set -euxo pipefail
 
-BUCKET="minio/reth-snapshots/reth-1-minimal-nightly-previous.tar.zst"
-DATADIR="$SCHELK_MOUNT/datadir"
-ETAG_FILE="$HOME/.reth-bench-snapshot-etag"
+: "${SCHELK_MOUNT:?SCHELK_MOUNT must be set}"
 
-# Get remote metadata via JSON for reliable parsing
-MC_STAT=$(mc stat --json "$BUCKET" 2>/dev/null || true)
-REMOTE_ETAG=$(echo "$MC_STAT" | jq -r '.etag // empty')
-if [ -z "$REMOTE_ETAG" ]; then
-  echo "::warning::Failed to get ETag from mc stat, will re-download"
-  REMOTE_ETAG="unknown-$(date +%s)"
+DATADIR_NAME="datadir"
+if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
+  DATADIR_NAME="datadir-big-blocks"
 fi
+DATADIR="$SCHELK_MOUNT/$DATADIR_NAME"
 
-LOCAL_ETAG=""
-[ -f "$ETAG_FILE" ] && LOCAL_ETAG=$(cat "$ETAG_FILE")
-
-if [ "$REMOTE_ETAG" = "$LOCAL_ETAG" ]; then
-  echo "Snapshot is up-to-date (ETag: ${REMOTE_ETAG})"
-  if [ "${1:-}" = "--check" ]; then
-    exit 0
+describe_snapshot() {
+  if [ -n "${BENCH_SNAPSHOT_NAME:-}" ]; then
+    printf '%s' "${BENCH_SNAPSHOT_NAME}"
+  elif [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
+    printf '%s' 'big-block weekly snapshot'
+  else
+    printf '%s' 'benchmark snapshot'
   fi
+}
+
+snapshot_ready() {
+  [ -d "$DATADIR/db" ] && [ -d "$DATADIR/static_files" ]
+}
+
+EXPECTED_SNAPSHOT="$(describe_snapshot)"
+
+sudo schelk recover -y --kill || sudo schelk full-recover -y || true
+sudo schelk mount -y || true
+
+if snapshot_ready; then
+  echo "Found local ${EXPECTED_SNAPSHOT} at ${DATADIR}"
   exit 0
 fi
 
-echo "Snapshot needs update (local: ${LOCAL_ETAG:-<none>}, remote: ${REMOTE_ETAG})"
+echo "::error::Missing local ${EXPECTED_SNAPSHOT} at ${DATADIR}. Benchmarks no longer download snapshots; pre-populate the local schelk data first."
+ls -la "$SCHELK_MOUNT" || true
+ls -la "$DATADIR" || true
+
 if [ "${1:-}" = "--check" ]; then
-  exit 1
+  exit 10
 fi
 
-# Get compressed size for progress tracking
-TOTAL_BYTES=$(echo "$MC_STAT" | jq -r '.size // empty')
-if [ -z "$TOTAL_BYTES" ] || [ "$TOTAL_BYTES" = "0" ]; then
-  echo "::error::Failed to get snapshot size from mc stat"
-  exit 1
-fi
-echo "Snapshot size: $TOTAL_BYTES bytes ($(numfmt --to=iec "$TOTAL_BYTES"))"
-
-# Prepare mount
-mountpoint -q "$SCHELK_MOUNT" && sudo schelk recover -y || true
-sudo schelk mount -y
-sudo rm -rf "$DATADIR"
-sudo mkdir -p "$DATADIR"
-
-update_comment() {
-  local pct="$1"
-  [ -z "${BENCH_COMMENT_ID:-}" ] && return 0
-  local status="Building binaries & downloading snapshot… ${pct}%"
-  local body
-  body="$(printf 'cc @%s\n\n🚀 Benchmark started! [View job](%s)\n\n⏳ **Status:** %s\n\n%s' \
-    "$BENCH_ACTOR" "$BENCH_JOB_URL" "$status" "$BENCH_CONFIG")"
-  curl -sf -X PATCH \
-    -H "Authorization: token ${GITHUB_TOKEN}" \
-    -H "Accept: application/vnd.github.v3+json" \
-    "https://api.github.com/repos/${BENCH_REPO}/issues/comments/${BENCH_COMMENT_ID}" \
-    -d "$(jq -nc --arg body "$body" '{body: $body}')" \
-    > /dev/null 2>&1 || true
-}
-
-# Track compressed bytes flowing through the pipe
-DL_BYTES_FILE=$(mktemp)
-echo 0 > "$DL_BYTES_FILE"
-
-# Start progress reporter in background
-(
-  while true; do
-    sleep 10
-    CURRENT=$(cat "$DL_BYTES_FILE" 2>/dev/null || echo 0)
-    if [ "$TOTAL_BYTES" -gt 0 ]; then
-      PCT=$(( CURRENT * 100 / TOTAL_BYTES ))
-      [ "$PCT" -gt 100 ] && PCT=100
-      echo "Snapshot download: $(numfmt --to=iec "$CURRENT") / $(numfmt --to=iec "$TOTAL_BYTES") (${PCT}%)"
-      update_comment "$PCT"
-    fi
-  done
-) &
-PROGRESS_PID=$!
-trap 'kill $PROGRESS_PID 2>/dev/null || true; rm -f "$DL_BYTES_FILE"' EXIT
-
-# Download and extract; python byte counter tracks compressed bytes received
-mc cat "$BUCKET" | python3 -c "
-import sys
-count = 0
-while True:
-    data = sys.stdin.buffer.read(1048576)
-    if not data:
-        break
-    count += len(data)
-    sys.stdout.buffer.write(data)
-    with open('$DL_BYTES_FILE', 'w') as f:
-        f.write(str(count))
-" | pzstd -d -p 6 | sudo tar -xf - -C "$DATADIR"
-
-# Stop progress reporter
-kill $PROGRESS_PID 2>/dev/null || true
-wait $PROGRESS_PID 2>/dev/null || true
-
-update_comment "100"
-echo "Snapshot download complete"
-
-# Promote the new snapshot to become the schelk baseline (virgin volume).
-# This copies changed blocks from scratch → virgin so that future
-# `schelk recover` calls restore to this new state.
-sync
-sudo schelk promote -y
-
-# Save ETag marker
-echo "$REMOTE_ETAG" > "$ETAG_FILE"
-echo "Snapshot promoted to schelk baseline (ETag: ${REMOTE_ETAG})"
+exit 1

.github/scripts/bench-reth-summary.py

@@ -111,6 +111,14 @@ def compute_stats(combined: list[dict]) -> dict:
     wall_clock_s = sum(total_latencies_ms) / 1_000
     mean_total_lat_ms = sum(total_latencies_ms) / n
 
+    # Persistence wait mean (for main table)
+    persist_values_ms = []
+    for r in combined:
+        v = r.get("persistence_wait_us")
+        if v is not None:
+            persist_values_ms.append(v / 1_000)
+    mean_persist_ms = sum(persist_values_ms) / len(persist_values_ms) if persist_values_ms else 0.0
+
     return {
         "n": n,
         "mean_ms": mean_lat,
@@ -121,6 +129,7 @@ def compute_stats(combined: list[dict]) -> dict:
         "mean_mgas_s": mean_mgas_s,
         "wall_clock_s": wall_clock_s,
         "mean_total_lat_ms": mean_total_lat_ms,
+        "mean_persist_ms": mean_persist_ms,
     }
 
 
@@ -145,7 +154,7 @@ def compute_wait_stats(combined: list[dict], field: str) -> dict:
 
 def _paired_data(
     baseline: list[dict], feature: list[dict]
-) -> tuple[list[tuple[float, float]], list[float], list[float], list[float]]:
+) -> tuple[list[tuple[float, float]], list[float], list[float], list[float], list[float]]:
     """Match blocks and return paired latencies and per-block diffs.
 
     Returns:
@@ -153,6 +162,7 @@ def _paired_data(
         lat_diffs_ms: list of feature − baseline latency diffs in ms
         mgas_diffs: list of feature − baseline Mgas/s diffs
         total_lat_diffs_ms: list of feature − baseline total latency diffs in ms
+        persist_diffs_ms: list of feature − baseline persistence wait diffs in ms
     """
     baseline_by_block = {r["block_number"]: r for r in baseline}
     feature_by_block = {r["block_number"]: r for r in feature}
@@ -162,6 +172,7 @@ def _paired_data(
     lat_diffs_ms = []
     mgas_diffs = []
     total_lat_diffs_ms = []
+    persist_diffs_ms = []
     for bn in common_blocks:
         b = baseline_by_block[bn]
         f = feature_by_block[bn]
@@ -179,7 +190,10 @@ def _paired_data(
         total_lat_diffs_ms.append(
             f["total_latency_us"] / 1_000 - b["total_latency_us"] / 1_000
         )
-    return pairs, lat_diffs_ms, mgas_diffs, total_lat_diffs_ms
+        b_persist = (b.get("persistence_wait_us") or 0) / 1_000
+        f_persist = (f.get("persistence_wait_us") or 0) / 1_000
+        persist_diffs_ms.append(f_persist - b_persist)
+    return pairs, lat_diffs_ms, mgas_diffs, total_lat_diffs_ms, persist_diffs_ms
 
 
 def compute_paired_stats(
@@ -195,13 +209,15 @@ def compute_paired_stats(
     all_lat_diffs = []
     all_mgas_diffs = []
     all_total_lat_diffs = []
+    all_persist_diffs = []
     blocks_per_pair = []
     for baseline, feature in zip(baseline_runs, feature_runs):
-        pairs, lat_diffs, mgas_diffs, total_lat_diffs = _paired_data(baseline, feature)
+        pairs, lat_diffs, mgas_diffs, total_lat_diffs, persist_diffs = _paired_data(baseline, feature)
         all_pairs.extend(pairs)
         all_lat_diffs.extend(lat_diffs)
         all_mgas_diffs.extend(mgas_diffs)
         all_total_lat_diffs.extend(total_lat_diffs)
+        all_persist_diffs.extend(persist_diffs)
         blocks_per_pair.append(len(pairs))
 
     if not all_lat_diffs:
@@ -245,6 +261,11 @@ def compute_paired_stats(
     total_se = std_total_diff / math.sqrt(len(all_total_lat_diffs)) if all_total_lat_diffs else 0.0
     wall_clock_ci_ms = T_CRITICAL * total_se
 
+    mean_persist_diff = sum(all_persist_diffs) / len(all_persist_diffs) if all_persist_diffs else 0.0
+    std_persist_diff = stddev(all_persist_diffs, mean_persist_diff) if len(all_persist_diffs) > 1 else 0.0
+    persist_se = std_persist_diff / math.sqrt(len(all_persist_diffs)) if all_persist_diffs else 0.0
+    persist_ci_ms = T_CRITICAL * persist_se
+
     return {
         "n": n,
         "mean_diff_ms": mean_diff,
@@ -258,6 +279,7 @@ def compute_paired_stats(
         "mean_mgas_diff": mean_mgas_diff,
         "mgas_ci": mgas_ci,
         "wall_clock_ci_ms": wall_clock_ci_ms,
+        "persist_ci_ms": persist_ci_ms,
         "blocks": max(blocks_per_pair),
     }
 
@@ -290,6 +312,14 @@ def fmt_s(v: float) -> str:
     return f"{v:.2f}s"
 
 
+def display_bal_mode(bal_mode: str | None) -> str | None:
+    if not bal_mode or bal_mode == "false":
+        return None
+    if bal_mode == "both":
+        return "true"
+    return bal_mode
+
+
 def significance(pct: float, ci_pct: float, lower_is_better: bool) -> str:
     """Return significance label: 'good', 'bad', or 'neutral'."""
     significant = abs(pct) > ci_pct
@@ -328,6 +358,7 @@ def compute_changes(
         ("p99", "p99_ms", "p99_ci_ms", "p99_ms", True),
         ("mgas_s", "mean_mgas_s", "mgas_ci", "mean_mgas_s", False),
         ("wall_clock", "wall_clock_s", "wall_clock_ci_ms", "mean_total_lat_ms", True),
+        ("persist_wait", "mean_persist_ms", "persist_ci_ms", "mean_persist_ms", True),
     ]
     changes = {}
     for name, stat_key, ci_key, base_key, lower_is_better in metrics:
@@ -351,6 +382,9 @@ def generate_comparison_table(
     feature_name: str,
     feature_sha: str,
     big_blocks: bool = False,
+    warmup_blocks: str | None = None,
+    wait_time: str | None = None,
+    bal_mode: str | None = None,
 ) -> str:
     """Generate a markdown comparison table between baseline and feature."""
     n = paired["blocks"]
@@ -366,6 +400,8 @@ def generate_comparison_table(
     p90_pct = pct(run1["p90_ms"], run2["p90_ms"])
     p99_pct = pct(run1["p99_ms"], run2["p99_ms"])
 
+    persist_pct = pct(run1["mean_persist_ms"], run2["mean_persist_ms"])
+
     # Bootstrap CIs as % of baseline percentile
     p50_ci_pct = paired["p50_ci_ms"] / run1["p50_ms"] * 100.0 if run1["p50_ms"] > 0 else 0.0
     p90_ci_pct = paired["p90_ci_ms"] / run1["p90_ms"] * 100.0 if run1["p90_ms"] > 0 else 0.0
@@ -375,6 +411,7 @@ def generate_comparison_table(
     lat_ci_pct = paired["ci_ms"] / run1["mean_ms"] * 100.0 if run1["mean_ms"] > 0 else 0.0
     mgas_ci_pct = paired["mgas_ci"] / run1["mean_mgas_s"] * 100.0 if run1["mean_mgas_s"] > 0 else 0.0
     wall_ci_pct = paired["wall_clock_ci_ms"] / run1["mean_total_lat_ms"] * 100.0 if run1["mean_total_lat_ms"] > 0 else 0.0
+    persist_ci_pct = paired["persist_ci_ms"] / run1["mean_persist_ms"] * 100.0 if run1["mean_persist_ms"] > 0 else 0.0
 
     base_url = f"https://github.com/{repo}/commit"
     baseline_label = f"[`{baseline_name}`]({base_url}/{baseline_ref})"
@@ -390,9 +427,18 @@ def generate_comparison_table(
         f"| P99 | {fmt_ms(run1['p99_ms'])} | {fmt_ms(run2['p99_ms'])} | {change_str(p99_pct, p99_ci_pct, lower_is_better=True)} |",
         f"| Mgas/s | {fmt_mgas(run1['mean_mgas_s'])} | {fmt_mgas(run2['mean_mgas_s'])} | {change_str(gas_pct, mgas_ci_pct, lower_is_better=False)} |",
         f"| Wall Clock | {fmt_s(run1['wall_clock_s'])} | {fmt_s(run2['wall_clock_s'])} | {change_str(wall_pct, wall_ci_pct, lower_is_better=True)} |",
+        f"| Persist Wait | {fmt_ms(run1['mean_persist_ms'])} | {fmt_ms(run2['mean_persist_ms'])} | {change_str(persist_pct, persist_ci_pct, lower_is_better=True)} |",
         "",
-        f"*{n} {'big blocks' if big_blocks else 'blocks'}*",
     ]
+    meta_parts = [f"{n} {'big blocks' if big_blocks else 'blocks'}"]
+    if warmup_blocks:
+        meta_parts.append(f"{warmup_blocks} warmup")
+    if wait_time:
+        meta_parts.append(f"wait time: {wait_time}")
+    display_mode = display_bal_mode(bal_mode)
+    if big_blocks and display_mode:
+        meta_parts.append(f"BAL: {display_mode}")
+    lines.append(f"*{', '.join(meta_parts)}*")
     return "\n".join(lines)
 
 
@@ -472,7 +518,9 @@ def main():
     parser.add_argument("--feature-ref", "--branch-sha", "--feature-sha", default=None, help="Feature commit SHA")
     parser.add_argument("--behind-baseline", "--behind-main", type=int, default=0, help="Commits behind baseline")
     parser.add_argument("--big-blocks", action="store_true", default=False, help="Big blocks mode")
-    parser.add_argument("--gas-ramp-blocks", type=int, default=0, help="Number of gas ramp blocks (big blocks mode)")
+    parser.add_argument("--warmup-blocks", default=None, help="Number of warmup blocks")
+    parser.add_argument("--wait-time", default=None, help="Wait time interval used between blocks")
+    parser.add_argument("--bal-mode", default=None, help="BAL mode (true, feature, baseline)")
     parser.add_argument("--grafana-url", default=None, help="Grafana dashboard URL for this benchmark run")
     args = parser.parse_args()
 
@@ -512,6 +560,7 @@ def main():
     baseline_name = args.baseline_name or "baseline"
     feature_name = args.feature_name or "feature"
     feature_sha = args.feature_ref or "unknown"
+    bal_mode = display_bal_mode(args.bal_mode)
 
     comparison_table = generate_comparison_table(
         baseline_stats,
@@ -523,6 +572,9 @@ def main():
         feature_name=feature_name,
         feature_sha=feature_sha,
         big_blocks=args.big_blocks,
+        warmup_blocks=args.warmup_blocks,
+        wait_time=args.wait_time,
+        bal_mode=bal_mode,
     )
     print(f"Generated comparison ({paired_stats['n']} paired blocks, "
           f"mean diff {paired_stats['mean_diff_ms']:+.3f}ms ± {paired_stats['ci_ms']:.3f}ms)")
@@ -554,7 +606,9 @@ def main():
     summary = {
         "blocks": paired_stats["blocks"],
         "big_blocks": args.big_blocks,
-        "gas_ramp_blocks": args.gas_ramp_blocks,
+        "warmup_blocks": args.warmup_blocks,
+        "wait_time": args.wait_time,
+        "bal_mode": bal_mode,
         "baseline": {
             "name": baseline_name,
             "ref": baseline_ref,

.github/scripts/bench-scheduled-refs.sh

@@ -0,0 +1,336 @@
+#!/usr/bin/env bash
+#
+# Resolves baseline and feature refs for scheduled benchmark runs.
+#
+# Supports three modes:
+#   nightly — Queries the latest successful scheduled docker.yml run via
+#             GitHub API to find the nightly Docker image commit. Compares
+#             with the last successful feature ref to detect staleness.
+#   hourly  — Compares origin/main HEAD against the last successfully
+#             benchmarked commit (falls back to HEAD~1 on first run).
+#             Checks for in-progress sibling runs to avoid overlap.
+#   release — Compares the latest GitHub release tag against the current
+#             nightly Docker build. Baseline is the release tag commit,
+#             feature is the nightly commit.
+#
+# Usage: bench-scheduled-refs.sh <force> <mode>
+#   force — "true" to run even if no new commit (bypass skip logic)
+#   mode  — "nightly", "hourly", or "release"
+#
+# Outputs (via GITHUB_OUTPUT):
+#   baseline-ref    — commit SHA for baseline
+#   feature-ref     — commit SHA for feature
+#   should-skip     — "true" if no new commit since last run or sibling in progress
+#   is-stale        — "true" if latest nightly build is >24h old (nightly only)
+#   stale-age-hours — age of the nightly build in hours (nightly only)
+#   nightly-created — ISO timestamp of the nightly build (nightly only)
+#   release-tag     — release tag name (release mode only, e.g. "v2.0.0")
+#
+# Reads:
+#   state/nightly-last-feature-ref  (nightly, from decofe/reth-bench-charts repo)
+#   state/hourly-last-feature-ref   (hourly, from decofe/reth-bench-charts repo)
+#   state/release-last-feature-ref  (release, from decofe/reth-bench-charts repo)
+#
+# Requires: gh (GitHub CLI), jq, date, git (hourly mode), curl, DEREK_TOKEN env
+set -euxo pipefail
+
+FORCE="${1:-false}"
+MODE="${2:-nightly}"
+REPO="${GITHUB_REPOSITORY:-paradigmxyz/reth}"
+
+echo "Mode: $MODE, Force: $FORCE"
+
+# ==========================================================================
+# Hourly mode: compare origin/main HEAD vs HEAD~1
+# ==========================================================================
+if [ "$MODE" = "hourly" ]; then
+
+  # --- Step 1: Resolve feature ref from git ---
+  echo "::group::Resolving hourly refs from git"
+  git fetch origin main --depth=2 --quiet
+  FEATURE_REF=$(git rev-parse origin/main)
+  echo "Feature (HEAD): $FEATURE_REF"
+  echo "::endgroup::"
+
+  # --- Step 2: Check for in-progress sibling runs ---
+  echo "::group::Checking for in-progress sibling runs"
+  CURRENT_RUN_ID="${GITHUB_RUN_ID:-0}"
+  IN_PROGRESS=$(gh run list \
+    -R "$REPO" \
+    --workflow=bench-scheduled.yml \
+    --status=in_progress \
+    --json databaseId \
+    --jq "[.[] | select(.databaseId != $CURRENT_RUN_ID)] | length")
+
+  SHOULD_SKIP="false"
+  if [ "$IN_PROGRESS" -gt 0 ]; then
+    echo "::warning::Previous bench run still in progress ($IN_PROGRESS sibling run(s) found). Skipping."
+    SHOULD_SKIP="true"
+    # Output a flag so the workflow can send a Slack alert
+    echo "long-running=true" >> "$GITHUB_OUTPUT"
+  else
+    echo "No in-progress sibling runs"
+    echo "long-running=false" >> "$GITHUB_OUTPUT"
+  fi
+  echo "::endgroup::"
+
+  # --- Step 3: Read last successful feature ref from charts repo ---
+  echo "::group::Reading persisted state"
+  LAST_FEATURE_REF=""
+  STATE_URL="https://raw.githubusercontent.com/decofe/reth-bench-charts/state/state/hourly-last-feature-ref"
+  if RAW=$(curl -sfL -H "Authorization: token ${DEREK_TOKEN}" "$STATE_URL"); then
+    LAST_FEATURE_REF=$(echo "$RAW" | tr -d '[:space:]')
+    echo "Previous feature ref: $LAST_FEATURE_REF"
+  else
+    echo "No persisted state found (first run)"
+  fi
+  echo "::endgroup::"
+
+  # --- Step 4: Determine baseline and skip logic ---
+  echo "::group::Resolving baseline and skip logic"
+  if [ "$SHOULD_SKIP" = "true" ]; then
+    BASELINE_REF=$(git rev-parse origin/main~1)
+    echo "Already marked skip (sibling in progress)"
+  elif [ -z "$LAST_FEATURE_REF" ]; then
+    # First run: no previous state, fall back to HEAD~1
+    BASELINE_REF=$(git rev-parse origin/main~1)
+    echo "First run — using HEAD~1 as baseline"
+  elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+    BASELINE_REF="$LAST_FEATURE_REF"
+    if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+      echo "No new commits on main, but force=true — running anyway"
+    else
+      SHOULD_SKIP="true"
+      echo "No new commits on main since last run — will skip"
+    fi
+  else
+    # Normal case: use last benchmarked commit as baseline
+    BASELINE_REF="$LAST_FEATURE_REF"
+    echo "New commit(s) on main detected — comparing against last benchmarked commit"
+  fi
+
+  echo "Baseline: $BASELINE_REF"
+  echo "Feature:  $FEATURE_REF"
+  echo "Skip:     $SHOULD_SKIP"
+  echo "::endgroup::"
+
+  # --- Step 5: Write outputs ---
+  {
+    echo "baseline-ref=$BASELINE_REF"
+    echo "feature-ref=$FEATURE_REF"
+    echo "should-skip=$SHOULD_SKIP"
+    echo "is-stale=false"
+    echo "stale-age-hours=0"
+    echo "nightly-created="
+  } >> "$GITHUB_OUTPUT"
+  exit 0
+fi
+
+# ==========================================================================
+# Release mode: compare latest GitHub release tag vs current nightly build
+# ==========================================================================
+if [ "$MODE" = "release" ]; then
+
+  # --- Step 1: Resolve feature ref from latest nightly Docker build ---
+  echo "::group::Querying latest nightly docker build"
+  RUNS_JSON=$(gh run list \
+    -R "$REPO" \
+    --workflow=docker.yml \
+    --event=schedule \
+    --status=completed \
+    --limit 5 \
+    --json headSha,createdAt,conclusion)
+
+  LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
+  if [ -z "$LATEST" ]; then
+    echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
+    exit 1
+  fi
+
+  FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
+  echo "Nightly commit (feature): $FEATURE_REF"
+  echo "::endgroup::"
+
+  # --- Step 2: Resolve baseline ref from latest GitHub release ---
+  echo "::group::Resolving latest release tag"
+  RELEASE_JSON=$(gh release view --repo "$REPO" --json tagName,targetCommitish,publishedAt 2>/dev/null || echo "{}")
+  RELEASE_TAG=$(echo "$RELEASE_JSON" | jq -r '.tagName // empty')
+
+  if [ -z "$RELEASE_TAG" ]; then
+    echo "::error::No release found on $REPO"
+    exit 1
+  fi
+
+  # Resolve the tag to a commit SHA
+  BASELINE_REF=$(gh api "repos/$REPO/git/ref/tags/$RELEASE_TAG" --jq '.object.sha' 2>/dev/null || true)
+
+  # If tag points to an annotated tag object, dereference to the commit
+  if [ -n "$BASELINE_REF" ]; then
+    OBJ_TYPE=$(gh api "repos/$REPO/git/tags/$BASELINE_REF" --jq '.object.type' 2>/dev/null || echo "commit")
+    if [ "$OBJ_TYPE" = "commit" ]; then
+      BASELINE_REF=$(gh api "repos/$REPO/git/tags/$BASELINE_REF" --jq '.object.sha' 2>/dev/null || echo "$BASELINE_REF")
+    fi
+  fi
+
+  if [ -z "$BASELINE_REF" ]; then
+    echo "::error::Could not resolve release tag $RELEASE_TAG to a commit"
+    exit 1
+  fi
+
+  echo "Release tag: $RELEASE_TAG"
+  echo "Release commit (baseline): $BASELINE_REF"
+  echo "::endgroup::"
+
+  # --- Step 3: Read last successful feature ref from charts repo ---
+  echo "::group::Reading persisted state"
+  LAST_FEATURE_REF=""
+  STATE_URL="https://raw.githubusercontent.com/decofe/reth-bench-charts/state/state/release-last-feature-ref"
+  if RAW=$(curl -sfL -H "Authorization: token ${DEREK_TOKEN}" "$STATE_URL"); then
+    LAST_FEATURE_REF=$(echo "$RAW" | tr -d '[:space:]')
+    echo "Previous feature ref: $LAST_FEATURE_REF"
+  else
+    echo "No persisted state found (first run)"
+  fi
+  echo "::endgroup::"
+
+  # --- Step 4: Skip logic ---
+  echo "::group::Resolving skip logic"
+  SHOULD_SKIP="false"
+  if [ -n "$LAST_FEATURE_REF" ] && [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+    if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+      echo "No new nightly, but force=true — running anyway"
+    else
+      SHOULD_SKIP="true"
+      echo "No new nightly since last release regression run — will skip"
+    fi
+  else
+    echo "New nightly detected or first run"
+  fi
+
+  echo "Baseline: $BASELINE_REF ($RELEASE_TAG)"
+  echo "Feature:  $FEATURE_REF"
+  echo "Skip:     $SHOULD_SKIP"
+  echo "::endgroup::"
+
+  # --- Step 5: Write outputs ---
+  {
+    echo "baseline-ref=$BASELINE_REF"
+    echo "feature-ref=$FEATURE_REF"
+    echo "should-skip=$SHOULD_SKIP"
+    echo "is-stale=false"
+    echo "stale-age-hours=0"
+    echo "nightly-created="
+    echo "long-running=false"
+    echo "release-tag=$RELEASE_TAG"
+  } >> "$GITHUB_OUTPUT"
+  exit 0
+fi
+
+# ==========================================================================
+# Nightly mode: query latest Docker nightly build (original logic)
+# ==========================================================================
+
+# --- Step 1: Query latest successful scheduled docker.yml run ---
+echo "::group::Querying latest nightly docker build"
+
+RUNS_JSON=$(gh run list \
+  -R "$REPO" \
+  --workflow=docker.yml \
+  --event=schedule \
+  --status=completed \
+  --limit 5 \
+  --json headSha,createdAt,conclusion)
+
+# Find the most recent successful run
+LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
+
+if [ -z "$LATEST" ]; then
+  echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
+  echo "Runs found: $RUNS_JSON"
+  exit 1
+fi
+
+FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
+CREATED_AT=$(echo "$LATEST" | jq -r '.createdAt')
+echo "Latest nightly commit: $FEATURE_REF"
+echo "Built at: $CREATED_AT"
+echo "::endgroup::"
+
+# --- Step 2: Staleness check ---
+echo "::group::Checking staleness"
+NOW_EPOCH=$(date +%s)
+# Handle both GNU date (-d) and BSD date (-j -f) for cross-platform compat
+CREATED_EPOCH=$(date -d "$CREATED_AT" +%s 2>/dev/null || \
+  date -j -f "%Y-%m-%dT%H:%M:%SZ" "$CREATED_AT" +%s 2>/dev/null || \
+  date -j -f "%Y-%m-%dT%T%z" "$CREATED_AT" +%s 2>/dev/null || \
+  { echo "::error::Cannot parse date: $CREATED_AT"; exit 1; })
+
+AGE_SECONDS=$(( NOW_EPOCH - CREATED_EPOCH ))
+AGE_HOURS=$(( AGE_SECONDS / 3600 ))
+IS_STALE="false"
+
+if [ "$AGE_HOURS" -gt 24 ]; then
+  IS_STALE="true"
+  echo "::warning::STALE NIGHTLY: Build is ${AGE_HOURS}h old (>24h threshold)"
+  echo "This indicates the nightly docker build failed — no new image was produced"
+else
+  echo "Nightly build age: ${AGE_HOURS}h (within 24h threshold)"
+fi
+echo "::endgroup::"
+
+# --- Step 3: Read last successful feature ref from charts repo ---
+echo "::group::Reading persisted state"
+LAST_FEATURE_REF=""
+STATE_URL="https://raw.githubusercontent.com/decofe/reth-bench-charts/state/state/nightly-last-feature-ref"
+if RAW=$(curl -sfL -H "Authorization: token ${DEREK_TOKEN}" "$STATE_URL"); then
+  LAST_FEATURE_REF=$(echo "$RAW" | tr -d '[:space:]')
+  echo "Previous feature ref: $LAST_FEATURE_REF"
+else
+  echo "No persisted state found (first run)"
+fi
+echo "::endgroup::"
+
+# --- Step 4: Determine baseline and skip logic ---
+echo "::group::Resolving refs"
+SHOULD_SKIP="false"
+BASELINE_REF="$FEATURE_REF"  # default for first run
+
+if [ "$IS_STALE" = "true" ]; then
+  # Stale = error path, don't skip (will alert and fail downstream)
+  SHOULD_SKIP="false"
+  BASELINE_REF="${LAST_FEATURE_REF:-$FEATURE_REF}"
+  echo "Stale nightly detected — will alert and fail"
+elif [ -z "$LAST_FEATURE_REF" ]; then
+  # First run: baseline = feature (self-comparison to establish baseline)
+  BASELINE_REF="$FEATURE_REF"
+  echo "First run — will benchmark nightly against itself to establish baseline"
+elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+  # No new nightly since last successful run
+  if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+    echo "No new nightly, but force=true — running anyway"
+    BASELINE_REF="$LAST_FEATURE_REF"
+  else
+    SHOULD_SKIP="true"
+    echo "No new nightly since last run — will skip"
+  fi
+else
+  # Normal case: new nightly available
+  BASELINE_REF="$LAST_FEATURE_REF"
+  echo "New nightly detected"
+fi
+
+echo "Baseline: $BASELINE_REF"
+echo "Feature:  $FEATURE_REF"
+echo "Skip:     $SHOULD_SKIP"
+echo "Stale:    $IS_STALE"
+echo "::endgroup::"
+
+# --- Step 5: Write outputs ---
+{
+  echo "baseline-ref=$BASELINE_REF"
+  echo "feature-ref=$FEATURE_REF"
+  echo "should-skip=$SHOULD_SKIP"
+  echo "is-stale=$IS_STALE"
+  echo "stale-age-hours=$AGE_HOURS"
+  echo "nightly-created=$CREATED_AT"
+} >> "$GITHUB_OUTPUT"

.github/scripts/bench-slack-notify.js

@@ -18,6 +18,7 @@
 
 const fs = require('fs');
 const path = require('path');
+const { fmtChange, fmtMs, verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
 
 const SLACK_API = 'https://slack.com/api/chat.postMessage';
 
@@ -61,41 +62,17 @@ function cell(text) {
   return { type: 'raw_text', text: s || ' ' };
 }
 
+// Slack shortcodes for verdict (Block Kit header doesn't support unicode emoji)
+const SLACK_VERDICT = {
+  '⚠️': ':warning:',
+  '❌': ':x:',
+  '✅': ':white_check_mark:',
+  '⚪': ':white_circle:',
+};
+
 function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, repo, samplyUrls }) {
-  const b = summary.baseline.stats;
-  const f = summary.feature.stats;
-  const c = summary.changes;
-
-  const sigEmoji = { good: '\u2705', bad: '\u274c', neutral: '\u26aa' };
-
-  function fmtMs(v) { return v.toFixed(2) + 'ms'; }
-  function fmtMgas(v) { return v.toFixed(2); }
-  function fmtS(v) { return v.toFixed(2) + 's'; }
-  function fmtChange(ch) {
-    if (!ch.pct && !ch.ci_pct) return ' ';
-    const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
-    const ciStr = ch.ci_pct ? ` (\u00b1${ch.ci_pct.toFixed(2)}%)` : '';
-    return `${pctStr}${ciStr} ${sigEmoji[ch.sig]}`;
-  }
-
-  // Overall result for header
-  const vals = Object.values(c);
-  const hasBad = vals.some(v => v.sig === 'bad');
-  const hasGood = vals.some(v => v.sig === 'good');
-  let headerEmoji, headerResult;
-  if (hasBad && hasGood) {
-    headerEmoji = ':warning:';
-    headerResult = 'Mixed Results';
-  } else if (hasBad) {
-    headerEmoji = ':x:';
-    headerResult = 'Regression';
-  } else if (hasGood) {
-    headerEmoji = ':white_check_mark:';
-    headerResult = 'Improvement';
-  } else {
-    headerEmoji = ':white_circle:';
-    headerResult = 'No Difference';
-  }
+  const { emoji, label } = verdict(summary.changes);
+  const headerEmoji = SLACK_VERDICT[emoji] || emoji;
 
   const prUrl = prNumber ? `https://github.com/${repo}/pull/${prNumber}` : '';
   const commitUrl = `https://github.com/${repo}/commit`;
@@ -120,19 +97,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
   if (fl1) featureLine += ` | <${fl1}|Samply 1>`;
   if (fl2) featureLine += ` | <${fl2}|Samply 2>`;
 
-  const cores = process.env.BENCH_CORES || '0';
-  const countsParts = [];
-  if (summary.big_blocks) {
-    const gasRamp = summary.gas_ramp_blocks || 0;
-    if (gasRamp > 0) countsParts.push(`*Gas Ramp:* ${gasRamp}`);
-    countsParts.push(`*Big Blocks:* ${summary.blocks}`);
-  } else {
-    const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
-    if (warmup) countsParts.push(`*Warmup:* ${warmup}`);
-    countsParts.push(`*Blocks:* ${summary.blocks}`);
-  }
-  if (cores !== '0') countsParts.push(`*Cores:* ${cores}`);
-  const countsLine = countsParts.join(' | ');
+  const countsLine = blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | ');
 
   const baselineArgs = process.env.BENCH_BASELINE_ARGS || '';
   const featureArgs = process.env.BENCH_FEATURE_ARGS || '';
@@ -159,10 +124,17 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
     },
   ];
 
+  // Build table rows from shared metricRows
+  const rows = metricRows(summary);
+  const tableRows = [
+    [cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
+    ...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
+  ];
+
   const blocks = [
     {
       type: 'header',
-      text: { type: 'plain_text', text: `${headerEmoji} ${headerResult}`, emoji: true },
+      text: { type: 'plain_text', text: `${headerEmoji} ${label}`, emoji: true },
     },
     {
       type: 'section',
@@ -176,16 +148,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
         { align: 'right' },
         { align: 'right' },
       ],
-      rows: [
-        [cell('Metric'),  cell('Baseline'), cell('Feature'), cell('Change')],
-        [cell('Mean'),     cell(fmtMs(b.mean_ms)),      cell(fmtMs(f.mean_ms)),      cell(fmtChange(c.mean))],
-        [cell('StdDev'),   cell(fmtMs(b.stddev_ms)),    cell(fmtMs(f.stddev_ms)),    cell(' ')],
-        [cell('P50'),      cell(fmtMs(b.p50_ms)),       cell(fmtMs(f.p50_ms)),       cell(fmtChange(c.p50))],
-        [cell('P90'),      cell(fmtMs(b.p90_ms)),       cell(fmtMs(f.p90_ms)),       cell(fmtChange(c.p90))],
-        [cell('P99'),      cell(fmtMs(b.p99_ms)),       cell(fmtMs(f.p99_ms)),       cell(fmtChange(c.p99))],
-        [cell('Mgas/s'),   cell(fmtMgas(b.mean_mgas_s)), cell(fmtMgas(f.mean_mgas_s)), cell(fmtChange(c.mgas_s))],
-        [cell('Wall Clock'), cell(fmtS(b.wall_clock_s)), cell(fmtS(f.wall_clock_s)), cell(fmtChange(c.wall_clock))],
-      ],
+      rows: tableRows,
     },
     {
       type: 'actions',
@@ -195,16 +158,12 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
 
   // Wait times as a separate table block (sent as threaded reply due to Slack one-table limit)
   const threadBlocks = [];
-  const waitTimes = summary.wait_times || {};
-  const waitKeys = Object.keys(waitTimes);
-  if (waitKeys.length > 0) {
-    const waitRows = [
+  const wtRows = waitTimeRows(summary);
+  if (wtRows.length > 0) {
+    const waitTableRows = [
       [cell('Wait Time'), cell('Baseline'), cell('Feature')],
+      ...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
     ];
-    for (const key of waitKeys) {
-      const wt = waitTimes[key];
-      waitRows.push([cell(wt.title), cell(fmtMs(wt.baseline.mean_ms)), cell(fmtMs(wt.feature.mean_ms))]);
-    }
     threadBlocks.push({
       type: 'table',
       column_settings: [
@@ -212,7 +171,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
         { align: 'right' },
         { align: 'right' },
       ],
-      rows: waitRows,
+      rows: waitTableRows,
     });
   }
 
@@ -274,16 +233,7 @@ async function success({ core, context }) {
   const jobUrl = process.env.BENCH_JOB_URL ||
     `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
 
-  // Load samply profile URLs (files exist when samply profiling was enabled)
-  const samplyUrls = {};
-  for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
-    try {
-      const url = fs.readFileSync(
-        path.join(process.env.BENCH_WORK_DIR, run, 'samply-profile-url.txt'), 'utf8'
-      ).trim();
-      if (url) samplyUrls[run] = url;
-    } catch {}
-  }
+  const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
 
   const slackUsers = loadSlackUsers(process.env.GITHUB_WORKSPACE || '.');
   const actorSlackId = slackUsers[actor];
@@ -300,6 +250,8 @@ async function success({ core, context }) {
     }
   }
 
+  const slackMode = process.env.BENCH_SLACK || 'always';
+
   // Post to public channel if any metric shows significant improvement or regression
   const channel = process.env.SLACK_BENCH_CHANNEL;
   let postedToChannel = false;
@@ -314,6 +266,14 @@ async function success({ core, context }) {
     }
   }
 
+  // In on-win mode, only notify on improvement — skip DM fallback entirely
+  if (slackMode === 'on-win') {
+    if (!postedToChannel) {
+      core.info('on-win mode: no improvement detected, skipping all notifications');
+    }
+    return;
+  }
+
   // DM the actor only when results were not posted to the public channel
   if (!postedToChannel) {
     if (actorSlackId) {

.github/scripts/bench-slack-users.json

@@ -20,5 +20,6 @@
   "SuperFluffy": "U095BKHB2Q4",
   "kamsz": "U0A2563UBRD",
   "zerosnacks": "U09FARPMN74",
-  "samczsun": "U096R14E4H3"
+  "samczsun": "U096R14E4H3",
+  "laibe": "U09FARE0B9Q"
 }

.github/scripts/bench-upload-clickhouse.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""Upload bench-scheduled summary.json results to ClickHouse.
+
+Reads the summary JSON produced by bench-reth-summary.py and inserts a row
+into the bench_dual_comparisons table so the PM dashboard can display results.
+
+Usage:
+    bench-upload-clickhouse.py \
+        --summary <summary.json> \
+        --workflow-name <name> \
+        --chain <chain>
+
+Environment variables:
+    CLICKHOUSE_HOST      ClickHouse host URL
+    CLICKHOUSE_USER      ClickHouse username
+    CLICKHOUSE_PASSWORD  ClickHouse password
+    CLICKHOUSE_DATABASE  ClickHouse database (default: "default")
+"""
+
+import argparse
+import json
+import os
+import sys
+import urllib.request
+import urllib.error
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Upload benchmark results to ClickHouse")
+    parser.add_argument("--summary", required=True, help="Path to summary.json")
+    parser.add_argument("--workflow-name", required=True, help="Workflow name for ClickHouse")
+    parser.add_argument("--chain", default="mainnet", help="Chain name")
+    parser.add_argument("--grafana-url", default="", help="Grafana dashboard URL")
+    parser.add_argument("--github-diff-url", default="", help="GitHub diff URL")
+    parser.add_argument("--job-url", default="", help="CI job URL")
+    args = parser.parse_args()
+
+    ch_host = os.environ.get("CLICKHOUSE_HOST", "")
+    ch_user = os.environ.get("CLICKHOUSE_USER", "")
+    ch_password = os.environ.get("CLICKHOUSE_PASSWORD", "")
+    ch_database = os.environ.get("CLICKHOUSE_DATABASE", "default")
+    ch_table = "bench_dual_comparisons"
+
+    if not ch_host or not ch_user or not ch_password:
+        print("Missing ClickHouse credentials, skipping upload", file=sys.stderr)
+        sys.exit(0)
+
+    with open(args.summary) as f:
+        summary = json.load(f)
+
+    baseline = summary["baseline"]
+    feature = summary["feature"]
+    b_stats = baseline["stats"]
+    f_stats = feature["stats"]
+    changes = summary["changes"]
+    blocks = summary["blocks"]
+
+    # Extract wait time data
+    wait_times = summary.get("wait_times", {})
+    def wait_mean(field):
+        wt = wait_times.get(field, {})
+        b = wt.get("baseline", {}).get("mean_ms", 0.0)
+        f = wt.get("feature", {}).get("mean_ms", 0.0)
+        return b, f
+
+    b_persist, f_persist = wait_mean("persistence_wait_us")
+    b_exec_cache, f_exec_cache = wait_mean("execution_cache_wait_us")
+    b_sparse, f_sparse = wait_mean("sparse_trie_wait_us")
+
+    # gas_per_second: summary uses mean_mgas_s (Mgas/s), ClickHouse stores gas/s
+    b_gas_per_second = b_stats["mean_mgas_s"] * 1_000_000
+    f_gas_per_second = f_stats["mean_mgas_s"] * 1_000_000
+
+    mean_change = changes.get("mean", {}).get("pct", 0.0)
+    gas_change = changes.get("mgas_s", {}).get("pct", 0.0)
+    latency_improved = 1 if mean_change < 0 else 0
+    throughput_improved = 1 if gas_change > 0 else 0
+
+    big_blocks = "true" if summary.get("big_blocks", False) else "false"
+    warmup_blocks = summary.get("warmup_blocks", 0) or 0
+
+    def esc(s):
+        return str(s).replace("'", "\\'")
+
+    insert = f"""
+    INSERT INTO {ch_database}.{ch_table} (
+        workflow_name, chain,
+        baseline_ref, baseline_commit,
+        feature_ref, feature_commit,
+        blocks,
+        baseline_total_latency_ms, baseline_gas_per_second,
+        baseline_latency_mean_ms, baseline_latency_median_ms,
+        baseline_latency_p90_ms, baseline_latency_p99_ms,
+        feature_total_latency_ms, feature_gas_per_second,
+        feature_latency_mean_ms, feature_latency_median_ms,
+        feature_latency_p90_ms, feature_latency_p99_ms,
+        mean_latency_change_percent, gas_per_second_change_percent,
+        latency_improved, throughput_improved,
+        warmup_blocks, big_blocks,
+        grafana_benchmark_url, github_diff_url, argo_workflow_url,
+        baseline_persistence_wait_mean_ms, baseline_execution_cache_wait_mean_ms,
+        baseline_sparse_trie_wait_mean_ms,
+        feature_persistence_wait_mean_ms, feature_execution_cache_wait_mean_ms,
+        feature_sparse_trie_wait_mean_ms
+    ) VALUES (
+        '{esc(args.workflow_name)}', '{esc(args.chain)}',
+        '{esc(baseline["ref"])}', '{esc(baseline["ref"])}',
+        '{esc(feature["ref"])}', '{esc(feature["ref"])}',
+        {blocks},
+        {b_stats.get("wall_clock_s", 0) * 1000}, {b_gas_per_second},
+        {b_stats["mean_ms"]}, {b_stats["p50_ms"]},
+        {b_stats["p90_ms"]}, {b_stats["p99_ms"]},
+        {f_stats.get("wall_clock_s", 0) * 1000}, {f_gas_per_second},
+        {f_stats["mean_ms"]}, {f_stats["p50_ms"]},
+        {f_stats["p90_ms"]}, {f_stats["p99_ms"]},
+        {mean_change}, {gas_change},
+        {latency_improved}, {throughput_improved},
+        {warmup_blocks}, '{big_blocks}',
+        '{esc(args.grafana_url)}', '{esc(args.github_diff_url)}', '{esc(args.job_url)}',
+        {b_persist}, {b_exec_cache}, {b_sparse},
+        {f_persist}, {f_exec_cache}, {f_sparse}
+    );
+    """
+
+    # Build ClickHouse HTTP URL (credentials via headers, never in URL)
+    host = ch_host.rstrip("/")
+    if not host.startswith("http"):
+        host = f"https://{host}:8443"
+
+    url = f"{host}/?database={ch_database}"
+
+    req = urllib.request.Request(url, data=insert.encode("utf-8"), method="POST")
+    req.add_header("Content-Type", "text/plain")
+    req.add_header("X-ClickHouse-User", ch_user)
+    req.add_header("X-ClickHouse-Key", ch_password)
+
+    try:
+        with urllib.request.urlopen(req) as resp:
+            body = resp.read().decode("utf-8")
+            if body.strip():
+                print(f"ClickHouse response: {body}")
+        print(f"Successfully uploaded benchmark results to ClickHouse ({args.workflow_name})")
+    except urllib.error.HTTPError as e:
+        body = e.read().decode("utf-8")
+        print(f"ClickHouse upload failed ({e.code}): {body}", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/bench-utils.js

@@ -0,0 +1,112 @@
+// Shared utilities for reth-bench result rendering.
+//
+// Used by bench-job-summary.js and bench-slack-notify.js.
+
+const fs = require('fs');
+const path = require('path');
+
+const SIG_EMOJI = { good: '✅', bad: '❌', neutral: '⚪' };
+
+function fmtMs(v) { return v.toFixed(2) + 'ms'; }
+function fmtMgas(v) { return v.toFixed(2); }
+function fmtS(v) { return v.toFixed(2) + 's'; }
+
+function fmtChange(ch) {
+  if (!ch || (!ch.pct && !ch.ci_pct)) return '';
+  const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
+  const ciStr = ch.ci_pct ? ` (±${ch.ci_pct.toFixed(2)}%)` : '';
+  return `${pctStr}${ciStr} ${SIG_EMOJI[ch.sig]}`;
+}
+
+function verdict(changes) {
+  const vals = Object.values(changes);
+  const hasBad = vals.some(v => v.sig === 'bad');
+  const hasGood = vals.some(v => v.sig === 'good');
+  if (hasBad && hasGood) return { emoji: '⚠️', label: 'Mixed Results' };
+  if (hasBad) return { emoji: '❌', label: 'Regression' };
+  if (hasGood) return { emoji: '✅', label: 'Improvement' };
+  return { emoji: '⚪', label: 'No Difference' };
+}
+
+function loadSamplyUrls(workDir) {
+  const urls = {};
+  for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
+    try {
+      const url = fs.readFileSync(path.join(workDir, run, 'samply-profile-url.txt'), 'utf8').trim();
+      if (url) urls[run] = url;
+    } catch {}
+  }
+  return urls;
+}
+
+function balModeLabel(mode) {
+  switch (mode) {
+    case 'true':
+    case 'feature':
+    case 'baseline':
+      return mode;
+    case 'both':
+      return 'true';
+    default:
+      return '';
+  }
+}
+
+function blocksLabel(summary) {
+  const parts = [];
+  if (summary.big_blocks) {
+    parts.push({ key: 'Big Blocks', value: summary.blocks });
+    const balMode = balModeLabel(summary.bal_mode || summary.bal || process.env.BENCH_BAL || 'false');
+    if (balMode) parts.push({ key: 'BAL', value: balMode });
+  } else {
+    const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
+    if (warmup) parts.push({ key: 'Warmup', value: warmup });
+    parts.push({ key: 'Blocks', value: summary.blocks });
+  }
+  const cores = process.env.BENCH_CORES || '0';
+  if (cores !== '0') parts.push({ key: 'Cores', value: cores });
+  if (summary.wait_time) parts.push({ key: 'Wait time', value: summary.wait_time });
+  return parts;
+}
+
+// The 7 metric rows shared by all renderers.
+// Returns an array of { label, baseline, feature, change } objects.
+function metricRows(summary) {
+  const b = summary.baseline.stats;
+  const f = summary.feature.stats;
+  const c = summary.changes;
+  return [
+    { label: 'Mean',       baseline: fmtMs(b.mean_ms),       feature: fmtMs(f.mean_ms),       change: fmtChange(c.mean) },
+    { label: 'StdDev',     baseline: fmtMs(b.stddev_ms),     feature: fmtMs(f.stddev_ms),     change: '' },
+    { label: 'P50',        baseline: fmtMs(b.p50_ms),        feature: fmtMs(f.p50_ms),        change: fmtChange(c.p50) },
+    { label: 'P90',        baseline: fmtMs(b.p90_ms),        feature: fmtMs(f.p90_ms),        change: fmtChange(c.p90) },
+    { label: 'P99',        baseline: fmtMs(b.p99_ms),        feature: fmtMs(f.p99_ms),        change: fmtChange(c.p99) },
+    { label: 'Mgas/s',     baseline: fmtMgas(b.mean_mgas_s), feature: fmtMgas(f.mean_mgas_s), change: fmtChange(c.mgas_s) },
+    { label: 'Wall Clock', baseline: fmtS(b.wall_clock_s),   feature: fmtS(f.wall_clock_s),   change: fmtChange(c.wall_clock) },
+    { label: 'Persist Wait', baseline: fmtMs(b.mean_persist_ms || 0), feature: fmtMs(f.mean_persist_ms || 0), change: fmtChange(c.persist_wait) },
+  ];
+}
+
+// Wait time rows: one row per metric showing mean values.
+function waitTimeRows(summary) {
+  const waitTimes = summary.wait_times || {};
+  const rows = [];
+  for (const key of Object.keys(waitTimes)) {
+    const wt = waitTimes[key];
+    rows.push({ title: wt.title, baseline: fmtMs(wt.baseline.mean_ms), feature: fmtMs(wt.feature.mean_ms) });
+  }
+  return rows;
+}
+
+module.exports = {
+  SIG_EMOJI,
+  fmtMs,
+  fmtMgas,
+  fmtS,
+  fmtChange,
+  verdict,
+  loadSamplyUrls,
+  blocksLabel,
+  metricRows,
+  waitTimeRows,
+};

.github/scripts/build_pgo_bolt.sh

@@ -1,414 +0,0 @@
-#!/usr/bin/env bash
-#
-# Full PGO+BOLT optimized build for reth using real reth-bench workloads.
-#
-# Phases:
-#   1. Build PGO-instrumented reth, run reth-bench → collect PGO profiles
-#   2. Build BOLT-instrumented reth (with PGO), run reth-bench → collect BOLT profiles
-#   3. Build final PGO+BOLT optimized binary
-#
-# Required environment variables:
-#   DATADIR    - Path to reth datadir (must already contain chain data)
-#   RPC_URL    - Source RPC URL for reth-bench to fetch payloads from
-#
-# Optional environment variables:
-#   PGO_BLOCKS      - Number of blocks for PGO profiling (default: 20)
-#   BOLT_BLOCKS     - Number of blocks for BOLT profiling (default: 20)
-#   SKIP_BOLT       - Temporarily skip BOLT phases (default: false)
-#   STRIP_SYMBOLS   - Strip debug symbols from output binary (default: true)
-#   COLLECT_PGO_ONLY - Stop after producing merged.profdata (default: false)
-#   PGO_PROFDATA    - Path to pre-collected merged.profdata (optional)
-#   PROFILE         - Cargo profile (default: maxperf-symbols)
-#   FEATURES        - Cargo features (default: jemalloc,asm-keccak,min-debug-logs)
-#   TARGET          - Target triple (default: auto-detected)
-#   EXTRA_RUSTFLAGS - Additional RUSTFLAGS (e.g. -C target-cpu=x86-64-v3)
-#
-# Output:
-#   target/$PROFILE_DIR/reth  — final optimized binary
-set -euo pipefail
-
-gha_section_start() {
-    local title="$1"
-    if [ -n "${GITHUB_ACTIONS:-}" ]; then
-        echo "::group::$title"
-    else
-        echo ""
-        echo "=== $title ==="
-    fi
-}
-
-gha_section_end() {
-    if [ -n "${GITHUB_ACTIONS:-}" ]; then
-        echo "::endgroup::"
-    fi
-}
-
-cd "$(dirname "$0")/../.."
-
-# ── Configuration ──────────────────────────────────────────────────────────────
-PGO_BLOCKS="${PGO_BLOCKS:-20}"
-BOLT_BLOCKS="${BOLT_BLOCKS:-20}"
-SKIP_BOLT="${SKIP_BOLT:-false}"
-STRIP_SYMBOLS="${STRIP_SYMBOLS:-true}"
-COLLECT_PGO_ONLY="${COLLECT_PGO_ONLY:-false}"
-PROFILE="${PROFILE:-maxperf-symbols}"
-FEATURES="${FEATURES:-jemalloc,asm-keccak,min-debug-logs}"
-TARGET="${TARGET:-$(rustc -Vv | grep host | cut -d' ' -f2)}"
-BASE_RUSTFLAGS="${RUSTFLAGS:-}"
-EXTRA_RUSTFLAGS="${EXTRA_RUSTFLAGS:-}"
-COMBINED_RUSTFLAGS="$BASE_RUSTFLAGS $EXTRA_RUSTFLAGS"
-PGO_PROFDATA="${PGO_PROFDATA:-}"
-DATADIR="${DATADIR:-}"
-RPC_URL="${RPC_URL:-}"
-
-SKIP_BOLT_BOOL=false
-if [[ "${SKIP_BOLT,,}" == "true" || "$SKIP_BOLT" == "1" ]]; then
-    SKIP_BOLT_BOOL=true
-fi
-
-STRIP_SYMBOLS_BOOL=false
-if [[ "${STRIP_SYMBOLS,,}" == "true" || "$STRIP_SYMBOLS" == "1" ]]; then
-    STRIP_SYMBOLS_BOOL=true
-fi
-
-COLLECT_PGO_ONLY_BOOL=false
-if [[ "${COLLECT_PGO_ONLY,,}" == "true" || "$COLLECT_PGO_ONLY" == "1" ]]; then
-    COLLECT_PGO_ONLY_BOOL=true
-fi
-
-USE_PRECOLLECTED_PGO=false
-if [ -n "$PGO_PROFDATA" ]; then
-    if [ ! -f "$PGO_PROFDATA" ]; then
-        echo "error: PGO_PROFDATA points to a missing file: $PGO_PROFDATA"
-        exit 1
-    fi
-    USE_PRECOLLECTED_PGO=true
-fi
-
-NEEDS_BENCH_WORKLOAD=true
-if [ "$USE_PRECOLLECTED_PGO" = true ] && [ "$SKIP_BOLT_BOOL" = true ]; then
-    NEEDS_BENCH_WORKLOAD=false
-fi
-
-if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
-    : "${DATADIR:?DATADIR must be set to the reth data directory}"
-    : "${RPC_URL:?RPC_URL must be set}"
-fi
-
-if [[ "$PROFILE" == dev ]]; then
-    PROFILE_DIR=debug
-else
-    PROFILE_DIR=$PROFILE
-fi
-
-MANIFEST_PATH="bin/reth"
-
-LLVM_VERSION=$(rustc -Vv | grep -oP 'LLVM version: \K\d+')
-PGO_DIR="$PWD/target/pgo-profiles"
-BOLT_DIR="$PWD/target/bolt-profiles"
-CARGO_ARGS=(--profile "$PROFILE" --features "$FEATURES" --manifest-path "$MANIFEST_PATH/Cargo.toml" --bin "reth" --locked)
-
-# Enable debug symbols for BOLT (requires symbols to reorder code).
-# Strip them at the end.
-PROFILE_UPPER=$(echo "$PROFILE" | tr '[:lower:]-' '[:upper:]_')
-export "CARGO_PROFILE_${PROFILE_UPPER}_STRIP=debuginfo"
-
-gha_section_start "Full PGO+BOLT Build"
-echo "Binary:      reth"
-echo "Manifest:    $MANIFEST_PATH"
-echo "Target:      $TARGET"
-echo "Profile:     $PROFILE"
-echo "Features:    $FEATURES"
-echo "LLVM:        $LLVM_VERSION"
-echo "PGO blocks:  $PGO_BLOCKS"
-echo "BOLT blocks: $BOLT_BLOCKS"
-echo "Skip BOLT:   $SKIP_BOLT"
-echo "Strip symbols: $STRIP_SYMBOLS"
-echo "Collect only: $COLLECT_PGO_ONLY"
-echo "PGO profdata: ${PGO_PROFDATA:-<collect with reth-bench>}"
-echo "RUSTFLAGS:   ${BASE_RUSTFLAGS:-<unset>}"
-echo "EXTRA_RUSTFLAGS: ${EXTRA_RUSTFLAGS:-<unset>}"
-if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
-    echo "Datadir:     $DATADIR"
-    echo "RPC URL:     $RPC_URL"
-else
-    echo "Datadir:     <not required>"
-    echo "RPC URL:     <not required>"
-fi
-gha_section_end
-
-# ── Prerequisites ──────────────────────────────────────────────────────────────
-gha_section_start "Installing prerequisites"
-rustup component add llvm-tools-preview
-
-LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
-if [ -z "$LLVM_PROFDATA" ]; then
-    echo "error: llvm-profdata not found"
-    exit 1
-fi
-
-install_bolt() {
-    if command -v llvm-bolt &>/dev/null; then
-        echo "BOLT already installed"
-        return
-    fi
-    echo "Installing BOLT from apt.llvm.org..."
-    wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc >/dev/null
-    CODENAME=$(lsb_release -cs)
-    echo "deb http://apt.llvm.org/$CODENAME/ llvm-toolchain-$CODENAME-$LLVM_VERSION main" | sudo tee /etc/apt/sources.list.d/llvm.list >/dev/null
-    sudo apt-get update -qq
-    sudo apt-get install -y -qq "bolt-$LLVM_VERSION"
-    sudo ln -sf "/usr/bin/llvm-bolt-$LLVM_VERSION" /usr/local/bin/llvm-bolt
-    sudo ln -sf "/usr/bin/merge-fdata-$LLVM_VERSION" /usr/local/bin/merge-fdata
-}
-if [ "$SKIP_BOLT_BOOL" = true ]; then
-    echo "Skipping BOLT installation (SKIP_BOLT=$SKIP_BOLT)"
-else
-    install_bolt
-fi
-gha_section_end
-
-if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
-    # Build reth-bench once (non-instrumented) — reused for both phases.
-    gha_section_start "Building reth-bench"
-    RUSTFLAGS="$COMBINED_RUSTFLAGS" \
-        cargo build --profile "$PROFILE" --features "$FEATURES" \
-        --manifest-path bin/reth-bench/Cargo.toml --bin reth-bench --locked
-    RETH_BENCH_BIN="$(find target -name reth-bench -type f -executable | head -1)"
-    echo "reth-bench: $RETH_BENCH_BIN"
-    gha_section_end
-else
-    gha_section_start "Building reth-bench"
-    echo "Skipping reth-bench build (pre-collected PGO with SKIP_BOLT=true)"
-    gha_section_end
-fi
-
-# ── Helpers ────────────────────────────────────────────────────────────────────
-RETH_PID=
-cleanup() {
-    if [ -n "${RETH_PID:-}" ] && kill -0 "$RETH_PID" 2>/dev/null; then
-        echo "Stopping reth (pid $RETH_PID)..."
-        sudo kill "$RETH_PID" 2>/dev/null || true
-        for i in $(seq 1 60); do
-            sudo kill -0 "$RETH_PID" 2>/dev/null || break
-            if [ $((i % 10)) -eq 0 ]; then
-                echo "  waiting... (${i}s)"
-            fi
-            sleep 1
-        done
-        sudo kill -9 "$RETH_PID" 2>/dev/null || true
-    fi
-}
-trap cleanup EXIT
-
-# Start reth, wait for RPC, run reth-bench, then stop reth.
-# Arguments: $1 = reth binary path, $2 = number of blocks, $3 = log label
-run_bench_workload() {
-    local reth_bin="$1" blocks="$2" label="$3"
-    local http_port=8545 authrpc_port=8551
-
-    echo "--- Starting reth ($label) ---"
-    sudo "$reth_bin" node \
-        --datadir "$DATADIR" \
-        --log.file.directory "/tmp/reth-${label}-logs" \
-        --engine.accept-execution-requests-hash \
-        --http --http.port "$http_port" \
-        --authrpc.port "$authrpc_port" \
-        --disable-discovery --no-persist-peers \
-        > "/tmp/reth-${label}.log" 2>&1 &
-    RETH_PID=$!
-
-    echo "Waiting for reth RPC..."
-    for i in $(seq 1 120); do
-        if curl -sf "http://127.0.0.1:$http_port" -X POST \
-            -H 'Content-Type: application/json' \
-            -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
-            > /dev/null 2>&1; then
-            echo "reth is ready after ${i}s"
-            break
-        fi
-        if [ "$i" -eq 120 ]; then
-            echo "error: reth failed to start within 120s"
-            cat "/tmp/reth-${label}.log"
-            exit 1
-        fi
-        sleep 1
-    done
-
-    echo "Running reth-bench ($blocks blocks)..."
-    "$RETH_BENCH_BIN" new-payload-fcu \
-        --rpc-url "$RPC_URL" \
-        --engine-rpc-url "http://127.0.0.1:$authrpc_port" \
-        --jwt-secret "$DATADIR/jwt.hex" \
-        --advance "$blocks" \
-        --reth-new-payload 2>&1 | sed -u "s/^/[$label] /"
-
-    echo "Stopping reth ($label)..."
-    sudo kill "$RETH_PID" 2>/dev/null || true
-    for i in $(seq 1 60); do
-        sudo kill -0 "$RETH_PID" 2>/dev/null || break
-        sleep 1
-    done
-    sudo kill -9 "$RETH_PID" 2>/dev/null || true
-    RETH_PID=
-}
-
-publish_binary() {
-    local source_bin="$1"
-    for out in "target/$TARGET/$PROFILE_DIR" "target/$PROFILE_DIR"; do
-        local destination="$out/reth"
-        mkdir -p "$out"
-        # Skip copying when source and destination resolve to the same inode.
-        if [ -e "$destination" ] && [ "$source_bin" -ef "$destination" ]; then
-            continue
-        fi
-        cp "$source_bin" "$destination"
-    done
-}
-
-if [ "$USE_PRECOLLECTED_PGO" = true ]; then
-    gha_section_start "Phase 1: Using Pre-Collected PGO Profile"
-    rm -rf "$PGO_DIR"
-    mkdir -p "$PGO_DIR"
-    cp "$PGO_PROFDATA" "$PGO_DIR/merged.profdata"
-    echo "Using pre-collected profile: $PGO_PROFDATA"
-    echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
-    gha_section_end
-else
-    # ── Phase 1: PGO profile collection ───────────────────────────────────────
-    gha_section_start "Phase 1: PGO Profile Collection"
-
-    rm -rf "$PGO_DIR"
-    mkdir -p "$PGO_DIR"
-
-    echo "Building PGO-instrumented binary..."
-    RUSTFLAGS="-Cprofile-generate=$PGO_DIR -Crelocation-model=pic $COMBINED_RUSTFLAGS" \
-        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
-
-    PGO_RETH_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
-    echo "Instrumented binary: $PGO_RETH_BIN ($(ls -lh "$PGO_RETH_BIN" | awk '{print $5}'))"
-
-    run_bench_workload "$PGO_RETH_BIN" "$PGO_BLOCKS" "pgo"
-
-    # Fix ownership if reth ran as root.
-    sudo chown -R "$(id -un):$(id -gn)" "$PGO_DIR" 2>/dev/null || true
-
-    # Merge PGO profiles.
-    echo "Merging PGO profiles..."
-    PROFRAW_COUNT=$(find "$PGO_DIR" -name '*.profraw' | wc -l)
-    echo "Found $PROFRAW_COUNT .profraw files"
-    if [ "$PROFRAW_COUNT" -eq 0 ]; then
-        echo "error: no .profraw files — instrumented binary did not produce profiles"
-        exit 1
-    fi
-    "$LLVM_PROFDATA" merge -o "$PGO_DIR/merged.profdata" "$PGO_DIR"/*.profraw
-    echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
-    gha_section_end
-fi
-
-if [ "$COLLECT_PGO_ONLY_BOOL" = true ]; then
-    gha_section_start "PGO Collection Complete"
-    echo "COLLECT_PGO_ONLY=true, skipping PGO/BOLT optimized binary build"
-    echo "Profile: $PGO_DIR/merged.profdata"
-    gha_section_end
-    exit 0
-fi
-
-if [ "$SKIP_BOLT_BOOL" = true ]; then
-    gha_section_start "BOLT Phase Skipped"
-    echo "SKIP_BOLT=$SKIP_BOLT, building PGO-only binary"
-    echo "Building PGO-optimized binary..."
-    RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata $COMBINED_RUSTFLAGS" \
-        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
-
-    BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
-    if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
-        echo "Stripping debug symbols..."
-        strip "$BUILT_BIN"
-    else
-        echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
-    fi
-    publish_binary "$BUILT_BIN"
-    gha_section_end
-else
-    # ── Phase 2: BOLT profile collection (with PGO) ──────────────────────────
-    gha_section_start "Phase 2: BOLT Profile Collection (with PGO)"
-
-    rm -rf "$BOLT_DIR"
-    mkdir -p "$BOLT_DIR"
-
-    echo "Building BOLT-instrumented binary with PGO..."
-    # --emit-relocs preserves relocation entries in the binary, required by llvm-bolt -instrument
-    RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
-        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
-
-    # Instrument with BOLT
-    BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
-    BOLT_INSTRUMENTED_BIN="$BUILT_BIN-bolt-instrumented"
-
-    echo "Instrumenting binary with BOLT..."
-    # --skip-funcs: skip compiler-generated drop_in_place functions that BOLT can't handle
-    # as split functions in relocation mode (triggered by --emit-relocs)
-    llvm-bolt "$BUILT_BIN" \
-        -instrument \
-        --instrumentation-file-append-pid \
-        --instrumentation-file="$BOLT_DIR/prof" \
-        --skip-funcs='.*drop_in_place.*' \
-        -o "$BOLT_INSTRUMENTED_BIN"
-    echo "BOLT-instrumented binary: $BOLT_INSTRUMENTED_BIN ($(ls -lh "$BOLT_INSTRUMENTED_BIN" | awk '{print $5}'))"
-
-    run_bench_workload "$BOLT_INSTRUMENTED_BIN" "$BOLT_BLOCKS" "bolt"
-
-    # Fix ownership for BOLT profiles
-    sudo chown -R "$(id -un):$(id -gn)" "$BOLT_DIR" 2>/dev/null || true
-
-    # Merge BOLT profiles
-    echo "Merging BOLT profiles..."
-    FDATA_COUNT=$(find "$BOLT_DIR" -name '*.fdata' | wc -l)
-    echo "Found $FDATA_COUNT .fdata files"
-    if [ "$FDATA_COUNT" -eq 0 ]; then
-        echo "error: no .fdata files — BOLT-instrumented binary did not produce profiles"
-        exit 1
-    fi
-    merge-fdata "$BOLT_DIR"/*.fdata > "$BOLT_DIR/merged.fdata"
-    echo "BOLT profile: $BOLT_DIR/merged.fdata ($(ls -lh "$BOLT_DIR/merged.fdata" | awk '{print $5}'))"
-    gha_section_end
-
-    # ── Phase 3: Final optimized build ───────────────────────────────────────
-    gha_section_start "Phase 3: Final PGO+BOLT Optimized Build"
-
-    echo "Building PGO-optimized binary..."
-    # --emit-relocs preserves relocation entries in the binary, required by llvm-bolt for code reordering
-    RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
-        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
-
-    BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
-    OPTIMIZED_BIN="$BUILT_BIN-bolt-optimized"
-
-    echo "Optimizing with BOLT..."
-    llvm-bolt "$BUILT_BIN" \
-        -o "$OPTIMIZED_BIN" \
-        --data "$BOLT_DIR/merged.fdata" \
-        -reorder-blocks=ext-tsp \
-        -reorder-functions=cdsort \
-        -split-functions \
-        -split-all-cold \
-        -dyno-stats \
-        -icf=1 \
-        -use-gnu-stack \
-        --skip-funcs='.*drop_in_place.*'
-
-    if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
-        echo "Stripping debug symbols..."
-        strip "$OPTIMIZED_BIN"
-    else
-        echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
-    fi
-    publish_binary "$OPTIMIZED_BIN"
-    gha_section_end
-fi
-
-gha_section_start "Build Complete"
-ls -lh "target/$PROFILE_DIR/reth"
-echo "Output: target/$PROFILE_DIR/reth"
-gha_section_end

.github/scripts/check_rv32imac.sh

@@ -1,10 +1,7 @@
 #!/usr/bin/env bash
-set -uo pipefail
+set -uxo pipefail
 
 crates_to_check=(
-    reth-codecs-derive
-    reth-primitives
-    reth-primitives-traits
     reth-network-peers
     reth-trie-common
     reth-trie-sparse

.github/scripts/check_wasm.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-set -uo pipefail
+set -uxo pipefail
 
 readarray -t crates < <(
   cargo metadata --format-version=1 --no-deps | jq -r '.packages[].name' | grep '^reth' | sort
@@ -22,6 +22,7 @@ exclude_crates=(
   reth-downloaders
   reth-e2e-test-utils
   reth-engine-service
+  reth-execution-cache
   reth-engine-tree
   reth-engine-util
   reth-eth-wire
@@ -55,6 +56,7 @@ exclude_crates=(
   reth-ress-provider
   # The following are not supposed to be working
   reth # all of the crates below
+  reth-bb # binary-only, uses tokio features unsupported on wasm
   reth-storage-rpc-provider
   reth-invalid-block-hooks # reth-provider
   reth-libmdbx # mdbx

.github/scripts/fetch-grafana-dashboard.py

@@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+"""
+Fetch a Grafana dashboard and convert it to the portable import format.
+
+Fetches the dashboard via API, replaces internal datasource/variable references
+with template variables, and adds __inputs/__requires/__elements so the JSON is
+importable on any Grafana instance.
+
+Usage:
+    export FETCH_GRAFANA_DASHBOARD_URL=https://<NAMESPACE>.grafana.net
+    export FETCH_GRAFANA_DASHBOARD_TOKEN=glsa_...
+
+    python3 .github/scripts/fetch-grafana-dashboard.py <dashboard-uid> > output.json
+"""
+
+import json
+import os
+import sys
+import urllib.request
+
+PANEL_TYPE_NAMES = {
+    "bargauge": "Bar gauge",
+    "gauge": "Gauge",
+    "heatmap": "Heatmap",
+    "piechart": "Pie chart",
+    "stat": "Stat",
+    "table": "Table",
+    "timeseries": "Time series",
+    "barchart": "Bar chart",
+    "text": "Text",
+    "dashlist": "Dashboard list",
+    "logs": "Logs",
+    "nodeGraph": "Node Graph",
+    "histogram": "Histogram",
+    "candlestick": "Candlestick",
+    "state-timeline": "State timeline",
+    "status-history": "Status history",
+    "geomap": "Geomap",
+    "canvas": "Canvas",
+    "news": "News",
+    "xychart": "XY Chart",
+    "trend": "Trend",
+    "datagrid": "Datagrid",
+    "flamegraph": "Flame Graph",
+    "traces": "Traces",
+}
+
+
+def fetch_json(base_url: str, token: str, path: str) -> dict:
+    url = f"{base_url}{path}"
+    req = urllib.request.Request(url, headers={"Authorization": f"Bearer {token}"})
+    with urllib.request.urlopen(req) as resp:
+        return json.loads(resp.read())
+
+
+def fetch_dashboard(base_url: str, token: str, uid: str) -> dict:
+    return fetch_json(base_url, token, f"/api/dashboards/uid/{uid}")
+
+
+def fetch_grafana_version(base_url: str) -> str:
+    req = urllib.request.Request(f"{base_url}/api/health")
+    with urllib.request.urlopen(req) as resp:
+        data = json.loads(resp.read())
+    # version string like "13.0.0-23940615780.patch2" -> take just the semver part
+    version = data.get("version", "")
+    # strip build metadata after the first hyphen if it looks like a pre-release
+    parts = version.split("-")
+    return parts[0] if parts else version
+
+
+def collect_panel_types(panels: list) -> set[str]:
+    types = set()
+    for panel in panels:
+        ptype = panel.get("type", "")
+        if ptype and ptype != "row":
+            types.add(ptype)
+        # nested panels inside collapsed rows
+        for sub in panel.get("panels", []):
+            sub_type = sub.get("type", "")
+            if sub_type and sub_type != "row":
+                types.add(sub_type)
+    return types
+
+
+def has_expression_datasource(dashboard: dict) -> bool:
+    return "__expr__" in json.dumps(dashboard)
+
+
+def make_exportable(dashboard: dict, grafana_version: str = "") -> dict:
+    dash = json.loads(json.dumps(dashboard))  # deep copy
+
+    # --- Strip internal fields ---
+    dash.pop("id", None)
+
+    # --- Rewrite links: point to the public repo instead of internal ---
+    dash["links"] = [
+        {
+            "asDropdown": False,
+            "icon": "external link",
+            "includeVars": False,
+            "keepTime": False,
+            "tags": [],
+            "targetBlank": True,
+            "title": "Source (GitHub)",
+            "tooltip": "View source file in repository",
+            "type": "link",
+            "url": "https://github.com/paradigmxyz/reth/tree/main/etc/grafana/dashboards",
+        }
+    ]
+
+    # --- Datasource: victoriametrics -> prometheus ---
+    dash_str = json.dumps(dash)
+    dash_str = dash_str.replace("victoriametrics-metrics-datasource", "prometheus")
+    dash = json.loads(dash_str)
+
+    # --- Templating: instance_label constant -> ${VAR_INSTANCE_LABEL} ---
+    # Also strip default-value fields the API returns that are not needed for import
+    STRIP_VAR_DEFAULTS = {"allowCustomValue", "regexApplyTo"}
+    for var in dash.get("templating", {}).get("list", []):
+        if var.get("name") == "instance_label" and var.get("type") == "constant":
+            var["query"] = "${VAR_INSTANCE_LABEL}"
+            var["current"] = {
+                "value": "${VAR_INSTANCE_LABEL}",
+                "text": "${VAR_INSTANCE_LABEL}",
+                "selected": False,
+            }
+            var["options"] = [
+                {
+                    "value": "${VAR_INSTANCE_LABEL}",
+                    "text": "${VAR_INSTANCE_LABEL}",
+                    "selected": False,
+                }
+            ]
+        # Clear current values for query/datasource vars (not meaningful for import)
+        elif var.get("type") in ("query", "datasource"):
+            var["current"] = {}
+        # Remove noisy default fields
+        for field in STRIP_VAR_DEFAULTS:
+            var.pop(field, None)
+        # Strip falsy defaults on query/datasource vars (API returns them, export omits them)
+        if var.get("type") in ("query", "datasource"):
+            for field in ("hide", "multi", "skipUrlSync"):
+                if not var.get(field):
+                    var.pop(field, None)
+
+    # --- Build __inputs ---
+    inputs = [
+        {
+            "name": "DS_PROMETHEUS",
+            "label": "Prometheus",
+            "description": "",
+            "type": "datasource",
+            "pluginId": "prometheus",
+            "pluginName": "Prometheus",
+        },
+    ]
+
+    if has_expression_datasource(dash):
+        inputs.append(
+            {
+                "name": "DS_EXPRESSION",
+                "label": "Expression",
+                "description": "",
+                "type": "datasource",
+                "pluginId": "__expr__",
+            }
+        )
+
+    inputs.append(
+        {
+            "name": "VAR_INSTANCE_LABEL",
+            "type": "constant",
+            "label": "Instance Label",
+            "value": "job",
+            "description": "",
+        }
+    )
+
+    # --- Build __requires ---
+    requires = []
+
+    if has_expression_datasource(dash):
+        requires.append({"type": "datasource", "id": "__expr__", "version": "1.0.0"})
+
+    panel_types = collect_panel_types(dash.get("panels", []))
+    for pt in sorted(panel_types):
+        requires.append(
+            {
+                "type": "panel",
+                "id": pt,
+                "name": PANEL_TYPE_NAMES.get(pt, pt),
+                "version": "",
+            }
+        )
+
+    requires.append(
+        {"type": "grafana", "id": "grafana", "name": "Grafana", "version": grafana_version}
+    )
+    requires.append(
+        {
+            "type": "datasource",
+            "id": "prometheus",
+            "name": "Prometheus",
+            "version": "1.0.0",
+        }
+    )
+
+    # --- Assemble output (with __inputs/__requires/__elements first) ---
+    output = {
+        "__inputs": inputs,
+        "__elements": {},
+        "__requires": requires,
+    }
+    output.update(dash)
+
+    return output
+
+
+def main():
+    if len(sys.argv) < 2:
+        print(f"Usage: {sys.argv[0]} <dashboard-uid>", file=sys.stderr)
+        sys.exit(1)
+
+    uid = sys.argv[1]
+    base_url = os.environ.get("FETCH_GRAFANA_DASHBOARD_URL", "").rstrip("/")
+    token = os.environ.get("FETCH_GRAFANA_DASHBOARD_TOKEN", "")
+
+    if not base_url or not token:
+        print(
+            "Error: FETCH_GRAFANA_DASHBOARD_URL and FETCH_GRAFANA_DASHBOARD_TOKEN env vars required",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    resp = fetch_dashboard(base_url, token, uid)
+    dashboard = resp["dashboard"]
+
+    grafana_version = fetch_grafana_version(base_url)
+    exported = make_exportable(dashboard, grafana_version)
+    print(json.dumps(exported, indent=2))
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/hive/Dockerfile

@@ -7,7 +7,7 @@ FROM lukemathwalker/cargo-chef:latest-rust-1 AS chef
 WORKDIR /app
 
 # Install system dependencies
-RUN apt-get update && apt-get -y upgrade && apt-get install -y libclang-dev pkg-config
+RUN apt-get update && apt-get install -y libclang-dev pkg-config
 
 # 
 # We prepare the build plan

.github/scripts/hive/build_simulators.sh

@@ -11,8 +11,14 @@ go build .
 
 # Run each hive command in the background for each simulator and wait
 echo "Building images"
-# TODO: test code has been moved from https://github.com/ethereum/execution-spec-tests to https://github.com/ethereum/execution-specs  we need to pin eels branch with `--sim.buildarg branch=<release-branch-name>` once we have the fusaka release tagged on the new repo
-./hive -client reth --sim "ethereum/eels" --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/v5.3.0/fixtures_develop.tar.gz -sim.timelimit 1s || true &
+./hive -client reth --sim "ethereum/eels/consume-engine" \
+    --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/v5.3.0/fixtures_develop.tar.gz \
+    --sim.buildarg branch=forks/osaka \
+    --sim.timelimit 1s || true &
+./hive -client reth --sim "ethereum/eels/consume-rlp" \
+    --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/v5.3.0/fixtures_develop.tar.gz \
+    --sim.buildarg branch=forks/osaka \
+    --sim.timelimit 1s || true &
 ./hive -client reth --sim "ethereum/engine" -sim.timelimit 1s || true &
 ./hive -client reth --sim "devp2p" -sim.timelimit 1s || true &
 ./hive -client reth --sim "ethereum/rpc-compat" -sim.timelimit 1s || true &

.github/scripts/hive/expected_failures.yaml

@@ -1,9 +1,6 @@
 # tracked by https://github.com/paradigmxyz/reth/issues/13879
 rpc-compat:
   - debug_getRawBlock/get-invalid-number (reth)
-  - debug_getRawHeader/get-invalid-number (reth)
-  - debug_getRawReceipts/get-invalid-number (reth)
-  - debug_getRawReceipts/get-block-n (reth)
   - debug_getRawTransaction/get-invalid-hash (reth)
 
   - eth_getStorageAt/get-storage-invalid-key-too-large (reth)

.github/scripts/hive/ignored_tests.yaml

@@ -21,7 +21,6 @@ engine-cancun:
   # Affects all clients, not just reth. Tracked: https://github.com/ethereum/hive/issues/1382
   - Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (reth)
   - Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (reth)
-  - Multiple New Payloads Extending Canonical Chain, Wait for Canonical Payload (Cancun) (reth)
 engine-api:
   - Transaction Re-Org, Re-Org Out (Paris) (reth)
   - Transaction Re-Org, Re-Org to Different Block (Paris) (reth)
@@ -31,5 +30,3 @@ engine-api:
   - Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Paris) (reth)
   - Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Paris) (reth)
   - Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P10 (Paris) (reth)
-  - Multiple New Payloads Extending Canonical Chain, Wait for Canonical Payload (Paris) (reth)
-  - Multiple New Payloads Extending Canonical Chain, Set Head to First Payload Received (Paris) (reth)

.github/scripts/hive/run_simulator.sh

@@ -13,7 +13,13 @@ if [[ "${sim}" == *"eels"* ]]; then
 fi
 
 run_hive() {
-    hive --sim "${sim}" --sim.limit "${limit}" --sim.parallelism "${parallelism}" --client reth 2>&1 | tee /tmp/log || true
+    hive \
+  --sim "${sim}" \
+  --sim.limit "${limit}" \
+  --sim.limit.exact=false \
+  --sim.parallelism "${parallelism}" \
+  --client reth \
+  2>&1 | tee /tmp/log || true
 }
 
 check_log() {

.github/scripts/install_geth.sh

@@ -2,7 +2,7 @@
 
 # Installs Geth (https://geth.ethereum.org) in $HOME/bin for x86_64 Linux.
 
-set -eo pipefail
+set -exo pipefail
 
 GETH_BUILD=${GETH_BUILD:-"1.13.4-3f907d6a"}
 

.github/scripts/verify_image_arch.sh

@@ -7,7 +7,7 @@
 # Environment:
 #   DRY_RUN=true  - Skip actual verification, just print what would be checked.
 
-set -euo pipefail
+set -euxo pipefail
 
 TARGETS="${1:-}"
 REGISTRY="${2:-}"

.github/workflows/bench-scheduled.yml

@@ -0,0 +1,995 @@
+# Scheduled regression benchmarks (nightly + hourly + release).
+#
+# Three modes:
+#   nightly — Compares the previous nightly Docker build against the current one.
+#             Runs daily after docker.yml produces a new nightly image.
+#   hourly  — Compares main HEAD against the last benchmarked commit to catch
+#             regressions quickly. Falls back to HEAD~1 on first run.
+#             Skips if no new commits or if a previous run is still in progress.
+#   release — Compares the latest GitHub release tag against the current nightly
+#             Docker build. Runs daily to track nightly vs release performance.
+#
+# State is persisted between runs via the decofe/reth-bench-charts repo: each
+# successful run saves the feature commit SHA so the next run knows what to
+# compare against.
+
+on:
+  schedule:
+    # Nightly: compares previous vs current nightly Docker build
+    - cron: "30 5 * * *"
+    # Hourly: compares main HEAD vs last benchmarked commit, skips if no new commits
+    - cron: "0 * * * *"
+    # Release: compares latest GitHub release tag vs current nightly Docker build
+    - cron: "0 9 * * *"
+  workflow_dispatch:
+    inputs:
+      force:
+        description: "Force run even if no new commit (bypass skip logic)"
+        required: false
+        default: false
+        type: boolean
+      slack:
+        description: "Slack notification policy"
+        required: false
+        default: "never"
+        type: choice
+        options:
+          - always
+          - on-win
+          - on-error
+          - never
+      mode:
+        description: "Benchmark mode"
+        required: false
+        default: "nightly"
+        type: choice
+        options:
+          - nightly
+          - hourly
+          - release
+
+env:
+  CARGO_TERM_COLOR: always
+  RUSTC_WRAPPER: "sccache"
+
+name: bench-scheduled
+
+permissions:
+  contents: read
+  actions: read
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Job 1: Resolve refs, check staleness, manage state
+  # ---------------------------------------------------------------------------
+  resolve-refs:
+    name: resolve-refs
+    runs-on: ubuntu-latest
+    outputs:
+      mode: ${{ steps.mode.outputs.mode }}
+      baseline-ref: ${{ steps.refs.outputs.baseline-ref }}
+      feature-ref: ${{ steps.refs.outputs.feature-ref }}
+      should-skip: ${{ steps.refs.outputs.should-skip }}
+      is-stale: ${{ steps.refs.outputs.is-stale }}
+      stale-age-hours: ${{ steps.refs.outputs.stale-age-hours }}
+      nightly-created: ${{ steps.refs.outputs.nightly-created }}
+      long-running: ${{ steps.refs.outputs.long-running }}
+      release-tag: ${{ steps.refs.outputs.release-tag }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          sparse-checkout: .github/scripts
+          sparse-checkout-cone-mode: true
+          fetch-depth: 2
+
+      - name: Detect mode
+        id: mode
+        run: |
+          # Maps cron schedules to modes (must match the schedule entries above)
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            MODE="${{ inputs.mode || 'nightly' }}"
+          elif [ "${{ github.event.schedule }}" = "30 5 * * *" ]; then
+            MODE="nightly"
+          elif [ "${{ github.event.schedule }}" = "0 9 * * *" ]; then
+            MODE="release"
+          else
+            MODE="hourly"
+          fi
+          echo "mode=$MODE" >> "$GITHUB_OUTPUT"
+          echo "Detected mode: $MODE"
+
+      - name: Resolve refs
+        id: refs
+        env:
+          GH_TOKEN: ${{ github.token }}
+          DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+        run: |
+          FORCE="${{ inputs.force || 'false' }}"
+          MODE="${{ steps.mode.outputs.mode }}"
+          .github/scripts/bench-scheduled-refs.sh "$FORCE" "$MODE"
+
+      - name: Alert on long-running hourly
+        if: steps.mode.outputs.mode == 'hourly' && steps.refs.outputs.long-running == 'true' && !(github.event_name == 'workflow_dispatch' && inputs.slack == 'never')
+        uses: actions/github-script@v9
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) return;
+
+            const repo = '${{ github.repository }}';
+            const runUrl = `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: ':warning: Hourly Bench: previous run still in progress', emoji: true },
+              },
+              {
+                type: 'section',
+                text: {
+                  type: 'mrkdwn',
+                  text: 'A previous hourly benchmark run is still in progress. This invocation will be skipped.\nThis may indicate a long-running or stuck job.',
+                },
+              },
+              {
+                type: 'actions',
+                elements: [{
+                  type: 'button',
+                  text: { type: 'plain_text', text: 'View Run :github:', emoji: true },
+                  url: runUrl,
+                  action_id: 'ci_button',
+                }],
+              },
+            ];
+            await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' },
+              body: JSON.stringify({ channel, blocks, text: 'Hourly bench: previous run still in progress', unfurl_links: false }),
+            });
+
+      - name: Alert on stale nightly
+        if: steps.mode.outputs.mode == 'nightly' && steps.refs.outputs.is-stale == 'true' && !(github.event_name == 'workflow_dispatch' && inputs.slack == 'never')
+        uses: actions/github-script@v9
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) {
+              core.warning('Slack credentials not set, skipping stale nightly alert');
+              return;
+            }
+
+            const ageHours = '${{ steps.refs.outputs.stale-age-hours }}';
+            const created = '${{ steps.refs.outputs.nightly-created }}';
+            const featureRef = '${{ steps.refs.outputs.feature-ref }}';
+            const shortSha = featureRef.slice(0, 8);
+            const repo = '${{ github.repository }}';
+            const runUrl = `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: ':rotating_light: Nightly Regression: nightly build is stale', emoji: true },
+              },
+              {
+                type: 'section',
+                text: {
+                  type: 'mrkdwn',
+                  text: [
+                    '*Nightly regression did not run* — nightly build is stale',
+                    '',
+                    `The latest nightly image was built from a commit that is *${ageHours}h old* (threshold: 24h).`,
+                    `This means today's nightly docker build likely failed and no new image was produced.`,
+                    '',
+                    `Stale commit: \`${shortSha}\` (built at ${created})`,
+                    '',
+                    '*Action required:* Check the <https://github.com/' + repo + '/actions/workflows/docker.yml|docker.yml> workflow for failures.',
+                  ].join('\n'),
+                },
+              },
+              {
+                type: 'actions',
+                elements: [
+                  {
+                    type: 'button',
+                    text: { type: 'plain_text', text: 'View Run :github:', emoji: true },
+                    url: runUrl,
+                    action_id: 'ci_button',
+                  },
+                ],
+              },
+            ];
+
+            const resp = await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: {
+                'Authorization': `Bearer ${token}`,
+                'Content-Type': 'application/json',
+              },
+              body: JSON.stringify({
+                channel,
+                blocks,
+                text: 'Nightly regression: nightly build is stale',
+                unfurl_links: false,
+              }),
+            });
+            const data = await resp.json();
+            if (!data.ok) {
+              core.warning(`Slack API error: ${JSON.stringify(data)}`);
+            }
+
+      - name: Fail on stale nightly
+        if: steps.mode.outputs.mode == 'nightly' && steps.refs.outputs.is-stale == 'true'
+        run: |
+          echo "::error::Nightly build is stale (>24h old). Aborting."
+          exit 1
+
+  # ---------------------------------------------------------------------------
+  # Job 2: Run the benchmark
+  # ---------------------------------------------------------------------------
+  bench-scheduled:
+    needs: resolve-refs
+    if: |
+      needs.resolve-refs.outputs.should-skip != 'true' &&
+      needs.resolve-refs.outputs.is-stale != 'true'
+    name: bench-scheduled
+    runs-on: [self-hosted, Linux, X64, available]
+    timeout-minutes: 120
+    env:
+      BENCH_RPC_URL: https://ethereum.reth.rs/rpc
+      SCHELK_MOUNT: /reth-bench
+      RETH_SCOPE: reth-bench.scope
+      BENCH_WORK_DIR: ${{ github.workspace }}/bench-work
+      BENCH_PR: ""
+      BENCH_MODE: ${{ needs.resolve-refs.outputs.mode }}
+      BENCH_ACTOR: "${{ needs.resolve-refs.outputs.mode }}-regression"
+      BENCH_BLOCKS: "2000"
+      BENCH_WARMUP_BLOCKS: "500"
+      BENCH_SAMPLY: "false"
+      BENCH_CORES: "0"
+      BENCH_BIG_BLOCKS: "false"
+      BENCH_WAIT_TIME: ""
+      BENCH_BASELINE_ARGS: ""
+      BENCH_FEATURE_ARGS: ""
+      BENCH_ABBA: "true"
+      BENCH_COMMENT_ID: ""
+      BENCH_SLACK: ${{ github.event_name == 'workflow_dispatch' && inputs.slack || 'always' }}
+      BENCH_METRICS_ADDR: "127.0.0.1:9100"
+      BENCH_OTLP_DISABLED: "true"
+      BASELINE_REF: ${{ needs.resolve-refs.outputs.baseline-ref }}
+      FEATURE_REF: ${{ needs.resolve-refs.outputs.feature-ref }}
+    steps:
+      - name: Clean up previous bench-work
+        run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
+
+      - uses: actions/checkout@v6
+        with:
+          submodules: true
+          fetch-depth: 0
+          ref: ${{ needs.resolve-refs.outputs.feature-ref }}
+
+      - name: Resolve job URL
+        id: job-url
+        uses: actions/github-script@v9
+        with:
+          script: |
+            const { data: jobs } = await github.rest.actions.listJobsForWorkflowRun({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              run_id: context.runId,
+            });
+            const job = jobs.jobs.find(j => j.name === 'bench-scheduled');
+            const jobUrl = job ? job.html_url : `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+            core.exportVariable('BENCH_JOB_URL', jobUrl);
+
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: mozilla-actions/sccache-action@v0.0.9
+        continue-on-error: true
+
+      - name: Install dependencies
+        env:
+          DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
+        run: |
+          mkdir -p "$HOME/.local/bin"
+
+          # apt packages
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends \
+            python3 make jq zstd curl dmsetup \
+            linux-tools-"$(uname -r)" || \
+            sudo apt-get install -y --no-install-recommends linux-tools-generic
+
+          # uv (Python package manager)
+          if ! command -v uv &>/dev/null; then
+            curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$HOME/.local/bin" sh
+          fi
+
+          # Configure git auth for private repos
+          git config --global url."https://x-access-token:${DEREK_TOKEN}@github.com/".insteadOf "https://github.com/"
+
+          # thin-provisioning-tools (era_invalidate, required by schelk)
+          if ! command -v era_invalidate &>/dev/null; then
+            git clone --depth 1 https://github.com/jthornber/thin-provisioning-tools /tmp/tpt
+            sudo make -C /tmp/tpt install
+            rm -rf /tmp/tpt
+          fi
+
+          # schelk (snapshot rollback tool, invoked via sudo)
+          if ! sudo sh -c 'command -v schelk' &>/dev/null; then
+            cargo install --git https://github.com/tempoxyz/schelk --locked
+            sudo install "$HOME/.cargo/bin/schelk" /usr/local/bin/
+          fi
+
+      - name: Check dependencies
+        run: |
+          export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
+          missing=()
+          for cmd in schelk cpupower taskset stdbuf python3 curl make uv jq; do
+            command -v "$cmd" &>/dev/null || missing+=("$cmd")
+          done
+          if [ ${#missing[@]} -gt 0 ]; then
+            echo "::error::Missing required tools: ${missing[*]}"
+            exit 1
+          fi
+          echo "All dependencies found"
+
+      - name: Resolve display names
+        id: refs
+        env:
+          RELEASE_TAG: ${{ needs.resolve-refs.outputs.release-tag }}
+        run: |
+          FEATURE_SHORT=$(echo "$FEATURE_REF" | cut -c1-8)
+          if [ "$BENCH_MODE" = "release" ] && [ -n "$RELEASE_TAG" ]; then
+            echo "baseline-name=${RELEASE_TAG}" >> "$GITHUB_OUTPUT"
+            echo "baseline-ref=${RELEASE_TAG}" >> "$GITHUB_OUTPUT"
+          else
+            BASELINE_SHORT=$(echo "$BASELINE_REF" | cut -c1-8)
+            echo "baseline-name=${BENCH_MODE}-${BASELINE_SHORT}" >> "$GITHUB_OUTPUT"
+            echo "baseline-ref=$BASELINE_REF" >> "$GITHUB_OUTPUT"
+          fi
+          echo "feature-name=${BENCH_MODE}-${FEATURE_SHORT}" >> "$GITHUB_OUTPUT"
+          echo "feature-ref=$FEATURE_REF" >> "$GITHUB_OUTPUT"
+
+      - name: Validate local snapshot
+        id: snapshot-check
+        run: .github/scripts/bench-reth-snapshot.sh
+
+      - name: Prepare source dirs
+        run: |
+          if [ -d ../reth-baseline ]; then
+            git -C ../reth-baseline fetch origin "$BASELINE_REF"
+          else
+            git clone . ../reth-baseline
+          fi
+          git -C ../reth-baseline checkout "$BASELINE_REF"
+
+          if [ -d ../reth-feature ]; then
+            git -C ../reth-feature fetch origin "$FEATURE_REF"
+          else
+            git clone . ../reth-feature
+          fi
+          git -C ../reth-feature checkout "$FEATURE_REF"
+
+      - name: Build binaries
+        id: build
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BENCH_REPO: ${{ github.repository }}
+        run: |
+          BASELINE_DIR="$(cd ../reth-baseline && pwd)"
+          FEATURE_DIR="$(cd ../reth-feature && pwd)"
+
+          .github/scripts/bench-reth-build.sh baseline "${BASELINE_DIR}" "$BASELINE_REF" &
+          PID_BASELINE=$!
+          .github/scripts/bench-reth-build.sh feature "${FEATURE_DIR}" "$FEATURE_REF" &
+          PID_FEATURE=$!
+
+          FAIL=0
+          wait $PID_BASELINE || FAIL=1
+          wait $PID_FEATURE || FAIL=1
+          if [ $FAIL -ne 0 ]; then
+            echo "::error::One or more build tasks failed"
+            exit 1
+          fi
+
+      # System tuning for reproducible benchmarks
+      - name: System setup
+        run: |
+          sudo cpupower frequency-set -g performance || true
+          # Disable turbo boost (Intel and AMD paths)
+          echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
+          echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
+          sudo swapoff -a || true
+          echo 0 | sudo tee /proc/sys/kernel/randomize_va_space || true
+          # Disable SMT (hyperthreading)
+          for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
+            first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
+            current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
+            if [ "$current" != "$first" ]; then
+              echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" || true
+            fi
+          done
+          echo "Online CPUs: $(nproc)"
+          # Disable transparent huge pages
+          for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
+            [ -d "$p" ] && echo never | sudo tee "$p/enabled" && echo never | sudo tee "$p/defrag" && break
+          done || true
+          # Prevent deep C-states
+          sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
+          # Move all IRQs to core 0
+          for irq in /proc/irq/*/smp_affinity_list; do
+            echo 0 | sudo tee "$irq" 2>/dev/null || true
+          done
+          # Stop noisy background services
+          sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
+          echo "=== Benchmark environment ==="
+          uname -r
+          lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA'
+          cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
+          cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
+          cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo "THP: unknown"
+          free -h
+
+      - name: Pre-flight cleanup
+        run: |
+          sudo systemctl stop "$RETH_SCOPE" 2>/dev/null || true
+          sudo systemctl reset-failed "$RETH_SCOPE" 2>/dev/null || true
+          sudo schelk recover -y --kill || sudo schelk full-recover -y || true
+          rm -rf "$BENCH_WORK_DIR"
+          mkdir -p "$BENCH_WORK_DIR"
+
+      - name: Start metrics proxy
+        run: |
+          BENCH_ID="${BENCH_MODE}-${{ github.run_id }}"
+          BENCH_REFERENCE_EPOCH=$(date +%s)
+          echo "BENCH_ID=${BENCH_ID}" >> "$GITHUB_ENV"
+          echo "BENCH_REFERENCE_EPOCH=${BENCH_REFERENCE_EPOCH}" >> "$GITHUB_ENV"
+
+          LABELS_FILE="/tmp/bench-metrics-labels.json"
+          echo '{}' > "$LABELS_FILE"
+          echo "BENCH_LABELS_FILE=${LABELS_FILE}" >> "$GITHUB_ENV"
+
+          python3 .github/scripts/bench-metrics-proxy.py \
+            --labels "$LABELS_FILE" \
+            --upstream "http://${BENCH_METRICS_ADDR}/" \
+            --subnet 10.10.0.0/24 \
+            --port 9090 &
+          PROXY_PID=$!
+          echo "BENCH_METRICS_PROXY_PID=${PROXY_PID}" >> "$GITHUB_ENV"
+          echo "Metrics proxy started (PID $PROXY_PID)"
+
+      # Interleaved run order (B-F-F-B) to reduce systematic bias
+      - name: "Run benchmark: baseline (1/2)"
+        id: run-baseline-1
+        run: |
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
+
+      - name: "Run benchmark: feature (1/2)"
+        id: run-feature-1
+        run: |
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
+
+      - name: "Run benchmark: feature (2/2)"
+        id: run-feature-2
+        run: |
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
+
+      - name: "Run benchmark: baseline (2/2)"
+        id: run-baseline-2
+        run: |
+          LAST_RUN_START=$(date +%s)
+          echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
+
+      - name: Stop metrics proxy & generate Grafana URL
+        id: metrics
+        if: "!cancelled()"
+        run: |
+          kill "$BENCH_METRICS_PROXY_PID" 2>/dev/null || true
+
+          LAST_RUN_DURATION=$(( $(date +%s) - BENCH_LAST_RUN_START ))
+          FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
+          TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
+          GRAFANA_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=ef57fux92e9z4e&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
+          echo "grafana-url=${GRAFANA_URL}" >> "$GITHUB_OUTPUT"
+          echo "Grafana URL: ${GRAFANA_URL}"
+
+      - name: Scan logs for errors
+        if: "!cancelled()"
+        run: |
+          ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
+          found=false
+          for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
+            LOG="$BENCH_WORK_DIR/$run_dir/node.log"
+            if [ ! -f "$LOG" ]; then continue; fi
+
+            panics=$(grep -c -E 'panicked at' "$LOG" || true)
+            errors=$(grep -c ' ERROR ' "$LOG" || true)
+
+            if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
+              if [ "$found" = false ]; then
+                printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
+                found=true
+              fi
+              printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' "$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
+              if [ "$panics" -gt 0 ]; then
+                printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
+                grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
+                printf '```\n' >> "$ERRORS_FILE"
+              fi
+              if [ "$errors" -gt 0 ]; then
+                printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
+                grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
+                printf '```\n' >> "$ERRORS_FILE"
+              fi
+              printf '\n</details>\n\n' >> "$ERRORS_FILE"
+            fi
+          done
+
+      - name: Parse results
+        id: results
+        if: success()
+        env:
+          BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
+          FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
+          BASELINE_REF_DISPLAY: ${{ steps.refs.outputs.baseline-ref }}
+        run: |
+          SUMMARY_ARGS="--output-summary $BENCH_WORK_DIR/summary.json"
+          SUMMARY_ARGS="$SUMMARY_ARGS --output-markdown $BENCH_WORK_DIR/comment.md"
+          SUMMARY_ARGS="$SUMMARY_ARGS --repo ${{ github.repository }}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --baseline-ref ${BASELINE_REF_DISPLAY}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --baseline-name ${BASELINE_NAME}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --feature-name ${FEATURE_NAME}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --feature-ref ${FEATURE_REF}"
+
+          BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
+          FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
+          BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+          FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
+          SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BASELINE_CSVS"
+          SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $FEATURE_CSVS"
+          SUMMARY_ARGS="$SUMMARY_ARGS --gas-csv $BENCH_WORK_DIR/feature-1/total_gas.csv"
+
+          GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
+          if [ -n "$GRAFANA_URL" ]; then
+            SUMMARY_ARGS="$SUMMARY_ARGS --grafana-url $GRAFANA_URL"
+          fi
+          # shellcheck disable=SC2086
+          python3 .github/scripts/bench-reth-summary.py $SUMMARY_ARGS
+
+      - name: Upload to ClickHouse
+        if: success()
+        env:
+          CLICKHOUSE_HOST: ${{ secrets.CLICKHOUSE_HOST }}
+          CLICKHOUSE_USER: ${{ secrets.CLICKHOUSE_USER }}
+          CLICKHOUSE_PASSWORD: ${{ secrets.CLICKHOUSE_PASSWORD }}
+        run: |
+          if [ "$BENCH_MODE" = "release" ]; then
+            WORKFLOW_NAME="workflows-release-regression-${{ github.run_id }}"
+          else
+            WORKFLOW_NAME="workflows-nightly-regression-${{ github.run_id }}"
+          fi
+          DIFF_URL="https://github.com/${{ github.repository }}/compare/${BASELINE_REF}...${FEATURE_REF}"
+          GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
+          JOB_URL="${BENCH_JOB_URL:-${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}}"
+
+          python3 .github/scripts/bench-upload-clickhouse.py \
+            --summary "$BENCH_WORK_DIR/summary.json" \
+            --workflow-name "$WORKFLOW_NAME" \
+            --chain mainnet \
+            --grafana-url "${GRAFANA_URL:-}" \
+            --github-diff-url "$DIFF_URL" \
+            --job-url "$JOB_URL"
+
+      - name: Generate charts
+        if: success() && env.BENCH_MODE != 'hourly'
+        env:
+          BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
+          FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
+        run: |
+          CHART_ARGS="--output-dir $BENCH_WORK_DIR/charts"
+          FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
+          BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
+          FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
+          BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+          CHART_ARGS="$CHART_ARGS --feature $FEATURE_CSVS"
+          CHART_ARGS="$CHART_ARGS --baseline $BASELINE_CSVS"
+          CHART_ARGS="$CHART_ARGS --baseline-name ${BASELINE_NAME}"
+          CHART_ARGS="$CHART_ARGS --feature-name ${FEATURE_NAME}"
+          # shellcheck disable=SC2086
+          uv run --with matplotlib python3 .github/scripts/bench-reth-charts.py $CHART_ARGS
+
+      - name: Upload results
+        if: "!cancelled()"
+        uses: actions/upload-artifact@v7
+        with:
+          name: bench-scheduled-results
+          path: ${{ env.BENCH_WORK_DIR }}
+
+      - name: Push charts
+        id: push-charts
+        if: success() && env.BENCH_MODE != 'hourly'
+        run: |
+          RUN_ID=${{ github.run_id }}
+          CHART_DIR="${BENCH_MODE}/${RUN_ID}"
+          CHARTS_REPO="https://x-access-token:${{ secrets.DEREK_TOKEN }}@github.com/decofe/reth-bench-charts.git"
+
+          TMP_DIR=$(mktemp -d)
+          if git clone --depth 1 "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
+            true
+          else
+            git init "${TMP_DIR}"
+            git -C "${TMP_DIR}" remote add origin "${CHARTS_REPO}"
+          fi
+
+          mkdir -p "${TMP_DIR}/${CHART_DIR}"
+          cp "$BENCH_WORK_DIR"/charts/*.png "${TMP_DIR}/${CHART_DIR}/"
+          git -C "${TMP_DIR}" add "${CHART_DIR}"
+          git -C "${TMP_DIR}" -c user.name="github-actions" -c user.email="github-actions@github.com" \
+            commit -m "nightly bench charts for run ${RUN_ID}"
+          git -C "${TMP_DIR}" push origin HEAD:main
+          echo "sha=$(git -C "${TMP_DIR}" rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+          rm -rf "${TMP_DIR}"
+
+      - name: Write job summary
+        if: success()
+        uses: actions/github-script@v9
+        with:
+          script: |
+            const fs = require('fs');
+            const { verdict, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
+
+            let summary;
+            try {
+              summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+            } catch (e) {
+              await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
+              return;
+            }
+
+            const repo = `${context.repo.owner}/${context.repo.repo}`;
+            const commitUrl = `https://github.com/${repo}/commit`;
+            const { emoji, label } = verdict(summary.changes);
+            const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
+            const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
+            const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+
+            const mode = process.env.BENCH_MODE || 'nightly';
+            const modeLabel = mode === 'hourly' ? 'Hourly Regression' : 'Nightly Regression';
+            let md = `# ${emoji} ${modeLabel}: ${label}\n\n`;
+            md += `**Baseline:** ${baselineLink}\n`;
+            md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
+            md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
+
+            const rows = metricRows(summary);
+            md += `| Metric | Baseline | Feature | Change |\n`;
+            md += `|--------|----------|---------|--------|\n`;
+            for (const r of rows) {
+              md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
+            }
+            md += '\n';
+
+            const wtRows = waitTimeRows(summary);
+            if (wtRows.length > 0) {
+              md += `### Wait Time Breakdown\n\n`;
+              md += `| Metric | Baseline | Feature |\n`;
+              md += `|--------|----------|--------|\n`;
+              for (const r of wtRows) {
+                md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
+              }
+              md += '\n';
+            }
+
+            // Charts
+            const chartSha = '${{ steps.push-charts.outputs.sha }}';
+            if (chartSha) {
+              const runId = '${{ github.run_id }}';
+              const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/nightly/${runId}`;
+              const charts = [
+                { file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
+                { file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
+                { file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
+              ];
+              md += `### Charts\n\n`;
+              for (const chart of charts) {
+                md += `<details><summary>${chart.label}</summary>\n\n`;
+                md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
+                md += `</details>\n\n`;
+              }
+            }
+
+            const grafanaUrl = '${{ steps.metrics.outputs.grafana-url }}';
+            if (grafanaUrl) {
+              md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
+            }
+
+            try {
+              const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
+              if (errors.trim()) md += '\n' + errors + '\n';
+            } catch {}
+
+            await core.summary.addRaw(md).write();
+
+      - name: Send Slack notification (success)
+        if: success() && (env.BENCH_SLACK == 'always' || env.BENCH_SLACK == 'on-win')
+        uses: actions/github-script@v9
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const fs = require('fs');
+            const { verdict, fmtChange, fmtMs, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
+
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) {
+              core.info('Slack credentials not set, skipping notification');
+              return;
+            }
+
+            let summary;
+            try {
+              summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+            } catch (e) {
+              core.warning('Could not read summary.json for Slack notification');
+              return;
+            }
+
+            // Filter notifications based on mode
+            const changes = summary.changes || {};
+            const mode = process.env.BENCH_MODE || 'nightly';
+            const slackMode = process.env.BENCH_SLACK || 'always';
+            const hasRegression = Object.values(changes).some(c => c.sig === 'bad');
+            const hasImprovement = Object.values(changes).some(c => c.sig === 'good');
+
+            // on-win mode: only notify on improvements
+            if (slackMode === 'on-win' && !hasImprovement) {
+              core.info('on-win mode: no improvement detected, skipping Slack notification');
+              return;
+            }
+
+            // Hourly mode: only notify on regressions
+            if (mode === 'hourly' && !hasRegression) {
+              core.info('Hourly mode: no regression detected, skipping Slack notification');
+              return;
+            }
+
+            // Nightly mode: always notify (report every run regardless of significance)
+
+            const SLACK_VERDICT = {
+              '⚠️': ':warning:',
+              '❌': ':x:',
+              '✅': ':white_check_mark:',
+              '⚪': ':white_circle:',
+            };
+
+            const repo = `${context.repo.owner}/${context.repo.repo}`;
+            const { emoji, label } = verdict(changes);
+            const headerEmoji = SLACK_VERDICT[emoji] || emoji;
+            const commitUrl = `https://github.com/${repo}/commit`;
+            const baselineLink = `<${commitUrl}/${summary.baseline.ref}|${summary.baseline.name}>`;
+            const featureLink = `<${commitUrl}/${summary.feature.ref}|${summary.feature.name}>`;
+            const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+            const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+
+            function cell(text) { return { type: 'raw_text', text: String(text) || ' ' }; }
+
+            const modeLabel = mode === 'release' ? 'Release Regression' : mode === 'hourly' ? 'Hourly Regression' : 'Nightly Regression';
+            const sectionText = [
+              `*${modeLabel}*`,
+              '',
+              `*Baseline:* ${baselineLink}`,
+              `*Feature:* ${featureLink}`,
+              blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | '),
+            ].join('\n');
+
+            const rows = metricRows(summary);
+            const tableRows = [
+              [cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
+              ...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
+            ];
+
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: `${headerEmoji} ${modeLabel}: ${label}`, emoji: true },
+              },
+              {
+                type: 'section',
+                text: { type: 'mrkdwn', text: sectionText },
+              },
+              {
+                type: 'table',
+                column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }, { align: 'right' }],
+                rows: tableRows,
+              },
+              {
+                type: 'actions',
+                elements: [
+                  {
+                    type: 'button',
+                    text: { type: 'plain_text', text: 'CI :github:', emoji: true },
+                    url: jobUrl,
+                    action_id: 'ci_button',
+                  },
+                  {
+                    type: 'button',
+                    text: { type: 'plain_text', text: 'Diff :github:', emoji: true },
+                    url: diffUrl,
+                    action_id: 'diff_button',
+                  },
+                ],
+              },
+            ];
+
+            const text = `${modeLabel}: ${summary.baseline.name} vs ${summary.feature.name}`;
+            const resp = await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: {
+                'Authorization': `Bearer ${token}`,
+                'Content-Type': 'application/json',
+              },
+              body: JSON.stringify({ channel, blocks, text, unfurl_links: false }),
+            });
+            const data = await resp.json();
+            if (!data.ok) {
+              core.warning(`Slack API error: ${JSON.stringify(data)}`);
+              return;
+            }
+
+            // Post wait time breakdown as threaded reply
+            const wtRows = waitTimeRows(summary);
+            if (data.ts && wtRows.length > 0) {
+              const waitTableRows = [
+                [cell('Wait Time'), cell('Baseline'), cell('Feature')],
+                ...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
+              ];
+              await fetch('https://slack.com/api/chat.postMessage', {
+                method: 'POST',
+                headers: {
+                  'Authorization': `Bearer ${token}`,
+                  'Content-Type': 'application/json',
+                },
+                body: JSON.stringify({
+                  channel,
+                  thread_ts: data.ts,
+                  blocks: [{
+                    type: 'table',
+                    column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }],
+                    rows: waitTableRows,
+                  }],
+                  text: 'Wait time breakdown',
+                  unfurl_links: false,
+                }),
+              });
+            }
+
+      - name: Send Slack notification (failure)
+        if: failure() && env.BENCH_SLACK != 'never' && env.BENCH_SLACK != 'on-win'
+        uses: actions/github-script@v9
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) return;
+
+            const steps_status = [
+              ['validating local snapshot', '${{ steps.snapshot-check.outcome }}'],
+              ['building binaries', '${{ steps.build.outcome }}'],
+              ['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
+              ['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
+              ['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
+              ['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
+            ];
+            const failed = steps_status.find(([, o]) => o === 'failure');
+            const failedStep = failed ? failed[0] : 'unknown step';
+
+            const repo = `${context.repo.owner}/${context.repo.repo}`;
+            const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+
+            const mode = process.env.BENCH_MODE || 'nightly';
+            const modeLabel = mode === 'release' ? 'Release' : mode === 'hourly' ? 'Hourly' : 'Nightly';
+
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: `:rotating_light: ${modeLabel} Bench Failed`, emoji: true },
+              },
+              {
+                type: 'section',
+                text: { type: 'mrkdwn', text: `*${modeLabel} regression* failed while *${failedStep}*\ncc <@U09FARE0B9Q> <@U09FAL2UMLJ>\n<@U0AAA8F0JEM> investigate this` },
+              },
+              {
+                type: 'actions',
+                elements: [{
+                  type: 'button',
+                  text: { type: 'plain_text', text: 'View Logs :github:', emoji: true },
+                  url: jobUrl,
+                  action_id: 'ci_button',
+                }],
+              },
+            ];
+
+            await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: {
+                'Authorization': `Bearer ${token}`,
+                'Content-Type': 'application/json',
+              },
+              body: JSON.stringify({
+                channel,
+                blocks,
+                text: `${modeLabel} bench failed while ${failedStep}`,
+                unfurl_links: false,
+              }),
+            });
+
+      - name: Clean build outputs
+        if: always()
+        run: |
+          sudo rm -rf ../reth-baseline/target ../reth-feature/target "$BENCH_WORK_DIR" 2>/dev/null || true
+
+      - name: Restore system settings
+        if: always()
+        run: |
+          sudo systemctl start irqbalance cron atd 2>/dev/null || true
+
+  # ---------------------------------------------------------------------------
+  # Job 3: Save state on success
+  # ---------------------------------------------------------------------------
+  save-state:
+    needs: [resolve-refs, bench-scheduled]
+    if: success()
+    name: save-state
+    runs-on: ubuntu-latest
+    steps:
+      - name: Push state to charts repo
+        env:
+          DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
+        run: |
+          MODE="${{ needs.resolve-refs.outputs.mode }}"
+          FEATURE_REF="${{ needs.resolve-refs.outputs.feature-ref }}"
+          CHARTS_REPO="https://x-access-token:${DEREK_TOKEN}@github.com/decofe/reth-bench-charts.git"
+
+          TMP_DIR=$(mktemp -d)
+          if git clone --depth 1 --branch state "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
+            true
+          else
+            git init "${TMP_DIR}"
+            git -C "${TMP_DIR}" remote add origin "${CHARTS_REPO}"
+          fi
+
+          mkdir -p "${TMP_DIR}/state"
+          echo "${FEATURE_REF}" > "${TMP_DIR}/state/${MODE}-last-feature-ref"
+          git -C "${TMP_DIR}" add state/
+          git -C "${TMP_DIR}" diff --cached --quiet && echo "No state change" && exit 0
+          git -C "${TMP_DIR}" -c user.name="github-actions" -c user.email="github-actions@github.com" \
+            commit -m "bench: update ${MODE} state to ${FEATURE_REF}"
+          git -C "${TMP_DIR}" push origin HEAD:state
+          rm -rf "${TMP_DIR}"

.github/workflows/bench.yml

@@ -12,14 +12,29 @@ on:
   workflow_dispatch:
     inputs:
       blocks:
-        description: "Number of blocks to benchmark (or 'big' for big blocks mode)"
+        description: "Number of blocks to benchmark"
         required: false
         default: "500"
         type: string
+      big_blocks:
+        description: "Use big blocks mode (pre-generated merged payloads with reth-bb)"
+        required: false
+        default: "false"
+        type: boolean
+      bal:
+        description: "Replay block access lists during big-block benchmarks"
+        required: false
+        default: "false"
+        type: choice
+        options:
+          - "false"
+          - "true"
+          - "feature"
+          - "baseline"
       warmup:
         description: "Number of warmup blocks"
         required: false
-        default: "100"
+        default: "200"
         type: string
       baseline:
         description: "Baseline git ref (default: merge-base)"
@@ -32,7 +47,7 @@ on:
         default: ""
         type: string
       wait_time:
-        description: "Fixed wait time between blocks (e.g. 500ms, 1s)"
+        description: "Minimum interval between block submissions (e.g. 500ms, 1s)"
         required: false
         default: ""
         type: string
@@ -51,31 +66,34 @@ on:
         required: false
         default: "false"
         type: boolean
-      reth_newPayload:
-        description: "Use reth_newPayload RPC (server-side timing)"
-        required: false
-        default: "true"
-        type: boolean
       cores:
         description: "Limit reth to N CPU cores (0 = all available)"
         required: false
         default: "0"
         type: string
-      no_slack:
-        description: "Suppress Slack notifications for benchmark results"
+      slack:
+        description: "Slack notification policy"
+        required: false
+        default: "never"
+        type: choice
+        options:
+          - always
+          - on-win
+          - on-error
+          - never
+      abba:
+        description: "Run ABBA (BFFB) interleaved order; false = single AB pass"
         required: false
         default: "true"
         type: boolean
-      abba:
-        description: "Run ABBA (BFFB) interleaved order; false = single AB pass"
+      otlp:
+        description: "Export OTLP traces and logs"
         required: false
         default: "true"
         type: boolean
 
 env:
   CARGO_TERM_COLOR: always
-  BASELINE: base
-  SEED: reth
   RUSTC_WRAPPER: "sccache"
   BENCH_RUNNERS: 2
 
@@ -102,19 +120,20 @@ jobs:
       baseline-name: ${{ steps.args.outputs.baseline-name }}
       feature-name: ${{ steps.args.outputs.feature-name }}
       samply: ${{ steps.args.outputs.samply }}
-      no-slack: ${{ steps.args.outputs.no-slack }}
+      slack: ${{ steps.args.outputs.slack }}
       cores: ${{ steps.args.outputs.cores }}
       big-blocks: ${{ steps.args.outputs.big-blocks }}
-      reth-new-payload: ${{ steps.args.outputs.reth-new-payload }}
+      bal: ${{ steps.args.outputs.bal }}
       wait-time: ${{ steps.args.outputs.wait-time }}
       baseline-args: ${{ steps.args.outputs.baseline-args }}
       feature-args: ${{ steps.args.outputs.feature-args }}
       abba: ${{ steps.args.outputs.abba }}
+      otlp: ${{ steps.args.outputs.otlp }}
       comment-id: ${{ steps.ack.outputs.comment-id }}
     steps:
       - name: Check org membership
         if: github.event_name == 'issue_comment'
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |
@@ -133,24 +152,30 @@ jobs:
 
       - name: Parse arguments
         id: args
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
-            let pr, actor, blocks, warmup, baseline, feature, samply, cores, bigBlocks;
+            const validBalModes = new Set(['false', 'true', 'feature', 'baseline']);
+            const validSlackModes = new Set(['always', 'on-win', 'on-error', 'never']);
+            const usage = '`@decofe bench [blocks=N] [big-blocks[=true|false]] [bal=true|false|feature|baseline] [warmup=N] [baseline=REF] [feature=REF] [samply] [slack=always|on-win|on-error|never] [cores=N] [abba=true|false] [otlp=true|false] [wait-time=DURATION] [baseline-args="..."] [feature-args="..."]`';
+            let pr, actor, blocks, warmup, baseline, feature, samply, cores, bigBlocks, bal;
+            let explicitWarmup = false;
 
             if (context.eventName === 'workflow_dispatch') {
               actor = '${{ github.actor }}';
               blocks = '${{ github.event.inputs.blocks }}' || '500';
-              warmup = '${{ github.event.inputs.warmup }}' || '100';
+              warmup = '${{ github.event.inputs.warmup }}' || '200';
+              if (warmup !== '200') explicitWarmup = true;
               baseline = '${{ github.event.inputs.baseline }}';
               feature = '${{ github.event.inputs.feature }}';
               samply = '${{ github.event.inputs.samply }}' === 'true' ? 'true' : 'false';
-              var noSlack = '${{ github.event.inputs.no_slack }}' !== 'false' ? 'true' : 'false';
+              var slack = '${{ github.event.inputs.slack }}' || 'never';
               cores = '${{ github.event.inputs.cores }}' || '0';
-              bigBlocks = blocks === 'big' ? 'true' : 'false';
-              var rethNewPayload = '${{ github.event.inputs.reth_newPayload }}' !== 'false' ? 'true' : 'false';
+              bigBlocks = '${{ github.event.inputs.big_blocks }}' === 'true' ? 'true' : 'false';
+              bal = '${{ github.event.inputs.bal }}' || 'false';
               var abba = '${{ github.event.inputs.abba }}' !== 'false' ? 'true' : 'false';
+              var otlp = '${{ github.event.inputs.otlp }}' !== 'false' ? 'true' : 'false';
               var waitTime = '${{ github.event.inputs.wait_time }}' || '';
               var baselineNodeArgs = '${{ github.event.inputs.baseline_args }}' || '';
               var featureNodeArgs = '${{ github.event.inputs.feature_args }}' || '';
@@ -173,14 +198,14 @@ jobs:
               actor = context.payload.comment.user.login;
 
               const body = context.payload.comment.body.trim();
-              const intArgs = new Set(['warmup', 'cores']);
-              const intOrKeywordArgs = new Map([['blocks', new Set(['big'])]]);
+              const intArgs = new Set(['warmup', 'cores', 'blocks']);
               const refArgs = new Set(['baseline', 'feature']);
-              const boolArgs = new Set(['samply', 'no-slack']);
-              const boolDefaultTrue = new Set(['reth_newPayload', 'abba']);
+              const boolArgs = new Set(['samply', 'big-blocks']);
+              const boolDefaultTrue = new Set(['abba', 'otlp']);
+              const enumArgs = new Map([['bal', validBalModes], ['slack', validSlackModes]]);
               const durationArgs = new Set(['wait-time']);
               const stringArgs = new Set(['baseline-args', 'feature-args']);
-              const defaults = { blocks: '500', warmup: '100', baseline: '', feature: '', samply: 'false', 'no-slack': 'false', cores: '0', reth_newPayload: 'true', abba: 'true', 'wait-time': '', 'baseline-args': '', 'feature-args': '' };
+              const defaults = { blocks: '500', warmup: '200', baseline: '', feature: '', samply: 'false', slack: 'always', 'big-blocks': 'false', bal: 'false', cores: '0', abba: 'true', otlp: 'true', 'wait-time': '', 'baseline-args': '', 'feature-args': '' };
               const unknown = [];
               const invalid = [];
               const args = body.replace(/^(?:@decofe|derek) bench\s*/, '');
@@ -207,7 +232,7 @@ jobs:
                 if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
                   value = value.slice(1, -1);
                 }
-                if (boolDefaultTrue.has(key)) {
+                if (boolArgs.has(key) || boolDefaultTrue.has(key)) {
                   if (value === 'true' || value === 'false') {
                     defaults[key] = value;
                   } else {
@@ -219,20 +244,18 @@ jobs:
                   } else {
                     invalid.push(`\`${key}=${value}\` (must be a duration like 500ms, 1s, 2m)`);
                   }
+                } else if (enumArgs.has(key)) {
+                  if (enumArgs.get(key).has(value)) {
+                    defaults[key] = value;
+                  } else {
+                    invalid.push(`\`${key}=${value}\` (must be true, false, feature, or baseline)`);
+                  }
                 } else if (intArgs.has(key)) {
                   if (!/^\d+$/.test(value)) {
                     invalid.push(`\`${key}=${value}\` (must be a positive integer)`);
                   } else {
                     defaults[key] = value;
-                  }
-                } else if (intOrKeywordArgs.has(key)) {
-                  const keywords = intOrKeywordArgs.get(key);
-                  if (keywords.has(value)) {
-                    defaults[key] = value;
-                  } else if (/^\d+$/.test(value)) {
-                    defaults[key] = value;
-                  } else {
-                    invalid.push(`\`${key}=${value}\` (must be a positive integer or one of: ${[...keywords].join(', ')})`);
+                    if (key === 'warmup') explicitWarmup = true;
                   }
                 } else if (refArgs.has(key)) {
                   if (!value) {
@@ -250,7 +273,7 @@ jobs:
               if (unknown.length) errors.push(`Unknown argument(s): \`${unknown.join('`, `')}\``);
               if (invalid.length) errors.push(`Invalid value(s): ${invalid.join(', ')}`);
               if (errors.length) {
-                const msg = `❌ **Invalid bench command**\n\n${errors.join('\n')}\n\n**Usage:** \`@decofe bench [blocks=N|big] [warmup=N] [baseline=REF] [feature=REF] [samply] [no-slack] [cores=N] [reth_newPayload=true|false] [abba=true|false] [wait-time=DURATION] [baseline-args="..."] [feature-args="..."]\``;
+                const msg = `❌ **Invalid bench command**\n\n${errors.join('\n')}\n\n**Usage:** ${usage}`;
                 await github.rest.issues.createComment({
                   owner: context.repo.owner,
                   repo: context.repo.repo,
@@ -265,16 +288,40 @@ jobs:
               baseline = defaults.baseline;
               feature = defaults.feature;
               samply = defaults.samply;
-              var noSlack = defaults['no-slack'];
+              var slack = defaults.slack;
               cores = defaults.cores;
-              bigBlocks = blocks === 'big' ? 'true' : 'false';
-              var rethNewPayload = defaults.reth_newPayload;
+              bigBlocks = defaults['big-blocks'];
+              bal = defaults.bal;
               var abba = defaults.abba;
+              var otlp = defaults.otlp;
               var waitTime = defaults['wait-time'];
               var baselineNodeArgs = defaults['baseline-args'];
               var featureNodeArgs = defaults['feature-args'];
             }
 
+            // Default warmup to 20 for big-blocks mode unless explicitly set
+            if (bigBlocks === 'true' && !explicitWarmup) {
+              warmup = '20';
+            }
+
+            if (!validBalModes.has(bal)) {
+              core.setFailed(`Invalid bal mode: ${bal}`);
+              return;
+            }
+            if (bal !== 'false' && bigBlocks !== 'true') {
+              const msg = `❌ **Invalid bench command**\n\n\`bal\` requires \`big-blocks=true\`.\n\n**Usage:** ${usage}`;
+              if (context.eventName === 'issue_comment') {
+                await github.rest.issues.createComment({
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  issue_number: context.issue.number,
+                  body: msg,
+                });
+              }
+              core.setFailed(msg);
+              return;
+            }
+
             // Resolve display names for baseline/feature
             let baselineName = baseline || 'main';
             let featureName = feature;
@@ -300,18 +347,19 @@ jobs:
             core.setOutput('baseline-name', baselineName);
             core.setOutput('feature-name', featureName);
             core.setOutput('samply', samply);
-            core.setOutput('no-slack', noSlack);
+            core.setOutput('slack', slack);
             core.setOutput('cores', cores);
             core.setOutput('big-blocks', bigBlocks);
-            core.setOutput('reth-new-payload', rethNewPayload);
+            core.setOutput('bal', bal);
             core.setOutput('wait-time', waitTime);
             core.setOutput('baseline-args', baselineNodeArgs);
             core.setOutput('feature-args', featureNodeArgs);
             core.setOutput('abba', abba);
+            core.setOutput('otlp', otlp);
 
       - name: Acknowledge request
         id: ack
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
@@ -365,16 +413,18 @@ jobs:
             const baseline = '${{ steps.args.outputs.baseline-name }}';
             const feature = '${{ steps.args.outputs.feature-name }}';
             const samply = '${{ steps.args.outputs.samply }}' === 'true';
-            const noSlack = '${{ steps.args.outputs.no-slack }}' === 'true';
+            const slack = '${{ steps.args.outputs.slack }}' || 'always';
             const bigBlocks = '${{ steps.args.outputs.big-blocks }}' === 'true';
+            const bal = '${{ steps.args.outputs.bal }}' || 'false';
             const samplyNote = samply ? ', samply: `enabled`' : '';
-            const noSlackNote = noSlack ? ', no-slack' : '';
+            const slackNote = slack !== 'always' ? `, slack: \`${slack}\`` : '';
+            const balNote = bigBlocks && bal !== 'false' ? `, BAL: \`${bal}\`` : '';
             const cores = '${{ steps.args.outputs.cores }}';
             const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
-            const rethNP = '${{ steps.args.outputs.reth-new-payload }}' !== 'false';
-            const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
             const abbaEnabled = '${{ steps.args.outputs.abba }}' !== 'false';
             const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
+            const otlpEnabled = '${{ steps.args.outputs.otlp }}' !== 'false';
+            const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
             const waitTimeVal = '${{ steps.args.outputs.wait-time }}';
             const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
             const baselineArgsVal = '${{ steps.args.outputs.baseline-args }}';
@@ -382,7 +432,7 @@ jobs:
             const featureArgsVal = '${{ steps.args.outputs.feature-args }}';
             const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
             const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
-            const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
+            const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${slackNote}${balNote}${coresNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
 
             const { data: comment } = await github.rest.issues.createComment({
               owner: context.repo.owner,
@@ -395,7 +445,7 @@ jobs:
 
       - name: Poll queue position
         if: steps.ack.outputs.comment-id && steps.ack.outputs.queue-position != '0'
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
@@ -407,16 +457,18 @@ jobs:
             const baseline = '${{ steps.args.outputs.baseline-name }}';
             const feature = '${{ steps.args.outputs.feature-name }}';
             const samply = '${{ steps.args.outputs.samply }}' === 'true';
-            const noSlack = '${{ steps.args.outputs.no-slack }}' === 'true';
+            const slack = '${{ steps.args.outputs.slack }}' || 'always';
             const bigBlocks = '${{ steps.args.outputs.big-blocks }}' === 'true';
+            const bal = '${{ steps.args.outputs.bal }}' || 'false';
             const samplyNote = samply ? ', samply: `enabled`' : '';
-            const noSlackNote = noSlack ? ', no-slack' : '';
+            const slackNote = slack !== 'always' ? `, slack: \`${slack}\`` : '';
+            const balNote = bigBlocks && bal !== 'false' ? `, BAL: \`${bal}\`` : '';
             const cores = '${{ steps.args.outputs.cores }}';
             const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
-            const rethNP = '${{ steps.args.outputs.reth-new-payload }}' !== 'false';
-            const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
             const abbaEnabled = '${{ steps.args.outputs.abba }}' !== 'false';
             const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
+            const otlpEnabled = '${{ steps.args.outputs.otlp }}' !== 'false';
+            const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
             const waitTimeVal = '${{ steps.args.outputs.wait-time }}';
             const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
             const baselineArgsVal = '${{ steps.args.outputs.baseline-args }}';
@@ -424,7 +476,7 @@ jobs:
             const featureArgsVal = '${{ steps.args.outputs.feature-args }}';
             const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
             const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
-            const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
+            const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${slackNote}${balNote}${coresNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
             const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
 
             const numRunners = parseInt(process.env.BENCH_RUNNERS) || 1;
@@ -481,6 +533,7 @@ jobs:
     env:
       BENCH_RPC_URL: https://ethereum.reth.rs/rpc
       SCHELK_MOUNT: /reth-bench
+      RETH_SCOPE: reth-bench.scope
       BENCH_WORK_DIR: ${{ github.workspace }}/bench-work
       BENCH_PR: ${{ needs.reth-bench-ack.outputs.pr }}
       BENCH_ACTOR: ${{ needs.reth-bench-ack.outputs.actor }}
@@ -489,23 +542,25 @@ jobs:
       BENCH_SAMPLY: ${{ needs.reth-bench-ack.outputs.samply }}
       BENCH_CORES: ${{ needs.reth-bench-ack.outputs.cores }}
       BENCH_BIG_BLOCKS: ${{ needs.reth-bench-ack.outputs.big-blocks }}
-      BENCH_RETH_NEW_PAYLOAD: ${{ needs.reth-bench-ack.outputs.reth-new-payload }}
+      BENCH_BAL: ${{ needs.reth-bench-ack.outputs.bal }}
       BENCH_WAIT_TIME: ${{ needs.reth-bench-ack.outputs.wait-time }}
       BENCH_BASELINE_ARGS: ${{ needs.reth-bench-ack.outputs.baseline-args }}
       BENCH_FEATURE_ARGS: ${{ needs.reth-bench-ack.outputs.feature-args }}
       BENCH_ABBA: ${{ needs.reth-bench-ack.outputs.abba }}
+      BENCH_OTLP: ${{ needs.reth-bench-ack.outputs.otlp }}
       BENCH_COMMENT_ID: ${{ needs.reth-bench-ack.outputs.comment-id }}
-      BENCH_NO_SLACK: ${{ needs.reth-bench-ack.outputs.no-slack }}
+      BENCH_SLACK: ${{ needs.reth-bench-ack.outputs.slack }}
+      BENCH_NODE_BIN: ${{ needs.reth-bench-ack.outputs.big-blocks == 'true' && 'reth-bb' || 'reth' }}
       BENCH_METRICS_ADDR: "127.0.0.1:9100"
-      BENCH_OTLP_TRACES_ENDPOINT: ${{ secrets.BENCH_OTLP_TRACES_ENDPOINT }}
-      BENCH_OTLP_LOGS_ENDPOINT: ${{ secrets.BENCH_OTLP_LOGS_ENDPOINT }}
+      BENCH_OTLP_TRACES_ENDPOINT: ${{ needs.reth-bench-ack.outputs.otlp != 'false' && secrets.BENCH_OTLP_TRACES_ENDPOINT || '' }}
+      BENCH_OTLP_LOGS_ENDPOINT: ${{ needs.reth-bench-ack.outputs.otlp != 'false' && secrets.BENCH_OTLP_LOGS_ENDPOINT || '' }}
     steps:
       - name: Clean up previous bench-work
         run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
 
       - name: Resolve checkout ref
         id: checkout-ref
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           script: |
             if (!process.env.BENCH_PR) {
@@ -531,7 +586,7 @@ jobs:
 
       - name: Resolve job URL and update status
         if: env.BENCH_COMMENT_ID
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
@@ -549,16 +604,18 @@ jobs:
             const baseline = '${{ needs.reth-bench-ack.outputs.baseline-name }}';
             const feature = '${{ needs.reth-bench-ack.outputs.feature-name }}';
             const samply = process.env.BENCH_SAMPLY === 'true';
-            const noSlack = process.env.BENCH_NO_SLACK === 'true';
+            const slack = process.env.BENCH_SLACK || 'always';
             const bigBlocks = process.env.BENCH_BIG_BLOCKS === 'true';
+            const bal = process.env.BENCH_BAL || 'false';
             const samplyNote = samply ? ', samply: `enabled`' : '';
-            const noSlackNote = noSlack ? ', no-slack' : '';
+            const slackNote = slack !== 'always' ? `, slack: \`${slack}\`` : '';
+            const balNote = bigBlocks && bal !== 'false' ? `, BAL: \`${bal}\`` : '';
             const cores = process.env.BENCH_CORES || '0';
             const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
-            const rethNP = (process.env.BENCH_RETH_NEW_PAYLOAD || 'true') !== 'false';
-            const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
             const abbaEnabled = (process.env.BENCH_ABBA || 'true') !== 'false';
             const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
+            const otlpEnabled = (process.env.BENCH_OTLP || 'true') !== 'false';
+            const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
             const waitTimeVal = process.env.BENCH_WAIT_TIME || '';
             const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
             const baselineArgsVal = process.env.BENCH_BASELINE_ARGS || '';
@@ -566,7 +623,7 @@ jobs:
             const featureArgsVal = process.env.BENCH_FEATURE_ARGS || '';
             const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
             const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
-            core.exportVariable('BENCH_CONFIG', `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`);
+            core.exportVariable('BENCH_CONFIG', `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${slackNote}${balNote}${coresNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`);
 
             const { buildBody } = require('./.github/scripts/bench-update-status.js');
             await github.rest.issues.updateComment({
@@ -593,12 +650,6 @@ jobs:
             linux-tools-"$(uname -r)" || \
             sudo apt-get install -y --no-install-recommends linux-tools-generic
 
-          # mc (MinIO client)
-          if ! command -v mc &>/dev/null; then
-            curl -sSfL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$HOME/.local/bin/mc"
-            chmod +x "$HOME/.local/bin/mc"
-          fi
-
           # uv (Python package manager)
           if ! command -v uv &>/dev/null; then
             curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$HOME/.local/bin" sh
@@ -633,7 +684,7 @@ jobs:
           echo "$HOME/.local/bin" >> "$GITHUB_PATH"
           echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
           missing=()
-          for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq; do
+          for cmd in schelk cpupower taskset stdbuf python3 curl make uv jq; do
             command -v "$cmd" &>/dev/null || missing+=("$cmd")
           done
           if [ ${#missing[@]} -gt 0 ]; then
@@ -645,7 +696,7 @@ jobs:
       # Build binaries
       - name: Resolve PR head branch
         id: pr-info
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           script: |
             if (process.env.BENCH_PR) {
@@ -663,7 +714,7 @@ jobs:
 
       - name: Resolve refs
         id: refs
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           script: |
             const { execSync } = require('child_process');
@@ -709,23 +760,45 @@ jobs:
             core.setOutput('feature-ref', featureRef);
             core.setOutput('feature-name', featureName);
 
-      - name: Check if snapshot needs update
-        id: snapshot-check
+      - name: Validate local big blocks
+        if: env.BENCH_BIG_BLOCKS == 'true'
+        id: big-blocks-check
         run: |
-          if .github/scripts/bench-reth-snapshot.sh --check; then
-            echo "needed=false" >> "$GITHUB_OUTPUT"
-          else
-            echo "needed=true" >> "$GITHUB_OUTPUT"
+          set -euo pipefail
+          BIG_BLOCKS_DIR="$HOME/.reth-bench-big-blocks"
+          PAYLOAD_DIR="$BIG_BLOCKS_DIR/payloads"
+          MANIFEST="$BIG_BLOCKS_DIR/manifest.json"
+          echo "BENCH_BIG_BLOCKS_DIR=${BIG_BLOCKS_DIR}" >> "$GITHUB_ENV"
+
+          if [ ! -f "$MANIFEST" ]; then
+            echo "::error::Missing local big-blocks manifest at $MANIFEST"
+            exit 1
           fi
 
-      - name: Update status (snapshot needed)
-        if: env.BENCH_COMMENT_ID && steps.snapshot-check.outputs.needed == 'true'
-        uses: actions/github-script@v8
-        with:
-          github-token: ${{ secrets.DEREK_PAT }}
-          script: |
-            const s = require('./.github/scripts/bench-update-status.js');
-            await s({github, context, status: 'Building binaries & downloading snapshot...'});
+          BASE_SNAPSHOT=$(jq -r '.base_snapshot // empty' "$MANIFEST")
+          if [ -z "$BASE_SNAPSHOT" ]; then
+            echo "::error::Big-blocks manifest missing base_snapshot field"
+            exit 1
+          fi
+
+          if [ ! -d "$PAYLOAD_DIR" ]; then
+            echo "::error::Missing local big-block payload directory at $PAYLOAD_DIR"
+            exit 1
+          fi
+
+          PAYLOAD_COUNT=$(find "$PAYLOAD_DIR" -name '*.json' | wc -l)
+          if [ "$PAYLOAD_COUNT" -eq 0 ]; then
+            echo "::error::No payload files found in $PAYLOAD_DIR"
+            exit 1
+          fi
+
+          echo "Big-blocks base snapshot: $BASE_SNAPSHOT"
+          echo "Payload files: $PAYLOAD_COUNT"
+          echo "BENCH_SNAPSHOT_NAME=${BASE_SNAPSHOT}" >> "$GITHUB_ENV"
+
+      - name: Validate local snapshot
+        id: snapshot-check
+        run: .github/scripts/bench-reth-snapshot.sh
 
       - name: Prepare source dirs
         run: |
@@ -745,12 +818,11 @@ jobs:
           fi
           git -C ../reth-feature checkout "$FEATURE_REF"
 
-      - name: Build binaries and download snapshot in parallel
+      - name: Build binaries
         id: build
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           BENCH_REPO: ${{ github.repository }}
-          SNAPSHOT_NEEDED: ${{ steps.snapshot-check.outputs.needed }}
         run: |
           BASELINE_DIR="$(cd ../reth-baseline && pwd)"
           FEATURE_DIR="$(cd ../reth-feature && pwd)"
@@ -760,28 +832,38 @@ jobs:
           .github/scripts/bench-reth-build.sh feature "${FEATURE_DIR}" "${{ steps.refs.outputs.feature-ref }}" &
           PID_FEATURE=$!
 
-          PID_SNAPSHOT=
-          if [ "$SNAPSHOT_NEEDED" = "true" ]; then
-            .github/scripts/bench-reth-snapshot.sh &
-            PID_SNAPSHOT=$!
-          fi
-
           FAIL=0
           wait $PID_BASELINE || FAIL=1
           wait $PID_FEATURE || FAIL=1
-          [ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
           if [ $FAIL -ne 0 ]; then
-            echo "::error::One or more parallel tasks failed (builds / snapshot download)"
+            echo "::error::One or more build tasks failed"
             exit 1
           fi
 
       # System tuning for reproducible benchmarks
       - name: System setup
         run: |
+          # Switch amd-pstate to passive mode so the kernel governor
+          # controls frequency directly (EPP is ignored in passive).
+          echo passive | sudo tee /sys/devices/system/cpu/amd_pstate/status 2>/dev/null || true
           sudo cpupower frequency-set -g performance || true
-          # Disable turbo boost (Intel and AMD paths)
-          echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
-          echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
+          # Pin all cores to the nominal (base) frequency from CPPC.
+          NOMINAL_KHZ=""
+          if [ -f /sys/devices/system/cpu/cpu0/acpi_cppc/nominal_freq ]; then
+            NOMINAL_MHZ=$(cat /sys/devices/system/cpu/cpu0/acpi_cppc/nominal_freq)
+            NOMINAL_KHZ=$((NOMINAL_MHZ * 1000))
+          elif [ -f /sys/devices/system/cpu/cpu0/cpufreq/base_frequency ]; then
+            NOMINAL_KHZ=$(cat /sys/devices/system/cpu/cpu0/cpufreq/base_frequency)
+          fi
+          if [ -n "$NOMINAL_KHZ" ] && [ "$NOMINAL_KHZ" -gt 0 ] 2>/dev/null; then
+            echo "Pinning all cores to nominal frequency: $((NOMINAL_KHZ / 1000)) MHz"
+            for f in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq; do
+              echo "$NOMINAL_KHZ" | sudo tee "$f" > /dev/null
+            done
+            for f in /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq; do
+              echo "$NOMINAL_KHZ" | sudo tee "$f" > /dev/null
+            done
+          fi
           sudo swapoff -a || true
           echo 0 | sudo tee /proc/sys/kernel/randomize_va_space || true
           # Disable SMT (hyperthreading)
@@ -797,8 +879,11 @@ jobs:
           for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
             [ -d "$p" ] && echo never | sudo tee "$p/enabled" && echo never | sudo tee "$p/defrag" && break
           done || true
+          # Replace any stale PM QoS holders left behind by earlier benchmark jobs.
+          sudo pkill -f '^bench-cpu-dma-latency' 2>/dev/null || true
           # Prevent deep C-states (avoids wake-up latency jitter)
-          sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
+          sudo bash -c 'exec 3<>/dev/cpu_dma_latency; printf "\0\0\0\0" >&3; exec -a bench-cpu-dma-latency sleep infinity' &
+          echo "BENCH_CPU_DMA_LATENCY_PID=$!" >> "$GITHUB_ENV"
           # Move all IRQs to core 0 (housekeeping core)
           for irq in /proc/irq/*/smp_affinity_list; do
             echo 0 | sudo tee "$irq" 2>/dev/null || true
@@ -811,40 +896,20 @@ jobs:
           lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA'
           cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
           cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
+          echo "scaling_min_freq: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq)"
+          echo "scaling_max_freq: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq)"
           cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo "THP: unknown"
           free -h
 
       # Clean up any leftover state
       - name: Pre-flight cleanup
         run: |
-          sudo pkill -9 reth || true
-          sleep 1
-          if mountpoint -q "$SCHELK_MOUNT"; then
-            sudo umount -l "$SCHELK_MOUNT" || true
-            sudo schelk recover -y || true
-          fi
+          sudo systemctl stop "$RETH_SCOPE" 2>/dev/null || true
+          sudo systemctl reset-failed "$RETH_SCOPE" 2>/dev/null || true
+          sudo schelk recover -y --kill || sudo schelk full-recover -y || true
           rm -rf "$BENCH_WORK_DIR"
           mkdir -p "$BENCH_WORK_DIR"
 
-      - name: Download big blocks
-        if: env.BENCH_BIG_BLOCKS == 'true'
-        run: |
-          set -euo pipefail
-          MC="mc --config-dir /home/ubuntu/.mc"
-          BUCKET="minio/reth-snapshots/reth-1-minimal-nightly-previous-big-blocks.tar.zst"
-          BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
-          rm -rf "$BIG_BLOCKS_DIR"; mkdir -p "$BIG_BLOCKS_DIR"
-          echo "Downloading big blocks from $BUCKET..."
-          $MC cat "$BUCKET" | pzstd -d -p 6 | tar -xf - -C "$BIG_BLOCKS_DIR"
-          echo "Big blocks downloaded to $BIG_BLOCKS_DIR"
-          # Verify expected directory structure
-          if [ ! -d "$BIG_BLOCKS_DIR/gas-ramp-dir" ] || [ ! -d "$BIG_BLOCKS_DIR/payloads" ]; then
-            echo "::error::Big blocks archive missing expected gas-ramp-dir/ or payloads/ directories"
-            ls -laR "$BIG_BLOCKS_DIR"
-            exit 1
-          fi
-          echo "Payload files: $(find "$BIG_BLOCKS_DIR/payloads" -name '*.json' | wc -l)"
-
       - name: Start metrics proxy
         run: |
           BENCH_ID="ci-${{ github.run_id }}"
@@ -867,7 +932,7 @@ jobs:
 
       - name: Update status (running benchmarks)
         if: success() && env.BENCH_COMMENT_ID
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
@@ -882,10 +947,12 @@ jobs:
           BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
           OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-1,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
         run: |
+          LAST_RUN_START=$(date +%s)
+          echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
           cat > "$BENCH_LABELS_FILE" <<LABELS
-          {"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          {"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
           LABELS
-          taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
+          taskset -c 0 .github/scripts/bench-reth-run.sh baseline "../reth-baseline/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/baseline-1"
 
       - name: "Run benchmark: feature (1/2)"
         id: run-feature-1
@@ -893,10 +960,12 @@ jobs:
           FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
           OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-1,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
         run: |
+          LAST_RUN_START=$(date +%s)
+          echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
           cat > "$BENCH_LABELS_FILE" <<LABELS
-          {"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          {"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
           LABELS
-          taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
+          taskset -c 0 .github/scripts/bench-reth-run.sh feature "../reth-feature/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/feature-1"
 
       - name: "Run benchmark: feature (2/2)"
         if: env.BENCH_ABBA != 'false'
@@ -905,10 +974,12 @@ jobs:
           FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
           OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-2,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
         run: |
+          LAST_RUN_START=$(date +%s)
+          echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
           cat > "$BENCH_LABELS_FILE" <<LABELS
-          {"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          {"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
           LABELS
-          taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
+          taskset -c 0 .github/scripts/bench-reth-run.sh feature "../reth-feature/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/feature-2"
 
       - name: "Run benchmark: baseline (2/2)"
         if: env.BENCH_ABBA != 'false'
@@ -922,7 +993,7 @@ jobs:
           cat > "$BENCH_LABELS_FILE" <<LABELS
           {"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
           LABELS
-          taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
+          taskset -c 0 .github/scripts/bench-reth-run.sh baseline "../reth-baseline/target/profiling/${BENCH_NODE_BIN}" "$BENCH_WORK_DIR/baseline-2"
 
       - name: Stop metrics proxy & generate Grafana URL
         id: metrics
@@ -1051,11 +1122,15 @@ jobs:
           fi
           if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
             SUMMARY_ARGS="$SUMMARY_ARGS --big-blocks"
-            # Read gas ramp blocks count from first baseline run (same for all runs)
-            GAS_RAMP_FILE="$BENCH_WORK_DIR/baseline-1/gas_ramp_blocks.txt"
-            if [ -f "$GAS_RAMP_FILE" ]; then
-              SUMMARY_ARGS="$SUMMARY_ARGS --gas-ramp-blocks $(cat "$GAS_RAMP_FILE" | tr -d '[:space:]')"
-            fi
+          fi
+          if [ -n "${BENCH_WARMUP_BLOCKS:-}" ]; then
+            SUMMARY_ARGS="$SUMMARY_ARGS --warmup-blocks $BENCH_WARMUP_BLOCKS"
+          fi
+          if [ -n "${BENCH_WAIT_TIME:-}" ]; then
+            SUMMARY_ARGS="$SUMMARY_ARGS --wait-time $BENCH_WAIT_TIME"
+          fi
+          if [ -n "${BENCH_BAL:-}" ] && [ "${BENCH_BAL}" != "false" ]; then
+            SUMMARY_ARGS="$SUMMARY_ARGS --bal-mode $BENCH_BAL"
           fi
           GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
           if [ -n "$GRAFANA_URL" ]; then
@@ -1119,7 +1194,7 @@ jobs:
 
       - name: Compare & comment
         if: success() && env.BENCH_COMMENT_ID
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
@@ -1195,14 +1270,25 @@ jobs:
                 comment_id: parseInt(ackCommentId),
                 body,
               });
-            } else {
-              // No PR — write results to job summary
-              await core.summary.addRaw(body).write();
             }
 
+      - name: Write job summary
+        if: success()
+        uses: actions/github-script@v9
+        with:
+          script: |
+            const jobSummary = require('./.github/scripts/bench-job-summary.js');
+            await jobSummary({
+              core,
+              context,
+              chartSha: '${{ steps.push-charts.outputs.sha }}',
+              grafanaUrl: '${{ steps.metrics.outputs.grafana-url }}',
+              runId: '${{ github.run_id }}',
+            });
+
       - name: Send Slack notification (success)
-        if: success() && env.BENCH_NO_SLACK != 'true'
-        uses: actions/github-script@v8
+        if: success() && (env.BENCH_SLACK == 'always' || env.BENCH_SLACK == 'on-win')
+        uses: actions/github-script@v9
         env:
           SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
           SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
@@ -1213,13 +1299,16 @@ jobs:
 
       - name: Update status (failed)
         if: failure() && env.BENCH_COMMENT_ID
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
             const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
+            const bigBlocks = process.env.BENCH_BIG_BLOCKS === 'true';
             const steps_status = [
-              ['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
+              ...(bigBlocks ? [['validating local big-block data', '${{ steps.big-blocks-check.outcome }}']] : []),
+              ['validating local snapshot', '${{ steps.snapshot-check.outcome }}'],
+              ['building binaries', '${{ steps.build.outcome }}'],
               ['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
               ['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
               ...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
@@ -1245,16 +1334,19 @@ jobs:
             });
 
       - name: Send Slack notification (failure)
-        if: failure()
-        uses: actions/github-script@v8
+        if: failure() && env.BENCH_SLACK != 'never' && env.BENCH_SLACK != 'on-win'
+        uses: actions/github-script@v9
         env:
           SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
           SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
         with:
           script: |
             const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
+            const bigBlocks = process.env.BENCH_BIG_BLOCKS === 'true';
             const steps_status = [
-              ['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
+              ...(bigBlocks ? [['validating local big-block data', '${{ steps.big-blocks-check.outcome }}']] : []),
+              ['validating local snapshot', '${{ steps.snapshot-check.outcome }}'],
+              ['building binaries', '${{ steps.build.outcome }}'],
               ['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
               ['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
               ...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
@@ -1267,7 +1359,7 @@ jobs:
 
       - name: Update status (cancelled)
         if: cancelled() && env.BENCH_COMMENT_ID
-        uses: actions/github-script@v8
+        uses: actions/github-script@v9
         with:
           github-token: ${{ secrets.DEREK_PAT }}
           script: |
@@ -1278,7 +1370,26 @@ jobs:
               body: `cc @${process.env.BENCH_ACTOR}\n\n⚠️ Benchmark cancelled. [View logs](${jobUrl})`,
             });
 
+      - name: Clean build outputs
+        if: always()
+        run: |
+          sudo rm -rf ../reth-baseline/target ../reth-feature/target "$BENCH_WORK_DIR" 2>/dev/null || true
+
       - name: Restore system settings
         if: always()
         run: |
+          # Restore frequency scaling to full range
+          for f in /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq; do
+            cat "$(dirname "$f")/cpuinfo_min_freq" | sudo tee "$f" > /dev/null 2>&1 || true
+          done
+          for f in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq; do
+            cat "$(dirname "$f")/cpuinfo_max_freq" | sudo tee "$f" > /dev/null 2>&1 || true
+          done
+          # Restore amd-pstate to active (EPP) mode with powersave governor
+          echo active | sudo tee /sys/devices/system/cpu/amd_pstate/status 2>/dev/null || true
+          if [ -n "${BENCH_CPU_DMA_LATENCY_PID:-}" ]; then
+            sudo kill "$BENCH_CPU_DMA_LATENCY_PID" 2>/dev/null || true
+          fi
+          sudo pkill -f '^bench-cpu-dma-latency' 2>/dev/null || true
+          sudo cpupower frequency-set -g powersave 2>/dev/null || true
           sudo systemctl start irqbalance cron atd 2>/dev/null || true