Bump revm (712dac7a), alloy-evm (b0eb7e6), revm-inspectors (24becc3)

- Update commit_transaction to return GasOutput per new BlockExecutor trait
- Re-add custom-beacon-withdrawals to workspace members
- Fix example to work with revm rakita/state-gas branch

.changelog/brave-dogs-fly.md

@@ -0,0 +1,10 @@
+---
+reth-engine-primitives: patch
+reth-engine-tree: patch
+reth-node-core: patch
+reth-trie-parallel: minor
+---
+
+Removed legacy proof calculation system and V2-specific configuration flags.
+
+Removed the legacy (non-V2) proof calculation code paths, simplified multiproof task architecture by removing the dual-mode system, and cleaned up V2-specific CLI flags (`--engine.disable-proof-v2`, `--engine.disable-trie-cache`) that are no longer needed. The codebase now exclusively uses V2 proofs with the sparse trie cache.

.changelog/calm-clams-buzz.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: patch
+---
+
+Refactored sparse trie node state tracking to use RLP nodes instead of hashes. Replaced `Option<B256>` hash fields with `SparseNodeState` enum that tracks either dirty nodes or cached RLP nodes with optional database storage flags. Added debug assertions to validate leaf path lengths and improved pruning logic to use node paths directly instead of path-hash tuples.

.changelog/cool-owls-rest.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: patch
+---
+
+Fixed a bug in `merge_subtrie_updates` where source insertions did not cancel destination removals (and vice versa), causing inconsistent trie updates accumulated across multiple `root()` calls without intermediate `take_updates()`. Added a test covering the cross-cancellation behavior.

.changelog/cool-suns-rest.md

@@ -0,0 +1,5 @@
+---
+reth-transaction-pool: minor
+---
+
+Added support for optional custom stateless and stateful validation hooks in `EthTransactionValidator` via `set_additional_stateless_validation` and `set_additional_stateful_validation` methods. Also implemented a manual `Debug` impl to handle the non-`Debug` function pointer fields.

.changelog/dark-ants-write.md

@@ -0,0 +1,5 @@
+---
+reth-tasks: patch
+---
+
+Added panic handler to all rayon thread pools that logs panics via `tracing::error` instead of aborting the process.

.changelog/dry-koalas-wink.md

@@ -0,0 +1,5 @@
+---
+reth-network-types: patch
+---
+
+Increased default maximum concurrent outbound dials from 15 to 30.

.changelog/dry-lakes-sleep.md

@@ -0,0 +1,6 @@
+---
+reth-trie-common: minor
+reth-trie: minor
+---
+
+Added `contains_range` method to `PrefixSet` for checking if any key falls within a half-open range. Added prefix set support to `ProofCalculator` via `with_prefix_set`, enabling stale cached hash invalidation and branch collapse detection when keys are inserted or removed; propagated storage prefix sets through `SyncAccountValueEncoder`.

.changelog/dull-clams-pack.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: patch
+---
+
+Refactored arena trie internals by adding a `BranchChildIdx::sibling()` helper, deduplicating `Index`/`NodeArena` type aliases, and replacing `is_empty()` with a `drop_root()` method. Fixed a bug where `cursor.pop()` was called before checking if the leaf was the root node, which could cause incorrect dirty-state propagation.

.changelog/dull-koalas-play.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: patch
+---
+
+Added recording of `SetRoot` operation in `ParallelSparseTrie::set_root` when the `trie-debug` feature is enabled.

.changelog/dull-seals-laugh.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: minor
+---
+
+Fixed a bug in `ArenaParallelSparseTrie` where subtrie updates that would completely empty a subtrie were incorrectly dispatched to parallel workers instead of being processed inline, preventing correct branch collapse detection when blinded siblings are present. Refactored the `SparseTrie` test suite to accept a `fn() -> T` factory instead of requiring `T: Default`, enabling a new `arena_parallel_sparse_trie_always_parallel` test variant that exercises all tests with parallelism thresholds set to 1. Added `test_branch_collapse_multi_empty_subtries_blinded_remaining` to cover the case where removing multiple revealed leaves empties their subtries and leaves a single blinded sibling requiring a proof.

.changelog/evil-fish-smile.md

@@ -0,0 +1,6 @@
+---
+reth-rpc-convert: minor
+reth-storage-rpc-provider: minor
+---
+
+Replaced the separate `TryFromBlockResponse`, `TryFromReceiptResponse`, and `TryFromTransactionResponse` traits with a unified `RpcResponseConverter` trait and default `EthRpcConverter` implementation. Removed the `op-alloy-network` dependency and refactored `RpcBlockchainProvider` to store a dynamic converter instance instead of relying on per-type trait bounds.

.changelog/fair-wolves-smile.md

@@ -0,0 +1,5 @@
+---
+reth-payload-builder: minor
+---
+
+Added observability metrics for payload resolve latency and new payload job creation latency to the payload builder service.

.changelog/fast-seals-play.md

@@ -0,0 +1,5 @@
+---
+reth-transaction-pool: minor
+---
+
+Added `consensus_ref` method to `PoolTransaction` trait for borrowing consensus transactions without cloning.

.changelog/fine-horses-whisper.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: patch
+---
+
+Fixed another branch collapse edge case where `check_subtrie_collapse_needs_proof` incorrectly compared removal count against total update count (including `Touched` entries), causing it to skip proof requests for blinded siblings and panic when the subtrie emptied. Added a regression test covering the removals + `Touched` + blinded sibling scenario.

.changelog/fine-koalas-shout.md

@@ -0,0 +1,10 @@
+---
+reth-chain-state: minor
+reth-engine-primitives: minor
+reth-engine-tree: minor
+reth-node-core: minor
+reth-node-events: minor
+reth: patch
+---
+
+Added configurable slow block logging (`--engine.slow-block-threshold`) that emits a structured `warn!` log with detailed timing, state-operation counts, and cache hit-rate metrics for blocks whose total processing time exceeds the threshold. Introduced `ExecutionTimingStats`, `CacheStats`, `StateProviderStats`, and `SlowBlockInfo` types to carry execution statistics from block validation through persistence, and refactored `PersistenceResult` to carry commit duration alongside the last persisted block.

.changelog/fix-simulate-revert-code.md

@@ -0,0 +1,5 @@
+---
+reth-rpc-eth-types: patch
+---
+
+Updated `eth_simulateV1` revert error code from `-32000` to `3` to be consistent with `eth_call`, per [execution-apis#748](https://github.com/ethereum/execution-apis/pull/748).

.changelog/icy-tigers-buzz.md

@@ -0,0 +1,6 @@
+---
+reth-trie-db: minor
+reth-engine-tree: minor
+---
+
+Added `PendingChangeset` and `PendingChangesetGuard` to `ChangesetCache` so concurrent readers wait for an in-progress computation instead of falling back to the expensive DB-based path. The guard automatically cancels the pending entry on drop (e.g. task panic), ensuring waiters always make progress.

.changelog/keen-geese-bake.md

@@ -0,0 +1,5 @@
+---
+reth-engine-tree: patch
+---
+
+Added sub-phase timing histograms to the sparse trie event loop, tracking channel wait, proof coalescing, multiproof reveal, and trie update durations separately.

.changelog/merry-bees-break.md

@@ -0,0 +1,8 @@
+---
+reth-engine-primitives: minor
+reth-engine-tree: minor
+reth-node-core: minor
+reth-trie-parallel: minor
+---
+
+Added `--engine.proof-jitter` CLI option behind the `trie-debug` feature flag. When set, each proof worker sleeps for a random duration up to the specified value before starting proof computation, useful for stress-testing timing-sensitive proof logic.

.changelog/merry-koalas-nod.md

@@ -0,0 +1,6 @@
+---
+reth-rpc-eth-api: minor
+reth-rpc-server-types: minor
+---
+
+Added `eth_getStorageValues` RPC method for batch storage slot retrieval across multiple addresses.

.changelog/mild-foxes-cook.md

@@ -0,0 +1,5 @@
+---
+reth-rpc-convert: patch
+---
+
+Updated `alloy-evm` dependency to git revision `9bc2dba` and adapted `TxEnvConverter` impl to the updated `TryIntoTxEnv` trait signature that now includes a `Spec` generic parameter.

.changelog/neat-ducks-whisper.md

@@ -0,0 +1,6 @@
+---
+reth-trie: patch
+reth-trie-sparse: patch
+---
+
+Refactored test harness for sparse trie tests by extracting `TrieTestHarness` into a shared `reth-trie` test utility, replacing duplicated inline harness code across multiple test modules. Updated `proof_v2` return type to include an optional root hash, and converted `original_root` and `storage` from public fields to accessor methods.

.changelog/nice-bears-cook.md

@@ -0,0 +1,5 @@
+---
+reth-trie: patch
+---
+
+Removed the local `increment_and_strip_trailing_zeros` function and `PATH_ALL_ZEROS` static in `proof_v2`, replacing them with the equivalent `Nibbles::next_without_prefix` and `Nibbles::is_zeroes` builtins. Also replaced manual `.get()` calls on `state_mask`/`hash_mask` with direct field access and switched to `Nibbles::unpack_array` over the unsafe `unpack_unchecked`.

.changelog/nice-whales-buzz.md

@@ -0,0 +1,5 @@
+---
+reth-engine-tree: patch
+---
+
+Added idle-time pre-computation of account trie upper hashes in the sparse trie payload processor when no pending proof results are available.

.changelog/odd-donkeys-chirp.md

@@ -0,0 +1,5 @@
+---
+reth-engine-tree: patch
+---
+
+Fixed `compare_trie_updates` to return `bool` indicating whether differences were found, and updated the caller to properly use the return value instead of treating all successful comparisons as having no differences.

.changelog/odd-frogs-break.md

@@ -0,0 +1,7 @@
+---
+reth-cli-commands: minor
+reth-node-core: minor
+reth: patch
+---
+
+Made v2 storage the default for all new databases, deprecating the `--storage.v2` flag to a hidden no-op kept for backwards compatibility. Updated CLI reference docs to remove the now-hidden flag from all command help pages.

.changelog/odd-snails-play.md

@@ -0,0 +1,5 @@
+---
+reth-node-core: minor
+---
+
+Added `with_dev_block_time` helper method to `NodeConfig` for configuring dev miner block production interval.

.changelog/proud-crabs-bark.md

@@ -0,0 +1,5 @@
+---
+reth-db-api: patch
+---
+
+Changed `StoredNibblesSubKey` encoding to use a stack-allocated `[u8; 65]` array instead of a heap-allocated `Vec<u8>`, avoiding unnecessary heap allocation.

.changelog/proud-slugs-fly.md

@@ -0,0 +1,7 @@
+---
+reth-engine-tree: patch
+reth-trie-sparse: patch
+reth-tasks: patch
+---
+
+Offloaded deallocation of expensive proof node buffers to a persistent background thread (`Runtime::spawn_drop`) to avoid blocking state root computation or lock-holding code.

.changelog/proud-wolves-spin.md

@@ -0,0 +1,5 @@
+---
+reth-storage-api: patch
+---
+
+Added `Arc` to `auto_impl` derive for storage-api traits to support automatic `Arc` wrapper implementations.

.changelog/quick-mice-jump.md

@@ -0,0 +1,8 @@
+---
+reth: patch
+reth-engine-tree: patch
+reth-node-builder: patch
+reth-trie-sparse: minor
+---
+
+Added `trie-debug` feature for recording sparse trie mutations to aid in debugging state root mismatches.

.changelog/rare-tigers-nod.md

@@ -0,0 +1,6 @@
+---
+reth-trie-sparse: patch
+reth-engine-tree: patch
+---
+
+Removed the `skip_proof_node_filtering` flag, `revealed_account_paths`/`revealed_paths` tracking, and the `filter_revealed_v2_proof_nodes` function from the sparse trie implementation. Also removed the corresponding skipped-nodes metrics, simplifying the proof node reveal path to always pass nodes directly to the sparse trie without pre-filtering.

.changelog/rare-whales-pack.md

@@ -0,0 +1,5 @@
+---
+reth-trie-sparse: minor
+---
+
+Added a comprehensive generic `SparseTrie` test suite covering `set_root`, `reveal_nodes`, `update_leaves`, `root`, `take_updates`, `commit_updates`, `prune`, `wipe`/`clear`, `get_leaf_value`, `find_leaf`, `size_hint`, and integration lifecycle scenarios. Tests are stamped out for all concrete `SparseTrie` implementations via a macro.

.changelog/rich-lakes-bark.md

@@ -0,0 +1,5 @@
+---
+reth-provider: patch
+---
+
+Fixed sender pruning during block reorg to skip when sender_recovery is fully pruned, preventing a fatal crash when no sender data exists in static files.

.changelog/safe-waves-read.md

@@ -0,0 +1,7 @@
+---
+reth-network-types: minor
+reth-network: minor
+reth-node-core: patch
+---
+
+Added `PersistedPeerInfo` struct to persist richer peer metadata (kind, fork ID, reputation) to disk. Updated `PeersConfig::with_basic_nodes_from_file` to support both the new `PersistedPeerInfo` format and the legacy `Vec<NodeRecord>` format with automatic conversion, and updated `write_peers_to_file` to exclude backed-off and banned peers.

.changelog/tall-fish-climb.md

@@ -0,0 +1,5 @@
+---
+reth-cli-commands: minor
+---
+
+Added `reth_version` field to `SnapshotManifest` to record the Reth version that produced a snapshot. The field is optional and populated automatically during manifest generation.

.changelog/tall-stars-shout.md

@@ -0,0 +1,5 @@
+---
+reth-network: minor
+---
+
+Added `fork_id` as a tiebreaker in peer selection when reputations are equal, preferring peers with a discovered `fork_id` as it indicates fork compatibility. Added a test to verify the tiebreaker behavior.

.changelog/vain-bats-bark.md

@@ -0,0 +1,8 @@
+---
+reth-trie-common: minor
+reth-trie: minor
+reth-trie-parallel: minor
+reth-engine-tree: patch
+---
+
+Moved `ProofV2Target`, `MultiProofTargetsV2`, and `ChunkedMultiProofTargetsV2` from `reth-trie-parallel::targets_v2` into a new `reth-trie-common::target_v2` module, making these types available at a lower level without pulling in the full parallel trie crate. Added a `multiproof_v2` method to `Proof` in `reth-trie` that generates a state multiproof using the V2 proof calculator with synchronous account value encoding.

.changelog/vain-birds-laugh.md

@@ -0,0 +1,14 @@
+---
+reth-engine-tree: patch
+reth-evm-ethereum: patch
+reth-evm: patch
+reth-primitives-traits: patch
+reth-revm: patch
+reth-rpc-eth-api: patch
+reth-rpc-eth-types: patch
+reth-provider: patch
+example-custom-evm: patch
+example-precompile-cache: patch
+---
+
+Bumped revm to v35.0.0, revm-inspectors to 0.35.0, and alloy-evm to 0.29.0. Updated call sites throughout the codebase to align with the new APIs, including `ExecutionResult` field changes (`gas_used` → `gas.used()`/`gas.final_refunded()`), removal of `.without_state_clear()`, updated `EthPrecompiles::new(spec)` constructor, and updated `block_hashes.lowest()` access.

.changelog/vain-goats-cook.md

@@ -0,0 +1,9 @@
+---
+reth-trie-sparse: minor
+reth-engine-primitives: minor
+reth-engine-tree: minor
+reth-node-core: minor
+reth-trie-common: patch
+---
+
+Added an arena-based sparse trie implementation (`ArenaParallelSparseTrie`) using `slotmap` arena allocation for node storage, enabling parallel subtrie mutation without per-node hashing overhead. Added `ConfigurableSparseTrie` enum to switch between the arena and hash-map implementations, and a `--engine.enable-arena-sparse-trie` CLI flag to opt in at runtime.

.changelog/warm-donkeys-dry.md

@@ -0,0 +1,6 @@
+---
+reth-engine-local: minor
+reth-node-builder: minor
+---
+
+Added trigger-based `MiningMode` variant that allows blocks to be built on-demand via custom streams, and exposed `with_mining_mode` method on `DebugNodeLauncherFuture` to override default mining configuration.

.changelog/warm-pandas-cook.md

@@ -0,0 +1,5 @@
+---
+reth-transaction-pool: patch
+---
+
+Fixed a bug where transactions from the same sender were added to the pending subpool out of nonce order. Ensured `process_updates` runs before `add_new_transaction` so that lower-nonce promotions are enqueued before the newly inserted higher-nonce transaction, preserving correct ordering for live `BestTransactions` iterators.

.changelog/young-horses-play.md

@@ -0,0 +1,7 @@
+---
+reth-trie: major
+reth-trie-db: major
+reth-provider: minor
+---
+
+Added `MaskedTrieCursorFactory` and `MaskedTrieCursor` to handle prefix-set-based hash invalidation at the cursor layer, replacing the `DatabaseTrieWitness` trait abstraction. Removed `with_prefix_sets_mut` from `TrieWitness` and deleted `DatabaseTrieWitness` — callers should now wrap their cursor factory with `MaskedTrieCursorFactory` to apply prefix sets during witness/proof computation.

.config/nextest.toml

@@ -1,6 +1,6 @@
 [profile.default]
 retries = { backoff = "exponential", count = 2, delay = "2s", jitter = true }
-slow-timeout = { period = "30s", terminate-after = 4 }
+slow-timeout = { period = "30s", terminate-after = 2 }
 
 [[profile.default.overrides]]
 filter = "test(general_state_tests)"

.dockerignore

@@ -20,6 +20,11 @@
 # include dist directory, where the reth binary is located after compilation
 !/dist
 
+# include PGO build helper used by Dockerfile.depot
+!/.github
+!/.github/scripts
+!/.github/scripts/build_pgo_bolt.sh
+
 # include licenses
 !LICENSE-*
 

.github/CODEOWNERS

@@ -1,7 +1,7 @@
 *                           @gakonst
 crates/chain-state/         @fgimenez @mattsse
 crates/chainspec/           @Rjected @joshieDo @mattsse
-crates/cli/                 @mattsse
+crates/cli/                 @mattsse @Rjected
 crates/config/              @shekhirin @mattsse @Rjected 
 crates/consensus/           @mattsse @Rjected
 crates/e2e-test-utils/      @mattsse @Rjected @klkvr @fgimenez

.github/actionlint.yaml

@@ -5,3 +5,4 @@ self-hosted-runner:
     - depot-ubuntu-latest-4
     - depot-ubuntu-latest-8
     - depot-ubuntu-latest-16
+    - available

.github/scripts/bench-job-summary.js

@@ -0,0 +1,106 @@
+// Generates a rich GitHub Actions job summary for reth-bench results.
+//
+// Reads from environment:
+//   BENCH_WORK_DIR       – Directory containing summary.json
+//   BENCH_PR             – PR number (may be empty)
+//   BENCH_ACTOR          – GitHub user who triggered the bench
+//   BENCH_CORES          – CPU core limit (0 = all)
+//   BENCH_WARMUP_BLOCKS  – Number of warmup blocks
+//   BENCH_SAMPLY         – 'true' if samply profiling was enabled
+//   BENCH_ABBA           – 'true' if ABBA interleaved order was used
+//
+// Usage from actions/github-script:
+//   const jobSummary = require('./.github/scripts/bench-job-summary.js');
+//   await jobSummary({ core, context, chartSha, grafanaUrl, runId });
+
+const fs = require('fs');
+const { verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
+
+module.exports = async function ({ core, context, chartSha, grafanaUrl, runId }) {
+  let summary;
+  try {
+    summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+  } catch (e) {
+    await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
+    return;
+  }
+
+  const repo = `${context.repo.owner}/${context.repo.repo}`;
+  const prNumber = process.env.BENCH_PR;
+  const actor = process.env.BENCH_ACTOR;
+  const commitUrl = `https://github.com/${repo}/commit`;
+
+  const { emoji, label } = verdict(summary.changes);
+  const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
+  const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
+  const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+
+  // Header & metadata
+  const metaParts = [];
+  if (prNumber) metaParts.push(`**[PR #${prNumber}](https://github.com/${repo}/pull/${prNumber})**`);
+  metaParts.push(`triggered by @${actor}`);
+
+  let md = `# ${emoji} ${label}\n\n`;
+  md += metaParts.join(' · ') + '\n\n';
+  md += `**Baseline:** ${baselineLink}\n`;
+  md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
+  md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
+
+  // Main comparison table
+  const rows = metricRows(summary);
+  md += `| Metric | Baseline | Feature | Change |\n`;
+  md += `|--------|----------|---------|--------|\n`;
+  for (const r of rows) {
+    md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
+  }
+  md += '\n';
+
+  // Wait time breakdown
+  const wtRows = waitTimeRows(summary);
+  if (wtRows.length > 0) {
+    md += `### Wait Time Breakdown\n\n`;
+    md += `| Metric | Baseline | Feature |\n`;
+    md += `|--------|----------|--------|\n`;
+    for (const r of wtRows) {
+      md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
+    }
+    md += '\n';
+  }
+
+  // Charts
+  if (chartSha) {
+    const prNum = prNumber || '0';
+    const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/pr/${prNum}/${runId}`;
+    const charts = [
+      { file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
+      { file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
+      { file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
+    ];
+    md += `### Charts\n\n`;
+    for (const chart of charts) {
+      md += `<details><summary>${chart.label}</summary>\n\n`;
+      md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
+      md += `</details>\n\n`;
+    }
+  }
+
+  // Samply profiles
+  const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
+  const samplyLinks = Object.entries(samplyUrls).map(([run, url]) => `- **${run}**: [Firefox Profiler](${url})`);
+  if (samplyLinks.length > 0) {
+    md += `### Samply Profiles\n\n${samplyLinks.join('\n')}\n\n`;
+  }
+
+  // Grafana
+  if (grafanaUrl) {
+    md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
+  }
+
+  // Node errors
+  try {
+    const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
+    if (errors.trim()) md += '\n' + errors + '\n';
+  } catch {}
+
+  await core.summary.addRaw(md).write();
+};

.github/scripts/bench-metrics-proxy.py

@@ -0,0 +1,276 @@
+#!/usr/bin/env python3
+"""
+Prometheus metrics proxy that fetches from a local reth node and
+re-exposes with additional benchmark labels.
+
+Reads labels from a JSON file (updated by local-reth-bench.sh between runs)
+and injects them into every Prometheus metric line.
+
+Returns empty 200 when reth is not running (clean Grafana gaps).
+"""
+import argparse
+import ipaddress
+import json
+import subprocess
+import sys
+import time
+from http.server import HTTPServer, BaseHTTPRequestHandler
+from urllib.request import urlopen
+from urllib.error import URLError
+
+
+def read_labels(path):
+    try:
+        with open(path) as f:
+            return json.load(f)
+    except (FileNotFoundError, json.JSONDecodeError):
+        return {}
+
+
+def inject_labels(metrics_bytes, label_str, label_names):
+    """Inject labels into Prometheus text format.
+
+    Operates on bytes and uses simple string ops instead of regex
+    for speed on large payloads (reth exposes thousands of metrics).
+
+    Skips injecting into lines that already contain any of the label names
+    to avoid duplicate labels (which Prometheus rejects).
+    """
+    if not label_str:
+        return metrics_bytes
+
+    label_bytes = label_str.encode("utf-8")
+    # Pre-encode label names for fast duplicate detection
+    label_name_bytes = [n.encode("utf-8") for n in label_names]
+    out = []
+    for line in metrics_bytes.split(b"\n"):
+        # Skip comments and blank lines
+        if line.startswith(b"#") or not line:
+            out.append(line)
+            continue
+
+        brace = line.find(b"{")
+        space = line.find(b" ")
+
+        if space == -1:
+            # Malformed, pass through
+            out.append(line)
+        elif brace != -1 and brace < space:
+            # Has labels: metric{existing="val"} 123
+            close = line.find(b"}", brace)
+            if close == -1:
+                out.append(line)
+                continue
+
+            # Filter out labels that already exist in this line
+            existing = line[brace + 1:close]
+            inject = label_bytes
+            if existing:
+                for name in label_name_bytes:
+                    if name + b"=" in existing:
+                        # Rebuild inject string excluding this label
+                        inject = _remove_label(inject, name)
+                if not inject:
+                    out.append(line)
+                    continue
+
+            if close == brace + 1:
+                # Empty braces: metric{} 123
+                out.append(line[:close] + inject + line[close:])
+            else:
+                out.append(line[:close] + b"," + inject + line[close:])
+        else:
+            # No labels: metric 123
+            out.append(line[:space] + b"{" + label_bytes + b"}" + line[space:])
+
+    return b"\n".join(out)
+
+
+def _remove_label(label_bytes, name):
+    """Remove a single label (name=\"...\") from a comma-separated label string."""
+    parts = []
+    for part in label_bytes.split(b","):
+        if not part.startswith(name + b"="):
+            parts.append(part)
+    return b",".join(parts)
+
+
+def build_label_str(labels):
+    """Pre-format the label injection string: key1="val1",key2="val2" """
+    if not labels:
+        return ""
+    return ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
+
+
+def build_elapsed_gauge(labels):
+    """Build a bench_elapsed_seconds gauge from run_start_epoch in labels."""
+    start = labels.get("run_start_epoch")
+    if not start:
+        return b""
+    try:
+        elapsed = time.time() - float(start)
+    except (ValueError, TypeError):
+        return b""
+    # Build labels excluding internal keys
+    display = {k: v for k, v in labels.items()
+               if k not in ("run_start_epoch", "reference_epoch")}
+    lstr = build_label_str(display)
+    return (
+        f"# HELP bench_elapsed_seconds Seconds since benchmark run started\n"
+        f"# TYPE bench_elapsed_seconds gauge\n"
+        f"bench_elapsed_seconds{{{lstr}}} {elapsed:.1f}\n"
+    ).encode("utf-8")
+
+
+def compute_timestamp_ms(labels):
+    """Compute a synthetic timestamp so all runs share a common time origin.
+
+    Returns the timestamp in milliseconds, or None if not enough info.
+    Uses: reference_epoch + (now - run_start_epoch) → all runs overlay at
+    the same Grafana time range.
+    """
+    ref = labels.get("reference_epoch")
+    start = labels.get("run_start_epoch")
+    if not ref or not start:
+        return None
+    try:
+        elapsed = time.time() - float(start)
+        return int((float(ref) + elapsed) * 1000)
+    except (ValueError, TypeError):
+        return None
+
+
+def inject_timestamps(metrics_bytes, timestamp_ms):
+    """Append a Prometheus timestamp (ms) to every data line.
+
+    Prometheus text format: metric{labels} value [timestamp_ms]
+    Adding timestamps causes Prometheus to store all runs' samples
+    at the same relative time, enabling natural overlay in Grafana.
+    """
+    if timestamp_ms is None:
+        return metrics_bytes
+
+    ts = str(timestamp_ms).encode("utf-8")
+    out = []
+    for line in metrics_bytes.split(b"\n"):
+        if line.startswith(b"#") or not line:
+            out.append(line)
+        else:
+            out.append(line + b" " + ts)
+    return b"\n".join(out)
+
+
+class MetricsHandler(BaseHTTPRequestHandler):
+    # Use HTTP/1.1 so Content-Length is respected and Prometheus
+    # doesn't have to rely on connection close to detect end of body.
+    protocol_version = "HTTP/1.1"
+
+    def do_GET(self):
+        src = self.client_address[0]
+        try:
+            resp = urlopen(self.server.upstream, timeout=2)
+            metrics = resp.read()
+        except (URLError, ConnectionError, OSError):
+            # reth not running — return empty 200
+            self._send(b"")
+            #print(f"  scrape from {src}: empty (reth not running)", flush=True)
+            return
+
+        all_labels = read_labels(self.server.labels_file)
+        # Internal keys — not injected as Prometheus labels
+        internal = ("run_start_epoch", "reference_epoch")
+        labels = {k: v for k, v in all_labels.items() if k not in internal}
+        label_str = build_label_str(labels)
+        label_names = sorted(labels.keys())
+
+        t0 = time.monotonic()
+        result = inject_labels(metrics, label_str, label_names)
+        result += build_elapsed_gauge(all_labels)
+        ts_ms = compute_timestamp_ms(all_labels)
+        result = inject_timestamps(result, ts_ms)
+        dt = time.monotonic() - t0
+
+        self._send(result)
+        print(f"  scrape from {src}: {len(metrics)} -> {len(result)} bytes, "
+              f"inject {dt*1000:.1f}ms", flush=True)
+
+    def _send(self, body):
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain; version=0.0.4")
+        self.send_header("Content-Length", str(len(body)))
+        self.send_header("Connection", "close")
+        self.end_headers()
+        if body:
+            self.wfile.write(body)
+
+    def log_message(self, format, *args):
+        pass  # suppress per-request logging
+
+
+def resolve_bind_address(subnet_cidr):
+    """Find the local IP address that belongs to the given subnet.
+
+    Uses ``ip -j addr show`` to enumerate interfaces and returns the first
+    address that falls within *subnet_cidr* (e.g. ``10.10.0.0/24``).
+    """
+    network = ipaddress.ip_network(subnet_cidr, strict=False)
+    try:
+        result = subprocess.run(
+            ["ip", "-j", "addr", "show"],
+            capture_output=True, text=True, check=True,
+        )
+        interfaces = json.loads(result.stdout)
+    except (subprocess.CalledProcessError, FileNotFoundError, json.JSONDecodeError) as exc:
+        print(f"Error: cannot enumerate interfaces: {exc}", file=sys.stderr)
+        sys.exit(1)
+
+    for iface in interfaces:
+        for addr_info in iface.get("addr_info", []):
+            try:
+                addr = ipaddress.ip_address(addr_info["local"])
+            except (KeyError, ValueError):
+                continue
+            if addr in network:
+                return str(addr)
+
+    print(f"Error: no interface address found in subnet {subnet_cidr}", file=sys.stderr)
+    sys.exit(1)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Prometheus metrics proxy with label injection")
+    parser.add_argument("--labels", default="/tmp/bench-metrics-labels.json",
+                        help="Path to JSON file with labels to inject (default: /tmp/bench-metrics-labels.json)")
+    parser.add_argument("--upstream", default="http://127.0.0.1:9100/",
+                        help="Upstream reth metrics URL (default: http://127.0.0.1:9100/)")
+
+    bind_group = parser.add_mutually_exclusive_group()
+    bind_group.add_argument("--bind", default=None,
+                            help="Address to bind the proxy (default: 0.0.0.0)")
+    bind_group.add_argument("--subnet", default=None,
+                            help="Auto-detect bind address from a local interface in this subnet (e.g. 10.10.0.0/24)")
+
+    parser.add_argument("--port", type=int, default=9090,
+                        help="Port to bind the proxy (default: 9090)")
+    args = parser.parse_args()
+
+    if args.subnet:
+        bind_addr = resolve_bind_address(args.subnet)
+    elif args.bind:
+        bind_addr = args.bind
+    else:
+        bind_addr = "0.0.0.0"
+
+    server = HTTPServer((bind_addr, args.port), MetricsHandler)
+    server.upstream = args.upstream
+    server.labels_file = args.labels
+
+    print(f"bench-metrics-proxy listening on {bind_addr}:{args.port}")
+    print(f"  upstream: {args.upstream}")
+    print(f"  labels:   {args.labels}")
+    sys.stdout.flush()
+    server.serve_forever()
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/bench-reth-build.sh

@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+#
+# Builds (or fetches from cache) reth binaries for benchmarking.
+#
+# Usage: bench-reth-build.sh <baseline|feature> <source-dir> <commit> [branch-sha]
+#
+#   baseline — build/fetch the baseline binary at <commit> (merge-base)
+#              source-dir must be checked out at <commit>
+#   feature  — build/fetch the candidate binary + reth-bench at <commit>
+#              source-dir must be checked out at <commit>
+#              optional branch-sha is the PR head commit for cache key
+#
+# Outputs:
+#   baseline: <source-dir>/target/profiling/reth
+#   feature:  <source-dir>/target/profiling/reth, reth-bench installed to cargo bin
+#
+# Required: mc (MinIO client) with a configured alias
+set -euo pipefail
+
+MC="mc"
+MODE="$1"
+SOURCE_DIR="$2"
+COMMIT="$3"
+
+# Tracy support: when BENCH_TRACY is "on" or "full", add Tracy cargo features
+# and frame pointers for accurate stack traces.
+EXTRA_FEATURES=""
+EXTRA_RUSTFLAGS=""
+if [ "${BENCH_TRACY:-off}" != "off" ]; then
+  EXTRA_FEATURES="tracy,tracy-client/ondemand"
+  EXTRA_RUSTFLAGS=" -C force-frame-pointers=yes"
+fi
+
+# Cache suffix: hash of features+rustflags so different build configs get separate cache entries
+if [ -n "$EXTRA_FEATURES" ] || [ -n "$EXTRA_RUSTFLAGS" ]; then
+  BUILD_SUFFIX="-$(echo "${EXTRA_FEATURES}${EXTRA_RUSTFLAGS}" | sha256sum | cut -c1-12)"
+else
+  BUILD_SUFFIX=""
+fi
+
+# Verify a cached reth binary was built from the expected commit.
+# `reth --version` outputs "Commit SHA: <full-sha>" on its own line.
+verify_binary() {
+  local binary="$1" expected_commit="$2"
+  local version binary_sha
+  version=$("$binary" --version 2>/dev/null) || return 1
+  binary_sha=$(echo "$version" | sed -n 's/^Commit SHA: *//p')
+  if [ -z "$binary_sha" ]; then
+    echo "Warning: could not extract commit SHA from version output"
+    return 1
+  fi
+  if [ "$binary_sha" = "$expected_commit" ]; then
+    return 0
+  fi
+  echo "Cache mismatch: binary built from ${binary_sha} but expected ${expected_commit}"
+  return 1
+}
+
+case "$MODE" in
+  baseline|main)
+    BUCKET="minio/reth-binaries/${COMMIT}${BUILD_SUFFIX}"
+    mkdir -p "${SOURCE_DIR}/target/profiling"
+
+    CACHE_VALID=false
+    if $MC stat "${BUCKET}/reth" &>/dev/null; then
+      echo "Cache hit for baseline (${COMMIT}), downloading binary..."
+      $MC cp "${BUCKET}/reth" "${SOURCE_DIR}/target/profiling/reth"
+      chmod +x "${SOURCE_DIR}/target/profiling/reth"
+      if verify_binary "${SOURCE_DIR}/target/profiling/reth" "${COMMIT}"; then
+        CACHE_VALID=true
+      else
+        echo "Cached baseline binary is stale, rebuilding..."
+      fi
+    fi
+    if [ "$CACHE_VALID" = false ]; then
+      echo "Building baseline (${COMMIT}) from source..."
+      cd "${SOURCE_DIR}"
+      FEATURES_ARG=""
+      WORKSPACE_ARG=""
+      if [ -n "$EXTRA_FEATURES" ]; then
+        # --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
+        FEATURES_ARG="--features ${EXTRA_FEATURES}"
+        WORKSPACE_ARG="--workspace"
+      fi
+      # shellcheck disable=SC2086
+      RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
+        cargo build --profile profiling --bin reth $WORKSPACE_ARG $FEATURES_ARG
+      $MC cp target/profiling/reth "${BUCKET}/reth"
+    fi
+    ;;
+
+  feature|branch)
+    BRANCH_SHA="${4:-$COMMIT}"
+    BUCKET="minio/reth-binaries/${BRANCH_SHA}${BUILD_SUFFIX}"
+
+    CACHE_VALID=false
+    if $MC stat "${BUCKET}/reth" &>/dev/null && $MC stat "${BUCKET}/reth-bench" &>/dev/null; then
+      echo "Cache hit for ${BRANCH_SHA}, downloading binaries..."
+      mkdir -p "${SOURCE_DIR}/target/profiling"
+      $MC cp "${BUCKET}/reth" "${SOURCE_DIR}/target/profiling/reth"
+      $MC cp "${BUCKET}/reth-bench" /home/ubuntu/.cargo/bin/reth-bench
+      chmod +x "${SOURCE_DIR}/target/profiling/reth" /home/ubuntu/.cargo/bin/reth-bench
+      if verify_binary "${SOURCE_DIR}/target/profiling/reth" "${COMMIT}"; then
+        CACHE_VALID=true
+      else
+        echo "Cached feature binary is stale, rebuilding..."
+      fi
+    fi
+    if [ "$CACHE_VALID" = false ]; then
+      echo "Building feature (${COMMIT}) from source..."
+      cd "${SOURCE_DIR}"
+      rustup show active-toolchain || rustup default stable
+      if [ -n "$EXTRA_FEATURES" ]; then
+        # Can't use `make profiling` when adding features; build explicitly
+        # --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
+        RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
+          cargo build --profile profiling --workspace --bin reth --features "${EXTRA_FEATURES}"
+      else
+        make profiling
+      fi
+      make install-reth-bench
+      $MC cp target/profiling/reth "${BUCKET}/reth"
+      $MC cp "$(which reth-bench)" "${BUCKET}/reth-bench"
+    fi
+    ;;
+
+  *)
+    echo "Usage: $0 <baseline|feature> <source-dir> <commit> [branch-sha]"
+    exit 1
+    ;;
+esac

.github/scripts/bench-reth-charts.py

@@ -0,0 +1,260 @@
+#!/usr/bin/env python3
+"""Generate benchmark charts from reth-bench CSV output.
+
+Usage:
+    bench-engine-charts.py <combined_csv> --output-dir <dir> [--baseline <baseline_csv>]
+
+Generates three PNG charts:
+  1. newPayload latency + Ggas/s per block (+ latency diff when baseline present)
+  2. Wait breakdown (persistence, execution cache, sparse trie) per block
+  3. Scatter plot of gas used vs latency
+
+When --baseline is provided, charts overlay both datasets for comparison.
+"""
+
+import argparse
+import csv
+import sys
+from pathlib import Path
+
+import numpy as np
+
+try:
+    import matplotlib
+
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt
+except ImportError:
+    print("matplotlib is required: pip install matplotlib", file=sys.stderr)
+    sys.exit(1)
+
+GIGAGAS = 1_000_000_000
+
+
+def parse_combined_csv(path: str) -> list[dict]:
+    rows = []
+    with open(path) as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            rows.append(
+                {
+                    "block_number": int(row["block_number"]),
+                    "gas_used": int(row["gas_used"]),
+                    "new_payload_latency_us": int(row["new_payload_latency"]),
+                    "persistence_wait_us": int(row["persistence_wait"])
+                    if row.get("persistence_wait")
+                    else None,
+                    "execution_cache_wait_us": int(row.get("execution_cache_wait", 0)),
+                    "sparse_trie_wait_us": int(row.get("sparse_trie_wait", 0)),
+                }
+            )
+    return rows
+
+
+def plot_latency_and_throughput(
+    feature: list[dict], baseline: list[dict] | None, out: Path,
+    baseline_name: str = "baseline", feature_name: str = "feature",
+):
+    num_plots = 3 if baseline else 2
+    fig, axes = plt.subplots(num_plots, 1, figsize=(12, 4 * num_plots), sharex=True)
+    ax1, ax2 = axes[0], axes[1]
+
+    feat_x = [r["block_number"] for r in feature]
+    feat_lat = [r["new_payload_latency_us"] / 1_000 for r in feature]
+    feat_ggas = []
+    for r in feature:
+        lat_s = r["new_payload_latency_us"] / 1_000_000
+        feat_ggas.append(r["gas_used"] / lat_s / GIGAGAS if lat_s > 0 else 0)
+
+    if baseline:
+        base_x = [r["block_number"] for r in baseline]
+        base_lat = [r["new_payload_latency_us"] / 1_000 for r in baseline]
+        base_ggas = []
+        for r in baseline:
+            lat_s = r["new_payload_latency_us"] / 1_000_000
+            base_ggas.append(r["gas_used"] / lat_s / GIGAGAS if lat_s > 0 else 0)
+        l, = ax1.plot(base_x, base_lat, linewidth=0.8, label=baseline_name, alpha=0.7)
+        ax1.axhline(np.median(base_lat), color=l.get_color(), linestyle="--", linewidth=1, alpha=0.7, label=f"{baseline_name} median")
+        l, = ax2.plot(base_x, base_ggas, linewidth=0.8, label=baseline_name, alpha=0.7)
+        ax2.axhline(np.median(base_ggas), color=l.get_color(), linestyle="--", linewidth=1, alpha=0.7, label=f"{baseline_name} median")
+
+    l, = ax1.plot(feat_x, feat_lat, linewidth=0.8, label=feature_name)
+    ax1.axhline(np.median(feat_lat), color=l.get_color(), linestyle="--", linewidth=1, label=f"{feature_name} median")
+    ax1.set_ylabel("Latency (ms)")
+    ax1.set_title("newPayload Latency per Block")
+    ax1.grid(True, alpha=0.3)
+    ax1.legend()
+
+    l, = ax2.plot(feat_x, feat_ggas, linewidth=0.8, label=feature_name)
+    ax2.axhline(np.median(feat_ggas), color=l.get_color(), linestyle="--", linewidth=1, label=f"{feature_name} median")
+    ax2.set_ylabel("Ggas/s")
+    ax2.set_title("Execution Throughput per Block")
+    ax2.grid(True, alpha=0.3)
+    ax2.legend()
+
+    if baseline:
+        ax3 = axes[2]
+        base_by_block = {r["block_number"]: r["new_payload_latency_us"] for r in baseline}
+        blocks, diffs = [], []
+        for r in feature:
+            bn = r["block_number"]
+            if bn in base_by_block and base_by_block[bn] > 0:
+                pct = (r["new_payload_latency_us"] - base_by_block[bn]) / base_by_block[bn] * 100
+                blocks.append(bn)
+                diffs.append(pct)
+        if blocks:
+            colors = ["green" if d <= 0 else "red" for d in diffs]
+            ax3.bar(blocks, diffs, width=1.0, color=colors, alpha=0.7, edgecolor="none")
+            ax3.axhline(0, color="black", linewidth=0.5)
+        ax3.set_ylabel("Δ Latency (%)")
+        ax3.set_title("Per-Block newPayload Latency Change (feature vs baseline)")
+        ax3.grid(True, alpha=0.3, axis="y")
+
+    axes[-1].set_xlabel("Block Number")
+    fig.tight_layout()
+    fig.savefig(out, dpi=150)
+    plt.close(fig)
+
+
+def plot_wait_breakdown(
+    feature: list[dict], baseline: list[dict] | None, out: Path,
+    baseline_name: str = "baseline", feature_name: str = "feature",
+):
+    series = [
+        ("Persistence Wait", "persistence_wait_us"),
+        ("State Cache Wait", "execution_cache_wait_us"),
+        ("Trie Cache Wait", "sparse_trie_wait_us"),
+    ]
+
+    fig, axes = plt.subplots(len(series), 1, figsize=(12, 3 * len(series)), sharex=True)
+    for ax, (label, key) in zip(axes, series):
+        if baseline:
+            bx = [r["block_number"] for r in baseline if r[key] is not None]
+            by = [r[key] / 1_000 for r in baseline if r[key] is not None]
+            if bx:
+                ax.plot(bx, by, linewidth=0.8, label=baseline_name, alpha=0.7)
+
+        fx = [r["block_number"] for r in feature if r[key] is not None]
+        fy = [r[key] / 1_000 for r in feature if r[key] is not None]
+        if fx:
+            ax.plot(fx, fy, linewidth=0.8, label=feature_name)
+
+        ax.set_ylabel("ms")
+        ax.set_title(label)
+        ax.grid(True, alpha=0.3)
+        if baseline:
+            ax.legend()
+
+    axes[-1].set_xlabel("Block Number")
+    fig.suptitle("Wait Time Breakdown per Block", fontsize=14, y=1.01)
+    fig.tight_layout()
+    fig.savefig(out, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+
+
+def _add_regression(ax, x, y, color, label):
+    """Add a linear regression line to the axes."""
+    if len(x) < 2:
+        return
+    xa, ya = np.array(x), np.array(y)
+    m, b = np.polyfit(xa, ya, 1)
+    x_range = np.linspace(xa.min(), xa.max(), 100)
+    ax.plot(x_range, m * x_range + b, color=color, linewidth=1.5, alpha=0.8,
+            label=label)
+
+
+def plot_gas_vs_latency(
+    feature: list[dict], baseline: list[dict] | None, out: Path,
+    baseline_name: str = "baseline", feature_name: str = "feature",
+):
+    fig, ax = plt.subplots(figsize=(8, 6))
+
+    if baseline:
+        bgas = [r["gas_used"] / 1_000_000 for r in baseline]
+        blat = [r["new_payload_latency_us"] / 1_000 for r in baseline]
+        ax.scatter(bgas, blat, s=8, alpha=0.5)
+        _add_regression(ax, bgas, blat, "tab:blue", baseline_name)
+
+    fgas = [r["gas_used"] / 1_000_000 for r in feature]
+    flat = [r["new_payload_latency_us"] / 1_000 for r in feature]
+    ax.scatter(fgas, flat, s=8, alpha=0.6)
+    _add_regression(ax, fgas, flat, "tab:orange", feature_name)
+
+    ax.set_xlabel("Gas Used (Mgas)")
+    ax.set_ylabel("newPayload Latency (ms)")
+    ax.set_title("Gas Used vs Latency")
+    ax.grid(True, alpha=0.3)
+    ax.legend()
+    fig.tight_layout()
+    fig.savefig(out, dpi=150)
+    plt.close(fig)
+
+
+def merge_csvs(paths: list[str]) -> list[dict]:
+    """Parse and merge multiple CSVs, averaging values for duplicate blocks."""
+    by_block: dict[int, list[dict]] = {}
+    for path in paths:
+        for row in parse_combined_csv(path):
+            by_block.setdefault(row["block_number"], []).append(row)
+
+    merged = []
+    for bn in sorted(by_block):
+        rows = by_block[bn]
+        if len(rows) == 1:
+            merged.append(rows[0])
+        else:
+            avg = {"block_number": bn}
+            for key in ("gas_used", "new_payload_latency_us"):
+                avg[key] = int(sum(r[key] for r in rows) / len(rows))
+            for key in ("persistence_wait_us", "execution_cache_wait_us", "sparse_trie_wait_us"):
+                vals = [r[key] for r in rows if r[key] is not None]
+                avg[key] = int(sum(vals) / len(vals)) if vals else None
+            merged.append(avg)
+    return merged
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate benchmark charts")
+    parser.add_argument(
+        "--feature", nargs="+", required=True,
+        help="Path(s) to feature combined_latency.csv",
+    )
+    parser.add_argument(
+        "--output-dir", required=True, help="Output directory for PNG charts"
+    )
+    parser.add_argument(
+        "--baseline", nargs="+", help="Path(s) to baseline combined_latency.csv"
+    )
+    parser.add_argument("--baseline-name", default="baseline", help="Label for baseline")
+    parser.add_argument("--feature-name", "--branch-name", default="feature", help="Label for feature")
+    args = parser.parse_args()
+
+    feature = merge_csvs(args.feature)
+    if not feature:
+        print("No results found in feature CSV(s)", file=sys.stderr)
+        sys.exit(1)
+
+    baseline = None
+    if args.baseline:
+        baseline = merge_csvs(args.baseline)
+        if not baseline:
+            print(
+                "Warning: no results in baseline CSV(s), skipping comparison",
+                file=sys.stderr,
+            )
+            baseline = None
+
+    out_dir = Path(args.output_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    bname = args.baseline_name
+    fname = args.feature_name
+    plot_latency_and_throughput(feature, baseline, out_dir / "latency_throughput.png", bname, fname)
+    plot_wait_breakdown(feature, baseline, out_dir / "wait_breakdown.png", bname, fname)
+    plot_gas_vs_latency(feature, baseline, out_dir / "gas_vs_latency.png", bname, fname)
+
+    print(f"Charts written to {out_dir}")
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/bench-reth-local.sh

@@ -0,0 +1,581 @@
+#!/usr/bin/env bash
+#
+# local-reth-bench.sh — Run the reth Engine API benchmark locally.
+#
+# Replicates the CI bench.yml workflow (build, snapshot, system tuning,
+# interleaved B-F-F-B execution, summary, charts) without any GitHub
+# Actions glue (no PR comments, no artifact upload, no Slack).
+#
+# Usage:
+#   local-reth-bench.sh <baseline-ref> <feature-ref> [options]
+#
+# Options:
+#   --blocks N      Number of blocks to benchmark (default: 500)
+#   --warmup N      Number of warmup blocks (default: 100)
+#   --cores N       Limit reth to N CPU cores, 0 = all available (default: 0)
+#   --samply        Enable samply profiling
+#   --tracy MODE    Tracy profiling: off, on, full (default: off)
+#   --tracy-filter F Tracy tracing filter (default: debug)
+#   --no-tune       Skip system tuning (useful on dev machines / macOS)
+#
+# Requires: the reth repo at RETH_REPO (default: ~/reth)
+#
+# Dependencies (install before first run):
+#   mc (MinIO client), schelk, cpupower, taskset, stdbuf, python3, curl,
+#   make, uv, pzstd, jq, Rust toolchain (cargo/rustup)
+#
+# The script delegates to the existing bench-reth-*.sh scripts in the reth
+# repo for the actual build, snapshot, and run steps.
+set -euo pipefail
+
+# ── PATH ──────────────────────────────────────────────────────────────
+# Ensure cargo and user-local bins (mc, uv) are visible
+export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
+
+# ── Defaults ──────────────────────────────────────────────────────────
+RETH_REPO="${RETH_REPO:-$HOME/reth}"
+BLOCKS=500
+WARMUP=100
+CORES=0
+SAMPLY=false
+TRACY="off"
+TRACY_FILTER="debug"
+TUNE=true
+BASELINE_REF=""
+FEATURE_REF=""
+
+# ── Parse arguments ──────────────────────────────────────────────────
+usage() {
+  cat <<EOF
+Usage: $(basename "$0") <baseline-ref> <feature-ref> [options]
+
+Options:
+  --blocks N         Number of blocks to benchmark (default: 500)
+  --warmup N         Number of warmup blocks (default: 100)
+  --cores N          Limit reth to N CPU cores (default: 0 = all)
+  --samply           Enable samply profiling
+  --tracy MODE       Tracy profiling: off, on, full (default: off)
+                       on   = tracing only (lower overhead)
+                       full = tracing + CPU sampling (higher overhead)
+  --tracy-filter F   Tracy tracing filter (default: debug)
+  --no-tune          Skip system tuning
+EOF
+  exit 1
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --blocks)       BLOCKS="$2"; shift 2 ;;
+    --warmup)       WARMUP="$2"; shift 2 ;;
+    --cores)        CORES="$2"; shift 2 ;;
+    --samply)       SAMPLY=true; shift ;;
+    --tracy)        TRACY="$2"; shift 2 ;;
+    --tracy-filter) TRACY_FILTER="$2"; shift 2 ;;
+    --no-tune)      TUNE=false; shift ;;
+    --help|-h)  usage ;;
+    -*)         echo "Unknown option: $1"; usage ;;
+    *)
+      if [ -z "$BASELINE_REF" ]; then
+        BASELINE_REF="$1"
+      elif [ -z "$FEATURE_REF" ]; then
+        FEATURE_REF="$1"
+      else
+        echo "Unexpected argument: $1"; usage
+      fi
+      shift
+      ;;
+  esac
+done
+
+if [ -z "$BASELINE_REF" ] || [ -z "$FEATURE_REF" ]; then
+  echo "Error: both <baseline-ref> and <feature-ref> are required."
+  usage
+fi
+
+# Validate --tracy value
+case "$TRACY" in
+  off|on|full) ;;
+  *) echo "Error: --tracy must be off, on, or full (got: $TRACY)"; usage ;;
+esac
+
+# Samply + tracy=full are mutually exclusive (both use perf sampling)
+if [ "$SAMPLY" = "true" ] && [ "$TRACY" = "full" ]; then
+  echo "Warning: samply and tracy=full both use perf sampling; downgrading tracy to 'on'."
+  TRACY="on"
+fi
+
+# ── Check dependencies ───────────────────────────────────────────────
+missing=()
+for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq cargo; do
+  command -v "$cmd" &>/dev/null || missing+=("$cmd")
+done
+if [ ${#missing[@]} -gt 0 ]; then
+  echo "Error: missing required tools: ${missing[*]}"
+  echo "See the CI 'Install dependencies' step in .github/workflows/bench.yml for install instructions."
+  exit 1
+fi
+
+if [ "$TRACY" != "off" ]; then
+  if ! command -v tracy-capture &>/dev/null; then
+    echo "Error: tracy-capture is required for --tracy $TRACY"
+    exit 1
+  fi
+fi
+
+# Ensure tools that run via sudo are in a sudo-visible path.
+# The bench scripts use `sudo schelk` / `sudo samply` but cargo installs
+# them to ~/.cargo/bin which sudo's secure_path doesn't include.
+for cmd in schelk samply; do
+  if command -v "$cmd" &>/dev/null && ! sudo sh -c "command -v $cmd" &>/dev/null; then
+    echo "Installing $cmd to /usr/local/bin (needed for sudo)..."
+    sudo install "$(command -v "$cmd")" /usr/local/bin/
+  fi
+done
+
+if [ ! -d "$RETH_REPO/.git" ]; then
+  echo "Error: RETH_REPO=$RETH_REPO is not a git repository."
+  echo "Set RETH_REPO or clone reth to ~/reth"
+  exit 1
+fi
+
+# ── Resolve paths ────────────────────────────────────────────────────
+SELF_DIR="$(cd "$(dirname "$0")" && pwd)"
+SCRIPTS_DIR="${RETH_REPO}/.github/scripts"
+BENCH_WORK_DIR="${RETH_REPO}/../bench-work-$(date +%Y%m%d-%H%M%S)"
+BASELINE_SRC="${RETH_REPO}/../reth-baseline"
+FEATURE_SRC="${RETH_REPO}/../reth-feature"
+
+mkdir -p "$BENCH_WORK_DIR"
+BENCH_WORK_DIR="$(cd "$BENCH_WORK_DIR" && pwd)"
+
+# ── Global cleanup trap (restores system tuning on any exit) ─────────
+TUNING_APPLIED=false
+CSTATE_PID=
+METRICS_PROXY_PID=
+cleanup_global() {
+  [ -n "$METRICS_PROXY_PID" ] && kill "$METRICS_PROXY_PID" 2>/dev/null || true
+  if [ "$TUNING_APPLIED" = true ]; then
+    echo
+    echo "▸ Restoring system settings..."
+    [ -n "$CSTATE_PID" ] && kill "$CSTATE_PID" 2>/dev/null || true
+    sudo systemctl start irqbalance cron atd 2>/dev/null || true
+    echo "  System settings restored."
+  fi
+}
+trap cleanup_global EXIT
+
+echo "═══════════════════════════════════════════════════════════"
+echo "  reth local benchmark"
+echo "═══════════════════════════════════════════════════════════"
+echo "  Baseline ref : $BASELINE_REF"
+echo "  Feature ref  : $FEATURE_REF"
+echo "  Blocks       : $BLOCKS"
+echo "  Warmup       : $WARMUP"
+echo "  Cores        : $CORES"
+echo "  Samply       : $SAMPLY"
+echo "  Tracy        : $TRACY"
+echo "  Tracy filter : $TRACY_FILTER"
+echo "  System tune  : $TUNE"
+echo "  Work dir     : $BENCH_WORK_DIR"
+echo "  Reth repo    : $RETH_REPO"
+echo "═══════════════════════════════════════════════════════════"
+echo
+
+# Enable sccache if available (matches CI's RUSTC_WRAPPER=sccache)
+if command -v sccache &>/dev/null; then
+  export RUSTC_WRAPPER="sccache"
+fi
+
+# Export env vars expected by the bench-reth-*.sh scripts
+export BENCH_BLOCKS="$BLOCKS"
+export BENCH_WARMUP_BLOCKS="$WARMUP"
+export BENCH_CORES="$CORES"
+export BENCH_SAMPLY="$SAMPLY"
+export BENCH_TRACY="$TRACY"
+export BENCH_TRACY_FILTER="$TRACY_FILTER"
+export BENCH_WORK_DIR
+export SCHELK_MOUNT="${SCHELK_MOUNT:-/reth-bench}"
+export BENCH_RPC_URL="${BENCH_RPC_URL:-https://ethereum.reth.rs/rpc}"
+export BENCH_METRICS_ADDR="127.0.0.1:9100"
+
+# ── Step 1: Resolve refs to full SHAs ────────────────────────────────
+echo "▸ Resolving git refs..."
+cd "$RETH_REPO"
+
+resolve_ref() {
+  local ref="$1"
+  git fetch origin "$ref" --quiet 2>/dev/null || true
+  git rev-parse "$ref" 2>/dev/null \
+    || git rev-parse "origin/$ref" 2>/dev/null \
+    || { echo "Error: cannot resolve ref '$ref'"; exit 1; }
+}
+
+BASELINE_SHA="$(resolve_ref "$BASELINE_REF")"
+FEATURE_SHA="$(resolve_ref "$FEATURE_REF")"
+echo "  Baseline SHA : $BASELINE_SHA"
+echo "  Feature SHA  : $FEATURE_SHA"
+echo
+
+# ── Step 2: Prepare source directories ───────────────────────────────
+echo "▸ Preparing source directories..."
+
+prepare_source() {
+  local src_dir="$1" ref="$2"
+  if [ -d "$src_dir" ]; then
+    git -C "$src_dir" fetch origin "$ref" 2>/dev/null || true
+  else
+    git clone --recurse-submodules "$RETH_REPO" "$src_dir"
+  fi
+  git -C "$src_dir" checkout "$ref" --force
+  git -C "$src_dir" submodule update --init --recursive
+}
+
+prepare_source "$BASELINE_SRC" "$BASELINE_SHA"
+prepare_source "$FEATURE_SRC" "$FEATURE_SHA"
+BASELINE_SRC="$(cd "$BASELINE_SRC" && pwd)"
+FEATURE_SRC="$(cd "$FEATURE_SRC" && pwd)"
+echo "  Baseline src : $BASELINE_SRC"
+echo "  Feature src  : $FEATURE_SRC"
+echo
+
+# ── Step 3: Check / download snapshot ────────────────────────────────
+echo "▸ Checking snapshot..."
+cd "$RETH_REPO"
+SNAPSHOT_NEEDED=false
+if ! "${SCRIPTS_DIR}/bench-reth-snapshot.sh" --check; then
+  SNAPSHOT_NEEDED=true
+  echo "  Snapshot needs update."
+else
+  echo "  Snapshot is up-to-date."
+fi
+echo
+
+# ── Step 4: Build binaries (+ snapshot download) in parallel ─────────
+echo "▸ Building binaries (parallel)..."
+cd "$RETH_REPO"
+
+FAIL=0
+
+"${SCRIPTS_DIR}/bench-reth-build.sh" baseline "$BASELINE_SRC" "$BASELINE_SHA" &
+PID_BASELINE=$!
+
+"${SCRIPTS_DIR}/bench-reth-build.sh" feature "$FEATURE_SRC" "$FEATURE_SHA" &
+PID_FEATURE=$!
+
+PID_SNAPSHOT=
+if [ "$SNAPSHOT_NEEDED" = "true" ]; then
+  echo "  Also downloading snapshot in parallel..."
+  "${SCRIPTS_DIR}/bench-reth-snapshot.sh" &
+  PID_SNAPSHOT=$!
+fi
+
+wait $PID_BASELINE || FAIL=1
+wait $PID_FEATURE  || FAIL=1
+[ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
+
+if [ $FAIL -ne 0 ]; then
+  echo "Error: one or more parallel tasks failed (builds / snapshot)"
+  exit 1
+fi
+echo "  Binaries built successfully."
+echo
+
+# ── Step 5: System tuning (optional) ────────────────────────────────
+if [ "$TUNE" = "true" ]; then
+  echo "▸ Applying system tuning..."
+
+  sudo cpupower frequency-set -g performance 2>/dev/null || true
+
+  # Disable turbo boost (Intel + AMD)
+  echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
+  echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
+
+  sudo swapoff -a 2>/dev/null || true
+  echo 0 | sudo tee /proc/sys/kernel/randomize_va_space 2>/dev/null || true
+
+  # Disable SMT (hyperthreading)
+  for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
+    [ -f "$cpu" ] || continue
+    first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
+    current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
+    if [ "$current" != "$first" ]; then
+      echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" 2>/dev/null || true
+    fi
+  done
+  echo "  Online CPUs: $(nproc)"
+
+  # Disable transparent huge pages
+  for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
+    if [ -d "$p" ]; then
+      echo never | sudo tee "$p/enabled" 2>/dev/null || true
+      echo never | sudo tee "$p/defrag" 2>/dev/null || true
+      break
+    fi
+  done
+
+  # Prevent deep C-states
+  sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
+  CSTATE_PID=$!
+
+  # Pin IRQs to core 0
+  for irq in /proc/irq/*/smp_affinity_list; do
+    echo 0 | sudo tee "$irq" 2>/dev/null || true
+  done
+
+  # Stop noisy background services
+  sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
+
+  TUNING_APPLIED=true
+
+  # Log environment for reproducibility (matches CI)
+  echo "  === Benchmark environment ==="
+  echo "  Kernel : $(uname -r)"
+  lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA' | sed 's/^/  /'
+  echo "  Governor : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo unknown)"
+  echo "  Freq     : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null || echo unknown)"
+  echo "  THP      : $(cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo unknown)"
+  free -h | sed 's/^/  /'
+  echo "  System tuning applied."
+  echo
+fi
+
+# ── Step 5b: Tracefs mount (tracy=full only) ─────────────────────────
+if [ "$TRACY" = "full" ] && [ "$(uname)" = "Linux" ]; then
+  echo "▸ Mounting tracefs for Tracy full mode..."
+  sudo mount -t tracefs tracefs /sys/kernel/tracing -o mode=755 2>/dev/null || true
+fi
+
+# ── Tracy upload & viewer helpers ────────────────────────────────────
+TRACY_VIEWER_BASE="${TRACY_VIEWER_BASE:-}"
+
+tracy_viewer_url() {
+  local profile_url="$1"
+  if [ -z "$TRACY_VIEWER_BASE" ]; then
+    echo ""
+    return
+  fi
+  local encoded
+  encoded=$(python3 -c "import urllib.parse, sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$profile_url")
+  echo "${TRACY_VIEWER_BASE}?profile_url=${encoded}"
+}
+
+upload_tracy() {
+  local label="$1" output_dir="$2" sha="$3"
+  local tracy_file="$output_dir/tracy-profile.tracy"
+
+  if [ ! -f "$tracy_file" ]; then
+    echo "  Tracy: no profile found, skipping upload."
+    return
+  fi
+
+  local timestamp short_sha remote_name bucket mc_alias
+  timestamp=$(date +%Y%m%d-%H%M%S)
+  short_sha="${sha:0:7}"
+  remote_name="${label}-${short_sha}-${timestamp}.tracy"
+  bucket="${TRACY_BUCKET:-tracy-profiles}"
+  mc_alias="${MC_ALIAS:-minio}"
+  local minio_base="${TRACY_MINIO_URL:-http://minio.minio.svc.cluster.local:9000}"
+
+  echo "  Tracy: uploading profile..."
+  if mc cp "$tracy_file" "${mc_alias}/${bucket}/${remote_name}"; then
+    local url="${minio_base}/${bucket}/${remote_name}"
+    echo "$url" > "$output_dir/tracy_url.txt"
+    local viewer
+    viewer=$(tracy_viewer_url "$url")
+    if [ -n "$viewer" ]; then
+      echo "$viewer" > "$output_dir/tracy_viewer_url.txt"
+      echo "  Tracy: uploaded → $viewer"
+    else
+      echo "  Tracy: uploaded → $url"
+    fi
+  else
+    echo "  Tracy: upload failed (non-fatal)."
+  fi
+
+  # Delete large profile to free disk
+  rm -f "$tracy_file"
+}
+
+# ── Step 6: Pre-flight cleanup ───────────────────────────────────────
+echo "▸ Pre-flight cleanup..."
+pkill -f bench-metrics-proxy 2>/dev/null || true
+sudo pkill -9 reth 2>/dev/null || true
+sleep 1
+if mountpoint -q "$SCHELK_MOUNT" 2>/dev/null; then
+  sudo umount -l "$SCHELK_MOUNT" 2>/dev/null || true
+  sudo schelk recover -y 2>/dev/null || true
+fi
+echo
+
+# ── Step 7: Interleaved benchmark runs (B-F-F-B) ────────────────────
+# This ordering reduces systematic bias from thermal drift and cache warming.
+BASELINE_BIN="${BASELINE_SRC}/target/profiling/reth"
+FEATURE_BIN="${FEATURE_SRC}/target/profiling/reth"
+
+# Start metrics proxy (reth → label injection → Prometheus)
+LABELS_FILE="/tmp/bench-metrics-labels.json"
+echo '{}' > "$LABELS_FILE"
+METRICS_SUBNET="${METRICS_SUBNET:-10.10.0.0/24}"
+METRICS_PORT="${METRICS_PORT:-9090}"
+python3 "${SELF_DIR}/bench-metrics-proxy.py" \
+  --labels "$LABELS_FILE" \
+  --upstream "http://${BENCH_METRICS_ADDR}/" \
+  --subnet "$METRICS_SUBNET" \
+  --port "$METRICS_PORT" &
+METRICS_PROXY_PID=$!
+echo "▸ Metrics proxy started (PID $METRICS_PROXY_PID) on subnet ${METRICS_SUBNET}, port ${METRICS_PORT}"
+
+# Unique benchmark ID: local-<timestamp> for local runs, ci-<run_id> for CI
+BENCH_ID="local-$(basename "$BENCH_WORK_DIR" | sed 's/bench-work-//')"
+# Reference epoch: shared time origin so all runs overlay in Grafana.
+# The proxy maps each run's elapsed time onto this common origin.
+BENCH_REFERENCE_EPOCH=$(date +%s)
+
+write_labels() {
+  local run_label="$1" run_type="$2" ref="$3" sha="$4"
+  LAST_RUN_START=$(date +%s)
+  cat > "$LABELS_FILE" <<-EOF
+	{"benchmark_run":"${run_label}","run_type":"${run_type}","git_ref":"${ref}","bench_sha":"${sha}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+	EOF
+}
+
+run_bench() {
+  local label="$1" binary="$2" output_dir="$3"
+  echo "▸ Running benchmark: ${label}..."
+  cd "$RETH_REPO"
+  if command -v taskset &>/dev/null; then
+    taskset -c 0 "${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
+  else
+    "${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
+  fi
+  echo "  ✓ ${label} complete."
+  echo
+}
+
+write_labels "baseline-1" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
+run_bench "baseline-1" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-1"
+
+write_labels "feature-1" "feature" "$FEATURE_REF" "$FEATURE_SHA"
+run_bench "feature-1"  "$FEATURE_BIN"  "$BENCH_WORK_DIR/feature-1"
+
+write_labels "feature-2" "feature" "$FEATURE_REF" "$FEATURE_SHA"
+run_bench "feature-2"  "$FEATURE_BIN"  "$BENCH_WORK_DIR/feature-2"
+
+write_labels "baseline-2" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
+run_bench "baseline-2" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-2"
+
+# ── Compute Grafana URL ──────────────────────────────────────────────
+GRAFANA_BASE_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr"
+GRAFANA_DATASOURCE="ef57fux92e9z4e"
+LAST_RUN_DURATION=$(( $(date +%s) - LAST_RUN_START ))
+FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
+TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
+GRAFANA_URL="${GRAFANA_BASE_URL}?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=${GRAFANA_DATASOURCE}&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
+
+# ── Step 8: Scan logs for errors ─────────────────────────────────────
+echo "▸ Scanning logs for errors..."
+ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
+found_errors=false
+for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
+  LOG="$BENCH_WORK_DIR/$run_dir/node.log"
+  [ -f "$LOG" ] || continue
+  panics=$(grep -c -E 'panicked at' "$LOG" 2>/dev/null || true)
+  errors=$(grep -c ' ERROR ' "$LOG" 2>/dev/null || true)
+  if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
+    if [ "$found_errors" = false ]; then
+      printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
+      found_errors=true
+    fi
+    printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' \
+      "$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
+    if [ "$panics" -gt 0 ]; then
+      printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
+      grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
+      printf '```\n' >> "$ERRORS_FILE"
+    fi
+    if [ "$errors" -gt 0 ]; then
+      printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
+      grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
+      printf '```\n' >> "$ERRORS_FILE"
+    fi
+    printf '\n</details>\n\n' >> "$ERRORS_FILE"
+  fi
+done
+if [ "$found_errors" = true ]; then
+  echo "  ⚠ Errors found — see $ERRORS_FILE"
+else
+  echo "  No errors found."
+fi
+echo
+
+# ── Step 9: Parse results ───────────────────────────────────────────
+echo "▸ Parsing results..."
+cd "$RETH_REPO"
+
+SUMMARY_ARGS=(
+  --output-summary "$BENCH_WORK_DIR/summary.json"
+  --output-markdown "$BENCH_WORK_DIR/comment.md"
+  --repo "paradigmxyz/reth"
+  --baseline-ref "$BASELINE_SHA"
+  --baseline-name "$BASELINE_REF"
+  --feature-name "$FEATURE_REF"
+  --feature-ref "$FEATURE_SHA"
+  --baseline-csv "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+  --feature-csv "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
+  --gas-csv "$BENCH_WORK_DIR/feature-1/total_gas.csv"
+  --grafana-url "$GRAFANA_URL"
+)
+
+python3 "${SCRIPTS_DIR}/bench-reth-summary.py" "${SUMMARY_ARGS[@]}"
+echo
+
+# ── Step 10: Generate charts ─────────────────────────────────────────
+echo "▸ Generating charts..."
+CHART_ARGS=(
+  --output-dir "$BENCH_WORK_DIR/charts"
+  --feature "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
+  --baseline "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+  --baseline-name "$BASELINE_REF"
+  --feature-name "$FEATURE_REF"
+)
+
+if python3 -c "import matplotlib" 2>/dev/null; then
+  python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
+elif command -v uv &>/dev/null; then
+  uv run --with matplotlib python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
+else
+  echo "  Warning: matplotlib not available, skipping chart generation."
+fi
+echo
+
+# ── Step 11: Upload Tracy profiles ────────────────────────────────────
+if [ "$TRACY" != "off" ]; then
+  echo "▸ Uploading Tracy profiles..."
+  upload_tracy "baseline-1" "$BENCH_WORK_DIR/baseline-1" "$BASELINE_SHA"
+  upload_tracy "feature-1"  "$BENCH_WORK_DIR/feature-1"  "$FEATURE_SHA"
+  upload_tracy "feature-2"  "$BENCH_WORK_DIR/feature-2"  "$FEATURE_SHA"
+  upload_tracy "baseline-2" "$BENCH_WORK_DIR/baseline-2" "$BASELINE_SHA"
+  echo
+fi
+
+# ── Done (system restore happens via EXIT trap) ─────────────────────
+echo "═══════════════════════════════════════════════════════════"
+echo "  Benchmark complete!"
+echo "═══════════════════════════════════════════════════════════"
+echo "  Results  : $BENCH_WORK_DIR/summary.json"
+echo "  Markdown : $BENCH_WORK_DIR/comment.md"
+echo "  Charts   : $BENCH_WORK_DIR/charts/"
+if [ -f "$ERRORS_FILE" ]; then
+  echo "  Errors   : $ERRORS_FILE"
+fi
+echo "  Grafana  : $GRAFANA_URL"
+if [ "$TRACY" != "off" ]; then
+  echo "  ─── Tracy Profiles ───"
+  for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
+    url_file="$BENCH_WORK_DIR/$run_dir/tracy_viewer_url.txt"
+    if [ -f "$url_file" ]; then
+      echo "  $run_dir : $(cat "$url_file")"
+    fi
+  done
+fi
+echo "═══════════════════════════════════════════════════════════"

.github/scripts/bench-reth-run.sh

@@ -0,0 +1,273 @@
+#!/usr/bin/env bash
+#
+# Runs a single reth-bench cycle: mount snapshot → start node → warmup →
+# benchmark → stop node → recover snapshot.
+#
+# Usage: bench-reth-run.sh <label> <binary> <output-dir>
+#
+# Required env: SCHELK_MOUNT, BENCH_RPC_URL, BENCH_BLOCKS, BENCH_WARMUP_BLOCKS
+# Optional env: BENCH_BIG_BLOCKS (true/false), BENCH_WORK_DIR (for big blocks path)
+#               BENCH_RETH_NEW_PAYLOAD (true/false, default true)
+#               BENCH_WAIT_TIME (duration like 500ms, default empty)
+#               BENCH_BASELINE_ARGS (extra reth node args for baseline runs)
+#               BENCH_FEATURE_ARGS (extra reth node args for feature runs)
+#               BENCH_OTLP_TRACES_ENDPOINT (OTLP HTTP endpoint for traces, e.g. https://host/insert/opentelemetry/v1/traces)
+#               BENCH_OTLP_LOGS_ENDPOINT (OTLP HTTP endpoint for logs, e.g. https://host/insert/opentelemetry/v1/logs)
+set -euo pipefail
+
+LABEL="$1"
+BINARY="$2"
+OUTPUT_DIR="$3"
+DATADIR="$SCHELK_MOUNT/datadir"
+mkdir -p "$OUTPUT_DIR"
+LOG="${OUTPUT_DIR}/node.log"
+
+cleanup() {
+  kill "$TAIL_PID" 2>/dev/null || true
+  # Stop tracy-capture first (SIGINT makes it disconnect and flush to disk)
+  # Must happen before killing reth, otherwise reth keeps streaming data.
+  if [ -n "${TRACY_PID:-}" ] && kill -0 "$TRACY_PID" 2>/dev/null; then
+    echo "Stopping tracy-capture..."
+    kill -INT "$TRACY_PID" 2>/dev/null || true
+    for i in $(seq 1 30); do
+      kill -0 "$TRACY_PID" 2>/dev/null || break
+      if [ $((i % 10)) -eq 0 ]; then
+        echo "Waiting for tracy-capture to finish writing... (${i}s)"
+      fi
+      sleep 1
+    done
+    if kill -0 "$TRACY_PID" 2>/dev/null; then
+      echo "tracy-capture still running after 30s, killing..."
+      kill -9 "$TRACY_PID" 2>/dev/null || true
+    fi
+    wait "$TRACY_PID" 2>/dev/null || true
+  fi
+  if [ -n "${RETH_PID:-}" ] && sudo kill -0 "$RETH_PID" 2>/dev/null; then
+    if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
+      # Send SIGINT to the inner reth process by exact name (not -f which
+      # would also match samply's cmdline containing "reth"). Samply will
+      # capture reth's exit and save the profile.
+      sudo pkill -INT -x reth 2>/dev/null || true
+      # Wait for samply to finish writing the profile and exit
+      for i in $(seq 1 120); do
+        sudo pgrep -x samply > /dev/null 2>&1 || break
+        if [ $((i % 10)) -eq 0 ]; then
+          echo "Waiting for samply to finish writing profile... (${i}s)"
+        fi
+        sleep 1
+      done
+      if sudo pgrep -x samply > /dev/null 2>&1; then
+        echo "Samply still running after 120s, sending SIGTERM..."
+        sudo pkill -x samply 2>/dev/null || true
+      fi
+    else
+      sudo kill "$RETH_PID"
+      for i in $(seq 1 30); do
+        sudo kill -0 "$RETH_PID" 2>/dev/null || break
+        sleep 1
+      done
+    fi
+    sudo kill -9 "$RETH_PID" 2>/dev/null || true
+    sleep 1
+  fi
+  # Fix ownership of reth-created files (reth runs as root)
+  sudo chown -R "$(id -un):$(id -gn)" "$OUTPUT_DIR" 2>/dev/null || true
+  if mountpoint -q "$SCHELK_MOUNT"; then
+    sudo umount -l "$SCHELK_MOUNT" || true
+    sudo schelk recover -y || true
+  fi
+}
+TAIL_PID=
+TRACY_PID=
+trap cleanup EXIT
+
+# Clean up stale schelk state from a previous cancelled run.
+# If schelk thinks it's still mounted (e.g. a cancelled run skipped cleanup),
+# recover first to reset state.
+sudo schelk recover -y -k || true
+
+# Mount
+sudo schelk mount -y
+sync
+sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
+echo "=== Cache state after drop ==="
+free -h
+grep Cached /proc/meminfo
+
+# Start reth
+# CPU layout: core 0 = OS/IRQs/reth-bench/aux, cores 1+ = reth node
+RETH_BENCH="$(which reth-bench)"
+ONLINE=$(nproc --all)
+MAX_RETH=$(( ONLINE - 1 ))
+if [ "${BENCH_CORES:-0}" -gt 0 ] && [ "$BENCH_CORES" -lt "$MAX_RETH" ]; then
+  MAX_RETH=$BENCH_CORES
+fi
+RETH_CPUS="1-${MAX_RETH}"
+
+BIG_BLOCKS="${BENCH_BIG_BLOCKS:-false}"
+
+RETH_ARGS=(
+  node
+  --datadir "$DATADIR"
+  --log.file.directory "$OUTPUT_DIR/reth-logs"
+  --engine.accept-execution-requests-hash
+  --http
+  --http.port 8545
+  --ws
+  --ws.api all
+  --authrpc.port 8551
+  --disable-discovery
+  --no-persist-peers
+)
+
+# Big blocks mode requires the testing API and skip-invalid-transactions
+if [ "$BIG_BLOCKS" = "true" ]; then
+  RETH_ARGS+=(--http.api eth,net,web3,reth,testing --testing.skip-invalid-transactions)
+fi
+
+# Append per-label extra node args (baseline or feature)
+EXTRA_NODE_ARGS=""
+case "$LABEL" in
+  baseline*) EXTRA_NODE_ARGS="${BENCH_BASELINE_ARGS:-}" ;;
+  feature*)  EXTRA_NODE_ARGS="${BENCH_FEATURE_ARGS:-}" ;;
+esac
+if [ -n "$EXTRA_NODE_ARGS" ]; then
+  # Word-split the string into individual args
+  # shellcheck disable=SC2206
+  RETH_ARGS+=($EXTRA_NODE_ARGS)
+fi
+
+if [ -n "${BENCH_METRICS_ADDR:-}" ]; then
+  RETH_ARGS+=(--metrics "$BENCH_METRICS_ADDR")
+fi
+
+# OTLP traces and logs export
+if [ -n "${BENCH_OTLP_TRACES_ENDPOINT:-}" ]; then
+  RETH_ARGS+=(--tracing-otlp="${BENCH_OTLP_TRACES_ENDPOINT}" --tracing-otlp.service-name=reth-bench)
+fi
+if [ -n "${BENCH_OTLP_LOGS_ENDPOINT:-}" ]; then
+  RETH_ARGS+=(--logs-otlp="${BENCH_OTLP_LOGS_ENDPOINT}" --logs-otlp.filter=debug)
+fi
+
+# Tracy profiling: add --log.tracy flags and set environment
+if [ "${BENCH_TRACY:-off}" != "off" ]; then
+  RETH_ARGS+=(--log.tracy --log.tracy.filter "${BENCH_TRACY_FILTER:-debug}")
+  if [ "${BENCH_TRACY}" = "on" ]; then
+    export TRACY_NO_SYS_TRACE=1
+  elif [ "${BENCH_TRACY}" = "full" ]; then
+    export TRACY_SAMPLING_HZ="${BENCH_TRACY_SAMPLING_HZ:-1}"
+  fi
+fi
+
+SUDO_ENV=()
+if [ -n "${OTEL_RESOURCE_ATTRIBUTES:-}" ]; then
+  SUDO_ENV+=("OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES}")
+  SUDO_ENV+=("OTEL_BSP_MAX_QUEUE_SIZE=65536" "OTEL_BLRP_MAX_QUEUE_SIZE=65536")
+fi
+
+# Limit reth memory to 95% of available RAM to prevent OOM kills
+TOTAL_MEM_KB=$(awk '/^MemTotal:/ {print $2}' /proc/meminfo)
+MEM_LIMIT=$(( TOTAL_MEM_KB * 95 / 100 * 1024 ))
+echo "Memory limit: $(( MEM_LIMIT / 1024 / 1024 ))MB (95% of $(( TOTAL_MEM_KB / 1024 ))MB)"
+
+if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
+  RETH_ARGS+=(--log.samply)
+  SAMPLY="$(which samply)"
+  sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
+    env "${SUDO_ENV[@]}" nice -n -20 \
+    "$SAMPLY" record --save-only --presymbolicate --rate 10000 \
+    --output "$OUTPUT_DIR/samply-profile.json.gz" \
+    -- "$BINARY" "${RETH_ARGS[@]}" \
+    > "$LOG" 2>&1 &
+else
+  sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
+    env "${SUDO_ENV[@]}" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
+    > "$LOG" 2>&1 &
+fi
+
+RETH_PID=$!
+stdbuf -oL tail -f "$LOG" | sed -u "s/^/[reth] /" &
+TAIL_PID=$!
+
+for i in $(seq 1 60); do
+  if curl -sf http://127.0.0.1:8545 -X POST \
+    -H 'Content-Type: application/json' \
+    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
+    > /dev/null 2>&1; then
+    echo "reth (${LABEL}) is ready after ${i}s"
+    break
+  fi
+  if [ "$i" -eq 60 ]; then
+    echo "::error::reth (${LABEL}) failed to start within 60s"
+    cat "$LOG"
+    exit 1
+  fi
+  sleep 1
+done
+
+# Run reth-bench with high priority but as the current user so output
+# files are not root-owned (avoids EACCES on next checkout).
+BENCH_NICE="sudo nice -n -20 sudo -u $(id -un)"
+
+# Build optional flags
+EXTRA_BENCH_ARGS=()
+if [ "${BENCH_RETH_NEW_PAYLOAD:-true}" != "false" ]; then
+  EXTRA_BENCH_ARGS+=(--reth-new-payload)
+fi
+if [ -n "${BENCH_WAIT_TIME:-}" ]; then
+  EXTRA_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
+fi
+
+if [ "$BIG_BLOCKS" = "true" ]; then
+  # Big blocks mode: replay pre-generated payloads with gas ramp
+  BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
+  # Count gas ramp blocks for reporting
+  GAS_RAMP_COUNT=$(find "$BIG_BLOCKS_DIR/gas-ramp-dir" -name '*.json' | wc -l)
+  echo "$GAS_RAMP_COUNT" > "$OUTPUT_DIR/gas_ramp_blocks.txt"
+  echo "Gas ramp blocks: $GAS_RAMP_COUNT"
+
+  # Start tracy-capture so profile only covers the benchmark
+  if [ "${BENCH_TRACY:-off}" != "off" ]; then
+    echo "Starting tracy-capture..."
+    tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
+    TRACY_PID=$!
+    sleep 0.5  # give tracy-capture time to connect
+  fi
+
+  echo "Running big blocks benchmark (replay-payloads)..."
+  $BENCH_NICE "$RETH_BENCH" replay-payloads \
+    "${EXTRA_BENCH_ARGS[@]}" \
+    --gas-ramp-dir "$BIG_BLOCKS_DIR/gas-ramp-dir" \
+    --payload-dir "$BIG_BLOCKS_DIR/payloads" \
+    --engine-rpc-url http://127.0.0.1:8551 \
+    --jwt-secret "$DATADIR/jwt.hex" \
+    --output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
+else
+  # Standard mode: warmup + new-payload-fcu
+  # Warmup
+  $BENCH_NICE "$RETH_BENCH" new-payload-fcu \
+    --rpc-url "$BENCH_RPC_URL" \
+    --engine-rpc-url http://127.0.0.1:8551 \
+    --jwt-secret "$DATADIR/jwt.hex" \
+    --advance "${BENCH_WARMUP_BLOCKS:-50}" \
+    "${EXTRA_BENCH_ARGS[@]}" 2>&1 | sed -u "s/^/[bench] /"
+
+  # Start tracy-capture after warmup so profile only covers the benchmark
+  if [ "${BENCH_TRACY:-off}" != "off" ]; then
+    echo "Starting tracy-capture..."
+    tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
+    TRACY_PID=$!
+    sleep 0.5  # give tracy-capture time to connect
+  fi
+
+  # Benchmark
+  $BENCH_NICE "$RETH_BENCH" new-payload-fcu \
+    --rpc-url "$BENCH_RPC_URL" \
+    --engine-rpc-url http://127.0.0.1:8551 \
+    --jwt-secret "$DATADIR/jwt.hex" \
+    --advance "$BENCH_BLOCKS" \
+    "${EXTRA_BENCH_ARGS[@]}" \
+    --output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
+fi
+
+# cleanup runs via trap

.github/scripts/bench-reth-snapshot.sh

@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+#
+# Downloads the latest nightly snapshot into the schelk volume with
+# progress reporting to the GitHub PR comment.
+#
+# Skips the download if the local ETag marker matches the remote one.
+#
+# Usage: bench-reth-snapshot.sh [--check]
+#   --check   Only check if a download is needed; exits 0 if up-to-date, 1 if not.
+#
+# Required env:
+#   SCHELK_MOUNT       – schelk mount point (e.g. /reth-bench)
+#   GITHUB_TOKEN       – token for GitHub API calls (only for download)
+#   BENCH_COMMENT_ID   – PR comment ID to update (optional)
+#   BENCH_REPO         – owner/repo (e.g. paradigmxyz/reth)
+#   BENCH_JOB_URL      – link to the Actions job
+#   BENCH_ACTOR        – user who triggered the benchmark
+#   BENCH_CONFIG       – config summary line
+set -euo pipefail
+
+BUCKET="minio/reth-snapshots/reth-1-minimal-nightly-previous.tar.zst"
+DATADIR="$SCHELK_MOUNT/datadir"
+ETAG_FILE="$HOME/.reth-bench-snapshot-etag"
+
+# Get remote metadata via JSON for reliable parsing
+MC_STAT=$(mc stat --json "$BUCKET" 2>/dev/null || true)
+REMOTE_ETAG=$(echo "$MC_STAT" | jq -r '.etag // empty')
+if [ -z "$REMOTE_ETAG" ]; then
+  echo "::warning::Failed to get ETag from mc stat, will re-download"
+  REMOTE_ETAG="unknown-$(date +%s)"
+fi
+
+LOCAL_ETAG=""
+[ -f "$ETAG_FILE" ] && LOCAL_ETAG=$(cat "$ETAG_FILE")
+
+if [ "$REMOTE_ETAG" = "$LOCAL_ETAG" ]; then
+  echo "Snapshot is up-to-date (ETag: ${REMOTE_ETAG})"
+  if [ "${1:-}" = "--check" ]; then
+    exit 0
+  fi
+  exit 0
+fi
+
+echo "Snapshot needs update (local: ${LOCAL_ETAG:-<none>}, remote: ${REMOTE_ETAG})"
+if [ "${1:-}" = "--check" ]; then
+  exit 1
+fi
+
+# Get compressed size for progress tracking
+TOTAL_BYTES=$(echo "$MC_STAT" | jq -r '.size // empty')
+if [ -z "$TOTAL_BYTES" ] || [ "$TOTAL_BYTES" = "0" ]; then
+  echo "::error::Failed to get snapshot size from mc stat"
+  exit 1
+fi
+echo "Snapshot size: $TOTAL_BYTES bytes ($(numfmt --to=iec "$TOTAL_BYTES"))"
+
+# Prepare mount
+mountpoint -q "$SCHELK_MOUNT" && sudo schelk recover -y || true
+sudo schelk mount -y
+sudo rm -rf "$DATADIR"
+sudo mkdir -p "$DATADIR"
+
+update_comment() {
+  local pct="$1"
+  [ -z "${BENCH_COMMENT_ID:-}" ] && return 0
+  local status="Building binaries & downloading snapshot… ${pct}%"
+  local body
+  body="$(printf 'cc @%s\n\n🚀 Benchmark started! [View job](%s)\n\n⏳ **Status:** %s\n\n%s' \
+    "$BENCH_ACTOR" "$BENCH_JOB_URL" "$status" "$BENCH_CONFIG")"
+  curl -sf -X PATCH \
+    -H "Authorization: token ${GITHUB_TOKEN}" \
+    -H "Accept: application/vnd.github.v3+json" \
+    "https://api.github.com/repos/${BENCH_REPO}/issues/comments/${BENCH_COMMENT_ID}" \
+    -d "$(jq -nc --arg body "$body" '{body: $body}')" \
+    > /dev/null 2>&1 || true
+}
+
+# Track compressed bytes flowing through the pipe
+DL_BYTES_FILE=$(mktemp)
+echo 0 > "$DL_BYTES_FILE"
+
+# Start progress reporter in background
+(
+  while true; do
+    sleep 10
+    CURRENT=$(cat "$DL_BYTES_FILE" 2>/dev/null || echo 0)
+    if [ "$TOTAL_BYTES" -gt 0 ]; then
+      PCT=$(( CURRENT * 100 / TOTAL_BYTES ))
+      [ "$PCT" -gt 100 ] && PCT=100
+      echo "Snapshot download: $(numfmt --to=iec "$CURRENT") / $(numfmt --to=iec "$TOTAL_BYTES") (${PCT}%)"
+      update_comment "$PCT"
+    fi
+  done
+) &
+PROGRESS_PID=$!
+trap 'kill $PROGRESS_PID 2>/dev/null || true; rm -f "$DL_BYTES_FILE"' EXIT
+
+# Download and extract; python byte counter tracks compressed bytes received
+mc cat "$BUCKET" | python3 -c "
+import sys
+count = 0
+while True:
+    data = sys.stdin.buffer.read(1048576)
+    if not data:
+        break
+    count += len(data)
+    sys.stdout.buffer.write(data)
+    with open('$DL_BYTES_FILE', 'w') as f:
+        f.write(str(count))
+" | pzstd -d -p 6 | sudo tar -xf - -C "$DATADIR"
+
+# Stop progress reporter
+kill $PROGRESS_PID 2>/dev/null || true
+wait $PROGRESS_PID 2>/dev/null || true
+
+update_comment "100"
+echo "Snapshot download complete"
+
+# Promote the new snapshot to become the schelk baseline (virgin volume).
+# This copies changed blocks from scratch → virgin so that future
+# `schelk recover` calls restore to this new state.
+sync
+sudo schelk promote -y
+
+# Save ETag marker
+echo "$REMOTE_ETAG" > "$ETAG_FILE"
+echo "Snapshot promoted to schelk baseline (ETag: ${REMOTE_ETAG})"

.github/scripts/bench-reth-summary.py

@@ -0,0 +1,592 @@
+#!/usr/bin/env python3
+"""Parse reth-bench CSV output and generate a summary JSON + markdown comparison.
+
+Usage:
+    bench-reth-summary.py <combined_csv> <gas_csv> \
+        --output-summary <summary.json> \
+        --output-markdown <comment.md> \
+        --baseline-csv <baseline_combined.csv> \
+        [--repo <owner/repo>] \
+        [--baseline-ref <sha>] \
+        [--feature-name <name>] \
+        [--feature-sha <sha>]
+
+Generates a paired statistical comparison between baseline and feature.
+Matches blocks by number and computes per-block diffs to cancel out gas
+variance. Fails if baseline or feature CSV is missing or empty.
+"""
+
+import argparse
+import csv
+import json
+import math
+import random
+import sys
+
+GIGAGAS = 1_000_000_000
+T_CRITICAL = 1.96  # two-tailed 95% confidence
+BOOTSTRAP_ITERATIONS = 10_000
+
+
+def _opt_int(row: dict, key: str) -> int | None:
+    """Return int value for a CSV field, or None if missing/empty."""
+    v = row.get(key)
+    if v is None or v == "":
+        return None
+    return int(v)
+
+
+def parse_combined_csv(path: str) -> list[dict]:
+    """Parse combined_latency.csv into a list of per-block dicts."""
+    rows = []
+    with open(path) as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            rows.append(
+                {
+                    "block_number": int(row["block_number"]),
+                    "gas_used": int(row["gas_used"]),
+                    "gas_limit": int(row["gas_limit"]),
+                    "transaction_count": int(row["transaction_count"]),
+                    "new_payload_latency_us": int(row["new_payload_latency"]),
+                    "fcu_latency_us": int(row["fcu_latency"]),
+                    "total_latency_us": int(row["total_latency"]),
+                    "persistence_wait_us": _opt_int(row, "persistence_wait"),
+                    "execution_cache_wait_us": _opt_int(row, "execution_cache_wait"),
+                    "sparse_trie_wait_us": _opt_int(row, "sparse_trie_wait"),
+                }
+            )
+    return rows
+
+
+def parse_gas_csv(path: str) -> list[dict]:
+    """Parse total_gas.csv into a list of per-block dicts."""
+    rows = []
+    with open(path) as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            rows.append(
+                {
+                    "block_number": int(row["block_number"]),
+                    "gas_used": int(row["gas_used"]),
+                    "time_us": int(row["time"]),
+                }
+            )
+    return rows
+
+
+def stddev(values: list[float], mean: float) -> float:
+    if len(values) < 2:
+        return 0.0
+    return math.sqrt(sum((v - mean) ** 2 for v in values) / (len(values) - 1))
+
+
+def percentile(sorted_vals: list[float], pct: int) -> float:
+    if not sorted_vals:
+        return 0.0
+    idx = int(len(sorted_vals) * pct / 100)
+    idx = min(idx, len(sorted_vals) - 1)
+    return sorted_vals[idx]
+
+
+def compute_stats(combined: list[dict]) -> dict:
+    """Compute per-run statistics from parsed CSV data."""
+    n = len(combined)
+    if n == 0:
+        return {}
+
+    latencies_ms = [r["new_payload_latency_us"] / 1_000 for r in combined]
+    sorted_lat = sorted(latencies_ms)
+    mean_lat = sum(latencies_ms) / n
+    std_lat = stddev(latencies_ms, mean_lat)
+
+    mgas_s_values = []
+    for r in combined:
+        lat_s = r["new_payload_latency_us"] / 1_000_000
+        if lat_s > 0:
+            mgas_s_values.append(r["gas_used"] / lat_s / 1_000_000)
+    mean_mgas_s = sum(mgas_s_values) / len(mgas_s_values) if mgas_s_values else 0
+
+    total_latencies_ms = [r["total_latency_us"] / 1_000 for r in combined]
+    wall_clock_s = sum(total_latencies_ms) / 1_000
+    mean_total_lat_ms = sum(total_latencies_ms) / n
+
+    return {
+        "n": n,
+        "mean_ms": mean_lat,
+        "stddev_ms": std_lat,
+        "p50_ms": percentile(sorted_lat, 50),
+        "p90_ms": percentile(sorted_lat, 90),
+        "p99_ms": percentile(sorted_lat, 99),
+        "mean_mgas_s": mean_mgas_s,
+        "wall_clock_s": wall_clock_s,
+        "mean_total_lat_ms": mean_total_lat_ms,
+    }
+
+
+def compute_wait_stats(combined: list[dict], field: str) -> dict:
+    """Compute mean/p50/p95 for a wait time field (in ms)."""
+    values_ms = []
+    for r in combined:
+        v = r.get(field)
+        if v is not None:
+            values_ms.append(v / 1_000)
+    if not values_ms:
+        return {}
+    n = len(values_ms)
+    mean_val = sum(values_ms) / n
+    sorted_vals = sorted(values_ms)
+    return {
+        "mean_ms": mean_val,
+        "p50_ms": percentile(sorted_vals, 50),
+        "p95_ms": percentile(sorted_vals, 95),
+    }
+
+
+def _paired_data(
+    baseline: list[dict], feature: list[dict]
+) -> tuple[list[tuple[float, float]], list[float], list[float], list[float]]:
+    """Match blocks and return paired latencies and per-block diffs.
+
+    Returns:
+        pairs: list of (baseline_ms, feature_ms) tuples
+        lat_diffs_ms: list of feature − baseline latency diffs in ms
+        mgas_diffs: list of feature − baseline Mgas/s diffs
+        total_lat_diffs_ms: list of feature − baseline total latency diffs in ms
+    """
+    baseline_by_block = {r["block_number"]: r for r in baseline}
+    feature_by_block = {r["block_number"]: r for r in feature}
+    common_blocks = sorted(set(baseline_by_block) & set(feature_by_block))
+
+    pairs = []
+    lat_diffs_ms = []
+    mgas_diffs = []
+    total_lat_diffs_ms = []
+    for bn in common_blocks:
+        b = baseline_by_block[bn]
+        f = feature_by_block[bn]
+        b_ms = b["new_payload_latency_us"] / 1_000
+        f_ms = f["new_payload_latency_us"] / 1_000
+        pairs.append((b_ms, f_ms))
+        lat_diffs_ms.append(f_ms - b_ms)
+        b_lat_s = b["new_payload_latency_us"] / 1_000_000
+        f_lat_s = f["new_payload_latency_us"] / 1_000_000
+        if b_lat_s > 0 and f_lat_s > 0:
+            mgas_diffs.append(
+                f["gas_used"] / f_lat_s / 1_000_000
+                - b["gas_used"] / b_lat_s / 1_000_000
+            )
+        total_lat_diffs_ms.append(
+            f["total_latency_us"] / 1_000 - b["total_latency_us"] / 1_000
+        )
+    return pairs, lat_diffs_ms, mgas_diffs, total_lat_diffs_ms
+
+
+def compute_paired_stats(
+    baseline_runs: list[list[dict]],
+    feature_runs: list[list[dict]],
+) -> dict:
+    """Compute paired statistics between baseline and feature runs.
+
+    Each pair (baseline_runs[i], feature_runs[i]) produces per-block diffs.
+    All diffs are pooled for the final CI.
+    """
+    all_pairs = []
+    all_lat_diffs = []
+    all_mgas_diffs = []
+    all_total_lat_diffs = []
+    blocks_per_pair = []
+    for baseline, feature in zip(baseline_runs, feature_runs):
+        pairs, lat_diffs, mgas_diffs, total_lat_diffs = _paired_data(baseline, feature)
+        all_pairs.extend(pairs)
+        all_lat_diffs.extend(lat_diffs)
+        all_mgas_diffs.extend(mgas_diffs)
+        all_total_lat_diffs.extend(total_lat_diffs)
+        blocks_per_pair.append(len(pairs))
+
+    if not all_lat_diffs:
+        return {}
+
+    n = len(all_lat_diffs)
+    mean_diff = sum(all_lat_diffs) / n
+    std_diff = stddev(all_lat_diffs, mean_diff)
+    se = std_diff / math.sqrt(n) if n > 0 else 0.0
+    ci = T_CRITICAL * se
+
+    # Bootstrap CI on difference-of-percentiles (resample paired blocks)
+    base_lats = sorted([p[0] for p in all_pairs])
+    feature_lats = sorted([p[1] for p in all_pairs])
+    p50_diff = percentile(feature_lats, 50) - percentile(base_lats, 50)
+    p90_diff = percentile(feature_lats, 90) - percentile(base_lats, 90)
+    p99_diff = percentile(feature_lats, 99) - percentile(base_lats, 99)
+
+    rng = random.Random(42)
+    p50_boot, p90_boot, p99_boot = [], [], []
+    for _ in range(BOOTSTRAP_ITERATIONS):
+        sample = rng.choices(all_pairs, k=n)
+        b_sorted = sorted(p[0] for p in sample)
+        f_sorted = sorted(p[1] for p in sample)
+        p50_boot.append(percentile(f_sorted, 50) - percentile(b_sorted, 50))
+        p90_boot.append(percentile(f_sorted, 90) - percentile(b_sorted, 90))
+        p99_boot.append(percentile(f_sorted, 99) - percentile(b_sorted, 99))
+    p50_boot.sort()
+    p90_boot.sort()
+    p99_boot.sort()
+    lo = int(BOOTSTRAP_ITERATIONS * 0.025)
+    hi = int(BOOTSTRAP_ITERATIONS * 0.975)
+
+    mean_mgas_diff = sum(all_mgas_diffs) / len(all_mgas_diffs) if all_mgas_diffs else 0.0
+    std_mgas_diff = stddev(all_mgas_diffs, mean_mgas_diff) if len(all_mgas_diffs) > 1 else 0.0
+    mgas_se = std_mgas_diff / math.sqrt(len(all_mgas_diffs)) if all_mgas_diffs else 0.0
+    mgas_ci = T_CRITICAL * mgas_se
+
+    mean_total_diff = sum(all_total_lat_diffs) / len(all_total_lat_diffs) if all_total_lat_diffs else 0.0
+    std_total_diff = stddev(all_total_lat_diffs, mean_total_diff) if len(all_total_lat_diffs) > 1 else 0.0
+    total_se = std_total_diff / math.sqrt(len(all_total_lat_diffs)) if all_total_lat_diffs else 0.0
+    wall_clock_ci_ms = T_CRITICAL * total_se
+
+    return {
+        "n": n,
+        "mean_diff_ms": mean_diff,
+        "ci_ms": ci,
+        "p50_diff_ms": p50_diff,
+        "p50_ci_ms": (p50_boot[hi] - p50_boot[lo]) / 2,
+        "p90_diff_ms": p90_diff,
+        "p90_ci_ms": (p90_boot[hi] - p90_boot[lo]) / 2,
+        "p99_diff_ms": p99_diff,
+        "p99_ci_ms": (p99_boot[hi] - p99_boot[lo]) / 2,
+        "mean_mgas_diff": mean_mgas_diff,
+        "mgas_ci": mgas_ci,
+        "wall_clock_ci_ms": wall_clock_ci_ms,
+        "blocks": max(blocks_per_pair),
+    }
+
+
+
+def format_duration(seconds: float) -> str:
+    if seconds >= 60:
+        return f"{seconds / 60:.1f}min"
+    return f"{seconds}s"
+
+
+def format_gas(gas: int) -> str:
+    if gas >= GIGAGAS:
+        return f"{gas / GIGAGAS:.1f}G"
+    if gas >= 1_000_000:
+        return f"{gas / 1_000_000:.1f}M"
+    return f"{gas:,}"
+
+
+
+def fmt_ms(v: float) -> str:
+    return f"{v:.2f}ms"
+
+
+def fmt_mgas(v: float) -> str:
+    return f"{v:.2f}"
+
+
+def fmt_s(v: float) -> str:
+    return f"{v:.2f}s"
+
+
+def significance(pct: float, ci_pct: float, lower_is_better: bool) -> str:
+    """Return significance label: 'good', 'bad', or 'neutral'."""
+    significant = abs(pct) > ci_pct
+    if not significant:
+        return "neutral"
+    elif (pct < 0) == lower_is_better:
+        return "good"
+    else:
+        return "bad"
+
+
+def change_str(pct: float, ci_pct: float, lower_is_better: bool) -> str:
+    """Format change% with paired CI significance.
+
+    Significant if the CI doesn't cross zero (i.e. |pct| > ci_pct).
+    """
+    sig = significance(pct, ci_pct, lower_is_better)
+    emoji = {"good": "✅", "bad": "❌", "neutral": "⚪"}[sig]
+    return f"{pct:+.2f}% {emoji} (±{ci_pct:.2f}%)"
+
+
+def compute_changes(
+    baseline_stats: dict, feature_stats: dict, paired_stats: dict
+) -> dict:
+    """Pre-compute change percentages and significance for each metric."""
+    def pct(base: float, feat: float) -> float:
+        return (feat - base) / base * 100.0 if base > 0 else 0.0
+
+    def ci_pct(ci_ms: float, base_ms: float) -> float:
+        return ci_ms / base_ms * 100.0 if base_ms > 0 else 0.0
+
+    metrics = [
+        ("mean", "mean_ms", "ci_ms", "mean_ms", True),
+        ("p50", "p50_ms", "p50_ci_ms", "p50_ms", True),
+        ("p90", "p90_ms", "p90_ci_ms", "p90_ms", True),
+        ("p99", "p99_ms", "p99_ci_ms", "p99_ms", True),
+        ("mgas_s", "mean_mgas_s", "mgas_ci", "mean_mgas_s", False),
+        ("wall_clock", "wall_clock_s", "wall_clock_ci_ms", "mean_total_lat_ms", True),
+    ]
+    changes = {}
+    for name, stat_key, ci_key, base_key, lower_is_better in metrics:
+        p = pct(baseline_stats[stat_key], feature_stats[stat_key])
+        c = ci_pct(paired_stats[ci_key], baseline_stats[base_key])
+        changes[name] = {
+            "pct": round(p, 4),
+            "ci_pct": round(c, 4),
+            "sig": significance(p, c, lower_is_better),
+        }
+    return changes
+
+
+def generate_comparison_table(
+    run1: dict,
+    run2: dict,
+    paired: dict,
+    repo: str,
+    baseline_ref: str,
+    baseline_name: str,
+    feature_name: str,
+    feature_sha: str,
+    big_blocks: bool = False,
+) -> str:
+    """Generate a markdown comparison table between baseline and feature."""
+    n = paired["blocks"]
+
+    def pct(base: float, feat: float) -> float:
+        return (feat - base) / base * 100.0 if base > 0 else 0.0
+
+    mean_pct = pct(run1["mean_ms"], run2["mean_ms"])
+    gas_pct = pct(run1["mean_mgas_s"], run2["mean_mgas_s"])
+    wall_pct = pct(run1["wall_clock_s"], run2["wall_clock_s"])
+
+    p50_pct = pct(run1["p50_ms"], run2["p50_ms"])
+    p90_pct = pct(run1["p90_ms"], run2["p90_ms"])
+    p99_pct = pct(run1["p99_ms"], run2["p99_ms"])
+
+    # Bootstrap CIs as % of baseline percentile
+    p50_ci_pct = paired["p50_ci_ms"] / run1["p50_ms"] * 100.0 if run1["p50_ms"] > 0 else 0.0
+    p90_ci_pct = paired["p90_ci_ms"] / run1["p90_ms"] * 100.0 if run1["p90_ms"] > 0 else 0.0
+    p99_ci_pct = paired["p99_ci_ms"] / run1["p99_ms"] * 100.0 if run1["p99_ms"] > 0 else 0.0
+
+    # CI as a percentage of baseline mean
+    lat_ci_pct = paired["ci_ms"] / run1["mean_ms"] * 100.0 if run1["mean_ms"] > 0 else 0.0
+    mgas_ci_pct = paired["mgas_ci"] / run1["mean_mgas_s"] * 100.0 if run1["mean_mgas_s"] > 0 else 0.0
+    wall_ci_pct = paired["wall_clock_ci_ms"] / run1["mean_total_lat_ms"] * 100.0 if run1["mean_total_lat_ms"] > 0 else 0.0
+
+    base_url = f"https://github.com/{repo}/commit"
+    baseline_label = f"[`{baseline_name}`]({base_url}/{baseline_ref})"
+    feature_label = f"[`{feature_name}`]({base_url}/{feature_sha})"
+
+    lines = [
+        f"| Metric | {baseline_label} | {feature_label} | Change |",
+        "|--------|------|--------|--------|",
+        f"| Mean | {fmt_ms(run1['mean_ms'])} | {fmt_ms(run2['mean_ms'])} | {change_str(mean_pct, lat_ci_pct, lower_is_better=True)} |",
+        f"| StdDev | {fmt_ms(run1['stddev_ms'])} | {fmt_ms(run2['stddev_ms'])} | |",
+        f"| P50 | {fmt_ms(run1['p50_ms'])} | {fmt_ms(run2['p50_ms'])} | {change_str(p50_pct, p50_ci_pct, lower_is_better=True)} |",
+        f"| P90 | {fmt_ms(run1['p90_ms'])} | {fmt_ms(run2['p90_ms'])} | {change_str(p90_pct, p90_ci_pct, lower_is_better=True)} |",
+        f"| P99 | {fmt_ms(run1['p99_ms'])} | {fmt_ms(run2['p99_ms'])} | {change_str(p99_pct, p99_ci_pct, lower_is_better=True)} |",
+        f"| Mgas/s | {fmt_mgas(run1['mean_mgas_s'])} | {fmt_mgas(run2['mean_mgas_s'])} | {change_str(gas_pct, mgas_ci_pct, lower_is_better=False)} |",
+        f"| Wall Clock | {fmt_s(run1['wall_clock_s'])} | {fmt_s(run2['wall_clock_s'])} | {change_str(wall_pct, wall_ci_pct, lower_is_better=True)} |",
+        "",
+        f"*{n} {'big blocks' if big_blocks else 'blocks'}*",
+    ]
+    return "\n".join(lines)
+
+
+def generate_wait_time_table(
+    title: str,
+    baseline_stats: dict,
+    feature_stats: dict,
+    baseline_label: str,
+    feature_label: str,
+) -> str:
+    """Generate a markdown table for a wait time metric."""
+    if not baseline_stats or not feature_stats:
+        return ""
+    lines = [
+        f"### {title}",
+        "",
+        f"| Metric | {baseline_label} | {feature_label} |",
+        "|--------|------|--------|",
+        f"| Mean | {fmt_ms(baseline_stats['mean_ms'])} | {fmt_ms(feature_stats['mean_ms'])} |",
+        f"| P50 | {fmt_ms(baseline_stats['p50_ms'])} | {fmt_ms(feature_stats['p50_ms'])} |",
+        f"| P95 | {fmt_ms(baseline_stats['p95_ms'])} | {fmt_ms(feature_stats['p95_ms'])} |",
+    ]
+    return "\n".join(lines)
+
+
+def generate_markdown(
+    summary: dict, comparison_table: str,
+    wait_time_tables: list[str] | None = None,
+    behind_baseline: int = 0, repo: str = "", baseline_ref: str = "", baseline_name: str = "",
+    grafana_url: str | None = None,
+) -> str:
+    """Generate a markdown comment body."""
+    lines = ["## Benchmark Results", ""]
+    if behind_baseline > 0:
+        s = "s" if behind_baseline > 1 else ""
+        diff_link = f"https://github.com/{repo}/compare/{baseline_ref[:12]}...{baseline_name}"
+        lines.append(f"> ⚠️ Feature is [**{behind_baseline} commit{s} behind `{baseline_name}`**]({diff_link}). Consider rebasing for accurate results.")
+        lines.append("")
+    lines.append(comparison_table)
+    if wait_time_tables:
+        lines.append("")
+        lines.append("<details>")
+        lines.append("<summary>Wait Time Breakdown</summary>")
+        lines.append("")
+        for table in wait_time_tables:
+            if table:
+                lines.append(table)
+                lines.append("")
+        lines.append("</details>")
+    if grafana_url:
+        lines.append("")
+        lines.append(f"**[Grafana Dashboard]({grafana_url})**")
+    return "\n".join(lines)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Parse reth-bench ABBA results")
+    parser.add_argument(
+        "--baseline-csv", nargs="+", required=True,
+        help="Baseline combined_latency.csv files (A1, A2)",
+    )
+    parser.add_argument(
+        "--feature-csv", "--branch-csv", nargs="+", required=True,
+        help="Feature combined_latency.csv files (B1, B2)",
+    )
+    parser.add_argument("--gas-csv", required=True, help="Path to total_gas.csv")
+    parser.add_argument(
+        "--output-summary", required=True, help="Output JSON summary path"
+    )
+    parser.add_argument("--output-markdown", required=True, help="Output markdown path")
+    parser.add_argument(
+        "--repo", default="paradigmxyz/reth", help="GitHub repo (owner/name)"
+    )
+    parser.add_argument("--baseline-ref", default=None, help="Baseline commit SHA")
+    parser.add_argument("--baseline-name", default=None, help="Baseline display name")
+    parser.add_argument("--feature-name", "--branch-name", default=None, help="Feature branch name")
+    parser.add_argument("--feature-ref", "--branch-sha", "--feature-sha", default=None, help="Feature commit SHA")
+    parser.add_argument("--behind-baseline", "--behind-main", type=int, default=0, help="Commits behind baseline")
+    parser.add_argument("--big-blocks", action="store_true", default=False, help="Big blocks mode")
+    parser.add_argument("--gas-ramp-blocks", type=int, default=0, help="Number of gas ramp blocks (big blocks mode)")
+    parser.add_argument("--grafana-url", default=None, help="Grafana dashboard URL for this benchmark run")
+    args = parser.parse_args()
+
+    if len(args.baseline_csv) != len(args.feature_csv):
+        print("Must provide equal number of baseline and feature CSVs", file=sys.stderr)
+        sys.exit(1)
+
+    baseline_runs = []
+    feature_runs = []
+    for path in args.baseline_csv:
+        data = parse_combined_csv(path)
+        if not data:
+            print(f"No results in {path}", file=sys.stderr)
+            sys.exit(1)
+        baseline_runs.append(data)
+    for path in args.feature_csv:
+        data = parse_combined_csv(path)
+        if not data:
+            print(f"No results in {path}", file=sys.stderr)
+            sys.exit(1)
+        feature_runs.append(data)
+
+    gas = parse_gas_csv(args.gas_csv)
+
+    all_baseline = [r for run in baseline_runs for r in run]
+    all_feature = [r for run in feature_runs for r in run]
+
+    baseline_stats = compute_stats(all_baseline)
+    feature_stats = compute_stats(all_feature)
+    paired_stats = compute_paired_stats(baseline_runs, feature_runs)
+
+    if not paired_stats:
+        print("No common blocks between baseline and feature runs", file=sys.stderr)
+        sys.exit(1)
+
+    baseline_ref = args.baseline_ref or "main"
+    baseline_name = args.baseline_name or "baseline"
+    feature_name = args.feature_name or "feature"
+    feature_sha = args.feature_ref or "unknown"
+
+    comparison_table = generate_comparison_table(
+        baseline_stats,
+        feature_stats,
+        paired_stats,
+        repo=args.repo,
+        baseline_ref=baseline_ref,
+        baseline_name=baseline_name,
+        feature_name=feature_name,
+        feature_sha=feature_sha,
+        big_blocks=args.big_blocks,
+    )
+    print(f"Generated comparison ({paired_stats['n']} paired blocks, "
+          f"mean diff {paired_stats['mean_diff_ms']:+.3f}ms ± {paired_stats['ci_ms']:.3f}ms)")
+
+    base_url = f"https://github.com/{args.repo}/commit"
+    baseline_label = f"[`{baseline_name}`]({base_url}/{baseline_ref})"
+    feature_label = f"[`{feature_name}`]({base_url}/{feature_sha})"
+
+    wait_fields = [
+        ("persistence_wait_us", "Persistence Wait"),
+        ("sparse_trie_wait_us", "Trie Cache Update Wait"),
+        ("execution_cache_wait_us", "Execution Cache Update Wait"),
+    ]
+    wait_time_tables = []
+    wait_time_data = {}
+    for field, title in wait_fields:
+        b_stats = compute_wait_stats(all_baseline, field)
+        f_stats = compute_wait_stats(all_feature, field)
+        if b_stats and f_stats:
+            wait_time_data[field] = {
+                "title": title,
+                "baseline": b_stats,
+                "feature": f_stats,
+            }
+        table = generate_wait_time_table(title, b_stats, f_stats, baseline_label, feature_label)
+        if table:
+            wait_time_tables.append(table)
+
+    summary = {
+        "blocks": paired_stats["blocks"],
+        "big_blocks": args.big_blocks,
+        "gas_ramp_blocks": args.gas_ramp_blocks,
+        "baseline": {
+            "name": baseline_name,
+            "ref": baseline_ref,
+            "stats": baseline_stats,
+        },
+        "feature": {
+            "name": feature_name,
+            "ref": feature_sha,
+            "stats": feature_stats,
+        },
+        "paired": paired_stats,
+        "changes": compute_changes(baseline_stats, feature_stats, paired_stats),
+        "wait_times": wait_time_data,
+    }
+    with open(args.output_summary, "w") as f:
+        json.dump(summary, f, indent=2)
+    print(f"Summary written to {args.output_summary}")
+
+    markdown = generate_markdown(
+        summary, comparison_table,
+        wait_time_tables=wait_time_tables,
+        behind_baseline=args.behind_baseline,
+        repo=args.repo,
+        baseline_ref=baseline_ref,
+        baseline_name=baseline_name,
+        grafana_url=args.grafana_url,
+    )
+
+    with open(args.output_markdown, "w") as f:
+        f.write(markdown)
+    print(f"Markdown written to {args.output_markdown}")
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/bench-scheduled-refs.sh

@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+#
+# Resolves baseline and feature refs for nightly regression benchmark runs.
+#
+# Queries the latest successful scheduled docker.yml run via GitHub API
+# to find the commit that built the nightly Docker image. Compares with
+# the last successful feature ref (from GH Actions cache) to determine
+# baseline, detect staleness, and decide whether to skip.
+#
+# Usage: bench-nightly-refs.sh [--force]
+#
+# Outputs (via GITHUB_OUTPUT):
+#   baseline-ref    — commit SHA for baseline
+#   feature-ref     — commit SHA for feature (current nightly)
+#   should-skip     — "true" if no new nightly since last run
+#   is-stale        — "true" if latest nightly build is >24h old
+#   stale-age-hours — age of the nightly build in hours (only if stale)
+#   nightly-created — ISO timestamp of the nightly build
+#
+# Reads:
+#   .nightly-state/last-feature-ref (from GH Actions cache, may not exist)
+#
+# Requires: gh (GitHub CLI), jq, date
+set -euo pipefail
+
+FORCE="${1:-false}"
+REPO="${GITHUB_REPOSITORY:-paradigmxyz/reth}"
+
+# --- Step 1: Query latest successful scheduled docker.yml run ---
+echo "::group::Querying latest nightly docker build"
+
+RUNS_JSON=$(gh run list \
+  -R "$REPO" \
+  --workflow=docker.yml \
+  --event=schedule \
+  --status=completed \
+  --limit 5 \
+  --json headSha,createdAt,conclusion)
+
+# Find the most recent successful run
+LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
+
+if [ -z "$LATEST" ]; then
+  echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
+  echo "Runs found: $RUNS_JSON"
+  exit 1
+fi
+
+FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
+CREATED_AT=$(echo "$LATEST" | jq -r '.createdAt')
+echo "Latest nightly commit: $FEATURE_REF"
+echo "Built at: $CREATED_AT"
+echo "::endgroup::"
+
+# --- Step 2: Staleness check ---
+echo "::group::Checking staleness"
+NOW_EPOCH=$(date +%s)
+# Handle both GNU date (-d) and BSD date (-j -f) for cross-platform compat
+CREATED_EPOCH=$(date -d "$CREATED_AT" +%s 2>/dev/null || \
+  date -j -f "%Y-%m-%dT%H:%M:%SZ" "$CREATED_AT" +%s 2>/dev/null || \
+  date -j -f "%Y-%m-%dT%T%z" "$CREATED_AT" +%s 2>/dev/null || \
+  { echo "::error::Cannot parse date: $CREATED_AT"; exit 1; })
+
+AGE_SECONDS=$(( NOW_EPOCH - CREATED_EPOCH ))
+AGE_HOURS=$(( AGE_SECONDS / 3600 ))
+IS_STALE="false"
+
+if [ "$AGE_HOURS" -gt 24 ]; then
+  IS_STALE="true"
+  echo "::warning::STALE NIGHTLY: Build is ${AGE_HOURS}h old (>24h threshold)"
+  echo "This indicates the nightly docker build failed — no new image was produced"
+else
+  echo "Nightly build age: ${AGE_HOURS}h (within 24h threshold)"
+fi
+echo "::endgroup::"
+
+# --- Step 3: Read last successful feature ref from cache ---
+echo "::group::Reading cached state"
+LAST_FEATURE_REF=""
+STATE_FILE=".nightly-state/last-feature-ref"
+if [ -f "$STATE_FILE" ]; then
+  LAST_FEATURE_REF=$(tr -d '[:space:]' < "$STATE_FILE")
+  echo "Previous feature ref: $LAST_FEATURE_REF"
+else
+  echo "No cached state found (first run)"
+fi
+echo "::endgroup::"
+
+# --- Step 4: Determine baseline and skip logic ---
+echo "::group::Resolving refs"
+SHOULD_SKIP="false"
+BASELINE_REF="$FEATURE_REF"  # default for first run
+
+if [ "$IS_STALE" = "true" ]; then
+  # Stale = error path, don't skip (will alert and fail downstream)
+  SHOULD_SKIP="false"
+  BASELINE_REF="${LAST_FEATURE_REF:-$FEATURE_REF}"
+  echo "Stale nightly detected — will alert and fail"
+elif [ -z "$LAST_FEATURE_REF" ]; then
+  # First run: baseline = feature (self-comparison to establish baseline)
+  BASELINE_REF="$FEATURE_REF"
+  echo "First run — will benchmark nightly against itself to establish baseline"
+elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+  # No new nightly since last successful run
+  if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+    echo "No new nightly, but force=true — running anyway"
+    BASELINE_REF="$LAST_FEATURE_REF"
+  else
+    SHOULD_SKIP="true"
+    echo "No new nightly since last run — will skip"
+  fi
+else
+  # Normal case: new nightly available
+  BASELINE_REF="$LAST_FEATURE_REF"
+  echo "New nightly detected"
+fi
+
+echo "Baseline: $BASELINE_REF"
+echo "Feature:  $FEATURE_REF"
+echo "Skip:     $SHOULD_SKIP"
+echo "Stale:    $IS_STALE"
+echo "::endgroup::"
+
+# --- Step 5: Write outputs ---
+{
+  echo "baseline-ref=$BASELINE_REF"
+  echo "feature-ref=$FEATURE_REF"
+  echo "should-skip=$SHOULD_SKIP"
+  echo "is-stale=$IS_STALE"
+  echo "stale-age-hours=$AGE_HOURS"
+  echo "nightly-created=$CREATED_AT"
+} >> "$GITHUB_OUTPUT"

.github/scripts/bench-slack-notify.js

@@ -0,0 +1,308 @@
+// Sends Slack notifications for reth-bench results.
+//
+// Reads from environment:
+//   SLACK_BENCH_BOT_TOKEN  – Slack Bot User OAuth Token (xoxb-...)
+//   SLACK_BENCH_CHANNEL    – Public channel ID for significant improvements
+//   BENCH_WORK_DIR         – Directory containing summary.json
+//   BENCH_PR               – PR number (may be empty)
+//   BENCH_ACTOR            – GitHub user who triggered the bench
+//   BENCH_JOB_URL          – URL to the Actions job page
+//   BENCH_BASELINE_ARGS    – Extra CLI args for the baseline reth node
+//   BENCH_FEATURE_ARGS     – Extra CLI args for the feature reth node
+//   BENCH_SAMPLY           – 'true' if samply profiling was enabled
+//
+// Usage from actions/github-script:
+//   const notify = require('./.github/scripts/bench-slack-notify.js');
+//   await notify.success({ core, context });
+//   await notify.failure({ core, context, failedStep: '...' });
+
+const fs = require('fs');
+const path = require('path');
+const { fmtChange, fmtMs, verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
+
+const SLACK_API = 'https://slack.com/api/chat.postMessage';
+
+function loadSlackUsers(repoRoot) {
+  try {
+    const raw = fs.readFileSync(path.join(repoRoot, '.github', 'scripts', 'bench-slack-users.json'), 'utf8');
+    const data = JSON.parse(raw);
+    // Filter out non-user-ID entries (like _comment)
+    const users = {};
+    for (const [k, v] of Object.entries(data)) {
+      if (!k.startsWith('_') && typeof v === 'string' && v.startsWith('U')) {
+        users[k] = v;
+      }
+    }
+    return users;
+  } catch {
+    return {};
+  }
+}
+
+async function postToSlack(token, channel, blocks, text, core, threadTs) {
+  const payload = { channel, blocks, text, unfurl_links: false };
+  if (threadTs) payload.thread_ts = threadTs;
+  const resp = await fetch(SLACK_API, {
+    method: 'POST',
+    headers: {
+      'Authorization': `Bearer ${token}`,
+      'Content-Type': 'application/json',
+    },
+    body: JSON.stringify(payload),
+  });
+  const data = await resp.json();
+  if (!data.ok) {
+    core.warning(`Slack API error (channel ${channel}): ${JSON.stringify(data)}`);
+  }
+  return data;
+}
+
+function cell(text) {
+  const s = String(text);
+  return { type: 'raw_text', text: s || ' ' };
+}
+
+// Slack shortcodes for verdict (Block Kit header doesn't support unicode emoji)
+const SLACK_VERDICT = {
+  '⚠️': ':warning:',
+  '❌': ':x:',
+  '✅': ':white_check_mark:',
+  '⚪': ':white_circle:',
+};
+
+function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, repo, samplyUrls }) {
+  const { emoji, label } = verdict(summary.changes);
+  const headerEmoji = SLACK_VERDICT[emoji] || emoji;
+
+  const prUrl = prNumber ? `https://github.com/${repo}/pull/${prNumber}` : '';
+  const commitUrl = `https://github.com/${repo}/commit`;
+  const baselineLink = `<${commitUrl}/${summary.baseline.ref}|${summary.baseline.name}>`;
+  const featureLink = `<${commitUrl}/${summary.feature.ref}|${summary.feature.name}>`;
+
+  // Meta line
+  const metaParts = [];
+  if (prNumber) metaParts.push(`*<${prUrl}|PR #${prNumber}>*`);
+  metaParts.push(`triggered by ${actorSlackId ? `<@${actorSlackId}>` : `@${actor}`}`);
+
+  // Baseline/feature lines with samply profile links
+  let baselineLine = `*Baseline:* ${baselineLink}`;
+  const bl1 = samplyUrls['baseline-1'];
+  const bl2 = samplyUrls['baseline-2'];
+  if (bl1) baselineLine += ` | <${bl1}|Samply 1>`;
+  if (bl2) baselineLine += ` | <${bl2}|Samply 2>`;
+
+  let featureLine = `*Feature:* ${featureLink}`;
+  const fl1 = samplyUrls['feature-1'];
+  const fl2 = samplyUrls['feature-2'];
+  if (fl1) featureLine += ` | <${fl1}|Samply 1>`;
+  if (fl2) featureLine += ` | <${fl2}|Samply 2>`;
+
+  const countsLine = blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | ');
+
+  const baselineArgs = process.env.BENCH_BASELINE_ARGS || '';
+  const featureArgs = process.env.BENCH_FEATURE_ARGS || '';
+  const argsLines = [];
+  if (baselineArgs) argsLines.push(`*Baseline Args:* \`${baselineArgs}\``);
+  if (featureArgs) argsLines.push(`*Feature Args:* \`${featureArgs}\``);
+
+  const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, ...argsLines, countsLine].join('\n');
+
+  // Action buttons
+  const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+  const buttons = [
+    {
+      type: 'button',
+      text: { type: 'plain_text', text: 'CI :github:', emoji: true },
+      url: jobUrl,
+      action_id: 'ci_button',
+    },
+    {
+      type: 'button',
+      text: { type: 'plain_text', text: 'Diff :github:', emoji: true },
+      url: diffUrl,
+      action_id: 'diff_button',
+    },
+  ];
+
+  // Build table rows from shared metricRows
+  const rows = metricRows(summary);
+  const tableRows = [
+    [cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
+    ...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
+  ];
+
+  const blocks = [
+    {
+      type: 'header',
+      text: { type: 'plain_text', text: `${headerEmoji} ${label}`, emoji: true },
+    },
+    {
+      type: 'section',
+      text: { type: 'mrkdwn', text: sectionText },
+    },
+    {
+      type: 'table',
+      column_settings: [
+        { align: 'left' },
+        { align: 'right' },
+        { align: 'right' },
+        { align: 'right' },
+      ],
+      rows: tableRows,
+    },
+    {
+      type: 'actions',
+      elements: buttons,
+    },
+  ];
+
+  // Wait times as a separate table block (sent as threaded reply due to Slack one-table limit)
+  const threadBlocks = [];
+  const wtRows = waitTimeRows(summary);
+  if (wtRows.length > 0) {
+    const waitTableRows = [
+      [cell('Wait Time'), cell('Baseline'), cell('Feature')],
+      ...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
+    ];
+    threadBlocks.push({
+      type: 'table',
+      column_settings: [
+        { align: 'left' },
+        { align: 'right' },
+        { align: 'right' },
+      ],
+      rows: waitTableRows,
+    });
+  }
+
+  return { blocks, threadBlocks };
+}
+
+function buildFailureBlocks({ prNumber, actor, actorSlackId, jobUrl, repo, failedStep }) {
+  const prUrl = prNumber ? `https://github.com/${repo}/pull/${prNumber}` : '';
+  const actorMention = actorSlackId ? `<@${actorSlackId}>` : `@${actor}`;
+  const parts = [
+    prNumber ? `*<${prUrl}|PR #${prNumber}>*` : '',
+    `by ${actorMention}`,
+    `failed while *${failedStep}*`,
+  ].filter(Boolean);
+
+  const buttons = [
+    {
+      type: 'button',
+      text: { type: 'plain_text', text: 'CI :github:', emoji: true },
+      url: jobUrl,
+      action_id: 'ci_button',
+    },
+  ];
+
+  return [
+    {
+      type: 'header',
+      text: { type: 'plain_text', text: ':rotating_light: Bench Failed', emoji: true },
+    },
+    {
+      type: 'section',
+      text: { type: 'mrkdwn', text: parts.join(' | ') },
+    },
+    {
+      type: 'actions',
+      elements: buttons,
+    },
+  ];
+}
+
+async function success({ core, context }) {
+  const token = process.env.SLACK_BENCH_BOT_TOKEN;
+  if (!token) {
+    core.info('SLACK_BENCH_BOT_TOKEN not set, skipping Slack notification');
+    return;
+  }
+
+  let summary;
+  try {
+    summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+  } catch (e) {
+    core.warning('Could not read summary.json for Slack notification');
+    return;
+  }
+
+  const repo = `${context.repo.owner}/${context.repo.repo}`;
+  const prNumber = process.env.BENCH_PR;
+  const actor = process.env.BENCH_ACTOR;
+  const jobUrl = process.env.BENCH_JOB_URL ||
+    `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+
+  const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
+
+  const slackUsers = loadSlackUsers(process.env.GITHUB_WORKSPACE || '.');
+  const actorSlackId = slackUsers[actor];
+
+  const { blocks, threadBlocks } = buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, repo, samplyUrls });
+  const text = `Bench: ${summary.baseline.name} vs ${summary.feature.name}`;
+
+  async function sendWithThread(ch) {
+    const res = await postToSlack(token, ch, blocks, text, core);
+    if (res.ok && res.ts && threadBlocks.length > 0) {
+      for (const tb of threadBlocks) {
+        await postToSlack(token, ch, [tb], 'Wait time breakdown', core, res.ts);
+      }
+    }
+  }
+
+  // Post to public channel if any metric shows significant improvement or regression
+  const channel = process.env.SLACK_BENCH_CHANNEL;
+  let postedToChannel = false;
+  if (channel) {
+    const changes = summary.changes || {};
+    const hasImprovement = Object.values(changes).some(c => c.sig === 'good');
+    if (hasImprovement) {
+      await sendWithThread(channel);
+      postedToChannel = true;
+    } else {
+      core.info('No significant improvement, skipping public channel notification');
+    }
+  }
+
+  // DM the actor only when results were not posted to the public channel
+  if (!postedToChannel) {
+    if (actorSlackId) {
+      await sendWithThread(actorSlackId);
+    } else {
+      core.info(`No Slack user mapping for GitHub user '${actor}', skipping DM`);
+    }
+  } else {
+    core.info(`Results posted to channel, skipping DM to ${actor}`);
+  }
+}
+
+async function failure({ core, context, failedStep }) {
+  const token = process.env.SLACK_BENCH_BOT_TOKEN;
+  if (!token) {
+    core.info('SLACK_BENCH_BOT_TOKEN not set, skipping Slack notification');
+    return;
+  }
+
+  const repo = `${context.repo.owner}/${context.repo.repo}`;
+  const prNumber = process.env.BENCH_PR;
+  const actor = process.env.BENCH_ACTOR;
+  const jobUrl = process.env.BENCH_JOB_URL ||
+    `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+
+  const slackUsers = loadSlackUsers(process.env.GITHUB_WORKSPACE || '.');
+  const actorSlackId = slackUsers[actor];
+
+  const blocks = buildFailureBlocks({ prNumber, actor, actorSlackId, jobUrl, repo, failedStep });
+  const text = `Bench failed while ${failedStep}`;
+
+  // Always DM the actor
+  if (actorSlackId) {
+    await postToSlack(token, actorSlackId, blocks, text, core);
+  } else {
+    core.info(`No Slack user mapping for GitHub user '${actor}', skipping DM`);
+  }
+
+  // Only DM for failures, don't post to public channel
+}
+
+module.exports = { success, failure };

.github/scripts/bench-slack-users.json

@@ -0,0 +1,24 @@
+{
+  "_comment": "Maps GitHub usernames to Slack user IDs. Find yours: Slack profile > ··· > Copy member ID.",
+  "shekhirin": "U09FAL2UMLJ",
+  "mattsse": "U09FQNPMRT3",
+  "klkvr": "U09FAK95FC2",
+  "joshieDo": "U09LHN6GYAU",
+  "mediocregopher": "U09FF75KMQU",
+  "yongkangc": "U09FB0ECTD4",
+  "gakonst": "U092SEPDM40",
+  "Rjected": "U09F6SCKRGT",
+  "DaniPopes": "U09FAT8EK2A",
+  "emmajam": "U0A34UN92HW",
+  "onbjerg": "U09FB0UK5AA",
+  "fgimenez": "U09G3GP7CSU",
+  "rakita": "U09FB3Z2M7Y",
+  "jxom": "U09F72MG083",
+  "tmm": "U0AD0U8E88N",
+  "pepyakin": "U0A7HKMGEHJ",
+  "grandizzy": "U09F8DBDDRT",
+  "SuperFluffy": "U095BKHB2Q4",
+  "kamsz": "U0A2563UBRD",
+  "zerosnacks": "U09FARPMN74",
+  "samczsun": "U096R14E4H3"
+}

.github/scripts/bench-update-status.js

@@ -0,0 +1,27 @@
+// Updates the reth-bench PR comment with current status.
+//
+// Reads from environment:
+//   BENCH_COMMENT_ID  – GitHub comment ID to update
+//   BENCH_JOB_URL     – URL to the Actions job page
+//   BENCH_CONFIG      – Config line (blocks, warmup, refs)
+//   BENCH_ACTOR       – User who triggered the benchmark
+//
+// Usage from actions/github-script:
+//   const s = require('./.github/scripts/bench-update-status.js');
+//   await s({github, context, status: 'Building baseline binary...'});
+
+function buildBody(status) {
+  return `cc @${process.env.BENCH_ACTOR}\n\n🚀 Benchmark started! [View job](${process.env.BENCH_JOB_URL})\n\n⏳ **Status:** ${status}\n\n${process.env.BENCH_CONFIG}`;
+}
+
+async function updateStatus({ github, context, status }) {
+  await github.rest.issues.updateComment({
+    owner: context.repo.owner,
+    repo: context.repo.repo,
+    comment_id: parseInt(process.env.BENCH_COMMENT_ID),
+    body: buildBody(status),
+  });
+}
+
+updateStatus.buildBody = buildBody;
+module.exports = updateStatus;

.github/scripts/bench-utils.js

@@ -0,0 +1,96 @@
+// Shared utilities for reth-bench result rendering.
+//
+// Used by bench-job-summary.js and bench-slack-notify.js.
+
+const fs = require('fs');
+const path = require('path');
+
+const SIG_EMOJI = { good: '✅', bad: '❌', neutral: '⚪' };
+
+function fmtMs(v) { return v.toFixed(2) + 'ms'; }
+function fmtMgas(v) { return v.toFixed(2); }
+function fmtS(v) { return v.toFixed(2) + 's'; }
+
+function fmtChange(ch) {
+  if (!ch || (!ch.pct && !ch.ci_pct)) return '';
+  const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
+  const ciStr = ch.ci_pct ? ` (±${ch.ci_pct.toFixed(2)}%)` : '';
+  return `${pctStr}${ciStr} ${SIG_EMOJI[ch.sig]}`;
+}
+
+function verdict(changes) {
+  const vals = Object.values(changes);
+  const hasBad = vals.some(v => v.sig === 'bad');
+  const hasGood = vals.some(v => v.sig === 'good');
+  if (hasBad && hasGood) return { emoji: '⚠️', label: 'Mixed Results' };
+  if (hasBad) return { emoji: '❌', label: 'Regression' };
+  if (hasGood) return { emoji: '✅', label: 'Improvement' };
+  return { emoji: '⚪', label: 'No Difference' };
+}
+
+function loadSamplyUrls(workDir) {
+  const urls = {};
+  for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
+    try {
+      const url = fs.readFileSync(path.join(workDir, run, 'samply-profile-url.txt'), 'utf8').trim();
+      if (url) urls[run] = url;
+    } catch {}
+  }
+  return urls;
+}
+
+function blocksLabel(summary) {
+  const parts = [];
+  if (summary.big_blocks) {
+    if (summary.gas_ramp_blocks) parts.push({ key: 'Gas Ramp', value: summary.gas_ramp_blocks });
+    parts.push({ key: 'Big Blocks', value: summary.blocks });
+  } else {
+    const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
+    if (warmup) parts.push({ key: 'Warmup', value: warmup });
+    parts.push({ key: 'Blocks', value: summary.blocks });
+  }
+  const cores = process.env.BENCH_CORES || '0';
+  if (cores !== '0') parts.push({ key: 'Cores', value: cores });
+  return parts;
+}
+
+// The 7 metric rows shared by all renderers.
+// Returns an array of { label, baseline, feature, change } objects.
+function metricRows(summary) {
+  const b = summary.baseline.stats;
+  const f = summary.feature.stats;
+  const c = summary.changes;
+  return [
+    { label: 'Mean',       baseline: fmtMs(b.mean_ms),       feature: fmtMs(f.mean_ms),       change: fmtChange(c.mean) },
+    { label: 'StdDev',     baseline: fmtMs(b.stddev_ms),     feature: fmtMs(f.stddev_ms),     change: '' },
+    { label: 'P50',        baseline: fmtMs(b.p50_ms),        feature: fmtMs(f.p50_ms),        change: fmtChange(c.p50) },
+    { label: 'P90',        baseline: fmtMs(b.p90_ms),        feature: fmtMs(f.p90_ms),        change: fmtChange(c.p90) },
+    { label: 'P99',        baseline: fmtMs(b.p99_ms),        feature: fmtMs(f.p99_ms),        change: fmtChange(c.p99) },
+    { label: 'Mgas/s',     baseline: fmtMgas(b.mean_mgas_s), feature: fmtMgas(f.mean_mgas_s), change: fmtChange(c.mgas_s) },
+    { label: 'Wall Clock', baseline: fmtS(b.wall_clock_s),   feature: fmtS(f.wall_clock_s),   change: fmtChange(c.wall_clock) },
+  ];
+}
+
+// Wait time rows: one row per metric showing mean values.
+function waitTimeRows(summary) {
+  const waitTimes = summary.wait_times || {};
+  const rows = [];
+  for (const key of Object.keys(waitTimes)) {
+    const wt = waitTimes[key];
+    rows.push({ title: wt.title, baseline: fmtMs(wt.baseline.mean_ms), feature: fmtMs(wt.feature.mean_ms) });
+  }
+  return rows;
+}
+
+module.exports = {
+  SIG_EMOJI,
+  fmtMs,
+  fmtMgas,
+  fmtS,
+  fmtChange,
+  verdict,
+  loadSamplyUrls,
+  blocksLabel,
+  metricRows,
+  waitTimeRows,
+};

.github/scripts/build_pgo_bolt.sh

@@ -0,0 +1,414 @@
+#!/usr/bin/env bash
+#
+# Full PGO+BOLT optimized build for reth using real reth-bench workloads.
+#
+# Phases:
+#   1. Build PGO-instrumented reth, run reth-bench → collect PGO profiles
+#   2. Build BOLT-instrumented reth (with PGO), run reth-bench → collect BOLT profiles
+#   3. Build final PGO+BOLT optimized binary
+#
+# Required environment variables:
+#   DATADIR    - Path to reth datadir (must already contain chain data)
+#   RPC_URL    - Source RPC URL for reth-bench to fetch payloads from
+#
+# Optional environment variables:
+#   PGO_BLOCKS      - Number of blocks for PGO profiling (default: 20)
+#   BOLT_BLOCKS     - Number of blocks for BOLT profiling (default: 20)
+#   SKIP_BOLT       - Temporarily skip BOLT phases (default: false)
+#   STRIP_SYMBOLS   - Strip debug symbols from output binary (default: true)
+#   COLLECT_PGO_ONLY - Stop after producing merged.profdata (default: false)
+#   PGO_PROFDATA    - Path to pre-collected merged.profdata (optional)
+#   PROFILE         - Cargo profile (default: maxperf-symbols)
+#   FEATURES        - Cargo features (default: jemalloc,asm-keccak,min-debug-logs)
+#   TARGET          - Target triple (default: auto-detected)
+#   EXTRA_RUSTFLAGS - Additional RUSTFLAGS (e.g. -C target-cpu=x86-64-v3)
+#
+# Output:
+#   target/$PROFILE_DIR/reth  — final optimized binary
+set -euo pipefail
+
+gha_section_start() {
+    local title="$1"
+    if [ -n "${GITHUB_ACTIONS:-}" ]; then
+        echo "::group::$title"
+    else
+        echo ""
+        echo "=== $title ==="
+    fi
+}
+
+gha_section_end() {
+    if [ -n "${GITHUB_ACTIONS:-}" ]; then
+        echo "::endgroup::"
+    fi
+}
+
+cd "$(dirname "$0")/../.."
+
+# ── Configuration ──────────────────────────────────────────────────────────────
+PGO_BLOCKS="${PGO_BLOCKS:-20}"
+BOLT_BLOCKS="${BOLT_BLOCKS:-20}"
+SKIP_BOLT="${SKIP_BOLT:-false}"
+STRIP_SYMBOLS="${STRIP_SYMBOLS:-true}"
+COLLECT_PGO_ONLY="${COLLECT_PGO_ONLY:-false}"
+PROFILE="${PROFILE:-maxperf-symbols}"
+FEATURES="${FEATURES:-jemalloc,asm-keccak,min-debug-logs}"
+TARGET="${TARGET:-$(rustc -Vv | grep host | cut -d' ' -f2)}"
+BASE_RUSTFLAGS="${RUSTFLAGS:-}"
+EXTRA_RUSTFLAGS="${EXTRA_RUSTFLAGS:-}"
+COMBINED_RUSTFLAGS="$BASE_RUSTFLAGS $EXTRA_RUSTFLAGS"
+PGO_PROFDATA="${PGO_PROFDATA:-}"
+DATADIR="${DATADIR:-}"
+RPC_URL="${RPC_URL:-}"
+
+SKIP_BOLT_BOOL=false
+if [[ "${SKIP_BOLT,,}" == "true" || "$SKIP_BOLT" == "1" ]]; then
+    SKIP_BOLT_BOOL=true
+fi
+
+STRIP_SYMBOLS_BOOL=false
+if [[ "${STRIP_SYMBOLS,,}" == "true" || "$STRIP_SYMBOLS" == "1" ]]; then
+    STRIP_SYMBOLS_BOOL=true
+fi
+
+COLLECT_PGO_ONLY_BOOL=false
+if [[ "${COLLECT_PGO_ONLY,,}" == "true" || "$COLLECT_PGO_ONLY" == "1" ]]; then
+    COLLECT_PGO_ONLY_BOOL=true
+fi
+
+USE_PRECOLLECTED_PGO=false
+if [ -n "$PGO_PROFDATA" ]; then
+    if [ ! -f "$PGO_PROFDATA" ]; then
+        echo "error: PGO_PROFDATA points to a missing file: $PGO_PROFDATA"
+        exit 1
+    fi
+    USE_PRECOLLECTED_PGO=true
+fi
+
+NEEDS_BENCH_WORKLOAD=true
+if [ "$USE_PRECOLLECTED_PGO" = true ] && [ "$SKIP_BOLT_BOOL" = true ]; then
+    NEEDS_BENCH_WORKLOAD=false
+fi
+
+if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
+    : "${DATADIR:?DATADIR must be set to the reth data directory}"
+    : "${RPC_URL:?RPC_URL must be set}"
+fi
+
+if [[ "$PROFILE" == dev ]]; then
+    PROFILE_DIR=debug
+else
+    PROFILE_DIR=$PROFILE
+fi
+
+MANIFEST_PATH="bin/reth"
+
+LLVM_VERSION=$(rustc -Vv | grep -oP 'LLVM version: \K\d+')
+PGO_DIR="$PWD/target/pgo-profiles"
+BOLT_DIR="$PWD/target/bolt-profiles"
+CARGO_ARGS=(--profile "$PROFILE" --features "$FEATURES" --manifest-path "$MANIFEST_PATH/Cargo.toml" --bin "reth" --locked)
+
+# Enable debug symbols for BOLT (requires symbols to reorder code).
+# Strip them at the end.
+PROFILE_UPPER=$(echo "$PROFILE" | tr '[:lower:]-' '[:upper:]_')
+export "CARGO_PROFILE_${PROFILE_UPPER}_STRIP=debuginfo"
+
+gha_section_start "Full PGO+BOLT Build"
+echo "Binary:      reth"
+echo "Manifest:    $MANIFEST_PATH"
+echo "Target:      $TARGET"
+echo "Profile:     $PROFILE"
+echo "Features:    $FEATURES"
+echo "LLVM:        $LLVM_VERSION"
+echo "PGO blocks:  $PGO_BLOCKS"
+echo "BOLT blocks: $BOLT_BLOCKS"
+echo "Skip BOLT:   $SKIP_BOLT"
+echo "Strip symbols: $STRIP_SYMBOLS"
+echo "Collect only: $COLLECT_PGO_ONLY"
+echo "PGO profdata: ${PGO_PROFDATA:-<collect with reth-bench>}"
+echo "RUSTFLAGS:   ${BASE_RUSTFLAGS:-<unset>}"
+echo "EXTRA_RUSTFLAGS: ${EXTRA_RUSTFLAGS:-<unset>}"
+if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
+    echo "Datadir:     $DATADIR"
+    echo "RPC URL:     $RPC_URL"
+else
+    echo "Datadir:     <not required>"
+    echo "RPC URL:     <not required>"
+fi
+gha_section_end
+
+# ── Prerequisites ──────────────────────────────────────────────────────────────
+gha_section_start "Installing prerequisites"
+rustup component add llvm-tools-preview
+
+LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
+if [ -z "$LLVM_PROFDATA" ]; then
+    echo "error: llvm-profdata not found"
+    exit 1
+fi
+
+install_bolt() {
+    if command -v llvm-bolt &>/dev/null; then
+        echo "BOLT already installed"
+        return
+    fi
+    echo "Installing BOLT from apt.llvm.org..."
+    wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc >/dev/null
+    CODENAME=$(lsb_release -cs)
+    echo "deb http://apt.llvm.org/$CODENAME/ llvm-toolchain-$CODENAME-$LLVM_VERSION main" | sudo tee /etc/apt/sources.list.d/llvm.list >/dev/null
+    sudo apt-get update -qq
+    sudo apt-get install -y -qq "bolt-$LLVM_VERSION"
+    sudo ln -sf "/usr/bin/llvm-bolt-$LLVM_VERSION" /usr/local/bin/llvm-bolt
+    sudo ln -sf "/usr/bin/merge-fdata-$LLVM_VERSION" /usr/local/bin/merge-fdata
+}
+if [ "$SKIP_BOLT_BOOL" = true ]; then
+    echo "Skipping BOLT installation (SKIP_BOLT=$SKIP_BOLT)"
+else
+    install_bolt
+fi
+gha_section_end
+
+if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
+    # Build reth-bench once (non-instrumented) — reused for both phases.
+    gha_section_start "Building reth-bench"
+    RUSTFLAGS="$COMBINED_RUSTFLAGS" \
+        cargo build --profile "$PROFILE" --features "$FEATURES" \
+        --manifest-path bin/reth-bench/Cargo.toml --bin reth-bench --locked
+    RETH_BENCH_BIN="$(find target -name reth-bench -type f -executable | head -1)"
+    echo "reth-bench: $RETH_BENCH_BIN"
+    gha_section_end
+else
+    gha_section_start "Building reth-bench"
+    echo "Skipping reth-bench build (pre-collected PGO with SKIP_BOLT=true)"
+    gha_section_end
+fi
+
+# ── Helpers ────────────────────────────────────────────────────────────────────
+RETH_PID=
+cleanup() {
+    if [ -n "${RETH_PID:-}" ] && kill -0 "$RETH_PID" 2>/dev/null; then
+        echo "Stopping reth (pid $RETH_PID)..."
+        sudo kill "$RETH_PID" 2>/dev/null || true
+        for i in $(seq 1 60); do
+            sudo kill -0 "$RETH_PID" 2>/dev/null || break
+            if [ $((i % 10)) -eq 0 ]; then
+                echo "  waiting... (${i}s)"
+            fi
+            sleep 1
+        done
+        sudo kill -9 "$RETH_PID" 2>/dev/null || true
+    fi
+}
+trap cleanup EXIT
+
+# Start reth, wait for RPC, run reth-bench, then stop reth.
+# Arguments: $1 = reth binary path, $2 = number of blocks, $3 = log label
+run_bench_workload() {
+    local reth_bin="$1" blocks="$2" label="$3"
+    local http_port=8545 authrpc_port=8551
+
+    echo "--- Starting reth ($label) ---"
+    sudo "$reth_bin" node \
+        --datadir "$DATADIR" \
+        --log.file.directory "/tmp/reth-${label}-logs" \
+        --engine.accept-execution-requests-hash \
+        --http --http.port "$http_port" \
+        --authrpc.port "$authrpc_port" \
+        --disable-discovery --no-persist-peers \
+        > "/tmp/reth-${label}.log" 2>&1 &
+    RETH_PID=$!
+
+    echo "Waiting for reth RPC..."
+    for i in $(seq 1 120); do
+        if curl -sf "http://127.0.0.1:$http_port" -X POST \
+            -H 'Content-Type: application/json' \
+            -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
+            > /dev/null 2>&1; then
+            echo "reth is ready after ${i}s"
+            break
+        fi
+        if [ "$i" -eq 120 ]; then
+            echo "error: reth failed to start within 120s"
+            cat "/tmp/reth-${label}.log"
+            exit 1
+        fi
+        sleep 1
+    done
+
+    echo "Running reth-bench ($blocks blocks)..."
+    "$RETH_BENCH_BIN" new-payload-fcu \
+        --rpc-url "$RPC_URL" \
+        --engine-rpc-url "http://127.0.0.1:$authrpc_port" \
+        --jwt-secret "$DATADIR/jwt.hex" \
+        --advance "$blocks" \
+        --reth-new-payload 2>&1 | sed -u "s/^/[$label] /"
+
+    echo "Stopping reth ($label)..."
+    sudo kill "$RETH_PID" 2>/dev/null || true
+    for i in $(seq 1 60); do
+        sudo kill -0 "$RETH_PID" 2>/dev/null || break
+        sleep 1
+    done
+    sudo kill -9 "$RETH_PID" 2>/dev/null || true
+    RETH_PID=
+}
+
+publish_binary() {
+    local source_bin="$1"
+    for out in "target/$TARGET/$PROFILE_DIR" "target/$PROFILE_DIR"; do
+        local destination="$out/reth"
+        mkdir -p "$out"
+        # Skip copying when source and destination resolve to the same inode.
+        if [ -e "$destination" ] && [ "$source_bin" -ef "$destination" ]; then
+            continue
+        fi
+        cp "$source_bin" "$destination"
+    done
+}
+
+if [ "$USE_PRECOLLECTED_PGO" = true ]; then
+    gha_section_start "Phase 1: Using Pre-Collected PGO Profile"
+    rm -rf "$PGO_DIR"
+    mkdir -p "$PGO_DIR"
+    cp "$PGO_PROFDATA" "$PGO_DIR/merged.profdata"
+    echo "Using pre-collected profile: $PGO_PROFDATA"
+    echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
+    gha_section_end
+else
+    # ── Phase 1: PGO profile collection ───────────────────────────────────────
+    gha_section_start "Phase 1: PGO Profile Collection"
+
+    rm -rf "$PGO_DIR"
+    mkdir -p "$PGO_DIR"
+
+    echo "Building PGO-instrumented binary..."
+    RUSTFLAGS="-Cprofile-generate=$PGO_DIR -Crelocation-model=pic $COMBINED_RUSTFLAGS" \
+        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
+
+    PGO_RETH_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
+    echo "Instrumented binary: $PGO_RETH_BIN ($(ls -lh "$PGO_RETH_BIN" | awk '{print $5}'))"
+
+    run_bench_workload "$PGO_RETH_BIN" "$PGO_BLOCKS" "pgo"
+
+    # Fix ownership if reth ran as root.
+    sudo chown -R "$(id -un):$(id -gn)" "$PGO_DIR" 2>/dev/null || true
+
+    # Merge PGO profiles.
+    echo "Merging PGO profiles..."
+    PROFRAW_COUNT=$(find "$PGO_DIR" -name '*.profraw' | wc -l)
+    echo "Found $PROFRAW_COUNT .profraw files"
+    if [ "$PROFRAW_COUNT" -eq 0 ]; then
+        echo "error: no .profraw files — instrumented binary did not produce profiles"
+        exit 1
+    fi
+    "$LLVM_PROFDATA" merge -o "$PGO_DIR/merged.profdata" "$PGO_DIR"/*.profraw
+    echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
+    gha_section_end
+fi
+
+if [ "$COLLECT_PGO_ONLY_BOOL" = true ]; then
+    gha_section_start "PGO Collection Complete"
+    echo "COLLECT_PGO_ONLY=true, skipping PGO/BOLT optimized binary build"
+    echo "Profile: $PGO_DIR/merged.profdata"
+    gha_section_end
+    exit 0
+fi
+
+if [ "$SKIP_BOLT_BOOL" = true ]; then
+    gha_section_start "BOLT Phase Skipped"
+    echo "SKIP_BOLT=$SKIP_BOLT, building PGO-only binary"
+    echo "Building PGO-optimized binary..."
+    RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata $COMBINED_RUSTFLAGS" \
+        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
+
+    BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
+    if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
+        echo "Stripping debug symbols..."
+        strip "$BUILT_BIN"
+    else
+        echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
+    fi
+    publish_binary "$BUILT_BIN"
+    gha_section_end
+else
+    # ── Phase 2: BOLT profile collection (with PGO) ──────────────────────────
+    gha_section_start "Phase 2: BOLT Profile Collection (with PGO)"
+
+    rm -rf "$BOLT_DIR"
+    mkdir -p "$BOLT_DIR"
+
+    echo "Building BOLT-instrumented binary with PGO..."
+    # --emit-relocs preserves relocation entries in the binary, required by llvm-bolt -instrument
+    RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
+        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
+
+    # Instrument with BOLT
+    BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
+    BOLT_INSTRUMENTED_BIN="$BUILT_BIN-bolt-instrumented"
+
+    echo "Instrumenting binary with BOLT..."
+    # --skip-funcs: skip compiler-generated drop_in_place functions that BOLT can't handle
+    # as split functions in relocation mode (triggered by --emit-relocs)
+    llvm-bolt "$BUILT_BIN" \
+        -instrument \
+        --instrumentation-file-append-pid \
+        --instrumentation-file="$BOLT_DIR/prof" \
+        --skip-funcs='.*drop_in_place.*' \
+        -o "$BOLT_INSTRUMENTED_BIN"
+    echo "BOLT-instrumented binary: $BOLT_INSTRUMENTED_BIN ($(ls -lh "$BOLT_INSTRUMENTED_BIN" | awk '{print $5}'))"
+
+    run_bench_workload "$BOLT_INSTRUMENTED_BIN" "$BOLT_BLOCKS" "bolt"
+
+    # Fix ownership for BOLT profiles
+    sudo chown -R "$(id -un):$(id -gn)" "$BOLT_DIR" 2>/dev/null || true
+
+    # Merge BOLT profiles
+    echo "Merging BOLT profiles..."
+    FDATA_COUNT=$(find "$BOLT_DIR" -name '*.fdata' | wc -l)
+    echo "Found $FDATA_COUNT .fdata files"
+    if [ "$FDATA_COUNT" -eq 0 ]; then
+        echo "error: no .fdata files — BOLT-instrumented binary did not produce profiles"
+        exit 1
+    fi
+    merge-fdata "$BOLT_DIR"/*.fdata > "$BOLT_DIR/merged.fdata"
+    echo "BOLT profile: $BOLT_DIR/merged.fdata ($(ls -lh "$BOLT_DIR/merged.fdata" | awk '{print $5}'))"
+    gha_section_end
+
+    # ── Phase 3: Final optimized build ───────────────────────────────────────
+    gha_section_start "Phase 3: Final PGO+BOLT Optimized Build"
+
+    echo "Building PGO-optimized binary..."
+    # --emit-relocs preserves relocation entries in the binary, required by llvm-bolt for code reordering
+    RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
+        cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
+
+    BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
+    OPTIMIZED_BIN="$BUILT_BIN-bolt-optimized"
+
+    echo "Optimizing with BOLT..."
+    llvm-bolt "$BUILT_BIN" \
+        -o "$OPTIMIZED_BIN" \
+        --data "$BOLT_DIR/merged.fdata" \
+        -reorder-blocks=ext-tsp \
+        -reorder-functions=cdsort \
+        -split-functions \
+        -split-all-cold \
+        -dyno-stats \
+        -icf=1 \
+        -use-gnu-stack \
+        --skip-funcs='.*drop_in_place.*'
+
+    if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
+        echo "Stripping debug symbols..."
+        strip "$OPTIMIZED_BIN"
+    else
+        echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
+    fi
+    publish_binary "$OPTIMIZED_BIN"
+    gha_section_end
+fi
+
+gha_section_start "Build Complete"
+ls -lh "target/$PROFILE_DIR/reth"
+echo "Output: target/$PROFILE_DIR/reth"
+gha_section_end

.github/scripts/hive/expected_failures.yaml

@@ -16,28 +16,11 @@ rpc-compat:
   # syncing mode, the test expects syncing to be false on start
   - eth_syncing/check-syncing (reth)
 
-# no fix due to https://github.com/paradigmxyz/reth/issues/8732
-engine-withdrawals:
-  - Withdrawals Fork On Genesis (Paris) (reth)
-  - Withdrawals Fork on Block 1 (Paris) (reth)
-  - Withdrawals Fork on Block 2 (Paris) (reth)
-  - Withdrawals Fork on Block 3 (Paris) (reth)
-  - Withdraw to a single account (Paris) (reth)
-  - Withdraw to two accounts (Paris) (reth)
-  - Withdraw many accounts (Paris) (reth)
-  - Withdraw zero amount (Paris) (reth)
-  - Empty Withdrawals (Paris) (reth)
-  - Corrupted Block Hash Payload (INVALID) (Paris) (reth)
-  - Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org (Paris) (reth)
+engine-withdrawals: [ ]
 
 engine-api: [ ]
 
-# no fix due to https://github.com/paradigmxyz/reth/issues/8732
-engine-cancun:
-  - Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True (Cancun) (reth)
-  # the test fails with older versions of the code for which it passed before, probably related to changes
-  # in hive or its dependencies
-  - Blob Transaction Ordering, Multiple Clients (Cancun) (reth)
+engine-cancun: [ ]
 
 sync: [ ]
 
@@ -49,7 +32,7 @@ engine-auth: [ ]
 #   The test artificially creates an empty account with storage, then tests EIP-7610's behavior.
 #   On mainnet, ~25 such accounts exist as contract addresses (derived from keccak(prefix, caller,
 #   nonce/salt), not from public keys). No private key exists for contract addresses. To trigger
-#   this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible. 
+#   this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible.
 #
 # tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_*
 #   Requires hash collision on create2 address to target already deployed accounts with storage.
@@ -59,10 +42,6 @@ engine-auth: [ ]
 #
 # System contract tests (already fixed and deployed):
 #
-# tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout and test_invalid_log_length
-#   System contract is already fixed and deployed; tests cover scenarios where contract is
-#   malformed which can't happen retroactively. No point in adding checks.
-#
 # tests/prague/eip7002_el_triggerable_withdrawals/test_contract_deployment.py::test_system_contract_deployment
 # tests/prague/eip7251_consolidations/test_contract_deployment.py::test_system_contract_deployment
 #   Post-fork system contract deployment tests. Should fix for spec compliance but not realistic
@@ -71,32 +50,8 @@ eels/consume-engine:
   - tests/prague/eip7702_set_code_tx/test_set_code_txs.py::test_set_code_to_non_empty_storage[fork_Prague-blockchain_test_engine-zero_nonce]-reth
   - tests/prague/eip7251_consolidations/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test_engine-deploy_after_fork-nonzero_balance]-reth
   - tests/prague/eip7251_consolidations/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test_engine-deploy_after_fork-zero_balance]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_amount_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_amount_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_index_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_index_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_pubkey_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_pubkey_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_signature_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_signature_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_withdrawal_credentials_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Prague-blockchain_test_engine-log_argument_withdrawal_credentials_size-value_zero]-reth
   - tests/prague/eip7002_el_triggerable_withdrawals/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test_engine-deploy_after_fork-nonzero_balance]-reth
   - tests/prague/eip7002_el_triggerable_withdrawals/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test_engine-deploy_after_fork-zero_balance]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_log_length[fork_Prague-blockchain_test_engine-slice_bytes_False]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_log_length[fork_Prague-blockchain_test_engine-slice_bytes_True]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_amount_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_amount_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_index_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_index_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_pubkey_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_pubkey_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_signature_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_signature_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_withdrawal_credentials_offset-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_layout[fork_Osaka-blockchain_test_engine-log_argument_withdrawal_credentials_size-value_zero]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_log_length[fork_Osaka-blockchain_test_engine-slice_bytes_False]-reth
-  - tests/prague/eip6110_deposits/test_modified_contract.py::test_invalid_log_length[fork_Osaka-blockchain_test_engine-slice_bytes_True]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Osaka-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Paris-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth

.github/scripts/hive/ignored_tests.yaml

@@ -11,17 +11,16 @@
 #
 # When a test should no longer be ignored, remove it from this list.
 
-# flaky
-engine-withdrawals:
-  - Withdrawals Fork on Block 1 - 8 Block Re-Org NewPayload (Paris) (reth)
-  - Withdrawals Fork on Block 8 - 10 Block Re-Org NewPayload (Paris) (reth)
-  - Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org (Paris) (reth)
-  - Sync after 128 blocks - Withdrawals on Block 2 - Multiple Withdrawal Accounts (Paris) (reth)
 engine-cancun:
   - Transaction Re-Org, New Payload on Revert Back (Cancun) (reth)
   - Transaction Re-Org, Re-Org to Different Block (Cancun) (reth)
   - Transaction Re-Org, Re-Org Out (Cancun) (reth)
   - Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=False, Invalid P9 (Cancun) (reth)
+  # Hive test infra bug: geth sidecar switched to PathScheme for state storage, which has
+  # strict trie integrity requirements incompatible with inserting intentionally invalid blocks.
+  # Affects all clients, not just reth. Tracked: https://github.com/ethereum/hive/issues/1382
+  - Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (reth)
+  - Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (reth)
   - Multiple New Payloads Extending Canonical Chain, Wait for Canonical Payload (Cancun) (reth)
 engine-api:
   - Transaction Re-Org, Re-Org Out (Paris) (reth)

.github/scripts/hive/run_simulator.sh

@@ -6,8 +6,14 @@ cd hivetests/
 sim="${1}"
 limit="${2}"
 
+# Use lower parallelism for eels tests to avoid OOM-killing the runner
+parallelism=16
+if [[ "${sim}" == *"eels"* ]]; then
+    parallelism=4
+fi
+
 run_hive() {
-    hive --sim "${sim}" --sim.limit "${limit}" --sim.parallelism 16 --client reth 2>&1 | tee /tmp/log || true
+    hive --sim "${sim}" --sim.limit "${limit}" --sim.parallelism "${parallelism}" --client reth 2>&1 | tee /tmp/log || true
 }
 
 check_log() {

.github/workflows/bench-scheduled.yml

@@ -0,0 +1,865 @@
+# Nightly regression benchmark.
+#
+# Compares the previous nightly build against the current nightly build to
+# detect performance regressions. Runs daily after docker.yml produces a new
+# nightly image at 01:00 UTC.
+#
+# State is persisted between runs via GitHub Actions cache: each successful
+# run saves the feature commit SHA so the next run knows what to compare against.
+
+on:
+  schedule:
+    - cron: "30 5 * * *" # 06:30 UTC daily
+  workflow_dispatch:
+    inputs:
+      force:
+        description: "Force run even if no new nightly (bypass skip logic)"
+        required: false
+        default: false
+        type: boolean
+      no_slack:
+        description: "Suppress Slack notifications"
+        required: false
+        default: true
+        type: boolean
+
+env:
+  CARGO_TERM_COLOR: always
+  RUSTC_WRAPPER: "sccache"
+
+name: bench-scheduled
+
+permissions:
+  contents: read
+  actions: read
+
+jobs:
+  # ---------------------------------------------------------------------------
+  # Job 1: Resolve nightly refs, check staleness, manage state
+  # ---------------------------------------------------------------------------
+  resolve-refs:
+    name: resolve-refs
+    runs-on: ubuntu-latest
+    outputs:
+      baseline-ref: ${{ steps.refs.outputs.baseline-ref }}
+      feature-ref: ${{ steps.refs.outputs.feature-ref }}
+      should-skip: ${{ steps.refs.outputs.should-skip }}
+      is-stale: ${{ steps.refs.outputs.is-stale }}
+      stale-age-hours: ${{ steps.refs.outputs.stale-age-hours }}
+      nightly-created: ${{ steps.refs.outputs.nightly-created }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          sparse-checkout: .github/scripts
+          sparse-checkout-cone-mode: true
+
+      - name: Restore nightly state
+        id: state-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: .nightly-state
+          key: bench-scheduled-state-dummy
+          restore-keys: |
+            bench-scheduled-state-
+
+      - name: Resolve nightly refs
+        id: refs
+        env:
+          GH_TOKEN: ${{ github.token }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+        run: |
+          FORCE="${{ inputs.force || 'false' }}"
+          .github/scripts/bench-scheduled-refs.sh "$FORCE"
+
+      - name: Alert on stale nightly
+        if: steps.refs.outputs.is-stale == 'true'
+        uses: actions/github-script@v8
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) {
+              core.warning('Slack credentials not set, skipping stale nightly alert');
+              return;
+            }
+
+            const ageHours = '${{ steps.refs.outputs.stale-age-hours }}';
+            const created = '${{ steps.refs.outputs.nightly-created }}';
+            const featureRef = '${{ steps.refs.outputs.feature-ref }}';
+            const shortSha = featureRef.slice(0, 8);
+            const repo = '${{ github.repository }}';
+            const runUrl = `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: ':rotating_light: Nightly Regression: nightly build is stale', emoji: true },
+              },
+              {
+                type: 'section',
+                text: {
+                  type: 'mrkdwn',
+                  text: [
+                    '*Nightly regression did not run* — nightly build is stale',
+                    '',
+                    `The latest nightly image was built from a commit that is *${ageHours}h old* (threshold: 24h).`,
+                    `This means today's nightly docker build likely failed and no new image was produced.`,
+                    '',
+                    `Stale commit: \`${shortSha}\` (built at ${created})`,
+                    '',
+                    '*Action required:* Check the <https://github.com/' + repo + '/actions/workflows/docker.yml|docker.yml> workflow for failures.',
+                  ].join('\n'),
+                },
+              },
+              {
+                type: 'actions',
+                elements: [
+                  {
+                    type: 'button',
+                    text: { type: 'plain_text', text: 'View Run :github:', emoji: true },
+                    url: runUrl,
+                    action_id: 'ci_button',
+                  },
+                ],
+              },
+            ];
+
+            const resp = await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: {
+                'Authorization': `Bearer ${token}`,
+                'Content-Type': 'application/json',
+              },
+              body: JSON.stringify({
+                channel,
+                blocks,
+                text: 'Nightly regression: nightly build is stale',
+                unfurl_links: false,
+              }),
+            });
+            const data = await resp.json();
+            if (!data.ok) {
+              core.warning(`Slack API error: ${JSON.stringify(data)}`);
+            }
+
+      - name: Fail on stale nightly
+        if: steps.refs.outputs.is-stale == 'true'
+        run: |
+          echo "::error::Nightly build is stale (>24h old). Aborting."
+          exit 1
+
+  # ---------------------------------------------------------------------------
+  # Job 2: Run the benchmark
+  # ---------------------------------------------------------------------------
+  bench-scheduled:
+    needs: resolve-refs
+    if: |
+      needs.resolve-refs.outputs.should-skip != 'true' &&
+      needs.resolve-refs.outputs.is-stale != 'true'
+    name: bench-scheduled
+    runs-on: [self-hosted, Linux, X64, available]
+    timeout-minutes: 120
+    env:
+      BENCH_RPC_URL: https://ethereum.reth.rs/rpc
+      SCHELK_MOUNT: /reth-bench
+      BENCH_WORK_DIR: ${{ github.workspace }}/bench-work
+      BENCH_PR: ""
+      BENCH_ACTOR: "nightly-regression"
+      BENCH_BLOCKS: "2000"
+      BENCH_WARMUP_BLOCKS: "500"
+      BENCH_SAMPLY: "false"
+      BENCH_CORES: "0"
+      BENCH_BIG_BLOCKS: "false"
+      BENCH_RETH_NEW_PAYLOAD: "true"
+      BENCH_WAIT_TIME: ""
+      BENCH_BASELINE_ARGS: ""
+      BENCH_FEATURE_ARGS: ""
+      BENCH_ABBA: "true"
+      BENCH_COMMENT_ID: ""
+      BENCH_NO_SLACK: ${{ github.event_name == 'workflow_dispatch' && inputs.no_slack == true && 'true' || 'false' }}
+      BENCH_METRICS_ADDR: "127.0.0.1:9100"
+      BENCH_OTLP_TRACES_ENDPOINT: ${{ secrets.BENCH_OTLP_TRACES_ENDPOINT }}
+      BENCH_OTLP_LOGS_ENDPOINT: ${{ secrets.BENCH_OTLP_LOGS_ENDPOINT }}
+      BASELINE_REF: ${{ needs.resolve-refs.outputs.baseline-ref }}
+      FEATURE_REF: ${{ needs.resolve-refs.outputs.feature-ref }}
+    steps:
+      - name: Clean up previous bench-work
+        run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
+
+      - uses: actions/checkout@v6
+        with:
+          submodules: true
+          fetch-depth: 0
+          ref: ${{ needs.resolve-refs.outputs.feature-ref }}
+
+      - name: Resolve job URL
+        id: job-url
+        uses: actions/github-script@v8
+        with:
+          script: |
+            const { data: jobs } = await github.rest.actions.listJobsForWorkflowRun({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              run_id: context.runId,
+            });
+            const job = jobs.jobs.find(j => j.name === 'bench-scheduled');
+            const jobUrl = job ? job.html_url : `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+            core.exportVariable('BENCH_JOB_URL', jobUrl);
+
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: mozilla-actions/sccache-action@v0.0.9
+        continue-on-error: true
+
+      - name: Install dependencies
+        env:
+          DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
+        run: |
+          mkdir -p "$HOME/.local/bin"
+
+          # apt packages
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends \
+            python3 make jq zstd curl dmsetup \
+            linux-tools-"$(uname -r)" || \
+            sudo apt-get install -y --no-install-recommends linux-tools-generic
+
+          # mc (MinIO client)
+          if ! command -v mc &>/dev/null; then
+            curl -sSfL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$HOME/.local/bin/mc"
+            chmod +x "$HOME/.local/bin/mc"
+          fi
+
+          # uv (Python package manager)
+          if ! command -v uv &>/dev/null; then
+            curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$HOME/.local/bin" sh
+          fi
+
+          # Configure git auth for private repos
+          git config --global url."https://x-access-token:${DEREK_TOKEN}@github.com/".insteadOf "https://github.com/"
+
+          # thin-provisioning-tools (era_invalidate, required by schelk)
+          if ! command -v era_invalidate &>/dev/null; then
+            git clone --depth 1 https://github.com/jthornber/thin-provisioning-tools /tmp/tpt
+            sudo make -C /tmp/tpt install
+            rm -rf /tmp/tpt
+          fi
+
+          # schelk (snapshot rollback tool, invoked via sudo)
+          if ! sudo sh -c 'command -v schelk' &>/dev/null; then
+            cargo install --git https://github.com/tempoxyz/schelk --locked
+            sudo install "$HOME/.cargo/bin/schelk" /usr/local/bin/
+          fi
+
+      - name: Check dependencies
+        run: |
+          export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
+          missing=()
+          for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq; do
+            command -v "$cmd" &>/dev/null || missing+=("$cmd")
+          done
+          if [ ${#missing[@]} -gt 0 ]; then
+            echo "::error::Missing required tools: ${missing[*]}"
+            exit 1
+          fi
+          echo "All dependencies found"
+
+      - name: Resolve display names
+        id: refs
+        run: |
+          BASELINE_SHORT=$(echo "$BASELINE_REF" | cut -c1-8)
+          FEATURE_SHORT=$(echo "$FEATURE_REF" | cut -c1-8)
+          echo "baseline-name=nightly-${BASELINE_SHORT}" >> "$GITHUB_OUTPUT"
+          echo "feature-name=nightly-${FEATURE_SHORT}" >> "$GITHUB_OUTPUT"
+          echo "baseline-ref=$BASELINE_REF" >> "$GITHUB_OUTPUT"
+          echo "feature-ref=$FEATURE_REF" >> "$GITHUB_OUTPUT"
+
+      - name: Check if snapshot needs update
+        id: snapshot-check
+        run: |
+          if .github/scripts/bench-reth-snapshot.sh --check; then
+            echo "needed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "needed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Prepare source dirs
+        run: |
+          if [ -d ../reth-baseline ]; then
+            git -C ../reth-baseline fetch origin "$BASELINE_REF"
+          else
+            git clone . ../reth-baseline
+          fi
+          git -C ../reth-baseline checkout "$BASELINE_REF"
+
+          if [ -d ../reth-feature ]; then
+            git -C ../reth-feature fetch origin "$FEATURE_REF"
+          else
+            git clone . ../reth-feature
+          fi
+          git -C ../reth-feature checkout "$FEATURE_REF"
+
+      - name: Build binaries and download snapshot in parallel
+        id: build
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BENCH_REPO: ${{ github.repository }}
+          SNAPSHOT_NEEDED: ${{ steps.snapshot-check.outputs.needed }}
+        run: |
+          BASELINE_DIR="$(cd ../reth-baseline && pwd)"
+          FEATURE_DIR="$(cd ../reth-feature && pwd)"
+
+          .github/scripts/bench-reth-build.sh baseline "${BASELINE_DIR}" "$BASELINE_REF" &
+          PID_BASELINE=$!
+          .github/scripts/bench-reth-build.sh feature "${FEATURE_DIR}" "$FEATURE_REF" &
+          PID_FEATURE=$!
+
+          PID_SNAPSHOT=
+          if [ "$SNAPSHOT_NEEDED" = "true" ]; then
+            .github/scripts/bench-reth-snapshot.sh &
+            PID_SNAPSHOT=$!
+          fi
+
+          FAIL=0
+          wait $PID_BASELINE || FAIL=1
+          wait $PID_FEATURE || FAIL=1
+          [ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
+          if [ $FAIL -ne 0 ]; then
+            echo "::error::One or more parallel tasks failed (builds / snapshot download)"
+            exit 1
+          fi
+
+      # System tuning for reproducible benchmarks
+      - name: System setup
+        run: |
+          sudo cpupower frequency-set -g performance || true
+          # Disable turbo boost (Intel and AMD paths)
+          echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
+          echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
+          sudo swapoff -a || true
+          echo 0 | sudo tee /proc/sys/kernel/randomize_va_space || true
+          # Disable SMT (hyperthreading)
+          for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
+            first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
+            current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
+            if [ "$current" != "$first" ]; then
+              echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" || true
+            fi
+          done
+          echo "Online CPUs: $(nproc)"
+          # Disable transparent huge pages
+          for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
+            [ -d "$p" ] && echo never | sudo tee "$p/enabled" && echo never | sudo tee "$p/defrag" && break
+          done || true
+          # Prevent deep C-states
+          sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
+          # Move all IRQs to core 0
+          for irq in /proc/irq/*/smp_affinity_list; do
+            echo 0 | sudo tee "$irq" 2>/dev/null || true
+          done
+          # Stop noisy background services
+          sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
+          echo "=== Benchmark environment ==="
+          uname -r
+          lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA'
+          cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
+          cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
+          cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo "THP: unknown"
+          free -h
+
+      - name: Pre-flight cleanup
+        run: |
+          sudo pkill -9 reth || true
+          sleep 1
+          if mountpoint -q "$SCHELK_MOUNT"; then
+            sudo umount -l "$SCHELK_MOUNT" || true
+            sudo schelk recover -y || true
+          fi
+          rm -rf "$BENCH_WORK_DIR"
+          mkdir -p "$BENCH_WORK_DIR"
+
+      - name: Start metrics proxy
+        run: |
+          BENCH_ID="nightly-${{ github.run_id }}"
+          BENCH_REFERENCE_EPOCH=$(date +%s)
+          echo "BENCH_ID=${BENCH_ID}" >> "$GITHUB_ENV"
+          echo "BENCH_REFERENCE_EPOCH=${BENCH_REFERENCE_EPOCH}" >> "$GITHUB_ENV"
+
+          LABELS_FILE="/tmp/bench-metrics-labels.json"
+          echo '{}' > "$LABELS_FILE"
+          echo "BENCH_LABELS_FILE=${LABELS_FILE}" >> "$GITHUB_ENV"
+
+          python3 .github/scripts/bench-metrics-proxy.py \
+            --labels "$LABELS_FILE" \
+            --upstream "http://${BENCH_METRICS_ADDR}/" \
+            --subnet 10.10.0.0/24 \
+            --port 9090 &
+          PROXY_PID=$!
+          echo "BENCH_METRICS_PROXY_PID=${PROXY_PID}" >> "$GITHUB_ENV"
+          echo "Metrics proxy started (PID $PROXY_PID)"
+
+      # Interleaved run order (B-F-F-B) to reduce systematic bias
+      - name: "Run benchmark: baseline (1/2)"
+        id: run-baseline-1
+        run: |
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
+
+      - name: "Run benchmark: feature (1/2)"
+        id: run-feature-1
+        run: |
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
+
+      - name: "Run benchmark: feature (2/2)"
+        id: run-feature-2
+        run: |
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
+
+      - name: "Run benchmark: baseline (2/2)"
+        id: run-baseline-2
+        run: |
+          LAST_RUN_START=$(date +%s)
+          echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
+          cat > "$BENCH_LABELS_FILE" <<LABELS
+          {"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+          LABELS
+          taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
+
+      - name: Stop metrics proxy & generate Grafana URL
+        id: metrics
+        if: "!cancelled()"
+        run: |
+          kill "$BENCH_METRICS_PROXY_PID" 2>/dev/null || true
+
+          LAST_RUN_DURATION=$(( $(date +%s) - BENCH_LAST_RUN_START ))
+          FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
+          TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
+          GRAFANA_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=ef57fux92e9z4e&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
+          echo "grafana-url=${GRAFANA_URL}" >> "$GITHUB_OUTPUT"
+          echo "Grafana URL: ${GRAFANA_URL}"
+
+      - name: Scan logs for errors
+        if: "!cancelled()"
+        run: |
+          ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
+          found=false
+          for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
+            LOG="$BENCH_WORK_DIR/$run_dir/node.log"
+            if [ ! -f "$LOG" ]; then continue; fi
+
+            panics=$(grep -c -E 'panicked at' "$LOG" || true)
+            errors=$(grep -c ' ERROR ' "$LOG" || true)
+
+            if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
+              if [ "$found" = false ]; then
+                printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
+                found=true
+              fi
+              printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' "$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
+              if [ "$panics" -gt 0 ]; then
+                printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
+                grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
+                printf '```\n' >> "$ERRORS_FILE"
+              fi
+              if [ "$errors" -gt 0 ]; then
+                printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
+                grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
+                printf '```\n' >> "$ERRORS_FILE"
+              fi
+              printf '\n</details>\n\n' >> "$ERRORS_FILE"
+            fi
+          done
+
+      - name: Parse results
+        id: results
+        if: success()
+        env:
+          BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
+          FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
+        run: |
+          SUMMARY_ARGS="--output-summary $BENCH_WORK_DIR/summary.json"
+          SUMMARY_ARGS="$SUMMARY_ARGS --output-markdown $BENCH_WORK_DIR/comment.md"
+          SUMMARY_ARGS="$SUMMARY_ARGS --repo ${{ github.repository }}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --baseline-ref ${BASELINE_REF}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --baseline-name ${BASELINE_NAME}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --feature-name ${FEATURE_NAME}"
+          SUMMARY_ARGS="$SUMMARY_ARGS --feature-ref ${FEATURE_REF}"
+
+          BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
+          FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
+          BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+          FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
+          SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BASELINE_CSVS"
+          SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $FEATURE_CSVS"
+          SUMMARY_ARGS="$SUMMARY_ARGS --gas-csv $BENCH_WORK_DIR/feature-1/total_gas.csv"
+
+          GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
+          if [ -n "$GRAFANA_URL" ]; then
+            SUMMARY_ARGS="$SUMMARY_ARGS --grafana-url $GRAFANA_URL"
+          fi
+          # shellcheck disable=SC2086
+          python3 .github/scripts/bench-reth-summary.py $SUMMARY_ARGS
+
+      - name: Generate charts
+        if: success()
+        env:
+          BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
+          FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
+        run: |
+          CHART_ARGS="--output-dir $BENCH_WORK_DIR/charts"
+          FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
+          BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
+          FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
+          BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+          CHART_ARGS="$CHART_ARGS --feature $FEATURE_CSVS"
+          CHART_ARGS="$CHART_ARGS --baseline $BASELINE_CSVS"
+          CHART_ARGS="$CHART_ARGS --baseline-name ${BASELINE_NAME}"
+          CHART_ARGS="$CHART_ARGS --feature-name ${FEATURE_NAME}"
+          # shellcheck disable=SC2086
+          uv run --with matplotlib python3 .github/scripts/bench-reth-charts.py $CHART_ARGS
+
+      - name: Upload results
+        if: "!cancelled()"
+        uses: actions/upload-artifact@v7
+        with:
+          name: bench-scheduled-results
+          path: ${{ env.BENCH_WORK_DIR }}
+
+      - name: Push charts
+        id: push-charts
+        if: success()
+        run: |
+          RUN_ID=${{ github.run_id }}
+          CHART_DIR="nightly/${RUN_ID}"
+          CHARTS_REPO="https://x-access-token:${{ secrets.DEREK_TOKEN }}@github.com/decofe/reth-bench-charts.git"
+
+          TMP_DIR=$(mktemp -d)
+          if git clone --depth 1 "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
+            true
+          else
+            git init "${TMP_DIR}"
+            git -C "${TMP_DIR}" remote add origin "${CHARTS_REPO}"
+          fi
+
+          mkdir -p "${TMP_DIR}/${CHART_DIR}"
+          cp "$BENCH_WORK_DIR"/charts/*.png "${TMP_DIR}/${CHART_DIR}/"
+          git -C "${TMP_DIR}" add "${CHART_DIR}"
+          git -C "${TMP_DIR}" -c user.name="github-actions" -c user.email="github-actions@github.com" \
+            commit -m "nightly bench charts for run ${RUN_ID}"
+          git -C "${TMP_DIR}" push origin HEAD:main
+          echo "sha=$(git -C "${TMP_DIR}" rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+          rm -rf "${TMP_DIR}"
+
+      - name: Write job summary
+        if: success()
+        uses: actions/github-script@v8
+        with:
+          script: |
+            const fs = require('fs');
+            const { verdict, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
+
+            let summary;
+            try {
+              summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+            } catch (e) {
+              await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
+              return;
+            }
+
+            const repo = `${context.repo.owner}/${context.repo.repo}`;
+            const commitUrl = `https://github.com/${repo}/commit`;
+            const { emoji, label } = verdict(summary.changes);
+            const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
+            const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
+            const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+
+            let md = `# ${emoji} Nightly Regression: ${label}\n\n`;
+            md += `**Baseline:** ${baselineLink}\n`;
+            md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
+            md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
+
+            const rows = metricRows(summary);
+            md += `| Metric | Baseline | Feature | Change |\n`;
+            md += `|--------|----------|---------|--------|\n`;
+            for (const r of rows) {
+              md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
+            }
+            md += '\n';
+
+            const wtRows = waitTimeRows(summary);
+            if (wtRows.length > 0) {
+              md += `### Wait Time Breakdown\n\n`;
+              md += `| Metric | Baseline | Feature |\n`;
+              md += `|--------|----------|--------|\n`;
+              for (const r of wtRows) {
+                md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
+              }
+              md += '\n';
+            }
+
+            // Charts
+            const chartSha = '${{ steps.push-charts.outputs.sha }}';
+            if (chartSha) {
+              const runId = '${{ github.run_id }}';
+              const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/nightly/${runId}`;
+              const charts = [
+                { file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
+                { file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
+                { file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
+              ];
+              md += `### Charts\n\n`;
+              for (const chart of charts) {
+                md += `<details><summary>${chart.label}</summary>\n\n`;
+                md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
+                md += `</details>\n\n`;
+              }
+            }
+
+            const grafanaUrl = '${{ steps.metrics.outputs.grafana-url }}';
+            if (grafanaUrl) {
+              md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
+            }
+
+            try {
+              const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
+              if (errors.trim()) md += '\n' + errors + '\n';
+            } catch {}
+
+            await core.summary.addRaw(md).write();
+
+      - name: Send Slack notification (success)
+        if: success() && env.BENCH_NO_SLACK != 'true'
+        uses: actions/github-script@v8
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const fs = require('fs');
+            const { verdict, fmtChange, fmtMs, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
+
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) {
+              core.info('Slack credentials not set, skipping notification');
+              return;
+            }
+
+            let summary;
+            try {
+              summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+            } catch (e) {
+              core.warning('Could not read summary.json for Slack notification');
+              return;
+            }
+
+            // Only notify on significant changes (regression OR improvement)
+            const changes = summary.changes || {};
+            const hasSignificant = Object.values(changes).some(c => c.sig === 'good' || c.sig === 'bad');
+            if (!hasSignificant) {
+              core.info('No significant changes detected, skipping nightly Slack notification');
+              return;
+            }
+
+            const SLACK_VERDICT = {
+              '⚠️': ':warning:',
+              '❌': ':x:',
+              '✅': ':white_check_mark:',
+              '⚪': ':white_circle:',
+            };
+
+            const repo = `${context.repo.owner}/${context.repo.repo}`;
+            const { emoji, label } = verdict(changes);
+            const headerEmoji = SLACK_VERDICT[emoji] || emoji;
+            const commitUrl = `https://github.com/${repo}/commit`;
+            const baselineLink = `<${commitUrl}/${summary.baseline.ref}|${summary.baseline.name}>`;
+            const featureLink = `<${commitUrl}/${summary.feature.ref}|${summary.feature.name}>`;
+            const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+            const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+
+            function cell(text) { return { type: 'raw_text', text: String(text) || ' ' }; }
+
+            const sectionText = [
+              '*Nightly Regression*',
+              '',
+              `*Baseline:* ${baselineLink}`,
+              `*Feature:* ${featureLink}`,
+              blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | '),
+            ].join('\n');
+
+            const rows = metricRows(summary);
+            const tableRows = [
+              [cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
+              ...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
+            ];
+
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: `${headerEmoji} Nightly: ${label}`, emoji: true },
+              },
+              {
+                type: 'section',
+                text: { type: 'mrkdwn', text: sectionText },
+              },
+              {
+                type: 'table',
+                column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }, { align: 'right' }],
+                rows: tableRows,
+              },
+              {
+                type: 'actions',
+                elements: [
+                  {
+                    type: 'button',
+                    text: { type: 'plain_text', text: 'CI :github:', emoji: true },
+                    url: jobUrl,
+                    action_id: 'ci_button',
+                  },
+                  {
+                    type: 'button',
+                    text: { type: 'plain_text', text: 'Diff :github:', emoji: true },
+                    url: diffUrl,
+                    action_id: 'diff_button',
+                  },
+                ],
+              },
+            ];
+
+            const text = `Nightly Regression: ${summary.baseline.name} vs ${summary.feature.name}`;
+            const resp = await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: {
+                'Authorization': `Bearer ${token}`,
+                'Content-Type': 'application/json',
+              },
+              body: JSON.stringify({ channel, blocks, text, unfurl_links: false }),
+            });
+            const data = await resp.json();
+            if (!data.ok) {
+              core.warning(`Slack API error: ${JSON.stringify(data)}`);
+              return;
+            }
+
+            // Post wait time breakdown as threaded reply
+            const wtRows = waitTimeRows(summary);
+            if (data.ts && wtRows.length > 0) {
+              const waitTableRows = [
+                [cell('Wait Time'), cell('Baseline'), cell('Feature')],
+                ...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
+              ];
+              await fetch('https://slack.com/api/chat.postMessage', {
+                method: 'POST',
+                headers: {
+                  'Authorization': `Bearer ${token}`,
+                  'Content-Type': 'application/json',
+                },
+                body: JSON.stringify({
+                  channel,
+                  thread_ts: data.ts,
+                  blocks: [{
+                    type: 'table',
+                    column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }],
+                    rows: waitTableRows,
+                  }],
+                  text: 'Wait time breakdown',
+                  unfurl_links: false,
+                }),
+              });
+            }
+
+      - name: Send Slack notification (failure)
+        if: failure()
+        uses: actions/github-script@v8
+        env:
+          SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
+          SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
+        with:
+          script: |
+            const token = process.env.SLACK_BENCH_BOT_TOKEN;
+            const channel = process.env.SLACK_BENCH_CHANNEL;
+            if (!token || !channel) return;
+
+            const steps_status = [
+              ['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
+              ['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
+              ['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
+              ['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
+              ['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
+            ];
+            const failed = steps_status.find(([, o]) => o === 'failure');
+            const failedStep = failed ? failed[0] : 'unknown step';
+
+            const repo = `${context.repo.owner}/${context.repo.repo}`;
+            const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
+
+            const blocks = [
+              {
+                type: 'header',
+                text: { type: 'plain_text', text: ':rotating_light: Nightly Bench Failed', emoji: true },
+              },
+              {
+                type: 'section',
+                text: { type: 'mrkdwn', text: `*Nightly regression* failed while *${failedStep}*` },
+              },
+              {
+                type: 'actions',
+                elements: [{
+                  type: 'button',
+                  text: { type: 'plain_text', text: 'View Logs :github:', emoji: true },
+                  url: jobUrl,
+                  action_id: 'ci_button',
+                }],
+              },
+            ];
+
+            await fetch('https://slack.com/api/chat.postMessage', {
+              method: 'POST',
+              headers: {
+                'Authorization': `Bearer ${token}`,
+                'Content-Type': 'application/json',
+              },
+              body: JSON.stringify({
+                channel,
+                blocks,
+                text: `Nightly bench failed while ${failedStep}`,
+                unfurl_links: false,
+              }),
+            });
+
+      - name: Restore system settings
+        if: always()
+        run: |
+          sudo systemctl start irqbalance cron atd 2>/dev/null || true
+
+  # ---------------------------------------------------------------------------
+  # Job 3: Save state on success
+  # ---------------------------------------------------------------------------
+  save-state:
+    needs: [resolve-refs, bench-scheduled]
+    if: success()
+    name: save-state
+    runs-on: ubuntu-latest
+    steps:
+      - name: Write state file
+        run: |
+          mkdir -p .nightly-state
+          echo "${{ needs.resolve-refs.outputs.feature-ref }}" > .nightly-state/last-feature-ref
+
+      - name: Save nightly state
+        uses: actions/cache/save@v4
+        with:
+          path: .nightly-state
+          key: bench-scheduled-state-${{ needs.resolve-refs.outputs.feature-ref }}

.github/workflows/book.yml

@@ -15,7 +15,7 @@ env:
 
 jobs:
   build:
-    runs-on: depot-ubuntu-latest-8
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
     timeout-minutes: 90
     steps:
       - name: Checkout

.github/workflows/check-alloy.yml

@@ -25,7 +25,7 @@ env:
 jobs:
   check:
     name: Check compilation with patched alloy
-    runs-on: depot-ubuntu-latest-16
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-16' || 'ubuntu-latest' }}
     timeout-minutes: 60
     steps:
       - uses: actions/checkout@v4

.github/workflows/compact.yml

@@ -18,7 +18,7 @@ env:
 name: compact-codec
 jobs:
   compact-codec:
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     strategy:
       matrix:
         bin:

.github/workflows/docker-test.yml

@@ -6,7 +6,7 @@ on:
       hive_target:
         required: true
         type: string
-        description: "Docker bake target to build (e.g. hive-stable, hive-edge)"
+        description: "Docker bake target to build (e.g. hive)"
       artifact_name:
         required: false
         type: string
@@ -15,7 +15,6 @@ on:
 
 jobs:
   build:
-    if: github.repository == 'paradigmxyz/reth'
     timeout-minutes: 45
     runs-on: ubuntu-latest
     permissions:
@@ -31,10 +30,22 @@ jobs:
           echo "sha=${{ github.sha }}" >> "$GITHUB_OUTPUT"
           echo "describe=$(git describe --always --tags)" >> "$GITHUB_OUTPUT"
 
+      - name: Detect fork
+        id: fork
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.repository }}" ]; then
+            echo "is_fork=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "is_fork=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      # Depot build (upstream only)
       - name: Set up Depot CLI
+        if: steps.fork.outputs.is_fork == 'false'
         uses: depot/setup-action@v1
 
-      - name: Build reth image
+      - name: Build reth image (Depot)
+        if: steps.fork.outputs.is_fork == 'false'
         uses: depot/bake-action@v1
         env:
           DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
@@ -46,8 +57,26 @@ jobs:
           targets: ${{ inputs.hive_target }}
           push: false
 
+      # Docker build (forks)
+      - name: Set up Docker Buildx
+        if: steps.fork.outputs.is_fork == 'true'
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build reth image (Docker)
+        if: steps.fork.outputs.is_fork == 'true'
+        uses: docker/bake-action@v6
+        env:
+          VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
+          VERGEN_GIT_DESCRIBE: ${{ steps.git.outputs.describe }}
+        with:
+          files: docker-bake.hcl
+          targets: ${{ inputs.hive_target }}
+          push: false
+          set: |
+            *.dockerfile=Dockerfile
+
       - name: Upload reth image
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
         with:
           name: ${{ inputs.artifact_name }}
           path: ./artifacts

.github/workflows/docker.yml

@@ -28,12 +28,30 @@ on:
         required: false
         type: boolean
         default: false
+      pgo:
+        description: "Enable PGO profiling"
+        required: false
+        type: boolean
+        default: false
+      pgo_blocks:
+        description: "Number of blocks to execute for PGO profiling"
+        required: false
+        type: string
+        default: "20"
 
 jobs:
+  collect-pgo-profile:
+    if: github.repository == 'paradigmxyz/reth' && github.event_name == 'workflow_dispatch' && inputs.pgo
+    uses: ./.github/workflows/pgo-profile.yml
+    with:
+      pgo_blocks: ${{ inputs.pgo_blocks || '20' }}
+    secrets: inherit
+
   build:
-    if: github.repository == 'paradigmxyz/reth'
+    if: github.repository == 'paradigmxyz/reth' && !failure() && !cancelled()
     name: Build Docker images
     runs-on: ubuntu-24.04
+    needs: collect-pgo-profile
     permissions:
       packages: write
       contents: read
@@ -45,7 +63,7 @@ jobs:
         uses: depot/setup-action@v1
 
       - name: Log in to GHCR
-        uses: docker/login-action@v3
+        uses: docker/login-action@v4
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -58,6 +76,30 @@ jobs:
           echo "describe=$(git describe --always --tags)" >> "$GITHUB_OUTPUT"
           echo "dirty=false" >> "$GITHUB_OUTPUT"
 
+      - name: Download pre-collected PGO profile
+        if: ${{ github.event_name == 'workflow_dispatch' && inputs.pgo }}
+        uses: actions/download-artifact@v7
+        with:
+          name: pgo-profdata
+          path: dist
+
+      - name: Configure PGO build args
+        id: pgo
+        run: |
+          if [[ "${{ github.event_name }}" == "workflow_dispatch" ]] && [[ "${{ inputs.pgo }}" == "true" ]]; then
+            if [ ! -f dist/merged.profdata ]; then
+              echo "::error::Expected dist/merged.profdata from collect-pgo-profile job"
+              exit 1
+            fi
+            echo "use_pgo_bolt=true" >> "$GITHUB_OUTPUT"
+            echo "pgo_profdata=dist/merged.profdata" >> "$GITHUB_OUTPUT"
+            echo "Using pre-collected PGO profile from collect-pgo-profile job"
+          else
+            echo "use_pgo_bolt=false" >> "$GITHUB_OUTPUT"
+            echo "pgo_profdata=" >> "$GITHUB_OUTPUT"
+            echo "PGO disabled"
+          fi
+
       - name: Determine build parameters
         id: params
         run: |
@@ -107,6 +149,9 @@ jobs:
           push: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }}
           set: |
             ${{ steps.params.outputs.ethereum_set }}
+            *.args.USE_PGO_BOLT=${{ steps.pgo.outputs.use_pgo_bolt }}
+            *.args.PGO_PROFDATA=${{ steps.pgo.outputs.pgo_profdata }}
+            *.args.STRIP_SYMBOLS=false
 
       - name: Verify image architectures
         env:

.github/workflows/e2e.yml

@@ -20,7 +20,7 @@ concurrency:
 jobs:
   test:
     name: e2e-testsuite
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     env:
       RUST_BACKTRACE: 1
     timeout-minutes: 90
@@ -47,7 +47,7 @@ jobs:
 
   rocksdb:
     name: e2e-rocksdb
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     env:
       RUST_BACKTRACE: 1
     timeout-minutes: 60
@@ -63,6 +63,6 @@ jobs:
         run: |
           cargo nextest run \
             --no-fail-fast \
-            --locked --features "edge" \
+            --locked \
             -p reth-e2e-test-utils \
             -E 'binary(rocksdb)'

.github/workflows/hive.yml

@@ -15,24 +15,17 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  build-reth-stable:
+  build-reth:
     uses: ./.github/workflows/docker-test.yml
     with:
-      hive_target: hive-stable
-      artifact_name: "reth-stable"
-    secrets: inherit
-
-  build-reth-edge:
-    uses: ./.github/workflows/docker-test.yml
-    with:
-      hive_target: hive-edge
-      artifact_name: "reth-edge"
+      hive_target: hive
+      artifact_name: "reth"
     secrets: inherit
 
   prepare-hive:
     if: github.repository == 'paradigmxyz/reth'
     timeout-minutes: 45
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     steps:
       - uses: actions/checkout@v6
       - name: Checkout hive tests
@@ -75,7 +68,7 @@ jobs:
           chmod +x hive
 
       - name: Upload hive assets
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
         with:
           name: hive_assets
           path: ./hive_assets
@@ -84,7 +77,6 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        storage: [stable, edge]
         # ethereum/rpc to be deprecated:
         # https://github.com/ethereum/hive/pull/1117
         scenario:
@@ -184,11 +176,11 @@ jobs:
           - sim: ethereum/eels/consume-rlp
             limit: .*tests/paris.*
     needs:
-      - build-reth-stable
-      - build-reth-edge
+      - build-reth
       - prepare-hive
-    name: ${{ matrix.storage }} / ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
-    runs-on: depot-ubuntu-latest-4
+    name: ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
+    # Use larger runners for eels tests to avoid OOM runner crashes
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
     permissions:
       issues: write
     steps:
@@ -197,15 +189,15 @@ jobs:
           fetch-depth: 0
 
       - name: Download hive assets
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@v8
         with:
           name: hive_assets
           path: /tmp
 
       - name: Download reth image
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@v8
         with:
-          name: reth-${{ matrix.storage }}
+          name: reth
           path: /tmp
 
       - name: Load Docker images

.github/workflows/integration.yml

@@ -22,15 +22,14 @@ concurrency:
 
 jobs:
   test:
-    name: test / ${{ matrix.network }} / ${{ matrix.storage }}
+    name: test / ${{ matrix.network }}
     if: github.event_name != 'schedule'
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     env:
       RUST_BACKTRACE: 1
     strategy:
       matrix:
         network: ["ethereum"]
-        storage: ["stable", "edge"]
     timeout-minutes: 60
     steps:
       - uses: actions/checkout@v6
@@ -47,7 +46,7 @@ jobs:
         run: |
           cargo nextest run \
             --no-fail-fast \
-            --locked --features "asm-keccak ${{ matrix.network }} ${{ matrix.storage == 'edge' && 'edge' || '' }}" \
+            --locked --features "asm-keccak ${{ matrix.network }}" \
             --workspace --exclude ef-tests \
             -E "kind(test) and not binary(e2e_testsuite)"
 

.github/workflows/kurtosis.yml

@@ -31,7 +31,7 @@ jobs:
     strategy:
       fail-fast: false
     name: run kurtosis
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     needs:
       - build-reth
     steps:
@@ -40,7 +40,7 @@ jobs:
           fetch-depth: 0
 
       - name: Download reth image
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@v8
         with:
           name: artifacts
           path: /tmp

.github/workflows/lint.yml

@@ -13,7 +13,7 @@ env:
 jobs:
   clippy-binaries:
     name: clippy binaries / ${{ matrix.type }}
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     strategy:
       matrix:
@@ -42,7 +42,7 @@ jobs:
 
   clippy:
     name: clippy
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -59,7 +59,7 @@ jobs:
           RUSTFLAGS: -D warnings
 
   wasm:
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -79,7 +79,7 @@ jobs:
           .github/scripts/check_wasm.sh
 
   riscv:
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 60
     steps:
       - uses: actions/checkout@v6
@@ -98,7 +98,7 @@ jobs:
 
   crate-checks:
     name: crate-checks (${{ matrix.partition }}/${{ matrix.total_partitions }})
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     strategy:
       matrix:
         partition: [1, 2, 3]
@@ -117,14 +117,14 @@ jobs:
 
   msrv:
     name: MSRV
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
       - uses: rui314/setup-mold@v1
       - uses: dtolnay/rust-toolchain@master
         with:
-          toolchain: "1.88" # MSRV
+          toolchain: "1.93" # MSRV
       - uses: mozilla-actions/sccache-action@v0.0.9
       - uses: Swatinem/rust-cache@v2
         with:
@@ -135,7 +135,7 @@ jobs:
 
   docs:
     name: docs
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -153,7 +153,7 @@ jobs:
 
   fmt:
     name: fmt
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -167,7 +167,7 @@ jobs:
 
   udeps:
     name: udeps
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -182,7 +182,7 @@ jobs:
 
   book:
     name: book
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -240,7 +240,7 @@ jobs:
   # Checks that selected crates can compile with power set of features
   features:
     name: features
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v6
@@ -264,7 +264,7 @@ jobs:
 
   # Check crates correctly propagate features
   feature-propagation:
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     timeout-minutes: 20
     steps:
       - uses: actions/checkout@v6

.github/workflows/pgo-profile.yml

@@ -0,0 +1,107 @@
+name: pgo-profile
+
+on:
+  workflow_call:
+    inputs:
+      pgo_blocks:
+        description: "Number of blocks to execute for PGO profiling"
+        required: false
+        type: string
+        default: "20"
+  workflow_dispatch:
+    inputs:
+      pgo_blocks:
+        description: "Number of blocks to execute for PGO profiling"
+        required: false
+        type: string
+        default: "20"
+
+jobs:
+  collect:
+    name: collect PGO profiles
+    runs-on: [self-hosted, Linux, X64]
+    timeout-minutes: 180
+    env:
+      SCHELK_MOUNT: /reth-bench
+      BENCH_RPC_URL: https://ethereum.reth.rs/rpc
+      RUSTC_WRAPPER: "sccache"
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          submodules: true
+      - uses: rui314/setup-mold@v1
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          target: x86_64-unknown-linux-gnu
+      - uses: mozilla-actions/sccache-action@v0.0.9
+        continue-on-error: true
+      - uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y --no-install-recommends \
+            dmsetup lsb-release wget linux-tools-"$(uname -r)" || \
+            sudo apt-get install -y --no-install-recommends linux-tools-generic
+
+      - name: Download snapshot if needed
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BENCH_REPO: ${{ github.repository }}
+        run: |
+          if ! .github/scripts/bench-reth-snapshot.sh --check; then
+            echo "Snapshot outdated or missing, downloading..."
+            .github/scripts/bench-reth-snapshot.sh
+          fi
+
+      - name: Mount snapshot
+        run: |
+          sudo pkill -9 reth || true
+          sleep 1
+          if mountpoint -q "$SCHELK_MOUNT"; then
+            sudo umount -l "$SCHELK_MOUNT" || true
+            sudo schelk recover -y || true
+          fi
+          sudo schelk mount -y
+          sync
+          sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
+
+      - name: Collect PGO profile
+        run: |
+          DATADIR="$SCHELK_MOUNT/datadir" \
+          RPC_URL="$BENCH_RPC_URL" \
+          PGO_BLOCKS="${{ inputs.pgo_blocks || '20' }}" \
+          BOLT_BLOCKS="${{ inputs.pgo_blocks || '20' }}" \
+          COLLECT_PGO_ONLY=true \
+          SKIP_BOLT=true \
+          PROFILE=maxperf-symbols \
+          FEATURES="jemalloc,asm-keccak,min-debug-logs" \
+          TARGET=x86_64-unknown-linux-gnu \
+          EXTRA_RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq" \
+            .github/scripts/build_pgo_bolt.sh
+
+      - name: Show PGO profile stats
+        run: |
+          LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
+          if [ -z "$LLVM_PROFDATA" ]; then
+            echo "::error::llvm-profdata not found in rust toolchain"
+            exit 1
+          fi
+          "$LLVM_PROFDATA" show --detailed-summary --topn=20 target/pgo-profiles/merged.profdata
+
+      - name: Upload PGO profile
+        uses: actions/upload-artifact@v7
+        with:
+          name: pgo-profdata
+          path: target/pgo-profiles/merged.profdata
+          retention-days: 1
+
+      - name: Recover snapshot
+        if: always()
+        run: |
+          if mountpoint -q "$SCHELK_MOUNT"; then
+            sudo umount -l "$SCHELK_MOUNT" || true
+            sudo schelk recover -y || true
+          fi

.github/workflows/pr-audit.yml

@@ -0,0 +1,30 @@
+name: Pull request audit
+
+on:
+  pull_request:
+    types: [labeled]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    if: github.event.label.name == 'cyclops'
+    steps:
+      - name: Publish event
+        run: |
+          set -euo pipefail
+
+          echo "${{ secrets.EVENTS_KEY }}" > ${{ runner.temp }}/key
+          echo "${{ secrets.EVENTS_CERT }}" > ${{ runner.temp }}/cert
+
+          curl -sf -o /dev/null -X POST ${{ secrets.EVENTS_ARGS }} \
+            -H "Content-Type: application/json" \
+            --key ${{ runner.temp }}/key \
+            --cert ${{ runner.temp }}/cert \
+            -d '{
+              "repository": "${{ github.repository }}",
+              "event": "pr_audit",
+              "data": {
+                "pr_number": ${{ github.event.pull_request.number }},
+                "sha": "${{ github.event.pull_request.head.sha }}"
+              }
+            }'

.github/workflows/release-reproducible.yml

@@ -52,7 +52,7 @@ jobs:
         uses: docker/setup-buildx-action@v3
 
       - name: Log in to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@v4
         with:
           registry: ghcr.io
           username: ${{ github.actor }}

.github/workflows/release.yml

@@ -13,6 +13,14 @@ on:
         description: "Enable dry run mode (builds artifacts but skips uploads and release creation)"
         type: boolean
         default: false
+      pgo:
+        description: "Enable PGO profiling"
+        type: boolean
+        default: false
+      pgo_blocks:
+        description: "Number of blocks to execute for PGO profiling on self-hosted runner"
+        type: string
+        default: "20"
 
 env:
   REPO_NAME: ${{ github.repository_owner }}/reth
@@ -37,7 +45,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Extract version
-        run: echo "VERSION=${GITHUB_REF_NAME}" >> $GITHUB_OUTPUT
+        run: echo "VERSION=${GITHUB_REF_NAME//\//-}" >> $GITHUB_OUTPUT
         id: extract_version
     outputs:
       VERSION: ${{ steps.extract_version.outputs.VERSION }}
@@ -69,26 +77,22 @@ jobs:
       fail-fast: true
       matrix:
         configs:
-          - target: x86_64-unknown-linux-gnu
-            os: ubuntu-24.04
-            profile: maxperf
-            allow_fail: false
           - target: aarch64-unknown-linux-gnu
-            os: ubuntu-24.04
+            os: ubuntu-24.04-arm
             profile: maxperf
             allow_fail: false
+            rustflags: ""
+            native: true
           - target: x86_64-apple-darwin
             os: macos-14
             profile: maxperf
             allow_fail: false
+            rustflags: "-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"
           - target: aarch64-apple-darwin
             os: macos-14
             profile: maxperf
             allow_fail: false
-          - target: riscv64gc-unknown-linux-gnu
-            os: ubuntu-24.04
-            profile: maxperf
-            allow_fail: true
+            rustflags: ""
         build:
           - command: build
             binary: reth
@@ -100,9 +104,10 @@ jobs:
           target: ${{ matrix.configs.target }}
       - uses: mozilla-actions/sccache-action@v0.0.9
       - name: Install cross main
+        if: ${{ !matrix.configs.native }}
         id: cross_main
         run: |
-          cargo install cross --git https://github.com/cross-rs/cross
+          cargo install cross --locked --git https://github.com/cross-rs/cross
       - uses: Swatinem/rust-cache@v2
         with:
           cache-on-failure: true
@@ -114,7 +119,12 @@ jobs:
           echo "MACOSX_DEPLOYMENT_TARGET=$(xcrun -sdk macosx --show-sdk-platform-version)" >> $GITHUB_ENV
 
       - name: Build Reth
-        run: make PROFILE=${{ matrix.configs.profile }} ${{ matrix.build.command }}-${{ matrix.configs.target }}
+        run: |
+          if [ "${{ matrix.configs.native }}" = "true" ]; then
+            make PROFILE=${{ matrix.configs.profile }} EXTRA_RUSTFLAGS="${{ matrix.configs.rustflags }}" ${{ matrix.build.command }}-native-${{ matrix.configs.target }}
+          else
+            make PROFILE=${{ matrix.configs.profile }} EXTRA_RUSTFLAGS="${{ matrix.configs.rustflags }}" ${{ matrix.build.command }}-${{ matrix.configs.target }}
+          fi
       - name: Move binary
         run: |
           mkdir artifacts
@@ -135,23 +145,105 @@ jobs:
 
       - name: Upload artifact
         if: ${{ github.event.inputs.dry_run != 'true' }}
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
         with:
           name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
           path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
 
       - name: Upload signature
         if: ${{ github.event.inputs.dry_run != 'true' }}
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
         with:
           name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
           path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
 
+  collect-pgo-profile:
+    if: github.event_name == 'workflow_dispatch' && inputs.pgo
+    uses: ./.github/workflows/pgo-profile.yml
+    with:
+      pgo_blocks: ${{ inputs.pgo_blocks || '20' }}
+    secrets: inherit
+
+  build-pgo:
+    if: github.event_name == 'workflow_dispatch' && inputs.pgo
+    name: build release (x86_64-linux PGO+BOLT)
+    runs-on: [self-hosted, Linux, X64]
+    needs: [extract-version, collect-pgo-profile]
+    timeout-minutes: 120
+    env:
+      RUSTC_WRAPPER: "sccache"
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          submodules: true
+      - uses: rui314/setup-mold@v1
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          target: x86_64-unknown-linux-gnu
+      - uses: mozilla-actions/sccache-action@v0.0.9
+        continue-on-error: true
+      - uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+
+      - name: Download pre-collected PGO profile
+        uses: actions/download-artifact@v7
+        with:
+          name: pgo-profdata
+          path: dist
+
+      - name: Verify PGO profile artifact
+        run: |
+          test -f dist/merged.profdata
+          ls -lh dist/merged.profdata
+
+      - name: Build Reth with PGO+BOLT
+        run: |
+          SKIP_BOLT=true \
+          PGO_PROFDATA="$PWD/dist/merged.profdata" \
+          PROFILE=maxperf-symbols \
+          FEATURES="jemalloc,asm-keccak,min-debug-logs" \
+          TARGET=x86_64-unknown-linux-gnu \
+          EXTRA_RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq" \
+            .github/scripts/build_pgo_bolt.sh
+
+      - name: Move binary
+        run: |
+          mkdir artifacts
+          mv target/maxperf-symbols/reth ./artifacts
+
+      - name: Configure GPG and create artifacts
+        env:
+          GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }}
+          GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
+        run: |
+          export GPG_TTY=$(tty)
+          echo -n "$GPG_SIGNING_KEY" | base64 --decode | gpg --batch --import
+          cd artifacts
+          tar -czf reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz reth*
+          echo "$GPG_PASSPHRASE" | gpg --passphrase-fd 0 --pinentry-mode loopback --batch -ab reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
+          mv *tar.gz* ..
+        shell: bash
+
+      - name: Upload artifact
+        if: ${{ github.event.inputs.dry_run != 'true' }}
+        uses: actions/upload-artifact@v6
+        with:
+          name: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
+          path: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
+
+      - name: Upload signature
+        if: ${{ github.event.inputs.dry_run != 'true' }}
+        uses: actions/upload-artifact@v6
+        with:
+          name: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz.asc
+          path: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz.asc
+
   draft-release:
     name: draft release
     runs-on: ubuntu-latest
-    needs: [build, extract-version]
-    if: ${{ github.event.inputs.dry_run != 'true' }}
+    needs: [build, build-pgo, extract-version]
+    if: ${{ !failure() && !cancelled() && github.event.inputs.dry_run != 'true' }}
     env:
       VERSION: ${{ needs.extract-version.outputs.VERSION }}
     permissions:
@@ -164,7 +256,7 @@ jobs:
         with:
           fetch-depth: 0
       - name: Download artifacts
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@v8
       - name: Generate full changelog
         id: changelog
         run: |

.github/workflows/reproducible-build.yml

@@ -43,7 +43,7 @@ jobs:
           echo "Binaries SHA256 on ${{ matrix.machine }}: $(cat checksum.sha256)"
 
       - name: Upload the hash
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@v7
         with:
           name: checksum-${{ matrix.machine }}
           path: |
@@ -56,12 +56,12 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Download artifacts from machine-1
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@v8
         with:
           name: checksum-machine-1
           path: machine-1/
       - name: Download artifacts from machine-2
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@v8
         with:
           name: checksum-machine-2
           path: machine-2/

.github/workflows/stage.yml

@@ -23,7 +23,7 @@ jobs:
     name: stage-run-test
     # Only run stage commands test in merge groups
     if: github.event_name == 'merge_group'
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     env:
       RUST_LOG: info,sync=error
       RUST_BACKTRACE: 1
@@ -38,7 +38,7 @@ jobs:
           cache-on-failure: true
       - name: Build reth
         run: |
-          cargo install --path bin/reth
+          cargo install --locked --path bin/reth
       - name: Run headers stage
         run: |
           reth stage run headers --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
@@ -51,15 +51,12 @@ jobs:
       - name: Run execution stage
         run: |
           reth stage run execution --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
-      - name: Run account-hashing stage
-        run: |
-          reth stage run account-hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
-      - name: Run storage hashing stage
-        run: |
-          reth stage run storage-hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
-      - name: Run hashing stage
-        run: |
-          reth stage run hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
+      # NOTE: account-hashing, storage-hashing, and hashing stages are omitted.
+      # With storage v2 (now default), these stages are no-ops because the
+      # execution stage writes directly to HashedAccounts/HashedStorages.
+      # Running them here is harmful: `stage run` unwinds before executing,
+      # and the unwind reverts the hashed state that execution wrote, but
+      # the no-op execute never restores it — causing merkle to fail.
       - name: Run merkle stage
         run: |
           reth stage run merkle --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints

.github/workflows/sync-era.yml

@@ -19,7 +19,7 @@ jobs:
   sync:
     if: github.repository == 'paradigmxyz/reth'
     name: sync (${{ matrix.chain.bin }})
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     env:
       RUST_LOG: info,sync=error
       RUST_BACKTRACE: 1

.github/workflows/sync.yml

@@ -19,7 +19,7 @@ jobs:
   sync:
     if: github.repository == 'paradigmxyz/reth'
     name: sync (${{ matrix.chain.bin }})
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     env:
       RUST_LOG: info,sync=error
       RUST_BACKTRACE: 1

.github/workflows/unit.yml

@@ -19,15 +19,13 @@ concurrency:
 
 jobs:
   test:
-    name: test / ${{ matrix.type }} / ${{ matrix.storage }}
-    runs-on: depot-ubuntu-latest-4
+    name: test / ${{ matrix.type }}
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
     env:
       RUST_BACKTRACE: 1
-      EDGE_FEATURES: ${{ matrix.storage == 'edge' && 'edge' || '' }}
     strategy:
       matrix:
         type: [ethereum]
-        storage: [stable, edge]
         include:
           - type: ethereum
             features: asm-keccak ethereum
@@ -50,14 +48,14 @@ jobs:
         run: |
           cargo nextest run \
             --no-fail-fast \
-            --features "${{ matrix.features }} $EDGE_FEATURES" --locked \
+            --features "${{ matrix.features }}" --locked \
             ${{ matrix.exclude_args }} --workspace \
             --exclude ef-tests --no-tests=warn \
             -E "!kind(test) and not binary(e2e_testsuite)"
 
   state:
     name: Ethereum state tests
-    runs-on: depot-ubuntu-latest-4
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
     env:
       RUST_LOG: info,sync=error
       RUST_BACKTRACE: 1
@@ -92,7 +90,7 @@ jobs:
 
   doc:
     name: doc tests
-    runs-on: depot-ubuntu-latest
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
     env:
       RUST_BACKTRACE: 1
     timeout-minutes: 30

CLAUDE.md

@@ -172,10 +172,97 @@ Before submitting changes, ensure:
 2. **Clippy**: No warnings
 3. **Tests Pass**: All unit and integration tests
 4. **Documentation**: Update relevant docs and add doc comments with `cargo docs --document-private-items`
-5. **Commit Messages**: Follow conventional format (feat:, fix:, chore:, etc.)
+5. **CLI Docs** (if CLI changed): Run `make update-book-cli` (see below)
+6. **Commit Messages**: Follow conventional format (feat:, fix:, chore:, etc.)
+
+### CLI Reference Docs (`book` CI Job)
+
+The CLI reference pages under `docs/vocs/docs/pages/cli/` are **auto-generated** from the `reth` binary's `--help` output. **Do not edit these files manually** — any hand edits will be overwritten and CI will fail regardless.
+
+When you add, remove, or modify CLI commands, subcommands, or flags, regenerate the CLI docs by running:
+
+```bash
+make update-book-cli
+```
+
+This builds `reth` in debug mode and runs `docs/cli/update.sh` to regenerate all CLI pages. Commit the resulting changes.
+
+The `book` CI job (`.github/workflows/lint.yml`) enforces this by regenerating the docs and running `git diff --exit-code`. If the committed docs don't match the generated output, CI fails. Manually editing these pages is never productive — always use `make update-book-cli`.
 
 ### Opening PRs against <https://github.com/paradigmxyz/reth>
 
+#### Titles
+
+Use [Conventional Commits](https://www.conventionalcommits.org/) with an optional scope:
+
+```
+<type>(<scope>): <short description>
+```
+
+**Types**: `feat`, `fix`, `perf`, `refactor`, `docs`, `test`, `chore`
+
+**Scope** (optional): crate or area, e.g. `evm`, `trie`, `rpc`, `engine`, `net`
+
+Examples:
+- `fix(rpc): correct gas estimation for ERC-20 transfers`
+- `perf: batch trie updates to reduce cursor overhead`
+- `feat(engine): add new_payload_interval metric`
+
+#### Descriptions
+
+Keep it short. Say what changed and why — nothing more.
+
+**Do:**
+- Write 1–3 sentences summarizing the change
+- Explain _why_ if the diff doesn't make it obvious
+- Link related issues or EIPs
+- Include benchmark numbers for perf changes
+
+**Don't:**
+- List every file changed — that's what the diff is for
+- Repeat the title in the body
+- Add "Files changed" or "Changes" sections
+- Write walls of text that go stale when the diff is updated
+- Use filler like "This PR introduces...", "comprehensive", "robust", "enhance", "leverage"
+
+**Template:**
+
+```
+Closes #<issue>
+
+<what changed, 1-3 sentences>
+
+<why, if not obvious from the diff>
+```
+
+**Good example:**
+
+```
+Closes #16800
+
+Adds fallback for external IP resolution so node startup doesn't fail
+when STUN is unreachable. Falls back to the configured default.
+```
+
+**Bad example:**
+
+```
+## Summary
+This PR introduces comprehensive improvements to the IP resolution system.
+
+## Changes
+- Modified `crates/net/discv4/src/lib.rs` to add fallback
+- Modified `crates/net/discv4/src/config.rs` to add default IP
+- Added tests in `crates/net/discv4/src/tests/ip.rs`
+
+## Files Changed
+- crates/net/discv4/src/lib.rs
+- crates/net/discv4/src/config.rs
+- crates/net/discv4/src/tests/ip.rs
+```
+
+#### Labels and CI
+
 Label PRs appropriately, first check the available labels and then apply the relevant ones:
 * when changes are RPC related, add A-rpc label
 * when changes are docs related, add C-docs label
@@ -313,6 +400,74 @@ GLOBAL_COUNTER.fetch_add(1, Ordering::SeqCst);
 Before adding a comment, ask: Would someone reading just the current code (no PR, no history) find this helpful?
 
 
+#### Rust Style Guides
+
+##### Type Ordering in Files
+
+When defining structs, traits, and functions in a file, follow this ordering convention. The file's primary type (matching the file name) comes first, followed by supporting public types, then private types and helpers.
+
+```rust
+use ...;
+
+/// The primary type of this file (matches filename).
+pub struct PayloadProcessor { ... }
+
+impl PayloadProcessor { ... }
+
+// Followed by public auxiliary types that support the primary type
+
+/// Configuration for the processor.
+pub struct PayloadProcessorConfig { ... }
+
+/// Result type returned by processor operations.
+pub struct ProcessorResult { ... }
+
+// Followed by public traits related to the primary type
+
+pub trait ProcessorExt { ... }
+
+// Followed by private helper types
+
+struct InternalState { ... }
+
+// Followed by private helper functions
+
+fn validate_input() { ... }
+```
+
+❌ **Bad**: Adding new traits and auxiliary types **above** the file's primary type (see [#22133](https://github.com/paradigmxyz/reth/pull/22133)):
+
+```rust
+use ...;
+
+// ❌ BAD - new auxiliary struct added before the file's main type
+pub struct CacheWaitDurations { ... }
+
+// ❌ BAD - new trait added before the file's main type  
+pub trait WaitForCaches { ... }
+
+// The file's primary type is buried below unrelated additions
+pub struct PayloadProcessor { ... }
+```
+
+✅ **Good**: New types go **after** the primary type:
+
+```rust
+use ...;
+
+// ✅ The file's primary type stays at the top
+pub struct PayloadProcessor { ... }
+
+impl PayloadProcessor { ... }
+
+// ✅ Auxiliary types follow the primary type
+pub struct CacheWaitDurations { ... }
+
+pub trait WaitForCaches { ... }
+
+impl WaitForCaches for PayloadProcessor { ... }
+```
+
 ### Example Contribution Workflow
 
 Let's say you want to fix a bug where external IP resolution fails on startup:
@@ -387,5 +542,8 @@ cargo build --release
 cargo check --workspace --all-features
 
 # Check documentation
-cargo docs --document-private-items 
+cargo docs --document-private-items
+
+# Regenerate CLI reference docs (after CLI changes)
+make update-book-cli
 ```

Cargo.toml

@@ -1,7 +1,7 @@
 [workspace.package]
 version = "1.11.3"
 edition = "2024"
-rust-version = "1.88"
+rust-version = "1.93"
 license = "MIT OR Apache-2.0"
 homepage = "https://paradigmxyz.github.io/reth"
 repository = "https://github.com/paradigmxyz/reth"
@@ -27,7 +27,6 @@ members = [
     "crates/engine/invalid-block-hooks/",
     "crates/engine/local",
     "crates/engine/primitives/",
-    "crates/engine/service",
     "crates/engine/tree/",
     "crates/engine/util/",
     "crates/era",
@@ -139,6 +138,7 @@ members = [
     "examples/exex-subscription",
     "examples/exex-test",
     "examples/full-contract-state",
+    "examples/migrate-trie-to-packed",
     "examples/manual-p2p/",
     "examples/network-txpool/",
     "examples/network/",
@@ -170,6 +170,7 @@ rust.rust_2018_idioms = { level = "deny", priority = -1 }
 rust.unreachable_pub = "warn"
 rust.unused_must_use = "deny"
 rust.rust_2024_incompatible_pat = "warn"
+rust.unexpected_cfgs = { level = "warn", check-cfg = ['cfg(tokio_unstable)'] }
 rustdoc.all = "warn"
 # rust.unnameable-types = "warn"
 
@@ -349,7 +350,6 @@ reth-ecies = { path = "crates/net/ecies" }
 reth-engine-local = { path = "crates/engine/local" }
 reth-engine-primitives = { path = "crates/engine/primitives", default-features = false }
 reth-engine-tree = { path = "crates/engine/tree" }
-reth-engine-service = { path = "crates/engine/service" }
 reth-engine-util = { path = "crates/engine/util" }
 reth-era = { path = "crates/era" }
 reth-era-downloader = { path = "crates/era-downloader" }
@@ -399,7 +399,9 @@ reth-payload-builder-primitives = { path = "crates/payload/builder-primitives" }
 reth-payload-primitives = { path = "crates/payload/primitives" }
 reth-payload-validator = { path = "crates/payload/validator" }
 reth-payload-util = { path = "crates/payload/util" }
-reth-primitives = { path = "crates/primitives", default-features = false }
+reth-primitives = { path = "crates/primitives", default-features = false, features = [
+    "__internal",
+] }
 reth-primitives-traits = { path = "crates/primitives-traits", default-features = false }
 reth-provider = { path = "crates/storage/provider" }
 reth-prune = { path = "crates/prune/prune" }
@@ -437,15 +439,14 @@ reth-trie-sparse = { path = "crates/trie/sparse", default-features = false }
 reth-zstd-compressors = { path = "crates/storage/zstd-compressors", default-features = false }
 
 # revm
-revm = { version = "34.0.0", default-features = false }
-revm-bytecode = { version = "8.0.0", default-features = false }
-revm-database = { version = "10.0.0", default-features = false }
-revm-state = { version = "9.0.0", default-features = false }
-revm-primitives = { version = "22.0.0", default-features = false }
-revm-interpreter = { version = "32.0.0", default-features = false }
-revm-database-interface = { version = "9.0.0", default-features = false }
-op-revm = { version = "15.0.0", default-features = false }
-revm-inspectors = "0.34.2"
+revm = { version = "36.0.0", default-features = false }
+revm-bytecode = { version = "9.0.0", default-features = false }
+revm-database = { version = "12.0.0", default-features = false }
+revm-state = { version = "10.0.0", default-features = false }
+revm-primitives = { version = "22.1.0", default-features = false }
+revm-interpreter = { version = "34.0.0", default-features = false }
+revm-database-interface = { version = "10.0.0", default-features = false }
+revm-inspectors = "0.36.1"
 
 # eth
 alloy-dyn-abi = "1.5.6"
@@ -457,7 +458,7 @@ alloy-sol-types = { version = "1.5.6", default-features = false }
 alloy-chains = { version = "0.2.5", default-features = false }
 alloy-eip2124 = { version = "0.2.0", default-features = false }
 alloy-eip7928 = { version = "0.3.0", default-features = false }
-alloy-evm = { version = "0.27.2", default-features = false }
+alloy-evm = { version = "0.29.2", default-features = false }
 alloy-rlp = { version = "0.3.13", default-features = false, features = [
     "core-net",
 ] }
@@ -465,50 +466,45 @@ alloy-trie = { version = "0.9.4", default-features = false }
 
 alloy-hardforks = "0.4.5"
 
-alloy-consensus = { version = "1.6.3", default-features = false }
-alloy-contract = { version = "1.6.3", default-features = false }
-alloy-eips = { version = "1.6.3", default-features = false }
-alloy-genesis = { version = "1.6.3", default-features = false }
-alloy-json-rpc = { version = "1.6.3", default-features = false }
-alloy-network = { version = "1.6.3", default-features = false }
-alloy-network-primitives = { version = "1.6.3", default-features = false }
-alloy-provider = { version = "1.6.3", features = [
+alloy-consensus = { version = "1.7.3", default-features = false }
+alloy-contract = { version = "1.7.3", default-features = false }
+alloy-eips = { version = "1.7.3", default-features = false }
+alloy-genesis = { version = "1.7.3", default-features = false }
+alloy-json-rpc = { version = "1.7.3", default-features = false }
+alloy-network = { version = "1.7.3", default-features = false }
+alloy-network-primitives = { version = "1.7.3", default-features = false }
+alloy-provider = { version = "1.7.3", features = [
     "reqwest",
     "debug-api",
 ], default-features = false }
-alloy-pubsub = { version = "1.6.3", default-features = false }
-alloy-rpc-client = { version = "1.6.3", default-features = false }
-alloy-rpc-types = { version = "1.6.3", features = [
+alloy-pubsub = { version = "1.7.3", default-features = false }
+alloy-rpc-client = { version = "1.7.3", default-features = false }
+alloy-rpc-types = { version = "1.7.3", features = [
     "eth",
 ], default-features = false }
-alloy-rpc-types-admin = { version = "1.6.3", default-features = false }
-alloy-rpc-types-anvil = { version = "1.6.3", default-features = false }
-alloy-rpc-types-beacon = { version = "1.6.3", default-features = false }
-alloy-rpc-types-debug = { version = "1.6.3", default-features = false }
-alloy-rpc-types-engine = { version = "1.6.3", default-features = false }
-alloy-rpc-types-eth = { version = "1.6.3", default-features = false }
-alloy-rpc-types-mev = { version = "1.6.3", default-features = false }
-alloy-rpc-types-trace = { version = "1.6.3", default-features = false }
-alloy-rpc-types-txpool = { version = "1.6.3", default-features = false }
-alloy-serde = { version = "1.6.3", default-features = false }
-alloy-signer = { version = "1.6.3", default-features = false }
-alloy-signer-local = { version = "1.6.3", default-features = false }
-alloy-transport = { version = "1.6.3" }
-alloy-transport-http = { version = "1.6.3", features = [
+alloy-rpc-types-admin = { version = "1.7.3", default-features = false }
+alloy-rpc-types-anvil = { version = "1.7.3", default-features = false }
+alloy-rpc-types-beacon = { version = "1.7.3", default-features = false }
+alloy-rpc-types-debug = { version = "1.7.3", default-features = false }
+alloy-rpc-types-engine = { version = "1.7.3", default-features = false }
+alloy-rpc-types-eth = { version = "1.7.3", default-features = false }
+alloy-rpc-types-mev = { version = "1.7.3", default-features = false }
+alloy-rpc-types-trace = { version = "1.7.3", default-features = false }
+alloy-rpc-types-txpool = { version = "1.7.3", default-features = false }
+alloy-serde = { version = "1.7.3", default-features = false }
+alloy-signer = { version = "1.7.3", default-features = false }
+alloy-signer-local = { version = "1.7.3", default-features = false }
+alloy-transport = { version = "1.7.3" }
+alloy-transport-http = { version = "1.7.3", features = [
     "reqwest-rustls-tls",
 ], default-features = false }
-alloy-transport-ipc = { version = "1.6.3", default-features = false }
-alloy-transport-ws = { version = "1.6.3", default-features = false }
+alloy-transport-ipc = { version = "1.7.3", default-features = false }
+alloy-transport-ws = { version = "1.7.3", default-features = false }
 
 # op
-alloy-op-evm = { version = "0.27.2", default-features = false }
-alloy-op-hardforks = "0.4.4"
-op-alloy-rpc-types = { version = "0.23.1", default-features = false }
-op-alloy-rpc-types-engine = { version = "0.23.1", default-features = false }
-op-alloy-network = { version = "0.23.1", default-features = false }
-op-alloy-consensus = { version = "0.23.1", default-features = false }
-op-alloy-rpc-jsonrpsee = { version = "0.23.1", default-features = false }
-op-alloy-flz = { version = "0.13.1", default-features = false }
+op-alloy-rpc-types = { version = "0.24.0", default-features = false }
+op-alloy-rpc-types-engine = { version = "0.24.0", default-features = false }
+op-alloy-consensus = { version = "0.24.0", default-features = false }
 
 # misc
 either = { version = "1.15.0", default-features = false }
@@ -523,6 +519,7 @@ bincode = "1.3"
 bitflags = "2.4"
 boyer-moore-magiclen = "0.2.16"
 bytes = { version = "1.11.1", default-features = false }
+blake3 = "1.8"
 brotli = "8"
 cfg-if = "1.0"
 clap = "4"
@@ -538,6 +535,7 @@ humantime = "2.1"
 humantime-serde = "1.1"
 itertools = { version = "0.14", default-features = false }
 linked_hash_set = "0.1"
+libc = "0.2"
 lz4 = "1.28.1"
 modular-bitfield = "0.13.1"
 notify = { version = "8.0.0", default-features = false, features = [
@@ -548,17 +546,19 @@ once_cell = { version = "1.19", default-features = false, features = [
     "critical-section",
 ] }
 parking_lot = "0.12"
+quanta = "0.12"
 paste = "1.0"
 rand = "0.9"
 rayon = "1.7"
+thread-priority = "3.0.0"
 rustc-hash = { version = "2.0", default-features = false }
 schnellru = "0.2"
 serde = { version = "1.0", default-features = false }
 serde_json = { version = "1.0", default-features = false, features = ["alloc"] }
 serde_with = { version = "3", default-features = false, features = ["macros"] }
 sha2 = { version = "0.10", default-features = false }
-shellexpand = "3.0.0"
 shlex = "1.3"
+slotmap = "1"
 smallvec = "1"
 strum = { version = "0.27", default-features = false }
 strum_macros = "0.27"
@@ -679,7 +679,7 @@ ethereum_ssz_derive = "0.10.1"
 jemalloc_pprof = { version = "0.8", default-features = false }
 tikv-jemalloc-ctl = "0.6"
 tikv-jemallocator = "0.6"
-tracy-client = "0.18.0"
+tracy-client = { version = "0.18.0", features = ["demangle"] }
 snmalloc-rs = { version = "0.3.7", features = ["build_cc"] }
 
 aes = "0.8.1"
@@ -692,6 +692,8 @@ cipher = "0.4.3"
 comfy-table = "7.0"
 concat-kdf = "0.1.0"
 crossbeam-channel = "0.5.13"
+crossbeam-queue = "0.3"
+crossbeam-utils = "0.8"
 crossterm = "0.29.0"
 csv = "1.3.0"
 ctrlc = "3.4"
@@ -767,10 +769,8 @@ ipnet = "2.11"
 # alloy-transport-ws = { git = "https://github.com/alloy-rs/alloy", rev = "3049f232fbb44d1909883e154eb38ec5962f53a3" }
 
 # op-alloy-consensus = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
-# op-alloy-network = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
 # op-alloy-rpc-types = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
 # op-alloy-rpc-types-engine = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
-# op-alloy-rpc-jsonrpsee = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
 #
 # revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "1207e33" }
 #
@@ -780,10 +780,15 @@ ipnet = "2.11"
 # jsonrpsee-http-client = { git = "https://github.com/paradigmxyz/jsonrpsee", branch = "matt/make-rpc-service-pub" }
 # jsonrpsee-types = { git = "https://github.com/paradigmxyz/jsonrpsee", branch = "matt/make-rpc-service-pub" }
 
-# alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "df124c0" }
-# alloy-op-evm = { git = "https://github.com/alloy-rs/evm", rev = "df124c0" }
+alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "b0eb7e6" }
 
-# revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "3020ea8" }
+revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "24becc3" }
 
-# alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "072c248" }
-# alloy-op-evm = { git = "https://github.com/alloy-rs/evm", rev = "072c248" }
+# revm from rakita/state-gas branch
+revm = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
+revm-bytecode = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
+revm-database = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
+revm-state = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
+revm-primitives = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
+revm-interpreter = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
+revm-database-interface = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }

Cross.toml

@@ -19,10 +19,11 @@ pre-build = [
 image = "ubuntu:24.04"
 pre-build = [
     "apt update",
-    "apt install --yes gcc gcc-riscv64-linux-gnu libclang-dev make",
+    "apt install --yes gcc gcc-riscv64-linux-gnu g++-riscv64-linux-gnu libclang-dev make",
 ]
 env.passthrough = [
     "CARGO_TARGET_RISCV64GC_UNKNOWN_LINUX_GNU_LINKER=riscv64-linux-gnu-gcc",
+    "CXX_riscv64gc_unknown_linux_gnu=riscv64-linux-gnu-g++",
 ]
 
 [build.env]

Dockerfile

@@ -1,6 +1,6 @@
 # syntax=docker.io/docker/dockerfile:1.7-labs
 
-FROM lukemathwalker/cargo-chef:latest-rust-1 AS chef
+FROM lukemathwalker/cargo-chef:latest-rust-1.93 AS chef
 WORKDIR /app
 
 LABEL org.opencontainers.image.source=https://github.com/paradigmxyz/reth

Dockerfile.depot

@@ -1,10 +1,12 @@
 # syntax=docker/dockerfile:1
 
 # Dockerfile for reth, optimized for Depot builds
+# Supports PGO+BOLT optimization for maximum performance
 # Usage:
 #   reth: --build-arg BINARY=reth
+#   PGO+BOLT: --build-arg USE_PGO_BOLT=true (Linux x86_64/aarch64 only)
 
-FROM rust:1 AS builder
+FROM rust:1.93 AS builder
 WORKDIR /app
 
 LABEL org.opencontainers.image.source=https://github.com/paradigmxyz/reth
@@ -43,6 +45,18 @@ ENV VERGEN_GIT_SHA=$VERGEN_GIT_SHA
 ENV VERGEN_GIT_DESCRIBE=$VERGEN_GIT_DESCRIBE
 ENV VERGEN_GIT_DIRTY=$VERGEN_GIT_DIRTY
 
+# Enable PGO+BOLT optimization (Linux only)
+ARG USE_PGO_BOLT=false
+ENV USE_PGO_BOLT=$USE_PGO_BOLT
+
+# Optional path to a pre-collected merged.profdata file in build context.
+ARG PGO_PROFDATA=""
+ENV PGO_PROFDATA=$PGO_PROFDATA
+
+# Whether to strip debug symbols from PGO-built binaries.
+ARG STRIP_SYMBOLS=true
+ENV STRIP_SYMBOLS=$STRIP_SYMBOLS
+
 # Build application
 # Platform-specific RUSTFLAGS: amd64 uses x86-64-v3 (Haswell+) with pclmulqdq for rocksdb
 ARG TARGETPLATFORM
@@ -53,12 +67,21 @@ RUN --mount=type=secret,id=DEPOT_TOKEN,env=SCCACHE_WEBDAV_TOKEN \
     --mount=type=cache,target=$SCCACHE_DIR,sharing=shared \
     export RUSTC_WRAPPER=sccache SCCACHE_WEBDAV_ENDPOINT=https://cache.depot.dev SCCACHE_DIR=/sccache && \
     sccache --start-server && \
-    if [ -n "$RUSTFLAGS" ]; then \
-        export RUSTFLAGS="$RUSTFLAGS"; \
-    elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
-        export RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"; \
+    if [ "$USE_PGO_BOLT" = "true" ] && [ "$TARGETPLATFORM" = "linux/amd64" ] && [ -n "$PGO_PROFDATA" ] && [ -f "$PGO_PROFDATA" ]; then \
+        apt-get update && apt-get install -y -qq lsb-release wget sudo && \
+        BINARY="$BINARY" PROFILE="$BUILD_PROFILE" FEATURES="$FEATURES" SKIP_BOLT=true STRIP_SYMBOLS="$STRIP_SYMBOLS" PGO_PROFDATA="$PGO_PROFDATA" \
+            ./.github/scripts/build_pgo_bolt.sh; \
+    else \
+        if [ "$USE_PGO_BOLT" = "true" ]; then \
+            echo "PGO requested but pre-collected profile missing at '${PGO_PROFDATA:-<unset>}' - falling back to non-PGO build"; \
+        fi; \
+        if [ -n "$RUSTFLAGS" ]; then \
+            export RUSTFLAGS="$RUSTFLAGS"; \
+        elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
+            export RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"; \
+        fi && \
+        cargo build --profile $BUILD_PROFILE --features "$FEATURES" --locked --bin $BINARY --manifest-path $MANIFEST_PATH/Cargo.toml; \
     fi && \
-    cargo build --profile $BUILD_PROFILE --features "$FEATURES" --locked --bin $BINARY --manifest-path $MANIFEST_PATH/Cargo.toml && \
     sccache --show-stats
 
 # Copy binary to a known location (ARG not resolved in COPY)

Makefile

@@ -17,6 +17,9 @@ FEATURES ?=
 # Cargo profile for builds. Default is for local builds, CI uses an override.
 PROFILE ?= release
 
+# Extra RUSTFLAGS to append to build targets (e.g., "-C target-cpu=x86-64-v3")
+EXTRA_RUSTFLAGS ?=
+
 # Extra flags for Cargo
 CARGO_INSTALL_EXTRA_FLAGS ?=
 
@@ -74,13 +77,13 @@ build-debug: ## Build the reth binary into `target/debug` directory.
 	cargo build --bin reth --features "$(FEATURES)"
 # Builds the reth binary natively.
 build-native-%:
-	cargo build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
+	$(if $(EXTRA_RUSTFLAGS),RUSTFLAGS="$(EXTRA_RUSTFLAGS)") cargo build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
 
 # The following commands use `cross` to build a cross-compile.
 #
 # These commands require that:
 #
-# - `cross` is installed (`cargo install cross`).
+# - `cross` is installed (`cargo install --locked cross`).
 # - Docker is running.
 # - The current user is in the `docker` group.
 #
@@ -92,11 +95,12 @@ build-native-%:
 # on other systems. JEMALLOC_SYS_WITH_LG_PAGE=16 tells jemalloc to use 64-KiB
 # pages. See: https://github.com/paradigmxyz/reth/issues/6742
 build-aarch64-unknown-linux-gnu: export JEMALLOC_SYS_WITH_LG_PAGE=16
+build-native-aarch64-unknown-linux-gnu: export JEMALLOC_SYS_WITH_LG_PAGE=16
 
 # Note: The additional rustc compiler flags are for intrinsics needed by MDBX.
 # See: https://github.com/cross-rs/cross/wiki/FAQ#undefined-reference-with-build-std
 build-%:
-	RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc" \
+	RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc $(EXTRA_RUSTFLAGS)" \
 		cross build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
 
 # Unfortunately we can't easily use cross to build for Darwin because of licensing issues.
@@ -261,7 +265,7 @@ lint-typos: ensure-typos
 
 ensure-typos:
 	@if ! command -v typos &> /dev/null; then \
-		echo "typos not found. Please install it by running the command 'cargo install typos-cli' or refer to the following link for more information: https://github.com/crate-ci/typos"; \
+		echo "typos not found. Please install it by running the command 'cargo install --locked typos-cli' or refer to the following link for more information: https://github.com/crate-ci/typos"; \
 		exit 1; \
     fi
 

README.md

@@ -93,7 +93,7 @@ When updating this, also update:
 - .github/workflows/lint.yml
 -->
 
-The Minimum Supported Rust Version (MSRV) of this project is [1.88.0](https://blog.rust-lang.org/2025/06/26/Rust-1.88.0/).
+The Minimum Supported Rust Version (MSRV) of this project is [1.93.0](https://blog.rust-lang.org/2026/01/22/Rust-1.93.0/).
 
 See the docs for detailed instructions on how to [build from source](https://reth.rs/installation/source/).
 

bin/reth-bench-compare/Cargo.toml

@@ -45,9 +45,6 @@ serde_json.workspace = true
 # Time handling
 chrono = { workspace = true, features = ["serde"] }
 
-# Path manipulation
-shellexpand.workspace = true
-
 # CSV handling
 csv.workspace = true
 

bin/reth-bench-compare/src/cli.rs

@@ -289,11 +289,7 @@ impl Args {
     /// Get the JWT secret path - either provided or derived from datadir
     pub(crate) fn jwt_secret_path(&self) -> PathBuf {
         match &self.jwt_secret {
-            Some(path) => {
-                let jwt_secret_str = path.to_string_lossy();
-                let expanded = shellexpand::tilde(&jwt_secret_str);
-                PathBuf::from(expanded.as_ref())
-            }
+            Some(path) => path.clone(),
             None => {
                 // Use the same logic as reth: <datadir>/<chain>/jwt.hex
                 let chain_path = self.datadir.clone().resolve_datadir(self.chain);
@@ -308,10 +304,9 @@ impl Args {
         chain_path.data_dir().to_path_buf()
     }
 
-    /// Get the expanded output directory path
+    /// Get the output directory path
     pub(crate) fn output_dir_path(&self) -> PathBuf {
-        let expanded = shellexpand::tilde(&self.output_dir);
-        PathBuf::from(expanded.as_ref())
+        PathBuf::from(&self.output_dir)
     }
 
     /// Get the effective warmup blocks value - either specified or defaults to blocks

bin/reth-bench/src/bench/context.rs

@@ -7,7 +7,7 @@ use alloy_primitives::address;
 use alloy_provider::{network::AnyNetwork, Provider, RootProvider};
 use alloy_rpc_client::ClientBuilder;
 use alloy_rpc_types_engine::JwtSecret;
-use alloy_transport::layers::RetryBackoffLayer;
+use alloy_transport::layers::{RateLimitRetryPolicy, RetryBackoffLayer};
 use reqwest::Url;
 use reth_node_core::args::BenchmarkArgs;
 use tracing::info;
@@ -29,6 +29,10 @@ pub(crate) struct BenchContext {
     pub(crate) next_block: u64,
     /// Whether the chain is an OP rollup.
     pub(crate) is_optimism: bool,
+    /// Whether to use `reth_newPayload` endpoint instead of `engine_newPayload*`.
+    pub(crate) use_reth_namespace: bool,
+    /// Whether to fetch and replay RLP-encoded blocks.
+    pub(crate) rlp_blocks: bool,
 }
 
 impl BenchContext {
@@ -49,9 +53,15 @@ impl BenchContext {
             }
         }
 
-        // set up alloy client for blocks
+        // set up alloy client for blocks, retrying on 429/503 (default) and 502
+        let retry_policy =
+            RateLimitRetryPolicy::default().or(|err: &alloy_transport::TransportError| -> bool {
+                err.as_transport_err()
+                    .and_then(|t| t.as_http_error())
+                    .is_some_and(|e| e.status == 502)
+            });
         let client = ClientBuilder::default()
-            .layer(RetryBackoffLayer::new(10, 800, u64::MAX))
+            .layer(RetryBackoffLayer::new_with_policy(10, 800, u64::MAX, retry_policy))
             .http(rpc_url.parse()?);
         let block_provider = RootProvider::<AnyNetwork>::new(client);
 
@@ -84,9 +94,11 @@ impl BenchContext {
 
         // Computes the block range for the benchmark.
         //
-        // - If `--advance` is provided, fetches the latest block and sets:
+        // - If `--advance` is provided, fetches the latest block from the engine and sets:
         //     - `from = head + 1`
         //     - `to = head + advance`
+        // - If only `--to` is provided, fetches the latest block from the engine and sets:
+        //     - `from = head`
         // - Otherwise, uses the values from `--from` and `--to`.
         let (from, to) = if let Some(advance) = bench_args.advance {
             if advance == 0 {
@@ -99,6 +111,14 @@ impl BenchContext {
                 .ok_or_else(|| eyre::eyre!("Failed to fetch latest block for --advance"))?;
             let head_number = head_block.header.number;
             (Some(head_number), Some(head_number + advance))
+        } else if bench_args.from.is_none() && bench_args.to.is_some() {
+            let head_block = auth_provider
+                .get_block_by_number(BlockNumberOrTag::Latest)
+                .await?
+                .ok_or_else(|| eyre::eyre!("Failed to fetch latest block from engine"))?;
+            let head_number = head_block.header.number;
+            info!(target: "reth-bench", "No --from provided, derived from engine head: {}", head_number);
+            (Some(head_number), bench_args.to)
         } else {
             (bench_args.from, bench_args.to)
         };
@@ -110,7 +130,7 @@ impl BenchContext {
             .full()
             .await?
             .ok_or_else(|| eyre::eyre!("Failed to fetch latest block from RPC"))?;
-        let mut benchmark_mode = BenchMode::new(from, to, latest_block.into_inner().number())?;
+        let mut benchmark_mode = BenchMode::new(from, to, latest_block.into_inner().number());
 
         let first_block = match benchmark_mode {
             BenchMode::Continuous(start) => {
@@ -140,6 +160,16 @@ impl BenchContext {
         };
 
         let next_block = first_block.header.number + 1;
-        Ok(Self { auth_provider, block_provider, benchmark_mode, next_block, is_optimism })
+        let rlp_blocks = bench_args.rlp_blocks;
+        let use_reth_namespace = bench_args.reth_new_payload || rlp_blocks;
+        Ok(Self {
+            auth_provider,
+            block_provider,
+            benchmark_mode,
+            next_block,
+            is_optimism,
+            use_reth_namespace,
+            rlp_blocks,
+        })
     }
 }

bin/reth-bench/src/bench/gas_limit_ramp.rs

@@ -6,7 +6,9 @@ use crate::{
         helpers::{build_payload, parse_gas_limit, prepare_payload_request, rpc_block_to_header},
         output::GasRampPayloadFile,
     },
-    valid_payload::{call_forkchoice_updated, call_new_payload, payload_to_new_payload},
+    valid_payload::{
+        call_forkchoice_updated_with_reth, call_new_payload_with_reth, payload_to_new_payload,
+    },
 };
 use alloy_eips::BlockNumberOrTag;
 use alloy_provider::{network::AnyNetwork, Provider, RootProvider};
@@ -19,6 +21,7 @@ use reth_chainspec::ChainSpec;
 use reth_cli_runner::CliContext;
 use reth_ethereum_primitives::TransactionSigned;
 use reth_primitives_traits::constants::{GAS_LIMIT_BOUND_DIVISOR, MAXIMUM_GAS_LIMIT_BLOCK};
+use reth_rpc_api::RethNewPayloadInput;
 use std::{path::PathBuf, time::Instant};
 use tracing::info;
 
@@ -47,6 +50,14 @@ pub struct Command {
     /// Output directory for benchmark results and generated payloads.
     #[arg(long, value_name = "OUTPUT")]
     output: PathBuf,
+
+    /// Use `reth_newPayload` endpoint instead of `engine_newPayload*`.
+    ///
+    /// The `reth_newPayload` endpoint is a reth-specific extension that takes `ExecutionData`
+    /// directly, waits for persistence and cache updates to complete before processing,
+    /// and returns server-side timing breakdowns (latency, persistence wait, cache wait).
+    #[arg(long, default_value = "false", verbatim_doc_comment)]
+    reth_new_payload: bool,
 }
 
 /// Mode for determining when to stop ramping.
@@ -138,6 +149,9 @@ impl Command {
                 );
             }
         }
+        if self.reth_new_payload {
+            info!("Using reth_newPayload and reth_forkchoiceUpdated endpoints");
+        }
 
         let mut blocks_processed = 0u64;
         let total_benchmark_duration = Instant::now();
@@ -163,7 +177,7 @@ impl Command {
             // Regenerate the payload from the modified block, but keep the original sidecar
             // which contains the actual execution requests data (not just the hash)
             let (payload, _) = ExecutionPayload::from_block_unchecked(block_hash, &block);
-            let (version, params) = payload_to_new_payload(
+            let (version, params, execution_data) = payload_to_new_payload(
                 payload,
                 sidecar,
                 false,
@@ -171,23 +185,32 @@ impl Command {
                 Some(new_payload_version),
             )?;
 
+            let (version, params) = if self.reth_new_payload {
+                (None, serde_json::to_value((RethNewPayloadInput::ExecutionData(execution_data),))?)
+            } else {
+                (Some(version), params)
+            };
+
             // Save payload to file with version info for replay
             let payload_path =
                 self.output.join(format!("payload_block_{}.json", block.header.number));
-            let file =
-                GasRampPayloadFile { version: version as u8, block_hash, params: params.clone() };
+            let file = GasRampPayloadFile {
+                version: version.map(|v| v as u8),
+                block_hash,
+                params: params.clone(),
+            };
             let payload_json = serde_json::to_string_pretty(&file)?;
             std::fs::write(&payload_path, &payload_json)?;
             info!(target: "reth-bench", block_number = block.header.number, path = %payload_path.display(), "Saved payload");
 
-            call_new_payload(&provider, version, params).await?;
+            let _ = call_new_payload_with_reth(&provider, version, params).await?;
 
             let forkchoice_state = ForkchoiceState {
                 head_block_hash: block_hash,
                 safe_block_hash: block_hash,
                 finalized_block_hash: block_hash,
             };
-            call_forkchoice_updated(&provider, version, forkchoice_state, None).await?;
+            call_forkchoice_updated_with_reth(&provider, version, forkchoice_state).await?;
 
             parent_header = block.header;
             parent_hash = block_hash;

bin/reth-bench/src/bench/metrics_scraper.rs

@@ -0,0 +1,156 @@
+//! Prometheus metrics scraper for reth-bench.
+//!
+//! Scrapes a node's Prometheus metrics endpoint after each block to record
+//! execution and state root durations with block-level granularity.
+
+use csv::Writer;
+use eyre::Context;
+use reqwest::Client;
+use serde::Serialize;
+use std::{path::Path, time::Duration};
+use tracing::info;
+
+/// Suffix for the metrics CSV output file.
+pub(crate) const METRICS_OUTPUT_SUFFIX: &str = "metrics.csv";
+
+/// A single row of scraped prometheus metrics for one block.
+#[derive(Debug, Clone, Serialize)]
+pub(crate) struct MetricsRow {
+    /// The block number.
+    pub(crate) block_number: u64,
+    /// EVM execution duration in seconds (from `sync_execution_execution_duration` gauge).
+    pub(crate) execution_duration_secs: Option<f64>,
+    /// State root computation duration in seconds (from
+    /// `sync_block_validation_state_root_duration` gauge).
+    pub(crate) state_root_duration_secs: Option<f64>,
+}
+
+/// Scrapes a Prometheus metrics endpoint after each block to collect
+/// execution and state root durations.
+pub(crate) struct MetricsScraper {
+    /// The full URL of the Prometheus metrics endpoint.
+    url: String,
+    /// Reusable HTTP client.
+    client: Client,
+    /// Collected metrics rows, one per block.
+    rows: Vec<MetricsRow>,
+}
+
+impl MetricsScraper {
+    /// Creates a new scraper if a URL is provided.
+    pub(crate) fn maybe_new(url: Option<String>) -> Option<Self> {
+        url.map(|url| {
+            info!(target: "reth-bench", %url, "Prometheus metrics scraping enabled");
+            let client = Client::builder()
+                .timeout(Duration::from_secs(5))
+                .build()
+                .expect("failed to build reqwest client");
+            Self { url, client, rows: Vec::new() }
+        })
+    }
+
+    /// Scrapes the metrics endpoint and records values for the given block.
+    pub(crate) async fn scrape_after_block(&mut self, block_number: u64) -> eyre::Result<()> {
+        let text = self
+            .client
+            .get(&self.url)
+            .send()
+            .await
+            .wrap_err("failed to fetch metrics endpoint")?
+            .error_for_status()
+            .wrap_err("metrics endpoint returned error status")?
+            .text()
+            .await
+            .wrap_err("failed to read metrics response body")?;
+
+        let execution = parse_gauge(&text, "sync_execution_execution_duration");
+        let state_root = parse_gauge(&text, "sync_block_validation_state_root_duration");
+
+        self.rows.push(MetricsRow {
+            block_number,
+            execution_duration_secs: execution,
+            state_root_duration_secs: state_root,
+        });
+        Ok(())
+    }
+
+    /// Writes collected metrics to a CSV file in the output directory.
+    pub(crate) fn write_csv(&self, output_dir: &Path) -> eyre::Result<()> {
+        let path = output_dir.join(METRICS_OUTPUT_SUFFIX);
+        info!(target: "reth-bench", "Writing scraped metrics to file: {:?}", path);
+        let mut writer = Writer::from_path(&path)?;
+        for row in &self.rows {
+            writer.serialize(row)?;
+        }
+        writer.flush()?;
+        Ok(())
+    }
+}
+
+/// Parses a Prometheus gauge value from exposition-format text.
+///
+/// Searches for lines starting with `name` followed by either a space or `{`
+/// (for labeled metrics), then parses the numeric value. Returns the last
+/// matching sample to handle metrics emitted with multiple label sets.
+fn parse_gauge(text: &str, name: &str) -> Option<f64> {
+    let mut result = None;
+    for line in text.lines() {
+        let line = line.trim();
+        if line.is_empty() || line.starts_with('#') {
+            continue;
+        }
+
+        if !line.starts_with(name) {
+            continue;
+        }
+
+        // Ensure we match the full metric name, not a prefix of another metric.
+        let rest = &line[name.len()..];
+        if !rest.starts_with(' ') && !rest.starts_with('{') {
+            continue;
+        }
+
+        // Format: `metric_name{labels} value [timestamp]` or `metric_name value [timestamp]`
+        // Value is always the second whitespace-separated token.
+        let mut parts = line.split_whitespace();
+        if let Some(value_str) = parts.nth(1) &&
+            let Ok(v) = value_str.parse::<f64>()
+        {
+            result = Some(v);
+        }
+    }
+    result
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_parse_gauge_simple() {
+        let text = r#"# HELP sync_execution_execution_duration Duration of execution
+# TYPE sync_execution_execution_duration gauge
+sync_execution_execution_duration 0.123456
+"#;
+        assert_eq!(parse_gauge(text, "sync_execution_execution_duration"), Some(0.123456));
+    }
+
+    #[test]
+    fn test_parse_gauge_missing() {
+        let text = "some_other_metric 1.0\n";
+        assert_eq!(parse_gauge(text, "sync_execution_execution_duration"), None);
+    }
+
+    #[test]
+    fn test_parse_gauge_with_labels() {
+        let text = "sync_block_validation_state_root_duration{instance=\"node1\"} 0.5\n";
+        assert_eq!(parse_gauge(text, "sync_block_validation_state_root_duration"), Some(0.5));
+    }
+
+    #[test]
+    fn test_parse_gauge_prefix_no_false_match() {
+        let text =
+            "sync_execution_execution_duration_total 99.0\nsync_execution_execution_duration 0.5\n";
+        assert_eq!(parse_gauge(text, "sync_execution_execution_duration"), Some(0.5));
+    }
+}

bin/reth-bench/src/bench/mod.rs

@@ -12,6 +12,7 @@ pub(crate) mod helpers;
 pub use generate_big_block::{
     RawTransaction, RpcTransactionSource, TransactionCollector, TransactionSource,
 };
+pub(crate) mod metrics_scraper;
 mod new_payload_fcu;
 mod new_payload_only;
 mod output;

bin/reth-bench/src/bench/new_payload_fcu.rs

@@ -13,6 +13,7 @@ use crate::{
     bench::{
         context::BenchContext,
         helpers::parse_duration,
+        metrics_scraper::MetricsScraper,
         output::{
             write_benchmark_results, CombinedResult, NewPayloadResult, TotalGasOutput, TotalGasRow,
         },
@@ -20,9 +21,11 @@ use crate::{
             derive_ws_rpc_url, setup_persistence_subscription, PersistenceWaiter,
         },
     },
-    valid_payload::{block_to_new_payload, call_forkchoice_updated, call_new_payload},
+    valid_payload::{
+        block_to_new_payload, call_forkchoice_updated_with_reth, call_new_payload_with_reth,
+    },
 };
-use alloy_provider::Provider;
+use alloy_provider::{ext::DebugApi, Provider};
 use alloy_rpc_types_engine::ForkchoiceState;
 use clap::Parser;
 use eyre::{Context, OptionExt};
@@ -30,7 +33,7 @@ use reth_cli_runner::CliContext;
 use reth_engine_primitives::config::DEFAULT_PERSISTENCE_THRESHOLD;
 use reth_node_core::args::BenchmarkArgs;
 use std::time::{Duration, Instant};
-use tracing::{debug, info};
+use tracing::{debug, info, warn};
 
 /// `reth benchmark new-payload-fcu` command
 #[derive(Debug, Parser)]
@@ -150,10 +153,18 @@ impl Command {
             auth_provider,
             mut next_block,
             is_optimism,
-            ..
+            use_reth_namespace,
+            rlp_blocks,
         } = BenchContext::new(&self.benchmark, self.rpc_url).await?;
 
         let total_blocks = benchmark_mode.total_blocks();
+
+        let mut metrics_scraper = MetricsScraper::maybe_new(self.benchmark.metrics_url.clone());
+
+        if use_reth_namespace {
+            info!("Using reth_newPayload and reth_forkchoiceUpdated endpoints");
+        }
+
         let buffer_size = self.rpc_block_buffer_size;
 
         // Use a oneshot channel to propagate errors from the spawned task
@@ -176,6 +187,21 @@ impl Command {
                     }
                 };
 
+                let rlp = if rlp_blocks {
+                    let rlp = match block_provider.debug_get_raw_block(next_block.into()).await {
+                        Ok(rlp) => rlp,
+                        Err(e) => {
+                            tracing::error!(target: "reth-bench", "Failed to fetch raw block {next_block}: {e}");
+                            let _ = error_sender
+                                .send(eyre::eyre!("Failed to fetch raw block {next_block}: {e}"));
+                            break;
+                        }
+                    };
+                    Some(rlp)
+                } else {
+                    None
+                };
+
                 let head_block_hash = block.header.hash;
                 let safe_block_hash = block_provider
                     .get_block_by_number(block.header.number.saturating_sub(32).into());
@@ -197,7 +223,7 @@ impl Command {
 
                 next_block += 1;
                 if let Err(e) = sender
-                    .send((block, head_block_hash, safe_block_hash, finalized_block_hash))
+                    .send((block, head_block_hash, safe_block_hash, finalized_block_hash, rlp))
                     .await
                 {
                     tracing::error!(target: "reth-bench", "Failed to send block data: {e}");
@@ -211,7 +237,7 @@ impl Command {
         let total_benchmark_duration = Instant::now();
         let mut total_wait_time = Duration::ZERO;
 
-        while let Some((block, head, safe, finalized)) = {
+        while let Some((block, head, safe, finalized, rlp)) = {
             let wait_start = Instant::now();
             let result = receiver.recv().await;
             total_wait_time += wait_start.elapsed();
@@ -230,16 +256,40 @@ impl Command {
                 finalized_block_hash: finalized,
             };
 
-            let (version, params) = block_to_new_payload(block, is_optimism)?;
+            let (version, params) =
+                block_to_new_payload(block, is_optimism, rlp, use_reth_namespace)?;
             let start = Instant::now();
-            call_new_payload(&auth_provider, version, params).await?;
+            let server_timings =
+                call_new_payload_with_reth(&auth_provider, version, params).await?;
 
-            let new_payload_result = NewPayloadResult { gas_used, latency: start.elapsed() };
+            let np_latency =
+                server_timings.as_ref().map(|t| t.latency).unwrap_or_else(|| start.elapsed());
+            let new_payload_result = NewPayloadResult {
+                gas_used,
+                latency: np_latency,
+                persistence_wait: server_timings.as_ref().and_then(|t| t.persistence_wait),
+                execution_cache_wait: server_timings
+                    .as_ref()
+                    .map(|t| t.execution_cache_wait)
+                    .unwrap_or_default(),
+                sparse_trie_wait: server_timings
+                    .as_ref()
+                    .map(|t| t.sparse_trie_wait)
+                    .unwrap_or_default(),
+            };
 
-            call_forkchoice_updated(&auth_provider, version, forkchoice_state, None).await?;
+            let fcu_start = Instant::now();
+            call_forkchoice_updated_with_reth(&auth_provider, version, forkchoice_state).await?;
+            let fcu_latency = fcu_start.elapsed();
 
-            let total_latency = start.elapsed();
-            let fcu_latency = total_latency - new_payload_result.latency;
+            let total_latency = if server_timings.is_some() {
+                // When using server-side latency for newPayload, derive total from the
+                // independently measured components to avoid mixing server-side and
+                // client-side (network-inclusive) timings.
+                np_latency + fcu_latency
+            } else {
+                start.elapsed()
+            };
             let combined_result = CombinedResult {
                 block_number,
                 gas_limit,
@@ -259,6 +309,12 @@ impl Command {
             };
             info!(target: "reth-bench", progress, %combined_result);
 
+            if let Some(scraper) = metrics_scraper.as_mut() &&
+                let Err(err) = scraper.scrape_after_block(block_number).await
+            {
+                warn!(target: "reth-bench", %err, block_number, "Failed to scrape metrics");
+            }
+
             if let Some(w) = &mut waiter {
                 w.on_block(block_number).await?;
             }
@@ -284,6 +340,10 @@ impl Command {
             write_benchmark_results(path, &gas_output_results, &combined_results)?;
         }
 
+        if let (Some(path), Some(scraper)) = (&self.benchmark.output, &metrics_scraper) {
+            scraper.write_csv(path)?;
+        }
+
         let gas_output =
             TotalGasOutput::with_combined_results(gas_output_results, &combined_results)?;