chore(mdbx): replace deprecated MDBX_NOTLS with MDBX_NOSTICKYTHREADS (#23378)

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>

.changelog/big-mules-cook.md

@@ -1,4 +0,0 @@
----
----
-
-Added site-level meta description for SEO.

.changelog/bold-frogs-run.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: patch
----
-
-Renamed and documented validation methods for clarity. `validate_one_no_state` and `validate_one_against_state` are now public methods `validate_stateless` and `validate_stateful` with improved documentation explaining their respective validation phases.

.changelog/brave-dogs-fly.md

@@ -1,10 +0,0 @@
----
-reth-engine-primitives: patch
-reth-engine-tree: patch
-reth-node-core: patch
-reth-trie-parallel: minor
----
-
-Removed legacy proof calculation system and V2-specific configuration flags.
-
-Removed the legacy (non-V2) proof calculation code paths, simplified multiproof task architecture by removing the dual-mode system, and cleaned up V2-specific CLI flags (`--engine.disable-proof-v2`, `--engine.disable-trie-cache`) that are no longer needed. The codebase now exclusively uses V2 proofs with the sparse trie cache.

.changelog/calm-clams-buzz.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Refactored sparse trie node state tracking to use RLP nodes instead of hashes. Replaced `Option<B256>` hash fields with `SparseNodeState` enum that tracks either dirty nodes or cached RLP nodes with optional database storage flags. Added debug assertions to validate leaf path lengths and improved pruning logic to use node paths directly instead of path-hash tuples.

.changelog/config.toml

@@ -1,20 +0,0 @@
-# Changelogs configuration for reth
-# https://github.com/wevm/changelogs
-
-# How to bump packages that depend on changed packages
-dependent_bump = "patch"
-
-[changelog]
-# Generate per-crate changelogs (vs single root changelog)
-format = "per-crate"
-
-# Fixed groups: all always share the same version
-# reth binaries share version
-[[fixed]]
-members = ["reth"]
-
-# Packages to ignore (internal/test-only crates)
-ignore = [
-    "reth-testing-utils",
-    "reth-bench",
-]

.changelog/cool-suns-rest.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: minor
----
-
-Added support for optional custom stateless and stateful validation hooks in `EthTransactionValidator` via `set_additional_stateless_validation` and `set_additional_stateful_validation` methods. Also implemented a manual `Debug` impl to handle the non-`Debug` function pointer fields.

.changelog/cute-bears-paint.md

@@ -1,17 +0,0 @@
----
-reth: minor
-reth-cli-commands: minor
-reth-e2e-test-utils: minor
-reth-ethereum-cli: minor
-reth-node-core: minor
-reth-optimism-bin: minor
-reth-optimism-cli: minor
-reth-prune: patch
-reth-stages: patch
-reth-storage-api: minor
-reth-storage-db-api: minor
-reth-storage-db-common: patch
-reth-storage-provider: patch
----
-
-Introduced `--storage.v2` flag to control storage mode defaults, replacing the `edge` feature flag with `rocksdb` feature. The new flag enables v2 storage settings (static files + RocksDB routing) while individual `--static-files.*` and `--rocksdb.*` flags can still override defaults. Updated feature gates from `edge` to `rocksdb` across all affected crates.

.changelog/dry-ants-bow.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Removed Windows platform support from the codebase, including the Windows cross-compilation Dockerfile, build targets in Cross.toml and Makefile, and Windows-specific options in the bug report template.

.changelog/dry-ducks-write.md

@@ -1,5 +0,0 @@
----
-reth-network: minor
----
-
-Added reason label to backed_off_peers metric. The metric now tracks backed off peers by reason (too_many_peers, graceful_close, connection_error) to improve observability.

.changelog/dull-koalas-play.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Added recording of `SetRoot` operation in `ParallelSparseTrie::set_root` when the `trie-debug` feature is enabled.

.changelog/eager-mules-fold.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse-parallel: patch
----
-
-Fixed parallel sparse trie to skip revealing disconnected leaves by checking parent branch reachability before inserting leaf nodes.

.changelog/easy-clouds-meow.md

@@ -1,5 +0,0 @@
----
-ef-tests: patch
----
-
-Removed reth-stateless crate and stateless validation from ef-tests.

.changelog/evil-fish-smile.md

@@ -1,6 +0,0 @@
----
-reth-rpc-convert: minor
-reth-storage-rpc-provider: minor
----
-
-Replaced the separate `TryFromBlockResponse`, `TryFromReceiptResponse`, and `TryFromTransactionResponse` traits with a unified `RpcResponseConverter` trait and default `EthRpcConverter` implementation. Removed the `op-alloy-network` dependency and refactored `RpcBlockchainProvider` to store a dynamic converter instance instead of relying on per-type trait bounds.

.changelog/evil-pigs-cry.md

@@ -1,6 +0,0 @@
----
-reth-engine-tree: patch
-reth-trie-sparse-parallel: patch
----
-
-Added tracing spans and debug logs to sparse trie operations for better observability during parallel state root computation.

.changelog/fair-winds-growl.md

@@ -1,6 +0,0 @@
----
-reth-exex: patch
-reth-exex-types: patch
----
-
-Added configurable backfill thresholds to ExEx notifications stream and added regression tests for state provider parity between pipeline and backfill execution paths.

.changelog/fast-fish-cry.md

@@ -1,4 +0,0 @@
----
----
-
-Added WebSocket subscription integration tests for eth_subscribe.

.changelog/fast-seals-play.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: minor
----
-
-Added `consensus_ref` method to `PoolTransaction` trait for borrowing consensus transactions without cloning.

.changelog/fast-waves-smile.md

@@ -1,4 +0,0 @@
----
----
-
-Improved nightly Docker build failure Slack notification with more detailed formatting and context.

.changelog/fix-simulate-revert-code.md

@@ -1,5 +0,0 @@
----
-reth-rpc-eth-types: patch
----
-
-Updated `eth_simulateV1` revert error code from `-32000` to `3` to be consistent with `eth_call`, per [execution-apis#748](https://github.com/ethereum/execution-apis/pull/748).

.changelog/gentle-moons-cry.md

@@ -1,5 +0,0 @@
----
-reth-engine-tree: patch
----
-
-Reordered cache size calculations in `ExecutionCache::new` to group related operations together.

.changelog/icy-lions-slide.md

@@ -1,7 +0,0 @@
----
-reth: patch
-reth-cli-commands: patch
-reth-node-core: patch
----
-
-Removed experimental ress protocol support for stateless Ethereum nodes.

.changelog/keen-geese-bake.md

@@ -1,5 +0,0 @@
----
-reth-engine-tree: patch
----
-
-Added sub-phase timing histograms to the sparse trie event loop, tracking channel wait, proof coalescing, multiproof reveal, and trie update durations separately.

.changelog/lazy-lakes-shout.md

@@ -1,5 +0,0 @@
----
-reth-node-builder: patch
----
-
-Removed biased select in engine service loop to allow fair scheduling of shutdown requests alongside event processing.

.changelog/lively-clams-drink.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: patch
----
-
-Fixed swapped arguments in `blob_tx_priority` function calls, correcting the parameter order to match the function signature.

.changelog/lively-clouds-bake.md

@@ -1,4 +0,0 @@
----
----
-
-Improved documentation overview page with better structure and clarity.

.changelog/lively-deer-shout.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Re-enabled changelog workflow to run automatically on pull requests.

.changelog/lively-foxes-play.md

@@ -1,5 +0,0 @@
----
-reth-node-events: patch
----
-
-Updated consensus engine log message to be more accurate about received updates.

.changelog/merry-koalas-nod.md

@@ -1,6 +0,0 @@
----
-reth-rpc-eth-api: minor
-reth-rpc-server-types: minor
----
-
-Added `eth_getStorageValues` RPC method for batch storage slot retrieval across multiple addresses.

.changelog/mild-foxes-bark.md

@@ -1,6 +0,0 @@
----
-reth-chainspec: minor
-reth-network-peers: minor
----
-
-Removed OP stack bootnodes from default chain configurations and network peers module.

.changelog/nice-trees-drink.md

@@ -1,9 +0,0 @@
----
-reth-network-api: minor
-reth-network-types: minor
-reth-network: minor
-reth-node-core: minor
-reth: minor
----
-
-Added optional ENR fork ID enforcement to filter out peers from incompatible networks during peer discovery, controlled by the `--enforce-enr-fork-id` CLI flag.

.changelog/nice-waves-bow.md

@@ -1,5 +0,0 @@
----
-reth-primitives: patch
----
-
-Moved feature-referenced dependencies from dev-dependencies to optional dependencies to ensure they are available when their corresponding features are enabled.

.changelog/odd-donkeys-chirp.md

@@ -1,5 +0,0 @@
----
-reth-engine-tree: patch
----
-
-Fixed `compare_trie_updates` to return `bool` indicating whether differences were found, and updated the caller to properly use the return value instead of treating all successful comparisons as having no differences.

.changelog/odd-snails-play.md

@@ -1,5 +0,0 @@
----
-reth-node-core: minor
----
-
-Added `with_dev_block_time` helper method to `NodeConfig` for configuring dev miner block production interval.

.changelog/old-dogs-shake.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: minor
----
-
-Added `IntoIter: Send` bounds to `validate_transactions` and `validate_transactions_with_origin` in the `TransactionValidator` trait, avoiding unnecessary `Vec` collects. Simplified default `validate_transactions_with_origin` to delegate to `validate_transactions`.

.changelog/proud-crabs-bark.md

@@ -1,5 +0,0 @@
----
-reth-db-api: patch
----
-
-Changed `StoredNibblesSubKey` encoding to use a stack-allocated `[u8; 65]` array instead of a heap-allocated `Vec<u8>`, avoiding unnecessary heap allocation.

.changelog/proud-wolves-spin.md

@@ -1,5 +0,0 @@
----
-reth-storage-api: patch
----
-
-Added `Arc` to `auto_impl` derive for storage-api traits to support automatic `Arc` wrapper implementations.

.changelog/quick-mice-jump.md

@@ -1,8 +0,0 @@
----
-reth: patch
-reth-engine-tree: patch
-reth-node-builder: patch
-reth-trie-sparse: minor
----
-
-Added `trie-debug` feature for recording sparse trie mutations to aid in debugging state root mismatches.

.changelog/quiet-foxes-dance.md

@@ -1,6 +0,0 @@
----
-reth-static-file-types: patch
-reth-provider: patch
----
-
-Move changeset offsets from segment header to external `.csoff` sidecar file for incremental writes and crash recovery.

.changelog/quiet-frogs-whisper.md

@@ -1,5 +0,0 @@
----
-reth-provider: patch
----
-
-Removed unused staging types from ProviderFactoryBuilder.

.changelog/rich-lakes-bark.md

@@ -1,5 +0,0 @@
----
-reth-provider: patch
----
-
-Fixed sender pruning during block reorg to skip when sender_recovery is fully pruned, preventing a fatal crash when no sender data exists in static files.

.changelog/safe-waves-read.md

@@ -1,7 +0,0 @@
----
-reth-network-types: minor
-reth-network: minor
-reth-node-core: patch
----
-
-Added `PersistedPeerInfo` struct to persist richer peer metadata (kind, fork ID, reputation) to disk. Updated `PeersConfig::with_basic_nodes_from_file` to support both the new `PersistedPeerInfo` format and the legacy `Vec<NodeRecord>` format with automatic conversion, and updated `write_peers_to_file` to exclude backed-off and banned peers.

.changelog/shy-tigers-dry.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Added automated changelog generation infrastructure using wevm/changelogs-rs with Claude Code integration. Configured per-crate changelog format with fixed version groups for reth binaries and exclusions for internal test utilities.

.changelog/swift-owls-fly.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: minor
----
-
-Removed `SerialSparseTrie` from the workspace, consolidating on `ParallelSparseTrie` as the single sparse trie implementation in `reth-trie-sparse`.

.changelog/tall-stars-shout.md

@@ -1,5 +0,0 @@
----
-reth-network: minor
----
-
-Added `fork_id` as a tiebreaker in peer selection when reputations are equal, preferring peers with a discovered `fork_id` as it indicates fork compatibility. Added a test to verify the tiebreaker behavior.

.changelog/tidy-stars-cry.md

@@ -1,5 +0,0 @@
----
-reth-trie-sparse: patch
----
-
-Fixed a bug where trie nodes could appear in both `updated_nodes` and `removed_nodes` simultaneously by removing entries from `removed_nodes` when a node is inserted as updated.

.changelog/vain-bats-bark.md

@@ -1,8 +0,0 @@
----
-reth-trie-common: minor
-reth-trie: minor
-reth-trie-parallel: minor
-reth-engine-tree: patch
----
-
-Moved `ProofV2Target`, `MultiProofTargetsV2`, and `ChunkedMultiProofTargetsV2` from `reth-trie-parallel::targets_v2` into a new `reth-trie-common::target_v2` module, making these types available at a lower level without pulling in the full parallel trie crate. Added a `multiproof_v2` method to `Proof` in `reth-trie` that generates a state multiproof using the V2 proof calculator with synchronous account value encoding.

.changelog/vain-lakes-cry.md

@@ -1,5 +0,0 @@
----
-reth: patch
----
-
-Updated Alloy dependencies from 1.5.2 to 1.6.1.

.changelog/vast-waves-fold.md

@@ -1,4 +0,0 @@
----
----
-
-Expanded CLI integration tests with subcommand help coverage, config TOML validation, genesis JSON validation, and send transaction round-trip test for dev mode.

.changelog/warm-donkeys-dry.md

@@ -1,6 +0,0 @@
----
-reth-engine-local: minor
-reth-node-builder: minor
----
-
-Added trigger-based `MiningMode` variant that allows blocks to be built on-demand via custom streams, and exposed `with_mining_mode` method on `DebugNodeLauncherFuture` to override default mining configuration.

.changelog/warm-foxes-glow.md

@@ -1,5 +0,0 @@
----
-reth-network: minor
----
-
-Added direction labels to `closed_sessions` and `pending_session_failures` metrics. Operators can now distinguish session closures and failures by direction (`active`, `incoming_pending`, `outgoing_pending` for closed sessions; `inbound`, `outbound` for pending session failures).

.changelog/warm-foxes-jump.md

@@ -1,4 +0,0 @@
----
----
-
-Moved Kurtosis CI failure notifications to the hive Slack channel.

.changelog/warm-geese-build.md

@@ -1,7 +0,0 @@
----
-reth-rpc-api: minor
-reth-rpc-builder: patch
-reth-rpc: minor
----
-
-Added `subscribeFinalizedChainNotifications` RPC endpoint that buffers committed chain notifications and emits them once a new finalized block is received.

.changelog/warm-pandas-cook.md

@@ -1,5 +0,0 @@
----
-reth-transaction-pool: patch
----
-
-Fixed a bug where transactions from the same sender were added to the pending subpool out of nonce order. Ensured `process_updates` runs before `add_new_transaction` so that lower-nonce promotions are enqueued before the newly inserted higher-nonce transaction, preserving correct ordering for live `BestTransactions` iterators.

.changelog/zesty-birds-smile.md

@@ -1,6 +0,0 @@
----
-reth-trie: minor
-reth-trie-parallel: minor
----
-
-Added `root_node` and `storage_root_node` methods to proof calculators for efficient root-only calculations. These methods directly return the root node without requiring dummy targets, replacing the previous workaround of passing fake targets to proof generation.

.config/zepter.yaml

@@ -12,7 +12,7 @@ workflows:
         # Check that `A` activates the features of `B`.
         "propagate-feature",
         # These are the features to check:
-        "--features=std,op,dev,asm-keccak,jemalloc,jemalloc-prof,tracy-allocator,tracy,serde-bincode-compat,serde,test-utils,arbitrary,bench,alloy-compat,min-error-logs,min-warn-logs,min-info-logs,min-debug-logs,min-trace-logs,otlp,otlp-logs,js-tracer,portable,keccak-cache-global",
+        "--features=std,op,dev,asm-keccak,jemalloc,jemalloc-prof,tracy-allocator,tracy,serde-bincode-compat,serde,test-utils,arbitrary,bench,alloy-compat,min-error-logs,min-warn-logs,min-info-logs,min-debug-logs,min-trace-logs,otlp,otlp-logs,js-tracer,portable,keccak-cache-global,trie-debug",
         # Do not try to add a new section to `[features]` of `A` only because `B` exposes that feature. There are edge-cases where this is still needed, but we can add them manually.
         "--left-side-feature-missing=ignore",
         # Ignore the case that `A` it outside of the workspace. Otherwise it will report errors in external dependencies that we have no influence on.

.github/actionlint.yaml

@@ -5,3 +5,4 @@ self-hosted-runner:
     - depot-ubuntu-latest-4
     - depot-ubuntu-latest-8
     - depot-ubuntu-latest-16
+    - available

.github/dependabot.yml

@@ -4,10 +4,14 @@ updates:
   directory: "/"
   schedule:
     interval: "weekly"
+  cooldown:
+    default-days: 7
 - package-ecosystem: "cargo"
   directory: "/"
   schedule:
     interval: "weekly"
+  cooldown:
+    default-days: 7
   labels:
     - "A-dependencies"
   commit-message:

.github/scripts/bench-job-summary.js

@@ -0,0 +1,106 @@
+// Generates a rich GitHub Actions job summary for reth-bench results.
+//
+// Reads from environment:
+//   BENCH_WORK_DIR       – Directory containing summary.json
+//   BENCH_PR             – PR number (may be empty)
+//   BENCH_ACTOR          – GitHub user who triggered the bench
+//   BENCH_CORES          – CPU core limit (0 = all)
+//   BENCH_WARMUP_BLOCKS  – Number of warmup blocks
+//   BENCH_SAMPLY         – 'true' if samply profiling was enabled
+//   BENCH_ABBA           – 'true' if ABBA interleaved order was used
+//
+// Usage from actions/github-script:
+//   const jobSummary = require('./.github/scripts/bench-job-summary.js');
+//   await jobSummary({ core, context, chartSha, grafanaUrl, runId });
+
+const fs = require('fs');
+const { verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
+
+module.exports = async function ({ core, context, chartSha, grafanaUrl, runId }) {
+  let summary;
+  try {
+    summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
+  } catch (e) {
+    await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
+    return;
+  }
+
+  const repo = `${context.repo.owner}/${context.repo.repo}`;
+  const prNumber = process.env.BENCH_PR;
+  const actor = process.env.BENCH_ACTOR;
+  const commitUrl = `https://github.com/${repo}/commit`;
+
+  const { emoji, label } = verdict(summary.changes);
+  const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
+  const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
+  const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
+
+  // Header & metadata
+  const metaParts = [];
+  if (prNumber) metaParts.push(`**[PR #${prNumber}](https://github.com/${repo}/pull/${prNumber})**`);
+  metaParts.push(`triggered by @${actor}`);
+
+  let md = `# ${emoji} ${label}\n\n`;
+  md += metaParts.join(' · ') + '\n\n';
+  md += `**Baseline:** ${baselineLink}\n`;
+  md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
+  md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
+
+  // Main comparison table
+  const rows = metricRows(summary);
+  md += `| Metric | Baseline | Feature | Change |\n`;
+  md += `|--------|----------|---------|--------|\n`;
+  for (const r of rows) {
+    md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
+  }
+  md += '\n';
+
+  // Wait time breakdown
+  const wtRows = waitTimeRows(summary);
+  if (wtRows.length > 0) {
+    md += `### Wait Time Breakdown\n\n`;
+    md += `| Metric | Baseline | Feature |\n`;
+    md += `|--------|----------|--------|\n`;
+    for (const r of wtRows) {
+      md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
+    }
+    md += '\n';
+  }
+
+  // Charts
+  if (chartSha) {
+    const prNum = prNumber || '0';
+    const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/pr/${prNum}/${runId}`;
+    const charts = [
+      { file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
+      { file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
+      { file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
+    ];
+    md += `### Charts\n\n`;
+    for (const chart of charts) {
+      md += `<details><summary>${chart.label}</summary>\n\n`;
+      md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
+      md += `</details>\n\n`;
+    }
+  }
+
+  // Samply profiles
+  const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
+  const samplyLinks = Object.entries(samplyUrls).map(([run, url]) => `- **${run}**: [Firefox Profiler](${url})`);
+  if (samplyLinks.length > 0) {
+    md += `### Samply Profiles\n\n${samplyLinks.join('\n')}\n\n`;
+  }
+
+  // Grafana
+  if (grafanaUrl) {
+    md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
+  }
+
+  // Node errors
+  try {
+    const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
+    if (errors.trim()) md += '\n' + errors + '\n';
+  } catch {}
+
+  await core.summary.addRaw(md).write();
+};

.github/scripts/bench-metrics-proxy.py

@@ -0,0 +1,276 @@
+#!/usr/bin/env python3
+"""
+Prometheus metrics proxy that fetches from a local reth node and
+re-exposes with additional benchmark labels.
+
+Reads labels from a JSON file (updated by local-reth-bench.sh between runs)
+and injects them into every Prometheus metric line.
+
+Returns empty 200 when reth is not running (clean Grafana gaps).
+"""
+import argparse
+import ipaddress
+import json
+import subprocess
+import sys
+import time
+from http.server import HTTPServer, BaseHTTPRequestHandler
+from urllib.request import urlopen
+from urllib.error import URLError
+
+
+def read_labels(path):
+    try:
+        with open(path) as f:
+            return json.load(f)
+    except (FileNotFoundError, json.JSONDecodeError):
+        return {}
+
+
+def inject_labels(metrics_bytes, label_str, label_names):
+    """Inject labels into Prometheus text format.
+
+    Operates on bytes and uses simple string ops instead of regex
+    for speed on large payloads (reth exposes thousands of metrics).
+
+    Skips injecting into lines that already contain any of the label names
+    to avoid duplicate labels (which Prometheus rejects).
+    """
+    if not label_str:
+        return metrics_bytes
+
+    label_bytes = label_str.encode("utf-8")
+    # Pre-encode label names for fast duplicate detection
+    label_name_bytes = [n.encode("utf-8") for n in label_names]
+    out = []
+    for line in metrics_bytes.split(b"\n"):
+        # Skip comments and blank lines
+        if line.startswith(b"#") or not line:
+            out.append(line)
+            continue
+
+        brace = line.find(b"{")
+        space = line.find(b" ")
+
+        if space == -1:
+            # Malformed, pass through
+            out.append(line)
+        elif brace != -1 and brace < space:
+            # Has labels: metric{existing="val"} 123
+            close = line.find(b"}", brace)
+            if close == -1:
+                out.append(line)
+                continue
+
+            # Filter out labels that already exist in this line
+            existing = line[brace + 1:close]
+            inject = label_bytes
+            if existing:
+                for name in label_name_bytes:
+                    if name + b"=" in existing:
+                        # Rebuild inject string excluding this label
+                        inject = _remove_label(inject, name)
+                if not inject:
+                    out.append(line)
+                    continue
+
+            if close == brace + 1:
+                # Empty braces: metric{} 123
+                out.append(line[:close] + inject + line[close:])
+            else:
+                out.append(line[:close] + b"," + inject + line[close:])
+        else:
+            # No labels: metric 123
+            out.append(line[:space] + b"{" + label_bytes + b"}" + line[space:])
+
+    return b"\n".join(out)
+
+
+def _remove_label(label_bytes, name):
+    """Remove a single label (name=\"...\") from a comma-separated label string."""
+    parts = []
+    for part in label_bytes.split(b","):
+        if not part.startswith(name + b"="):
+            parts.append(part)
+    return b",".join(parts)
+
+
+def build_label_str(labels):
+    """Pre-format the label injection string: key1="val1",key2="val2" """
+    if not labels:
+        return ""
+    return ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
+
+
+def build_elapsed_gauge(labels):
+    """Build a bench_elapsed_seconds gauge from run_start_epoch in labels."""
+    start = labels.get("run_start_epoch")
+    if not start:
+        return b""
+    try:
+        elapsed = time.time() - float(start)
+    except (ValueError, TypeError):
+        return b""
+    # Build labels excluding internal keys
+    display = {k: v for k, v in labels.items()
+               if k not in ("run_start_epoch", "reference_epoch")}
+    lstr = build_label_str(display)
+    return (
+        f"# HELP bench_elapsed_seconds Seconds since benchmark run started\n"
+        f"# TYPE bench_elapsed_seconds gauge\n"
+        f"bench_elapsed_seconds{{{lstr}}} {elapsed:.1f}\n"
+    ).encode("utf-8")
+
+
+def compute_timestamp_ms(labels):
+    """Compute a synthetic timestamp so all runs share a common time origin.
+
+    Returns the timestamp in milliseconds, or None if not enough info.
+    Uses: reference_epoch + (now - run_start_epoch) → all runs overlay at
+    the same Grafana time range.
+    """
+    ref = labels.get("reference_epoch")
+    start = labels.get("run_start_epoch")
+    if not ref or not start:
+        return None
+    try:
+        elapsed = time.time() - float(start)
+        return int((float(ref) + elapsed) * 1000)
+    except (ValueError, TypeError):
+        return None
+
+
+def inject_timestamps(metrics_bytes, timestamp_ms):
+    """Append a Prometheus timestamp (ms) to every data line.
+
+    Prometheus text format: metric{labels} value [timestamp_ms]
+    Adding timestamps causes Prometheus to store all runs' samples
+    at the same relative time, enabling natural overlay in Grafana.
+    """
+    if timestamp_ms is None:
+        return metrics_bytes
+
+    ts = str(timestamp_ms).encode("utf-8")
+    out = []
+    for line in metrics_bytes.split(b"\n"):
+        if line.startswith(b"#") or not line:
+            out.append(line)
+        else:
+            out.append(line + b" " + ts)
+    return b"\n".join(out)
+
+
+class MetricsHandler(BaseHTTPRequestHandler):
+    # Use HTTP/1.1 so Content-Length is respected and Prometheus
+    # doesn't have to rely on connection close to detect end of body.
+    protocol_version = "HTTP/1.1"
+
+    def do_GET(self):
+        src = self.client_address[0]
+        try:
+            resp = urlopen(self.server.upstream, timeout=2)
+            metrics = resp.read()
+        except (URLError, ConnectionError, OSError):
+            # reth not running — return empty 200
+            self._send(b"")
+            #print(f"  scrape from {src}: empty (reth not running)", flush=True)
+            return
+
+        all_labels = read_labels(self.server.labels_file)
+        # Internal keys — not injected as Prometheus labels
+        internal = ("run_start_epoch", "reference_epoch")
+        labels = {k: v for k, v in all_labels.items() if k not in internal}
+        label_str = build_label_str(labels)
+        label_names = sorted(labels.keys())
+
+        t0 = time.monotonic()
+        result = inject_labels(metrics, label_str, label_names)
+        result += build_elapsed_gauge(all_labels)
+        ts_ms = compute_timestamp_ms(all_labels)
+        result = inject_timestamps(result, ts_ms)
+        dt = time.monotonic() - t0
+
+        self._send(result)
+        print(f"  scrape from {src}: {len(metrics)} -> {len(result)} bytes, "
+              f"inject {dt*1000:.1f}ms", flush=True)
+
+    def _send(self, body):
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain; version=0.0.4")
+        self.send_header("Content-Length", str(len(body)))
+        self.send_header("Connection", "close")
+        self.end_headers()
+        if body:
+            self.wfile.write(body)
+
+    def log_message(self, format, *args):
+        pass  # suppress per-request logging
+
+
+def resolve_bind_address(subnet_cidr):
+    """Find the local IP address that belongs to the given subnet.
+
+    Uses ``ip -j addr show`` to enumerate interfaces and returns the first
+    address that falls within *subnet_cidr* (e.g. ``10.10.0.0/24``).
+    """
+    network = ipaddress.ip_network(subnet_cidr, strict=False)
+    try:
+        result = subprocess.run(
+            ["ip", "-j", "addr", "show"],
+            capture_output=True, text=True, check=True,
+        )
+        interfaces = json.loads(result.stdout)
+    except (subprocess.CalledProcessError, FileNotFoundError, json.JSONDecodeError) as exc:
+        print(f"Error: cannot enumerate interfaces: {exc}", file=sys.stderr)
+        sys.exit(1)
+
+    for iface in interfaces:
+        for addr_info in iface.get("addr_info", []):
+            try:
+                addr = ipaddress.ip_address(addr_info["local"])
+            except (KeyError, ValueError):
+                continue
+            if addr in network:
+                return str(addr)
+
+    print(f"Error: no interface address found in subnet {subnet_cidr}", file=sys.stderr)
+    sys.exit(1)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Prometheus metrics proxy with label injection")
+    parser.add_argument("--labels", default="/tmp/bench-metrics-labels.json",
+                        help="Path to JSON file with labels to inject (default: /tmp/bench-metrics-labels.json)")
+    parser.add_argument("--upstream", default="http://127.0.0.1:9100/",
+                        help="Upstream reth metrics URL (default: http://127.0.0.1:9100/)")
+
+    bind_group = parser.add_mutually_exclusive_group()
+    bind_group.add_argument("--bind", default=None,
+                            help="Address to bind the proxy (default: 0.0.0.0)")
+    bind_group.add_argument("--subnet", default=None,
+                            help="Auto-detect bind address from a local interface in this subnet (e.g. 10.10.0.0/24)")
+
+    parser.add_argument("--port", type=int, default=9090,
+                        help="Port to bind the proxy (default: 9090)")
+    args = parser.parse_args()
+
+    if args.subnet:
+        bind_addr = resolve_bind_address(args.subnet)
+    elif args.bind:
+        bind_addr = args.bind
+    else:
+        bind_addr = "0.0.0.0"
+
+    server = HTTPServer((bind_addr, args.port), MetricsHandler)
+    server.upstream = args.upstream
+    server.labels_file = args.labels
+
+    print(f"bench-metrics-proxy listening on {bind_addr}:{args.port}")
+    print(f"  upstream: {args.upstream}")
+    print(f"  labels:   {args.labels}")
+    sys.stdout.flush()
+    server.serve_forever()
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/bench-reth-build.sh

@@ -1,99 +1,76 @@
 #!/usr/bin/env bash
 #
-# Builds (or fetches from cache) reth binaries for benchmarking.
+# Builds reth binaries for benchmarking from local source only.
 #
-# Usage: bench-reth-build.sh <baseline|feature> <source-dir> <commit> [branch-sha]
+# Usage: bench-reth-build.sh <baseline|feature> <source-dir> <commit>
 #
-#   baseline — build/fetch the baseline binary at <commit> (merge-base)
+#   baseline — build the baseline binary at <commit> (merge-base)
 #              source-dir must be checked out at <commit>
-#   feature  — build/fetch the candidate binary + reth-bench at <commit>
+#   feature  — build the candidate binary + reth-bench at <commit>
 #              source-dir must be checked out at <commit>
-#              optional branch-sha is the PR head commit for cache key
 #
 # Outputs:
-#   baseline: <source-dir>/target/profiling/reth
-#   feature:  <source-dir>/target/profiling/reth, reth-bench installed to cargo bin
+#   baseline: <source-dir>/target/profiling/reth (or reth-bb if BENCH_BIG_BLOCKS=true)
+#   feature:  <source-dir>/target/profiling/reth (or reth-bb), reth-bench installed to cargo bin
 #
-# Required: mc (MinIO client) with a configured alias
-set -euo pipefail
+# Optional env: BENCH_BIG_BLOCKS (true/false) — build reth-bb instead of reth
+set -euxo pipefail
 
-MC="mc"
 MODE="$1"
 SOURCE_DIR="$2"
 COMMIT="$3"
 
-# Verify a cached reth binary was built from the expected commit.
-# `reth --version` outputs "Commit SHA: <full-sha>" on its own line.
-verify_binary() {
-  local binary="$1" expected_commit="$2"
-  local version binary_sha
-  version=$("$binary" --version 2>/dev/null) || return 1
-  binary_sha=$(echo "$version" | sed -n 's/^Commit SHA: *//p')
-  if [ -z "$binary_sha" ]; then
-    echo "Warning: could not extract commit SHA from version output"
-    return 1
+BIG_BLOCKS="${BENCH_BIG_BLOCKS:-false}"
+# The node binary to build: reth-bb for big blocks, reth otherwise
+if [ "$BIG_BLOCKS" = "true" ]; then
+  NODE_BIN="reth-bb"
+  NODE_PKG="-p reth-bb"
+else
+  NODE_BIN="reth"
+  NODE_PKG="--bin reth"
+fi
+
+# Tracy support: when BENCH_TRACY is "on" or "full", add Tracy cargo features
+# and frame pointers for accurate stack traces.
+EXTRA_FEATURES=""
+EXTRA_RUSTFLAGS=""
+if [ "${BENCH_TRACY:-off}" != "off" ]; then
+  EXTRA_FEATURES="tracy,tracy-client/ondemand"
+  EXTRA_RUSTFLAGS=" -C force-frame-pointers=yes"
+fi
+
+# Build the requested node binary with the benchmark profile.
+build_node_binary() {
+  local features_arg=""
+  local workspace_arg=""
+
+  cd "$SOURCE_DIR"
+  if [ -n "$EXTRA_FEATURES" ]; then
+    # --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
+    features_arg="--features ${EXTRA_FEATURES}"
+    workspace_arg="--workspace"
   fi
-  if [ "$binary_sha" = "$expected_commit" ]; then
-    return 0
-  fi
-  echo "Cache mismatch: binary built from ${binary_sha} but expected ${expected_commit}"
-  return 1
+
+  # shellcheck disable=SC2086
+  RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
+    cargo build --locked --profile profiling $NODE_PKG $workspace_arg $features_arg
 }
 
 case "$MODE" in
   baseline|main)
-    BUCKET="minio/reth-binaries/${COMMIT}"
-    mkdir -p "${SOURCE_DIR}/target/profiling"
-
-    CACHE_VALID=false
-    if $MC stat "${BUCKET}/reth" &>/dev/null; then
-      echo "Cache hit for baseline (${COMMIT}), downloading binary..."
-      $MC cp "${BUCKET}/reth" "${SOURCE_DIR}/target/profiling/reth"
-      chmod +x "${SOURCE_DIR}/target/profiling/reth"
-      if verify_binary "${SOURCE_DIR}/target/profiling/reth" "${COMMIT}"; then
-        CACHE_VALID=true
-      else
-        echo "Cached baseline binary is stale, rebuilding..."
-      fi
-    fi
-    if [ "$CACHE_VALID" = false ]; then
-      echo "Building baseline (${COMMIT}) from source..."
-      cd "${SOURCE_DIR}"
-      cargo build --profile profiling --bin reth
-      $MC cp target/profiling/reth "${BUCKET}/reth"
-    fi
+    echo "Building baseline ${NODE_BIN} (${COMMIT}) from source..."
+    build_node_binary
     ;;
 
   feature|branch)
-    BRANCH_SHA="${4:-$COMMIT}"
-    BUCKET="minio/reth-binaries/${BRANCH_SHA}"
-
-    CACHE_VALID=false
-    if $MC stat "${BUCKET}/reth" &>/dev/null && $MC stat "${BUCKET}/reth-bench" &>/dev/null; then
-      echo "Cache hit for ${BRANCH_SHA}, downloading binaries..."
-      mkdir -p "${SOURCE_DIR}/target/profiling"
-      $MC cp "${BUCKET}/reth" "${SOURCE_DIR}/target/profiling/reth"
-      $MC cp "${BUCKET}/reth-bench" /home/ubuntu/.cargo/bin/reth-bench
-      chmod +x "${SOURCE_DIR}/target/profiling/reth" /home/ubuntu/.cargo/bin/reth-bench
-      if verify_binary "${SOURCE_DIR}/target/profiling/reth" "${COMMIT}"; then
-        CACHE_VALID=true
-      else
-        echo "Cached feature binary is stale, rebuilding..."
-      fi
-    fi
-    if [ "$CACHE_VALID" = false ]; then
-      echo "Building feature (${COMMIT}) from source..."
-      cd "${SOURCE_DIR}"
-      rustup show active-toolchain || rustup default stable
-      make profiling
-      make install-reth-bench
-      $MC cp target/profiling/reth "${BUCKET}/reth"
-      $MC cp "$(which reth-bench)" "${BUCKET}/reth-bench"
-    fi
+    echo "Building feature ${NODE_BIN} (${COMMIT}) from source..."
+    rustup show active-toolchain || rustup default stable
+    build_node_binary
+    make -C "$SOURCE_DIR" install-reth-bench
     ;;
 
   *)
-    echo "Usage: $0 <baseline|feature> <source-dir> <commit> [branch-sha]"
+    echo "Usage: $0 <baseline|feature> <source-dir> <commit>"
     exit 1
     ;;
 esac

.github/scripts/bench-reth-local.sh

@@ -0,0 +1,567 @@
+#!/usr/bin/env bash
+#
+# local-reth-bench.sh — Run the reth Engine API benchmark locally.
+#
+# Replicates the CI bench.yml workflow (build, local snapshot validation, system tuning,
+# interleaved B-F-F-B execution, summary, charts) without any GitHub
+# Actions glue (no PR comments, no artifact upload, no Slack).
+#
+# Usage:
+#   local-reth-bench.sh <baseline-ref> <feature-ref> [options]
+#
+# Options:
+#   --blocks N      Number of blocks to benchmark (default: 500)
+#   --warmup N      Number of warmup blocks (default: 100)
+#   --cores N       Limit reth to N CPU cores, 0 = all available (default: 0)
+#   --samply        Enable samply profiling
+#   --tracy MODE    Tracy profiling: off, on, full (default: off)
+#   --tracy-filter F Tracy tracing filter (default: debug)
+#   --no-tune       Skip system tuning (useful on dev machines / macOS)
+#
+# Requires: the reth repo at RETH_REPO (default: ~/reth)
+#
+# Dependencies (install before first run):
+#   schelk, cpupower, taskset, stdbuf, python3, curl,
+#   make, uv, jq, Rust toolchain (cargo/rustup)
+# Optional:
+#   mc for Tracy profile upload
+#
+# The script delegates to the existing bench-reth-*.sh scripts in the reth
+# repo for the actual build, snapshot, and run steps.
+set -euxo pipefail
+
+# ── PATH ──────────────────────────────────────────────────────────────
+# Ensure cargo and user-local bins (uv) are visible
+export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
+
+# ── Defaults ──────────────────────────────────────────────────────────
+RETH_REPO="${RETH_REPO:-$HOME/reth}"
+BLOCKS=500
+WARMUP=100
+CORES=0
+SAMPLY=false
+TRACY="off"
+TRACY_FILTER="debug"
+TUNE=true
+BASELINE_REF=""
+FEATURE_REF=""
+
+# ── Parse arguments ──────────────────────────────────────────────────
+usage() {
+  cat <<EOF
+Usage: $(basename "$0") <baseline-ref> <feature-ref> [options]
+
+Options:
+  --blocks N         Number of blocks to benchmark (default: 500)
+  --warmup N         Number of warmup blocks (default: 100)
+  --cores N          Limit reth to N CPU cores (default: 0 = all)
+  --samply           Enable samply profiling
+  --tracy MODE       Tracy profiling: off, on, full (default: off)
+                       on   = tracing only (lower overhead)
+                       full = tracing + CPU sampling (higher overhead)
+  --tracy-filter F   Tracy tracing filter (default: debug)
+  --no-tune          Skip system tuning
+EOF
+  exit 1
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --blocks)       BLOCKS="$2"; shift 2 ;;
+    --warmup)       WARMUP="$2"; shift 2 ;;
+    --cores)        CORES="$2"; shift 2 ;;
+    --samply)       SAMPLY=true; shift ;;
+    --tracy)        TRACY="$2"; shift 2 ;;
+    --tracy-filter) TRACY_FILTER="$2"; shift 2 ;;
+    --no-tune)      TUNE=false; shift ;;
+    --help|-h)  usage ;;
+    -*)         echo "Unknown option: $1"; usage ;;
+    *)
+      if [ -z "$BASELINE_REF" ]; then
+        BASELINE_REF="$1"
+      elif [ -z "$FEATURE_REF" ]; then
+        FEATURE_REF="$1"
+      else
+        echo "Unexpected argument: $1"; usage
+      fi
+      shift
+      ;;
+  esac
+done
+
+if [ -z "$BASELINE_REF" ] || [ -z "$FEATURE_REF" ]; then
+  echo "Error: both <baseline-ref> and <feature-ref> are required."
+  usage
+fi
+
+# Validate --tracy value
+case "$TRACY" in
+  off|on|full) ;;
+  *) echo "Error: --tracy must be off, on, or full (got: $TRACY)"; usage ;;
+esac
+
+# Samply + tracy=full are mutually exclusive (both use perf sampling)
+if [ "$SAMPLY" = "true" ] && [ "$TRACY" = "full" ]; then
+  echo "Warning: samply and tracy=full both use perf sampling; downgrading tracy to 'on'."
+  TRACY="on"
+fi
+
+# ── Check dependencies ───────────────────────────────────────────────
+missing=()
+for cmd in schelk cpupower taskset stdbuf python3 curl make uv jq cargo; do
+  command -v "$cmd" &>/dev/null || missing+=("$cmd")
+done
+if [ ${#missing[@]} -gt 0 ]; then
+  echo "Error: missing required tools: ${missing[*]}"
+  echo "See the CI 'Install dependencies' step in .github/workflows/bench.yml for install instructions."
+  exit 1
+fi
+
+if [ "$TRACY" != "off" ]; then
+  if ! command -v tracy-capture &>/dev/null; then
+    echo "Error: tracy-capture is required for --tracy $TRACY"
+    exit 1
+  fi
+fi
+
+# Ensure tools that run via sudo are in a sudo-visible path.
+# The bench scripts use `sudo schelk` / `sudo samply` but cargo installs
+# them to ~/.cargo/bin which sudo's secure_path doesn't include.
+for cmd in schelk samply; do
+  if command -v "$cmd" &>/dev/null && ! sudo sh -c "command -v $cmd" &>/dev/null; then
+    echo "Installing $cmd to /usr/local/bin (needed for sudo)..."
+    sudo install "$(command -v "$cmd")" /usr/local/bin/
+  fi
+done
+
+if [ ! -d "$RETH_REPO/.git" ]; then
+  echo "Error: RETH_REPO=$RETH_REPO is not a git repository."
+  echo "Set RETH_REPO or clone reth to ~/reth"
+  exit 1
+fi
+
+# ── Resolve paths ────────────────────────────────────────────────────
+SELF_DIR="$(cd "$(dirname "$0")" && pwd)"
+SCRIPTS_DIR="${RETH_REPO}/.github/scripts"
+BENCH_WORK_DIR="${RETH_REPO}/../bench-work-$(date +%Y%m%d-%H%M%S)"
+BASELINE_SRC="${RETH_REPO}/../reth-baseline"
+FEATURE_SRC="${RETH_REPO}/../reth-feature"
+
+mkdir -p "$BENCH_WORK_DIR"
+BENCH_WORK_DIR="$(cd "$BENCH_WORK_DIR" && pwd)"
+
+# ── Global cleanup trap (restores system tuning on any exit) ─────────
+TUNING_APPLIED=false
+CSTATE_PID=
+METRICS_PROXY_PID=
+cleanup_global() {
+  [ -n "$METRICS_PROXY_PID" ] && kill "$METRICS_PROXY_PID" 2>/dev/null || true
+  if [ "$TUNING_APPLIED" = true ]; then
+    echo
+    echo "▸ Restoring system settings..."
+    [ -n "$CSTATE_PID" ] && kill "$CSTATE_PID" 2>/dev/null || true
+    sudo systemctl start irqbalance cron atd 2>/dev/null || true
+    echo "  System settings restored."
+  fi
+}
+trap cleanup_global EXIT
+
+echo "═══════════════════════════════════════════════════════════"
+echo "  reth local benchmark"
+echo "═══════════════════════════════════════════════════════════"
+echo "  Baseline ref : $BASELINE_REF"
+echo "  Feature ref  : $FEATURE_REF"
+echo "  Blocks       : $BLOCKS"
+echo "  Warmup       : $WARMUP"
+echo "  Cores        : $CORES"
+echo "  Samply       : $SAMPLY"
+echo "  Tracy        : $TRACY"
+echo "  Tracy filter : $TRACY_FILTER"
+echo "  System tune  : $TUNE"
+echo "  Work dir     : $BENCH_WORK_DIR"
+echo "  Reth repo    : $RETH_REPO"
+echo "═══════════════════════════════════════════════════════════"
+echo
+
+# Enable sccache if available (matches CI's RUSTC_WRAPPER=sccache)
+if command -v sccache &>/dev/null; then
+  export RUSTC_WRAPPER="sccache"
+fi
+
+# Export env vars expected by the bench-reth-*.sh scripts
+export BENCH_BLOCKS="$BLOCKS"
+export BENCH_WARMUP_BLOCKS="$WARMUP"
+export BENCH_CORES="$CORES"
+export BENCH_SAMPLY="$SAMPLY"
+export BENCH_TRACY="$TRACY"
+export BENCH_TRACY_FILTER="$TRACY_FILTER"
+export BENCH_WORK_DIR
+export SCHELK_MOUNT="${SCHELK_MOUNT:-/reth-bench}"
+export BENCH_RPC_URL="${BENCH_RPC_URL:-https://ethereum.reth.rs/rpc}"
+export BENCH_METRICS_ADDR="127.0.0.1:9100"
+
+# ── Step 1: Resolve refs to full SHAs ────────────────────────────────
+echo "▸ Resolving git refs..."
+cd "$RETH_REPO"
+
+resolve_ref() {
+  local ref="$1"
+  git fetch origin "$ref" --quiet 2>/dev/null || true
+  git rev-parse "$ref" 2>/dev/null \
+    || git rev-parse "origin/$ref" 2>/dev/null \
+    || { echo "Error: cannot resolve ref '$ref'"; exit 1; }
+}
+
+BASELINE_SHA="$(resolve_ref "$BASELINE_REF")"
+FEATURE_SHA="$(resolve_ref "$FEATURE_REF")"
+echo "  Baseline SHA : $BASELINE_SHA"
+echo "  Feature SHA  : $FEATURE_SHA"
+echo
+
+# ── Step 2: Prepare source directories ───────────────────────────────
+echo "▸ Preparing source directories..."
+
+prepare_source() {
+  local src_dir="$1" ref="$2"
+  if [ -d "$src_dir" ]; then
+    git -C "$src_dir" fetch origin "$ref" 2>/dev/null || true
+  else
+    git clone --recurse-submodules "$RETH_REPO" "$src_dir"
+  fi
+  git -C "$src_dir" checkout "$ref" --force
+  git -C "$src_dir" submodule update --init --recursive
+}
+
+prepare_source "$BASELINE_SRC" "$BASELINE_SHA"
+prepare_source "$FEATURE_SRC" "$FEATURE_SHA"
+BASELINE_SRC="$(cd "$BASELINE_SRC" && pwd)"
+FEATURE_SRC="$(cd "$FEATURE_SRC" && pwd)"
+echo "  Baseline src : $BASELINE_SRC"
+echo "  Feature src  : $FEATURE_SRC"
+echo
+
+# ── Step 3: Validate local snapshot ──────────────────────────────────
+echo "▸ Validating local snapshot..."
+cd "$RETH_REPO"
+"${SCRIPTS_DIR}/bench-reth-snapshot.sh"
+echo "  Snapshot is ready."
+echo
+
+# ── Step 4: Build binaries in parallel ───────────────────────────────
+echo "▸ Building binaries (parallel)..."
+cd "$RETH_REPO"
+
+FAIL=0
+
+"${SCRIPTS_DIR}/bench-reth-build.sh" baseline "$BASELINE_SRC" "$BASELINE_SHA" &
+PID_BASELINE=$!
+
+"${SCRIPTS_DIR}/bench-reth-build.sh" feature "$FEATURE_SRC" "$FEATURE_SHA" &
+PID_FEATURE=$!
+
+wait $PID_BASELINE || FAIL=1
+wait $PID_FEATURE  || FAIL=1
+
+if [ $FAIL -ne 0 ]; then
+  echo "Error: one or more build tasks failed"
+  exit 1
+fi
+echo "  Binaries built successfully."
+echo
+
+# ── Step 5: System tuning (optional) ────────────────────────────────
+if [ "$TUNE" = "true" ]; then
+  echo "▸ Applying system tuning..."
+
+  sudo cpupower frequency-set -g performance 2>/dev/null || true
+
+  # Disable turbo boost (Intel + AMD)
+  echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
+  echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
+
+  sudo swapoff -a 2>/dev/null || true
+  echo 0 | sudo tee /proc/sys/kernel/randomize_va_space 2>/dev/null || true
+
+  # Disable SMT (hyperthreading)
+  for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
+    [ -f "$cpu" ] || continue
+    first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
+    current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
+    if [ "$current" != "$first" ]; then
+      echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" 2>/dev/null || true
+    fi
+  done
+  echo "  Online CPUs: $(nproc)"
+
+  # Disable transparent huge pages
+  for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
+    if [ -d "$p" ]; then
+      echo never | sudo tee "$p/enabled" 2>/dev/null || true
+      echo never | sudo tee "$p/defrag" 2>/dev/null || true
+      break
+    fi
+  done
+
+  # Prevent deep C-states
+  sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
+  CSTATE_PID=$!
+
+  # Pin IRQs to core 0
+  for irq in /proc/irq/*/smp_affinity_list; do
+    echo 0 | sudo tee "$irq" 2>/dev/null || true
+  done
+
+  # Stop noisy background services
+  sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
+
+  TUNING_APPLIED=true
+
+  # Log environment for reproducibility (matches CI)
+  echo "  === Benchmark environment ==="
+  echo "  Kernel : $(uname -r)"
+  lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA' | sed 's/^/  /'
+  echo "  Governor : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo unknown)"
+  echo "  Freq     : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null || echo unknown)"
+  echo "  THP      : $(cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo unknown)"
+  free -h | sed 's/^/  /'
+  echo "  System tuning applied."
+  echo
+fi
+
+# ── Step 5b: Tracefs mount (tracy=full only) ─────────────────────────
+if [ "$TRACY" = "full" ] && [ "$(uname)" = "Linux" ]; then
+  echo "▸ Mounting tracefs for Tracy full mode..."
+  sudo mount -t tracefs tracefs /sys/kernel/tracing -o mode=755 2>/dev/null || true
+fi
+
+# ── Tracy upload & viewer helpers ────────────────────────────────────
+TRACY_VIEWER_BASE="${TRACY_VIEWER_BASE:-}"
+
+tracy_viewer_url() {
+  local profile_url="$1"
+  if [ -z "$TRACY_VIEWER_BASE" ]; then
+    echo ""
+    return
+  fi
+  local encoded
+  encoded=$(python3 -c "import urllib.parse, sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$profile_url")
+  echo "${TRACY_VIEWER_BASE}?profile_url=${encoded}"
+}
+
+upload_tracy() {
+  local label="$1" output_dir="$2" sha="$3"
+  local tracy_file="$output_dir/tracy-profile.tracy"
+
+  if [ ! -f "$tracy_file" ]; then
+    echo "  Tracy: no profile found, skipping upload."
+    return
+  fi
+
+  local timestamp short_sha remote_name bucket mc_alias
+  timestamp=$(date +%Y%m%d-%H%M%S)
+  short_sha="${sha:0:7}"
+  remote_name="${label}-${short_sha}-${timestamp}.tracy"
+  bucket="${TRACY_BUCKET:-tracy-profiles}"
+  mc_alias="${MC_ALIAS:-minio}"
+  local minio_base="${TRACY_MINIO_URL:-http://minio.minio.svc.cluster.local:9000}"
+
+  echo "  Tracy: uploading profile..."
+  if mc cp "$tracy_file" "${mc_alias}/${bucket}/${remote_name}"; then
+    local url="${minio_base}/${bucket}/${remote_name}"
+    echo "$url" > "$output_dir/tracy_url.txt"
+    local viewer
+    viewer=$(tracy_viewer_url "$url")
+    if [ -n "$viewer" ]; then
+      echo "$viewer" > "$output_dir/tracy_viewer_url.txt"
+      echo "  Tracy: uploaded → $viewer"
+    else
+      echo "  Tracy: uploaded → $url"
+    fi
+  else
+    echo "  Tracy: upload failed (non-fatal)."
+  fi
+
+  # Delete large profile to free disk
+  rm -f "$tracy_file"
+}
+
+# ── Step 6: Pre-flight cleanup ───────────────────────────────────────
+echo "▸ Pre-flight cleanup..."
+pkill -f bench-metrics-proxy 2>/dev/null || true
+sudo systemctl stop "${RETH_SCOPE:-reth-bench.scope}" 2>/dev/null || true
+sudo systemctl reset-failed "${RETH_SCOPE:-reth-bench.scope}" 2>/dev/null || true
+sudo schelk recover -y --kill || sudo schelk full-recover -y || true
+echo
+
+# ── Step 7: Interleaved benchmark runs (B-F-F-B) ────────────────────
+# This ordering reduces systematic bias from thermal drift and cache warming.
+BASELINE_BIN="${BASELINE_SRC}/target/profiling/reth"
+FEATURE_BIN="${FEATURE_SRC}/target/profiling/reth"
+
+# Start metrics proxy (reth → label injection → Prometheus)
+LABELS_FILE="/tmp/bench-metrics-labels.json"
+echo '{}' > "$LABELS_FILE"
+METRICS_SUBNET="${METRICS_SUBNET:-10.10.0.0/24}"
+METRICS_PORT="${METRICS_PORT:-9090}"
+python3 "${SELF_DIR}/bench-metrics-proxy.py" \
+  --labels "$LABELS_FILE" \
+  --upstream "http://${BENCH_METRICS_ADDR}/" \
+  --subnet "$METRICS_SUBNET" \
+  --port "$METRICS_PORT" &
+METRICS_PROXY_PID=$!
+echo "▸ Metrics proxy started (PID $METRICS_PROXY_PID) on subnet ${METRICS_SUBNET}, port ${METRICS_PORT}"
+
+# Unique benchmark ID: local-<timestamp> for local runs, ci-<run_id> for CI
+BENCH_ID="local-$(basename "$BENCH_WORK_DIR" | sed 's/bench-work-//')"
+# Reference epoch: shared time origin so all runs overlay in Grafana.
+# The proxy maps each run's elapsed time onto this common origin.
+BENCH_REFERENCE_EPOCH=$(date +%s)
+
+write_labels() {
+  local run_label="$1" run_type="$2" ref="$3" sha="$4"
+  LAST_RUN_START=$(date +%s)
+  cat > "$LABELS_FILE" <<-EOF
+	{"benchmark_run":"${run_label}","run_type":"${run_type}","git_ref":"${ref}","bench_sha":"${sha}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
+	EOF
+}
+
+run_bench() {
+  local label="$1" binary="$2" output_dir="$3"
+  echo "▸ Running benchmark: ${label}..."
+  cd "$RETH_REPO"
+  if command -v taskset &>/dev/null; then
+    taskset -c 0 "${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
+  else
+    "${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
+  fi
+  echo "  ✓ ${label} complete."
+  echo
+}
+
+write_labels "baseline-1" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
+run_bench "baseline-1" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-1"
+
+write_labels "feature-1" "feature" "$FEATURE_REF" "$FEATURE_SHA"
+run_bench "feature-1"  "$FEATURE_BIN"  "$BENCH_WORK_DIR/feature-1"
+
+write_labels "feature-2" "feature" "$FEATURE_REF" "$FEATURE_SHA"
+run_bench "feature-2"  "$FEATURE_BIN"  "$BENCH_WORK_DIR/feature-2"
+
+write_labels "baseline-2" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
+run_bench "baseline-2" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-2"
+
+# ── Compute Grafana URL ──────────────────────────────────────────────
+GRAFANA_BASE_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr"
+GRAFANA_DATASOURCE="ef57fux92e9z4e"
+LAST_RUN_DURATION=$(( $(date +%s) - LAST_RUN_START ))
+FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
+TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
+GRAFANA_URL="${GRAFANA_BASE_URL}?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=${GRAFANA_DATASOURCE}&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
+
+# ── Step 8: Scan logs for errors ─────────────────────────────────────
+echo "▸ Scanning logs for errors..."
+ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
+found_errors=false
+for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
+  LOG="$BENCH_WORK_DIR/$run_dir/node.log"
+  [ -f "$LOG" ] || continue
+  panics=$(grep -c -E 'panicked at' "$LOG" 2>/dev/null || true)
+  errors=$(grep -c ' ERROR ' "$LOG" 2>/dev/null || true)
+  if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
+    if [ "$found_errors" = false ]; then
+      printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
+      found_errors=true
+    fi
+    printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' \
+      "$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
+    if [ "$panics" -gt 0 ]; then
+      printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
+      grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
+      printf '```\n' >> "$ERRORS_FILE"
+    fi
+    if [ "$errors" -gt 0 ]; then
+      printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
+      grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
+      printf '```\n' >> "$ERRORS_FILE"
+    fi
+    printf '\n</details>\n\n' >> "$ERRORS_FILE"
+  fi
+done
+if [ "$found_errors" = true ]; then
+  echo "  ⚠ Errors found — see $ERRORS_FILE"
+else
+  echo "  No errors found."
+fi
+echo
+
+# ── Step 9: Parse results ───────────────────────────────────────────
+echo "▸ Parsing results..."
+cd "$RETH_REPO"
+
+SUMMARY_ARGS=(
+  --output-summary "$BENCH_WORK_DIR/summary.json"
+  --output-markdown "$BENCH_WORK_DIR/comment.md"
+  --repo "paradigmxyz/reth"
+  --baseline-ref "$BASELINE_SHA"
+  --baseline-name "$BASELINE_REF"
+  --feature-name "$FEATURE_REF"
+  --feature-ref "$FEATURE_SHA"
+  --baseline-csv "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+  --feature-csv "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
+  --gas-csv "$BENCH_WORK_DIR/feature-1/total_gas.csv"
+  --grafana-url "$GRAFANA_URL"
+)
+
+python3 "${SCRIPTS_DIR}/bench-reth-summary.py" "${SUMMARY_ARGS[@]}"
+echo
+
+# ── Step 10: Generate charts ─────────────────────────────────────────
+echo "▸ Generating charts..."
+CHART_ARGS=(
+  --output-dir "$BENCH_WORK_DIR/charts"
+  --feature "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
+  --baseline "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
+  --baseline-name "$BASELINE_REF"
+  --feature-name "$FEATURE_REF"
+)
+
+if python3 -c "import matplotlib" 2>/dev/null; then
+  python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
+elif command -v uv &>/dev/null; then
+  uv run --with matplotlib python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
+else
+  echo "  Warning: matplotlib not available, skipping chart generation."
+fi
+echo
+
+# ── Step 11: Upload Tracy profiles ────────────────────────────────────
+if [ "$TRACY" != "off" ]; then
+  echo "▸ Uploading Tracy profiles..."
+  upload_tracy "baseline-1" "$BENCH_WORK_DIR/baseline-1" "$BASELINE_SHA"
+  upload_tracy "feature-1"  "$BENCH_WORK_DIR/feature-1"  "$FEATURE_SHA"
+  upload_tracy "feature-2"  "$BENCH_WORK_DIR/feature-2"  "$FEATURE_SHA"
+  upload_tracy "baseline-2" "$BENCH_WORK_DIR/baseline-2" "$BASELINE_SHA"
+  echo
+fi
+
+# ── Done (system restore happens via EXIT trap) ─────────────────────
+echo "═══════════════════════════════════════════════════════════"
+echo "  Benchmark complete!"
+echo "═══════════════════════════════════════════════════════════"
+echo "  Results  : $BENCH_WORK_DIR/summary.json"
+echo "  Markdown : $BENCH_WORK_DIR/comment.md"
+echo "  Charts   : $BENCH_WORK_DIR/charts/"
+if [ -f "$ERRORS_FILE" ]; then
+  echo "  Errors   : $ERRORS_FILE"
+fi
+echo "  Grafana  : $GRAFANA_URL"
+if [ "$TRACY" != "off" ]; then
+  echo "  ─── Tracy Profiles ───"
+  for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
+    url_file="$BENCH_WORK_DIR/$run_dir/tracy_viewer_url.txt"
+    if [ -f "$url_file" ]; then
+      echo "  $run_dir : $(cat "$url_file")"
+    fi
+  done
+fi
+echo "═══════════════════════════════════════════════════════════"

.github/scripts/bench-reth-run.sh

@@ -6,18 +6,50 @@
 # Usage: bench-reth-run.sh <label> <binary> <output-dir>
 #
 # Required env: SCHELK_MOUNT, BENCH_RPC_URL, BENCH_BLOCKS, BENCH_WARMUP_BLOCKS
-set -euo pipefail
+# Optional env: BENCH_BIG_BLOCKS (true/false), BENCH_WORK_DIR (for big blocks path)
+#               BENCH_BAL (false/true/feature/baseline; only used with big blocks)
+#               BENCH_WAIT_TIME (duration like 500ms, default empty)
+#               BENCH_BASELINE_ARGS (extra reth node args for baseline runs)
+#               BENCH_FEATURE_ARGS (extra reth node args for feature runs)
+#               BENCH_OTLP_TRACES_ENDPOINT (OTLP HTTP endpoint for traces, e.g. https://host/insert/opentelemetry/v1/traces)
+#               BENCH_OTLP_LOGS_ENDPOINT (OTLP HTTP endpoint for logs, e.g. https://host/insert/opentelemetry/v1/logs)
+#               BENCH_OTLP_DISABLED (true to skip OTLP export even if endpoints are set)
+set -euxo pipefail
 
 LABEL="$1"
 BINARY="$2"
 OUTPUT_DIR="$3"
-DATADIR="$SCHELK_MOUNT/datadir"
+DATADIR_NAME="datadir"
+if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
+  DATADIR_NAME="datadir-big-blocks"
+fi
+DATADIR="$SCHELK_MOUNT/$DATADIR_NAME"
 mkdir -p "$OUTPUT_DIR"
 LOG="${OUTPUT_DIR}/node.log"
 
+RETH_SCOPE="${RETH_SCOPE:-reth-bench.scope}"
+
 cleanup() {
   kill "$TAIL_PID" 2>/dev/null || true
-  if [ -n "${RETH_PID:-}" ] && sudo kill -0 "$RETH_PID" 2>/dev/null; then
+  # Stop tracy-capture first (SIGINT makes it disconnect and flush to disk)
+  # Must happen before killing reth, otherwise reth keeps streaming data.
+  if [ -n "${TRACY_PID:-}" ] && kill -0 "$TRACY_PID" 2>/dev/null; then
+    echo "Stopping tracy-capture..."
+    kill -INT "$TRACY_PID" 2>/dev/null || true
+    for i in $(seq 1 30); do
+      kill -0 "$TRACY_PID" 2>/dev/null || break
+      if [ $((i % 10)) -eq 0 ]; then
+        echo "Waiting for tracy-capture to finish writing... (${i}s)"
+      fi
+      sleep 1
+    done
+    if kill -0 "$TRACY_PID" 2>/dev/null; then
+      echo "tracy-capture still running after 30s, killing..."
+      kill -9 "$TRACY_PID" 2>/dev/null || true
+    fi
+    wait "$TRACY_PID" 2>/dev/null || true
+  fi
+  if sudo systemctl is-active "$RETH_SCOPE" >/dev/null 2>&1; then
     if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
       # Send SIGINT to the inner reth process by exact name (not -f which
       # would also match samply's cmdline containing "reth"). Samply will
@@ -35,28 +67,37 @@ cleanup() {
         echo "Samply still running after 120s, sending SIGTERM..."
         sudo pkill -x samply 2>/dev/null || true
       fi
-    else
-      sudo kill "$RETH_PID"
-      for i in $(seq 1 30); do
-        sudo kill -0 "$RETH_PID" 2>/dev/null || break
-        sleep 1
-      done
     fi
-    sudo kill -9 "$RETH_PID" 2>/dev/null || true
+    # Stop the entire systemd scope — kills all processes in the cgroup.
+    # This is reliable regardless of process reparenting or PID wrapper issues.
+    sudo systemctl stop "$RETH_SCOPE" 2>/dev/null || true
     sleep 1
   fi
+  sudo systemctl reset-failed "$RETH_SCOPE" 2>/dev/null || true
   # Fix ownership of reth-created files (reth runs as root)
   sudo chown -R "$(id -un):$(id -gn)" "$OUTPUT_DIR" 2>/dev/null || true
-  if mountpoint -q "$SCHELK_MOUNT"; then
-    sudo umount -l "$SCHELK_MOUNT" || true
-    sudo schelk recover -y || true
-  fi
+  # Let schelk recover the mounted volume in place so dm-era can restore only
+  # the changed blocks and clean up its own state.
+  sudo schelk recover -y --kill || true
 }
 TAIL_PID=
+TRACY_PID=
 trap cleanup EXIT
 
+# Clean up stale state from a previous cancelled run.
+# Stop any leftover reth process in the scope, then recover schelk state.
+sudo systemctl stop "$RETH_SCOPE" 2>/dev/null || true
+sudo systemctl reset-failed "$RETH_SCOPE" 2>/dev/null || true
+sudo schelk recover -y --kill || sudo schelk full-recover -y || true
+
 # Mount
-sudo schelk mount -y
+sudo schelk mount -y || true
+if [ ! -d "$DATADIR/db" ] || [ ! -d "$DATADIR/static_files" ]; then
+  echo "::error::Failed to mount benchmark datadir at ${DATADIR}"
+  ls -la "$SCHELK_MOUNT" || true
+  ls -la "$DATADIR" || true
+  exit 1
+fi
 sync
 sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
 echo "=== Cache state after drop ==="
@@ -73,6 +114,8 @@ if [ "${BENCH_CORES:-0}" -gt 0 ] && [ "$BENCH_CORES" -lt "$MAX_RETH" ]; then
 fi
 RETH_CPUS="1-${MAX_RETH}"
 
+BIG_BLOCKS="${BENCH_BIG_BLOCKS:-false}"
+
 RETH_ARGS=(
   node
   --datadir "$DATADIR"
@@ -87,20 +130,77 @@ RETH_ARGS=(
   --no-persist-peers
 )
 
+# Gate flag on binary support (older baselines may not have it).
+# Uses --help which exits immediately via clap without node init.
+SYNC_STATE_IDLE=false
+if "$BINARY" node --help 2>/dev/null | grep -qF -- '--debug.startup-sync-state-idle'; then
+  RETH_ARGS+=(--debug.startup-sync-state-idle)
+  SYNC_STATE_IDLE=true
+fi
+
+# Append per-label extra node args (baseline or feature)
+EXTRA_NODE_ARGS=""
+case "$LABEL" in
+  baseline*) EXTRA_NODE_ARGS="${BENCH_BASELINE_ARGS:-}" ;;
+  feature*)  EXTRA_NODE_ARGS="${BENCH_FEATURE_ARGS:-}" ;;
+esac
+if [ -n "$EXTRA_NODE_ARGS" ]; then
+  # Word-split the string into individual args
+  # shellcheck disable=SC2206
+  RETH_ARGS+=($EXTRA_NODE_ARGS)
+fi
+
+if [ -n "${BENCH_METRICS_ADDR:-}" ]; then
+  RETH_ARGS+=(--metrics "$BENCH_METRICS_ADDR")
+fi
+
+# OTLP traces and logs export
+if [ "${BENCH_OTLP_DISABLED:-false}" != "true" ]; then
+  if [ -n "${BENCH_OTLP_TRACES_ENDPOINT:-}" ]; then
+    RETH_ARGS+=(--tracing-otlp="${BENCH_OTLP_TRACES_ENDPOINT}" --tracing-otlp.service-name=reth-bench)
+  fi
+  if [ -n "${BENCH_OTLP_LOGS_ENDPOINT:-}" ]; then
+    RETH_ARGS+=(--logs-otlp="${BENCH_OTLP_LOGS_ENDPOINT}" --logs-otlp.filter=debug)
+  fi
+fi
+
+# Tracy profiling: add --log.tracy flags and set environment
+if [ "${BENCH_TRACY:-off}" != "off" ]; then
+  RETH_ARGS+=(--log.tracy --log.tracy.filter "${BENCH_TRACY_FILTER:-debug}")
+  if [ "${BENCH_TRACY}" = "on" ]; then
+    export TRACY_NO_SYS_TRACE=1
+  elif [ "${BENCH_TRACY}" = "full" ]; then
+    export TRACY_SAMPLING_HZ="${BENCH_TRACY_SAMPLING_HZ:-1}"
+  fi
+fi
+
+SUDO_ENV=()
+if [ -n "${OTEL_RESOURCE_ATTRIBUTES:-}" ]; then
+  SUDO_ENV+=("OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES}")
+  SUDO_ENV+=("OTEL_BSP_MAX_QUEUE_SIZE=65536" "OTEL_BLRP_MAX_QUEUE_SIZE=65536")
+fi
+
+# Limit reth memory to 95% of available RAM to prevent OOM kills
+TOTAL_MEM_KB=$(awk '/^MemTotal:/ {print $2}' /proc/meminfo)
+MEM_LIMIT=$(( TOTAL_MEM_KB * 95 / 100 * 1024 ))
+echo "Memory limit: $(( MEM_LIMIT / 1024 / 1024 ))MB (95% of $(( TOTAL_MEM_KB / 1024 ))MB)"
+
 if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
   RETH_ARGS+=(--log.samply)
   SAMPLY="$(which samply)"
-  sudo taskset -c "$RETH_CPUS" nice -n -20 \
+  sudo systemd-run --quiet --scope --collect --unit="$RETH_SCOPE" \
+    -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
+    env "${SUDO_ENV[@]}" nice -n -20 \
     "$SAMPLY" record --save-only --presymbolicate --rate 10000 \
     --output "$OUTPUT_DIR/samply-profile.json.gz" \
     -- "$BINARY" "${RETH_ARGS[@]}" \
     > "$LOG" 2>&1 &
 else
-  sudo taskset -c "$RETH_CPUS" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
+  sudo systemd-run --quiet --scope --collect --unit="$RETH_SCOPE" \
+    -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
+    env "${SUDO_ENV[@]}" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
     > "$LOG" 2>&1 &
 fi
-
-RETH_PID=$!
 stdbuf -oL tail -f "$LOG" | sed -u "s/^/[reth] /" &
 TAIL_PID=$!
 
@@ -109,7 +209,7 @@ for i in $(seq 1 60); do
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
     > /dev/null 2>&1; then
-    echo "reth (${LABEL}) is ready after ${i}s"
+    echo "reth (${LABEL}) RPC is up after ${i}s"
     break
   fi
   if [ "$i" -eq 60 ]; then
@@ -120,25 +220,138 @@ for i in $(seq 1 60); do
   sleep 1
 done
 
+# Wait for the pipeline to finish (eth_syncing returns false) so the
+# engine is in live mode and can accept newPayload calls.
+# Only possible when --debug.startup-sync-state-idle is supported.
+if [ "$SYNC_STATE_IDLE" = "true" ]; then
+  for i in $(seq 1 300); do
+    SYNC_RESULT=$(curl -sf http://127.0.0.1:8545 -X POST \
+      -H 'Content-Type: application/json' \
+      -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' 2>/dev/null || true)
+    if [ -n "$SYNC_RESULT" ] && jq -e '.result == false' <<< "$SYNC_RESULT" > /dev/null 2>&1; then
+      echo "reth (${LABEL}) pipeline finished after ${i}s, engine is live"
+      break
+    fi
+    if [ "$i" -eq 300 ]; then
+      echo "::error::reth (${LABEL}) pipeline did not finish within 300s"
+      cat "$LOG"
+      exit 1
+    fi
+    sleep 1
+  done
+else
+  echo "reth (${LABEL}) binary does not support --debug.startup-sync-state-idle, skipping sync wait"
+fi
+
 # Run reth-bench with high priority but as the current user so output
 # files are not root-owned (avoids EACCES on next checkout).
 BENCH_NICE="sudo nice -n -20 sudo -u $(id -un)"
 
-# Warmup
-$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
-  --rpc-url "$BENCH_RPC_URL" \
-  --engine-rpc-url http://127.0.0.1:8551 \
-  --jwt-secret "$DATADIR/jwt.hex" \
-  --advance "${BENCH_WARMUP_BLOCKS:-50}" \
-  --reth-new-payload 2>&1 | sed -u "s/^/[bench] /"
+# Build optional flags
+EXTRA_BENCH_ARGS=(--reth-new-payload)
+if [ -n "${BENCH_WAIT_TIME:-}" ]; then
+  EXTRA_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
+fi
 
-# Benchmark
-$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
-  --rpc-url "$BENCH_RPC_URL" \
-  --engine-rpc-url http://127.0.0.1:8551 \
-  --jwt-secret "$DATADIR/jwt.hex" \
-  --advance "$BENCH_BLOCKS" \
-  --reth-new-payload \
-  --output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
+if [ "$BIG_BLOCKS" = "true" ]; then
+  # Big blocks mode: replay pre-generated payloads
+  BIG_BLOCKS_DIR="${BENCH_BIG_BLOCKS_DIR:-${BENCH_WORK_DIR}/big-blocks}"
+  BENCH_BAL_MODE="${BENCH_BAL:-false}"
+
+  BB_BENCH_ARGS=(--reth-new-payload)
+  if [ -n "${BENCH_WAIT_TIME:-}" ]; then
+    BB_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
+  fi
+  case "$BENCH_BAL_MODE" in
+    false)
+      ;;
+    true)
+      BB_BENCH_ARGS+=(--bal)
+      ;;
+    baseline)
+      if [[ "$LABEL" == baseline* ]]; then
+        BB_BENCH_ARGS+=(--bal)
+      fi
+      ;;
+    feature)
+      if [[ "$LABEL" == feature* ]]; then
+        BB_BENCH_ARGS+=(--bal)
+      fi
+      ;;
+    *)
+      echo "::error::Unknown BENCH_BAL value: $BENCH_BAL_MODE"
+      exit 1
+      ;;
+  esac
+
+  # Warmup
+  WARMUP="${BENCH_WARMUP_BLOCKS:-50}"
+  if [ "$WARMUP" -gt 0 ] 2>/dev/null; then
+    echo "Running big blocks warmup (${WARMUP} payloads)..."
+    $BENCH_NICE "$RETH_BENCH" replay-payloads \
+      "${BB_BENCH_ARGS[@]}" \
+      --count "$WARMUP" \
+      --payload-dir "$BIG_BLOCKS_DIR/payloads" \
+      --engine-rpc-url http://127.0.0.1:8551 \
+      --jwt-secret "$DATADIR/jwt.hex" 2>&1 | sed -u "s/^/[bench] /"
+  fi
+
+  # Start tracy-capture after warmup so profile only covers the benchmark
+  if [ "${BENCH_TRACY:-off}" != "off" ]; then
+    echo "Starting tracy-capture..."
+    tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
+    TRACY_PID=$!
+    sleep 0.5  # give tracy-capture time to connect
+  fi
+
+  # Benchmark — skip warmup payloads so they aren't measured
+  BB_SKIP=0
+  if [ "$WARMUP" -gt 0 ] 2>/dev/null; then
+    BB_SKIP="$WARMUP"
+  fi
+  if [ "${BENCH_BLOCKS:-0}" -gt 0 ] 2>/dev/null; then
+    BB_BENCH_ARGS+=(--count "$BENCH_BLOCKS")
+  fi
+
+  echo "Running big blocks benchmark (replay-payloads, skip=${BB_SKIP})..."
+  $BENCH_NICE "$RETH_BENCH" replay-payloads \
+    "${BB_BENCH_ARGS[@]}" \
+    --skip "$BB_SKIP" \
+    --payload-dir "$BIG_BLOCKS_DIR/payloads" \
+    --engine-rpc-url http://127.0.0.1:8551 \
+    --jwt-secret "$DATADIR/jwt.hex" \
+    --output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
+else
+  # Standard mode: warmup + new-payload-fcu
+  WARMUP="${BENCH_WARMUP_BLOCKS:-50}"
+  if [ "$WARMUP" -gt 0 ] 2>/dev/null; then
+    # Warm up the node before measuring the benchmark window.
+    $BENCH_NICE "$RETH_BENCH" new-payload-fcu \
+      --rpc-url "$BENCH_RPC_URL" \
+      --engine-rpc-url http://127.0.0.1:8551 \
+      --jwt-secret "$DATADIR/jwt.hex" \
+      --advance "$WARMUP" \
+      "${EXTRA_BENCH_ARGS[@]}" 2>&1 | sed -u "s/^/[bench] /"
+  else
+    echo "Skipping warmup (0 blocks)..."
+  fi
+
+  # Start tracy-capture after warmup so profile only covers the benchmark
+  if [ "${BENCH_TRACY:-off}" != "off" ]; then
+    echo "Starting tracy-capture..."
+    tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
+    TRACY_PID=$!
+    sleep 0.5  # give tracy-capture time to connect
+  fi
+
+  # Benchmark
+  $BENCH_NICE "$RETH_BENCH" new-payload-fcu \
+    --rpc-url "$BENCH_RPC_URL" \
+    --engine-rpc-url http://127.0.0.1:8551 \
+    --jwt-secret "$DATADIR/jwt.hex" \
+    --advance "$BENCH_BLOCKS" \
+    "${EXTRA_BENCH_ARGS[@]}" \
+    --output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
+fi
 
 # cleanup runs via trap

.github/scripts/bench-reth-snapshot.sh

@@ -1,127 +1,56 @@
 #!/usr/bin/env bash
 #
-# Downloads the latest nightly snapshot into the schelk volume with
-# progress reporting to the GitHub PR comment.
-#
-# Skips the download if the local ETag marker matches the remote one.
+# Validates that the benchmark snapshot has already been populated into the
+# local schelk volume.
 #
 # Usage: bench-reth-snapshot.sh [--check]
-#   --check   Only check if a download is needed; exits 0 if up-to-date, 1 if not.
+#   --check   Exit 0 if the local snapshot is ready, 10 if it is missing.
 #
 # Required env:
-#   SCHELK_MOUNT       – schelk mount point (e.g. /reth-bench)
-#   GITHUB_TOKEN       – token for GitHub API calls (only for download)
-#   BENCH_COMMENT_ID   – PR comment ID to update (optional)
-#   BENCH_REPO         – owner/repo (e.g. paradigmxyz/reth)
-#   BENCH_JOB_URL      – link to the Actions job
-#   BENCH_ACTOR        – user who triggered the benchmark
-#   BENCH_CONFIG       – config summary line
-set -euo pipefail
+#   SCHELK_MOUNT – schelk mount point (e.g. /reth-bench)
+# Optional env:
+#   BENCH_BIG_BLOCKS   – true when validating the big-blocks snapshot datadir
+#   BENCH_SNAPSHOT_NAME – expected snapshot label for log/error output
+set -euxo pipefail
 
-BUCKET="minio/reth-snapshots/reth-1-minimal-nightly-previous.tar.zst"
-DATADIR="$SCHELK_MOUNT/datadir"
-ETAG_FILE="$HOME/.reth-bench-snapshot-etag"
+: "${SCHELK_MOUNT:?SCHELK_MOUNT must be set}"
 
-# Get remote metadata via JSON for reliable parsing
-MC_STAT=$(mc stat --json "$BUCKET" 2>/dev/null || true)
-REMOTE_ETAG=$(echo "$MC_STAT" | jq -r '.etag // empty')
-if [ -z "$REMOTE_ETAG" ]; then
-  echo "::warning::Failed to get ETag from mc stat, will re-download"
-  REMOTE_ETAG="unknown-$(date +%s)"
+DATADIR_NAME="datadir"
+if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
+  DATADIR_NAME="datadir-big-blocks"
 fi
+DATADIR="$SCHELK_MOUNT/$DATADIR_NAME"
 
-LOCAL_ETAG=""
-[ -f "$ETAG_FILE" ] && LOCAL_ETAG=$(cat "$ETAG_FILE")
-
-if [ "$REMOTE_ETAG" = "$LOCAL_ETAG" ]; then
-  echo "Snapshot is up-to-date (ETag: ${REMOTE_ETAG})"
-  if [ "${1:-}" = "--check" ]; then
-    exit 0
+describe_snapshot() {
+  if [ -n "${BENCH_SNAPSHOT_NAME:-}" ]; then
+    printf '%s' "${BENCH_SNAPSHOT_NAME}"
+  elif [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
+    printf '%s' 'big-block weekly snapshot'
+  else
+    printf '%s' 'benchmark snapshot'
   fi
+}
+
+snapshot_ready() {
+  [ -d "$DATADIR/db" ] && [ -d "$DATADIR/static_files" ]
+}
+
+EXPECTED_SNAPSHOT="$(describe_snapshot)"
+
+sudo schelk recover -y --kill || sudo schelk full-recover -y || true
+sudo schelk mount -y || true
+
+if snapshot_ready; then
+  echo "Found local ${EXPECTED_SNAPSHOT} at ${DATADIR}"
   exit 0
 fi
 
-echo "Snapshot needs update (local: ${LOCAL_ETAG:-<none>}, remote: ${REMOTE_ETAG})"
+echo "::error::Missing local ${EXPECTED_SNAPSHOT} at ${DATADIR}. Benchmarks no longer download snapshots; pre-populate the local schelk data first."
+ls -la "$SCHELK_MOUNT" || true
+ls -la "$DATADIR" || true
+
 if [ "${1:-}" = "--check" ]; then
-  exit 1
+  exit 10
 fi
 
-# Get compressed size for progress tracking
-TOTAL_BYTES=$(echo "$MC_STAT" | jq -r '.size // empty')
-if [ -z "$TOTAL_BYTES" ] || [ "$TOTAL_BYTES" = "0" ]; then
-  echo "::error::Failed to get snapshot size from mc stat"
-  exit 1
-fi
-echo "Snapshot size: $TOTAL_BYTES bytes ($(numfmt --to=iec "$TOTAL_BYTES"))"
-
-# Prepare mount
-mountpoint -q "$SCHELK_MOUNT" && sudo schelk recover -y || true
-sudo schelk mount -y
-sudo rm -rf "$DATADIR"
-sudo mkdir -p "$DATADIR"
-
-update_comment() {
-  local pct="$1"
-  [ -z "${BENCH_COMMENT_ID:-}" ] && return 0
-  local status="Building binaries & downloading snapshot… ${pct}%"
-  local body
-  body="$(printf 'cc @%s\n\n🚀 Benchmark started! [View job](%s)\n\n⏳ **Status:** %s\n\n%s' \
-    "$BENCH_ACTOR" "$BENCH_JOB_URL" "$status" "$BENCH_CONFIG")"
-  curl -sf -X PATCH \
-    -H "Authorization: token ${GITHUB_TOKEN}" \
-    -H "Accept: application/vnd.github.v3+json" \
-    "https://api.github.com/repos/${BENCH_REPO}/issues/comments/${BENCH_COMMENT_ID}" \
-    -d "$(jq -nc --arg body "$body" '{body: $body}')" \
-    > /dev/null 2>&1 || true
-}
-
-# Track compressed bytes flowing through the pipe
-DL_BYTES_FILE=$(mktemp)
-echo 0 > "$DL_BYTES_FILE"
-
-# Start progress reporter in background
-(
-  while true; do
-    sleep 10
-    CURRENT=$(cat "$DL_BYTES_FILE" 2>/dev/null || echo 0)
-    if [ "$TOTAL_BYTES" -gt 0 ]; then
-      PCT=$(( CURRENT * 100 / TOTAL_BYTES ))
-      [ "$PCT" -gt 100 ] && PCT=100
-      echo "Snapshot download: $(numfmt --to=iec "$CURRENT") / $(numfmt --to=iec "$TOTAL_BYTES") (${PCT}%)"
-      update_comment "$PCT"
-    fi
-  done
-) &
-PROGRESS_PID=$!
-trap 'kill $PROGRESS_PID 2>/dev/null || true; rm -f "$DL_BYTES_FILE"' EXIT
-
-# Download and extract; python byte counter tracks compressed bytes received
-mc cat "$BUCKET" | python3 -c "
-import sys
-count = 0
-while True:
-    data = sys.stdin.buffer.read(1048576)
-    if not data:
-        break
-    count += len(data)
-    sys.stdout.buffer.write(data)
-    with open('$DL_BYTES_FILE', 'w') as f:
-        f.write(str(count))
-" | pzstd -d -p 6 | sudo tar -xf - -C "$DATADIR"
-
-# Stop progress reporter
-kill $PROGRESS_PID 2>/dev/null || true
-wait $PROGRESS_PID 2>/dev/null || true
-
-update_comment "100"
-echo "Snapshot download complete"
-
-# Promote the new snapshot to become the schelk baseline (virgin volume).
-# This copies changed blocks from scratch → virgin so that future
-# `schelk recover` calls restore to this new state.
-sync
-sudo schelk promote -y
-
-# Save ETag marker
-echo "$REMOTE_ETAG" > "$ETAG_FILE"
-echo "Snapshot promoted to schelk baseline (ETag: ${REMOTE_ETAG})"
+exit 1

.github/scripts/bench-reth-summary.py

@@ -107,6 +107,18 @@ def compute_stats(combined: list[dict]) -> dict:
             mgas_s_values.append(r["gas_used"] / lat_s / 1_000_000)
     mean_mgas_s = sum(mgas_s_values) / len(mgas_s_values) if mgas_s_values else 0
 
+    total_latencies_ms = [r["total_latency_us"] / 1_000 for r in combined]
+    wall_clock_s = sum(total_latencies_ms) / 1_000
+    mean_total_lat_ms = sum(total_latencies_ms) / n
+
+    # Persistence wait mean (for main table)
+    persist_values_ms = []
+    for r in combined:
+        v = r.get("persistence_wait_us")
+        if v is not None:
+            persist_values_ms.append(v / 1_000)
+    mean_persist_ms = sum(persist_values_ms) / len(persist_values_ms) if persist_values_ms else 0.0
+
     return {
         "n": n,
         "mean_ms": mean_lat,
@@ -115,6 +127,9 @@ def compute_stats(combined: list[dict]) -> dict:
         "p90_ms": percentile(sorted_lat, 90),
         "p99_ms": percentile(sorted_lat, 99),
         "mean_mgas_s": mean_mgas_s,
+        "wall_clock_s": wall_clock_s,
+        "mean_total_lat_ms": mean_total_lat_ms,
+        "mean_persist_ms": mean_persist_ms,
     }
 
 
@@ -139,13 +154,15 @@ def compute_wait_stats(combined: list[dict], field: str) -> dict:
 
 def _paired_data(
     baseline: list[dict], feature: list[dict]
-) -> tuple[list[tuple[float, float]], list[float], list[float]]:
+) -> tuple[list[tuple[float, float]], list[float], list[float], list[float], list[float]]:
     """Match blocks and return paired latencies and per-block diffs.
 
     Returns:
         pairs: list of (baseline_ms, feature_ms) tuples
         lat_diffs_ms: list of feature − baseline latency diffs in ms
         mgas_diffs: list of feature − baseline Mgas/s diffs
+        total_lat_diffs_ms: list of feature − baseline total latency diffs in ms
+        persist_diffs_ms: list of feature − baseline persistence wait diffs in ms
     """
     baseline_by_block = {r["block_number"]: r for r in baseline}
     feature_by_block = {r["block_number"]: r for r in feature}
@@ -154,6 +171,8 @@ def _paired_data(
     pairs = []
     lat_diffs_ms = []
     mgas_diffs = []
+    total_lat_diffs_ms = []
+    persist_diffs_ms = []
     for bn in common_blocks:
         b = baseline_by_block[bn]
         f = feature_by_block[bn]
@@ -168,7 +187,13 @@ def _paired_data(
                 f["gas_used"] / f_lat_s / 1_000_000
                 - b["gas_used"] / b_lat_s / 1_000_000
             )
-    return pairs, lat_diffs_ms, mgas_diffs
+        total_lat_diffs_ms.append(
+            f["total_latency_us"] / 1_000 - b["total_latency_us"] / 1_000
+        )
+        b_persist = (b.get("persistence_wait_us") or 0) / 1_000
+        f_persist = (f.get("persistence_wait_us") or 0) / 1_000
+        persist_diffs_ms.append(f_persist - b_persist)
+    return pairs, lat_diffs_ms, mgas_diffs, total_lat_diffs_ms, persist_diffs_ms
 
 
 def compute_paired_stats(
@@ -183,12 +208,16 @@ def compute_paired_stats(
     all_pairs = []
     all_lat_diffs = []
     all_mgas_diffs = []
+    all_total_lat_diffs = []
+    all_persist_diffs = []
     blocks_per_pair = []
     for baseline, feature in zip(baseline_runs, feature_runs):
-        pairs, lat_diffs, mgas_diffs = _paired_data(baseline, feature)
+        pairs, lat_diffs, mgas_diffs, total_lat_diffs, persist_diffs = _paired_data(baseline, feature)
         all_pairs.extend(pairs)
         all_lat_diffs.extend(lat_diffs)
         all_mgas_diffs.extend(mgas_diffs)
+        all_total_lat_diffs.extend(total_lat_diffs)
+        all_persist_diffs.extend(persist_diffs)
         blocks_per_pair.append(len(pairs))
 
     if not all_lat_diffs:
@@ -227,6 +256,16 @@ def compute_paired_stats(
     mgas_se = std_mgas_diff / math.sqrt(len(all_mgas_diffs)) if all_mgas_diffs else 0.0
     mgas_ci = T_CRITICAL * mgas_se
 
+    mean_total_diff = sum(all_total_lat_diffs) / len(all_total_lat_diffs) if all_total_lat_diffs else 0.0
+    std_total_diff = stddev(all_total_lat_diffs, mean_total_diff) if len(all_total_lat_diffs) > 1 else 0.0
+    total_se = std_total_diff / math.sqrt(len(all_total_lat_diffs)) if all_total_lat_diffs else 0.0
+    wall_clock_ci_ms = T_CRITICAL * total_se
+
+    mean_persist_diff = sum(all_persist_diffs) / len(all_persist_diffs) if all_persist_diffs else 0.0
+    std_persist_diff = stddev(all_persist_diffs, mean_persist_diff) if len(all_persist_diffs) > 1 else 0.0
+    persist_se = std_persist_diff / math.sqrt(len(all_persist_diffs)) if all_persist_diffs else 0.0
+    persist_ci_ms = T_CRITICAL * persist_se
+
     return {
         "n": n,
         "mean_diff_ms": mean_diff,
@@ -239,6 +278,8 @@ def compute_paired_stats(
         "p99_ci_ms": (p99_boot[hi] - p99_boot[lo]) / 2,
         "mean_mgas_diff": mean_mgas_diff,
         "mgas_ci": mgas_ci,
+        "wall_clock_ci_ms": wall_clock_ci_ms,
+        "persist_ci_ms": persist_ci_ms,
         "blocks": max(blocks_per_pair),
     }
 
@@ -267,6 +308,18 @@ def fmt_mgas(v: float) -> str:
     return f"{v:.2f}"
 
 
+def fmt_s(v: float) -> str:
+    return f"{v:.2f}s"
+
+
+def display_bal_mode(bal_mode: str | None) -> str | None:
+    if not bal_mode or bal_mode == "false":
+        return None
+    if bal_mode == "both":
+        return "true"
+    return bal_mode
+
+
 def significance(pct: float, ci_pct: float, lower_is_better: bool) -> str:
     """Return significance label: 'good', 'bad', or 'neutral'."""
     significant = abs(pct) > ci_pct
@@ -304,6 +357,8 @@ def compute_changes(
         ("p90", "p90_ms", "p90_ci_ms", "p90_ms", True),
         ("p99", "p99_ms", "p99_ci_ms", "p99_ms", True),
         ("mgas_s", "mean_mgas_s", "mgas_ci", "mean_mgas_s", False),
+        ("wall_clock", "wall_clock_s", "wall_clock_ci_ms", "mean_total_lat_ms", True),
+        ("persist_wait", "mean_persist_ms", "persist_ci_ms", "mean_persist_ms", True),
     ]
     changes = {}
     for name, stat_key, ci_key, base_key, lower_is_better in metrics:
@@ -326,6 +381,10 @@ def generate_comparison_table(
     baseline_name: str,
     feature_name: str,
     feature_sha: str,
+    big_blocks: bool = False,
+    warmup_blocks: str | None = None,
+    wait_time: str | None = None,
+    bal_mode: str | None = None,
 ) -> str:
     """Generate a markdown comparison table between baseline and feature."""
     n = paired["blocks"]
@@ -335,11 +394,14 @@ def generate_comparison_table(
 
     mean_pct = pct(run1["mean_ms"], run2["mean_ms"])
     gas_pct = pct(run1["mean_mgas_s"], run2["mean_mgas_s"])
+    wall_pct = pct(run1["wall_clock_s"], run2["wall_clock_s"])
 
     p50_pct = pct(run1["p50_ms"], run2["p50_ms"])
     p90_pct = pct(run1["p90_ms"], run2["p90_ms"])
     p99_pct = pct(run1["p99_ms"], run2["p99_ms"])
 
+    persist_pct = pct(run1["mean_persist_ms"], run2["mean_persist_ms"])
+
     # Bootstrap CIs as % of baseline percentile
     p50_ci_pct = paired["p50_ci_ms"] / run1["p50_ms"] * 100.0 if run1["p50_ms"] > 0 else 0.0
     p90_ci_pct = paired["p90_ci_ms"] / run1["p90_ms"] * 100.0 if run1["p90_ms"] > 0 else 0.0
@@ -348,6 +410,8 @@ def generate_comparison_table(
     # CI as a percentage of baseline mean
     lat_ci_pct = paired["ci_ms"] / run1["mean_ms"] * 100.0 if run1["mean_ms"] > 0 else 0.0
     mgas_ci_pct = paired["mgas_ci"] / run1["mean_mgas_s"] * 100.0 if run1["mean_mgas_s"] > 0 else 0.0
+    wall_ci_pct = paired["wall_clock_ci_ms"] / run1["mean_total_lat_ms"] * 100.0 if run1["mean_total_lat_ms"] > 0 else 0.0
+    persist_ci_pct = paired["persist_ci_ms"] / run1["mean_persist_ms"] * 100.0 if run1["mean_persist_ms"] > 0 else 0.0
 
     base_url = f"https://github.com/{repo}/commit"
     baseline_label = f"[`{baseline_name}`]({base_url}/{baseline_ref})"
@@ -362,9 +426,19 @@ def generate_comparison_table(
         f"| P90 | {fmt_ms(run1['p90_ms'])} | {fmt_ms(run2['p90_ms'])} | {change_str(p90_pct, p90_ci_pct, lower_is_better=True)} |",
         f"| P99 | {fmt_ms(run1['p99_ms'])} | {fmt_ms(run2['p99_ms'])} | {change_str(p99_pct, p99_ci_pct, lower_is_better=True)} |",
         f"| Mgas/s | {fmt_mgas(run1['mean_mgas_s'])} | {fmt_mgas(run2['mean_mgas_s'])} | {change_str(gas_pct, mgas_ci_pct, lower_is_better=False)} |",
+        f"| Wall Clock | {fmt_s(run1['wall_clock_s'])} | {fmt_s(run2['wall_clock_s'])} | {change_str(wall_pct, wall_ci_pct, lower_is_better=True)} |",
+        f"| Persist Wait | {fmt_ms(run1['mean_persist_ms'])} | {fmt_ms(run2['mean_persist_ms'])} | {change_str(persist_pct, persist_ci_pct, lower_is_better=True)} |",
         "",
-        f"*{n} blocks*",
     ]
+    meta_parts = [f"{n} {'big blocks' if big_blocks else 'blocks'}"]
+    if warmup_blocks:
+        meta_parts.append(f"{warmup_blocks} warmup")
+    if wait_time:
+        meta_parts.append(f"wait time: {wait_time}")
+    display_mode = display_bal_mode(bal_mode)
+    if big_blocks and display_mode:
+        meta_parts.append(f"BAL: {display_mode}")
+    lines.append(f"*{', '.join(meta_parts)}*")
     return "\n".join(lines)
 
 
@@ -394,6 +468,7 @@ def generate_markdown(
     summary: dict, comparison_table: str,
     wait_time_tables: list[str] | None = None,
     behind_baseline: int = 0, repo: str = "", baseline_ref: str = "", baseline_name: str = "",
+    grafana_url: str | None = None,
 ) -> str:
     """Generate a markdown comment body."""
     lines = ["## Benchmark Results", ""]
@@ -413,6 +488,9 @@ def generate_markdown(
                 lines.append(table)
                 lines.append("")
         lines.append("</details>")
+    if grafana_url:
+        lines.append("")
+        lines.append(f"**[Grafana Dashboard]({grafana_url})**")
     return "\n".join(lines)
 
 
@@ -439,6 +517,11 @@ def main():
     parser.add_argument("--feature-name", "--branch-name", default=None, help="Feature branch name")
     parser.add_argument("--feature-ref", "--branch-sha", "--feature-sha", default=None, help="Feature commit SHA")
     parser.add_argument("--behind-baseline", "--behind-main", type=int, default=0, help="Commits behind baseline")
+    parser.add_argument("--big-blocks", action="store_true", default=False, help="Big blocks mode")
+    parser.add_argument("--warmup-blocks", default=None, help="Number of warmup blocks")
+    parser.add_argument("--wait-time", default=None, help="Wait time interval used between blocks")
+    parser.add_argument("--bal-mode", default=None, help="BAL mode (true, feature, baseline)")
+    parser.add_argument("--grafana-url", default=None, help="Grafana dashboard URL for this benchmark run")
     args = parser.parse_args()
 
     if len(args.baseline_csv) != len(args.feature_csv):
@@ -477,6 +560,7 @@ def main():
     baseline_name = args.baseline_name or "baseline"
     feature_name = args.feature_name or "feature"
     feature_sha = args.feature_ref or "unknown"
+    bal_mode = display_bal_mode(args.bal_mode)
 
     comparison_table = generate_comparison_table(
         baseline_stats,
@@ -487,6 +571,10 @@ def main():
         baseline_name=baseline_name,
         feature_name=feature_name,
         feature_sha=feature_sha,
+        big_blocks=args.big_blocks,
+        warmup_blocks=args.warmup_blocks,
+        wait_time=args.wait_time,
+        bal_mode=bal_mode,
     )
     print(f"Generated comparison ({paired_stats['n']} paired blocks, "
           f"mean diff {paired_stats['mean_diff_ms']:+.3f}ms ± {paired_stats['ci_ms']:.3f}ms)")
@@ -517,6 +605,10 @@ def main():
 
     summary = {
         "blocks": paired_stats["blocks"],
+        "big_blocks": args.big_blocks,
+        "warmup_blocks": args.warmup_blocks,
+        "wait_time": args.wait_time,
+        "bal_mode": bal_mode,
         "baseline": {
             "name": baseline_name,
             "ref": baseline_ref,
@@ -542,6 +634,7 @@ def main():
         repo=args.repo,
         baseline_ref=baseline_ref,
         baseline_name=baseline_name,
+        grafana_url=args.grafana_url,
     )
 
     with open(args.output_markdown, "w") as f:

.github/scripts/bench-scheduled-refs.sh

@@ -0,0 +1,336 @@
+#!/usr/bin/env bash
+#
+# Resolves baseline and feature refs for scheduled benchmark runs.
+#
+# Supports three modes:
+#   nightly — Queries the latest successful scheduled docker.yml run via
+#             GitHub API to find the nightly Docker image commit. Compares
+#             with the last successful feature ref to detect staleness.
+#   hourly  — Compares origin/main HEAD against the last successfully
+#             benchmarked commit (falls back to HEAD~1 on first run).
+#             Checks for in-progress sibling runs to avoid overlap.
+#   release — Compares the latest GitHub release tag against the current
+#             nightly Docker build. Baseline is the release tag commit,
+#             feature is the nightly commit.
+#
+# Usage: bench-scheduled-refs.sh <force> <mode>
+#   force — "true" to run even if no new commit (bypass skip logic)
+#   mode  — "nightly", "hourly", or "release"
+#
+# Outputs (via GITHUB_OUTPUT):
+#   baseline-ref    — commit SHA for baseline
+#   feature-ref     — commit SHA for feature
+#   should-skip     — "true" if no new commit since last run or sibling in progress
+#   is-stale        — "true" if latest nightly build is >24h old (nightly only)
+#   stale-age-hours — age of the nightly build in hours (nightly only)
+#   nightly-created — ISO timestamp of the nightly build (nightly only)
+#   release-tag     — release tag name (release mode only, e.g. "v2.0.0")
+#
+# Reads:
+#   state/nightly-last-feature-ref  (nightly, from decofe/reth-bench-charts repo)
+#   state/hourly-last-feature-ref   (hourly, from decofe/reth-bench-charts repo)
+#   state/release-last-feature-ref  (release, from decofe/reth-bench-charts repo)
+#
+# Requires: gh (GitHub CLI), jq, date, git (hourly mode), curl, DEREK_TOKEN env
+set -euxo pipefail
+
+FORCE="${1:-false}"
+MODE="${2:-nightly}"
+REPO="${GITHUB_REPOSITORY:-paradigmxyz/reth}"
+
+echo "Mode: $MODE, Force: $FORCE"
+
+# ==========================================================================
+# Hourly mode: compare origin/main HEAD vs HEAD~1
+# ==========================================================================
+if [ "$MODE" = "hourly" ]; then
+
+  # --- Step 1: Resolve feature ref from git ---
+  echo "::group::Resolving hourly refs from git"
+  git fetch origin main --depth=2 --quiet
+  FEATURE_REF=$(git rev-parse origin/main)
+  echo "Feature (HEAD): $FEATURE_REF"
+  echo "::endgroup::"
+
+  # --- Step 2: Check for in-progress sibling runs ---
+  echo "::group::Checking for in-progress sibling runs"
+  CURRENT_RUN_ID="${GITHUB_RUN_ID:-0}"
+  IN_PROGRESS=$(gh run list \
+    -R "$REPO" \
+    --workflow=bench-scheduled.yml \
+    --status=in_progress \
+    --json databaseId \
+    --jq "[.[] | select(.databaseId != $CURRENT_RUN_ID)] | length")
+
+  SHOULD_SKIP="false"
+  if [ "$IN_PROGRESS" -gt 0 ]; then
+    echo "::warning::Previous bench run still in progress ($IN_PROGRESS sibling run(s) found). Skipping."
+    SHOULD_SKIP="true"
+    # Output a flag so the workflow can send a Slack alert
+    echo "long-running=true" >> "$GITHUB_OUTPUT"
+  else
+    echo "No in-progress sibling runs"
+    echo "long-running=false" >> "$GITHUB_OUTPUT"
+  fi
+  echo "::endgroup::"
+
+  # --- Step 3: Read last successful feature ref from charts repo ---
+  echo "::group::Reading persisted state"
+  LAST_FEATURE_REF=""
+  STATE_URL="https://raw.githubusercontent.com/decofe/reth-bench-charts/state/state/hourly-last-feature-ref"
+  if RAW=$(curl -sfL -H "Authorization: token ${DEREK_TOKEN}" "$STATE_URL"); then
+    LAST_FEATURE_REF=$(echo "$RAW" | tr -d '[:space:]')
+    echo "Previous feature ref: $LAST_FEATURE_REF"
+  else
+    echo "No persisted state found (first run)"
+  fi
+  echo "::endgroup::"
+
+  # --- Step 4: Determine baseline and skip logic ---
+  echo "::group::Resolving baseline and skip logic"
+  if [ "$SHOULD_SKIP" = "true" ]; then
+    BASELINE_REF=$(git rev-parse origin/main~1)
+    echo "Already marked skip (sibling in progress)"
+  elif [ -z "$LAST_FEATURE_REF" ]; then
+    # First run: no previous state, fall back to HEAD~1
+    BASELINE_REF=$(git rev-parse origin/main~1)
+    echo "First run — using HEAD~1 as baseline"
+  elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+    BASELINE_REF="$LAST_FEATURE_REF"
+    if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+      echo "No new commits on main, but force=true — running anyway"
+    else
+      SHOULD_SKIP="true"
+      echo "No new commits on main since last run — will skip"
+    fi
+  else
+    # Normal case: use last benchmarked commit as baseline
+    BASELINE_REF="$LAST_FEATURE_REF"
+    echo "New commit(s) on main detected — comparing against last benchmarked commit"
+  fi
+
+  echo "Baseline: $BASELINE_REF"
+  echo "Feature:  $FEATURE_REF"
+  echo "Skip:     $SHOULD_SKIP"
+  echo "::endgroup::"
+
+  # --- Step 5: Write outputs ---
+  {
+    echo "baseline-ref=$BASELINE_REF"
+    echo "feature-ref=$FEATURE_REF"
+    echo "should-skip=$SHOULD_SKIP"
+    echo "is-stale=false"
+    echo "stale-age-hours=0"
+    echo "nightly-created="
+  } >> "$GITHUB_OUTPUT"
+  exit 0
+fi
+
+# ==========================================================================
+# Release mode: compare latest GitHub release tag vs current nightly build
+# ==========================================================================
+if [ "$MODE" = "release" ]; then
+
+  # --- Step 1: Resolve feature ref from latest nightly Docker build ---
+  echo "::group::Querying latest nightly docker build"
+  RUNS_JSON=$(gh run list \
+    -R "$REPO" \
+    --workflow=docker.yml \
+    --event=schedule \
+    --status=completed \
+    --limit 5 \
+    --json headSha,createdAt,conclusion)
+
+  LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
+  if [ -z "$LATEST" ]; then
+    echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
+    exit 1
+  fi
+
+  FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
+  echo "Nightly commit (feature): $FEATURE_REF"
+  echo "::endgroup::"
+
+  # --- Step 2: Resolve baseline ref from latest GitHub release ---
+  echo "::group::Resolving latest release tag"
+  RELEASE_JSON=$(gh release view --repo "$REPO" --json tagName,targetCommitish,publishedAt 2>/dev/null || echo "{}")
+  RELEASE_TAG=$(echo "$RELEASE_JSON" | jq -r '.tagName // empty')
+
+  if [ -z "$RELEASE_TAG" ]; then
+    echo "::error::No release found on $REPO"
+    exit 1
+  fi
+
+  # Resolve the tag to a commit SHA
+  BASELINE_REF=$(gh api "repos/$REPO/git/ref/tags/$RELEASE_TAG" --jq '.object.sha' 2>/dev/null || true)
+
+  # If tag points to an annotated tag object, dereference to the commit
+  if [ -n "$BASELINE_REF" ]; then
+    OBJ_TYPE=$(gh api "repos/$REPO/git/tags/$BASELINE_REF" --jq '.object.type' 2>/dev/null || echo "commit")
+    if [ "$OBJ_TYPE" = "commit" ]; then
+      BASELINE_REF=$(gh api "repos/$REPO/git/tags/$BASELINE_REF" --jq '.object.sha' 2>/dev/null || echo "$BASELINE_REF")
+    fi
+  fi
+
+  if [ -z "$BASELINE_REF" ]; then
+    echo "::error::Could not resolve release tag $RELEASE_TAG to a commit"
+    exit 1
+  fi
+
+  echo "Release tag: $RELEASE_TAG"
+  echo "Release commit (baseline): $BASELINE_REF"
+  echo "::endgroup::"
+
+  # --- Step 3: Read last successful feature ref from charts repo ---
+  echo "::group::Reading persisted state"
+  LAST_FEATURE_REF=""
+  STATE_URL="https://raw.githubusercontent.com/decofe/reth-bench-charts/state/state/release-last-feature-ref"
+  if RAW=$(curl -sfL -H "Authorization: token ${DEREK_TOKEN}" "$STATE_URL"); then
+    LAST_FEATURE_REF=$(echo "$RAW" | tr -d '[:space:]')
+    echo "Previous feature ref: $LAST_FEATURE_REF"
+  else
+    echo "No persisted state found (first run)"
+  fi
+  echo "::endgroup::"
+
+  # --- Step 4: Skip logic ---
+  echo "::group::Resolving skip logic"
+  SHOULD_SKIP="false"
+  if [ -n "$LAST_FEATURE_REF" ] && [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+    if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+      echo "No new nightly, but force=true — running anyway"
+    else
+      SHOULD_SKIP="true"
+      echo "No new nightly since last release regression run — will skip"
+    fi
+  else
+    echo "New nightly detected or first run"
+  fi
+
+  echo "Baseline: $BASELINE_REF ($RELEASE_TAG)"
+  echo "Feature:  $FEATURE_REF"
+  echo "Skip:     $SHOULD_SKIP"
+  echo "::endgroup::"
+
+  # --- Step 5: Write outputs ---
+  {
+    echo "baseline-ref=$BASELINE_REF"
+    echo "feature-ref=$FEATURE_REF"
+    echo "should-skip=$SHOULD_SKIP"
+    echo "is-stale=false"
+    echo "stale-age-hours=0"
+    echo "nightly-created="
+    echo "long-running=false"
+    echo "release-tag=$RELEASE_TAG"
+  } >> "$GITHUB_OUTPUT"
+  exit 0
+fi
+
+# ==========================================================================
+# Nightly mode: query latest Docker nightly build (original logic)
+# ==========================================================================
+
+# --- Step 1: Query latest successful scheduled docker.yml run ---
+echo "::group::Querying latest nightly docker build"
+
+RUNS_JSON=$(gh run list \
+  -R "$REPO" \
+  --workflow=docker.yml \
+  --event=schedule \
+  --status=completed \
+  --limit 5 \
+  --json headSha,createdAt,conclusion)
+
+# Find the most recent successful run
+LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
+
+if [ -z "$LATEST" ]; then
+  echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
+  echo "Runs found: $RUNS_JSON"
+  exit 1
+fi
+
+FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
+CREATED_AT=$(echo "$LATEST" | jq -r '.createdAt')
+echo "Latest nightly commit: $FEATURE_REF"
+echo "Built at: $CREATED_AT"
+echo "::endgroup::"
+
+# --- Step 2: Staleness check ---
+echo "::group::Checking staleness"
+NOW_EPOCH=$(date +%s)
+# Handle both GNU date (-d) and BSD date (-j -f) for cross-platform compat
+CREATED_EPOCH=$(date -d "$CREATED_AT" +%s 2>/dev/null || \
+  date -j -f "%Y-%m-%dT%H:%M:%SZ" "$CREATED_AT" +%s 2>/dev/null || \
+  date -j -f "%Y-%m-%dT%T%z" "$CREATED_AT" +%s 2>/dev/null || \
+  { echo "::error::Cannot parse date: $CREATED_AT"; exit 1; })
+
+AGE_SECONDS=$(( NOW_EPOCH - CREATED_EPOCH ))
+AGE_HOURS=$(( AGE_SECONDS / 3600 ))
+IS_STALE="false"
+
+if [ "$AGE_HOURS" -gt 24 ]; then
+  IS_STALE="true"
+  echo "::warning::STALE NIGHTLY: Build is ${AGE_HOURS}h old (>24h threshold)"
+  echo "This indicates the nightly docker build failed — no new image was produced"
+else
+  echo "Nightly build age: ${AGE_HOURS}h (within 24h threshold)"
+fi
+echo "::endgroup::"
+
+# --- Step 3: Read last successful feature ref from charts repo ---
+echo "::group::Reading persisted state"
+LAST_FEATURE_REF=""
+STATE_URL="https://raw.githubusercontent.com/decofe/reth-bench-charts/state/state/nightly-last-feature-ref"
+if RAW=$(curl -sfL -H "Authorization: token ${DEREK_TOKEN}" "$STATE_URL"); then
+  LAST_FEATURE_REF=$(echo "$RAW" | tr -d '[:space:]')
+  echo "Previous feature ref: $LAST_FEATURE_REF"
+else
+  echo "No persisted state found (first run)"
+fi
+echo "::endgroup::"
+
+# --- Step 4: Determine baseline and skip logic ---
+echo "::group::Resolving refs"
+SHOULD_SKIP="false"
+BASELINE_REF="$FEATURE_REF"  # default for first run
+
+if [ "$IS_STALE" = "true" ]; then
+  # Stale = error path, don't skip (will alert and fail downstream)
+  SHOULD_SKIP="false"
+  BASELINE_REF="${LAST_FEATURE_REF:-$FEATURE_REF}"
+  echo "Stale nightly detected — will alert and fail"
+elif [ -z "$LAST_FEATURE_REF" ]; then
+  # First run: baseline = feature (self-comparison to establish baseline)
+  BASELINE_REF="$FEATURE_REF"
+  echo "First run — will benchmark nightly against itself to establish baseline"
+elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
+  # No new nightly since last successful run
+  if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
+    echo "No new nightly, but force=true — running anyway"
+    BASELINE_REF="$LAST_FEATURE_REF"
+  else
+    SHOULD_SKIP="true"
+    echo "No new nightly since last run — will skip"
+  fi
+else
+  # Normal case: new nightly available
+  BASELINE_REF="$LAST_FEATURE_REF"
+  echo "New nightly detected"
+fi
+
+echo "Baseline: $BASELINE_REF"
+echo "Feature:  $FEATURE_REF"
+echo "Skip:     $SHOULD_SKIP"
+echo "Stale:    $IS_STALE"
+echo "::endgroup::"
+
+# --- Step 5: Write outputs ---
+{
+  echo "baseline-ref=$BASELINE_REF"
+  echo "feature-ref=$FEATURE_REF"
+  echo "should-skip=$SHOULD_SKIP"
+  echo "is-stale=$IS_STALE"
+  echo "stale-age-hours=$AGE_HOURS"
+  echo "nightly-created=$CREATED_AT"
+} >> "$GITHUB_OUTPUT"

.github/scripts/bench-slack-notify.js

@@ -7,6 +7,8 @@
 //   BENCH_PR               – PR number (may be empty)
 //   BENCH_ACTOR            – GitHub user who triggered the bench
 //   BENCH_JOB_URL          – URL to the Actions job page
+//   BENCH_BASELINE_ARGS    – Extra CLI args for the baseline reth node
+//   BENCH_FEATURE_ARGS     – Extra CLI args for the feature reth node
 //   BENCH_SAMPLY           – 'true' if samply profiling was enabled
 //
 // Usage from actions/github-script:
@@ -16,6 +18,7 @@
 
 const fs = require('fs');
 const path = require('path');
+const { fmtChange, fmtMs, verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
 
 const SLACK_API = 'https://slack.com/api/chat.postMessage';
 
@@ -59,40 +62,17 @@ function cell(text) {
   return { type: 'raw_text', text: s || ' ' };
 }
 
+// Slack shortcodes for verdict (Block Kit header doesn't support unicode emoji)
+const SLACK_VERDICT = {
+  '⚠️': ':warning:',
+  '❌': ':x:',
+  '✅': ':white_check_mark:',
+  '⚪': ':white_circle:',
+};
+
 function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, repo, samplyUrls }) {
-  const b = summary.baseline.stats;
-  const f = summary.feature.stats;
-  const c = summary.changes;
-
-  const sigEmoji = { good: '\u2705', bad: '\u274c', neutral: '\u26aa' };
-
-  function fmtMs(v) { return v.toFixed(2) + 'ms'; }
-  function fmtMgas(v) { return v.toFixed(2); }
-  function fmtChange(ch) {
-    if (!ch.pct && !ch.ci_pct) return ' ';
-    const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
-    const ciStr = ch.ci_pct ? ` (\u00b1${ch.ci_pct.toFixed(2)}%)` : '';
-    return `${pctStr}${ciStr} ${sigEmoji[ch.sig]}`;
-  }
-
-  // Overall result for header
-  const vals = Object.values(c);
-  const hasBad = vals.some(v => v.sig === 'bad');
-  const hasGood = vals.some(v => v.sig === 'good');
-  let headerEmoji, headerResult;
-  if (hasBad && hasGood) {
-    headerEmoji = ':warning:';
-    headerResult = 'Mixed Results';
-  } else if (hasBad) {
-    headerEmoji = ':x:';
-    headerResult = 'Regression';
-  } else if (hasGood) {
-    headerEmoji = ':white_check_mark:';
-    headerResult = 'Improvement';
-  } else {
-    headerEmoji = ':white_circle:';
-    headerResult = 'No Difference';
-  }
+  const { emoji, label } = verdict(summary.changes);
+  const headerEmoji = SLACK_VERDICT[emoji] || emoji;
 
   const prUrl = prNumber ? `https://github.com/${repo}/pull/${prNumber}` : '';
   const commitUrl = `https://github.com/${repo}/commit`;
@@ -117,15 +97,15 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
   if (fl1) featureLine += ` | <${fl1}|Samply 1>`;
   if (fl2) featureLine += ` | <${fl2}|Samply 2>`;
 
-  const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
-  const cores = process.env.BENCH_CORES || '0';
-  const countsParts = [];
-  if (warmup) countsParts.push(`*Warmup:* ${warmup}`);
-  countsParts.push(`*Blocks:* ${summary.blocks}`);
-  if (cores !== '0') countsParts.push(`*Cores:* ${cores}`);
-  const countsLine = countsParts.join(' | ');
+  const countsLine = blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | ');
 
-  const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, countsLine].join('\n');
+  const baselineArgs = process.env.BENCH_BASELINE_ARGS || '';
+  const featureArgs = process.env.BENCH_FEATURE_ARGS || '';
+  const argsLines = [];
+  if (baselineArgs) argsLines.push(`*Baseline Args:* \`${baselineArgs}\``);
+  if (featureArgs) argsLines.push(`*Feature Args:* \`${featureArgs}\``);
+
+  const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, ...argsLines, countsLine].join('\n');
 
   // Action buttons
   const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
@@ -144,10 +124,17 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
     },
   ];
 
+  // Build table rows from shared metricRows
+  const rows = metricRows(summary);
+  const tableRows = [
+    [cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
+    ...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
+  ];
+
   const blocks = [
     {
       type: 'header',
-      text: { type: 'plain_text', text: `${headerEmoji} ${headerResult}`, emoji: true },
+      text: { type: 'plain_text', text: `${headerEmoji} ${label}`, emoji: true },
     },
     {
       type: 'section',
@@ -161,15 +148,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
         { align: 'right' },
         { align: 'right' },
       ],
-      rows: [
-        [cell('Metric'),  cell('Baseline'), cell('Feature'), cell('Change')],
-        [cell('Mean'),     cell(fmtMs(b.mean_ms)),      cell(fmtMs(f.mean_ms)),      cell(fmtChange(c.mean))],
-        [cell('StdDev'),   cell(fmtMs(b.stddev_ms)),    cell(fmtMs(f.stddev_ms)),    cell(' ')],
-        [cell('P50'),      cell(fmtMs(b.p50_ms)),       cell(fmtMs(f.p50_ms)),       cell(fmtChange(c.p50))],
-        [cell('P90'),      cell(fmtMs(b.p90_ms)),       cell(fmtMs(f.p90_ms)),       cell(fmtChange(c.p90))],
-        [cell('P99'),      cell(fmtMs(b.p99_ms)),       cell(fmtMs(f.p99_ms)),       cell(fmtChange(c.p99))],
-        [cell('Mgas/s'),   cell(fmtMgas(b.mean_mgas_s)), cell(fmtMgas(f.mean_mgas_s)), cell(fmtChange(c.mgas_s))],
-      ],
+      rows: tableRows,
     },
     {
       type: 'actions',
@@ -179,16 +158,12 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
 
   // Wait times as a separate table block (sent as threaded reply due to Slack one-table limit)
   const threadBlocks = [];
-  const waitTimes = summary.wait_times || {};
-  const waitKeys = Object.keys(waitTimes);
-  if (waitKeys.length > 0) {
-    const waitRows = [
+  const wtRows = waitTimeRows(summary);
+  if (wtRows.length > 0) {
+    const waitTableRows = [
       [cell('Wait Time'), cell('Baseline'), cell('Feature')],
+      ...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
     ];
-    for (const key of waitKeys) {
-      const wt = waitTimes[key];
-      waitRows.push([cell(wt.title), cell(fmtMs(wt.baseline.mean_ms)), cell(fmtMs(wt.feature.mean_ms))]);
-    }
     threadBlocks.push({
       type: 'table',
       column_settings: [
@@ -196,7 +171,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
         { align: 'right' },
         { align: 'right' },
       ],
-      rows: waitRows,
+      rows: waitTableRows,
     });
   }
 
@@ -258,16 +233,7 @@ async function success({ core, context }) {
   const jobUrl = process.env.BENCH_JOB_URL ||
     `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
 
-  // Load samply profile URLs (files exist when samply profiling was enabled)
-  const samplyUrls = {};
-  for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
-    try {
-      const url = fs.readFileSync(
-        path.join(process.env.BENCH_WORK_DIR, run, 'samply-profile-url.txt'), 'utf8'
-      ).trim();
-      if (url) samplyUrls[run] = url;
-    } catch {}
-  }
+  const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
 
   const slackUsers = loadSlackUsers(process.env.GITHUB_WORKSPACE || '.');
   const actorSlackId = slackUsers[actor];
@@ -284,6 +250,8 @@ async function success({ core, context }) {
     }
   }
 
+  const slackMode = process.env.BENCH_SLACK || 'always';
+
   // Post to public channel if any metric shows significant improvement or regression
   const channel = process.env.SLACK_BENCH_CHANNEL;
   let postedToChannel = false;
@@ -298,6 +266,14 @@ async function success({ core, context }) {
     }
   }
 
+  // In on-win mode, only notify on improvement — skip DM fallback entirely
+  if (slackMode === 'on-win') {
+    if (!postedToChannel) {
+      core.info('on-win mode: no improvement detected, skipping all notifications');
+    }
+    return;
+  }
+
   // DM the actor only when results were not posted to the public channel
   if (!postedToChannel) {
     if (actorSlackId) {

.github/scripts/bench-slack-users.json

@@ -9,5 +9,17 @@
   "gakonst": "U092SEPDM40",
   "Rjected": "U09F6SCKRGT",
   "DaniPopes": "U09FAT8EK2A",
-  "emmajam": "U0A34UN92HW"
+  "emmajam": "U0A34UN92HW",
+  "onbjerg": "U09FB0UK5AA",
+  "fgimenez": "U09G3GP7CSU",
+  "rakita": "U09FB3Z2M7Y",
+  "jxom": "U09F72MG083",
+  "tmm": "U0AD0U8E88N",
+  "pepyakin": "U0A7HKMGEHJ",
+  "grandizzy": "U09F8DBDDRT",
+  "SuperFluffy": "U095BKHB2Q4",
+  "kamsz": "U0A2563UBRD",
+  "zerosnacks": "U09FARPMN74",
+  "samczsun": "U096R14E4H3",
+  "laibe": "U09FARE0B9Q"
 }

.github/scripts/bench-upload-clickhouse.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""Upload bench-scheduled summary.json results to ClickHouse.
+
+Reads the summary JSON produced by bench-reth-summary.py and inserts a row
+into the bench_dual_comparisons table so the PM dashboard can display results.
+
+Usage:
+    bench-upload-clickhouse.py \
+        --summary <summary.json> \
+        --workflow-name <name> \
+        --chain <chain>
+
+Environment variables:
+    CLICKHOUSE_HOST      ClickHouse host URL
+    CLICKHOUSE_USER      ClickHouse username
+    CLICKHOUSE_PASSWORD  ClickHouse password
+    CLICKHOUSE_DATABASE  ClickHouse database (default: "default")
+"""
+
+import argparse
+import json
+import os
+import sys
+import urllib.request
+import urllib.error
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Upload benchmark results to ClickHouse")
+    parser.add_argument("--summary", required=True, help="Path to summary.json")
+    parser.add_argument("--workflow-name", required=True, help="Workflow name for ClickHouse")
+    parser.add_argument("--chain", default="mainnet", help="Chain name")
+    parser.add_argument("--grafana-url", default="", help="Grafana dashboard URL")
+    parser.add_argument("--github-diff-url", default="", help="GitHub diff URL")
+    parser.add_argument("--job-url", default="", help="CI job URL")
+    args = parser.parse_args()
+
+    ch_host = os.environ.get("CLICKHOUSE_HOST", "")
+    ch_user = os.environ.get("CLICKHOUSE_USER", "")
+    ch_password = os.environ.get("CLICKHOUSE_PASSWORD", "")
+    ch_database = os.environ.get("CLICKHOUSE_DATABASE", "default")
+    ch_table = "bench_dual_comparisons"
+
+    if not ch_host or not ch_user or not ch_password:
+        print("Missing ClickHouse credentials, skipping upload", file=sys.stderr)
+        sys.exit(0)
+
+    with open(args.summary) as f:
+        summary = json.load(f)
+
+    baseline = summary["baseline"]
+    feature = summary["feature"]
+    b_stats = baseline["stats"]
+    f_stats = feature["stats"]
+    changes = summary["changes"]
+    blocks = summary["blocks"]
+
+    # Extract wait time data
+    wait_times = summary.get("wait_times", {})
+    def wait_mean(field):
+        wt = wait_times.get(field, {})
+        b = wt.get("baseline", {}).get("mean_ms", 0.0)
+        f = wt.get("feature", {}).get("mean_ms", 0.0)
+        return b, f
+
+    b_persist, f_persist = wait_mean("persistence_wait_us")
+    b_exec_cache, f_exec_cache = wait_mean("execution_cache_wait_us")
+    b_sparse, f_sparse = wait_mean("sparse_trie_wait_us")
+
+    # gas_per_second: summary uses mean_mgas_s (Mgas/s), ClickHouse stores gas/s
+    b_gas_per_second = b_stats["mean_mgas_s"] * 1_000_000
+    f_gas_per_second = f_stats["mean_mgas_s"] * 1_000_000
+
+    mean_change = changes.get("mean", {}).get("pct", 0.0)
+    gas_change = changes.get("mgas_s", {}).get("pct", 0.0)
+    latency_improved = 1 if mean_change < 0 else 0
+    throughput_improved = 1 if gas_change > 0 else 0
+
+    big_blocks = "true" if summary.get("big_blocks", False) else "false"
+    warmup_blocks = summary.get("warmup_blocks", 0) or 0
+
+    def esc(s):
+        return str(s).replace("'", "\\'")
+
+    insert = f"""
+    INSERT INTO {ch_database}.{ch_table} (
+        workflow_name, chain,
+        baseline_ref, baseline_commit,
+        feature_ref, feature_commit,
+        blocks,
+        baseline_total_latency_ms, baseline_gas_per_second,
+        baseline_latency_mean_ms, baseline_latency_median_ms,
+        baseline_latency_p90_ms, baseline_latency_p99_ms,
+        feature_total_latency_ms, feature_gas_per_second,
+        feature_latency_mean_ms, feature_latency_median_ms,
+        feature_latency_p90_ms, feature_latency_p99_ms,
+        mean_latency_change_percent, gas_per_second_change_percent,
+        latency_improved, throughput_improved,
+        warmup_blocks, big_blocks,
+        grafana_benchmark_url, github_diff_url, argo_workflow_url,
+        baseline_persistence_wait_mean_ms, baseline_execution_cache_wait_mean_ms,
+        baseline_sparse_trie_wait_mean_ms,
+        feature_persistence_wait_mean_ms, feature_execution_cache_wait_mean_ms,
+        feature_sparse_trie_wait_mean_ms
+    ) VALUES (
+        '{esc(args.workflow_name)}', '{esc(args.chain)}',
+        '{esc(baseline["ref"])}', '{esc(baseline["ref"])}',
+        '{esc(feature["ref"])}', '{esc(feature["ref"])}',
+        {blocks},
+        {b_stats.get("wall_clock_s", 0) * 1000}, {b_gas_per_second},
+        {b_stats["mean_ms"]}, {b_stats["p50_ms"]},
+        {b_stats["p90_ms"]}, {b_stats["p99_ms"]},
+        {f_stats.get("wall_clock_s", 0) * 1000}, {f_gas_per_second},
+        {f_stats["mean_ms"]}, {f_stats["p50_ms"]},
+        {f_stats["p90_ms"]}, {f_stats["p99_ms"]},
+        {mean_change}, {gas_change},
+        {latency_improved}, {throughput_improved},
+        {warmup_blocks}, '{big_blocks}',
+        '{esc(args.grafana_url)}', '{esc(args.github_diff_url)}', '{esc(args.job_url)}',
+        {b_persist}, {b_exec_cache}, {b_sparse},
+        {f_persist}, {f_exec_cache}, {f_sparse}
+    );
+    """
+
+    # Build ClickHouse HTTP URL (credentials via headers, never in URL)
+    host = ch_host.rstrip("/")
+    if not host.startswith("http"):
+        host = f"https://{host}:8443"
+
+    url = f"{host}/?database={ch_database}"
+
+    req = urllib.request.Request(url, data=insert.encode("utf-8"), method="POST")
+    req.add_header("Content-Type", "text/plain")
+    req.add_header("X-ClickHouse-User", ch_user)
+    req.add_header("X-ClickHouse-Key", ch_password)
+
+    try:
+        with urllib.request.urlopen(req) as resp:
+            body = resp.read().decode("utf-8")
+            if body.strip():
+                print(f"ClickHouse response: {body}")
+        print(f"Successfully uploaded benchmark results to ClickHouse ({args.workflow_name})")
+    except urllib.error.HTTPError as e:
+        body = e.read().decode("utf-8")
+        print(f"ClickHouse upload failed ({e.code}): {body}", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/bench-utils.js

@@ -0,0 +1,112 @@
+// Shared utilities for reth-bench result rendering.
+//
+// Used by bench-job-summary.js and bench-slack-notify.js.
+
+const fs = require('fs');
+const path = require('path');
+
+const SIG_EMOJI = { good: '✅', bad: '❌', neutral: '⚪' };
+
+function fmtMs(v) { return v.toFixed(2) + 'ms'; }
+function fmtMgas(v) { return v.toFixed(2); }
+function fmtS(v) { return v.toFixed(2) + 's'; }
+
+function fmtChange(ch) {
+  if (!ch || (!ch.pct && !ch.ci_pct)) return '';
+  const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
+  const ciStr = ch.ci_pct ? ` (±${ch.ci_pct.toFixed(2)}%)` : '';
+  return `${pctStr}${ciStr} ${SIG_EMOJI[ch.sig]}`;
+}
+
+function verdict(changes) {
+  const vals = Object.values(changes);
+  const hasBad = vals.some(v => v.sig === 'bad');
+  const hasGood = vals.some(v => v.sig === 'good');
+  if (hasBad && hasGood) return { emoji: '⚠️', label: 'Mixed Results' };
+  if (hasBad) return { emoji: '❌', label: 'Regression' };
+  if (hasGood) return { emoji: '✅', label: 'Improvement' };
+  return { emoji: '⚪', label: 'No Difference' };
+}
+
+function loadSamplyUrls(workDir) {
+  const urls = {};
+  for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
+    try {
+      const url = fs.readFileSync(path.join(workDir, run, 'samply-profile-url.txt'), 'utf8').trim();
+      if (url) urls[run] = url;
+    } catch {}
+  }
+  return urls;
+}
+
+function balModeLabel(mode) {
+  switch (mode) {
+    case 'true':
+    case 'feature':
+    case 'baseline':
+      return mode;
+    case 'both':
+      return 'true';
+    default:
+      return '';
+  }
+}
+
+function blocksLabel(summary) {
+  const parts = [];
+  if (summary.big_blocks) {
+    parts.push({ key: 'Big Blocks', value: summary.blocks });
+    const balMode = balModeLabel(summary.bal_mode || summary.bal || process.env.BENCH_BAL || 'false');
+    if (balMode) parts.push({ key: 'BAL', value: balMode });
+  } else {
+    const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
+    if (warmup) parts.push({ key: 'Warmup', value: warmup });
+    parts.push({ key: 'Blocks', value: summary.blocks });
+  }
+  const cores = process.env.BENCH_CORES || '0';
+  if (cores !== '0') parts.push({ key: 'Cores', value: cores });
+  if (summary.wait_time) parts.push({ key: 'Wait time', value: summary.wait_time });
+  return parts;
+}
+
+// The 7 metric rows shared by all renderers.
+// Returns an array of { label, baseline, feature, change } objects.
+function metricRows(summary) {
+  const b = summary.baseline.stats;
+  const f = summary.feature.stats;
+  const c = summary.changes;
+  return [
+    { label: 'Mean',       baseline: fmtMs(b.mean_ms),       feature: fmtMs(f.mean_ms),       change: fmtChange(c.mean) },
+    { label: 'StdDev',     baseline: fmtMs(b.stddev_ms),     feature: fmtMs(f.stddev_ms),     change: '' },
+    { label: 'P50',        baseline: fmtMs(b.p50_ms),        feature: fmtMs(f.p50_ms),        change: fmtChange(c.p50) },
+    { label: 'P90',        baseline: fmtMs(b.p90_ms),        feature: fmtMs(f.p90_ms),        change: fmtChange(c.p90) },
+    { label: 'P99',        baseline: fmtMs(b.p99_ms),        feature: fmtMs(f.p99_ms),        change: fmtChange(c.p99) },
+    { label: 'Mgas/s',     baseline: fmtMgas(b.mean_mgas_s), feature: fmtMgas(f.mean_mgas_s), change: fmtChange(c.mgas_s) },
+    { label: 'Wall Clock', baseline: fmtS(b.wall_clock_s),   feature: fmtS(f.wall_clock_s),   change: fmtChange(c.wall_clock) },
+    { label: 'Persist Wait', baseline: fmtMs(b.mean_persist_ms || 0), feature: fmtMs(f.mean_persist_ms || 0), change: fmtChange(c.persist_wait) },
+  ];
+}
+
+// Wait time rows: one row per metric showing mean values.
+function waitTimeRows(summary) {
+  const waitTimes = summary.wait_times || {};
+  const rows = [];
+  for (const key of Object.keys(waitTimes)) {
+    const wt = waitTimes[key];
+    rows.push({ title: wt.title, baseline: fmtMs(wt.baseline.mean_ms), feature: fmtMs(wt.feature.mean_ms) });
+  }
+  return rows;
+}
+
+module.exports = {
+  SIG_EMOJI,
+  fmtMs,
+  fmtMgas,
+  fmtS,
+  fmtChange,
+  verdict,
+  loadSamplyUrls,
+  blocksLabel,
+  metricRows,
+  waitTimeRows,
+};

.github/scripts/check_rv32imac.sh

@@ -1,10 +1,7 @@
 #!/usr/bin/env bash
-set -uo pipefail
+set -uxo pipefail
 
 crates_to_check=(
-    reth-codecs-derive
-    reth-primitives
-    reth-primitives-traits
     reth-network-peers
     reth-trie-common
     reth-trie-sparse

.github/scripts/check_wasm.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-set -uo pipefail
+set -uxo pipefail
 
 readarray -t crates < <(
   cargo metadata --format-version=1 --no-deps | jq -r '.packages[].name' | grep '^reth' | sort
@@ -22,6 +22,7 @@ exclude_crates=(
   reth-downloaders
   reth-e2e-test-utils
   reth-engine-service
+  reth-execution-cache
   reth-engine-tree
   reth-engine-util
   reth-eth-wire
@@ -55,6 +56,7 @@ exclude_crates=(
   reth-ress-provider
   # The following are not supposed to be working
   reth # all of the crates below
+  reth-bb # binary-only, uses tokio features unsupported on wasm
   reth-storage-rpc-provider
   reth-invalid-block-hooks # reth-provider
   reth-libmdbx # mdbx

.github/scripts/fetch-grafana-dashboard.py

@@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+"""
+Fetch a Grafana dashboard and convert it to the portable import format.
+
+Fetches the dashboard via API, replaces internal datasource/variable references
+with template variables, and adds __inputs/__requires/__elements so the JSON is
+importable on any Grafana instance.
+
+Usage:
+    export FETCH_GRAFANA_DASHBOARD_URL=https://<NAMESPACE>.grafana.net
+    export FETCH_GRAFANA_DASHBOARD_TOKEN=glsa_...
+
+    python3 .github/scripts/fetch-grafana-dashboard.py <dashboard-uid> > output.json
+"""
+
+import json
+import os
+import sys
+import urllib.request
+
+PANEL_TYPE_NAMES = {
+    "bargauge": "Bar gauge",
+    "gauge": "Gauge",
+    "heatmap": "Heatmap",
+    "piechart": "Pie chart",
+    "stat": "Stat",
+    "table": "Table",
+    "timeseries": "Time series",
+    "barchart": "Bar chart",
+    "text": "Text",
+    "dashlist": "Dashboard list",
+    "logs": "Logs",
+    "nodeGraph": "Node Graph",
+    "histogram": "Histogram",
+    "candlestick": "Candlestick",
+    "state-timeline": "State timeline",
+    "status-history": "Status history",
+    "geomap": "Geomap",
+    "canvas": "Canvas",
+    "news": "News",
+    "xychart": "XY Chart",
+    "trend": "Trend",
+    "datagrid": "Datagrid",
+    "flamegraph": "Flame Graph",
+    "traces": "Traces",
+}
+
+
+def fetch_json(base_url: str, token: str, path: str) -> dict:
+    url = f"{base_url}{path}"
+    req = urllib.request.Request(url, headers={"Authorization": f"Bearer {token}"})
+    with urllib.request.urlopen(req) as resp:
+        return json.loads(resp.read())
+
+
+def fetch_dashboard(base_url: str, token: str, uid: str) -> dict:
+    return fetch_json(base_url, token, f"/api/dashboards/uid/{uid}")
+
+
+def fetch_grafana_version(base_url: str) -> str:
+    req = urllib.request.Request(f"{base_url}/api/health")
+    with urllib.request.urlopen(req) as resp:
+        data = json.loads(resp.read())
+    # version string like "13.0.0-23940615780.patch2" -> take just the semver part
+    version = data.get("version", "")
+    # strip build metadata after the first hyphen if it looks like a pre-release
+    parts = version.split("-")
+    return parts[0] if parts else version
+
+
+def collect_panel_types(panels: list) -> set[str]:
+    types = set()
+    for panel in panels:
+        ptype = panel.get("type", "")
+        if ptype and ptype != "row":
+            types.add(ptype)
+        # nested panels inside collapsed rows
+        for sub in panel.get("panels", []):
+            sub_type = sub.get("type", "")
+            if sub_type and sub_type != "row":
+                types.add(sub_type)
+    return types
+
+
+def has_expression_datasource(dashboard: dict) -> bool:
+    return "__expr__" in json.dumps(dashboard)
+
+
+def make_exportable(dashboard: dict, grafana_version: str = "") -> dict:
+    dash = json.loads(json.dumps(dashboard))  # deep copy
+
+    # --- Strip internal fields ---
+    dash.pop("id", None)
+
+    # --- Rewrite links: point to the public repo instead of internal ---
+    dash["links"] = [
+        {
+            "asDropdown": False,
+            "icon": "external link",
+            "includeVars": False,
+            "keepTime": False,
+            "tags": [],
+            "targetBlank": True,
+            "title": "Source (GitHub)",
+            "tooltip": "View source file in repository",
+            "type": "link",
+            "url": "https://github.com/paradigmxyz/reth/tree/main/etc/grafana/dashboards",
+        }
+    ]
+
+    # --- Datasource: victoriametrics -> prometheus ---
+    dash_str = json.dumps(dash)
+    dash_str = dash_str.replace("victoriametrics-metrics-datasource", "prometheus")
+    dash = json.loads(dash_str)
+
+    # --- Templating: instance_label constant -> ${VAR_INSTANCE_LABEL} ---
+    # Also strip default-value fields the API returns that are not needed for import
+    STRIP_VAR_DEFAULTS = {"allowCustomValue", "regexApplyTo"}
+    for var in dash.get("templating", {}).get("list", []):
+        if var.get("name") == "instance_label" and var.get("type") == "constant":
+            var["query"] = "${VAR_INSTANCE_LABEL}"
+            var["current"] = {
+                "value": "${VAR_INSTANCE_LABEL}",
+                "text": "${VAR_INSTANCE_LABEL}",
+                "selected": False,
+            }
+            var["options"] = [
+                {
+                    "value": "${VAR_INSTANCE_LABEL}",
+                    "text": "${VAR_INSTANCE_LABEL}",
+                    "selected": False,
+                }
+            ]
+        # Clear current values for query/datasource vars (not meaningful for import)
+        elif var.get("type") in ("query", "datasource"):
+            var["current"] = {}
+        # Remove noisy default fields
+        for field in STRIP_VAR_DEFAULTS:
+            var.pop(field, None)
+        # Strip falsy defaults on query/datasource vars (API returns them, export omits them)
+        if var.get("type") in ("query", "datasource"):
+            for field in ("hide", "multi", "skipUrlSync"):
+                if not var.get(field):
+                    var.pop(field, None)
+
+    # --- Build __inputs ---
+    inputs = [
+        {
+            "name": "DS_PROMETHEUS",
+            "label": "Prometheus",
+            "description": "",
+            "type": "datasource",
+            "pluginId": "prometheus",
+            "pluginName": "Prometheus",
+        },
+    ]
+
+    if has_expression_datasource(dash):
+        inputs.append(
+            {
+                "name": "DS_EXPRESSION",
+                "label": "Expression",
+                "description": "",
+                "type": "datasource",
+                "pluginId": "__expr__",
+            }
+        )
+
+    inputs.append(
+        {
+            "name": "VAR_INSTANCE_LABEL",
+            "type": "constant",
+            "label": "Instance Label",
+            "value": "job",
+            "description": "",
+        }
+    )
+
+    # --- Build __requires ---
+    requires = []
+
+    if has_expression_datasource(dash):
+        requires.append({"type": "datasource", "id": "__expr__", "version": "1.0.0"})
+
+    panel_types = collect_panel_types(dash.get("panels", []))
+    for pt in sorted(panel_types):
+        requires.append(
+            {
+                "type": "panel",
+                "id": pt,
+                "name": PANEL_TYPE_NAMES.get(pt, pt),
+                "version": "",
+            }
+        )
+
+    requires.append(
+        {"type": "grafana", "id": "grafana", "name": "Grafana", "version": grafana_version}
+    )
+    requires.append(
+        {
+            "type": "datasource",
+            "id": "prometheus",
+            "name": "Prometheus",
+            "version": "1.0.0",
+        }
+    )
+
+    # --- Assemble output (with __inputs/__requires/__elements first) ---
+    output = {
+        "__inputs": inputs,
+        "__elements": {},
+        "__requires": requires,
+    }
+    output.update(dash)
+
+    return output
+
+
+def main():
+    if len(sys.argv) < 2:
+        print(f"Usage: {sys.argv[0]} <dashboard-uid>", file=sys.stderr)
+        sys.exit(1)
+
+    uid = sys.argv[1]
+    base_url = os.environ.get("FETCH_GRAFANA_DASHBOARD_URL", "").rstrip("/")
+    token = os.environ.get("FETCH_GRAFANA_DASHBOARD_TOKEN", "")
+
+    if not base_url or not token:
+        print(
+            "Error: FETCH_GRAFANA_DASHBOARD_URL and FETCH_GRAFANA_DASHBOARD_TOKEN env vars required",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    resp = fetch_dashboard(base_url, token, uid)
+    dashboard = resp["dashboard"]
+
+    grafana_version = fetch_grafana_version(base_url)
+    exported = make_exportable(dashboard, grafana_version)
+    print(json.dumps(exported, indent=2))
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/hive/Dockerfile

@@ -7,7 +7,7 @@ FROM lukemathwalker/cargo-chef:latest-rust-1 AS chef
 WORKDIR /app
 
 # Install system dependencies
-RUN apt-get update && apt-get -y upgrade && apt-get install -y libclang-dev pkg-config
+RUN apt-get update && apt-get install -y libclang-dev pkg-config
 
 # 
 # We prepare the build plan

.github/scripts/hive/build_simulators.sh

@@ -1,6 +1,23 @@
 #!/usr/bin/env bash
 set -eo pipefail
 
+fixture_variant="${1:-osaka}"
+
+case "${fixture_variant}" in
+    amsterdam)
+        eels_fixtures="https://github.com/ethereum/execution-spec-tests/releases/download/snobal-devnet-5@v8037.0.0/fixtures_snobal-devnet-5.tar.gz"
+        eels_branch="devnets/snobal/5"
+        ;;
+    osaka)
+        eels_fixtures="https://github.com/ethereum/execution-spec-tests/releases/download/v5.3.0/fixtures_develop.tar.gz"
+        eels_branch="forks/osaka"
+        ;;
+    *)
+        echo "unknown hive fixture variant: ${fixture_variant}"
+        exit 1
+        ;;
+esac
+
 # Create the hive_assets directory
 mkdir hive_assets/
 
@@ -11,8 +28,14 @@ go build .
 
 # Run each hive command in the background for each simulator and wait
 echo "Building images"
-# TODO: test code has been moved from https://github.com/ethereum/execution-spec-tests to https://github.com/ethereum/execution-specs  we need to pin eels branch with `--sim.buildarg branch=<release-branch-name>` once we have the fusaka release tagged on the new repo
-./hive -client reth --sim "ethereum/eels" --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/v5.3.0/fixtures_develop.tar.gz -sim.timelimit 1s || true &
+./hive -client reth --sim "ethereum/eels/consume-engine" \
+    --sim.buildarg fixtures="${eels_fixtures}" \
+    --sim.buildarg branch="${eels_branch}" \
+    --sim.timelimit 1s || true &
+./hive -client reth --sim "ethereum/eels/consume-rlp" \
+    --sim.buildarg fixtures="${eels_fixtures}" \
+    --sim.buildarg branch="${eels_branch}" \
+    --sim.timelimit 1s || true &
 ./hive -client reth --sim "ethereum/engine" -sim.timelimit 1s || true &
 ./hive -client reth --sim "devp2p" -sim.timelimit 1s || true &
 ./hive -client reth --sim "ethereum/rpc-compat" -sim.timelimit 1s || true &

.github/scripts/hive/expected_failures.yaml

@@ -1,9 +1,6 @@
 # tracked by https://github.com/paradigmxyz/reth/issues/13879
 rpc-compat:
   - debug_getRawBlock/get-invalid-number (reth)
-  - debug_getRawHeader/get-invalid-number (reth)
-  - debug_getRawReceipts/get-invalid-number (reth)
-  - debug_getRawReceipts/get-block-n (reth)
   - debug_getRawTransaction/get-invalid-hash (reth)
 
   - eth_getStorageAt/get-storage-invalid-key-too-large (reth)
@@ -16,32 +13,15 @@ rpc-compat:
   # syncing mode, the test expects syncing to be false on start
   - eth_syncing/check-syncing (reth)
 
-# no fix due to https://github.com/paradigmxyz/reth/issues/8732
-engine-withdrawals:
-  - Withdrawals Fork On Genesis (Paris) (reth)
-  - Withdrawals Fork on Block 1 (Paris) (reth)
-  - Withdrawals Fork on Block 2 (Paris) (reth)
-  - Withdrawals Fork on Block 3 (Paris) (reth)
-  - Withdraw to a single account (Paris) (reth)
-  - Withdraw to two accounts (Paris) (reth)
-  - Withdraw many accounts (Paris) (reth)
-  - Withdraw zero amount (Paris) (reth)
-  - Empty Withdrawals (Paris) (reth)
-  - Corrupted Block Hash Payload (INVALID) (Paris) (reth)
-  - Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org (Paris) (reth)
+engine-withdrawals: []
 
-engine-api: [ ]
+engine-api: []
 
-# no fix due to https://github.com/paradigmxyz/reth/issues/8732
-engine-cancun:
-  - Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True (Cancun) (reth)
-  # the test fails with older versions of the code for which it passed before, probably related to changes
-  # in hive or its dependencies
-  - Blob Transaction Ordering, Multiple Clients (Cancun) (reth)
+engine-cancun: []
 
-sync: [ ]
+sync: []
 
-engine-auth: [ ]
+engine-auth: []
 
 # EIP-7610 related tests (Revert creation in case of non-empty storage):
 #
@@ -49,7 +29,7 @@ engine-auth: [ ]
 #   The test artificially creates an empty account with storage, then tests EIP-7610's behavior.
 #   On mainnet, ~25 such accounts exist as contract addresses (derived from keccak(prefix, caller,
 #   nonce/salt), not from public keys). No private key exists for contract addresses. To trigger
-#   this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible. 
+#   this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible.
 #
 # tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_*
 #   Requires hash collision on create2 address to target already deployed accounts with storage.
@@ -119,6 +99,40 @@ eels/consume-engine:
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_2-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Shanghai-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE2-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Amsterdam-blockchain_test_engine_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE2-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_engine_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_engine_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_engine_from_state_test-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_engine_from_state_test-opcode_CREATE-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_engine_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Prague-blockchain_test_engine_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Shanghai-blockchain_test_engine_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Paris-blockchain_test_engine_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_engine_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Cancun-blockchain_test_engine_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_engine_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_engine_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_engine_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_engine_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Osaka-blockchain_test_engine_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_engine_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_engine_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_engine_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_engine_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_engine_from_state_test-empty-initcode]-reth
 
 # Blob limit tests:
 #
@@ -213,3 +227,37 @@ eels/consume-rlp:
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_1-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Prague-tx_type_2-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
   - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Shanghai-tx_type_0-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_2-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Amsterdam-blockchain_test_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_1-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_from_state_test-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_tx[fork_Amsterdam-tx_type_0-blockchain_test_from_state_test-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE2-non-empty-balance-correct-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_create_opcode[fork_Amsterdam-blockchain_test_from_state_test-opcode_CREATE2-non-empty-balance-revert-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Amsterdam-blockchain_test_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Prague-blockchain_test_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Shanghai-blockchain_test_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Paris-blockchain_test_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Cancun-blockchain_test_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Shanghai-blockchain_test_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_from_state_test-initcode-with-deploy]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_collision_with_create2_revert_in_initcode[fork_Osaka-blockchain_test_from_state_test]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Prague-blockchain_test_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Cancun-blockchain_test_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Paris-blockchain_test_from_state_test-sstore-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_from_state_test-empty-initcode]-reth
+  - tests/paris/eip7610_create_collision/test_revert_in_create.py::test_create2_collision_storage[fork_Osaka-blockchain_test_from_state_test-initcode-with-deploy]-reth

.github/scripts/hive/ignored_tests.yaml

@@ -21,7 +21,6 @@ engine-cancun:
   # Affects all clients, not just reth. Tracked: https://github.com/ethereum/hive/issues/1382
   - Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (reth)
   - Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (reth)
-  - Multiple New Payloads Extending Canonical Chain, Wait for Canonical Payload (Cancun) (reth)
 engine-api:
   - Transaction Re-Org, Re-Org Out (Paris) (reth)
   - Transaction Re-Org, Re-Org to Different Block (Paris) (reth)
@@ -31,5 +30,3 @@ engine-api:
   - Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Paris) (reth)
   - Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Paris) (reth)
   - Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P10 (Paris) (reth)
-  - Multiple New Payloads Extending Canonical Chain, Wait for Canonical Payload (Paris) (reth)
-  - Multiple New Payloads Extending Canonical Chain, Set Head to First Payload Received (Paris) (reth)

.github/scripts/hive/run_simulator.sh

@@ -5,6 +5,12 @@ cd hivetests/
 
 sim="${1}"
 limit="${2}"
+fixture_variant="${3:-}"
+
+if [[ "${fixture_variant}" == "osaka" && "${sim}" == *"eels"* && "${limit}" == *"tests/amsterdam"* ]]; then
+    echo "osaka fixtures do not support amsterdam tests"
+    exit 1
+fi
 
 # Use lower parallelism for eels tests to avoid OOM-killing the runner
 parallelism=16
@@ -13,7 +19,13 @@ if [[ "${sim}" == *"eels"* ]]; then
 fi
 
 run_hive() {
-    hive --sim "${sim}" --sim.limit "${limit}" --sim.parallelism "${parallelism}" --client reth 2>&1 | tee /tmp/log || true
+    hive \
+  --sim "${sim}" \
+  --sim.limit "${limit}" \
+  --sim.limit.exact=false \
+  --sim.parallelism "${parallelism}" \
+  --client reth \
+  2>&1 | tee /tmp/log || true
 }
 
 check_log() {

.github/scripts/install_geth.sh

@@ -2,7 +2,7 @@
 
 # Installs Geth (https://geth.ethereum.org) in $HOME/bin for x86_64 Linux.
 
-set -eo pipefail
+set -exo pipefail
 
 GETH_BUILD=${GETH_BUILD:-"1.13.4-3f907d6a"}
 

.github/scripts/verify_image_arch.sh

@@ -7,7 +7,7 @@
 # Environment:
 #   DRY_RUN=true  - Skip actual verification, just print what would be checked.
 
-set -euo pipefail
+set -euxo pipefail
 
 TARGETS="${1:-}"
 REGISTRY="${2:-}"

.github/workflows/book.yml

@@ -10,19 +10,22 @@ on:
     types: [opened, reopened, synchronize, closed]
   merge_group:
 
-env:
-  RUSTC_WRAPPER: "sccache"
+permissions: {}
 
 jobs:
   build:
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 90
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
 
       - name: Install bun
-        uses: oven-sh/setup-bun@v2
+        uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
         with:
           bun-version: v1.2.23
 
@@ -36,8 +39,6 @@ jobs:
       - name: Install Rust nightly
         uses: dtolnay/rust-toolchain@nightly
 
-      - uses: mozilla-actions/sccache-action@v0.0.9
-
       - name: Build docs
         run: cd docs/vocs && bash scripts/build-cargo-docs.sh
 
@@ -47,10 +48,10 @@ jobs:
           echo "Vocs Build Complete"
 
       - name: Setup Pages
-        uses: actions/configure-pages@v5
+        uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
 
       - name: Upload artifact
-        uses: actions/upload-pages-artifact@v4
+        uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
         with:
           path: "./docs/vocs/docs/dist"
 
@@ -74,4 +75,4 @@ jobs:
     steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v4
+        uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0

.github/workflows/changelog.yml

@@ -1,25 +0,0 @@
-name: Changelog
-
-on:
-  pull_request:
-    types: [opened, synchronize]
-
-jobs:
-  changelog:
-    # Skip for fork PRs since they can't access secrets
-    if: github.event.pull_request.head.repo.full_name == github.repository
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-      pull-requests: write
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-          ref: ${{ github.event.pull_request.head.sha }}
-      - run: npm install -g @anthropic-ai/claude-code
-      - uses: wevm/changelogs/check@master
-        with:
-          ai: 'claude -p'
-        env:
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

.github/workflows/check-alloy.yml

@@ -22,31 +22,41 @@ on:
 env:
   CARGO_TERM_COLOR: always
 
+permissions: {}
+
 jobs:
   check:
     name: Check compilation with patched alloy
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-16' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
 
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: Swatinem/rust-cache@v2
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
 
       - name: Apply alloy patches
+        env:
+          ALLOY_BRANCH: ${{ inputs.alloy_branch }}
+          ALLOY_EVM_BRANCH: ${{ inputs.alloy_evm_branch }}
+          OP_ALLOY_BRANCH: ${{ inputs.op_alloy_branch }}
         run: |
           ARGS=""
-          if [ -n "${{ inputs.alloy_branch }}" ]; then
-            ARGS="$ARGS --alloy ${{ inputs.alloy_branch }}"
+          if [ -n "$ALLOY_BRANCH" ]; then
+            ARGS="$ARGS --alloy $ALLOY_BRANCH"
           fi
-          if [ -n "${{ inputs.alloy_evm_branch }}" ]; then
-            ARGS="$ARGS --evm ${{ inputs.alloy_evm_branch }}"
+          if [ -n "$ALLOY_EVM_BRANCH" ]; then
+            ARGS="$ARGS --evm $ALLOY_EVM_BRANCH"
           fi
-          if [ -n "${{ inputs.op_alloy_branch }}" ]; then
-            ARGS="$ARGS --op ${{ inputs.op_alloy_branch }}"
+          if [ -n "$OP_ALLOY_BRANCH" ]; then
+            ARGS="$ARGS --op $OP_ALLOY_BRANCH"
           fi
           
           if [ -z "$ARGS" ]; then

.github/workflows/compact.yml

@@ -16,32 +16,38 @@ env:
   RUSTC_WRAPPER: "sccache"
 
 name: compact-codec
+permissions: {}
+
 jobs:
   compact-codec:
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     strategy:
       matrix:
         bin:
           - cargo run --bin reth --features "dev"
     steps:
-      - uses: rui314/setup-mold@v1
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - name: Checkout base
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           ref: ${{ github.base_ref || 'main' }}
+          persist-credentials: false
       # On `main` branch, generates test vectors and serializes them to disk using `Compact`.
       - name: Generate compact vectors
         run: |
           ${{ matrix.bin }} -- test-vectors compact --write
       - name: Checkout PR
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           clean: false
+          persist-credentials: false
       # On incoming merge try to read and decode previously generated vectors with `Compact`
       - name: Read vectors
         run: ${{ matrix.bin }} -- test-vectors compact --read

.github/workflows/dependencies.yml

@@ -9,13 +9,14 @@ on:
   workflow_dispatch:
     # Needed so we can run it manually
 
-permissions:
-  contents: write
-  pull-requests: write
+permissions: {}
 
 jobs:
   update:
     if: github.repository == 'paradigmxyz/reth'
+    permissions:
+      contents: write
+      pull-requests: write
     uses: tempoxyz/ci/.github/workflows/cargo-update-pr.yml@main
     secrets:
-      token: ${{ secrets.GITHUB_TOKEN }}
+      token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/docker-tag-latest.yml

@@ -17,6 +17,8 @@ on:
 env:
   DOCKER_USERNAME: ${{ github.actor }}
 
+permissions: {}
+
 jobs:
   tag-reth-latest:
     name: Tag reth as latest
@@ -27,16 +29,22 @@ jobs:
       contents: read
     steps:
       - name: Log in to Docker
+        env:
+          DOCKER_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
         run: |
-          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io --username ${DOCKER_USERNAME} --password-stdin
+          echo "$DOCKER_PASSWORD" | docker login ghcr.io --username "${DOCKER_USERNAME}" --password-stdin
 
       - name: Pull reth release image
+        env:
+          VERSION: ${{ inputs.version }}
         run: |
-          docker pull ghcr.io/${{ github.repository_owner }}/reth:${{ inputs.version }}
+          docker pull ghcr.io/${{ github.repository_owner }}/reth:${VERSION}
 
       - name: Tag reth as latest
+        env:
+          VERSION: ${{ inputs.version }}
         run: |
-          docker tag ghcr.io/${{ github.repository_owner }}/reth:${{ inputs.version }} ghcr.io/${{ github.repository_owner }}/reth:latest
+          docker tag ghcr.io/${{ github.repository_owner }}/reth:${VERSION} ghcr.io/${{ github.repository_owner }}/reth:latest
 
       - name: Push reth latest tag
         run: |

.github/workflows/docker-test.yml

@@ -6,13 +6,15 @@ on:
       hive_target:
         required: true
         type: string
-        description: "Docker bake target to build (e.g. hive-stable, hive-edge)"
+        description: "Docker bake target to build (e.g. hive)"
       artifact_name:
         required: false
         type: string
         default: "artifacts"
         description: "Name for the uploaded artifact"
 
+permissions: {}
+
 jobs:
   build:
     timeout-minutes: 45
@@ -21,7 +23,9 @@ jobs:
       id-token: write
       contents: read
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - run: mkdir -p artifacts
 
       - name: Get git info
@@ -32,8 +36,12 @@ jobs:
 
       - name: Detect fork
         id: fork
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
+          REPO: ${{ github.repository }}
         run: |
-          if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.repository }}" ]; then
+          if [ "$EVENT_NAME" = "pull_request" ] && [ "$HEAD_REPO" != "$REPO" ]; then
             echo "is_fork=true" >> "$GITHUB_OUTPUT"
           else
             echo "is_fork=false" >> "$GITHUB_OUTPUT"
@@ -42,11 +50,11 @@ jobs:
       # Depot build (upstream only)
       - name: Set up Depot CLI
         if: steps.fork.outputs.is_fork == 'false'
-        uses: depot/setup-action@v1
+        uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1.7.1
 
       - name: Build reth image (Depot)
         if: steps.fork.outputs.is_fork == 'false'
-        uses: depot/bake-action@v1
+        uses: depot/bake-action@1d58c2668346981089b088b7ef36755b206b20e9 # v1.13.0
         env:
           DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
           VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
@@ -60,11 +68,11 @@ jobs:
       # Docker build (forks)
       - name: Set up Docker Buildx
         if: steps.fork.outputs.is_fork == 'true'
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
 
       - name: Build reth image (Docker)
         if: steps.fork.outputs.is_fork == 'true'
-        uses: docker/bake-action@v6
+        uses: docker/bake-action@82490499d2e5613fcead7e128237ef0b0ea210f7 # v7.0.0
         env:
           VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
           VERGEN_GIT_DESCRIBE: ${{ steps.git.outputs.describe }}
@@ -76,7 +84,7 @@ jobs:
             *.dockerfile=Dockerfile
 
       - name: Upload reth image
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
         with:
           name: ${{ inputs.artifact_name }}
           path: ./artifacts

.github/workflows/docker.yml

@@ -29,6 +29,8 @@ on:
         type: boolean
         default: false
 
+permissions: {}
+
 jobs:
   build:
     if: github.repository == 'paradigmxyz/reth'
@@ -39,13 +41,15 @@ jobs:
       contents: read
       id-token: write
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
 
       - name: Set up Depot CLI
-        uses: depot/setup-action@v1
+        uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1.7.1
 
       - name: Log in to GHCR
-        uses: docker/login-action@v3
+        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -60,10 +64,13 @@ jobs:
 
       - name: Determine build parameters
         id: params
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          BUILD_TYPE: ${{ inputs.build_type }}
         run: |
           REGISTRY="ghcr.io/${{ github.repository_owner }}"
 
-          if [[ "${{ github.event_name }}" == "push" ]]; then
+          if [[ "${EVENT_NAME}" == "push" ]]; then
             VERSION="${GITHUB_REF#refs/tags/}"
             echo "targets=ethereum" >> "$GITHUB_OUTPUT"
 
@@ -81,7 +88,7 @@ jobs:
               echo "ethereum_set=ethereum.tags=${REGISTRY}/reth:${VERSION}" >> "$GITHUB_OUTPUT"
             fi
 
-          elif [[ "${{ github.event_name }}" == "schedule" ]] || [[ "${{ inputs.build_type }}" == "nightly" ]]; then
+          elif [[ "${EVENT_NAME}" == "schedule" ]] || [[ "${BUILD_TYPE}" == "nightly" ]]; then
             echo "targets=nightly" >> "$GITHUB_OUTPUT"
             echo "ethereum_tags=${REGISTRY}/reth:nightly" >> "$GITHUB_OUTPUT"
             echo "ethereum_set=ethereum.tags=${REGISTRY}/reth:nightly" >> "$GITHUB_OUTPUT"
@@ -94,7 +101,7 @@ jobs:
           fi
 
       - name: Build and push images
-        uses: depot/bake-action@v1
+        uses: depot/bake-action@1d58c2668346981089b088b7ef36755b206b20e9 # v1.13.0
         env:
           VERGEN_GIT_SHA: ${{ steps.git.outputs.sha }}
           VERGEN_GIT_DESCRIBE: ${{ steps.git.outputs.describe }}
@@ -105,6 +112,8 @@ jobs:
           files: docker-bake.hcl
           targets: ${{ steps.params.outputs.targets }}
           push: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }}
+          save: false
+          load: false
           set: |
             ${{ steps.params.outputs.ethereum_set }}
 
@@ -124,7 +133,7 @@ jobs:
     if: failure() && github.event_name == 'schedule'
     steps:
       - name: Slack Webhook Action
-        uses: rtCamp/action-slack-notify@v2
+        uses: rtCamp/action-slack-notify@e31e87e03dd19038e411e38ae27cbad084a90661 # v2.3.3
         env:
           SLACK_COLOR: danger
           SLACK_ICON_EMOJI: ":rotating_light:"
@@ -138,6 +147,8 @@ jobs:
             *Run:* <https://github.com/paradigmxyz/reth/actions/runs/${{ github.run_id }}|View logs>
 
             *Action required:* Re-run the workflow or investigate the build failure.
+
+            <@U0AAA8F0JEM> investigate and re-run if flaky
           SLACK_FOOTER: "paradigmxyz/reth · docker.yml"
           MSG_MINIMAL: true
           SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}

.github/workflows/e2e.yml

@@ -17,19 +17,27 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
+permissions: {}
+
 jobs:
   test:
     name: e2e-testsuite
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     env:
       RUST_BACKTRACE: 1
     timeout-minutes: 90
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - uses: dtolnay/rust-toolchain@stable
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: taiki-e/install-action@nextest
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: nextest
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - name: Run e2e tests
@@ -48,21 +56,27 @@ jobs:
   rocksdb:
     name: e2e-rocksdb
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     env:
       RUST_BACKTRACE: 1
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - uses: dtolnay/rust-toolchain@stable
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: taiki-e/install-action@nextest
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: nextest
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - name: Run RocksDB e2e tests
         run: |
           cargo nextest run \
             --no-fail-fast \
-            --locked --features "edge" \
+            --locked \
             -p reth-e2e-test-utils \
             -E 'binary(rocksdb)'

.github/workflows/fetch-grafana-dashboard.yml

@@ -0,0 +1,72 @@
+name: Fetch Grafana Dashboard
+
+on:
+  workflow_dispatch:
+    inputs:
+      dashboard_uid:
+        description: "Grafana dashboard UID to export"
+        required: true
+        default: "2k8BXz24x"
+      target_path:
+        description: "Target file path in the repo (e.g. etc/grafana/dashboards/overview.json)"
+        required: true
+        default: "etc/grafana/dashboards/overview.json"
+
+permissions: {}
+
+jobs:
+  fetch:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: "3.12"
+
+      - name: Fetch dashboard from Grafana
+        env:
+          FETCH_GRAFANA_DASHBOARD_URL: ${{ secrets.FETCH_GRAFANA_DASHBOARD_URL }}
+          FETCH_GRAFANA_DASHBOARD_TOKEN: ${{ secrets.FETCH_GRAFANA_DASHBOARD_TOKEN }}
+          DASHBOARD_UID: ${{ inputs.dashboard_uid }}
+          TARGET_PATH: ${{ inputs.target_path }}
+        run: |
+          python3 .github/scripts/fetch-grafana-dashboard.py "${DASHBOARD_UID}" \
+            > "${TARGET_PATH}"
+
+      - name: Check for changes
+        id: diff
+        env:
+          TARGET_PATH: ${{ inputs.target_path }}
+        run: |
+          if git diff --quiet "${TARGET_PATH}"; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+            echo "No changes detected."
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Create pull request
+        if: steps.diff.outputs.changed == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          DASHBOARD_UID: ${{ inputs.dashboard_uid }}
+          TARGET_PATH: ${{ inputs.target_path }}
+        run: |
+          TARGET="${TARGET_PATH}"
+          FILENAME="$(basename "$TARGET")"
+          BRANCH="chore/sync-grafana-${FILENAME%.*}-$(date +%Y%m%d-%H%M%S)"
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git checkout -b "$BRANCH"
+          git add "$TARGET"
+          git commit -m "chore: update Grafana dashboard ${FILENAME}"
+          git push origin "$BRANCH"
+          gh pr create \
+            --title "chore: update Grafana dashboard ${FILENAME}" \
+            --body "Automated export from Grafana (dashboard UID: \`${DASHBOARD_UID}\`, target: \`${TARGET}\`)."

.github/workflows/grafana.yml

@@ -6,16 +6,33 @@ on:
   push:
     branches: [main]
 
+permissions: {}
+
 jobs:
   check-dashboard:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     steps:
-      - uses: actions/checkout@v6
-      - name: Check for ${DS_PROMETHEUS} in overview.json
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - name: Validate dashboard format
         run: |
-          if grep -Fn '${DS_PROMETHEUS}' etc/grafana/dashboards/overview.json; then
-            echo "Error: overview.json contains '\${DS_PROMETHEUS}' placeholder"
-            echo "Please replace it with '\${datasource}'"
-            exit 1
-          fi
-          echo "✓ overview.json does not contain '\${DS_PROMETHEUS}' placeholder"
+          python3 -c "
+          import json, sys
+          with open('etc/grafana/dashboards/overview.json') as f:
+              d = json.load(f)
+          errors = []
+          if '__inputs' not in d:
+              errors.append('missing __inputs')
+          if '__requires' not in d:
+              errors.append('missing __requires')
+          if d.get('id') is not None:
+              errors.append('contains internal id field — use export-dashboard.py')
+          if errors:
+              for e in errors:
+                  print(f'Error: {e}', file=sys.stderr)
+              sys.exit(1)
+          print('✓ overview.json is a valid exported dashboard')
+          "

.github/workflows/hive.yml

@@ -14,52 +14,62 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
-jobs:
-  build-reth-stable:
-    uses: ./.github/workflows/docker-test.yml
-    with:
-      hive_target: hive-stable
-      artifact_name: "reth-stable"
-    secrets: inherit
+permissions: {}
 
-  build-reth-edge:
+jobs:
+  build-reth:
+    permissions:
+      contents: read
+      id-token: write
     uses: ./.github/workflows/docker-test.yml
     with:
-      hive_target: hive-edge
-      artifact_name: "reth-edge"
+      hive_target: hive
+      artifact_name: "reth"
     secrets: inherit
 
   prepare-hive:
     if: github.repository == 'paradigmxyz/reth'
     timeout-minutes: 45
-    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-16' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        variant:
+          - amsterdam
+          - osaka
+    name: Prepare Hive - ${{ matrix.variant == 'amsterdam' && 'Amsterdam' || 'Osaka' }}
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - name: Checkout hive tests
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           repository: ethereum/hive
           path: hivetests
+          persist-credentials: false
 
       - name: Get hive commit hash
         id: hive-commit
         run: echo "hash=$(cd hivetests && git rev-parse HEAD)" >> $GITHUB_OUTPUT
 
-      - uses: actions/setup-go@v6
+      - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
         with:
           go-version: "^1.13.1"
       - run: go version
 
       - name: Restore hive assets cache
         id: cache-hive
-        uses: actions/cache@v5
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
         with:
           path: ./hive_assets
-          key: hive-assets-${{ steps.hive-commit.outputs.hash }}-${{ hashFiles('.github/scripts/hive/build_simulators.sh') }}
+          key: hive-assets-${{ matrix.variant }}-${{ steps.hive-commit.outputs.hash }}-${{ hashFiles('.github/scripts/hive/build_simulators.sh') }}
 
       - name: Build hive assets
         if: steps.cache-hive.outputs.cache-hit != 'true'
-        run: .github/scripts/hive/build_simulators.sh
+        run: .github/scripts/hive/build_simulators.sh ${{ matrix.variant }}
 
       - name: Load cached Docker images
         if: steps.cache-hive.outputs.cache-hit == 'true'
@@ -75,16 +85,199 @@ jobs:
           chmod +x hive
 
       - name: Upload hive assets
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
         with:
-          name: hive_assets
+          name: hive_assets_${{ matrix.variant }}
           path: ./hive_assets
-  test:
+  test-amsterdam:
+    timeout-minutes: 120
+    strategy:
+      fail-fast: false
+      matrix:
+        # ethereum/rpc to be deprecated:
+        # https://github.com/ethereum/hive/pull/1117
+        scenario:
+          - sim: smoke/genesis
+          - sim: smoke/network
+          - sim: ethereum/sync
+          - sim: devp2p
+            limit: discv4
+          # started failing after https://github.com/ethereum/go-ethereum/pull/31843, no
+          # action on our side, remove from here when we get unexpected passes on these tests
+          # - sim: devp2p
+          #  limit: eth
+          #  include:
+          #    - MaliciousHandshake
+          #    # failures tracked in https://github.com/paradigmxyz/reth/issues/14825
+          #    - Status
+          #    - GetBlockHeaders
+          #    - ZeroRequestID
+          #    - GetBlockBodies
+          #    - Transaction
+          #    - NewPooledTxs
+          - sim: devp2p
+            limit: discv5
+            include:
+              # failures tracked at https://github.com/paradigmxyz/reth/issues/14825
+              - PingLargeRequestID
+          - sim: ethereum/engine
+            limit: engine-exchange-capabilities
+          - sim: ethereum/engine
+            limit: engine-withdrawals
+          - sim: ethereum/engine
+            limit: engine-auth
+          - sim: ethereum/engine
+            limit: engine-api
+          - sim: ethereum/engine
+            limit: cancun
+          # eth_ rpc methods
+          - sim: ethereum/rpc-compat
+            include:
+              - eth_blockNumber
+              - eth_call
+              - eth_chainId
+              - eth_createAccessList
+              - eth_estimateGas
+              - eth_feeHistory
+              - eth_getBalance
+              - eth_getBlockBy
+              - eth_getBlockTransactionCountBy
+              - eth_getCode
+              - eth_getProof
+              - eth_getStorage
+              - eth_getTransactionBy
+              - eth_getTransactionCount
+              - eth_getTransactionReceipt
+              - eth_sendRawTransaction
+              - eth_syncing
+              # debug_ rpc methods
+              - debug_
+
+          # consume-engine
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/amsterdam.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/osaka.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/prague.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/cancun.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/shanghai.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/berlin.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/istanbul.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/homestead.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/frontier.*
+          - sim: ethereum/eels/consume-engine
+            limit: .*tests/paris.*
+
+          # consume-rlp
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/amsterdam.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/osaka.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/prague.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/cancun.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/shanghai.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/berlin.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/istanbul.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/homestead.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/frontier.*
+          - sim: ethereum/eels/consume-rlp
+            limit: .*tests/paris.*
+    needs:
+      - build-reth
+      - prepare-hive
+    name: Hive-Amsterdam / ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
+    # Use larger runners for eels tests to avoid OOM runner crashes
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Download hive assets
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: hive_assets_amsterdam
+          path: /tmp
+
+      - name: Download reth image
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: reth
+          path: /tmp
+
+      - name: Load Docker images
+        run: .github/scripts/hive/load_images.sh
+
+      - name: Move hive binary
+        run: |
+          mv /tmp/hive /usr/local/bin
+          chmod +x /usr/local/bin/hive
+
+      - name: Checkout hive tests
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          repository: ethereum/hive
+          ref: master
+          path: hivetests
+          persist-credentials: false
+
+      - name: Run simulator
+        env:
+          SCENARIO_SIM: ${{ matrix.scenario.sim }}
+          SCENARIO_LIMIT: ${{ matrix.scenario.limit }}
+          SCENARIO_TESTS: ${{ join(matrix.scenario.include, '|') }}
+        run: |
+          LIMIT="$SCENARIO_LIMIT"
+          TESTS="$SCENARIO_TESTS"
+          if [ -n "$LIMIT" ] && [ -n "$TESTS" ]; then
+            FILTER="$LIMIT/$TESTS"
+          elif [ -n "$LIMIT" ]; then
+            FILTER="$LIMIT"
+          elif [ -n "$TESTS" ]; then
+            FILTER="/$TESTS"
+          else
+            FILTER="/"
+          fi
+          echo "filter: $FILTER"
+          .github/scripts/hive/run_simulator.sh "$SCENARIO_SIM" "$FILTER" "amsterdam"
+
+      - name: Parse hive output
+        run: |
+          find hivetests/workspace/logs -type f -name "*.json" ! -name "hive.json" | xargs -I {} python .github/scripts/hive/parse.py {} --exclusion .github/scripts/hive/expected_failures.yaml --ignored .github/scripts/hive/ignored_tests.yaml
+
+      - name: Print simulator output
+        if: ${{ failure() }}
+        run: |
+          cat hivetests/workspace/logs/*simulator*.log
+
+      - name: Print reth client logs
+        if: ${{ failure() }}
+        run: |
+          cat hivetests/workspace/logs/reth/client-*.log
+
+  test-osaka:
     timeout-minutes: 120
     strategy:
       fail-fast: false
       matrix:
-        storage: [stable, edge]
         # ethereum/rpc to be deprecated:
         # https://github.com/ethereum/hive/pull/1117
         scenario:
@@ -184,29 +377,30 @@ jobs:
           - sim: ethereum/eels/consume-rlp
             limit: .*tests/paris.*
     needs:
-      - build-reth-stable
-      - build-reth-edge
+      - build-reth
       - prepare-hive
-    name: ${{ matrix.storage }} / ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
+    name: Hive-Osaka / ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
     # Use larger runners for eels tests to avoid OOM runner crashes
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
     permissions:
+      contents: read
       issues: write
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
+          persist-credentials: false
           fetch-depth: 0
 
       - name: Download hive assets
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
         with:
-          name: hive_assets
+          name: hive_assets_osaka
           path: /tmp
 
       - name: Download reth image
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
         with:
-          name: reth-${{ matrix.storage }}
+          name: reth
           path: /tmp
 
       - name: Load Docker images
@@ -218,16 +412,21 @@ jobs:
           chmod +x /usr/local/bin/hive
 
       - name: Checkout hive tests
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           repository: ethereum/hive
           ref: master
           path: hivetests
+          persist-credentials: false
 
       - name: Run simulator
+        env:
+          SCENARIO_SIM: ${{ matrix.scenario.sim }}
+          SCENARIO_LIMIT: ${{ matrix.scenario.limit }}
+          SCENARIO_TESTS: ${{ join(matrix.scenario.include, '|') }}
         run: |
-          LIMIT="${{ matrix.scenario.limit }}"
-          TESTS="${{ join(matrix.scenario.include, '|') }}"
+          LIMIT="$SCENARIO_LIMIT"
+          TESTS="$SCENARIO_TESTS"
           if [ -n "$LIMIT" ] && [ -n "$TESTS" ]; then
             FILTER="$LIMIT/$TESTS"
           elif [ -n "$LIMIT" ]; then
@@ -238,7 +437,7 @@ jobs:
             FILTER="/"
           fi
           echo "filter: $FILTER"
-          .github/scripts/hive/run_simulator.sh "${{ matrix.scenario.sim }}" "$FILTER"
+          .github/scripts/hive/run_simulator.sh "$SCENARIO_SIM" "$FILTER" "osaka"
 
       - name: Parse hive output
         run: |
@@ -254,12 +453,14 @@ jobs:
         run: |
           cat hivetests/workspace/logs/reth/client-*.log
   notify-on-error:
-    needs: test
+    needs:
+      - test-amsterdam
+      - test-osaka
     if: failure()
     runs-on: ubuntu-latest
     steps:
       - name: Slack Webhook Action
-        uses: rtCamp/action-slack-notify@v2
+        uses: rtCamp/action-slack-notify@e31e87e03dd19038e411e38ae27cbad084a90661 # v2.3.3
         env:
           SLACK_COLOR: ${{ job.status }}
           SLACK_MESSAGE: "Failed run: https://github.com/paradigmxyz/reth/actions/runs/${{ github.run_id }}"

.github/workflows/integration.yml

@@ -20,34 +20,41 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
+permissions: {}
+
 jobs:
   test:
-    name: test / ${{ matrix.network }} / ${{ matrix.storage }}
+    name: test / ${{ matrix.network }}
     if: github.event_name != 'schedule'
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     env:
       RUST_BACKTRACE: 1
     strategy:
       matrix:
         network: ["ethereum"]
-        storage: ["stable", "edge"]
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
       - name: Install Geth
         run: .github/scripts/install_geth.sh
-      - uses: taiki-e/install-action@nextest
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: nextest
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - name: Run tests
         run: |
           cargo nextest run \
             --no-fail-fast \
-            --locked --features "asm-keccak ${{ matrix.network }} ${{ matrix.storage == 'edge' && 'edge' || '' }}" \
+            --locked --features "asm-keccak ${{ matrix.network }}" \
             --workspace --exclude ef-tests \
             -E "kind(test) and not binary(e2e_testsuite)"
 
@@ -59,7 +66,7 @@ jobs:
     timeout-minutes: 30
     steps:
       - name: Decide whether the needed jobs succeeded or failed
-        uses: re-actors/alls-green@release/v1
+        uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe # release/v1
         with:
           jobs: ${{ toJSON(needs) }}
 
@@ -67,13 +74,19 @@ jobs:
     name: era1 file integration tests once a day
     if: github.event_name == 'schedule' && github.repository == 'paradigmxyz/reth'
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
-      - uses: taiki-e/install-action@nextest
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: nextest
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - name: run era1 files integration tests

.github/workflows/kurtosis.yml

@@ -18,9 +18,14 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
+permissions: {}
+
 jobs:
   build-reth:
     if: github.repository == 'paradigmxyz/reth'
+    permissions:
+      contents: read
+      id-token: write
     uses: ./.github/workflows/docker-test.yml
     with:
       hive_target: kurtosis
@@ -32,15 +37,18 @@ jobs:
       fail-fast: false
     name: run kurtosis
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     needs:
       - build-reth
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
+          persist-credentials: false
           fetch-depth: 0
 
       - name: Download reth image
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
         with:
           name: artifacts
           path: /tmp
@@ -52,7 +60,7 @@ jobs:
           docker image ls -a
 
       - name: Run kurtosis
-        uses: ethpandaops/kurtosis-assertoor-github-action@v1
+        uses: ethpandaops/kurtosis-assertoor-github-action@f64942cbc780df731a731ea9f45765b161d2c8df # v1.0.1
         with:
           ethereum_package_args: ".github/assets/kurtosis_network_params.yaml"
 
@@ -62,7 +70,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Slack Webhook Action
-        uses: rtCamp/action-slack-notify@v2
+        uses: rtCamp/action-slack-notify@e31e87e03dd19038e411e38ae27cbad084a90661 # v2.3.3
         env:
           SLACK_COLOR: ${{ job.status }}
           SLACK_MESSAGE: "Failed run: https://github.com/paradigmxyz/reth/actions/runs/${{ github.run_id }}"

.github/workflows/label-pr.yml

@@ -4,19 +4,23 @@ on:
   pull_request:
     types: [opened]
 
+permissions: {}
+
 jobs:
   label_prs:
     runs-on: ubuntu-latest
     permissions:
+      contents: read
       issues: write
       pull-requests: write
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
+          persist-credentials: false
           fetch-depth: 0
 
       - name: Label PRs
-        uses: actions/github-script@v8
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
         with:
           script: |
             const label_pr = require('./.github/scripts/label_pr.js')

.github/workflows/lint-actions.yml

@@ -8,11 +8,17 @@ on:
     paths:
     - '.github/**'
 
+permissions: {}
+
 jobs:
   actionlint:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - name: Download actionlint
         id: get_actionlint
         run: bash <(curl https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash)

.github/workflows/lint.yml

@@ -10,10 +10,14 @@ env:
   CARGO_TERM_COLOR: always
   RUSTC_WRAPPER: "sccache"
 
+permissions: {}
+
 jobs:
   clippy-binaries:
     name: clippy binaries / ${{ matrix.type }}
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     strategy:
       matrix:
@@ -22,17 +26,19 @@ jobs:
             args: --workspace --lib --examples --tests --benches --locked
             features: "ethereum asm-keccak jemalloc jemalloc-prof min-error-logs min-warn-logs min-info-logs min-debug-logs min-trace-logs"
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@clippy
         with:
           components: clippy
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - if: "${{ matrix.type == 'book' }}"
-        uses: arduino/setup-protoc@v3
+        uses: arduino/setup-protoc@c65c819552d16ad3c9b72d9dfd5ba5237b9c906b # v3.0.0
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
       - name: Run clippy on binaries
@@ -43,15 +49,19 @@ jobs:
   clippy:
     name: clippy
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@nightly
         with:
           components: clippy
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - run: cargo clippy --workspace --lib --examples --tests --benches --all-features --locked
@@ -60,19 +70,25 @@ jobs:
 
   wasm:
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
         with:
           target: wasm32-wasip1
-      - uses: taiki-e/install-action@cargo-hack
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: cargo-hack
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
-      - uses: dcarbone/install-jq-action@v3
+      - uses: dcarbone/install-jq-action@b7ef57d46ece78760b4019dbc4080a1ba2a40b45 # v3.2.0
       - name: Run Wasm checks
         run: |
           sudo apt update && sudo apt install gcc-multilib
@@ -80,53 +96,69 @@ jobs:
 
   riscv:
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
         with:
           target: riscv32imac-unknown-none-elf
-      - uses: taiki-e/install-action@cargo-hack
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: cargo-hack
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
-      - uses: dcarbone/install-jq-action@v3
+      - uses: dcarbone/install-jq-action@b7ef57d46ece78760b4019dbc4080a1ba2a40b45 # v3.2.0
       - name: Run RISC-V checks
         run: .github/scripts/check_rv32imac.sh
 
   crate-checks:
     name: crate-checks (${{ matrix.partition }}/${{ matrix.total_partitions }})
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     strategy:
       matrix:
         partition: [1, 2, 3]
         total_partitions: [3]
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
-      - uses: taiki-e/install-action@cargo-hack
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: cargo-hack
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - run: cargo hack check --workspace --partition ${{ matrix.partition }}/${{ matrix.total_partitions }}
 
   msrv:
     name: MSRV
-    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@master
         with:
           toolchain: "1.93" # MSRV
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - run: cargo build --bin reth --workspace
@@ -136,13 +168,17 @@ jobs:
   docs:
     name: docs
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@nightly
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - run: cargo docs --document-private-items
@@ -154,42 +190,56 @@ jobs:
   fmt:
     name: fmt
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@nightly
         with:
           components: rustfmt
-      - uses: mozilla-actions/sccache-action@v0.0.9
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
       - name: Run fmt
         run: cargo fmt --all --check
 
   udeps:
     name: udeps
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@nightly
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
-      - uses: taiki-e/install-action@cargo-udeps
+      - uses: taiki-e/install-action@1f2425cdb59f8fffb99ee16a5968edf6f57a2b93 # v2.75.24
+        with:
+          tool: cargo-udeps
       - run: cargo udeps --workspace --lib --examples --tests --benches --all-features --locked
 
   book:
     name: book
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@nightly
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
         with:
           cache-on-failure: true
       - run: cargo build --bin reth --workspace
@@ -201,77 +251,72 @@ jobs:
 
   typos:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: crate-ci/typos@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: crate-ci/typos@02ea592e44b3a53c302f697cddca7641cd051c3d # v1.45.0
 
   check-toml:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - name: Run dprint
-        uses: dprint/check@v2.3
+        uses: dprint/check@9cb3a2b17a8e606d37aae341e49df3654933fc23 # v2.3
         with:
           config-path: dprint.json
 
   grafana:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - name: Check dashboard JSON with jq
-        uses: sergeysova/jq-action@v2
+        uses: sergeysova/jq-action@a3f0d4ff59cc1dddf023fc0b325dd75b10deec58 # v2.3.0
         with:
           cmd: jq empty etc/grafana/dashboards/overview.json
 
   no-test-deps:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
       - uses: dtolnay/rust-toolchain@stable
       - name: Ensure no arbitrary or proptest dependency on default build
         run: cargo tree --package reth -e=features,no-dev | grep -Eq "arbitrary|proptest" && exit 1 || exit 0
 
-  # Checks that selected crates can compile with power set of features
-  features:
-    name: features
-    runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
-    timeout-minutes: 30
-    steps:
-      - uses: actions/checkout@v6
-      - uses: rui314/setup-mold@v1
-      - uses: dtolnay/rust-toolchain@clippy
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: Swatinem/rust-cache@v2
-        with:
-          cache-on-failure: true
-      - name: cargo install cargo-hack
-        uses: taiki-e/install-action@cargo-hack
-      - run: |
-          cargo hack check \
-            --package reth-codecs \
-            --package reth-primitives-traits \
-            --package reth-primitives \
-            --feature-powerset \
-            --depth 2
-        env:
-          RUSTFLAGS: -D warnings
-
   # Check crates correctly propagate features
   feature-propagation:
     runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest' || 'ubuntu-latest' }}
+    permissions:
+      contents: read
     timeout-minutes: 20
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
       - uses: dtolnay/rust-toolchain@stable
-      - uses: mozilla-actions/sccache-action@v0.0.9
-      - uses: rui314/setup-mold@v1
-      - uses: taiki-e/cache-cargo-install-action@v3
+      - uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9
+      - uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
+      - uses: taiki-e/cache-cargo-install-action@a8b9ecf8e0c0ea09d7481cfc583a5203ecd585b5 # v3.0.5
         with:
           tool: zepter
       - name: Eagerly pull dependencies
@@ -279,6 +324,8 @@ jobs:
       - run: zepter run check
 
   deny:
+    permissions:
+      contents: read
     uses: tempoxyz/ci/.github/workflows/deny.yml@main
 
   lint-success:
@@ -297,12 +344,11 @@ jobs:
       - typos
       - grafana
       - no-test-deps
-      - features
       - feature-propagation
       - deny
     timeout-minutes: 30
     steps:
       - name: Decide whether the needed jobs succeeded or failed
-        uses: re-actors/alls-green@release/v1
+        uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe # release/v1
         with:
           jobs: ${{ toJSON(needs) }}

.github/workflows/pr-audit.yml

@@ -0,0 +1,39 @@
+name: Pull request audit
+
+on:
+  pull_request:
+    types: [labeled]
+
+permissions: {}
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    if: github.event.label.name == 'cyclops'
+    permissions: {}
+    steps:
+      - name: Publish event
+        env:
+          EVENTS_KEY: ${{ secrets.EVENTS_KEY }}
+          EVENTS_CERT: ${{ secrets.EVENTS_CERT }}
+          EVENTS_ARGS: ${{ secrets.EVENTS_ARGS }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          PR_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+
+          echo "$EVENTS_KEY" > "${{ runner.temp }}/key"
+          echo "$EVENTS_CERT" > "${{ runner.temp }}/cert"
+
+          curl -sf -o /dev/null -X POST $EVENTS_ARGS \
+            -H "Content-Type: application/json" \
+            --key "${{ runner.temp }}/key" \
+            --cert "${{ runner.temp }}/cert" \
+            -d '{
+              "repository": "${{ github.repository }}",
+              "event": "pr_audit",
+              "data": {
+                "pr_number": '"$PR_NUMBER"',
+                "sha": "'"$PR_SHA"'"
+              }
+            }'

.github/workflows/pr-title.yml

@@ -8,20 +8,19 @@ on:
       - edited
       - synchronize
 
-permissions:
-  pull-requests: read
-  contents: read
+permissions: {}
 
 jobs:
   conventional-title:
     name: Validate PR title is Conventional Commit
     runs-on: ubuntu-latest
     permissions:
+      contents: read
       pull-requests: write
     steps:
       - name: Check title
         id: lint_pr_title
-        uses: amannn/action-semantic-pull-request@v6
+        uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
@@ -40,7 +39,7 @@ jobs:
         continue-on-error: true
       - name: Add PR Comment for Invalid Title
         if: steps.lint_pr_title.outcome == 'failure'
-        uses: marocchino/sticky-pull-request-comment@v2
+        uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff # v3.0.3
         with:
           header: pr-title-lint-error
           message: |
@@ -76,7 +75,7 @@ jobs:
 
       - name: Remove Comment for Valid Title
         if: steps.lint_pr_title.outcome == 'success'
-        uses: marocchino/sticky-pull-request-comment@v2
+        uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff # v3.0.3
         with:
           header: pr-title-lint-error
           delete: true