Compare commits

..

1 Commits

Author SHA1 Message Date
Emma Jamieson-Hoare
f04db57d3a chore(ci): continue on error when codspeed is flakey/cancelled on main merge 2026-02-27 09:49:17 +00:00
444 changed files with 11993 additions and 28268 deletions

View File

@@ -1,5 +0,0 @@
---
reth-trie-sparse: patch
---
Fixed a bug in `merge_subtrie_updates` where source insertions did not cancel destination removals (and vice versa), causing inconsistent trie updates accumulated across multiple `root()` calls without intermediate `take_updates()`. Added a test covering the cross-cancellation behavior.

View File

@@ -1,5 +0,0 @@
---
reth-tasks: patch
---
Added panic handler to all rayon thread pools that logs panics via `tracing::error` instead of aborting the process.

View File

@@ -1,5 +0,0 @@
---
reth-network-types: patch
---
Increased default maximum concurrent outbound dials from 15 to 30.

View File

@@ -1,6 +0,0 @@
---
reth-trie-common: minor
reth-trie: minor
---
Added `contains_range` method to `PrefixSet` for checking if any key falls within a half-open range. Added prefix set support to `ProofCalculator` via `with_prefix_set`, enabling stale cached hash invalidation and branch collapse detection when keys are inserted or removed; propagated storage prefix sets through `SyncAccountValueEncoder`.

View File

@@ -1,5 +0,0 @@
---
reth-trie-sparse: patch
---
Refactored arena trie internals by adding a `BranchChildIdx::sibling()` helper, deduplicating `Index`/`NodeArena` type aliases, and replacing `is_empty()` with a `drop_root()` method. Fixed a bug where `cursor.pop()` was called before checking if the leaf was the root node, which could cause incorrect dirty-state propagation.

View File

@@ -1,5 +0,0 @@
---
reth-trie-sparse: minor
---
Fixed a bug in `ArenaParallelSparseTrie` where subtrie updates that would completely empty a subtrie were incorrectly dispatched to parallel workers instead of being processed inline, preventing correct branch collapse detection when blinded siblings are present. Refactored the `SparseTrie` test suite to accept a `fn() -> T` factory instead of requiring `T: Default`, enabling a new `arena_parallel_sparse_trie_always_parallel` test variant that exercises all tests with parallelism thresholds set to 1. Added `test_branch_collapse_multi_empty_subtries_blinded_remaining` to cover the case where removing multiple revealed leaves empties their subtries and leaves a single blinded sibling requiring a proof.

View File

@@ -1,5 +0,0 @@
---
reth-payload-builder: minor
---
Added observability metrics for payload resolve latency and new payload job creation latency to the payload builder service.

View File

@@ -1,5 +0,0 @@
---
reth-trie-sparse: patch
---
Fixed another branch collapse edge case where `check_subtrie_collapse_needs_proof` incorrectly compared removal count against total update count (including `Touched` entries), causing it to skip proof requests for blinded siblings and panic when the subtrie emptied. Added a regression test covering the removals + `Touched` + blinded sibling scenario.

View File

@@ -1,10 +0,0 @@
---
reth-chain-state: minor
reth-engine-primitives: minor
reth-engine-tree: minor
reth-node-core: minor
reth-node-events: minor
reth: patch
---
Added configurable slow block logging (`--engine.slow-block-threshold`) that emits a structured `warn!` log with detailed timing, state-operation counts, and cache hit-rate metrics for blocks whose total processing time exceeds the threshold. Introduced `ExecutionTimingStats`, `CacheStats`, `StateProviderStats`, and `SlowBlockInfo` types to carry execution statistics from block validation through persistence, and refactored `PersistenceResult` to carry commit duration alongside the last persisted block.

View File

@@ -1,6 +0,0 @@
---
reth-trie-db: minor
reth-engine-tree: minor
---
Added `PendingChangeset` and `PendingChangesetGuard` to `ChangesetCache` so concurrent readers wait for an in-progress computation instead of falling back to the expensive DB-based path. The guard automatically cancels the pending entry on drop (e.g. task panic), ensuring waiters always make progress.

View File

@@ -1,8 +0,0 @@
---
reth-engine-primitives: minor
reth-engine-tree: minor
reth-node-core: minor
reth-trie-parallel: minor
---
Added `--engine.proof-jitter` CLI option behind the `trie-debug` feature flag. When set, each proof worker sleeps for a random duration up to the specified value before starting proof computation, useful for stress-testing timing-sensitive proof logic.

View File

@@ -1,5 +0,0 @@
---
reth-rpc-convert: patch
---
Updated `alloy-evm` dependency to git revision `9bc2dba` and adapted `TxEnvConverter` impl to the updated `TryIntoTxEnv` trait signature that now includes a `Spec` generic parameter.

View File

@@ -1,6 +0,0 @@
---
reth-trie: patch
reth-trie-sparse: patch
---
Refactored test harness for sparse trie tests by extracting `TrieTestHarness` into a shared `reth-trie` test utility, replacing duplicated inline harness code across multiple test modules. Updated `proof_v2` return type to include an optional root hash, and converted `original_root` and `storage` from public fields to accessor methods.

View File

@@ -1,5 +0,0 @@
---
reth-trie: patch
---
Removed the local `increment_and_strip_trailing_zeros` function and `PATH_ALL_ZEROS` static in `proof_v2`, replacing them with the equivalent `Nibbles::next_without_prefix` and `Nibbles::is_zeroes` builtins. Also replaced manual `.get()` calls on `state_mask`/`hash_mask` with direct field access and switched to `Nibbles::unpack_array` over the unsafe `unpack_unchecked`.

View File

@@ -1,5 +0,0 @@
---
reth-engine-tree: patch
---
Added idle-time pre-computation of account trie upper hashes in the sparse trie payload processor when no pending proof results are available.

View File

@@ -1,7 +0,0 @@
---
reth-cli-commands: minor
reth-node-core: minor
reth: patch
---
Made v2 storage the default for all new databases, deprecating the `--storage.v2` flag to a hidden no-op kept for backwards compatibility. Updated CLI reference docs to remove the now-hidden flag from all command help pages.

View File

@@ -1,7 +0,0 @@
---
reth-engine-tree: patch
reth-trie-sparse: patch
reth-tasks: patch
---
Offloaded deallocation of expensive proof node buffers to a persistent background thread (`Runtime::spawn_drop`) to avoid blocking state root computation or lock-holding code.

View File

@@ -1,6 +0,0 @@
---
reth-trie-sparse: patch
reth-engine-tree: patch
---
Removed the `skip_proof_node_filtering` flag, `revealed_account_paths`/`revealed_paths` tracking, and the `filter_revealed_v2_proof_nodes` function from the sparse trie implementation. Also removed the corresponding skipped-nodes metrics, simplifying the proof node reveal path to always pass nodes directly to the sparse trie without pre-filtering.

View File

@@ -1,5 +0,0 @@
---
reth-trie-sparse: minor
---
Added a comprehensive generic `SparseTrie` test suite covering `set_root`, `reveal_nodes`, `update_leaves`, `root`, `take_updates`, `commit_updates`, `prune`, `wipe`/`clear`, `get_leaf_value`, `find_leaf`, `size_hint`, and integration lifecycle scenarios. Tests are stamped out for all concrete `SparseTrie` implementations via a macro.

View File

@@ -1,5 +0,0 @@
---
reth-cli-commands: minor
---
Added `reth_version` field to `SnapshotManifest` to record the Reth version that produced a snapshot. The field is optional and populated automatically during manifest generation.

View File

@@ -1,8 +0,0 @@
---
reth-trie-common: minor
reth-trie: minor
reth-trie-parallel: minor
reth-engine-tree: patch
---
Moved `ProofV2Target`, `MultiProofTargetsV2`, and `ChunkedMultiProofTargetsV2` from `reth-trie-parallel::targets_v2` into a new `reth-trie-common::target_v2` module, making these types available at a lower level without pulling in the full parallel trie crate. Added a `multiproof_v2` method to `Proof` in `reth-trie` that generates a state multiproof using the V2 proof calculator with synchronous account value encoding.

View File

@@ -1,14 +0,0 @@
---
reth-engine-tree: patch
reth-evm-ethereum: patch
reth-evm: patch
reth-primitives-traits: patch
reth-revm: patch
reth-rpc-eth-api: patch
reth-rpc-eth-types: patch
reth-provider: patch
example-custom-evm: patch
example-precompile-cache: patch
---
Bumped revm to v35.0.0, revm-inspectors to 0.35.0, and alloy-evm to 0.29.0. Updated call sites throughout the codebase to align with the new APIs, including `ExecutionResult` field changes (`gas_used``gas.used()`/`gas.final_refunded()`), removal of `.without_state_clear()`, updated `EthPrecompiles::new(spec)` constructor, and updated `block_hashes.lowest()` access.

View File

@@ -1,9 +0,0 @@
---
reth-trie-sparse: minor
reth-engine-primitives: minor
reth-engine-tree: minor
reth-node-core: minor
reth-trie-common: patch
---
Added an arena-based sparse trie implementation (`ArenaParallelSparseTrie`) using `slotmap` arena allocation for node storage, enabling parallel subtrie mutation without per-node hashing overhead. Added `ConfigurableSparseTrie` enum to switch between the arena and hash-map implementations, and a `--engine.enable-arena-sparse-trie` CLI flag to opt in at runtime.

View File

@@ -1,7 +0,0 @@
---
reth-trie: major
reth-trie-db: major
reth-provider: minor
---
Added `MaskedTrieCursorFactory` and `MaskedTrieCursor` to handle prefix-set-based hash invalidation at the cursor layer, replacing the `DatabaseTrieWitness` trait abstraction. Removed `with_prefix_sets_mut` from `TrieWitness` and deleted `DatabaseTrieWitness` — callers should now wrap their cursor factory with `MaskedTrieCursorFactory` to apply prefix sets during witness/proof computation.

View File

@@ -1,5 +0,0 @@
---
reth-trie: patch
---
Fixed a potential panic in `ProofCalculator` by clearing internal computation state (`branch_stack`, `child_stack`, `branch_path`, etc.) after errors, preventing stale state from causing `usize` underflow panics when the calculator is reused. Added a test verifying correct behavior after simulated mid-computation errors.

View File

@@ -20,11 +20,6 @@
# include dist directory, where the reth binary is located after compilation
!/dist
# include PGO build helper used by Dockerfile.depot
!/.github
!/.github/scripts
!/.github/scripts/build_pgo_bolt.sh
# include licenses
!LICENSE-*

View File

@@ -5,4 +5,3 @@ self-hosted-runner:
- depot-ubuntu-latest-4
- depot-ubuntu-latest-8
- depot-ubuntu-latest-16
- available

View File

@@ -1,106 +0,0 @@
// Generates a rich GitHub Actions job summary for reth-bench results.
//
// Reads from environment:
// BENCH_WORK_DIR Directory containing summary.json
// BENCH_PR PR number (may be empty)
// BENCH_ACTOR GitHub user who triggered the bench
// BENCH_CORES CPU core limit (0 = all)
// BENCH_WARMUP_BLOCKS Number of warmup blocks
// BENCH_SAMPLY 'true' if samply profiling was enabled
// BENCH_ABBA 'true' if ABBA interleaved order was used
//
// Usage from actions/github-script:
// const jobSummary = require('./.github/scripts/bench-job-summary.js');
// await jobSummary({ core, context, chartSha, grafanaUrl, runId });
const fs = require('fs');
const { verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
module.exports = async function ({ core, context, chartSha, grafanaUrl, runId }) {
let summary;
try {
summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
} catch (e) {
await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
return;
}
const repo = `${context.repo.owner}/${context.repo.repo}`;
const prNumber = process.env.BENCH_PR;
const actor = process.env.BENCH_ACTOR;
const commitUrl = `https://github.com/${repo}/commit`;
const { emoji, label } = verdict(summary.changes);
const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
// Header & metadata
const metaParts = [];
if (prNumber) metaParts.push(`**[PR #${prNumber}](https://github.com/${repo}/pull/${prNumber})**`);
metaParts.push(`triggered by @${actor}`);
let md = `# ${emoji} ${label}\n\n`;
md += metaParts.join(' · ') + '\n\n';
md += `**Baseline:** ${baselineLink}\n`;
md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
// Main comparison table
const rows = metricRows(summary);
md += `| Metric | Baseline | Feature | Change |\n`;
md += `|--------|----------|---------|--------|\n`;
for (const r of rows) {
md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
}
md += '\n';
// Wait time breakdown
const wtRows = waitTimeRows(summary);
if (wtRows.length > 0) {
md += `### Wait Time Breakdown\n\n`;
md += `| Metric | Baseline | Feature |\n`;
md += `|--------|----------|--------|\n`;
for (const r of wtRows) {
md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
}
md += '\n';
}
// Charts
if (chartSha) {
const prNum = prNumber || '0';
const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/pr/${prNum}/${runId}`;
const charts = [
{ file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
{ file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
{ file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
];
md += `### Charts\n\n`;
for (const chart of charts) {
md += `<details><summary>${chart.label}</summary>\n\n`;
md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
md += `</details>\n\n`;
}
}
// Samply profiles
const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
const samplyLinks = Object.entries(samplyUrls).map(([run, url]) => `- **${run}**: [Firefox Profiler](${url})`);
if (samplyLinks.length > 0) {
md += `### Samply Profiles\n\n${samplyLinks.join('\n')}\n\n`;
}
// Grafana
if (grafanaUrl) {
md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
}
// Node errors
try {
const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
if (errors.trim()) md += '\n' + errors + '\n';
} catch {}
await core.summary.addRaw(md).write();
};

View File

@@ -1,276 +0,0 @@
#!/usr/bin/env python3
"""
Prometheus metrics proxy that fetches from a local reth node and
re-exposes with additional benchmark labels.
Reads labels from a JSON file (updated by local-reth-bench.sh between runs)
and injects them into every Prometheus metric line.
Returns empty 200 when reth is not running (clean Grafana gaps).
"""
import argparse
import ipaddress
import json
import subprocess
import sys
import time
from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib.request import urlopen
from urllib.error import URLError
def read_labels(path):
try:
with open(path) as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return {}
def inject_labels(metrics_bytes, label_str, label_names):
"""Inject labels into Prometheus text format.
Operates on bytes and uses simple string ops instead of regex
for speed on large payloads (reth exposes thousands of metrics).
Skips injecting into lines that already contain any of the label names
to avoid duplicate labels (which Prometheus rejects).
"""
if not label_str:
return metrics_bytes
label_bytes = label_str.encode("utf-8")
# Pre-encode label names for fast duplicate detection
label_name_bytes = [n.encode("utf-8") for n in label_names]
out = []
for line in metrics_bytes.split(b"\n"):
# Skip comments and blank lines
if line.startswith(b"#") or not line:
out.append(line)
continue
brace = line.find(b"{")
space = line.find(b" ")
if space == -1:
# Malformed, pass through
out.append(line)
elif brace != -1 and brace < space:
# Has labels: metric{existing="val"} 123
close = line.find(b"}", brace)
if close == -1:
out.append(line)
continue
# Filter out labels that already exist in this line
existing = line[brace + 1:close]
inject = label_bytes
if existing:
for name in label_name_bytes:
if name + b"=" in existing:
# Rebuild inject string excluding this label
inject = _remove_label(inject, name)
if not inject:
out.append(line)
continue
if close == brace + 1:
# Empty braces: metric{} 123
out.append(line[:close] + inject + line[close:])
else:
out.append(line[:close] + b"," + inject + line[close:])
else:
# No labels: metric 123
out.append(line[:space] + b"{" + label_bytes + b"}" + line[space:])
return b"\n".join(out)
def _remove_label(label_bytes, name):
"""Remove a single label (name=\"...\") from a comma-separated label string."""
parts = []
for part in label_bytes.split(b","):
if not part.startswith(name + b"="):
parts.append(part)
return b",".join(parts)
def build_label_str(labels):
"""Pre-format the label injection string: key1="val1",key2="val2" """
if not labels:
return ""
return ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
def build_elapsed_gauge(labels):
"""Build a bench_elapsed_seconds gauge from run_start_epoch in labels."""
start = labels.get("run_start_epoch")
if not start:
return b""
try:
elapsed = time.time() - float(start)
except (ValueError, TypeError):
return b""
# Build labels excluding internal keys
display = {k: v for k, v in labels.items()
if k not in ("run_start_epoch", "reference_epoch")}
lstr = build_label_str(display)
return (
f"# HELP bench_elapsed_seconds Seconds since benchmark run started\n"
f"# TYPE bench_elapsed_seconds gauge\n"
f"bench_elapsed_seconds{{{lstr}}} {elapsed:.1f}\n"
).encode("utf-8")
def compute_timestamp_ms(labels):
"""Compute a synthetic timestamp so all runs share a common time origin.
Returns the timestamp in milliseconds, or None if not enough info.
Uses: reference_epoch + (now - run_start_epoch) → all runs overlay at
the same Grafana time range.
"""
ref = labels.get("reference_epoch")
start = labels.get("run_start_epoch")
if not ref or not start:
return None
try:
elapsed = time.time() - float(start)
return int((float(ref) + elapsed) * 1000)
except (ValueError, TypeError):
return None
def inject_timestamps(metrics_bytes, timestamp_ms):
"""Append a Prometheus timestamp (ms) to every data line.
Prometheus text format: metric{labels} value [timestamp_ms]
Adding timestamps causes Prometheus to store all runs' samples
at the same relative time, enabling natural overlay in Grafana.
"""
if timestamp_ms is None:
return metrics_bytes
ts = str(timestamp_ms).encode("utf-8")
out = []
for line in metrics_bytes.split(b"\n"):
if line.startswith(b"#") or not line:
out.append(line)
else:
out.append(line + b" " + ts)
return b"\n".join(out)
class MetricsHandler(BaseHTTPRequestHandler):
# Use HTTP/1.1 so Content-Length is respected and Prometheus
# doesn't have to rely on connection close to detect end of body.
protocol_version = "HTTP/1.1"
def do_GET(self):
src = self.client_address[0]
try:
resp = urlopen(self.server.upstream, timeout=2)
metrics = resp.read()
except (URLError, ConnectionError, OSError):
# reth not running — return empty 200
self._send(b"")
#print(f" scrape from {src}: empty (reth not running)", flush=True)
return
all_labels = read_labels(self.server.labels_file)
# Internal keys — not injected as Prometheus labels
internal = ("run_start_epoch", "reference_epoch")
labels = {k: v for k, v in all_labels.items() if k not in internal}
label_str = build_label_str(labels)
label_names = sorted(labels.keys())
t0 = time.monotonic()
result = inject_labels(metrics, label_str, label_names)
result += build_elapsed_gauge(all_labels)
ts_ms = compute_timestamp_ms(all_labels)
result = inject_timestamps(result, ts_ms)
dt = time.monotonic() - t0
self._send(result)
print(f" scrape from {src}: {len(metrics)} -> {len(result)} bytes, "
f"inject {dt*1000:.1f}ms", flush=True)
def _send(self, body):
self.send_response(200)
self.send_header("Content-Type", "text/plain; version=0.0.4")
self.send_header("Content-Length", str(len(body)))
self.send_header("Connection", "close")
self.end_headers()
if body:
self.wfile.write(body)
def log_message(self, format, *args):
pass # suppress per-request logging
def resolve_bind_address(subnet_cidr):
"""Find the local IP address that belongs to the given subnet.
Uses ``ip -j addr show`` to enumerate interfaces and returns the first
address that falls within *subnet_cidr* (e.g. ``10.10.0.0/24``).
"""
network = ipaddress.ip_network(subnet_cidr, strict=False)
try:
result = subprocess.run(
["ip", "-j", "addr", "show"],
capture_output=True, text=True, check=True,
)
interfaces = json.loads(result.stdout)
except (subprocess.CalledProcessError, FileNotFoundError, json.JSONDecodeError) as exc:
print(f"Error: cannot enumerate interfaces: {exc}", file=sys.stderr)
sys.exit(1)
for iface in interfaces:
for addr_info in iface.get("addr_info", []):
try:
addr = ipaddress.ip_address(addr_info["local"])
except (KeyError, ValueError):
continue
if addr in network:
return str(addr)
print(f"Error: no interface address found in subnet {subnet_cidr}", file=sys.stderr)
sys.exit(1)
def main():
parser = argparse.ArgumentParser(description="Prometheus metrics proxy with label injection")
parser.add_argument("--labels", default="/tmp/bench-metrics-labels.json",
help="Path to JSON file with labels to inject (default: /tmp/bench-metrics-labels.json)")
parser.add_argument("--upstream", default="http://127.0.0.1:9100/",
help="Upstream reth metrics URL (default: http://127.0.0.1:9100/)")
bind_group = parser.add_mutually_exclusive_group()
bind_group.add_argument("--bind", default=None,
help="Address to bind the proxy (default: 0.0.0.0)")
bind_group.add_argument("--subnet", default=None,
help="Auto-detect bind address from a local interface in this subnet (e.g. 10.10.0.0/24)")
parser.add_argument("--port", type=int, default=9090,
help="Port to bind the proxy (default: 9090)")
args = parser.parse_args()
if args.subnet:
bind_addr = resolve_bind_address(args.subnet)
elif args.bind:
bind_addr = args.bind
else:
bind_addr = "0.0.0.0"
server = HTTPServer((bind_addr, args.port), MetricsHandler)
server.upstream = args.upstream
server.labels_file = args.labels
print(f"bench-metrics-proxy listening on {bind_addr}:{args.port}")
print(f" upstream: {args.upstream}")
print(f" labels: {args.labels}")
sys.stdout.flush()
server.serve_forever()
if __name__ == "__main__":
main()

View File

@@ -22,22 +22,6 @@ MODE="$1"
SOURCE_DIR="$2"
COMMIT="$3"
# Tracy support: when BENCH_TRACY is "on" or "full", add Tracy cargo features
# and frame pointers for accurate stack traces.
EXTRA_FEATURES=""
EXTRA_RUSTFLAGS=""
if [ "${BENCH_TRACY:-off}" != "off" ]; then
EXTRA_FEATURES="tracy,tracy-client/ondemand"
EXTRA_RUSTFLAGS=" -C force-frame-pointers=yes"
fi
# Cache suffix: hash of features+rustflags so different build configs get separate cache entries
if [ -n "$EXTRA_FEATURES" ] || [ -n "$EXTRA_RUSTFLAGS" ]; then
BUILD_SUFFIX="-$(echo "${EXTRA_FEATURES}${EXTRA_RUSTFLAGS}" | sha256sum | cut -c1-12)"
else
BUILD_SUFFIX=""
fi
# Verify a cached reth binary was built from the expected commit.
# `reth --version` outputs "Commit SHA: <full-sha>" on its own line.
verify_binary() {
@@ -58,7 +42,7 @@ verify_binary() {
case "$MODE" in
baseline|main)
BUCKET="minio/reth-binaries/${COMMIT}${BUILD_SUFFIX}"
BUCKET="minio/reth-binaries/${COMMIT}"
mkdir -p "${SOURCE_DIR}/target/profiling"
CACHE_VALID=false
@@ -75,23 +59,14 @@ case "$MODE" in
if [ "$CACHE_VALID" = false ]; then
echo "Building baseline (${COMMIT}) from source..."
cd "${SOURCE_DIR}"
FEATURES_ARG=""
WORKSPACE_ARG=""
if [ -n "$EXTRA_FEATURES" ]; then
# --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
FEATURES_ARG="--features ${EXTRA_FEATURES}"
WORKSPACE_ARG="--workspace"
fi
# shellcheck disable=SC2086
RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
cargo build --profile profiling --bin reth $WORKSPACE_ARG $FEATURES_ARG
cargo build --profile profiling --bin reth
$MC cp target/profiling/reth "${BUCKET}/reth"
fi
;;
feature|branch)
BRANCH_SHA="${4:-$COMMIT}"
BUCKET="minio/reth-binaries/${BRANCH_SHA}${BUILD_SUFFIX}"
BUCKET="minio/reth-binaries/${BRANCH_SHA}"
CACHE_VALID=false
if $MC stat "${BUCKET}/reth" &>/dev/null && $MC stat "${BUCKET}/reth-bench" &>/dev/null; then
@@ -110,14 +85,7 @@ case "$MODE" in
echo "Building feature (${COMMIT}) from source..."
cd "${SOURCE_DIR}"
rustup show active-toolchain || rustup default stable
if [ -n "$EXTRA_FEATURES" ]; then
# Can't use `make profiling` when adding features; build explicitly
# --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
cargo build --profile profiling --workspace --bin reth --features "${EXTRA_FEATURES}"
else
make profiling
fi
make profiling
make install-reth-bench
$MC cp target/profiling/reth "${BUCKET}/reth"
$MC cp "$(which reth-bench)" "${BUCKET}/reth-bench"

View File

@@ -1,581 +0,0 @@
#!/usr/bin/env bash
#
# local-reth-bench.sh — Run the reth Engine API benchmark locally.
#
# Replicates the CI bench.yml workflow (build, snapshot, system tuning,
# interleaved B-F-F-B execution, summary, charts) without any GitHub
# Actions glue (no PR comments, no artifact upload, no Slack).
#
# Usage:
# local-reth-bench.sh <baseline-ref> <feature-ref> [options]
#
# Options:
# --blocks N Number of blocks to benchmark (default: 500)
# --warmup N Number of warmup blocks (default: 100)
# --cores N Limit reth to N CPU cores, 0 = all available (default: 0)
# --samply Enable samply profiling
# --tracy MODE Tracy profiling: off, on, full (default: off)
# --tracy-filter F Tracy tracing filter (default: debug)
# --no-tune Skip system tuning (useful on dev machines / macOS)
#
# Requires: the reth repo at RETH_REPO (default: ~/reth)
#
# Dependencies (install before first run):
# mc (MinIO client), schelk, cpupower, taskset, stdbuf, python3, curl,
# make, uv, pzstd, jq, Rust toolchain (cargo/rustup)
#
# The script delegates to the existing bench-reth-*.sh scripts in the reth
# repo for the actual build, snapshot, and run steps.
set -euo pipefail
# ── PATH ──────────────────────────────────────────────────────────────
# Ensure cargo and user-local bins (mc, uv) are visible
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
# ── Defaults ──────────────────────────────────────────────────────────
RETH_REPO="${RETH_REPO:-$HOME/reth}"
BLOCKS=500
WARMUP=100
CORES=0
SAMPLY=false
TRACY="off"
TRACY_FILTER="debug"
TUNE=true
BASELINE_REF=""
FEATURE_REF=""
# ── Parse arguments ──────────────────────────────────────────────────
usage() {
cat <<EOF
Usage: $(basename "$0") <baseline-ref> <feature-ref> [options]
Options:
--blocks N Number of blocks to benchmark (default: 500)
--warmup N Number of warmup blocks (default: 100)
--cores N Limit reth to N CPU cores (default: 0 = all)
--samply Enable samply profiling
--tracy MODE Tracy profiling: off, on, full (default: off)
on = tracing only (lower overhead)
full = tracing + CPU sampling (higher overhead)
--tracy-filter F Tracy tracing filter (default: debug)
--no-tune Skip system tuning
EOF
exit 1
}
while [[ $# -gt 0 ]]; do
case "$1" in
--blocks) BLOCKS="$2"; shift 2 ;;
--warmup) WARMUP="$2"; shift 2 ;;
--cores) CORES="$2"; shift 2 ;;
--samply) SAMPLY=true; shift ;;
--tracy) TRACY="$2"; shift 2 ;;
--tracy-filter) TRACY_FILTER="$2"; shift 2 ;;
--no-tune) TUNE=false; shift ;;
--help|-h) usage ;;
-*) echo "Unknown option: $1"; usage ;;
*)
if [ -z "$BASELINE_REF" ]; then
BASELINE_REF="$1"
elif [ -z "$FEATURE_REF" ]; then
FEATURE_REF="$1"
else
echo "Unexpected argument: $1"; usage
fi
shift
;;
esac
done
if [ -z "$BASELINE_REF" ] || [ -z "$FEATURE_REF" ]; then
echo "Error: both <baseline-ref> and <feature-ref> are required."
usage
fi
# Validate --tracy value
case "$TRACY" in
off|on|full) ;;
*) echo "Error: --tracy must be off, on, or full (got: $TRACY)"; usage ;;
esac
# Samply + tracy=full are mutually exclusive (both use perf sampling)
if [ "$SAMPLY" = "true" ] && [ "$TRACY" = "full" ]; then
echo "Warning: samply and tracy=full both use perf sampling; downgrading tracy to 'on'."
TRACY="on"
fi
# ── Check dependencies ───────────────────────────────────────────────
missing=()
for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq cargo; do
command -v "$cmd" &>/dev/null || missing+=("$cmd")
done
if [ ${#missing[@]} -gt 0 ]; then
echo "Error: missing required tools: ${missing[*]}"
echo "See the CI 'Install dependencies' step in .github/workflows/bench.yml for install instructions."
exit 1
fi
if [ "$TRACY" != "off" ]; then
if ! command -v tracy-capture &>/dev/null; then
echo "Error: tracy-capture is required for --tracy $TRACY"
exit 1
fi
fi
# Ensure tools that run via sudo are in a sudo-visible path.
# The bench scripts use `sudo schelk` / `sudo samply` but cargo installs
# them to ~/.cargo/bin which sudo's secure_path doesn't include.
for cmd in schelk samply; do
if command -v "$cmd" &>/dev/null && ! sudo sh -c "command -v $cmd" &>/dev/null; then
echo "Installing $cmd to /usr/local/bin (needed for sudo)..."
sudo install "$(command -v "$cmd")" /usr/local/bin/
fi
done
if [ ! -d "$RETH_REPO/.git" ]; then
echo "Error: RETH_REPO=$RETH_REPO is not a git repository."
echo "Set RETH_REPO or clone reth to ~/reth"
exit 1
fi
# ── Resolve paths ────────────────────────────────────────────────────
SELF_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPTS_DIR="${RETH_REPO}/.github/scripts"
BENCH_WORK_DIR="${RETH_REPO}/../bench-work-$(date +%Y%m%d-%H%M%S)"
BASELINE_SRC="${RETH_REPO}/../reth-baseline"
FEATURE_SRC="${RETH_REPO}/../reth-feature"
mkdir -p "$BENCH_WORK_DIR"
BENCH_WORK_DIR="$(cd "$BENCH_WORK_DIR" && pwd)"
# ── Global cleanup trap (restores system tuning on any exit) ─────────
TUNING_APPLIED=false
CSTATE_PID=
METRICS_PROXY_PID=
cleanup_global() {
[ -n "$METRICS_PROXY_PID" ] && kill "$METRICS_PROXY_PID" 2>/dev/null || true
if [ "$TUNING_APPLIED" = true ]; then
echo
echo "▸ Restoring system settings..."
[ -n "$CSTATE_PID" ] && kill "$CSTATE_PID" 2>/dev/null || true
sudo systemctl start irqbalance cron atd 2>/dev/null || true
echo " System settings restored."
fi
}
trap cleanup_global EXIT
echo "═══════════════════════════════════════════════════════════"
echo " reth local benchmark"
echo "═══════════════════════════════════════════════════════════"
echo " Baseline ref : $BASELINE_REF"
echo " Feature ref : $FEATURE_REF"
echo " Blocks : $BLOCKS"
echo " Warmup : $WARMUP"
echo " Cores : $CORES"
echo " Samply : $SAMPLY"
echo " Tracy : $TRACY"
echo " Tracy filter : $TRACY_FILTER"
echo " System tune : $TUNE"
echo " Work dir : $BENCH_WORK_DIR"
echo " Reth repo : $RETH_REPO"
echo "═══════════════════════════════════════════════════════════"
echo
# Enable sccache if available (matches CI's RUSTC_WRAPPER=sccache)
if command -v sccache &>/dev/null; then
export RUSTC_WRAPPER="sccache"
fi
# Export env vars expected by the bench-reth-*.sh scripts
export BENCH_BLOCKS="$BLOCKS"
export BENCH_WARMUP_BLOCKS="$WARMUP"
export BENCH_CORES="$CORES"
export BENCH_SAMPLY="$SAMPLY"
export BENCH_TRACY="$TRACY"
export BENCH_TRACY_FILTER="$TRACY_FILTER"
export BENCH_WORK_DIR
export SCHELK_MOUNT="${SCHELK_MOUNT:-/reth-bench}"
export BENCH_RPC_URL="${BENCH_RPC_URL:-https://ethereum.reth.rs/rpc}"
export BENCH_METRICS_ADDR="127.0.0.1:9100"
# ── Step 1: Resolve refs to full SHAs ────────────────────────────────
echo "▸ Resolving git refs..."
cd "$RETH_REPO"
resolve_ref() {
local ref="$1"
git fetch origin "$ref" --quiet 2>/dev/null || true
git rev-parse "$ref" 2>/dev/null \
|| git rev-parse "origin/$ref" 2>/dev/null \
|| { echo "Error: cannot resolve ref '$ref'"; exit 1; }
}
BASELINE_SHA="$(resolve_ref "$BASELINE_REF")"
FEATURE_SHA="$(resolve_ref "$FEATURE_REF")"
echo " Baseline SHA : $BASELINE_SHA"
echo " Feature SHA : $FEATURE_SHA"
echo
# ── Step 2: Prepare source directories ───────────────────────────────
echo "▸ Preparing source directories..."
prepare_source() {
local src_dir="$1" ref="$2"
if [ -d "$src_dir" ]; then
git -C "$src_dir" fetch origin "$ref" 2>/dev/null || true
else
git clone --recurse-submodules "$RETH_REPO" "$src_dir"
fi
git -C "$src_dir" checkout "$ref" --force
git -C "$src_dir" submodule update --init --recursive
}
prepare_source "$BASELINE_SRC" "$BASELINE_SHA"
prepare_source "$FEATURE_SRC" "$FEATURE_SHA"
BASELINE_SRC="$(cd "$BASELINE_SRC" && pwd)"
FEATURE_SRC="$(cd "$FEATURE_SRC" && pwd)"
echo " Baseline src : $BASELINE_SRC"
echo " Feature src : $FEATURE_SRC"
echo
# ── Step 3: Check / download snapshot ────────────────────────────────
echo "▸ Checking snapshot..."
cd "$RETH_REPO"
SNAPSHOT_NEEDED=false
if ! "${SCRIPTS_DIR}/bench-reth-snapshot.sh" --check; then
SNAPSHOT_NEEDED=true
echo " Snapshot needs update."
else
echo " Snapshot is up-to-date."
fi
echo
# ── Step 4: Build binaries (+ snapshot download) in parallel ─────────
echo "▸ Building binaries (parallel)..."
cd "$RETH_REPO"
FAIL=0
"${SCRIPTS_DIR}/bench-reth-build.sh" baseline "$BASELINE_SRC" "$BASELINE_SHA" &
PID_BASELINE=$!
"${SCRIPTS_DIR}/bench-reth-build.sh" feature "$FEATURE_SRC" "$FEATURE_SHA" &
PID_FEATURE=$!
PID_SNAPSHOT=
if [ "$SNAPSHOT_NEEDED" = "true" ]; then
echo " Also downloading snapshot in parallel..."
"${SCRIPTS_DIR}/bench-reth-snapshot.sh" &
PID_SNAPSHOT=$!
fi
wait $PID_BASELINE || FAIL=1
wait $PID_FEATURE || FAIL=1
[ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
if [ $FAIL -ne 0 ]; then
echo "Error: one or more parallel tasks failed (builds / snapshot)"
exit 1
fi
echo " Binaries built successfully."
echo
# ── Step 5: System tuning (optional) ────────────────────────────────
if [ "$TUNE" = "true" ]; then
echo "▸ Applying system tuning..."
sudo cpupower frequency-set -g performance 2>/dev/null || true
# Disable turbo boost (Intel + AMD)
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
sudo swapoff -a 2>/dev/null || true
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space 2>/dev/null || true
# Disable SMT (hyperthreading)
for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
[ -f "$cpu" ] || continue
first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
if [ "$current" != "$first" ]; then
echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" 2>/dev/null || true
fi
done
echo " Online CPUs: $(nproc)"
# Disable transparent huge pages
for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
if [ -d "$p" ]; then
echo never | sudo tee "$p/enabled" 2>/dev/null || true
echo never | sudo tee "$p/defrag" 2>/dev/null || true
break
fi
done
# Prevent deep C-states
sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
CSTATE_PID=$!
# Pin IRQs to core 0
for irq in /proc/irq/*/smp_affinity_list; do
echo 0 | sudo tee "$irq" 2>/dev/null || true
done
# Stop noisy background services
sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
TUNING_APPLIED=true
# Log environment for reproducibility (matches CI)
echo " === Benchmark environment ==="
echo " Kernel : $(uname -r)"
lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA' | sed 's/^/ /'
echo " Governor : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo unknown)"
echo " Freq : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null || echo unknown)"
echo " THP : $(cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo unknown)"
free -h | sed 's/^/ /'
echo " System tuning applied."
echo
fi
# ── Step 5b: Tracefs mount (tracy=full only) ─────────────────────────
if [ "$TRACY" = "full" ] && [ "$(uname)" = "Linux" ]; then
echo "▸ Mounting tracefs for Tracy full mode..."
sudo mount -t tracefs tracefs /sys/kernel/tracing -o mode=755 2>/dev/null || true
fi
# ── Tracy upload & viewer helpers ────────────────────────────────────
TRACY_VIEWER_BASE="${TRACY_VIEWER_BASE:-}"
tracy_viewer_url() {
local profile_url="$1"
if [ -z "$TRACY_VIEWER_BASE" ]; then
echo ""
return
fi
local encoded
encoded=$(python3 -c "import urllib.parse, sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$profile_url")
echo "${TRACY_VIEWER_BASE}?profile_url=${encoded}"
}
upload_tracy() {
local label="$1" output_dir="$2" sha="$3"
local tracy_file="$output_dir/tracy-profile.tracy"
if [ ! -f "$tracy_file" ]; then
echo " Tracy: no profile found, skipping upload."
return
fi
local timestamp short_sha remote_name bucket mc_alias
timestamp=$(date +%Y%m%d-%H%M%S)
short_sha="${sha:0:7}"
remote_name="${label}-${short_sha}-${timestamp}.tracy"
bucket="${TRACY_BUCKET:-tracy-profiles}"
mc_alias="${MC_ALIAS:-minio}"
local minio_base="${TRACY_MINIO_URL:-http://minio.minio.svc.cluster.local:9000}"
echo " Tracy: uploading profile..."
if mc cp "$tracy_file" "${mc_alias}/${bucket}/${remote_name}"; then
local url="${minio_base}/${bucket}/${remote_name}"
echo "$url" > "$output_dir/tracy_url.txt"
local viewer
viewer=$(tracy_viewer_url "$url")
if [ -n "$viewer" ]; then
echo "$viewer" > "$output_dir/tracy_viewer_url.txt"
echo " Tracy: uploaded → $viewer"
else
echo " Tracy: uploaded → $url"
fi
else
echo " Tracy: upload failed (non-fatal)."
fi
# Delete large profile to free disk
rm -f "$tracy_file"
}
# ── Step 6: Pre-flight cleanup ───────────────────────────────────────
echo "▸ Pre-flight cleanup..."
pkill -f bench-metrics-proxy 2>/dev/null || true
sudo pkill -9 reth 2>/dev/null || true
sleep 1
if mountpoint -q "$SCHELK_MOUNT" 2>/dev/null; then
sudo umount -l "$SCHELK_MOUNT" 2>/dev/null || true
sudo schelk recover -y 2>/dev/null || true
fi
echo
# ── Step 7: Interleaved benchmark runs (B-F-F-B) ────────────────────
# This ordering reduces systematic bias from thermal drift and cache warming.
BASELINE_BIN="${BASELINE_SRC}/target/profiling/reth"
FEATURE_BIN="${FEATURE_SRC}/target/profiling/reth"
# Start metrics proxy (reth → label injection → Prometheus)
LABELS_FILE="/tmp/bench-metrics-labels.json"
echo '{}' > "$LABELS_FILE"
METRICS_SUBNET="${METRICS_SUBNET:-10.10.0.0/24}"
METRICS_PORT="${METRICS_PORT:-9090}"
python3 "${SELF_DIR}/bench-metrics-proxy.py" \
--labels "$LABELS_FILE" \
--upstream "http://${BENCH_METRICS_ADDR}/" \
--subnet "$METRICS_SUBNET" \
--port "$METRICS_PORT" &
METRICS_PROXY_PID=$!
echo "▸ Metrics proxy started (PID $METRICS_PROXY_PID) on subnet ${METRICS_SUBNET}, port ${METRICS_PORT}"
# Unique benchmark ID: local-<timestamp> for local runs, ci-<run_id> for CI
BENCH_ID="local-$(basename "$BENCH_WORK_DIR" | sed 's/bench-work-//')"
# Reference epoch: shared time origin so all runs overlay in Grafana.
# The proxy maps each run's elapsed time onto this common origin.
BENCH_REFERENCE_EPOCH=$(date +%s)
write_labels() {
local run_label="$1" run_type="$2" ref="$3" sha="$4"
LAST_RUN_START=$(date +%s)
cat > "$LABELS_FILE" <<-EOF
{"benchmark_run":"${run_label}","run_type":"${run_type}","git_ref":"${ref}","bench_sha":"${sha}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
EOF
}
run_bench() {
local label="$1" binary="$2" output_dir="$3"
echo "▸ Running benchmark: ${label}..."
cd "$RETH_REPO"
if command -v taskset &>/dev/null; then
taskset -c 0 "${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
else
"${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
fi
echo "${label} complete."
echo
}
write_labels "baseline-1" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
run_bench "baseline-1" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-1"
write_labels "feature-1" "feature" "$FEATURE_REF" "$FEATURE_SHA"
run_bench "feature-1" "$FEATURE_BIN" "$BENCH_WORK_DIR/feature-1"
write_labels "feature-2" "feature" "$FEATURE_REF" "$FEATURE_SHA"
run_bench "feature-2" "$FEATURE_BIN" "$BENCH_WORK_DIR/feature-2"
write_labels "baseline-2" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
run_bench "baseline-2" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-2"
# ── Compute Grafana URL ──────────────────────────────────────────────
GRAFANA_BASE_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr"
GRAFANA_DATASOURCE="ef57fux92e9z4e"
LAST_RUN_DURATION=$(( $(date +%s) - LAST_RUN_START ))
FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
GRAFANA_URL="${GRAFANA_BASE_URL}?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=${GRAFANA_DATASOURCE}&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
# ── Step 8: Scan logs for errors ─────────────────────────────────────
echo "▸ Scanning logs for errors..."
ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
found_errors=false
for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
LOG="$BENCH_WORK_DIR/$run_dir/node.log"
[ -f "$LOG" ] || continue
panics=$(grep -c -E 'panicked at' "$LOG" 2>/dev/null || true)
errors=$(grep -c ' ERROR ' "$LOG" 2>/dev/null || true)
if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
if [ "$found_errors" = false ]; then
printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
found_errors=true
fi
printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' \
"$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
if [ "$panics" -gt 0 ]; then
printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
printf '```\n' >> "$ERRORS_FILE"
fi
if [ "$errors" -gt 0 ]; then
printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
printf '```\n' >> "$ERRORS_FILE"
fi
printf '\n</details>\n\n' >> "$ERRORS_FILE"
fi
done
if [ "$found_errors" = true ]; then
echo " ⚠ Errors found — see $ERRORS_FILE"
else
echo " No errors found."
fi
echo
# ── Step 9: Parse results ───────────────────────────────────────────
echo "▸ Parsing results..."
cd "$RETH_REPO"
SUMMARY_ARGS=(
--output-summary "$BENCH_WORK_DIR/summary.json"
--output-markdown "$BENCH_WORK_DIR/comment.md"
--repo "paradigmxyz/reth"
--baseline-ref "$BASELINE_SHA"
--baseline-name "$BASELINE_REF"
--feature-name "$FEATURE_REF"
--feature-ref "$FEATURE_SHA"
--baseline-csv "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
--feature-csv "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
--gas-csv "$BENCH_WORK_DIR/feature-1/total_gas.csv"
--grafana-url "$GRAFANA_URL"
)
python3 "${SCRIPTS_DIR}/bench-reth-summary.py" "${SUMMARY_ARGS[@]}"
echo
# ── Step 10: Generate charts ─────────────────────────────────────────
echo "▸ Generating charts..."
CHART_ARGS=(
--output-dir "$BENCH_WORK_DIR/charts"
--feature "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
--baseline "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
--baseline-name "$BASELINE_REF"
--feature-name "$FEATURE_REF"
)
if python3 -c "import matplotlib" 2>/dev/null; then
python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
elif command -v uv &>/dev/null; then
uv run --with matplotlib python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
else
echo " Warning: matplotlib not available, skipping chart generation."
fi
echo
# ── Step 11: Upload Tracy profiles ────────────────────────────────────
if [ "$TRACY" != "off" ]; then
echo "▸ Uploading Tracy profiles..."
upload_tracy "baseline-1" "$BENCH_WORK_DIR/baseline-1" "$BASELINE_SHA"
upload_tracy "feature-1" "$BENCH_WORK_DIR/feature-1" "$FEATURE_SHA"
upload_tracy "feature-2" "$BENCH_WORK_DIR/feature-2" "$FEATURE_SHA"
upload_tracy "baseline-2" "$BENCH_WORK_DIR/baseline-2" "$BASELINE_SHA"
echo
fi
# ── Done (system restore happens via EXIT trap) ─────────────────────
echo "═══════════════════════════════════════════════════════════"
echo " Benchmark complete!"
echo "═══════════════════════════════════════════════════════════"
echo " Results : $BENCH_WORK_DIR/summary.json"
echo " Markdown : $BENCH_WORK_DIR/comment.md"
echo " Charts : $BENCH_WORK_DIR/charts/"
if [ -f "$ERRORS_FILE" ]; then
echo " Errors : $ERRORS_FILE"
fi
echo " Grafana : $GRAFANA_URL"
if [ "$TRACY" != "off" ]; then
echo " ─── Tracy Profiles ───"
for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
url_file="$BENCH_WORK_DIR/$run_dir/tracy_viewer_url.txt"
if [ -f "$url_file" ]; then
echo " $run_dir : $(cat "$url_file")"
fi
done
fi
echo "═══════════════════════════════════════════════════════════"

View File

@@ -6,13 +6,6 @@
# Usage: bench-reth-run.sh <label> <binary> <output-dir>
#
# Required env: SCHELK_MOUNT, BENCH_RPC_URL, BENCH_BLOCKS, BENCH_WARMUP_BLOCKS
# Optional env: BENCH_BIG_BLOCKS (true/false), BENCH_WORK_DIR (for big blocks path)
# BENCH_RETH_NEW_PAYLOAD (true/false, default true)
# BENCH_WAIT_TIME (duration like 500ms, default empty)
# BENCH_BASELINE_ARGS (extra reth node args for baseline runs)
# BENCH_FEATURE_ARGS (extra reth node args for feature runs)
# BENCH_OTLP_TRACES_ENDPOINT (OTLP HTTP endpoint for traces, e.g. https://host/insert/opentelemetry/v1/traces)
# BENCH_OTLP_LOGS_ENDPOINT (OTLP HTTP endpoint for logs, e.g. https://host/insert/opentelemetry/v1/logs)
set -euo pipefail
LABEL="$1"
@@ -24,24 +17,6 @@ LOG="${OUTPUT_DIR}/node.log"
cleanup() {
kill "$TAIL_PID" 2>/dev/null || true
# Stop tracy-capture first (SIGINT makes it disconnect and flush to disk)
# Must happen before killing reth, otherwise reth keeps streaming data.
if [ -n "${TRACY_PID:-}" ] && kill -0 "$TRACY_PID" 2>/dev/null; then
echo "Stopping tracy-capture..."
kill -INT "$TRACY_PID" 2>/dev/null || true
for i in $(seq 1 30); do
kill -0 "$TRACY_PID" 2>/dev/null || break
if [ $((i % 10)) -eq 0 ]; then
echo "Waiting for tracy-capture to finish writing... (${i}s)"
fi
sleep 1
done
if kill -0 "$TRACY_PID" 2>/dev/null; then
echo "tracy-capture still running after 30s, killing..."
kill -9 "$TRACY_PID" 2>/dev/null || true
fi
wait "$TRACY_PID" 2>/dev/null || true
fi
if [ -n "${RETH_PID:-}" ] && sudo kill -0 "$RETH_PID" 2>/dev/null; then
if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
# Send SIGINT to the inner reth process by exact name (not -f which
@@ -78,14 +53,8 @@ cleanup() {
fi
}
TAIL_PID=
TRACY_PID=
trap cleanup EXIT
# Clean up stale schelk state from a previous cancelled run.
# If schelk thinks it's still mounted (e.g. a cancelled run skipped cleanup),
# recover first to reset state.
sudo schelk recover -y -k || true
# Mount
sudo schelk mount -y
sync
@@ -104,8 +73,6 @@ if [ "${BENCH_CORES:-0}" -gt 0 ] && [ "$BENCH_CORES" -lt "$MAX_RETH" ]; then
fi
RETH_CPUS="1-${MAX_RETH}"
BIG_BLOCKS="${BENCH_BIG_BLOCKS:-false}"
RETH_ARGS=(
node
--datadir "$DATADIR"
@@ -120,68 +87,16 @@ RETH_ARGS=(
--no-persist-peers
)
# Big blocks mode requires the testing API and skip-invalid-transactions
if [ "$BIG_BLOCKS" = "true" ]; then
RETH_ARGS+=(--http.api eth,net,web3,reth,testing --testing.skip-invalid-transactions)
fi
# Append per-label extra node args (baseline or feature)
EXTRA_NODE_ARGS=""
case "$LABEL" in
baseline*) EXTRA_NODE_ARGS="${BENCH_BASELINE_ARGS:-}" ;;
feature*) EXTRA_NODE_ARGS="${BENCH_FEATURE_ARGS:-}" ;;
esac
if [ -n "$EXTRA_NODE_ARGS" ]; then
# Word-split the string into individual args
# shellcheck disable=SC2206
RETH_ARGS+=($EXTRA_NODE_ARGS)
fi
if [ -n "${BENCH_METRICS_ADDR:-}" ]; then
RETH_ARGS+=(--metrics "$BENCH_METRICS_ADDR")
fi
# OTLP traces and logs export
if [ -n "${BENCH_OTLP_TRACES_ENDPOINT:-}" ]; then
RETH_ARGS+=(--tracing-otlp="${BENCH_OTLP_TRACES_ENDPOINT}" --tracing-otlp.service-name=reth-bench)
fi
if [ -n "${BENCH_OTLP_LOGS_ENDPOINT:-}" ]; then
RETH_ARGS+=(--logs-otlp="${BENCH_OTLP_LOGS_ENDPOINT}" --logs-otlp.filter=debug)
fi
# Tracy profiling: add --log.tracy flags and set environment
if [ "${BENCH_TRACY:-off}" != "off" ]; then
RETH_ARGS+=(--log.tracy --log.tracy.filter "${BENCH_TRACY_FILTER:-debug}")
if [ "${BENCH_TRACY}" = "on" ]; then
export TRACY_NO_SYS_TRACE=1
elif [ "${BENCH_TRACY}" = "full" ]; then
export TRACY_SAMPLING_HZ="${BENCH_TRACY_SAMPLING_HZ:-1}"
fi
fi
SUDO_ENV=()
if [ -n "${OTEL_RESOURCE_ATTRIBUTES:-}" ]; then
SUDO_ENV+=("OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES}")
SUDO_ENV+=("OTEL_BSP_MAX_QUEUE_SIZE=65536" "OTEL_BLRP_MAX_QUEUE_SIZE=65536")
fi
# Limit reth memory to 95% of available RAM to prevent OOM kills
TOTAL_MEM_KB=$(awk '/^MemTotal:/ {print $2}' /proc/meminfo)
MEM_LIMIT=$(( TOTAL_MEM_KB * 95 / 100 * 1024 ))
echo "Memory limit: $(( MEM_LIMIT / 1024 / 1024 ))MB (95% of $(( TOTAL_MEM_KB / 1024 ))MB)"
if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
RETH_ARGS+=(--log.samply)
SAMPLY="$(which samply)"
sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
env "${SUDO_ENV[@]}" nice -n -20 \
sudo taskset -c "$RETH_CPUS" nice -n -20 \
"$SAMPLY" record --save-only --presymbolicate --rate 10000 \
--output "$OUTPUT_DIR/samply-profile.json.gz" \
-- "$BINARY" "${RETH_ARGS[@]}" \
> "$LOG" 2>&1 &
else
sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
env "${SUDO_ENV[@]}" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
sudo taskset -c "$RETH_CPUS" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
> "$LOG" 2>&1 &
fi
@@ -209,65 +124,21 @@ done
# files are not root-owned (avoids EACCES on next checkout).
BENCH_NICE="sudo nice -n -20 sudo -u $(id -un)"
# Build optional flags
EXTRA_BENCH_ARGS=()
if [ "${BENCH_RETH_NEW_PAYLOAD:-true}" != "false" ]; then
EXTRA_BENCH_ARGS+=(--reth-new-payload)
fi
if [ -n "${BENCH_WAIT_TIME:-}" ]; then
EXTRA_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
fi
# Warmup
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
--rpc-url "$BENCH_RPC_URL" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "${BENCH_WARMUP_BLOCKS:-50}" \
--reth-new-payload 2>&1 | sed -u "s/^/[bench] /"
if [ "$BIG_BLOCKS" = "true" ]; then
# Big blocks mode: replay pre-generated payloads with gas ramp
BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
# Count gas ramp blocks for reporting
GAS_RAMP_COUNT=$(find "$BIG_BLOCKS_DIR/gas-ramp-dir" -name '*.json' | wc -l)
echo "$GAS_RAMP_COUNT" > "$OUTPUT_DIR/gas_ramp_blocks.txt"
echo "Gas ramp blocks: $GAS_RAMP_COUNT"
# Start tracy-capture so profile only covers the benchmark
if [ "${BENCH_TRACY:-off}" != "off" ]; then
echo "Starting tracy-capture..."
tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
TRACY_PID=$!
sleep 0.5 # give tracy-capture time to connect
fi
echo "Running big blocks benchmark (replay-payloads)..."
$BENCH_NICE "$RETH_BENCH" replay-payloads \
"${EXTRA_BENCH_ARGS[@]}" \
--gas-ramp-dir "$BIG_BLOCKS_DIR/gas-ramp-dir" \
--payload-dir "$BIG_BLOCKS_DIR/payloads" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
else
# Standard mode: warmup + new-payload-fcu
# Warmup
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
--rpc-url "$BENCH_RPC_URL" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "${BENCH_WARMUP_BLOCKS:-50}" \
"${EXTRA_BENCH_ARGS[@]}" 2>&1 | sed -u "s/^/[bench] /"
# Start tracy-capture after warmup so profile only covers the benchmark
if [ "${BENCH_TRACY:-off}" != "off" ]; then
echo "Starting tracy-capture..."
tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
TRACY_PID=$!
sleep 0.5 # give tracy-capture time to connect
fi
# Benchmark
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
--rpc-url "$BENCH_RPC_URL" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "$BENCH_BLOCKS" \
"${EXTRA_BENCH_ARGS[@]}" \
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
fi
# Benchmark
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
--rpc-url "$BENCH_RPC_URL" \
--engine-rpc-url http://127.0.0.1:8551 \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "$BENCH_BLOCKS" \
--reth-new-payload \
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
# cleanup runs via trap

View File

@@ -107,10 +107,6 @@ def compute_stats(combined: list[dict]) -> dict:
mgas_s_values.append(r["gas_used"] / lat_s / 1_000_000)
mean_mgas_s = sum(mgas_s_values) / len(mgas_s_values) if mgas_s_values else 0
total_latencies_ms = [r["total_latency_us"] / 1_000 for r in combined]
wall_clock_s = sum(total_latencies_ms) / 1_000
mean_total_lat_ms = sum(total_latencies_ms) / n
return {
"n": n,
"mean_ms": mean_lat,
@@ -119,8 +115,6 @@ def compute_stats(combined: list[dict]) -> dict:
"p90_ms": percentile(sorted_lat, 90),
"p99_ms": percentile(sorted_lat, 99),
"mean_mgas_s": mean_mgas_s,
"wall_clock_s": wall_clock_s,
"mean_total_lat_ms": mean_total_lat_ms,
}
@@ -145,14 +139,13 @@ def compute_wait_stats(combined: list[dict], field: str) -> dict:
def _paired_data(
baseline: list[dict], feature: list[dict]
) -> tuple[list[tuple[float, float]], list[float], list[float], list[float]]:
) -> tuple[list[tuple[float, float]], list[float], list[float]]:
"""Match blocks and return paired latencies and per-block diffs.
Returns:
pairs: list of (baseline_ms, feature_ms) tuples
lat_diffs_ms: list of feature baseline latency diffs in ms
mgas_diffs: list of feature baseline Mgas/s diffs
total_lat_diffs_ms: list of feature baseline total latency diffs in ms
"""
baseline_by_block = {r["block_number"]: r for r in baseline}
feature_by_block = {r["block_number"]: r for r in feature}
@@ -161,7 +154,6 @@ def _paired_data(
pairs = []
lat_diffs_ms = []
mgas_diffs = []
total_lat_diffs_ms = []
for bn in common_blocks:
b = baseline_by_block[bn]
f = feature_by_block[bn]
@@ -176,10 +168,7 @@ def _paired_data(
f["gas_used"] / f_lat_s / 1_000_000
- b["gas_used"] / b_lat_s / 1_000_000
)
total_lat_diffs_ms.append(
f["total_latency_us"] / 1_000 - b["total_latency_us"] / 1_000
)
return pairs, lat_diffs_ms, mgas_diffs, total_lat_diffs_ms
return pairs, lat_diffs_ms, mgas_diffs
def compute_paired_stats(
@@ -194,14 +183,12 @@ def compute_paired_stats(
all_pairs = []
all_lat_diffs = []
all_mgas_diffs = []
all_total_lat_diffs = []
blocks_per_pair = []
for baseline, feature in zip(baseline_runs, feature_runs):
pairs, lat_diffs, mgas_diffs, total_lat_diffs = _paired_data(baseline, feature)
pairs, lat_diffs, mgas_diffs = _paired_data(baseline, feature)
all_pairs.extend(pairs)
all_lat_diffs.extend(lat_diffs)
all_mgas_diffs.extend(mgas_diffs)
all_total_lat_diffs.extend(total_lat_diffs)
blocks_per_pair.append(len(pairs))
if not all_lat_diffs:
@@ -240,11 +227,6 @@ def compute_paired_stats(
mgas_se = std_mgas_diff / math.sqrt(len(all_mgas_diffs)) if all_mgas_diffs else 0.0
mgas_ci = T_CRITICAL * mgas_se
mean_total_diff = sum(all_total_lat_diffs) / len(all_total_lat_diffs) if all_total_lat_diffs else 0.0
std_total_diff = stddev(all_total_lat_diffs, mean_total_diff) if len(all_total_lat_diffs) > 1 else 0.0
total_se = std_total_diff / math.sqrt(len(all_total_lat_diffs)) if all_total_lat_diffs else 0.0
wall_clock_ci_ms = T_CRITICAL * total_se
return {
"n": n,
"mean_diff_ms": mean_diff,
@@ -257,7 +239,6 @@ def compute_paired_stats(
"p99_ci_ms": (p99_boot[hi] - p99_boot[lo]) / 2,
"mean_mgas_diff": mean_mgas_diff,
"mgas_ci": mgas_ci,
"wall_clock_ci_ms": wall_clock_ci_ms,
"blocks": max(blocks_per_pair),
}
@@ -286,10 +267,6 @@ def fmt_mgas(v: float) -> str:
return f"{v:.2f}"
def fmt_s(v: float) -> str:
return f"{v:.2f}s"
def significance(pct: float, ci_pct: float, lower_is_better: bool) -> str:
"""Return significance label: 'good', 'bad', or 'neutral'."""
significant = abs(pct) > ci_pct
@@ -327,7 +304,6 @@ def compute_changes(
("p90", "p90_ms", "p90_ci_ms", "p90_ms", True),
("p99", "p99_ms", "p99_ci_ms", "p99_ms", True),
("mgas_s", "mean_mgas_s", "mgas_ci", "mean_mgas_s", False),
("wall_clock", "wall_clock_s", "wall_clock_ci_ms", "mean_total_lat_ms", True),
]
changes = {}
for name, stat_key, ci_key, base_key, lower_is_better in metrics:
@@ -350,7 +326,6 @@ def generate_comparison_table(
baseline_name: str,
feature_name: str,
feature_sha: str,
big_blocks: bool = False,
) -> str:
"""Generate a markdown comparison table between baseline and feature."""
n = paired["blocks"]
@@ -360,7 +335,6 @@ def generate_comparison_table(
mean_pct = pct(run1["mean_ms"], run2["mean_ms"])
gas_pct = pct(run1["mean_mgas_s"], run2["mean_mgas_s"])
wall_pct = pct(run1["wall_clock_s"], run2["wall_clock_s"])
p50_pct = pct(run1["p50_ms"], run2["p50_ms"])
p90_pct = pct(run1["p90_ms"], run2["p90_ms"])
@@ -374,7 +348,6 @@ def generate_comparison_table(
# CI as a percentage of baseline mean
lat_ci_pct = paired["ci_ms"] / run1["mean_ms"] * 100.0 if run1["mean_ms"] > 0 else 0.0
mgas_ci_pct = paired["mgas_ci"] / run1["mean_mgas_s"] * 100.0 if run1["mean_mgas_s"] > 0 else 0.0
wall_ci_pct = paired["wall_clock_ci_ms"] / run1["mean_total_lat_ms"] * 100.0 if run1["mean_total_lat_ms"] > 0 else 0.0
base_url = f"https://github.com/{repo}/commit"
baseline_label = f"[`{baseline_name}`]({base_url}/{baseline_ref})"
@@ -389,9 +362,8 @@ def generate_comparison_table(
f"| P90 | {fmt_ms(run1['p90_ms'])} | {fmt_ms(run2['p90_ms'])} | {change_str(p90_pct, p90_ci_pct, lower_is_better=True)} |",
f"| P99 | {fmt_ms(run1['p99_ms'])} | {fmt_ms(run2['p99_ms'])} | {change_str(p99_pct, p99_ci_pct, lower_is_better=True)} |",
f"| Mgas/s | {fmt_mgas(run1['mean_mgas_s'])} | {fmt_mgas(run2['mean_mgas_s'])} | {change_str(gas_pct, mgas_ci_pct, lower_is_better=False)} |",
f"| Wall Clock | {fmt_s(run1['wall_clock_s'])} | {fmt_s(run2['wall_clock_s'])} | {change_str(wall_pct, wall_ci_pct, lower_is_better=True)} |",
"",
f"*{n} {'big blocks' if big_blocks else 'blocks'}*",
f"*{n} blocks*",
]
return "\n".join(lines)
@@ -422,7 +394,6 @@ def generate_markdown(
summary: dict, comparison_table: str,
wait_time_tables: list[str] | None = None,
behind_baseline: int = 0, repo: str = "", baseline_ref: str = "", baseline_name: str = "",
grafana_url: str | None = None,
) -> str:
"""Generate a markdown comment body."""
lines = ["## Benchmark Results", ""]
@@ -442,9 +413,6 @@ def generate_markdown(
lines.append(table)
lines.append("")
lines.append("</details>")
if grafana_url:
lines.append("")
lines.append(f"**[Grafana Dashboard]({grafana_url})**")
return "\n".join(lines)
@@ -471,9 +439,6 @@ def main():
parser.add_argument("--feature-name", "--branch-name", default=None, help="Feature branch name")
parser.add_argument("--feature-ref", "--branch-sha", "--feature-sha", default=None, help="Feature commit SHA")
parser.add_argument("--behind-baseline", "--behind-main", type=int, default=0, help="Commits behind baseline")
parser.add_argument("--big-blocks", action="store_true", default=False, help="Big blocks mode")
parser.add_argument("--gas-ramp-blocks", type=int, default=0, help="Number of gas ramp blocks (big blocks mode)")
parser.add_argument("--grafana-url", default=None, help="Grafana dashboard URL for this benchmark run")
args = parser.parse_args()
if len(args.baseline_csv) != len(args.feature_csv):
@@ -522,7 +487,6 @@ def main():
baseline_name=baseline_name,
feature_name=feature_name,
feature_sha=feature_sha,
big_blocks=args.big_blocks,
)
print(f"Generated comparison ({paired_stats['n']} paired blocks, "
f"mean diff {paired_stats['mean_diff_ms']:+.3f}ms ± {paired_stats['ci_ms']:.3f}ms)")
@@ -553,8 +517,6 @@ def main():
summary = {
"blocks": paired_stats["blocks"],
"big_blocks": args.big_blocks,
"gas_ramp_blocks": args.gas_ramp_blocks,
"baseline": {
"name": baseline_name,
"ref": baseline_ref,
@@ -580,7 +542,6 @@ def main():
repo=args.repo,
baseline_ref=baseline_ref,
baseline_name=baseline_name,
grafana_url=args.grafana_url,
)
with open(args.output_markdown, "w") as f:

View File

@@ -1,132 +0,0 @@
#!/usr/bin/env bash
#
# Resolves baseline and feature refs for nightly regression benchmark runs.
#
# Queries the latest successful scheduled docker.yml run via GitHub API
# to find the commit that built the nightly Docker image. Compares with
# the last successful feature ref (from GH Actions cache) to determine
# baseline, detect staleness, and decide whether to skip.
#
# Usage: bench-nightly-refs.sh [--force]
#
# Outputs (via GITHUB_OUTPUT):
# baseline-ref — commit SHA for baseline
# feature-ref — commit SHA for feature (current nightly)
# should-skip — "true" if no new nightly since last run
# is-stale — "true" if latest nightly build is >24h old
# stale-age-hours — age of the nightly build in hours (only if stale)
# nightly-created — ISO timestamp of the nightly build
#
# Reads:
# .nightly-state/last-feature-ref (from GH Actions cache, may not exist)
#
# Requires: gh (GitHub CLI), jq, date
set -euo pipefail
FORCE="${1:-false}"
REPO="${GITHUB_REPOSITORY:-paradigmxyz/reth}"
# --- Step 1: Query latest successful scheduled docker.yml run ---
echo "::group::Querying latest nightly docker build"
RUNS_JSON=$(gh run list \
-R "$REPO" \
--workflow=docker.yml \
--event=schedule \
--status=completed \
--limit 5 \
--json headSha,createdAt,conclusion)
# Find the most recent successful run
LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
if [ -z "$LATEST" ]; then
echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
echo "Runs found: $RUNS_JSON"
exit 1
fi
FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
CREATED_AT=$(echo "$LATEST" | jq -r '.createdAt')
echo "Latest nightly commit: $FEATURE_REF"
echo "Built at: $CREATED_AT"
echo "::endgroup::"
# --- Step 2: Staleness check ---
echo "::group::Checking staleness"
NOW_EPOCH=$(date +%s)
# Handle both GNU date (-d) and BSD date (-j -f) for cross-platform compat
CREATED_EPOCH=$(date -d "$CREATED_AT" +%s 2>/dev/null || \
date -j -f "%Y-%m-%dT%H:%M:%SZ" "$CREATED_AT" +%s 2>/dev/null || \
date -j -f "%Y-%m-%dT%T%z" "$CREATED_AT" +%s 2>/dev/null || \
{ echo "::error::Cannot parse date: $CREATED_AT"; exit 1; })
AGE_SECONDS=$(( NOW_EPOCH - CREATED_EPOCH ))
AGE_HOURS=$(( AGE_SECONDS / 3600 ))
IS_STALE="false"
if [ "$AGE_HOURS" -gt 24 ]; then
IS_STALE="true"
echo "::warning::STALE NIGHTLY: Build is ${AGE_HOURS}h old (>24h threshold)"
echo "This indicates the nightly docker build failed — no new image was produced"
else
echo "Nightly build age: ${AGE_HOURS}h (within 24h threshold)"
fi
echo "::endgroup::"
# --- Step 3: Read last successful feature ref from cache ---
echo "::group::Reading cached state"
LAST_FEATURE_REF=""
STATE_FILE=".nightly-state/last-feature-ref"
if [ -f "$STATE_FILE" ]; then
LAST_FEATURE_REF=$(tr -d '[:space:]' < "$STATE_FILE")
echo "Previous feature ref: $LAST_FEATURE_REF"
else
echo "No cached state found (first run)"
fi
echo "::endgroup::"
# --- Step 4: Determine baseline and skip logic ---
echo "::group::Resolving refs"
SHOULD_SKIP="false"
BASELINE_REF="$FEATURE_REF" # default for first run
if [ "$IS_STALE" = "true" ]; then
# Stale = error path, don't skip (will alert and fail downstream)
SHOULD_SKIP="false"
BASELINE_REF="${LAST_FEATURE_REF:-$FEATURE_REF}"
echo "Stale nightly detected — will alert and fail"
elif [ -z "$LAST_FEATURE_REF" ]; then
# First run: baseline = feature (self-comparison to establish baseline)
BASELINE_REF="$FEATURE_REF"
echo "First run — will benchmark nightly against itself to establish baseline"
elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
# No new nightly since last successful run
if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
echo "No new nightly, but force=true — running anyway"
BASELINE_REF="$LAST_FEATURE_REF"
else
SHOULD_SKIP="true"
echo "No new nightly since last run — will skip"
fi
else
# Normal case: new nightly available
BASELINE_REF="$LAST_FEATURE_REF"
echo "New nightly detected"
fi
echo "Baseline: $BASELINE_REF"
echo "Feature: $FEATURE_REF"
echo "Skip: $SHOULD_SKIP"
echo "Stale: $IS_STALE"
echo "::endgroup::"
# --- Step 5: Write outputs ---
{
echo "baseline-ref=$BASELINE_REF"
echo "feature-ref=$FEATURE_REF"
echo "should-skip=$SHOULD_SKIP"
echo "is-stale=$IS_STALE"
echo "stale-age-hours=$AGE_HOURS"
echo "nightly-created=$CREATED_AT"
} >> "$GITHUB_OUTPUT"

View File

@@ -7,8 +7,6 @@
// BENCH_PR PR number (may be empty)
// BENCH_ACTOR GitHub user who triggered the bench
// BENCH_JOB_URL URL to the Actions job page
// BENCH_BASELINE_ARGS Extra CLI args for the baseline reth node
// BENCH_FEATURE_ARGS Extra CLI args for the feature reth node
// BENCH_SAMPLY 'true' if samply profiling was enabled
//
// Usage from actions/github-script:
@@ -18,7 +16,6 @@
const fs = require('fs');
const path = require('path');
const { fmtChange, fmtMs, verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
const SLACK_API = 'https://slack.com/api/chat.postMessage';
@@ -62,17 +59,40 @@ function cell(text) {
return { type: 'raw_text', text: s || ' ' };
}
// Slack shortcodes for verdict (Block Kit header doesn't support unicode emoji)
const SLACK_VERDICT = {
'⚠️': ':warning:',
'❌': ':x:',
'✅': ':white_check_mark:',
'⚪': ':white_circle:',
};
function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, repo, samplyUrls }) {
const { emoji, label } = verdict(summary.changes);
const headerEmoji = SLACK_VERDICT[emoji] || emoji;
const b = summary.baseline.stats;
const f = summary.feature.stats;
const c = summary.changes;
const sigEmoji = { good: '\u2705', bad: '\u274c', neutral: '\u26aa' };
function fmtMs(v) { return v.toFixed(2) + 'ms'; }
function fmtMgas(v) { return v.toFixed(2); }
function fmtChange(ch) {
if (!ch.pct && !ch.ci_pct) return ' ';
const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
const ciStr = ch.ci_pct ? ` (\u00b1${ch.ci_pct.toFixed(2)}%)` : '';
return `${pctStr}${ciStr} ${sigEmoji[ch.sig]}`;
}
// Overall result for header
const vals = Object.values(c);
const hasBad = vals.some(v => v.sig === 'bad');
const hasGood = vals.some(v => v.sig === 'good');
let headerEmoji, headerResult;
if (hasBad && hasGood) {
headerEmoji = ':warning:';
headerResult = 'Mixed Results';
} else if (hasBad) {
headerEmoji = ':x:';
headerResult = 'Regression';
} else if (hasGood) {
headerEmoji = ':white_check_mark:';
headerResult = 'Improvement';
} else {
headerEmoji = ':white_circle:';
headerResult = 'No Difference';
}
const prUrl = prNumber ? `https://github.com/${repo}/pull/${prNumber}` : '';
const commitUrl = `https://github.com/${repo}/commit`;
@@ -97,15 +117,15 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
if (fl1) featureLine += ` | <${fl1}|Samply 1>`;
if (fl2) featureLine += ` | <${fl2}|Samply 2>`;
const countsLine = blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | ');
const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
const cores = process.env.BENCH_CORES || '0';
const countsParts = [];
if (warmup) countsParts.push(`*Warmup:* ${warmup}`);
countsParts.push(`*Blocks:* ${summary.blocks}`);
if (cores !== '0') countsParts.push(`*Cores:* ${cores}`);
const countsLine = countsParts.join(' | ');
const baselineArgs = process.env.BENCH_BASELINE_ARGS || '';
const featureArgs = process.env.BENCH_FEATURE_ARGS || '';
const argsLines = [];
if (baselineArgs) argsLines.push(`*Baseline Args:* \`${baselineArgs}\``);
if (featureArgs) argsLines.push(`*Feature Args:* \`${featureArgs}\``);
const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, ...argsLines, countsLine].join('\n');
const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, countsLine].join('\n');
// Action buttons
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
@@ -124,17 +144,10 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
},
];
// Build table rows from shared metricRows
const rows = metricRows(summary);
const tableRows = [
[cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
];
const blocks = [
{
type: 'header',
text: { type: 'plain_text', text: `${headerEmoji} ${label}`, emoji: true },
text: { type: 'plain_text', text: `${headerEmoji} ${headerResult}`, emoji: true },
},
{
type: 'section',
@@ -148,7 +161,15 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
{ align: 'right' },
{ align: 'right' },
],
rows: tableRows,
rows: [
[cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
[cell('Mean'), cell(fmtMs(b.mean_ms)), cell(fmtMs(f.mean_ms)), cell(fmtChange(c.mean))],
[cell('StdDev'), cell(fmtMs(b.stddev_ms)), cell(fmtMs(f.stddev_ms)), cell(' ')],
[cell('P50'), cell(fmtMs(b.p50_ms)), cell(fmtMs(f.p50_ms)), cell(fmtChange(c.p50))],
[cell('P90'), cell(fmtMs(b.p90_ms)), cell(fmtMs(f.p90_ms)), cell(fmtChange(c.p90))],
[cell('P99'), cell(fmtMs(b.p99_ms)), cell(fmtMs(f.p99_ms)), cell(fmtChange(c.p99))],
[cell('Mgas/s'), cell(fmtMgas(b.mean_mgas_s)), cell(fmtMgas(f.mean_mgas_s)), cell(fmtChange(c.mgas_s))],
],
},
{
type: 'actions',
@@ -158,12 +179,16 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
// Wait times as a separate table block (sent as threaded reply due to Slack one-table limit)
const threadBlocks = [];
const wtRows = waitTimeRows(summary);
if (wtRows.length > 0) {
const waitTableRows = [
const waitTimes = summary.wait_times || {};
const waitKeys = Object.keys(waitTimes);
if (waitKeys.length > 0) {
const waitRows = [
[cell('Wait Time'), cell('Baseline'), cell('Feature')],
...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
];
for (const key of waitKeys) {
const wt = waitTimes[key];
waitRows.push([cell(wt.title), cell(fmtMs(wt.baseline.mean_ms)), cell(fmtMs(wt.feature.mean_ms))]);
}
threadBlocks.push({
type: 'table',
column_settings: [
@@ -171,7 +196,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
{ align: 'right' },
{ align: 'right' },
],
rows: waitTableRows,
rows: waitRows,
});
}
@@ -233,7 +258,16 @@ async function success({ core, context }) {
const jobUrl = process.env.BENCH_JOB_URL ||
`${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
// Load samply profile URLs (files exist when samply profiling was enabled)
const samplyUrls = {};
for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
try {
const url = fs.readFileSync(
path.join(process.env.BENCH_WORK_DIR, run, 'samply-profile-url.txt'), 'utf8'
).trim();
if (url) samplyUrls[run] = url;
} catch {}
}
const slackUsers = loadSlackUsers(process.env.GITHUB_WORKSPACE || '.');
const actorSlackId = slackUsers[actor];

View File

@@ -9,16 +9,5 @@
"gakonst": "U092SEPDM40",
"Rjected": "U09F6SCKRGT",
"DaniPopes": "U09FAT8EK2A",
"emmajam": "U0A34UN92HW",
"onbjerg": "U09FB0UK5AA",
"fgimenez": "U09G3GP7CSU",
"rakita": "U09FB3Z2M7Y",
"jxom": "U09F72MG083",
"tmm": "U0AD0U8E88N",
"pepyakin": "U0A7HKMGEHJ",
"grandizzy": "U09F8DBDDRT",
"SuperFluffy": "U095BKHB2Q4",
"kamsz": "U0A2563UBRD",
"zerosnacks": "U09FARPMN74",
"samczsun": "U096R14E4H3"
"emmajam": "U0A34UN92HW"
}

View File

@@ -1,96 +0,0 @@
// Shared utilities for reth-bench result rendering.
//
// Used by bench-job-summary.js and bench-slack-notify.js.
const fs = require('fs');
const path = require('path');
const SIG_EMOJI = { good: '✅', bad: '❌', neutral: '⚪' };
function fmtMs(v) { return v.toFixed(2) + 'ms'; }
function fmtMgas(v) { return v.toFixed(2); }
function fmtS(v) { return v.toFixed(2) + 's'; }
function fmtChange(ch) {
if (!ch || (!ch.pct && !ch.ci_pct)) return '';
const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
const ciStr = ch.ci_pct ? `${ch.ci_pct.toFixed(2)}%)` : '';
return `${pctStr}${ciStr} ${SIG_EMOJI[ch.sig]}`;
}
function verdict(changes) {
const vals = Object.values(changes);
const hasBad = vals.some(v => v.sig === 'bad');
const hasGood = vals.some(v => v.sig === 'good');
if (hasBad && hasGood) return { emoji: '⚠️', label: 'Mixed Results' };
if (hasBad) return { emoji: '❌', label: 'Regression' };
if (hasGood) return { emoji: '✅', label: 'Improvement' };
return { emoji: '⚪', label: 'No Difference' };
}
function loadSamplyUrls(workDir) {
const urls = {};
for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
try {
const url = fs.readFileSync(path.join(workDir, run, 'samply-profile-url.txt'), 'utf8').trim();
if (url) urls[run] = url;
} catch {}
}
return urls;
}
function blocksLabel(summary) {
const parts = [];
if (summary.big_blocks) {
if (summary.gas_ramp_blocks) parts.push({ key: 'Gas Ramp', value: summary.gas_ramp_blocks });
parts.push({ key: 'Big Blocks', value: summary.blocks });
} else {
const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
if (warmup) parts.push({ key: 'Warmup', value: warmup });
parts.push({ key: 'Blocks', value: summary.blocks });
}
const cores = process.env.BENCH_CORES || '0';
if (cores !== '0') parts.push({ key: 'Cores', value: cores });
return parts;
}
// The 7 metric rows shared by all renderers.
// Returns an array of { label, baseline, feature, change } objects.
function metricRows(summary) {
const b = summary.baseline.stats;
const f = summary.feature.stats;
const c = summary.changes;
return [
{ label: 'Mean', baseline: fmtMs(b.mean_ms), feature: fmtMs(f.mean_ms), change: fmtChange(c.mean) },
{ label: 'StdDev', baseline: fmtMs(b.stddev_ms), feature: fmtMs(f.stddev_ms), change: '' },
{ label: 'P50', baseline: fmtMs(b.p50_ms), feature: fmtMs(f.p50_ms), change: fmtChange(c.p50) },
{ label: 'P90', baseline: fmtMs(b.p90_ms), feature: fmtMs(f.p90_ms), change: fmtChange(c.p90) },
{ label: 'P99', baseline: fmtMs(b.p99_ms), feature: fmtMs(f.p99_ms), change: fmtChange(c.p99) },
{ label: 'Mgas/s', baseline: fmtMgas(b.mean_mgas_s), feature: fmtMgas(f.mean_mgas_s), change: fmtChange(c.mgas_s) },
{ label: 'Wall Clock', baseline: fmtS(b.wall_clock_s), feature: fmtS(f.wall_clock_s), change: fmtChange(c.wall_clock) },
];
}
// Wait time rows: one row per metric showing mean values.
function waitTimeRows(summary) {
const waitTimes = summary.wait_times || {};
const rows = [];
for (const key of Object.keys(waitTimes)) {
const wt = waitTimes[key];
rows.push({ title: wt.title, baseline: fmtMs(wt.baseline.mean_ms), feature: fmtMs(wt.feature.mean_ms) });
}
return rows;
}
module.exports = {
SIG_EMOJI,
fmtMs,
fmtMgas,
fmtS,
fmtChange,
verdict,
loadSamplyUrls,
blocksLabel,
metricRows,
waitTimeRows,
};

View File

@@ -1,414 +0,0 @@
#!/usr/bin/env bash
#
# Full PGO+BOLT optimized build for reth using real reth-bench workloads.
#
# Phases:
# 1. Build PGO-instrumented reth, run reth-bench → collect PGO profiles
# 2. Build BOLT-instrumented reth (with PGO), run reth-bench → collect BOLT profiles
# 3. Build final PGO+BOLT optimized binary
#
# Required environment variables:
# DATADIR - Path to reth datadir (must already contain chain data)
# RPC_URL - Source RPC URL for reth-bench to fetch payloads from
#
# Optional environment variables:
# PGO_BLOCKS - Number of blocks for PGO profiling (default: 20)
# BOLT_BLOCKS - Number of blocks for BOLT profiling (default: 20)
# SKIP_BOLT - Temporarily skip BOLT phases (default: false)
# STRIP_SYMBOLS - Strip debug symbols from output binary (default: true)
# COLLECT_PGO_ONLY - Stop after producing merged.profdata (default: false)
# PGO_PROFDATA - Path to pre-collected merged.profdata (optional)
# PROFILE - Cargo profile (default: maxperf-symbols)
# FEATURES - Cargo features (default: jemalloc,asm-keccak,min-debug-logs)
# TARGET - Target triple (default: auto-detected)
# EXTRA_RUSTFLAGS - Additional RUSTFLAGS (e.g. -C target-cpu=x86-64-v3)
#
# Output:
# target/$PROFILE_DIR/reth — final optimized binary
set -euo pipefail
gha_section_start() {
local title="$1"
if [ -n "${GITHUB_ACTIONS:-}" ]; then
echo "::group::$title"
else
echo ""
echo "=== $title ==="
fi
}
gha_section_end() {
if [ -n "${GITHUB_ACTIONS:-}" ]; then
echo "::endgroup::"
fi
}
cd "$(dirname "$0")/../.."
# ── Configuration ──────────────────────────────────────────────────────────────
PGO_BLOCKS="${PGO_BLOCKS:-20}"
BOLT_BLOCKS="${BOLT_BLOCKS:-20}"
SKIP_BOLT="${SKIP_BOLT:-false}"
STRIP_SYMBOLS="${STRIP_SYMBOLS:-true}"
COLLECT_PGO_ONLY="${COLLECT_PGO_ONLY:-false}"
PROFILE="${PROFILE:-maxperf-symbols}"
FEATURES="${FEATURES:-jemalloc,asm-keccak,min-debug-logs}"
TARGET="${TARGET:-$(rustc -Vv | grep host | cut -d' ' -f2)}"
BASE_RUSTFLAGS="${RUSTFLAGS:-}"
EXTRA_RUSTFLAGS="${EXTRA_RUSTFLAGS:-}"
COMBINED_RUSTFLAGS="$BASE_RUSTFLAGS $EXTRA_RUSTFLAGS"
PGO_PROFDATA="${PGO_PROFDATA:-}"
DATADIR="${DATADIR:-}"
RPC_URL="${RPC_URL:-}"
SKIP_BOLT_BOOL=false
if [[ "${SKIP_BOLT,,}" == "true" || "$SKIP_BOLT" == "1" ]]; then
SKIP_BOLT_BOOL=true
fi
STRIP_SYMBOLS_BOOL=false
if [[ "${STRIP_SYMBOLS,,}" == "true" || "$STRIP_SYMBOLS" == "1" ]]; then
STRIP_SYMBOLS_BOOL=true
fi
COLLECT_PGO_ONLY_BOOL=false
if [[ "${COLLECT_PGO_ONLY,,}" == "true" || "$COLLECT_PGO_ONLY" == "1" ]]; then
COLLECT_PGO_ONLY_BOOL=true
fi
USE_PRECOLLECTED_PGO=false
if [ -n "$PGO_PROFDATA" ]; then
if [ ! -f "$PGO_PROFDATA" ]; then
echo "error: PGO_PROFDATA points to a missing file: $PGO_PROFDATA"
exit 1
fi
USE_PRECOLLECTED_PGO=true
fi
NEEDS_BENCH_WORKLOAD=true
if [ "$USE_PRECOLLECTED_PGO" = true ] && [ "$SKIP_BOLT_BOOL" = true ]; then
NEEDS_BENCH_WORKLOAD=false
fi
if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
: "${DATADIR:?DATADIR must be set to the reth data directory}"
: "${RPC_URL:?RPC_URL must be set}"
fi
if [[ "$PROFILE" == dev ]]; then
PROFILE_DIR=debug
else
PROFILE_DIR=$PROFILE
fi
MANIFEST_PATH="bin/reth"
LLVM_VERSION=$(rustc -Vv | grep -oP 'LLVM version: \K\d+')
PGO_DIR="$PWD/target/pgo-profiles"
BOLT_DIR="$PWD/target/bolt-profiles"
CARGO_ARGS=(--profile "$PROFILE" --features "$FEATURES" --manifest-path "$MANIFEST_PATH/Cargo.toml" --bin "reth" --locked)
# Enable debug symbols for BOLT (requires symbols to reorder code).
# Strip them at the end.
PROFILE_UPPER=$(echo "$PROFILE" | tr '[:lower:]-' '[:upper:]_')
export "CARGO_PROFILE_${PROFILE_UPPER}_STRIP=debuginfo"
gha_section_start "Full PGO+BOLT Build"
echo "Binary: reth"
echo "Manifest: $MANIFEST_PATH"
echo "Target: $TARGET"
echo "Profile: $PROFILE"
echo "Features: $FEATURES"
echo "LLVM: $LLVM_VERSION"
echo "PGO blocks: $PGO_BLOCKS"
echo "BOLT blocks: $BOLT_BLOCKS"
echo "Skip BOLT: $SKIP_BOLT"
echo "Strip symbols: $STRIP_SYMBOLS"
echo "Collect only: $COLLECT_PGO_ONLY"
echo "PGO profdata: ${PGO_PROFDATA:-<collect with reth-bench>}"
echo "RUSTFLAGS: ${BASE_RUSTFLAGS:-<unset>}"
echo "EXTRA_RUSTFLAGS: ${EXTRA_RUSTFLAGS:-<unset>}"
if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
echo "Datadir: $DATADIR"
echo "RPC URL: $RPC_URL"
else
echo "Datadir: <not required>"
echo "RPC URL: <not required>"
fi
gha_section_end
# ── Prerequisites ──────────────────────────────────────────────────────────────
gha_section_start "Installing prerequisites"
rustup component add llvm-tools-preview
LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
if [ -z "$LLVM_PROFDATA" ]; then
echo "error: llvm-profdata not found"
exit 1
fi
install_bolt() {
if command -v llvm-bolt &>/dev/null; then
echo "BOLT already installed"
return
fi
echo "Installing BOLT from apt.llvm.org..."
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc >/dev/null
CODENAME=$(lsb_release -cs)
echo "deb http://apt.llvm.org/$CODENAME/ llvm-toolchain-$CODENAME-$LLVM_VERSION main" | sudo tee /etc/apt/sources.list.d/llvm.list >/dev/null
sudo apt-get update -qq
sudo apt-get install -y -qq "bolt-$LLVM_VERSION"
sudo ln -sf "/usr/bin/llvm-bolt-$LLVM_VERSION" /usr/local/bin/llvm-bolt
sudo ln -sf "/usr/bin/merge-fdata-$LLVM_VERSION" /usr/local/bin/merge-fdata
}
if [ "$SKIP_BOLT_BOOL" = true ]; then
echo "Skipping BOLT installation (SKIP_BOLT=$SKIP_BOLT)"
else
install_bolt
fi
gha_section_end
if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
# Build reth-bench once (non-instrumented) — reused for both phases.
gha_section_start "Building reth-bench"
RUSTFLAGS="$COMBINED_RUSTFLAGS" \
cargo build --profile "$PROFILE" --features "$FEATURES" \
--manifest-path bin/reth-bench/Cargo.toml --bin reth-bench --locked
RETH_BENCH_BIN="$(find target -name reth-bench -type f -executable | head -1)"
echo "reth-bench: $RETH_BENCH_BIN"
gha_section_end
else
gha_section_start "Building reth-bench"
echo "Skipping reth-bench build (pre-collected PGO with SKIP_BOLT=true)"
gha_section_end
fi
# ── Helpers ────────────────────────────────────────────────────────────────────
RETH_PID=
cleanup() {
if [ -n "${RETH_PID:-}" ] && kill -0 "$RETH_PID" 2>/dev/null; then
echo "Stopping reth (pid $RETH_PID)..."
sudo kill "$RETH_PID" 2>/dev/null || true
for i in $(seq 1 60); do
sudo kill -0 "$RETH_PID" 2>/dev/null || break
if [ $((i % 10)) -eq 0 ]; then
echo " waiting... (${i}s)"
fi
sleep 1
done
sudo kill -9 "$RETH_PID" 2>/dev/null || true
fi
}
trap cleanup EXIT
# Start reth, wait for RPC, run reth-bench, then stop reth.
# Arguments: $1 = reth binary path, $2 = number of blocks, $3 = log label
run_bench_workload() {
local reth_bin="$1" blocks="$2" label="$3"
local http_port=8545 authrpc_port=8551
echo "--- Starting reth ($label) ---"
sudo "$reth_bin" node \
--datadir "$DATADIR" \
--log.file.directory "/tmp/reth-${label}-logs" \
--engine.accept-execution-requests-hash \
--http --http.port "$http_port" \
--authrpc.port "$authrpc_port" \
--disable-discovery --no-persist-peers \
> "/tmp/reth-${label}.log" 2>&1 &
RETH_PID=$!
echo "Waiting for reth RPC..."
for i in $(seq 1 120); do
if curl -sf "http://127.0.0.1:$http_port" -X POST \
-H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
> /dev/null 2>&1; then
echo "reth is ready after ${i}s"
break
fi
if [ "$i" -eq 120 ]; then
echo "error: reth failed to start within 120s"
cat "/tmp/reth-${label}.log"
exit 1
fi
sleep 1
done
echo "Running reth-bench ($blocks blocks)..."
"$RETH_BENCH_BIN" new-payload-fcu \
--rpc-url "$RPC_URL" \
--engine-rpc-url "http://127.0.0.1:$authrpc_port" \
--jwt-secret "$DATADIR/jwt.hex" \
--advance "$blocks" \
--reth-new-payload 2>&1 | sed -u "s/^/[$label] /"
echo "Stopping reth ($label)..."
sudo kill "$RETH_PID" 2>/dev/null || true
for i in $(seq 1 60); do
sudo kill -0 "$RETH_PID" 2>/dev/null || break
sleep 1
done
sudo kill -9 "$RETH_PID" 2>/dev/null || true
RETH_PID=
}
publish_binary() {
local source_bin="$1"
for out in "target/$TARGET/$PROFILE_DIR" "target/$PROFILE_DIR"; do
local destination="$out/reth"
mkdir -p "$out"
# Skip copying when source and destination resolve to the same inode.
if [ -e "$destination" ] && [ "$source_bin" -ef "$destination" ]; then
continue
fi
cp "$source_bin" "$destination"
done
}
if [ "$USE_PRECOLLECTED_PGO" = true ]; then
gha_section_start "Phase 1: Using Pre-Collected PGO Profile"
rm -rf "$PGO_DIR"
mkdir -p "$PGO_DIR"
cp "$PGO_PROFDATA" "$PGO_DIR/merged.profdata"
echo "Using pre-collected profile: $PGO_PROFDATA"
echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
gha_section_end
else
# ── Phase 1: PGO profile collection ───────────────────────────────────────
gha_section_start "Phase 1: PGO Profile Collection"
rm -rf "$PGO_DIR"
mkdir -p "$PGO_DIR"
echo "Building PGO-instrumented binary..."
RUSTFLAGS="-Cprofile-generate=$PGO_DIR -Crelocation-model=pic $COMBINED_RUSTFLAGS" \
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
PGO_RETH_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
echo "Instrumented binary: $PGO_RETH_BIN ($(ls -lh "$PGO_RETH_BIN" | awk '{print $5}'))"
run_bench_workload "$PGO_RETH_BIN" "$PGO_BLOCKS" "pgo"
# Fix ownership if reth ran as root.
sudo chown -R "$(id -un):$(id -gn)" "$PGO_DIR" 2>/dev/null || true
# Merge PGO profiles.
echo "Merging PGO profiles..."
PROFRAW_COUNT=$(find "$PGO_DIR" -name '*.profraw' | wc -l)
echo "Found $PROFRAW_COUNT .profraw files"
if [ "$PROFRAW_COUNT" -eq 0 ]; then
echo "error: no .profraw files — instrumented binary did not produce profiles"
exit 1
fi
"$LLVM_PROFDATA" merge -o "$PGO_DIR/merged.profdata" "$PGO_DIR"/*.profraw
echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
gha_section_end
fi
if [ "$COLLECT_PGO_ONLY_BOOL" = true ]; then
gha_section_start "PGO Collection Complete"
echo "COLLECT_PGO_ONLY=true, skipping PGO/BOLT optimized binary build"
echo "Profile: $PGO_DIR/merged.profdata"
gha_section_end
exit 0
fi
if [ "$SKIP_BOLT_BOOL" = true ]; then
gha_section_start "BOLT Phase Skipped"
echo "SKIP_BOLT=$SKIP_BOLT, building PGO-only binary"
echo "Building PGO-optimized binary..."
RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata $COMBINED_RUSTFLAGS" \
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
echo "Stripping debug symbols..."
strip "$BUILT_BIN"
else
echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
fi
publish_binary "$BUILT_BIN"
gha_section_end
else
# ── Phase 2: BOLT profile collection (with PGO) ──────────────────────────
gha_section_start "Phase 2: BOLT Profile Collection (with PGO)"
rm -rf "$BOLT_DIR"
mkdir -p "$BOLT_DIR"
echo "Building BOLT-instrumented binary with PGO..."
# --emit-relocs preserves relocation entries in the binary, required by llvm-bolt -instrument
RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
# Instrument with BOLT
BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
BOLT_INSTRUMENTED_BIN="$BUILT_BIN-bolt-instrumented"
echo "Instrumenting binary with BOLT..."
# --skip-funcs: skip compiler-generated drop_in_place functions that BOLT can't handle
# as split functions in relocation mode (triggered by --emit-relocs)
llvm-bolt "$BUILT_BIN" \
-instrument \
--instrumentation-file-append-pid \
--instrumentation-file="$BOLT_DIR/prof" \
--skip-funcs='.*drop_in_place.*' \
-o "$BOLT_INSTRUMENTED_BIN"
echo "BOLT-instrumented binary: $BOLT_INSTRUMENTED_BIN ($(ls -lh "$BOLT_INSTRUMENTED_BIN" | awk '{print $5}'))"
run_bench_workload "$BOLT_INSTRUMENTED_BIN" "$BOLT_BLOCKS" "bolt"
# Fix ownership for BOLT profiles
sudo chown -R "$(id -un):$(id -gn)" "$BOLT_DIR" 2>/dev/null || true
# Merge BOLT profiles
echo "Merging BOLT profiles..."
FDATA_COUNT=$(find "$BOLT_DIR" -name '*.fdata' | wc -l)
echo "Found $FDATA_COUNT .fdata files"
if [ "$FDATA_COUNT" -eq 0 ]; then
echo "error: no .fdata files — BOLT-instrumented binary did not produce profiles"
exit 1
fi
merge-fdata "$BOLT_DIR"/*.fdata > "$BOLT_DIR/merged.fdata"
echo "BOLT profile: $BOLT_DIR/merged.fdata ($(ls -lh "$BOLT_DIR/merged.fdata" | awk '{print $5}'))"
gha_section_end
# ── Phase 3: Final optimized build ───────────────────────────────────────
gha_section_start "Phase 3: Final PGO+BOLT Optimized Build"
echo "Building PGO-optimized binary..."
# --emit-relocs preserves relocation entries in the binary, required by llvm-bolt for code reordering
RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
OPTIMIZED_BIN="$BUILT_BIN-bolt-optimized"
echo "Optimizing with BOLT..."
llvm-bolt "$BUILT_BIN" \
-o "$OPTIMIZED_BIN" \
--data "$BOLT_DIR/merged.fdata" \
-reorder-blocks=ext-tsp \
-reorder-functions=cdsort \
-split-functions \
-split-all-cold \
-dyno-stats \
-icf=1 \
-use-gnu-stack \
--skip-funcs='.*drop_in_place.*'
if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
echo "Stripping debug symbols..."
strip "$OPTIMIZED_BIN"
else
echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
fi
publish_binary "$OPTIMIZED_BIN"
gha_section_end
fi
gha_section_start "Build Complete"
ls -lh "target/$PROFILE_DIR/reth"
echo "Output: target/$PROFILE_DIR/reth"
gha_section_end

View File

@@ -16,11 +16,28 @@ rpc-compat:
# syncing mode, the test expects syncing to be false on start
- eth_syncing/check-syncing (reth)
engine-withdrawals: [ ]
# no fix due to https://github.com/paradigmxyz/reth/issues/8732
engine-withdrawals:
- Withdrawals Fork On Genesis (Paris) (reth)
- Withdrawals Fork on Block 1 (Paris) (reth)
- Withdrawals Fork on Block 2 (Paris) (reth)
- Withdrawals Fork on Block 3 (Paris) (reth)
- Withdraw to a single account (Paris) (reth)
- Withdraw to two accounts (Paris) (reth)
- Withdraw many accounts (Paris) (reth)
- Withdraw zero amount (Paris) (reth)
- Empty Withdrawals (Paris) (reth)
- Corrupted Block Hash Payload (INVALID) (Paris) (reth)
- Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org (Paris) (reth)
engine-api: [ ]
engine-cancun: [ ]
# no fix due to https://github.com/paradigmxyz/reth/issues/8732
engine-cancun:
- Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True (Cancun) (reth)
# the test fails with older versions of the code for which it passed before, probably related to changes
# in hive or its dependencies
- Blob Transaction Ordering, Multiple Clients (Cancun) (reth)
sync: [ ]
@@ -32,7 +49,7 @@ engine-auth: [ ]
# The test artificially creates an empty account with storage, then tests EIP-7610's behavior.
# On mainnet, ~25 such accounts exist as contract addresses (derived from keccak(prefix, caller,
# nonce/salt), not from public keys). No private key exists for contract addresses. To trigger
# this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible.
# this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible.
#
# tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_*
# Requires hash collision on create2 address to target already deployed accounts with storage.

View File

@@ -1,865 +0,0 @@
# Nightly regression benchmark.
#
# Compares the previous nightly build against the current nightly build to
# detect performance regressions. Runs daily after docker.yml produces a new
# nightly image at 01:00 UTC.
#
# State is persisted between runs via GitHub Actions cache: each successful
# run saves the feature commit SHA so the next run knows what to compare against.
on:
schedule:
- cron: "30 5 * * *" # 06:30 UTC daily
workflow_dispatch:
inputs:
force:
description: "Force run even if no new nightly (bypass skip logic)"
required: false
default: false
type: boolean
no_slack:
description: "Suppress Slack notifications"
required: false
default: true
type: boolean
env:
CARGO_TERM_COLOR: always
RUSTC_WRAPPER: "sccache"
name: bench-scheduled
permissions:
contents: read
actions: read
jobs:
# ---------------------------------------------------------------------------
# Job 1: Resolve nightly refs, check staleness, manage state
# ---------------------------------------------------------------------------
resolve-refs:
name: resolve-refs
runs-on: ubuntu-latest
outputs:
baseline-ref: ${{ steps.refs.outputs.baseline-ref }}
feature-ref: ${{ steps.refs.outputs.feature-ref }}
should-skip: ${{ steps.refs.outputs.should-skip }}
is-stale: ${{ steps.refs.outputs.is-stale }}
stale-age-hours: ${{ steps.refs.outputs.stale-age-hours }}
nightly-created: ${{ steps.refs.outputs.nightly-created }}
steps:
- uses: actions/checkout@v6
with:
sparse-checkout: .github/scripts
sparse-checkout-cone-mode: true
- name: Restore nightly state
id: state-cache
uses: actions/cache/restore@v4
with:
path: .nightly-state
key: bench-scheduled-state-dummy
restore-keys: |
bench-scheduled-state-
- name: Resolve nightly refs
id: refs
env:
GH_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
FORCE="${{ inputs.force || 'false' }}"
.github/scripts/bench-scheduled-refs.sh "$FORCE"
- name: Alert on stale nightly
if: steps.refs.outputs.is-stale == 'true'
uses: actions/github-script@v8
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
with:
script: |
const token = process.env.SLACK_BENCH_BOT_TOKEN;
const channel = process.env.SLACK_BENCH_CHANNEL;
if (!token || !channel) {
core.warning('Slack credentials not set, skipping stale nightly alert');
return;
}
const ageHours = '${{ steps.refs.outputs.stale-age-hours }}';
const created = '${{ steps.refs.outputs.nightly-created }}';
const featureRef = '${{ steps.refs.outputs.feature-ref }}';
const shortSha = featureRef.slice(0, 8);
const repo = '${{ github.repository }}';
const runUrl = `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
const blocks = [
{
type: 'header',
text: { type: 'plain_text', text: ':rotating_light: Nightly Regression: nightly build is stale', emoji: true },
},
{
type: 'section',
text: {
type: 'mrkdwn',
text: [
'*Nightly regression did not run* — nightly build is stale',
'',
`The latest nightly image was built from a commit that is *${ageHours}h old* (threshold: 24h).`,
`This means today's nightly docker build likely failed and no new image was produced.`,
'',
`Stale commit: \`${shortSha}\` (built at ${created})`,
'',
'*Action required:* Check the <https://github.com/' + repo + '/actions/workflows/docker.yml|docker.yml> workflow for failures.',
].join('\n'),
},
},
{
type: 'actions',
elements: [
{
type: 'button',
text: { type: 'plain_text', text: 'View Run :github:', emoji: true },
url: runUrl,
action_id: 'ci_button',
},
],
},
];
const resp = await fetch('https://slack.com/api/chat.postMessage', {
method: 'POST',
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
channel,
blocks,
text: 'Nightly regression: nightly build is stale',
unfurl_links: false,
}),
});
const data = await resp.json();
if (!data.ok) {
core.warning(`Slack API error: ${JSON.stringify(data)}`);
}
- name: Fail on stale nightly
if: steps.refs.outputs.is-stale == 'true'
run: |
echo "::error::Nightly build is stale (>24h old). Aborting."
exit 1
# ---------------------------------------------------------------------------
# Job 2: Run the benchmark
# ---------------------------------------------------------------------------
bench-scheduled:
needs: resolve-refs
if: |
needs.resolve-refs.outputs.should-skip != 'true' &&
needs.resolve-refs.outputs.is-stale != 'true'
name: bench-scheduled
runs-on: [self-hosted, Linux, X64, available]
timeout-minutes: 120
env:
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
SCHELK_MOUNT: /reth-bench
BENCH_WORK_DIR: ${{ github.workspace }}/bench-work
BENCH_PR: ""
BENCH_ACTOR: "nightly-regression"
BENCH_BLOCKS: "2000"
BENCH_WARMUP_BLOCKS: "500"
BENCH_SAMPLY: "false"
BENCH_CORES: "0"
BENCH_BIG_BLOCKS: "false"
BENCH_RETH_NEW_PAYLOAD: "true"
BENCH_WAIT_TIME: ""
BENCH_BASELINE_ARGS: ""
BENCH_FEATURE_ARGS: ""
BENCH_ABBA: "true"
BENCH_COMMENT_ID: ""
BENCH_NO_SLACK: ${{ github.event_name == 'workflow_dispatch' && inputs.no_slack == true && 'true' || 'false' }}
BENCH_METRICS_ADDR: "127.0.0.1:9100"
BENCH_OTLP_TRACES_ENDPOINT: ${{ secrets.BENCH_OTLP_TRACES_ENDPOINT }}
BENCH_OTLP_LOGS_ENDPOINT: ${{ secrets.BENCH_OTLP_LOGS_ENDPOINT }}
BASELINE_REF: ${{ needs.resolve-refs.outputs.baseline-ref }}
FEATURE_REF: ${{ needs.resolve-refs.outputs.feature-ref }}
steps:
- name: Clean up previous bench-work
run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
- uses: actions/checkout@v6
with:
submodules: true
fetch-depth: 0
ref: ${{ needs.resolve-refs.outputs.feature-ref }}
- name: Resolve job URL
id: job-url
uses: actions/github-script@v8
with:
script: |
const { data: jobs } = await github.rest.actions.listJobsForWorkflowRun({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.runId,
});
const job = jobs.jobs.find(j => j.name === 'bench-scheduled');
const jobUrl = job ? job.html_url : `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
core.exportVariable('BENCH_JOB_URL', jobUrl);
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@v0.0.9
continue-on-error: true
- name: Install dependencies
env:
DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
run: |
mkdir -p "$HOME/.local/bin"
# apt packages
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends \
python3 make jq zstd curl dmsetup \
linux-tools-"$(uname -r)" || \
sudo apt-get install -y --no-install-recommends linux-tools-generic
# mc (MinIO client)
if ! command -v mc &>/dev/null; then
curl -sSfL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$HOME/.local/bin/mc"
chmod +x "$HOME/.local/bin/mc"
fi
# uv (Python package manager)
if ! command -v uv &>/dev/null; then
curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$HOME/.local/bin" sh
fi
# Configure git auth for private repos
git config --global url."https://x-access-token:${DEREK_TOKEN}@github.com/".insteadOf "https://github.com/"
# thin-provisioning-tools (era_invalidate, required by schelk)
if ! command -v era_invalidate &>/dev/null; then
git clone --depth 1 https://github.com/jthornber/thin-provisioning-tools /tmp/tpt
sudo make -C /tmp/tpt install
rm -rf /tmp/tpt
fi
# schelk (snapshot rollback tool, invoked via sudo)
if ! sudo sh -c 'command -v schelk' &>/dev/null; then
cargo install --git https://github.com/tempoxyz/schelk --locked
sudo install "$HOME/.cargo/bin/schelk" /usr/local/bin/
fi
- name: Check dependencies
run: |
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
missing=()
for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq; do
command -v "$cmd" &>/dev/null || missing+=("$cmd")
done
if [ ${#missing[@]} -gt 0 ]; then
echo "::error::Missing required tools: ${missing[*]}"
exit 1
fi
echo "All dependencies found"
- name: Resolve display names
id: refs
run: |
BASELINE_SHORT=$(echo "$BASELINE_REF" | cut -c1-8)
FEATURE_SHORT=$(echo "$FEATURE_REF" | cut -c1-8)
echo "baseline-name=nightly-${BASELINE_SHORT}" >> "$GITHUB_OUTPUT"
echo "feature-name=nightly-${FEATURE_SHORT}" >> "$GITHUB_OUTPUT"
echo "baseline-ref=$BASELINE_REF" >> "$GITHUB_OUTPUT"
echo "feature-ref=$FEATURE_REF" >> "$GITHUB_OUTPUT"
- name: Check if snapshot needs update
id: snapshot-check
run: |
if .github/scripts/bench-reth-snapshot.sh --check; then
echo "needed=false" >> "$GITHUB_OUTPUT"
else
echo "needed=true" >> "$GITHUB_OUTPUT"
fi
- name: Prepare source dirs
run: |
if [ -d ../reth-baseline ]; then
git -C ../reth-baseline fetch origin "$BASELINE_REF"
else
git clone . ../reth-baseline
fi
git -C ../reth-baseline checkout "$BASELINE_REF"
if [ -d ../reth-feature ]; then
git -C ../reth-feature fetch origin "$FEATURE_REF"
else
git clone . ../reth-feature
fi
git -C ../reth-feature checkout "$FEATURE_REF"
- name: Build binaries and download snapshot in parallel
id: build
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BENCH_REPO: ${{ github.repository }}
SNAPSHOT_NEEDED: ${{ steps.snapshot-check.outputs.needed }}
run: |
BASELINE_DIR="$(cd ../reth-baseline && pwd)"
FEATURE_DIR="$(cd ../reth-feature && pwd)"
.github/scripts/bench-reth-build.sh baseline "${BASELINE_DIR}" "$BASELINE_REF" &
PID_BASELINE=$!
.github/scripts/bench-reth-build.sh feature "${FEATURE_DIR}" "$FEATURE_REF" &
PID_FEATURE=$!
PID_SNAPSHOT=
if [ "$SNAPSHOT_NEEDED" = "true" ]; then
.github/scripts/bench-reth-snapshot.sh &
PID_SNAPSHOT=$!
fi
FAIL=0
wait $PID_BASELINE || FAIL=1
wait $PID_FEATURE || FAIL=1
[ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
if [ $FAIL -ne 0 ]; then
echo "::error::One or more parallel tasks failed (builds / snapshot download)"
exit 1
fi
# System tuning for reproducible benchmarks
- name: System setup
run: |
sudo cpupower frequency-set -g performance || true
# Disable turbo boost (Intel and AMD paths)
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
sudo swapoff -a || true
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space || true
# Disable SMT (hyperthreading)
for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
if [ "$current" != "$first" ]; then
echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" || true
fi
done
echo "Online CPUs: $(nproc)"
# Disable transparent huge pages
for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
[ -d "$p" ] && echo never | sudo tee "$p/enabled" && echo never | sudo tee "$p/defrag" && break
done || true
# Prevent deep C-states
sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
# Move all IRQs to core 0
for irq in /proc/irq/*/smp_affinity_list; do
echo 0 | sudo tee "$irq" 2>/dev/null || true
done
# Stop noisy background services
sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
echo "=== Benchmark environment ==="
uname -r
lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA'
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo "THP: unknown"
free -h
- name: Pre-flight cleanup
run: |
sudo pkill -9 reth || true
sleep 1
if mountpoint -q "$SCHELK_MOUNT"; then
sudo umount -l "$SCHELK_MOUNT" || true
sudo schelk recover -y || true
fi
rm -rf "$BENCH_WORK_DIR"
mkdir -p "$BENCH_WORK_DIR"
- name: Start metrics proxy
run: |
BENCH_ID="nightly-${{ github.run_id }}"
BENCH_REFERENCE_EPOCH=$(date +%s)
echo "BENCH_ID=${BENCH_ID}" >> "$GITHUB_ENV"
echo "BENCH_REFERENCE_EPOCH=${BENCH_REFERENCE_EPOCH}" >> "$GITHUB_ENV"
LABELS_FILE="/tmp/bench-metrics-labels.json"
echo '{}' > "$LABELS_FILE"
echo "BENCH_LABELS_FILE=${LABELS_FILE}" >> "$GITHUB_ENV"
python3 .github/scripts/bench-metrics-proxy.py \
--labels "$LABELS_FILE" \
--upstream "http://${BENCH_METRICS_ADDR}/" \
--subnet 10.10.0.0/24 \
--port 9090 &
PROXY_PID=$!
echo "BENCH_METRICS_PROXY_PID=${PROXY_PID}" >> "$GITHUB_ENV"
echo "Metrics proxy started (PID $PROXY_PID)"
# Interleaved run order (B-F-F-B) to reduce systematic bias
- name: "Run benchmark: baseline (1/2)"
id: run-baseline-1
run: |
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
- name: "Run benchmark: feature (1/2)"
id: run-feature-1
run: |
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
- name: "Run benchmark: feature (2/2)"
id: run-feature-2
run: |
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
- name: "Run benchmark: baseline (2/2)"
id: run-baseline-2
run: |
LAST_RUN_START=$(date +%s)
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
- name: Stop metrics proxy & generate Grafana URL
id: metrics
if: "!cancelled()"
run: |
kill "$BENCH_METRICS_PROXY_PID" 2>/dev/null || true
LAST_RUN_DURATION=$(( $(date +%s) - BENCH_LAST_RUN_START ))
FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
GRAFANA_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=ef57fux92e9z4e&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
echo "grafana-url=${GRAFANA_URL}" >> "$GITHUB_OUTPUT"
echo "Grafana URL: ${GRAFANA_URL}"
- name: Scan logs for errors
if: "!cancelled()"
run: |
ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
found=false
for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
LOG="$BENCH_WORK_DIR/$run_dir/node.log"
if [ ! -f "$LOG" ]; then continue; fi
panics=$(grep -c -E 'panicked at' "$LOG" || true)
errors=$(grep -c ' ERROR ' "$LOG" || true)
if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
if [ "$found" = false ]; then
printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
found=true
fi
printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' "$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
if [ "$panics" -gt 0 ]; then
printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
printf '```\n' >> "$ERRORS_FILE"
fi
if [ "$errors" -gt 0 ]; then
printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
printf '```\n' >> "$ERRORS_FILE"
fi
printf '\n</details>\n\n' >> "$ERRORS_FILE"
fi
done
- name: Parse results
id: results
if: success()
env:
BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
run: |
SUMMARY_ARGS="--output-summary $BENCH_WORK_DIR/summary.json"
SUMMARY_ARGS="$SUMMARY_ARGS --output-markdown $BENCH_WORK_DIR/comment.md"
SUMMARY_ARGS="$SUMMARY_ARGS --repo ${{ github.repository }}"
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-ref ${BASELINE_REF}"
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-name ${BASELINE_NAME}"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-name ${FEATURE_NAME}"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-ref ${FEATURE_REF}"
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BASELINE_CSVS"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $FEATURE_CSVS"
SUMMARY_ARGS="$SUMMARY_ARGS --gas-csv $BENCH_WORK_DIR/feature-1/total_gas.csv"
GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
if [ -n "$GRAFANA_URL" ]; then
SUMMARY_ARGS="$SUMMARY_ARGS --grafana-url $GRAFANA_URL"
fi
# shellcheck disable=SC2086
python3 .github/scripts/bench-reth-summary.py $SUMMARY_ARGS
- name: Generate charts
if: success()
env:
BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
run: |
CHART_ARGS="--output-dir $BENCH_WORK_DIR/charts"
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
CHART_ARGS="$CHART_ARGS --feature $FEATURE_CSVS"
CHART_ARGS="$CHART_ARGS --baseline $BASELINE_CSVS"
CHART_ARGS="$CHART_ARGS --baseline-name ${BASELINE_NAME}"
CHART_ARGS="$CHART_ARGS --feature-name ${FEATURE_NAME}"
# shellcheck disable=SC2086
uv run --with matplotlib python3 .github/scripts/bench-reth-charts.py $CHART_ARGS
- name: Upload results
if: "!cancelled()"
uses: actions/upload-artifact@v7
with:
name: bench-scheduled-results
path: ${{ env.BENCH_WORK_DIR }}
- name: Push charts
id: push-charts
if: success()
run: |
RUN_ID=${{ github.run_id }}
CHART_DIR="nightly/${RUN_ID}"
CHARTS_REPO="https://x-access-token:${{ secrets.DEREK_TOKEN }}@github.com/decofe/reth-bench-charts.git"
TMP_DIR=$(mktemp -d)
if git clone --depth 1 "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
true
else
git init "${TMP_DIR}"
git -C "${TMP_DIR}" remote add origin "${CHARTS_REPO}"
fi
mkdir -p "${TMP_DIR}/${CHART_DIR}"
cp "$BENCH_WORK_DIR"/charts/*.png "${TMP_DIR}/${CHART_DIR}/"
git -C "${TMP_DIR}" add "${CHART_DIR}"
git -C "${TMP_DIR}" -c user.name="github-actions" -c user.email="github-actions@github.com" \
commit -m "nightly bench charts for run ${RUN_ID}"
git -C "${TMP_DIR}" push origin HEAD:main
echo "sha=$(git -C "${TMP_DIR}" rev-parse HEAD)" >> "$GITHUB_OUTPUT"
rm -rf "${TMP_DIR}"
- name: Write job summary
if: success()
uses: actions/github-script@v8
with:
script: |
const fs = require('fs');
const { verdict, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
let summary;
try {
summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
} catch (e) {
await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
return;
}
const repo = `${context.repo.owner}/${context.repo.repo}`;
const commitUrl = `https://github.com/${repo}/commit`;
const { emoji, label } = verdict(summary.changes);
const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
let md = `# ${emoji} Nightly Regression: ${label}\n\n`;
md += `**Baseline:** ${baselineLink}\n`;
md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
const rows = metricRows(summary);
md += `| Metric | Baseline | Feature | Change |\n`;
md += `|--------|----------|---------|--------|\n`;
for (const r of rows) {
md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
}
md += '\n';
const wtRows = waitTimeRows(summary);
if (wtRows.length > 0) {
md += `### Wait Time Breakdown\n\n`;
md += `| Metric | Baseline | Feature |\n`;
md += `|--------|----------|--------|\n`;
for (const r of wtRows) {
md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
}
md += '\n';
}
// Charts
const chartSha = '${{ steps.push-charts.outputs.sha }}';
if (chartSha) {
const runId = '${{ github.run_id }}';
const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/nightly/${runId}`;
const charts = [
{ file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
{ file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
{ file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
];
md += `### Charts\n\n`;
for (const chart of charts) {
md += `<details><summary>${chart.label}</summary>\n\n`;
md += `![${chart.label}](${baseUrl}/${chart.file})\n\n`;
md += `</details>\n\n`;
}
}
const grafanaUrl = '${{ steps.metrics.outputs.grafana-url }}';
if (grafanaUrl) {
md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
}
try {
const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
if (errors.trim()) md += '\n' + errors + '\n';
} catch {}
await core.summary.addRaw(md).write();
- name: Send Slack notification (success)
if: success() && env.BENCH_NO_SLACK != 'true'
uses: actions/github-script@v8
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
with:
script: |
const fs = require('fs');
const { verdict, fmtChange, fmtMs, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
const token = process.env.SLACK_BENCH_BOT_TOKEN;
const channel = process.env.SLACK_BENCH_CHANNEL;
if (!token || !channel) {
core.info('Slack credentials not set, skipping notification');
return;
}
let summary;
try {
summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
} catch (e) {
core.warning('Could not read summary.json for Slack notification');
return;
}
// Only notify on significant changes (regression OR improvement)
const changes = summary.changes || {};
const hasSignificant = Object.values(changes).some(c => c.sig === 'good' || c.sig === 'bad');
if (!hasSignificant) {
core.info('No significant changes detected, skipping nightly Slack notification');
return;
}
const SLACK_VERDICT = {
'⚠️': ':warning:',
'❌': ':x:',
'✅': ':white_check_mark:',
'⚪': ':white_circle:',
};
const repo = `${context.repo.owner}/${context.repo.repo}`;
const { emoji, label } = verdict(changes);
const headerEmoji = SLACK_VERDICT[emoji] || emoji;
const commitUrl = `https://github.com/${repo}/commit`;
const baselineLink = `<${commitUrl}/${summary.baseline.ref}|${summary.baseline.name}>`;
const featureLink = `<${commitUrl}/${summary.feature.ref}|${summary.feature.name}>`;
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
function cell(text) { return { type: 'raw_text', text: String(text) || ' ' }; }
const sectionText = [
'*Nightly Regression*',
'',
`*Baseline:* ${baselineLink}`,
`*Feature:* ${featureLink}`,
blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | '),
].join('\n');
const rows = metricRows(summary);
const tableRows = [
[cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
];
const blocks = [
{
type: 'header',
text: { type: 'plain_text', text: `${headerEmoji} Nightly: ${label}`, emoji: true },
},
{
type: 'section',
text: { type: 'mrkdwn', text: sectionText },
},
{
type: 'table',
column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }, { align: 'right' }],
rows: tableRows,
},
{
type: 'actions',
elements: [
{
type: 'button',
text: { type: 'plain_text', text: 'CI :github:', emoji: true },
url: jobUrl,
action_id: 'ci_button',
},
{
type: 'button',
text: { type: 'plain_text', text: 'Diff :github:', emoji: true },
url: diffUrl,
action_id: 'diff_button',
},
],
},
];
const text = `Nightly Regression: ${summary.baseline.name} vs ${summary.feature.name}`;
const resp = await fetch('https://slack.com/api/chat.postMessage', {
method: 'POST',
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({ channel, blocks, text, unfurl_links: false }),
});
const data = await resp.json();
if (!data.ok) {
core.warning(`Slack API error: ${JSON.stringify(data)}`);
return;
}
// Post wait time breakdown as threaded reply
const wtRows = waitTimeRows(summary);
if (data.ts && wtRows.length > 0) {
const waitTableRows = [
[cell('Wait Time'), cell('Baseline'), cell('Feature')],
...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
];
await fetch('https://slack.com/api/chat.postMessage', {
method: 'POST',
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
channel,
thread_ts: data.ts,
blocks: [{
type: 'table',
column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }],
rows: waitTableRows,
}],
text: 'Wait time breakdown',
unfurl_links: false,
}),
});
}
- name: Send Slack notification (failure)
if: failure()
uses: actions/github-script@v8
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
with:
script: |
const token = process.env.SLACK_BENCH_BOT_TOKEN;
const channel = process.env.SLACK_BENCH_CHANNEL;
if (!token || !channel) return;
const steps_status = [
['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
];
const failed = steps_status.find(([, o]) => o === 'failure');
const failedStep = failed ? failed[0] : 'unknown step';
const repo = `${context.repo.owner}/${context.repo.repo}`;
const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
const blocks = [
{
type: 'header',
text: { type: 'plain_text', text: ':rotating_light: Nightly Bench Failed', emoji: true },
},
{
type: 'section',
text: { type: 'mrkdwn', text: `*Nightly regression* failed while *${failedStep}*` },
},
{
type: 'actions',
elements: [{
type: 'button',
text: { type: 'plain_text', text: 'View Logs :github:', emoji: true },
url: jobUrl,
action_id: 'ci_button',
}],
},
];
await fetch('https://slack.com/api/chat.postMessage', {
method: 'POST',
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
channel,
blocks,
text: `Nightly bench failed while ${failedStep}`,
unfurl_links: false,
}),
});
- name: Restore system settings
if: always()
run: |
sudo systemctl start irqbalance cron atd 2>/dev/null || true
# ---------------------------------------------------------------------------
# Job 3: Save state on success
# ---------------------------------------------------------------------------
save-state:
needs: [resolve-refs, bench-scheduled]
if: success()
name: save-state
runs-on: ubuntu-latest
steps:
- name: Write state file
run: |
mkdir -p .nightly-state
echo "${{ needs.resolve-refs.outputs.feature-ref }}" > .nightly-state/last-feature-ref
- name: Save nightly state
uses: actions/cache/save@v4
with:
path: .nightly-state
key: bench-scheduled-state-${{ needs.resolve-refs.outputs.feature-ref }}

View File

@@ -7,12 +7,16 @@
# same block range (snapshot recovered between runs) to compare performance.
on:
# TODO: Disabled temporarily for https://github.com/CodSpeedHQ/runner/issues/55
# merge_group:
push:
branches: [main]
issue_comment:
types: [created]
types: [created, edited]
workflow_dispatch:
inputs:
blocks:
description: "Number of blocks to benchmark (or 'big' for big blocks mode)"
description: "Number of blocks to benchmark"
required: false
default: "500"
type: string
@@ -31,51 +35,16 @@ on:
required: false
default: ""
type: string
wait_time:
description: "Fixed wait time between blocks (e.g. 500ms, 1s)"
required: false
default: ""
type: string
baseline_args:
description: "Extra CLI args for the baseline reth node"
required: false
default: ""
type: string
feature_args:
description: "Extra CLI args for the feature reth node"
required: false
default: ""
type: string
samply:
description: "Enable samply profiling"
required: false
default: "false"
type: boolean
reth_newPayload:
description: "Use reth_newPayload RPC (server-side timing)"
required: false
default: "true"
type: boolean
cores:
description: "Limit reth to N CPU cores (0 = all available)"
required: false
default: "0"
type: string
no_slack:
description: "Suppress Slack notifications for benchmark results"
required: false
default: "true"
type: boolean
abba:
description: "Run ABBA (BFFB) interleaved order; false = single AB pass"
required: false
default: "true"
type: boolean
otlp:
description: "Export OTLP traces and logs"
required: false
default: "true"
type: boolean
env:
CARGO_TERM_COLOR: always
@@ -91,6 +60,47 @@ permissions:
pull-requests: write
jobs:
codspeed:
if: github.event_name == 'push'
continue-on-error: true
runs-on: depot-ubuntu-latest
concurrency:
group: bench-codspeed-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
strategy:
matrix:
partition: [1, 2]
total_partitions: [2]
include:
- partition: 1
crates: "-p reth-primitives -p reth-trie-common -p reth-trie-sparse"
- partition: 2
crates: "-p reth-trie"
name: codspeed (${{ matrix.partition }}/${{ matrix.total_partitions }})
steps:
- uses: actions/checkout@v6
with:
submodules: true
ref: ${{ github.event_name == 'issue_comment' && format('refs/pull/{0}/merge', github.event.issue.number) || '' }}
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
- uses: mozilla-actions/sccache-action@v0.0.9
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Install cargo-codspeed
uses: taiki-e/install-action@v2
with:
tool: cargo-codspeed
- name: Build the benchmark target(s)
run: cargo codspeed build --profile profiling --features test-utils ${{ matrix.crates }}
- name: Run the benchmarks
uses: CodSpeedHQ/action@v4
with:
run: cargo codspeed run ${{ matrix.crates }}
mode: instrumentation
token: ${{ secrets.CODSPEED_TOKEN }}
reth-bench-ack:
if: |
(github.event_name == 'issue_comment' && github.event.issue.pull_request && (startsWith(github.event.comment.body, '@decofe bench') || startsWith(github.event.comment.body, 'derek bench'))) ||
@@ -107,15 +117,7 @@ jobs:
baseline-name: ${{ steps.args.outputs.baseline-name }}
feature-name: ${{ steps.args.outputs.feature-name }}
samply: ${{ steps.args.outputs.samply }}
no-slack: ${{ steps.args.outputs.no-slack }}
cores: ${{ steps.args.outputs.cores }}
big-blocks: ${{ steps.args.outputs.big-blocks }}
reth-new-payload: ${{ steps.args.outputs.reth-new-payload }}
wait-time: ${{ steps.args.outputs.wait-time }}
baseline-args: ${{ steps.args.outputs.baseline-args }}
feature-args: ${{ steps.args.outputs.feature-args }}
abba: ${{ steps.args.outputs.abba }}
otlp: ${{ steps.args.outputs.otlp }}
comment-id: ${{ steps.ack.outputs.comment-id }}
steps:
- name: Check org membership
@@ -143,7 +145,7 @@ jobs:
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
let pr, actor, blocks, warmup, baseline, feature, samply, cores, bigBlocks;
let pr, actor, blocks, warmup, baseline, feature, samply, cores;
if (context.eventName === 'workflow_dispatch') {
actor = '${{ github.actor }}';
@@ -152,15 +154,7 @@ jobs:
baseline = '${{ github.event.inputs.baseline }}';
feature = '${{ github.event.inputs.feature }}';
samply = '${{ github.event.inputs.samply }}' === 'true' ? 'true' : 'false';
var noSlack = '${{ github.event.inputs.no_slack }}' !== 'false' ? 'true' : 'false';
cores = '${{ github.event.inputs.cores }}' || '0';
bigBlocks = blocks === 'big' ? 'true' : 'false';
var rethNewPayload = '${{ github.event.inputs.reth_newPayload }}' !== 'false' ? 'true' : 'false';
var abba = '${{ github.event.inputs.abba }}' !== 'false' ? 'true' : 'false';
var otlp = '${{ github.event.inputs.otlp }}' !== 'false' ? 'true' : 'false';
var waitTime = '${{ github.event.inputs.wait_time }}' || '';
var baselineNodeArgs = '${{ github.event.inputs.baseline_args }}' || '';
var featureNodeArgs = '${{ github.event.inputs.feature_args }}' || '';
// Find PR for the selected branch
const branch = '${{ github.ref_name }}';
@@ -180,75 +174,37 @@ jobs:
actor = context.payload.comment.user.login;
const body = context.payload.comment.body.trim();
const intArgs = new Set(['warmup', 'cores']);
const intOrKeywordArgs = new Map([['blocks', new Set(['big'])]]);
const intArgs = new Set(['blocks', 'warmup', 'cores']);
const refArgs = new Set(['baseline', 'feature']);
const boolArgs = new Set(['samply', 'no-slack']);
const boolDefaultTrue = new Set(['reth_newPayload', 'abba', 'otlp']);
const durationArgs = new Set(['wait-time']);
const stringArgs = new Set(['baseline-args', 'feature-args']);
const defaults = { blocks: '500', warmup: '100', baseline: '', feature: '', samply: 'false', 'no-slack': 'false', cores: '0', reth_newPayload: 'true', abba: 'true', otlp: 'true', 'wait-time': '', 'baseline-args': '', 'feature-args': '' };
const boolArgs = new Set(['samply']);
const defaults = { blocks: '500', warmup: '100', baseline: '', feature: '', samply: 'false', cores: '0' };
const unknown = [];
const invalid = [];
const args = body.replace(/^(?:@decofe|derek) bench\s*/, '');
// Parse args, handling quoted values like key="value with spaces"
const parts = [];
const argRegex = /(\S+?="[^"]*"|\S+?='[^']*'|\S+)/g;
let m;
while ((m = argRegex.exec(args)) !== null) parts.push(m[1]);
for (const part of parts) {
for (const part of args.split(/\s+/).filter(Boolean)) {
const eq = part.indexOf('=');
if (eq === -1) {
if (boolArgs.has(part)) {
defaults[part] = 'true';
} else if (boolDefaultTrue.has(part)) {
defaults[part] = 'true';
} else {
unknown.push(part);
}
continue;
}
const key = part.slice(0, eq);
let value = part.slice(eq + 1);
// Strip surrounding quotes
if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
value = value.slice(1, -1);
}
if (boolDefaultTrue.has(key)) {
if (value === 'true' || value === 'false') {
defaults[key] = value;
} else {
invalid.push(`\`${key}=${value}\` (must be true or false)`);
}
} else if (durationArgs.has(key)) {
if (/^\d+(ms|s|m)$/.test(value)) {
defaults[key] = value;
} else {
invalid.push(`\`${key}=${value}\` (must be a duration like 500ms, 1s, 2m)`);
}
} else if (intArgs.has(key)) {
const value = part.slice(eq + 1);
if (intArgs.has(key)) {
if (!/^\d+$/.test(value)) {
invalid.push(`\`${key}=${value}\` (must be a positive integer)`);
} else {
defaults[key] = value;
}
} else if (intOrKeywordArgs.has(key)) {
const keywords = intOrKeywordArgs.get(key);
if (keywords.has(value)) {
defaults[key] = value;
} else if (/^\d+$/.test(value)) {
defaults[key] = value;
} else {
invalid.push(`\`${key}=${value}\` (must be a positive integer or one of: ${[...keywords].join(', ')})`);
}
} else if (refArgs.has(key)) {
if (!value) {
invalid.push(`\`${key}=\` (must be a git ref)`);
} else {
defaults[key] = value;
}
} else if (stringArgs.has(key)) {
defaults[key] = value;
} else {
unknown.push(key);
}
@@ -257,7 +213,7 @@ jobs:
if (unknown.length) errors.push(`Unknown argument(s): \`${unknown.join('`, `')}\``);
if (invalid.length) errors.push(`Invalid value(s): ${invalid.join(', ')}`);
if (errors.length) {
const msg = `❌ **Invalid bench command**\n\n${errors.join('\n')}\n\n**Usage:** \`@decofe bench [blocks=N|big] [warmup=N] [baseline=REF] [feature=REF] [samply] [no-slack] [cores=N] [reth_newPayload=true|false] [abba=true|false] [otlp=true|false] [wait-time=DURATION] [baseline-args="..."] [feature-args="..."]\``;
const msg = `❌ **Invalid bench command**\n\n${errors.join('\n')}\n\n**Usage:** \`@decofe bench [blocks=N] [warmup=N] [baseline=REF] [feature=REF] [samply] [cores=N]\``;
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
@@ -272,15 +228,7 @@ jobs:
baseline = defaults.baseline;
feature = defaults.feature;
samply = defaults.samply;
var noSlack = defaults['no-slack'];
cores = defaults.cores;
bigBlocks = blocks === 'big' ? 'true' : 'false';
var rethNewPayload = defaults.reth_newPayload;
var abba = defaults.abba;
var otlp = defaults.otlp;
var waitTime = defaults['wait-time'];
var baselineNodeArgs = defaults['baseline-args'];
var featureNodeArgs = defaults['feature-args'];
}
// Resolve display names for baseline/feature
@@ -308,15 +256,7 @@ jobs:
core.setOutput('baseline-name', baselineName);
core.setOutput('feature-name', featureName);
core.setOutput('samply', samply);
core.setOutput('no-slack', noSlack);
core.setOutput('cores', cores);
core.setOutput('big-blocks', bigBlocks);
core.setOutput('reth-new-payload', rethNewPayload);
core.setOutput('wait-time', waitTime);
core.setOutput('baseline-args', baselineNodeArgs);
core.setOutput('feature-args', featureNodeArgs);
core.setOutput('abba', abba);
core.setOutput('otlp', otlp);
- name: Acknowledge request
id: ack
@@ -356,6 +296,7 @@ jobs:
});
allRuns.push(...r);
}
// Only count runs that trigger reth-bench (not push-triggered codspeed runs)
const benchRuns = allRuns.filter(r => r.event === 'issue_comment' || r.event === 'workflow_dispatch');
const thisRun = benchRuns.find(r => r.id === context.runId);
const thisCreatedAt = thisRun ? new Date(thisRun.created_at) : new Date();
@@ -374,26 +315,10 @@ jobs:
const baseline = '${{ steps.args.outputs.baseline-name }}';
const feature = '${{ steps.args.outputs.feature-name }}';
const samply = '${{ steps.args.outputs.samply }}' === 'true';
const noSlack = '${{ steps.args.outputs.no-slack }}' === 'true';
const bigBlocks = '${{ steps.args.outputs.big-blocks }}' === 'true';
const samplyNote = samply ? ', samply: `enabled`' : '';
const noSlackNote = noSlack ? ', no-slack' : '';
const cores = '${{ steps.args.outputs.cores }}';
const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
const rethNP = '${{ steps.args.outputs.reth-new-payload }}' !== 'false';
const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
const abbaEnabled = '${{ steps.args.outputs.abba }}' !== 'false';
const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
const otlpEnabled = '${{ steps.args.outputs.otlp }}' !== 'false';
const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
const waitTimeVal = '${{ steps.args.outputs.wait-time }}';
const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
const baselineArgsVal = '${{ steps.args.outputs.baseline-args }}';
const baselineArgsNote = baselineArgsVal ? `, baseline-args: \`${baselineArgsVal}\`` : '';
const featureArgsVal = '${{ steps.args.outputs.feature-args }}';
const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
const config = `**Config:** ${blocks} blocks, ${warmup} warmup blocks, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${coresNote}`;
const { data: comment } = await github.rest.issues.createComment({
owner: context.repo.owner,
@@ -418,26 +343,10 @@ jobs:
const baseline = '${{ steps.args.outputs.baseline-name }}';
const feature = '${{ steps.args.outputs.feature-name }}';
const samply = '${{ steps.args.outputs.samply }}' === 'true';
const noSlack = '${{ steps.args.outputs.no-slack }}' === 'true';
const bigBlocks = '${{ steps.args.outputs.big-blocks }}' === 'true';
const samplyNote = samply ? ', samply: `enabled`' : '';
const noSlackNote = noSlack ? ', no-slack' : '';
const cores = '${{ steps.args.outputs.cores }}';
const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
const rethNP = '${{ steps.args.outputs.reth-new-payload }}' !== 'false';
const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
const abbaEnabled = '${{ steps.args.outputs.abba }}' !== 'false';
const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
const otlpEnabled = '${{ steps.args.outputs.otlp }}' !== 'false';
const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
const waitTimeVal = '${{ steps.args.outputs.wait-time }}';
const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
const baselineArgsVal = '${{ steps.args.outputs.baseline-args }}';
const baselineArgsNote = baselineArgsVal ? `, baseline-args: \`${baselineArgsVal}\`` : '';
const featureArgsVal = '${{ steps.args.outputs.feature-args }}';
const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
const config = `**Config:** ${blocks} blocks, ${warmup} warmup blocks, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${coresNote}`;
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const numRunners = parseInt(process.env.BENCH_RUNNERS) || 1;
@@ -489,7 +398,7 @@ jobs:
reth-bench:
needs: reth-bench-ack
name: reth-bench
runs-on: [self-hosted, Linux, X64, available]
runs-on: [self-hosted, Linux, X64]
timeout-minutes: 120
env:
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
@@ -501,22 +410,8 @@ jobs:
BENCH_WARMUP_BLOCKS: ${{ needs.reth-bench-ack.outputs.warmup }}
BENCH_SAMPLY: ${{ needs.reth-bench-ack.outputs.samply }}
BENCH_CORES: ${{ needs.reth-bench-ack.outputs.cores }}
BENCH_BIG_BLOCKS: ${{ needs.reth-bench-ack.outputs.big-blocks }}
BENCH_RETH_NEW_PAYLOAD: ${{ needs.reth-bench-ack.outputs.reth-new-payload }}
BENCH_WAIT_TIME: ${{ needs.reth-bench-ack.outputs.wait-time }}
BENCH_BASELINE_ARGS: ${{ needs.reth-bench-ack.outputs.baseline-args }}
BENCH_FEATURE_ARGS: ${{ needs.reth-bench-ack.outputs.feature-args }}
BENCH_ABBA: ${{ needs.reth-bench-ack.outputs.abba }}
BENCH_OTLP: ${{ needs.reth-bench-ack.outputs.otlp }}
BENCH_COMMENT_ID: ${{ needs.reth-bench-ack.outputs.comment-id }}
BENCH_NO_SLACK: ${{ needs.reth-bench-ack.outputs.no-slack }}
BENCH_METRICS_ADDR: "127.0.0.1:9100"
BENCH_OTLP_TRACES_ENDPOINT: ${{ needs.reth-bench-ack.outputs.otlp != 'false' && secrets.BENCH_OTLP_TRACES_ENDPOINT || '' }}
BENCH_OTLP_LOGS_ENDPOINT: ${{ needs.reth-bench-ack.outputs.otlp != 'false' && secrets.BENCH_OTLP_LOGS_ENDPOINT || '' }}
steps:
- name: Clean up previous bench-work
run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
- name: Resolve checkout ref
id: checkout-ref
uses: actions/github-script@v8
@@ -531,11 +426,13 @@ jobs:
repo: context.repo.repo,
pull_number: parseInt(process.env.BENCH_PR),
});
// Always use head SHA — the merge ref (refs/pull/N/merge) may not
// exist if the PR has conflicts, was force-pushed, or was
// merged/closed between this step and checkout.
core.info(`PR #${process.env.BENCH_PR} (${pr.state}), using head SHA ${pr.head.sha}`);
core.setOutput('ref', pr.head.sha);
// For closed/merged PRs, the merge ref doesn't exist — use head SHA
if (pr.state !== 'open') {
core.info(`PR #${process.env.BENCH_PR} is ${pr.state}, using head SHA ${pr.head.sha}`);
core.setOutput('ref', pr.head.sha);
} else {
core.setOutput('ref', `refs/pull/${process.env.BENCH_PR}/merge`);
}
- uses: actions/checkout@v6
with:
@@ -563,26 +460,10 @@ jobs:
const baseline = '${{ needs.reth-bench-ack.outputs.baseline-name }}';
const feature = '${{ needs.reth-bench-ack.outputs.feature-name }}';
const samply = process.env.BENCH_SAMPLY === 'true';
const noSlack = process.env.BENCH_NO_SLACK === 'true';
const bigBlocks = process.env.BENCH_BIG_BLOCKS === 'true';
const samplyNote = samply ? ', samply: `enabled`' : '';
const noSlackNote = noSlack ? ', no-slack' : '';
const cores = process.env.BENCH_CORES || '0';
const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
const rethNP = (process.env.BENCH_RETH_NEW_PAYLOAD || 'true') !== 'false';
const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
const abbaEnabled = (process.env.BENCH_ABBA || 'true') !== 'false';
const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
const otlpEnabled = (process.env.BENCH_OTLP || 'true') !== 'false';
const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
const waitTimeVal = process.env.BENCH_WAIT_TIME || '';
const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
const baselineArgsVal = process.env.BENCH_BASELINE_ARGS || '';
const baselineArgsNote = baselineArgsVal ? `, baseline-args: \`${baselineArgsVal}\`` : '';
const featureArgsVal = process.env.BENCH_FEATURE_ARGS || '';
const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
core.exportVariable('BENCH_CONFIG', `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`);
core.exportVariable('BENCH_CONFIG', `**Config:** ${blocks} blocks, ${warmup} warmup blocks, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${coresNote}`);
const { buildBody } = require('./.github/scripts/bench-update-status.js');
await github.rest.issues.updateComment({
@@ -842,45 +723,6 @@ jobs:
rm -rf "$BENCH_WORK_DIR"
mkdir -p "$BENCH_WORK_DIR"
- name: Download big blocks
if: env.BENCH_BIG_BLOCKS == 'true'
run: |
set -euo pipefail
MC="mc --config-dir /home/ubuntu/.mc"
BUCKET="minio/reth-snapshots/reth-1-minimal-nightly-previous-big-blocks.tar.zst"
BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
rm -rf "$BIG_BLOCKS_DIR"; mkdir -p "$BIG_BLOCKS_DIR"
echo "Downloading big blocks from $BUCKET..."
$MC cat "$BUCKET" | pzstd -d -p 6 | tar -xf - -C "$BIG_BLOCKS_DIR"
echo "Big blocks downloaded to $BIG_BLOCKS_DIR"
# Verify expected directory structure
if [ ! -d "$BIG_BLOCKS_DIR/gas-ramp-dir" ] || [ ! -d "$BIG_BLOCKS_DIR/payloads" ]; then
echo "::error::Big blocks archive missing expected gas-ramp-dir/ or payloads/ directories"
ls -laR "$BIG_BLOCKS_DIR"
exit 1
fi
echo "Payload files: $(find "$BIG_BLOCKS_DIR/payloads" -name '*.json' | wc -l)"
- name: Start metrics proxy
run: |
BENCH_ID="ci-${{ github.run_id }}"
BENCH_REFERENCE_EPOCH=$(date +%s)
echo "BENCH_ID=${BENCH_ID}" >> "$GITHUB_ENV"
echo "BENCH_REFERENCE_EPOCH=${BENCH_REFERENCE_EPOCH}" >> "$GITHUB_ENV"
LABELS_FILE="/tmp/bench-metrics-labels.json"
echo '{}' > "$LABELS_FILE"
echo "BENCH_LABELS_FILE=${LABELS_FILE}" >> "$GITHUB_ENV"
python3 .github/scripts/bench-metrics-proxy.py \
--labels "$LABELS_FILE" \
--upstream "http://${BENCH_METRICS_ADDR}/" \
--subnet 10.10.0.0/24 \
--port 9090 &
PROXY_PID=$!
echo "BENCH_METRICS_PROXY_PID=${PROXY_PID}" >> "$GITHUB_ENV"
echo "Metrics proxy started (PID $PROXY_PID)"
- name: Update status (running benchmarks)
if: success() && env.BENCH_COMMENT_ID
uses: actions/github-script@v8
@@ -894,64 +736,19 @@ jobs:
# thermal drift and cache warming.
- name: "Run benchmark: baseline (1/2)"
id: run-baseline-1
env:
BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-1,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
run: |
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
run: taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
- name: "Run benchmark: feature (1/2)"
id: run-feature-1
env:
FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-1,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
run: |
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
run: taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
- name: "Run benchmark: feature (2/2)"
if: env.BENCH_ABBA != 'false'
id: run-feature-2
env:
FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-2,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
run: |
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
run: taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
- name: "Run benchmark: baseline (2/2)"
if: env.BENCH_ABBA != 'false'
id: run-baseline-2
env:
BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-2,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
run: |
LAST_RUN_START=$(date +%s)
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
cat > "$BENCH_LABELS_FILE" <<LABELS
{"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
LABELS
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
- name: Stop metrics proxy & generate Grafana URL
id: metrics
if: "!cancelled()"
run: |
kill "$BENCH_METRICS_PROXY_PID" 2>/dev/null || true
LAST_RUN_DURATION=$(( $(date +%s) - BENCH_LAST_RUN_START ))
FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
GRAFANA_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=ef57fux92e9z4e&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
echo "grafana-url=${GRAFANA_URL}" >> "$GITHUB_OUTPUT"
echo "Grafana URL: ${GRAFANA_URL}"
run: taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
- name: Scan logs for errors
if: "!cancelled()"
@@ -1053,30 +850,12 @@ jobs:
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-name ${BASELINE_NAME}"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-name ${FEATURE_NAME}"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-ref ${FEATURE_REF}"
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
if [ "${BENCH_ABBA:-true}" = "true" ]; then
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
fi
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BASELINE_CSVS"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $FEATURE_CSVS"
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BENCH_WORK_DIR/baseline-1/combined_latency.csv $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $BENCH_WORK_DIR/feature-1/combined_latency.csv $BENCH_WORK_DIR/feature-2/combined_latency.csv"
SUMMARY_ARGS="$SUMMARY_ARGS --gas-csv $BENCH_WORK_DIR/feature-1/total_gas.csv"
if [ "$BEHIND_BASELINE" -gt 0 ]; then
SUMMARY_ARGS="$SUMMARY_ARGS --behind-baseline $BEHIND_BASELINE"
fi
if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
SUMMARY_ARGS="$SUMMARY_ARGS --big-blocks"
# Read gas ramp blocks count from first baseline run (same for all runs)
GAS_RAMP_FILE="$BENCH_WORK_DIR/baseline-1/gas_ramp_blocks.txt"
if [ -f "$GAS_RAMP_FILE" ]; then
SUMMARY_ARGS="$SUMMARY_ARGS --gas-ramp-blocks $(cat "$GAS_RAMP_FILE" | tr -d '[:space:]')"
fi
fi
GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
if [ -n "$GRAFANA_URL" ]; then
SUMMARY_ARGS="$SUMMARY_ARGS --grafana-url $GRAFANA_URL"
fi
# shellcheck disable=SC2086
python3 .github/scripts/bench-reth-summary.py $SUMMARY_ARGS
@@ -1087,14 +866,8 @@ jobs:
FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
run: |
CHART_ARGS="--output-dir $BENCH_WORK_DIR/charts"
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
if [ "${BENCH_ABBA:-true}" = "true" ]; then
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
fi
CHART_ARGS="$CHART_ARGS --feature $FEATURE_CSVS"
CHART_ARGS="$CHART_ARGS --baseline $BASELINE_CSVS"
CHART_ARGS="$CHART_ARGS --feature $BENCH_WORK_DIR/feature-1/combined_latency.csv $BENCH_WORK_DIR/feature-2/combined_latency.csv"
CHART_ARGS="$CHART_ARGS --baseline $BENCH_WORK_DIR/baseline-1/combined_latency.csv $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
CHART_ARGS="$CHART_ARGS --baseline-name ${BASELINE_NAME}"
CHART_ARGS="$CHART_ARGS --feature-name ${FEATURE_NAME}"
# shellcheck disable=SC2086
@@ -1102,7 +875,7 @@ jobs:
- name: Upload results
if: "!cancelled()"
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v6
with:
name: bench-reth-results
path: ${{ env.BENCH_WORK_DIR }}
@@ -1134,7 +907,7 @@ jobs:
rm -rf "${TMP_DIR}"
- name: Compare & comment
if: success() && env.BENCH_COMMENT_ID
if: success()
uses: actions/github-script@v8
with:
github-token: ${{ secrets.DEREK_PAT }}
@@ -1170,8 +943,7 @@ jobs:
// Samply profile links (URLs point directly to Firefox Profiler)
if (process.env.BENCH_SAMPLY === 'true') {
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
const runs = abba ? ['baseline-1', 'feature-1', 'feature-2', 'baseline-2'] : ['baseline-1', 'feature-1'];
const runs = ['baseline-1', 'feature-1', 'feature-2', 'baseline-2'];
const links = [];
for (const run of runs) {
try {
@@ -1186,12 +958,6 @@ jobs:
}
}
// Grafana dashboard link
const grafanaUrl = '${{ steps.metrics.outputs.grafana-url }}';
if (grafanaUrl) {
comment += `\n\n### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n`;
}
// Node errors (panics / ERROR logs)
try {
const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
@@ -1211,24 +977,13 @@ jobs:
comment_id: parseInt(ackCommentId),
body,
});
} else {
// No PR — write results to job summary
await core.summary.addRaw(body).write();
}
- name: Write job summary
if: success()
uses: actions/github-script@v8
with:
script: |
const jobSummary = require('./.github/scripts/bench-job-summary.js');
await jobSummary({
core,
context,
chartSha: '${{ steps.push-charts.outputs.sha }}',
grafanaUrl: '${{ steps.metrics.outputs.grafana-url }}',
runId: '${{ github.run_id }}',
});
- name: Send Slack notification (success)
if: success() && env.BENCH_NO_SLACK != 'true'
if: success()
uses: actions/github-script@v8
env:
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
@@ -1244,13 +999,12 @@ jobs:
with:
github-token: ${{ secrets.DEREK_PAT }}
script: |
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
const steps_status = [
['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
];
const failed = steps_status.find(([, o]) => o === 'failure');
const failedStep = failed ? failed[0] : 'unknown step';
@@ -1279,13 +1033,12 @@ jobs:
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
with:
script: |
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
const steps_status = [
['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
];
const failed = steps_status.find(([, o]) => o === 'failure');
const failedStep = failed ? failed[0] : 'unknown step';

View File

@@ -6,7 +6,7 @@ on:
hive_target:
required: true
type: string
description: "Docker bake target to build (e.g. hive)"
description: "Docker bake target to build (e.g. hive-stable, hive-edge)"
artifact_name:
required: false
type: string
@@ -76,7 +76,7 @@ jobs:
*.dockerfile=Dockerfile
- name: Upload reth image
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v6
with:
name: ${{ inputs.artifact_name }}
path: ./artifacts

View File

@@ -28,30 +28,12 @@ on:
required: false
type: boolean
default: false
pgo:
description: "Enable PGO profiling"
required: false
type: boolean
default: false
pgo_blocks:
description: "Number of blocks to execute for PGO profiling"
required: false
type: string
default: "20"
jobs:
collect-pgo-profile:
if: github.repository == 'paradigmxyz/reth' && github.event_name == 'workflow_dispatch' && inputs.pgo
uses: ./.github/workflows/pgo-profile.yml
with:
pgo_blocks: ${{ inputs.pgo_blocks || '20' }}
secrets: inherit
build:
if: github.repository == 'paradigmxyz/reth' && !failure() && !cancelled()
if: github.repository == 'paradigmxyz/reth'
name: Build Docker images
runs-on: ubuntu-24.04
needs: collect-pgo-profile
permissions:
packages: write
contents: read
@@ -63,7 +45,7 @@ jobs:
uses: depot/setup-action@v1
- name: Log in to GHCR
uses: docker/login-action@v4
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -76,30 +58,6 @@ jobs:
echo "describe=$(git describe --always --tags)" >> "$GITHUB_OUTPUT"
echo "dirty=false" >> "$GITHUB_OUTPUT"
- name: Download pre-collected PGO profile
if: ${{ github.event_name == 'workflow_dispatch' && inputs.pgo }}
uses: actions/download-artifact@v7
with:
name: pgo-profdata
path: dist
- name: Configure PGO build args
id: pgo
run: |
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]] && [[ "${{ inputs.pgo }}" == "true" ]]; then
if [ ! -f dist/merged.profdata ]; then
echo "::error::Expected dist/merged.profdata from collect-pgo-profile job"
exit 1
fi
echo "use_pgo_bolt=true" >> "$GITHUB_OUTPUT"
echo "pgo_profdata=dist/merged.profdata" >> "$GITHUB_OUTPUT"
echo "Using pre-collected PGO profile from collect-pgo-profile job"
else
echo "use_pgo_bolt=false" >> "$GITHUB_OUTPUT"
echo "pgo_profdata=" >> "$GITHUB_OUTPUT"
echo "PGO disabled"
fi
- name: Determine build parameters
id: params
run: |
@@ -149,9 +107,6 @@ jobs:
push: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }}
set: |
${{ steps.params.outputs.ethereum_set }}
*.args.USE_PGO_BOLT=${{ steps.pgo.outputs.use_pgo_bolt }}
*.args.PGO_PROFDATA=${{ steps.pgo.outputs.pgo_profdata }}
*.args.STRIP_SYMBOLS=false
- name: Verify image architectures
env:

View File

@@ -63,6 +63,6 @@ jobs:
run: |
cargo nextest run \
--no-fail-fast \
--locked \
--locked --features "edge" \
-p reth-e2e-test-utils \
-E 'binary(rocksdb)'

View File

@@ -15,11 +15,18 @@ concurrency:
cancel-in-progress: true
jobs:
build-reth:
build-reth-stable:
uses: ./.github/workflows/docker-test.yml
with:
hive_target: hive
artifact_name: "reth"
hive_target: hive-stable
artifact_name: "reth-stable"
secrets: inherit
build-reth-edge:
uses: ./.github/workflows/docker-test.yml
with:
hive_target: hive-edge
artifact_name: "reth-edge"
secrets: inherit
prepare-hive:
@@ -68,7 +75,7 @@ jobs:
chmod +x hive
- name: Upload hive assets
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v6
with:
name: hive_assets
path: ./hive_assets
@@ -77,6 +84,7 @@ jobs:
strategy:
fail-fast: false
matrix:
storage: [stable, edge]
# ethereum/rpc to be deprecated:
# https://github.com/ethereum/hive/pull/1117
scenario:
@@ -176,9 +184,10 @@ jobs:
- sim: ethereum/eels/consume-rlp
limit: .*tests/paris.*
needs:
- build-reth
- build-reth-stable
- build-reth-edge
- prepare-hive
name: ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
name: ${{ matrix.storage }} / ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
# Use larger runners for eels tests to avoid OOM runner crashes
runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
permissions:
@@ -189,15 +198,15 @@ jobs:
fetch-depth: 0
- name: Download hive assets
uses: actions/download-artifact@v8
uses: actions/download-artifact@v7
with:
name: hive_assets
path: /tmp
- name: Download reth image
uses: actions/download-artifact@v8
uses: actions/download-artifact@v7
with:
name: reth
name: reth-${{ matrix.storage }}
path: /tmp
- name: Load Docker images

View File

@@ -22,7 +22,7 @@ concurrency:
jobs:
test:
name: test / ${{ matrix.network }}
name: test / ${{ matrix.network }} / ${{ matrix.storage }}
if: github.event_name != 'schedule'
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
env:
@@ -30,6 +30,7 @@ jobs:
strategy:
matrix:
network: ["ethereum"]
storage: ["stable", "edge"]
timeout-minutes: 60
steps:
- uses: actions/checkout@v6
@@ -46,7 +47,7 @@ jobs:
run: |
cargo nextest run \
--no-fail-fast \
--locked --features "asm-keccak ${{ matrix.network }}" \
--locked --features "asm-keccak ${{ matrix.network }} ${{ matrix.storage == 'edge' && 'edge' || '' }}" \
--workspace --exclude ef-tests \
-E "kind(test) and not binary(e2e_testsuite)"

View File

@@ -40,7 +40,7 @@ jobs:
fetch-depth: 0
- name: Download reth image
uses: actions/download-artifact@v8
uses: actions/download-artifact@v7
with:
name: artifacts
path: /tmp

View File

@@ -1,107 +0,0 @@
name: pgo-profile
on:
workflow_call:
inputs:
pgo_blocks:
description: "Number of blocks to execute for PGO profiling"
required: false
type: string
default: "20"
workflow_dispatch:
inputs:
pgo_blocks:
description: "Number of blocks to execute for PGO profiling"
required: false
type: string
default: "20"
jobs:
collect:
name: collect PGO profiles
runs-on: [self-hosted, Linux, X64]
timeout-minutes: 180
env:
SCHELK_MOUNT: /reth-bench
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
RUSTC_WRAPPER: "sccache"
steps:
- uses: actions/checkout@v6
with:
submodules: true
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
with:
target: x86_64-unknown-linux-gnu
- uses: mozilla-actions/sccache-action@v0.0.9
continue-on-error: true
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Install dependencies
run: |
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends \
dmsetup lsb-release wget linux-tools-"$(uname -r)" || \
sudo apt-get install -y --no-install-recommends linux-tools-generic
- name: Download snapshot if needed
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BENCH_REPO: ${{ github.repository }}
run: |
if ! .github/scripts/bench-reth-snapshot.sh --check; then
echo "Snapshot outdated or missing, downloading..."
.github/scripts/bench-reth-snapshot.sh
fi
- name: Mount snapshot
run: |
sudo pkill -9 reth || true
sleep 1
if mountpoint -q "$SCHELK_MOUNT"; then
sudo umount -l "$SCHELK_MOUNT" || true
sudo schelk recover -y || true
fi
sudo schelk mount -y
sync
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
- name: Collect PGO profile
run: |
DATADIR="$SCHELK_MOUNT/datadir" \
RPC_URL="$BENCH_RPC_URL" \
PGO_BLOCKS="${{ inputs.pgo_blocks || '20' }}" \
BOLT_BLOCKS="${{ inputs.pgo_blocks || '20' }}" \
COLLECT_PGO_ONLY=true \
SKIP_BOLT=true \
PROFILE=maxperf-symbols \
FEATURES="jemalloc,asm-keccak,min-debug-logs" \
TARGET=x86_64-unknown-linux-gnu \
EXTRA_RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq" \
.github/scripts/build_pgo_bolt.sh
- name: Show PGO profile stats
run: |
LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
if [ -z "$LLVM_PROFDATA" ]; then
echo "::error::llvm-profdata not found in rust toolchain"
exit 1
fi
"$LLVM_PROFDATA" show --detailed-summary --topn=20 target/pgo-profiles/merged.profdata
- name: Upload PGO profile
uses: actions/upload-artifact@v7
with:
name: pgo-profdata
path: target/pgo-profiles/merged.profdata
retention-days: 1
- name: Recover snapshot
if: always()
run: |
if mountpoint -q "$SCHELK_MOUNT"; then
sudo umount -l "$SCHELK_MOUNT" || true
sudo schelk recover -y || true
fi

View File

@@ -1,30 +0,0 @@
name: Pull request audit
on:
pull_request:
types: [labeled]
jobs:
publish:
runs-on: ubuntu-latest
if: github.event.label.name == 'cyclops'
steps:
- name: Publish event
run: |
set -euo pipefail
echo "${{ secrets.EVENTS_KEY }}" > ${{ runner.temp }}/key
echo "${{ secrets.EVENTS_CERT }}" > ${{ runner.temp }}/cert
curl -sf -o /dev/null -X POST ${{ secrets.EVENTS_ARGS }} \
-H "Content-Type: application/json" \
--key ${{ runner.temp }}/key \
--cert ${{ runner.temp }}/cert \
-d '{
"repository": "${{ github.repository }}",
"event": "pr_audit",
"data": {
"pr_number": ${{ github.event.pull_request.number }},
"sha": "${{ github.event.pull_request.head.sha }}"
}
}'

View File

@@ -52,7 +52,7 @@ jobs:
uses: docker/setup-buildx-action@v3
- name: Log in to GitHub Container Registry
uses: docker/login-action@v4
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}

View File

@@ -13,14 +13,6 @@ on:
description: "Enable dry run mode (builds artifacts but skips uploads and release creation)"
type: boolean
default: false
pgo:
description: "Enable PGO profiling"
type: boolean
default: false
pgo_blocks:
description: "Number of blocks to execute for PGO profiling on self-hosted runner"
type: string
default: "20"
env:
REPO_NAME: ${{ github.repository_owner }}/reth
@@ -45,7 +37,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Extract version
run: echo "VERSION=${GITHUB_REF_NAME//\//-}" >> $GITHUB_OUTPUT
run: echo "VERSION=${GITHUB_REF_NAME}" >> $GITHUB_OUTPUT
id: extract_version
outputs:
VERSION: ${{ steps.extract_version.outputs.VERSION }}
@@ -77,22 +69,22 @@ jobs:
fail-fast: true
matrix:
configs:
- target: x86_64-unknown-linux-gnu
os: ubuntu-24.04
profile: maxperf
allow_fail: false
- target: aarch64-unknown-linux-gnu
os: ubuntu-24.04-arm
profile: maxperf
allow_fail: false
rustflags: ""
native: true
- target: x86_64-apple-darwin
os: macos-14
profile: maxperf
allow_fail: false
rustflags: "-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"
- target: aarch64-apple-darwin
os: macos-14
profile: maxperf
allow_fail: false
rustflags: ""
build:
- command: build
binary: reth
@@ -104,7 +96,6 @@ jobs:
target: ${{ matrix.configs.target }}
- uses: mozilla-actions/sccache-action@v0.0.9
- name: Install cross main
if: ${{ !matrix.configs.native }}
id: cross_main
run: |
cargo install cross --locked --git https://github.com/cross-rs/cross
@@ -119,12 +110,7 @@ jobs:
echo "MACOSX_DEPLOYMENT_TARGET=$(xcrun -sdk macosx --show-sdk-platform-version)" >> $GITHUB_ENV
- name: Build Reth
run: |
if [ "${{ matrix.configs.native }}" = "true" ]; then
make PROFILE=${{ matrix.configs.profile }} EXTRA_RUSTFLAGS="${{ matrix.configs.rustflags }}" ${{ matrix.build.command }}-native-${{ matrix.configs.target }}
else
make PROFILE=${{ matrix.configs.profile }} EXTRA_RUSTFLAGS="${{ matrix.configs.rustflags }}" ${{ matrix.build.command }}-${{ matrix.configs.target }}
fi
run: make PROFILE=${{ matrix.configs.profile }} ${{ matrix.build.command }}-${{ matrix.configs.target }}
- name: Move binary
run: |
mkdir artifacts
@@ -145,105 +131,23 @@ jobs:
- name: Upload artifact
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v6
with:
name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
- name: Upload signature
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v6
with:
name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
collect-pgo-profile:
if: github.event_name == 'workflow_dispatch' && inputs.pgo
uses: ./.github/workflows/pgo-profile.yml
with:
pgo_blocks: ${{ inputs.pgo_blocks || '20' }}
secrets: inherit
build-pgo:
if: github.event_name == 'workflow_dispatch' && inputs.pgo
name: build release (x86_64-linux PGO+BOLT)
runs-on: [self-hosted, Linux, X64]
needs: [extract-version, collect-pgo-profile]
timeout-minutes: 120
env:
RUSTC_WRAPPER: "sccache"
steps:
- uses: actions/checkout@v6
with:
submodules: true
- uses: rui314/setup-mold@v1
- uses: dtolnay/rust-toolchain@stable
with:
target: x86_64-unknown-linux-gnu
- uses: mozilla-actions/sccache-action@v0.0.9
continue-on-error: true
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
- name: Download pre-collected PGO profile
uses: actions/download-artifact@v7
with:
name: pgo-profdata
path: dist
- name: Verify PGO profile artifact
run: |
test -f dist/merged.profdata
ls -lh dist/merged.profdata
- name: Build Reth with PGO+BOLT
run: |
SKIP_BOLT=true \
PGO_PROFDATA="$PWD/dist/merged.profdata" \
PROFILE=maxperf-symbols \
FEATURES="jemalloc,asm-keccak,min-debug-logs" \
TARGET=x86_64-unknown-linux-gnu \
EXTRA_RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq" \
.github/scripts/build_pgo_bolt.sh
- name: Move binary
run: |
mkdir artifacts
mv target/maxperf-symbols/reth ./artifacts
- name: Configure GPG and create artifacts
env:
GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }}
GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
run: |
export GPG_TTY=$(tty)
echo -n "$GPG_SIGNING_KEY" | base64 --decode | gpg --batch --import
cd artifacts
tar -czf reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz reth*
echo "$GPG_PASSPHRASE" | gpg --passphrase-fd 0 --pinentry-mode loopback --batch -ab reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
mv *tar.gz* ..
shell: bash
- name: Upload artifact
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: actions/upload-artifact@v6
with:
name: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
path: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
- name: Upload signature
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: actions/upload-artifact@v6
with:
name: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz.asc
path: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz.asc
draft-release:
name: draft release
runs-on: ubuntu-latest
needs: [build, build-pgo, extract-version]
if: ${{ !failure() && !cancelled() && github.event.inputs.dry_run != 'true' }}
needs: [build, extract-version]
if: ${{ github.event.inputs.dry_run != 'true' }}
env:
VERSION: ${{ needs.extract-version.outputs.VERSION }}
permissions:
@@ -256,7 +160,7 @@ jobs:
with:
fetch-depth: 0
- name: Download artifacts
uses: actions/download-artifact@v8
uses: actions/download-artifact@v7
- name: Generate full changelog
id: changelog
run: |

View File

@@ -43,7 +43,7 @@ jobs:
echo "Binaries SHA256 on ${{ matrix.machine }}: $(cat checksum.sha256)"
- name: Upload the hash
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v6
with:
name: checksum-${{ matrix.machine }}
path: |
@@ -56,12 +56,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Download artifacts from machine-1
uses: actions/download-artifact@v8
uses: actions/download-artifact@v7
with:
name: checksum-machine-1
path: machine-1/
- name: Download artifacts from machine-2
uses: actions/download-artifact@v8
uses: actions/download-artifact@v7
with:
name: checksum-machine-2
path: machine-2/

View File

@@ -51,12 +51,15 @@ jobs:
- name: Run execution stage
run: |
reth stage run execution --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
# NOTE: account-hashing, storage-hashing, and hashing stages are omitted.
# With storage v2 (now default), these stages are no-ops because the
# execution stage writes directly to HashedAccounts/HashedStorages.
# Running them here is harmful: `stage run` unwinds before executing,
# and the unwind reverts the hashed state that execution wrote, but
# the no-op execute never restores it — causing merkle to fail.
- name: Run account-hashing stage
run: |
reth stage run account-hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
- name: Run storage hashing stage
run: |
reth stage run storage-hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
- name: Run hashing stage
run: |
reth stage run hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
- name: Run merkle stage
run: |
reth stage run merkle --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints

View File

@@ -19,13 +19,15 @@ concurrency:
jobs:
test:
name: test / ${{ matrix.type }}
name: test / ${{ matrix.type }} / ${{ matrix.storage }}
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
env:
RUST_BACKTRACE: 1
EDGE_FEATURES: ${{ matrix.storage == 'edge' && 'edge' || '' }}
strategy:
matrix:
type: [ethereum]
storage: [stable, edge]
include:
- type: ethereum
features: asm-keccak ethereum
@@ -48,14 +50,14 @@ jobs:
run: |
cargo nextest run \
--no-fail-fast \
--features "${{ matrix.features }}" --locked \
--features "${{ matrix.features }} $EDGE_FEATURES" --locked \
${{ matrix.exclude_args }} --workspace \
--exclude ef-tests --no-tests=warn \
-E "!kind(test) and not binary(e2e_testsuite)"
state:
name: Ethereum state tests
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
env:
RUST_LOG: info,sync=error
RUST_BACKTRACE: 1

972
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,5 @@
[workspace.package]
version = "1.11.3"
version = "1.11.1"
edition = "2024"
rust-version = "1.93"
license = "MIT OR Apache-2.0"
@@ -170,7 +170,6 @@ rust.rust_2018_idioms = { level = "deny", priority = -1 }
rust.unreachable_pub = "warn"
rust.unused_must_use = "deny"
rust.rust_2024_incompatible_pat = "warn"
rust.unexpected_cfgs = { level = "warn", check-cfg = ['cfg(tokio_unstable)'] }
rustdoc.all = "warn"
# rust.unnameable-types = "warn"
@@ -399,9 +398,7 @@ reth-payload-builder-primitives = { path = "crates/payload/builder-primitives" }
reth-payload-primitives = { path = "crates/payload/primitives" }
reth-payload-validator = { path = "crates/payload/validator" }
reth-payload-util = { path = "crates/payload/util" }
reth-primitives = { path = "crates/primitives", default-features = false, features = [
"__internal",
] }
reth-primitives = { path = "crates/primitives", default-features = false, features = ["__internal"] }
reth-primitives-traits = { path = "crates/primitives-traits", default-features = false }
reth-provider = { path = "crates/storage/provider" }
reth-prune = { path = "crates/prune/prune" }
@@ -439,29 +436,26 @@ reth-trie-sparse = { path = "crates/trie/sparse", default-features = false }
reth-zstd-compressors = { path = "crates/storage/zstd-compressors", default-features = false }
# revm
revm = { version = "36.0.0", default-features = false }
revm-bytecode = { version = "9.0.0", default-features = false }
revm-database = { version = "12.0.0", default-features = false }
revm-state = { version = "10.0.0", default-features = false }
revm-primitives = { version = "22.1.0", default-features = false }
revm-interpreter = { version = "34.0.0", default-features = false }
revm-database-interface = { version = "10.0.0", default-features = false }
revm-inspectors = "0.36.1"
revm = { version = "34.0.0", default-features = false }
revm-bytecode = { version = "8.0.0", default-features = false }
revm-database = { version = "10.0.0", default-features = false }
revm-state = { version = "9.0.0", default-features = false }
revm-primitives = { version = "22.0.0", default-features = false }
revm-interpreter = { version = "32.0.0", default-features = false }
revm-database-interface = { version = "9.0.0", default-features = false }
op-revm = { version = "15.0.0", default-features = false }
revm-inspectors = "0.34.2"
# eth
alloy-dyn-abi = "1.5.6"
alloy-primitives = { version = "1.5.6", default-features = false, features = [
"map-foldhash",
] }
alloy-primitives = { version = "1.5.6", default-features = false, features = ["map-foldhash"] }
alloy-sol-types = { version = "1.5.6", default-features = false }
alloy-chains = { version = "0.2.5", default-features = false }
alloy-eip2124 = { version = "0.2.0", default-features = false }
alloy-eip7928 = { version = "0.3.0", default-features = false }
alloy-evm = { version = "0.29.2", default-features = false }
alloy-rlp = { version = "0.3.13", default-features = false, features = [
"core-net",
] }
alloy-evm = { version = "0.27.2", default-features = false }
alloy-rlp = { version = "0.3.13", default-features = false, features = ["core-net"] }
alloy-trie = { version = "0.9.4", default-features = false }
alloy-hardforks = "0.4.5"
@@ -473,15 +467,10 @@ alloy-genesis = { version = "1.7.3", default-features = false }
alloy-json-rpc = { version = "1.7.3", default-features = false }
alloy-network = { version = "1.7.3", default-features = false }
alloy-network-primitives = { version = "1.7.3", default-features = false }
alloy-provider = { version = "1.7.3", features = [
"reqwest",
"debug-api",
], default-features = false }
alloy-provider = { version = "1.7.3", features = ["reqwest", "debug-api"], default-features = false }
alloy-pubsub = { version = "1.7.3", default-features = false }
alloy-rpc-client = { version = "1.7.3", default-features = false }
alloy-rpc-types = { version = "1.7.3", features = [
"eth",
], default-features = false }
alloy-rpc-types = { version = "1.7.3", features = ["eth"], default-features = false }
alloy-rpc-types-admin = { version = "1.7.3", default-features = false }
alloy-rpc-types-anvil = { version = "1.7.3", default-features = false }
alloy-rpc-types-beacon = { version = "1.7.3", default-features = false }
@@ -495,31 +484,30 @@ alloy-serde = { version = "1.7.3", default-features = false }
alloy-signer = { version = "1.7.3", default-features = false }
alloy-signer-local = { version = "1.7.3", default-features = false }
alloy-transport = { version = "1.7.3" }
alloy-transport-http = { version = "1.7.3", features = [
"reqwest-rustls-tls",
], default-features = false }
alloy-transport-http = { version = "1.7.3", features = ["reqwest-rustls-tls"], default-features = false }
alloy-transport-ipc = { version = "1.7.3", default-features = false }
alloy-transport-ws = { version = "1.7.3", default-features = false }
# op
op-alloy-rpc-types = { version = "0.24.0", default-features = false }
op-alloy-rpc-types-engine = { version = "0.24.0", default-features = false }
op-alloy-consensus = { version = "0.24.0", default-features = false }
alloy-op-evm = { version = "0.27.2", default-features = false }
alloy-op-hardforks = "0.4.4"
op-alloy-rpc-types = { version = "0.23.1", default-features = false }
op-alloy-rpc-types-engine = { version = "0.23.1", default-features = false }
op-alloy-network = { version = "0.23.1", default-features = false }
op-alloy-consensus = { version = "0.23.1", default-features = false }
op-alloy-rpc-jsonrpsee = { version = "0.23.1", default-features = false }
op-alloy-flz = { version = "0.13.1", default-features = false }
# misc
either = { version = "1.15.0", default-features = false }
arrayvec = { version = "0.7.6", default-features = false }
aquamarine = "0.6"
auto_impl = "1"
backon = { version = "1.2", default-features = false, features = [
"std-blocking-sleep",
"tokio-sleep",
] }
backon = { version = "1.2", default-features = false, features = ["std-blocking-sleep", "tokio-sleep"] }
bincode = "1.3"
bitflags = "2.4"
boyer-moore-magiclen = "0.2.16"
bytes = { version = "1.11.1", default-features = false }
blake3 = "1.8"
brotli = "8"
cfg-if = "1.0"
clap = "4"
@@ -535,16 +523,11 @@ humantime = "2.1"
humantime-serde = "1.1"
itertools = { version = "0.14", default-features = false }
linked_hash_set = "0.1"
libc = "0.2"
lz4 = "1.28.1"
modular-bitfield = "0.13.1"
notify = { version = "8.0.0", default-features = false, features = [
"macos_fsevent",
] }
notify = { version = "8.0.0", default-features = false, features = ["macos_fsevent"] }
nybbles = { version = "0.4.8", default-features = false }
once_cell = { version = "1.19", default-features = false, features = [
"critical-section",
] }
once_cell = { version = "1.19", default-features = false, features = ["critical-section"] }
parking_lot = "0.12"
quanta = "0.12"
paste = "1.0"
@@ -558,16 +541,13 @@ serde_json = { version = "1.0", default-features = false, features = ["alloc"] }
serde_with = { version = "3", default-features = false, features = ["macros"] }
sha2 = { version = "0.10", default-features = false }
shlex = "1.3"
slotmap = "1"
smallvec = "1"
strum = { version = "0.27", default-features = false }
strum_macros = "0.27"
syn = "2.0"
thiserror = { version = "2.0.0", default-features = false }
tar = "0.4.44"
tracing = { version = "0.1.0", default-features = false, features = [
"attributes",
] }
tracing = { version = "0.1.0", default-features = false, features = ["attributes"] }
tracing-appender = "0.2"
url = { version = "2.3", default-features = false }
zstd = "0.13"
@@ -605,11 +585,7 @@ futures-util = { version = "0.3", default-features = false }
hyper = "1.3"
hyper-util = "0.1.5"
pin-project = "1.0.12"
reqwest = { version = "0.12", default-features = false, features = [
"rustls-tls",
"rustls-tls-native-roots",
"stream",
] }
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "rustls-tls-native-roots", "stream"] }
tracing-futures = "0.2"
tower = "0.5"
tower-http = "0.6"
@@ -634,10 +610,7 @@ proptest-arbitrary-interop = "0.1.0"
# crypto
enr = { version = "0.13", default-features = false }
k256 = { version = "0.13", default-features = false, features = ["ecdsa"] }
secp256k1 = { version = "0.30", default-features = false, features = [
"global-context",
"recovery",
] }
secp256k1 = { version = "0.30", default-features = false, features = ["global-context", "recovery"] }
# rand 8 for secp256k1
rand_08 = { package = "rand", version = "0.8" }
@@ -692,7 +665,6 @@ cipher = "0.4.3"
comfy-table = "7.0"
concat-kdf = "0.1.0"
crossbeam-channel = "0.5.13"
crossbeam-queue = "0.3"
crossbeam-utils = "0.8"
crossterm = "0.29.0"
csv = "1.3.0"
@@ -769,8 +741,10 @@ ipnet = "2.11"
# alloy-transport-ws = { git = "https://github.com/alloy-rs/alloy", rev = "3049f232fbb44d1909883e154eb38ec5962f53a3" }
# op-alloy-consensus = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
# op-alloy-network = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
# op-alloy-rpc-types = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
# op-alloy-rpc-types-engine = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
# op-alloy-rpc-jsonrpsee = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
#
# revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "1207e33" }
#
@@ -780,15 +754,10 @@ ipnet = "2.11"
# jsonrpsee-http-client = { git = "https://github.com/paradigmxyz/jsonrpsee", branch = "matt/make-rpc-service-pub" }
# jsonrpsee-types = { git = "https://github.com/paradigmxyz/jsonrpsee", branch = "matt/make-rpc-service-pub" }
alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "b0eb7e6" }
# alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "df124c0" }
# alloy-op-evm = { git = "https://github.com/alloy-rs/evm", rev = "df124c0" }
revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "24becc3" }
# revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "3020ea8" }
# revm from rakita/state-gas branch
revm = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
revm-bytecode = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
revm-database = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
revm-state = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
revm-primitives = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
revm-interpreter = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
revm-database-interface = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
# alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "072c248" }
# alloy-op-evm = { git = "https://github.com/alloy-rs/evm", rev = "072c248" }

View File

@@ -1,10 +1,8 @@
# syntax=docker/dockerfile:1
# Dockerfile for reth, optimized for Depot builds
# Supports PGO+BOLT optimization for maximum performance
# Usage:
# reth: --build-arg BINARY=reth
# PGO+BOLT: --build-arg USE_PGO_BOLT=true (Linux x86_64/aarch64 only)
FROM rust:1.93 AS builder
WORKDIR /app
@@ -45,18 +43,6 @@ ENV VERGEN_GIT_SHA=$VERGEN_GIT_SHA
ENV VERGEN_GIT_DESCRIBE=$VERGEN_GIT_DESCRIBE
ENV VERGEN_GIT_DIRTY=$VERGEN_GIT_DIRTY
# Enable PGO+BOLT optimization (Linux only)
ARG USE_PGO_BOLT=false
ENV USE_PGO_BOLT=$USE_PGO_BOLT
# Optional path to a pre-collected merged.profdata file in build context.
ARG PGO_PROFDATA=""
ENV PGO_PROFDATA=$PGO_PROFDATA
# Whether to strip debug symbols from PGO-built binaries.
ARG STRIP_SYMBOLS=true
ENV STRIP_SYMBOLS=$STRIP_SYMBOLS
# Build application
# Platform-specific RUSTFLAGS: amd64 uses x86-64-v3 (Haswell+) with pclmulqdq for rocksdb
ARG TARGETPLATFORM
@@ -67,21 +53,12 @@ RUN --mount=type=secret,id=DEPOT_TOKEN,env=SCCACHE_WEBDAV_TOKEN \
--mount=type=cache,target=$SCCACHE_DIR,sharing=shared \
export RUSTC_WRAPPER=sccache SCCACHE_WEBDAV_ENDPOINT=https://cache.depot.dev SCCACHE_DIR=/sccache && \
sccache --start-server && \
if [ "$USE_PGO_BOLT" = "true" ] && [ "$TARGETPLATFORM" = "linux/amd64" ] && [ -n "$PGO_PROFDATA" ] && [ -f "$PGO_PROFDATA" ]; then \
apt-get update && apt-get install -y -qq lsb-release wget sudo && \
BINARY="$BINARY" PROFILE="$BUILD_PROFILE" FEATURES="$FEATURES" SKIP_BOLT=true STRIP_SYMBOLS="$STRIP_SYMBOLS" PGO_PROFDATA="$PGO_PROFDATA" \
./.github/scripts/build_pgo_bolt.sh; \
else \
if [ "$USE_PGO_BOLT" = "true" ]; then \
echo "PGO requested but pre-collected profile missing at '${PGO_PROFDATA:-<unset>}' - falling back to non-PGO build"; \
fi; \
if [ -n "$RUSTFLAGS" ]; then \
export RUSTFLAGS="$RUSTFLAGS"; \
elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
export RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"; \
fi && \
cargo build --profile $BUILD_PROFILE --features "$FEATURES" --locked --bin $BINARY --manifest-path $MANIFEST_PATH/Cargo.toml; \
if [ -n "$RUSTFLAGS" ]; then \
export RUSTFLAGS="$RUSTFLAGS"; \
elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
export RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"; \
fi && \
cargo build --profile $BUILD_PROFILE --features "$FEATURES" --locked --bin $BINARY --manifest-path $MANIFEST_PATH/Cargo.toml && \
sccache --show-stats
# Copy binary to a known location (ARG not resolved in COPY)

View File

@@ -17,9 +17,6 @@ FEATURES ?=
# Cargo profile for builds. Default is for local builds, CI uses an override.
PROFILE ?= release
# Extra RUSTFLAGS to append to build targets (e.g., "-C target-cpu=x86-64-v3")
EXTRA_RUSTFLAGS ?=
# Extra flags for Cargo
CARGO_INSTALL_EXTRA_FLAGS ?=
@@ -77,7 +74,7 @@ build-debug: ## Build the reth binary into `target/debug` directory.
cargo build --bin reth --features "$(FEATURES)"
# Builds the reth binary natively.
build-native-%:
$(if $(EXTRA_RUSTFLAGS),RUSTFLAGS="$(EXTRA_RUSTFLAGS)") cargo build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
cargo build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
# The following commands use `cross` to build a cross-compile.
#
@@ -95,12 +92,11 @@ build-native-%:
# on other systems. JEMALLOC_SYS_WITH_LG_PAGE=16 tells jemalloc to use 64-KiB
# pages. See: https://github.com/paradigmxyz/reth/issues/6742
build-aarch64-unknown-linux-gnu: export JEMALLOC_SYS_WITH_LG_PAGE=16
build-native-aarch64-unknown-linux-gnu: export JEMALLOC_SYS_WITH_LG_PAGE=16
# Note: The additional rustc compiler flags are for intrinsics needed by MDBX.
# See: https://github.com/cross-rs/cross/wiki/FAQ#undefined-reference-with-build-std
build-%:
RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc $(EXTRA_RUSTFLAGS)" \
RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc" \
cross build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
# Unfortunately we can't easily use cross to build for Darwin because of licensing issues.

View File

@@ -7,7 +7,7 @@ use alloy_primitives::address;
use alloy_provider::{network::AnyNetwork, Provider, RootProvider};
use alloy_rpc_client::ClientBuilder;
use alloy_rpc_types_engine::JwtSecret;
use alloy_transport::layers::{RateLimitRetryPolicy, RetryBackoffLayer};
use alloy_transport::layers::RetryBackoffLayer;
use reqwest::Url;
use reth_node_core::args::BenchmarkArgs;
use tracing::info;
@@ -53,15 +53,9 @@ impl BenchContext {
}
}
// set up alloy client for blocks, retrying on 429/503 (default) and 502
let retry_policy =
RateLimitRetryPolicy::default().or(|err: &alloy_transport::TransportError| -> bool {
err.as_transport_err()
.and_then(|t| t.as_http_error())
.is_some_and(|e| e.status == 502)
});
// set up alloy client for blocks
let client = ClientBuilder::default()
.layer(RetryBackoffLayer::new_with_policy(10, 800, u64::MAX, retry_policy))
.layer(RetryBackoffLayer::new(10, 800, u64::MAX))
.http(rpc_url.parse()?);
let block_provider = RootProvider::<AnyNetwork>::new(client);
@@ -94,11 +88,9 @@ impl BenchContext {
// Computes the block range for the benchmark.
//
// - If `--advance` is provided, fetches the latest block from the engine and sets:
// - If `--advance` is provided, fetches the latest block and sets:
// - `from = head + 1`
// - `to = head + advance`
// - If only `--to` is provided, fetches the latest block from the engine and sets:
// - `from = head`
// - Otherwise, uses the values from `--from` and `--to`.
let (from, to) = if let Some(advance) = bench_args.advance {
if advance == 0 {
@@ -111,14 +103,6 @@ impl BenchContext {
.ok_or_else(|| eyre::eyre!("Failed to fetch latest block for --advance"))?;
let head_number = head_block.header.number;
(Some(head_number), Some(head_number + advance))
} else if bench_args.from.is_none() && bench_args.to.is_some() {
let head_block = auth_provider
.get_block_by_number(BlockNumberOrTag::Latest)
.await?
.ok_or_else(|| eyre::eyre!("Failed to fetch latest block from engine"))?;
let head_number = head_block.header.number;
info!(target: "reth-bench", "No --from provided, derived from engine head: {}", head_number);
(Some(head_number), bench_args.to)
} else {
(bench_args.from, bench_args.to)
};
@@ -130,7 +114,7 @@ impl BenchContext {
.full()
.await?
.ok_or_else(|| eyre::eyre!("Failed to fetch latest block from RPC"))?;
let mut benchmark_mode = BenchMode::new(from, to, latest_block.into_inner().number());
let mut benchmark_mode = BenchMode::new(from, to, latest_block.into_inner().number())?;
let first_block = match benchmark_mode {
BenchMode::Continuous(start) => {

View File

@@ -24,7 +24,7 @@ pub(crate) struct GasRampPayloadFile {
/// Engine API version (1-5).
///
/// `None` indicates that `reth_newPayload` should be used.
#[serde(default, skip_serializing_if = "Option::is_none")]
#[serde(skip_serializing_if = "Option::is_none")]
pub(crate) version: Option<u8>,
/// The block hash for FCU.
pub(crate) block_hash: B256,
@@ -114,27 +114,16 @@ impl CombinedResult {
impl std::fmt::Display for CombinedResult {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let np = &self.new_payload_result;
write!(
f,
"Block {} processed at {:.4} Ggas/s, used {} total gas. Combined: {:.4} Ggas/s. fcu: {:?}, newPayload: {:?}",
self.block_number,
np.gas_per_second() / GIGAGAS as f64,
np.gas_used,
self.new_payload_result.gas_per_second() / GIGAGAS as f64,
self.new_payload_result.gas_used,
self.combined_gas_per_second() / GIGAGAS as f64,
self.fcu_latency,
np.latency,
)?;
if !np.execution_cache_wait.is_zero() {
write!(f, ", execution cache wait: {:?}", np.execution_cache_wait)?;
}
if !np.sparse_trie_wait.is_zero() {
write!(f, ", trie cache wait: {:?}", np.sparse_trie_wait)?;
}
if let Some(d) = np.persistence_wait {
write!(f, ", persistence wait: {d:?}")?;
}
Ok(())
self.new_payload_result.latency
)
}
}

View File

@@ -292,6 +292,10 @@ impl Command {
info!(target: "reth-bench", gas_ramp_payload = i + 1, "Gas ramp payload executed successfully");
if let Some(w) = &mut waiter {
w.on_block(payload.block_number).await?;
}
parent_hash = payload.file.block_hash;
}

View File

@@ -34,16 +34,17 @@ impl BenchMode {
}
/// Create a [`BenchMode`] from optional `from` and `to` fields.
///
/// If only `--to` is provided, `from` is derived as `latest_block + 1`.
pub const fn new(from: Option<u64>, to: Option<u64>, latest_block: u64) -> Self {
pub fn new(from: Option<u64>, to: Option<u64>, latest_block: u64) -> Result<Self, eyre::Error> {
// If neither `--from` nor `--to` are provided, we will run the benchmark continuously,
// starting at the latest block.
match (from, to) {
(Some(from), Some(to)) => Self::Range(from..=to),
(None, None) => Self::Continuous(latest_block),
(Some(start), None) => Self::Continuous(start),
(None, Some(to)) => Self::Range(latest_block + 1..=to),
(Some(from), Some(to)) => Ok(Self::Range(from..=to)),
(None, None) => Ok(Self::Continuous(latest_block)),
(Some(start), None) => Ok(Self::Continuous(start)),
_ => {
// both or neither are allowed, everything else is ambiguous
Err(eyre::eyre!("`from` and `to` must be provided together, or not at all."))
}
}
}
}

View File

@@ -235,8 +235,8 @@ pub(crate) fn payload_to_new_payload(
))?,
)
} else {
// Preserve the original RequestsOrHash payload for engine_newPayloadV4.
let requests = prague.requests.clone();
// Extract actual Requests from RequestsOrHash
let requests = prague.requests.requests_hash();
(
version,
serde_json::to_value((

View File

@@ -89,6 +89,7 @@ default = [
"keccak-cache-global",
"asm-keccak",
"min-debug-logs",
"rocksdb",
]
otlp = [
@@ -124,7 +125,6 @@ jemalloc = [
"reth-node-core/jemalloc",
"reth-node-metrics/jemalloc",
"reth-ethereum-cli/jemalloc",
"reth-provider/jemalloc",
]
jemalloc-prof = [
"reth-cli-util/jemalloc",
@@ -191,6 +191,8 @@ min-trace-logs = [
]
trie-debug = ["reth-node-builder/trie-debug", "reth-node-core/trie-debug"]
rocksdb = ["reth-ethereum-cli/rocksdb", "reth-node-core/rocksdb"]
edge = ["rocksdb"]
[[bin]]
name = "reth"

View File

@@ -56,6 +56,7 @@ alloy-signer.workspace = true
alloy-signer-local.workspace = true
rand.workspace = true
revm-state.workspace = true
criterion.workspace = true
[features]
serde = [
@@ -85,3 +86,8 @@ test-utils = [
"reth-ethereum-primitives/test-utils",
]
rayon = ["dep:rayon"]
[[bench]]
name = "canonical_hashes_range"
harness = false
required-features = ["test-utils"]

View File

@@ -0,0 +1,96 @@
#![allow(missing_docs)]
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use reth_chain_state::{
test_utils::TestBlockBuilder, ExecutedBlock, MemoryOverlayStateProviderRef,
};
use reth_ethereum_primitives::EthPrimitives;
use reth_storage_api::{noop::NoopProvider, BlockHashReader};
criterion_group!(benches, bench_canonical_hashes_range);
criterion_main!(benches);
fn bench_canonical_hashes_range(c: &mut Criterion) {
let mut group = c.benchmark_group("canonical_hashes_range");
let scenarios = [("small", 10), ("medium", 100), ("large", 1000)];
for (name, num_blocks) in scenarios {
group.bench_function(format!("{}_blocks_{}", name, num_blocks), |b| {
let (provider, blocks) = setup_provider_with_blocks(num_blocks);
let start_block = blocks[0].recovered_block().number;
let end_block = blocks[num_blocks / 2].recovered_block().number;
b.iter(|| {
black_box(
provider
.canonical_hashes_range(black_box(start_block), black_box(end_block))
.unwrap(),
)
})
});
}
let (provider, blocks) = setup_provider_with_blocks(500);
let base_block = blocks[100].recovered_block().number;
let range_sizes = [1, 10, 50, 100, 250];
for range_size in range_sizes {
group.bench_function(format!("range_size_{}", range_size), |b| {
let end_block = base_block + range_size;
b.iter(|| {
black_box(
provider
.canonical_hashes_range(black_box(base_block), black_box(end_block))
.unwrap(),
)
})
});
}
// Benchmark edge cases
group.bench_function("no_in_memory_matches", |b| {
let (provider, blocks) = setup_provider_with_blocks(100);
let first_block = blocks[0].recovered_block().number;
let start_block = first_block - 50;
let end_block = first_block - 10;
b.iter(|| {
black_box(
provider
.canonical_hashes_range(black_box(start_block), black_box(end_block))
.unwrap(),
)
})
});
group.bench_function("all_in_memory_matches", |b| {
let (provider, blocks) = setup_provider_with_blocks(100);
let first_block = blocks[0].recovered_block().number;
let last_block = blocks[blocks.len() - 1].recovered_block().number;
b.iter(|| {
black_box(
provider
.canonical_hashes_range(black_box(first_block), black_box(last_block + 1))
.unwrap(),
)
})
});
group.finish();
}
fn setup_provider_with_blocks(
num_blocks: usize,
) -> (MemoryOverlayStateProviderRef<'static, EthPrimitives>, Vec<ExecutedBlock<EthPrimitives>>) {
let mut builder = TestBlockBuilder::<EthPrimitives>::default();
let blocks: Vec<_> = builder.get_executed_blocks(1000..1000 + num_blocks as u64).collect();
let historical = Box::new(NoopProvider::default());
let provider = MemoryOverlayStateProviderRef::new(historical, blocks.clone());
(provider, blocks)
}

View File

@@ -9,7 +9,7 @@ use std::{
fmt,
sync::{Arc, LazyLock},
};
use tracing::{debug_span, instrument};
use tracing::instrument;
/// Shared handle to asynchronously populated trie data.
///
@@ -163,8 +163,6 @@ impl DeferredTrieData {
anchor_hash: B256,
ancestors: &[Self],
) -> ComputedTrieData {
let _span = debug_span!(target: "engine::tree::deferred_trie", "sort_inputs").entered();
#[cfg(feature = "rayon")]
let (sorted_hashed_state, sorted_trie_updates) = rayon::join(
|| match Arc::try_unwrap(hashed_state) {
@@ -189,10 +187,6 @@ impl DeferredTrieData {
},
);
drop(_span);
let _span = debug_span!(target: "engine::tree::deferred_trie", "build_overlay").entered();
// Reuse parent's overlay if available and anchors match.
// We can only reuse the parent's overlay if it was built on top of the same
// persisted anchor. If the anchor has changed (e.g., due to persistence),
@@ -211,9 +205,6 @@ impl DeferredTrieData {
Arc::clone(&trie_input.state),
Default::default(), // prefix_sets are per-block, not cumulative
);
let _span =
debug_span!(target: "engine::tree::deferred_trie", "extend_overlay")
.entered();
// Only trigger COW clone if there's actually data to add.
#[cfg(feature = "rayon")]
{
@@ -281,7 +272,6 @@ impl DeferredTrieData {
sorted_hashed_state: &HashedPostStateSorted,
sorted_trie_updates: &TrieUpdatesSorted,
) -> TrieInputSorted {
let _span = debug_span!(target: "engine::tree::deferred_trie", "merge_ancestors", num_ancestors = ancestors.len()).entered();
let mut overlay = TrieInputSorted::default();
let state_mut = Arc::make_mut(&mut overlay.state);

View File

@@ -1,69 +0,0 @@
//! Execution timing statistics for detailed block logging.
//!
//! This module provides types for collecting and passing execution timing statistics
//! through the block processing pipeline, enabling unified detailed block logging after
//! database commit.
use std::time::Duration;
use alloy_primitives::B256;
/// Statistics collected during block execution for cross-client performance analysis.
///
/// These statistics are populated during block validation and carried through to
/// persistence, where they are used to emit a single unified log entry that includes
/// complete timing information (including commit time).
#[derive(Debug, Clone, Default)]
pub struct ExecutionTimingStats {
/// Block number
pub block_number: u64,
/// Block hash
pub block_hash: B256,
/// Total gas used by the block
pub gas_used: u64,
/// Number of transactions in the block
pub tx_count: usize,
/// Time spent executing transactions (includes state reads)
pub execution_duration: Duration,
/// Time spent fetching state during execution (subset of `execution_duration`, includes cache
/// hits)
pub state_read_duration: Duration,
/// Time spent computing state root hash
pub state_hash_duration: Duration,
/// Number of accounts read during execution
pub accounts_read: usize,
/// Number of storage slots read (SLOAD operations)
pub storage_read: usize,
/// Number of code reads (EXTCODE* operations)
pub code_read: usize,
/// Total bytes of code read
pub code_bytes_read: usize,
/// Number of accounts changed (balance/nonce updates)
pub accounts_changed: usize,
/// Number of accounts deleted (SELFDESTRUCT)
pub accounts_deleted: usize,
/// Number of storage slots changed (SSTORE operations)
pub storage_slots_changed: usize,
/// Number of storage slots deleted (set to zero)
pub storage_slots_deleted: usize,
/// Number of bytecodes created/changed (contract deployments)
pub bytecodes_changed: usize,
/// Total bytes of code written
pub code_bytes_written: usize,
/// Number of EIP-7702 delegations set
pub eip7702_delegations_set: usize,
/// Number of EIP-7702 delegations cleared
pub eip7702_delegations_cleared: usize,
/// Account cache hits
pub account_cache_hits: usize,
/// Account cache misses
pub account_cache_misses: usize,
/// Storage cache hits
pub storage_cache_hits: usize,
/// Storage cache misses
pub storage_cache_misses: usize,
/// Code cache hits
pub code_cache_hits: usize,
/// Code cache misses
pub code_cache_misses: usize,
}

View File

@@ -992,7 +992,7 @@ impl<N: NodePrimitives<SignedTx: SignedTransaction>> NewCanonicalChain<N> {
///
/// Returns the new tip for [`Self::Reorg`] and [`Self::Commit`] variants which commit at least
/// 1 new block.
pub fn tip(&self) -> &RecoveredBlock<N::Block> {
pub fn tip(&self) -> &SealedBlock<N::Block> {
match self {
Self::Commit { new } | Self::Reorg { new, .. } => {
new.last().expect("non empty blocks").recovered_block()

View File

@@ -8,9 +8,6 @@
#![cfg_attr(not(test), warn(unused_crate_dependencies))]
#![cfg_attr(docsrs, feature(doc_cfg))]
mod execution_stats;
pub use execution_stats::ExecutionTimingStats;
mod in_memory;
pub use in_memory::*;

View File

@@ -36,6 +36,15 @@ pub use spec::{
DepositContract, ForkBaseFeeParams, DEV, HOLESKY, HOODI, MAINNET, SEPOLIA,
};
use reth_primitives_traits::sync::OnceLock;
/// Simple utility to create a thread-safe sync cell with a value set.
pub fn once_cell_set<T>(value: T) -> OnceLock<T> {
let once = OnceLock::new();
let _ = once.set(value);
once
}
#[cfg(test)]
mod tests {
use super::*;

View File

@@ -21,7 +21,7 @@ use crate::chainspec::ChainSpecParser;
///
/// This trait is supposed to be implemented by the main struct of the CLI.
///
/// It provides commonly used functionality for running commands and information about the CLI, such
/// It provides commonly used functionality for running commands and information about the CL, such
/// as the name and version.
pub trait RethCli: Sized {
/// The associated `ChainSpecParser` type

View File

@@ -46,7 +46,7 @@ reth-prune.workspace = true
reth-prune-types.workspace = true
reth-revm.workspace = true
reth-stages.workspace = true
reth-stages-types.workspace = true
reth-stages-types = { workspace = true, optional = true }
reth-static-file-types = { workspace = true, features = ["clap"] }
reth-static-file.workspace = true
reth-tasks.workspace = true
@@ -87,8 +87,6 @@ tokio-stream.workspace = true
reqwest.workspace = true
url.workspace = true
metrics.workspace = true
blake3.workspace = true
rayon.workspace = true
# io
fdlimit.workspace = true
@@ -110,6 +108,7 @@ reth-provider = { workspace = true, features = ["test-utils"] }
tempfile.workspace = true
[features]
default = []
arbitrary = [
"dep:proptest",
"dep:arbitrary",
@@ -128,9 +127,12 @@ arbitrary = [
"reth-trie-common?/test-utils",
"reth-codecs/arbitrary",
"reth-prune-types/arbitrary",
"reth-stages-types/arbitrary",
"reth-stages-types?/arbitrary",
"reth-trie-common?/arbitrary",
"alloy-consensus/arbitrary",
"reth-primitives-traits/arbitrary",
"reth-ethereum-primitives/arbitrary",
]
rocksdb = ["reth-db-common/rocksdb", "reth-stages/rocksdb", "reth-provider/rocksdb", "reth-prune/rocksdb"]
edge = ["rocksdb"]

View File

@@ -73,12 +73,17 @@ pub struct EnvironmentArgs<C: ChainSpecParser> {
}
impl<C: ChainSpecParser> EnvironmentArgs<C> {
/// Returns the storage settings for new database initialization.
/// Returns the effective storage settings derived from `--storage.v2`.
///
/// Always returns [`StorageSettings::v2()`] — v2 is the default for all new
/// databases. Existing databases use the settings persisted in their metadata.
/// The base storage mode is determined by `--storage.v2`:
/// - When `--storage.v2` is set: uses [`StorageSettings::v2()`] defaults
/// - Otherwise: uses [`StorageSettings::base()`] defaults
pub fn storage_settings(&self) -> StorageSettings {
StorageSettings::v2()
if self.storage.v2 {
StorageSettings::v2()
} else {
StorageSettings::base()
}
}
/// Initializes environment according to [`AccessRights`] and returns an instance of
@@ -187,6 +192,7 @@ impl<C: ChainSpecParser> EnvironmentArgs<C> {
where
C: ChainSpecParser<ChainSpec = N::ChainSpec>,
{
let prune_modes = config.prune.segments.clone();
let factory = ProviderFactory::<NodeTypesWithDBAdapter<N, DatabaseEnv>>::new(
db,
self.chain.clone(),
@@ -194,8 +200,7 @@ impl<C: ChainSpecParser> EnvironmentArgs<C> {
rocksdb_provider,
runtime,
)?
.with_prune_modes(config.prune.segments.clone())
.with_minimum_pruning_distance(config.prune.minimum_pruning_distance);
.with_prune_modes(prune_modes.clone());
// Check for consistency between database and static files.
if !access.is_read_only_inconsistent() &&
@@ -229,13 +234,10 @@ impl<C: ChainSpecParser> EnvironmentArgs<C> {
NoopBodiesDownloader::default(),
NoopEvmConfig::<N::Evm>::default(),
config.stages.clone(),
config.prune.segments.clone(),
prune_modes.clone(),
None,
))
.build(
factory.clone(),
StaticFileProducer::new(factory.clone(), config.prune.segments.clone()),
);
.build(factory.clone(), StaticFileProducer::new(factory.clone(), prune_modes));
// Move all applicable data from database to static files.
pipeline.move_to_static_files()?;

View File

@@ -21,6 +21,7 @@ use std::{
};
use tracing::{info, warn};
#[cfg(all(unix, feature = "rocksdb"))]
mod rocksdb;
/// Interval for logging progress during checksum computation.
@@ -72,6 +73,7 @@ enum Subcommand {
limit: Option<usize>,
},
/// Calculates the checksum of a RocksDB table
#[cfg(all(unix, feature = "rocksdb"))]
Rocksdb {
/// The RocksDB table
#[arg(value_enum)]
@@ -98,6 +100,7 @@ impl Command {
Subcommand::StaticFile { segment, start_block, end_block, limit } => {
checksum_static_file(tool, segment, start_block, end_block, limit)?;
}
#[cfg(all(unix, feature = "rocksdb"))]
Subcommand::Rocksdb { table, limit } => {
rocksdb::checksum_rocksdb(tool, table, limit)?;
}

View File

@@ -102,14 +102,14 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + EthereumHardforks>> Command<C>
let static_files_path = data_dir.static_files();
let exex_wal_path = data_dir.exex_wal();
// ensure the provided datadir exists
// ensure the provided datadir exist
eyre::ensure!(
data_dir.data_dir().is_dir(),
"Datadir does not exist: {:?}",
data_dir.data_dir()
);
// ensure the provided database exists
// ensure the provided database exist
eyre::ensure!(db_path.is_dir(), "Database does not exist: {:?}", db_path);
match self.command {

View File

@@ -9,7 +9,7 @@ use reth_db_api::{
};
use reth_db_common::DbTool;
use reth_node_builder::NodeTypesWithDB;
use reth_provider::{providers::ProviderNodeTypes, StaticFileProviderFactory};
use reth_provider::providers::ProviderNodeTypes;
use reth_storage_api::{BlockNumReader, StateProvider, StorageSettingsCache};
use reth_tasks::spawn_scoped_os_thread;
use std::{
@@ -17,9 +17,9 @@ use std::{
thread,
time::{Duration, Instant},
};
use tracing::info;
use tracing::{error, info};
/// Log progress every 30 seconds
/// Log progress every 5 seconds
const LOG_INTERVAL: Duration = Duration::from_secs(30);
/// The arguments for the `reth db state` command
@@ -152,11 +152,17 @@ impl Command {
let mut storage_keys = BTreeSet::new();
if history_in_rocksdb {
self.collect_staticfile_storage_keys(tool, address, &mut storage_keys)?;
} else {
self.collect_mdbx_storage_keys_parallel(tool, address, &mut storage_keys)?;
error!(
target: "reth::cli",
"Historical storage queries with RocksDB backend are not yet supported. \
Use MDBX for storage history or query current state without --block."
);
return Ok(());
}
// Collect keys from MDBX StorageChangeSets using parallel scanning
self.collect_mdbx_storage_keys_parallel(tool, address, &mut storage_keys)?;
info!(
target: "reth::cli",
address = %address,
@@ -201,63 +207,6 @@ impl Command {
Ok(())
}
/// Collects storage keys from static file StorageChangeSets (storage_v2).
fn collect_staticfile_storage_keys<N: NodeTypesWithDB + ProviderNodeTypes>(
&self,
tool: &DbTool<N>,
address: Address,
keys: &mut BTreeSet<B256>,
) -> eyre::Result<()> {
let tip = tool.provider_factory.provider()?.best_block_number()?;
if tip == 0 {
return Ok(());
}
info!(
target: "reth::cli",
address = %address,
tip,
"Scanning static file storage changesets"
);
let static_file_provider = tool.provider_factory.static_file_provider();
let walker = static_file_provider.walk_storage_changeset_range(0..=tip);
let mut total_scanned = 0usize;
let mut last_log = Instant::now();
for changeset_result in walker {
let (block_addr, storage_entry) = changeset_result?;
total_scanned += 1;
if block_addr.address() == address {
keys.insert(storage_entry.key);
}
if last_log.elapsed() >= LOG_INTERVAL {
info!(
target: "reth::cli",
address = %address,
entries_scanned = total_scanned,
unique_keys = keys.len(),
"Scanning static file storage changesets"
);
last_log = Instant::now();
}
}
info!(
target: "reth::cli",
address = %address,
total_entries = total_scanned,
unique_keys = keys.len(),
"Finished static file storage changeset scan"
);
Ok(())
}
/// Collects storage keys from MDBX StorageChangeSets using parallel block range scanning.
fn collect_mdbx_storage_keys_parallel<N: NodeTypesWithDB + ProviderNodeTypes>(
&self,

View File

@@ -0,0 +1,630 @@
use crate::common::EnvironmentArgs;
use clap::Parser;
use eyre::Result;
use lz4::Decoder;
use reqwest::{blocking::Client as BlockingClient, header::RANGE, Client, StatusCode};
use reth_chainspec::{EthChainSpec, EthereumHardforks};
use reth_cli::chainspec::ChainSpecParser;
use reth_fs_util as fs;
use std::{
borrow::Cow,
fs::OpenOptions,
io::{self, BufWriter, Read, Write},
path::{Path, PathBuf},
sync::{Arc, OnceLock},
time::{Duration, Instant},
};
use tar::Archive;
use tokio::task;
use tracing::info;
use url::Url;
use zstd::stream::read::Decoder as ZstdDecoder;
const BYTE_UNITS: [&str; 4] = ["B", "KB", "MB", "GB"];
const MERKLE_BASE_URL: &str = "https://downloads.merkle.io";
const EXTENSION_TAR_LZ4: &str = ".tar.lz4";
const EXTENSION_TAR_ZSTD: &str = ".tar.zst";
/// Global static download defaults
static DOWNLOAD_DEFAULTS: OnceLock<DownloadDefaults> = OnceLock::new();
/// Download configuration defaults
///
/// Global defaults can be set via [`DownloadDefaults::try_init`].
#[derive(Debug, Clone)]
pub struct DownloadDefaults {
/// List of available snapshot sources
pub available_snapshots: Vec<Cow<'static, str>>,
/// Default base URL for snapshots
pub default_base_url: Cow<'static, str>,
/// Default base URL for chain-aware snapshots.
///
/// When set, the chain ID is appended to form the full URL: `{base_url}/{chain_id}`.
/// For example, given a base URL of `https://snapshots.example.com` and chain ID `1`,
/// the resulting URL would be `https://snapshots.example.com/1`.
///
/// Falls back to [`default_base_url`](Self::default_base_url) when `None`.
pub default_chain_aware_base_url: Option<Cow<'static, str>>,
/// Optional custom long help text that overrides the generated help
pub long_help: Option<String>,
}
impl DownloadDefaults {
/// Initialize the global download defaults with this configuration
pub fn try_init(self) -> Result<(), Self> {
DOWNLOAD_DEFAULTS.set(self)
}
/// Get a reference to the global download defaults
pub fn get_global() -> &'static DownloadDefaults {
DOWNLOAD_DEFAULTS.get_or_init(DownloadDefaults::default_download_defaults)
}
/// Default download configuration with defaults from merkle.io and publicnode
pub fn default_download_defaults() -> Self {
Self {
available_snapshots: vec![
Cow::Borrowed("https://www.merkle.io/snapshots (default, mainnet archive)"),
Cow::Borrowed("https://publicnode.com/snapshots (full nodes & testnets)"),
],
default_base_url: Cow::Borrowed(MERKLE_BASE_URL),
default_chain_aware_base_url: None,
long_help: None,
}
}
/// Generates the long help text for the download URL argument using these defaults.
///
/// If a custom long_help is set, it will be returned. Otherwise, help text is generated
/// from the available_snapshots list.
pub fn long_help(&self) -> String {
if let Some(ref custom_help) = self.long_help {
return custom_help.clone();
}
let mut help = String::from(
"Specify a snapshot URL or let the command propose a default one.\n\nAvailable snapshot sources:\n",
);
for source in &self.available_snapshots {
help.push_str("- ");
help.push_str(source);
help.push('\n');
}
help.push_str(
"\nIf no URL is provided, the latest archive snapshot for the selected chain\nwill be proposed for download from ",
);
help.push_str(
self.default_chain_aware_base_url.as_deref().unwrap_or(&self.default_base_url),
);
help.push_str(
".\n\nLocal file:// URLs are also supported for extracting snapshots from disk.",
);
help
}
/// Add a snapshot source to the list
pub fn with_snapshot(mut self, source: impl Into<Cow<'static, str>>) -> Self {
self.available_snapshots.push(source.into());
self
}
/// Replace all snapshot sources
pub fn with_snapshots(mut self, sources: Vec<Cow<'static, str>>) -> Self {
self.available_snapshots = sources;
self
}
/// Set the default base URL, e.g. `https://downloads.merkle.io`.
pub fn with_base_url(mut self, url: impl Into<Cow<'static, str>>) -> Self {
self.default_base_url = url.into();
self
}
/// Set the default chain-aware base URL.
pub fn with_chain_aware_base_url(mut self, url: impl Into<Cow<'static, str>>) -> Self {
self.default_chain_aware_base_url = Some(url.into());
self
}
/// Builder: Set custom long help text, overriding the generated help
pub fn with_long_help(mut self, help: impl Into<String>) -> Self {
self.long_help = Some(help.into());
self
}
}
impl Default for DownloadDefaults {
fn default() -> Self {
Self::default_download_defaults()
}
}
#[derive(Debug, Parser)]
pub struct DownloadCommand<C: ChainSpecParser> {
#[command(flatten)]
env: EnvironmentArgs<C>,
/// Custom URL to download the snapshot from
#[arg(long, short, long_help = DownloadDefaults::get_global().long_help())]
url: Option<String>,
}
impl<C: ChainSpecParser<ChainSpec: EthChainSpec + EthereumHardforks>> DownloadCommand<C> {
pub async fn execute<N>(self) -> Result<()> {
let data_dir = self.env.datadir.resolve_datadir(self.env.chain.chain());
fs::create_dir_all(&data_dir)?;
let url = match self.url {
Some(url) => url,
None => {
let url = get_latest_snapshot_url(self.env.chain.chain().id()).await?;
info!(target: "reth::cli", "Using default snapshot URL: {}", url);
url
}
};
info!(target: "reth::cli",
chain = %self.env.chain.chain(),
dir = ?data_dir.data_dir(),
url = %url,
"Starting snapshot download and extraction"
);
stream_and_extract(&url, data_dir.data_dir()).await?;
info!(target: "reth::cli", "Snapshot downloaded and extracted successfully");
Ok(())
}
}
impl<C: ChainSpecParser> DownloadCommand<C> {
/// Returns the underlying chain being used to run this command
pub fn chain_spec(&self) -> Option<&Arc<C::ChainSpec>> {
Some(&self.env.chain)
}
}
// Monitor process status and display progress every 100ms
// to avoid overwhelming stdout
struct DownloadProgress {
downloaded: u64,
total_size: u64,
last_displayed: Instant,
started_at: Instant,
}
impl DownloadProgress {
/// Creates new progress tracker with given total size
fn new(total_size: u64) -> Self {
let now = Instant::now();
Self { downloaded: 0, total_size, last_displayed: now, started_at: now }
}
/// Converts bytes to human readable format (B, KB, MB, GB)
fn format_size(size: u64) -> String {
let mut size = size as f64;
let mut unit_index = 0;
while size >= 1024.0 && unit_index < BYTE_UNITS.len() - 1 {
size /= 1024.0;
unit_index += 1;
}
format!("{:.2} {}", size, BYTE_UNITS[unit_index])
}
/// Format duration as human readable string
fn format_duration(duration: Duration) -> String {
let secs = duration.as_secs();
if secs < 60 {
format!("{secs}s")
} else if secs < 3600 {
format!("{}m {}s", secs / 60, secs % 60)
} else {
format!("{}h {}m", secs / 3600, (secs % 3600) / 60)
}
}
/// Updates progress bar
fn update(&mut self, chunk_size: u64) -> Result<()> {
self.downloaded += chunk_size;
// Only update display at most 10 times per second for efficiency
if self.last_displayed.elapsed() >= Duration::from_millis(100) {
let formatted_downloaded = Self::format_size(self.downloaded);
let formatted_total = Self::format_size(self.total_size);
let progress = (self.downloaded as f64 / self.total_size as f64) * 100.0;
// Calculate ETA based on current speed
let elapsed = self.started_at.elapsed();
let eta = if self.downloaded > 0 {
let remaining = self.total_size.saturating_sub(self.downloaded);
let speed = self.downloaded as f64 / elapsed.as_secs_f64();
if speed > 0.0 {
Duration::from_secs_f64(remaining as f64 / speed)
} else {
Duration::ZERO
}
} else {
Duration::ZERO
};
let eta_str = Self::format_duration(eta);
// Pad with spaces to clear any previous longer line
print!(
"\rDownloading and extracting... {progress:.2}% ({formatted_downloaded} / {formatted_total}) ETA: {eta_str} ",
);
io::stdout().flush()?;
self.last_displayed = Instant::now();
}
Ok(())
}
}
/// Adapter to track progress while reading
struct ProgressReader<R> {
reader: R,
progress: DownloadProgress,
}
impl<R: Read> ProgressReader<R> {
fn new(reader: R, total_size: u64) -> Self {
Self { reader, progress: DownloadProgress::new(total_size) }
}
}
impl<R: Read> Read for ProgressReader<R> {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
let bytes = self.reader.read(buf)?;
if bytes > 0 &&
let Err(e) = self.progress.update(bytes as u64)
{
return Err(io::Error::other(e));
}
Ok(bytes)
}
}
/// Supported compression formats for snapshots
#[derive(Debug, Clone, Copy)]
enum CompressionFormat {
Lz4,
Zstd,
}
impl CompressionFormat {
/// Detect compression format from file extension
fn from_url(url: &str) -> Result<Self> {
let path =
Url::parse(url).map(|u| u.path().to_string()).unwrap_or_else(|_| url.to_string());
if path.ends_with(EXTENSION_TAR_LZ4) {
Ok(Self::Lz4)
} else if path.ends_with(EXTENSION_TAR_ZSTD) {
Ok(Self::Zstd)
} else {
Err(eyre::eyre!(
"Unsupported file format. Expected .tar.lz4 or .tar.zst, got: {}",
path
))
}
}
}
/// Extracts a compressed tar archive to the target directory with progress tracking.
fn extract_archive<R: Read>(
reader: R,
total_size: u64,
format: CompressionFormat,
target_dir: &Path,
) -> Result<()> {
let progress_reader = ProgressReader::new(reader, total_size);
match format {
CompressionFormat::Lz4 => {
let decoder = Decoder::new(progress_reader)?;
Archive::new(decoder).unpack(target_dir)?;
}
CompressionFormat::Zstd => {
let decoder = ZstdDecoder::new(progress_reader)?;
Archive::new(decoder).unpack(target_dir)?;
}
}
info!(target: "reth::cli", "Extraction complete.");
Ok(())
}
/// Extracts a snapshot from a local file.
fn extract_from_file(path: &Path, format: CompressionFormat, target_dir: &Path) -> Result<()> {
let file = std::fs::File::open(path)?;
let total_size = file.metadata()?.len();
extract_archive(file, total_size, format, target_dir)
}
const MAX_DOWNLOAD_RETRIES: u32 = 10;
const RETRY_BACKOFF_SECS: u64 = 5;
/// Wrapper that tracks download progress while writing data.
/// Used with [`io::copy`] to display progress during downloads.
struct ProgressWriter<W> {
inner: W,
progress: DownloadProgress,
}
impl<W: Write> Write for ProgressWriter<W> {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.inner.write(buf)?;
let _ = self.progress.update(n as u64);
Ok(n)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
/// Downloads a file with resume support using HTTP Range requests.
/// Automatically retries on failure, resuming from where it left off.
/// Returns the path to the downloaded file and its total size.
fn resumable_download(url: &str, target_dir: &Path) -> Result<(PathBuf, u64)> {
let file_name = Url::parse(url)
.ok()
.and_then(|u| u.path_segments()?.next_back().map(|s| s.to_string()))
.unwrap_or_else(|| "snapshot.tar".to_string());
let final_path = target_dir.join(&file_name);
let part_path = target_dir.join(format!("{file_name}.part"));
let client = BlockingClient::builder().timeout(Duration::from_secs(30)).build()?;
let mut total_size: Option<u64> = None;
let mut last_error: Option<eyre::Error> = None;
let finalize_download = |size: u64| -> Result<(PathBuf, u64)> {
fs::rename(&part_path, &final_path)?;
info!(target: "reth::cli", "Download complete: {}", final_path.display());
Ok((final_path.clone(), size))
};
for attempt in 1..=MAX_DOWNLOAD_RETRIES {
let existing_size = fs::metadata(&part_path).map(|m| m.len()).unwrap_or(0);
if let Some(total) = total_size &&
existing_size >= total
{
return finalize_download(total);
}
if attempt > 1 {
info!(target: "reth::cli",
"Retry attempt {}/{} - resuming from {} bytes",
attempt, MAX_DOWNLOAD_RETRIES, existing_size
);
}
let mut request = client.get(url);
if existing_size > 0 {
request = request.header(RANGE, format!("bytes={existing_size}-"));
if attempt == 1 {
info!(target: "reth::cli", "Resuming download from {} bytes", existing_size);
}
}
let response = match request.send().and_then(|r| r.error_for_status()) {
Ok(r) => r,
Err(e) => {
last_error = Some(e.into());
if attempt < MAX_DOWNLOAD_RETRIES {
info!(target: "reth::cli",
"Download failed, retrying in {} seconds...", RETRY_BACKOFF_SECS
);
std::thread::sleep(Duration::from_secs(RETRY_BACKOFF_SECS));
}
continue;
}
};
let is_partial = response.status() == StatusCode::PARTIAL_CONTENT;
let size = if is_partial {
response
.headers()
.get("Content-Range")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.split('/').next_back())
.and_then(|v| v.parse().ok())
} else {
response.content_length()
};
if total_size.is_none() {
total_size = size;
}
let current_total = total_size.ok_or_else(|| {
eyre::eyre!("Server did not provide Content-Length or Content-Range header")
})?;
let file = if is_partial && existing_size > 0 {
OpenOptions::new()
.append(true)
.open(&part_path)
.map_err(|e| fs::FsPathError::open(e, &part_path))?
} else {
fs::create_file(&part_path)?
};
let start_offset = if is_partial { existing_size } else { 0 };
let mut progress = DownloadProgress::new(current_total);
progress.downloaded = start_offset;
let mut writer = ProgressWriter { inner: BufWriter::new(file), progress };
let mut reader = response;
let copy_result = io::copy(&mut reader, &mut writer);
let flush_result = writer.inner.flush();
println!();
if let Err(e) = copy_result.and(flush_result) {
last_error = Some(e.into());
if attempt < MAX_DOWNLOAD_RETRIES {
info!(target: "reth::cli",
"Download interrupted, retrying in {} seconds...", RETRY_BACKOFF_SECS
);
std::thread::sleep(Duration::from_secs(RETRY_BACKOFF_SECS));
}
continue;
}
return finalize_download(current_total);
}
Err(last_error
.unwrap_or_else(|| eyre::eyre!("Download failed after {} attempts", MAX_DOWNLOAD_RETRIES)))
}
/// Fetches the snapshot from a remote URL with resume support, then extracts it.
fn download_and_extract(url: &str, format: CompressionFormat, target_dir: &Path) -> Result<()> {
let (downloaded_path, total_size) = resumable_download(url, target_dir)?;
info!(target: "reth::cli", "Extracting snapshot...");
let file = fs::open(&downloaded_path)?;
extract_archive(file, total_size, format, target_dir)?;
fs::remove_file(&downloaded_path)?;
info!(target: "reth::cli", "Removed downloaded archive");
Ok(())
}
/// Downloads and extracts a snapshot, blocking until finished.
///
/// Supports both `file://` URLs for local files and HTTP(S) URLs for remote downloads.
fn blocking_download_and_extract(url: &str, target_dir: &Path) -> Result<()> {
let format = CompressionFormat::from_url(url)?;
if let Ok(parsed_url) = Url::parse(url) &&
parsed_url.scheme() == "file"
{
let file_path = parsed_url
.to_file_path()
.map_err(|_| eyre::eyre!("Invalid file:// URL path: {}", url))?;
extract_from_file(&file_path, format, target_dir)
} else {
download_and_extract(url, format, target_dir)
}
}
async fn stream_and_extract(url: &str, target_dir: &Path) -> Result<()> {
let target_dir = target_dir.to_path_buf();
let url = url.to_string();
task::spawn_blocking(move || blocking_download_and_extract(&url, &target_dir)).await??;
Ok(())
}
// Builds default URL for latest mainnet archive snapshot using configured defaults
async fn get_latest_snapshot_url(chain_id: u64) -> Result<String> {
let defaults = DownloadDefaults::get_global();
let base_url = match &defaults.default_chain_aware_base_url {
Some(url) => format!("{url}/{chain_id}"),
None => defaults.default_base_url.to_string(),
};
let latest_url = format!("{base_url}/latest.txt");
let filename = Client::new()
.get(latest_url)
.send()
.await?
.error_for_status()?
.text()
.await?
.trim()
.to_string();
Ok(format!("{base_url}/{filename}"))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_download_defaults_builder() {
let defaults = DownloadDefaults::default()
.with_snapshot("https://example.com/snapshots (example)")
.with_base_url("https://example.com");
assert_eq!(defaults.default_base_url, "https://example.com");
assert_eq!(defaults.available_snapshots.len(), 3); // 2 defaults + 1 added
}
#[test]
fn test_download_defaults_replace_snapshots() {
let defaults = DownloadDefaults::default().with_snapshots(vec![
Cow::Borrowed("https://custom1.com"),
Cow::Borrowed("https://custom2.com"),
]);
assert_eq!(defaults.available_snapshots.len(), 2);
assert_eq!(defaults.available_snapshots[0], "https://custom1.com");
}
#[test]
fn test_long_help_generation() {
let defaults = DownloadDefaults::default();
let help = defaults.long_help();
assert!(help.contains("Available snapshot sources:"));
assert!(help.contains("merkle.io"));
assert!(help.contains("publicnode.com"));
assert!(help.contains("file://"));
}
#[test]
fn test_long_help_override() {
let custom_help = "This is custom help text for downloading snapshots.";
let defaults = DownloadDefaults::default().with_long_help(custom_help);
let help = defaults.long_help();
assert_eq!(help, custom_help);
assert!(!help.contains("Available snapshot sources:"));
}
#[test]
fn test_builder_chaining() {
let defaults = DownloadDefaults::default()
.with_base_url("https://custom.example.com")
.with_snapshot("https://snapshot1.com")
.with_snapshot("https://snapshot2.com")
.with_long_help("Custom help for snapshots");
assert_eq!(defaults.default_base_url, "https://custom.example.com");
assert_eq!(defaults.available_snapshots.len(), 4); // 2 defaults + 2 added
assert_eq!(defaults.long_help, Some("Custom help for snapshots".to_string()));
}
#[test]
fn test_compression_format_detection() {
assert!(matches!(
CompressionFormat::from_url("https://example.com/snapshot.tar.lz4"),
Ok(CompressionFormat::Lz4)
));
assert!(matches!(
CompressionFormat::from_url("https://example.com/snapshot.tar.zst"),
Ok(CompressionFormat::Zstd)
));
assert!(matches!(
CompressionFormat::from_url("file:///path/to/snapshot.tar.lz4"),
Ok(CompressionFormat::Lz4)
));
assert!(matches!(
CompressionFormat::from_url("file:///path/to/snapshot.tar.zst"),
Ok(CompressionFormat::Zstd)
));
assert!(CompressionFormat::from_url("https://example.com/snapshot.tar.gz").is_err());
}
}

View File

@@ -1,610 +0,0 @@
use crate::download::{
manifest::{ComponentManifest, ComponentSelection, SnapshotComponentType, SnapshotManifest},
SelectionPreset,
};
use reth_chainspec::{EthereumHardfork, EthereumHardforks};
use reth_config::config::{BlocksPerFileConfig, Config, PruneConfig, StaticFilesConfig};
use reth_db::tables;
use reth_db_api::transaction::{DbTx, DbTxMut};
use reth_node_core::args::DefaultPruningValues;
use reth_prune_types::{PruneCheckpoint, PruneMode, PruneSegment};
use reth_stages_types::StageCheckpoint;
use std::{collections::BTreeMap, path::Path};
use tracing::info;
/// Minimum blocks to keep for receipts, matching `--minimal` prune settings.
const MINIMUM_RECEIPTS_DISTANCE: u64 = 64;
/// Minimum blocks to keep for history/bodies, matching `--minimal` prune settings
/// (`MINIMUM_UNWIND_SAFE_DISTANCE`).
const MINIMUM_HISTORY_DISTANCE: u64 = 10064;
/// Writes a [`Config`] as TOML to `<data_dir>/reth.toml`.
///
/// If the file already exists, it is not overwritten. Returns `true` if the file was written.
pub fn write_config(config: &Config, data_dir: &Path) -> eyre::Result<bool> {
let config_path = data_dir.join("reth.toml");
if config_path.exists() {
info!(target: "reth::cli",
path = ?config_path,
"reth.toml already exists, skipping config generation"
);
return Ok(false);
}
let toml_str = toml::to_string_pretty(config)?;
reth_fs_util::write(&config_path, toml_str)?;
info!(target: "reth::cli",
path = ?config_path,
"Generated reth.toml based on downloaded components"
);
Ok(true)
}
/// Writes prune checkpoints to the provided write transaction.
pub(crate) fn write_prune_checkpoints_tx<Tx>(
tx: &Tx,
config: &Config,
snapshot_block: u64,
) -> eyre::Result<()>
where
Tx: DbTx + DbTxMut,
{
let segments = &config.prune.segments;
// Collect (segment, mode) pairs for all configured prune segments
let checkpoints: Vec<(PruneSegment, PruneMode)> = [
(PruneSegment::SenderRecovery, segments.sender_recovery),
(PruneSegment::TransactionLookup, segments.transaction_lookup),
(PruneSegment::Receipts, segments.receipts),
(PruneSegment::AccountHistory, segments.account_history),
(PruneSegment::StorageHistory, segments.storage_history),
(PruneSegment::Bodies, segments.bodies_history),
]
.into_iter()
.filter_map(|(segment, mode)| mode.map(|m| (segment, m)))
.collect();
if checkpoints.is_empty() {
return Ok(());
}
// Look up the last tx number for the snapshot block from BlockBodyIndices
let tx_number =
tx.get::<tables::BlockBodyIndices>(snapshot_block)?.map(|indices| indices.last_tx_num());
for (segment, prune_mode) in &checkpoints {
let checkpoint = PruneCheckpoint {
block_number: Some(snapshot_block),
tx_number,
prune_mode: *prune_mode,
};
tx.put::<tables::PruneCheckpoints>(*segment, checkpoint)?;
info!(target: "reth::cli",
segment = %segment,
block = snapshot_block,
tx = ?tx_number,
mode = ?prune_mode,
"Set prune checkpoint"
);
}
Ok(())
}
/// Stage IDs for index stages whose output is stored in RocksDB and is never
/// distributed in snapshots.
const INDEX_STAGE_IDS: [&str; 3] =
["TransactionLookup", "IndexAccountHistory", "IndexStorageHistory"];
/// Prune segments that correspond to the index stages.
const INDEX_PRUNE_SEGMENTS: [PruneSegment; 3] =
[PruneSegment::TransactionLookup, PruneSegment::AccountHistory, PruneSegment::StorageHistory];
/// Resets stage and prune checkpoints for stages whose output is not included
/// in the snapshot inside an existing write transaction.
///
/// A snapshot's mdbx comes from a fully synced node, so it has stage checkpoints
/// at the tip for `TransactionLookup`, `IndexAccountHistory`, and
/// `IndexStorageHistory`. Since we don't distribute the rocksdb indices those
/// stages produced, we must reset their checkpoints to block 0. Otherwise the
/// pipeline would see "already done" and skip rebuilding entirely.
///
/// We intentionally do not reset `SenderRecovery`: sender static files are
/// distributed for archive downloads, and non-archive downloads rely on the
/// configured prune checkpoints for this segment.
pub(crate) fn reset_index_stage_checkpoints_tx<Tx>(tx: &Tx) -> eyre::Result<()>
where
Tx: DbTx + DbTxMut,
{
for stage_id in INDEX_STAGE_IDS {
tx.put::<tables::StageCheckpoints>(stage_id.to_string(), StageCheckpoint::default())?;
// Also clear any stage-specific progress data
tx.delete::<tables::StageCheckpointProgresses>(stage_id.to_string(), None)?;
info!(target: "reth::cli", stage = stage_id, "Reset stage checkpoint to block 0");
}
// Clear corresponding prune checkpoints so the pruner doesn't inherit
// state from the source node
for segment in INDEX_PRUNE_SEGMENTS {
tx.delete::<tables::PruneCheckpoints>(segment, None)?;
}
Ok(())
}
/// Generates a [`Config`] from per-component range selections.
///
/// When all data components are selected as `All`, no pruning is configured (archive node).
/// Otherwise, `--minimal` style pruning is applied for missing/partial components.
pub(crate) fn config_for_selections(
selections: &BTreeMap<SnapshotComponentType, ComponentSelection>,
manifest: &SnapshotManifest,
preset: Option<SelectionPreset>,
chain_spec: Option<&impl EthereumHardforks>,
) -> Config {
let selection_for = |ty| selections.get(&ty).copied().unwrap_or(ComponentSelection::None);
let tx_sel = selection_for(SnapshotComponentType::Transactions);
let senders_sel = selection_for(SnapshotComponentType::TransactionSenders);
let receipt_sel = selection_for(SnapshotComponentType::Receipts);
let account_cs_sel = selection_for(SnapshotComponentType::AccountChangesets);
let storage_cs_sel = selection_for(SnapshotComponentType::StorageChangesets);
// Archive node — all data components present, no pruning
let is_archive = [tx_sel, senders_sel, receipt_sel, account_cs_sel, storage_cs_sel]
.iter()
.all(|s| *s == ComponentSelection::All);
// Extract blocks_per_file from manifest for all component types
let blocks_per_file = |ty: SnapshotComponentType| -> Option<u64> {
match manifest.component(ty)? {
ComponentManifest::Chunked(c) => Some(c.blocks_per_file),
ComponentManifest::Single(_) => None,
}
};
let static_files = StaticFilesConfig {
blocks_per_file: BlocksPerFileConfig {
headers: blocks_per_file(SnapshotComponentType::Headers),
transactions: blocks_per_file(SnapshotComponentType::Transactions),
receipts: blocks_per_file(SnapshotComponentType::Receipts),
transaction_senders: blocks_per_file(SnapshotComponentType::TransactionSenders),
account_change_sets: blocks_per_file(SnapshotComponentType::AccountChangesets),
storage_change_sets: blocks_per_file(SnapshotComponentType::StorageChangesets),
},
};
if is_archive || matches!(preset, Some(SelectionPreset::Archive)) {
return Config { static_files, ..Default::default() };
}
if matches!(preset, Some(SelectionPreset::Full)) {
let defaults = DefaultPruningValues::get_global();
let mut segments = defaults.full_prune_modes.clone();
if defaults.full_bodies_history_use_pre_merge {
segments.bodies_history = chain_spec.and_then(|chain_spec| {
chain_spec
.ethereum_fork_activation(EthereumHardfork::Paris)
.block_number()
.map(PruneMode::Before)
});
}
return Config {
prune: PruneConfig { segments, ..Default::default() },
static_files,
..Default::default()
};
}
let mut config = Config::default();
let mut prune = PruneConfig::default();
if senders_sel != ComponentSelection::All {
prune.segments.sender_recovery = Some(PruneMode::Full);
}
prune.segments.transaction_lookup = Some(PruneMode::Full);
if let Some(mode) = selection_to_prune_mode(tx_sel, Some(MINIMUM_HISTORY_DISTANCE)) {
prune.segments.bodies_history = Some(mode);
}
if let Some(mode) = selection_to_prune_mode(receipt_sel, Some(MINIMUM_RECEIPTS_DISTANCE)) {
prune.segments.receipts = Some(mode);
}
if let Some(mode) = selection_to_prune_mode(account_cs_sel, Some(MINIMUM_HISTORY_DISTANCE)) {
prune.segments.account_history = Some(mode);
}
if let Some(mode) = selection_to_prune_mode(storage_cs_sel, Some(MINIMUM_HISTORY_DISTANCE)) {
prune.segments.storage_history = Some(mode);
}
config.prune = prune;
config.static_files = static_files;
config
}
/// Converts a [`ComponentSelection`] to an optional [`PruneMode`].
///
/// `min_distance` enforces the minimum blocks required for this segment.
/// When set, `None` and distances below the minimum are clamped to it
/// instead of producing `PruneMode::Full` which reth would reject.
fn selection_to_prune_mode(
sel: ComponentSelection,
min_distance: Option<u64>,
) -> Option<PruneMode> {
match sel {
ComponentSelection::All => None,
ComponentSelection::Distance(d) => {
Some(PruneMode::Distance(min_distance.map_or(d, |min| d.max(min))))
}
ComponentSelection::None => Some(min_distance.map_or(PruneMode::Full, PruneMode::Distance)),
}
}
/// Human-readable prune config summary.
pub(crate) fn describe_prune_config(config: &Config) -> Vec<String> {
let segments = &config.prune.segments;
[
("sender_recovery", segments.sender_recovery),
("transaction_lookup", segments.transaction_lookup),
("bodies_history", segments.bodies_history),
("receipts", segments.receipts),
("account_history", segments.account_history),
("storage_history", segments.storage_history),
]
.into_iter()
.filter_map(|(name, mode)| mode.map(|m| format!("{name}={}", format_mode(&m))))
.collect()
}
fn format_mode(mode: &PruneMode) -> String {
match mode {
PruneMode::Full => "\"full\"".to_string(),
PruneMode::Distance(d) => format!("{{ distance = {d} }}"),
PruneMode::Before(b) => format!("{{ before = {b} }}"),
}
}
#[cfg(test)]
mod tests {
use super::*;
use reth_db::Database;
/// Empty manifest for tests that only care about prune config.
fn empty_manifest() -> SnapshotManifest {
SnapshotManifest {
block: 0,
chain_id: 1,
storage_version: 2,
timestamp: 0,
base_url: None,
reth_version: None,
components: BTreeMap::new(),
}
}
#[test]
fn write_prune_checkpoints_sets_all_segments() {
let dir = tempfile::tempdir().unwrap();
let db = reth_db::init_db(dir.path(), reth_db::mdbx::DatabaseArguments::default()).unwrap();
let mut selections = BTreeMap::new();
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
let snapshot_block = 21_000_000;
{
let tx = db.tx_mut().unwrap();
write_prune_checkpoints_tx(&tx, &config, snapshot_block).unwrap();
tx.commit().unwrap();
}
// Verify all expected segments have checkpoints
let tx = db.tx().unwrap();
for segment in [
PruneSegment::SenderRecovery,
PruneSegment::TransactionLookup,
PruneSegment::Receipts,
PruneSegment::AccountHistory,
PruneSegment::StorageHistory,
PruneSegment::Bodies,
] {
let checkpoint = tx
.get::<tables::PruneCheckpoints>(segment)
.unwrap()
.unwrap_or_else(|| panic!("expected checkpoint for {segment}"));
assert_eq!(checkpoint.block_number, Some(snapshot_block));
// No BlockBodyIndices in empty DB, so tx_number should be None
assert_eq!(checkpoint.tx_number, None);
}
}
#[test]
fn write_prune_checkpoints_archive_no_checkpoints() {
let dir = tempfile::tempdir().unwrap();
let db = reth_db::init_db(dir.path(), reth_db::mdbx::DatabaseArguments::default()).unwrap();
// Archive node — no pruning configured, so no checkpoints written
let mut selections = BTreeMap::new();
for ty in SnapshotComponentType::ALL {
selections.insert(ty, ComponentSelection::All);
}
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
{
let tx = db.tx_mut().unwrap();
write_prune_checkpoints_tx(&tx, &config, 21_000_000).unwrap();
tx.commit().unwrap();
}
let tx = db.tx().unwrap();
for segment in [PruneSegment::SenderRecovery, PruneSegment::TransactionLookup] {
assert!(
tx.get::<tables::PruneCheckpoints>(segment).unwrap().is_none(),
"expected no checkpoint for {segment} on archive node"
);
}
}
#[test]
fn selections_all_no_pruning() {
let mut selections = BTreeMap::new();
for ty in SnapshotComponentType::ALL {
selections.insert(ty, ComponentSelection::All);
}
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
// Archive node — nothing pruned
assert_eq!(config.prune.segments.transaction_lookup, None);
assert_eq!(config.prune.segments.sender_recovery, None);
assert_eq!(config.prune.segments.bodies_history, None);
assert_eq!(config.prune.segments.receipts, None);
assert_eq!(config.prune.segments.account_history, None);
assert_eq!(config.prune.segments.storage_history, None);
}
#[test]
fn selections_none_clamps_to_minimum_distance() {
let mut selections = BTreeMap::new();
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
assert_eq!(config.prune.segments.transaction_lookup, Some(PruneMode::Full));
assert_eq!(config.prune.segments.sender_recovery, Some(PruneMode::Full));
// All segments clamped to their minimum distances
assert_eq!(
config.prune.segments.bodies_history,
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
);
assert_eq!(
config.prune.segments.receipts,
Some(PruneMode::Distance(MINIMUM_RECEIPTS_DISTANCE))
);
assert_eq!(
config.prune.segments.account_history,
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
);
assert_eq!(
config.prune.segments.storage_history,
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
);
}
#[test]
fn selections_distance_maps_bodies_history() {
let mut selections = BTreeMap::new();
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
selections
.insert(SnapshotComponentType::Transactions, ComponentSelection::Distance(10_064));
selections.insert(SnapshotComponentType::Receipts, ComponentSelection::None);
selections
.insert(SnapshotComponentType::AccountChangesets, ComponentSelection::Distance(10_064));
selections
.insert(SnapshotComponentType::StorageChangesets, ComponentSelection::Distance(10_064));
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
assert_eq!(config.prune.segments.transaction_lookup, Some(PruneMode::Full));
assert_eq!(config.prune.segments.sender_recovery, Some(PruneMode::Full));
// Bodies follows tx selection
assert_eq!(config.prune.segments.bodies_history, Some(PruneMode::Distance(10_064)));
assert_eq!(
config.prune.segments.receipts,
Some(PruneMode::Distance(MINIMUM_RECEIPTS_DISTANCE))
);
assert_eq!(config.prune.segments.account_history, Some(PruneMode::Distance(10_064)));
assert_eq!(config.prune.segments.storage_history, Some(PruneMode::Distance(10_064)));
}
#[test]
fn full_preset_matches_default_full_prune_config() {
let mut selections = BTreeMap::new();
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
selections
.insert(SnapshotComponentType::Transactions, ComponentSelection::Distance(500_000));
selections.insert(SnapshotComponentType::Receipts, ComponentSelection::Distance(10_064));
let chain_spec = reth_chainspec::MAINNET.clone();
let config = config_for_selections(
&selections,
&empty_manifest(),
Some(SelectionPreset::Full),
Some(chain_spec.as_ref()),
);
assert_eq!(config.prune.segments.sender_recovery, Some(PruneMode::Full));
assert_eq!(config.prune.segments.transaction_lookup, None);
assert_eq!(
config.prune.segments.receipts,
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
);
assert_eq!(
config.prune.segments.account_history,
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
);
assert_eq!(
config.prune.segments.storage_history,
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
);
let paris_block = chain_spec
.ethereum_fork_activation(EthereumHardfork::Paris)
.block_number()
.expect("mainnet Paris block should be known");
assert_eq!(config.prune.segments.bodies_history, Some(PruneMode::Before(paris_block)));
}
#[test]
fn describe_selections_all_no_pruning() {
let mut selections = BTreeMap::new();
for ty in SnapshotComponentType::ALL {
selections.insert(ty, ComponentSelection::All);
}
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
let desc = describe_prune_config(&config);
// Archive node — no prune segments described
assert!(desc.is_empty());
}
#[test]
fn describe_selections_with_distances() {
let mut selections = BTreeMap::new();
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
selections
.insert(SnapshotComponentType::Transactions, ComponentSelection::Distance(10_064));
selections.insert(SnapshotComponentType::Receipts, ComponentSelection::None);
let config = config_for_selections(
&selections,
&empty_manifest(),
None,
None::<&reth_chainspec::ChainSpec>,
);
let desc = describe_prune_config(&config);
assert!(desc.contains(&"sender_recovery=\"full\"".to_string()));
// Bodies follows tx selection
assert!(desc.contains(&"bodies_history={ distance = 10064 }".to_string()));
assert!(desc.contains(&"receipts={ distance = 64 }".to_string()));
}
#[test]
fn reset_index_stage_checkpoints_clears_only_rocksdb_index_stages() {
let dir = tempfile::tempdir().unwrap();
let db = reth_db::init_db(dir.path(), reth_db::mdbx::DatabaseArguments::default()).unwrap();
// Simulate a fully synced node: set stage checkpoints at tip
let tip_checkpoint = StageCheckpoint::new(24_500_000);
{
let tx = db.tx_mut().unwrap();
for stage_id in INDEX_STAGE_IDS {
tx.put::<tables::StageCheckpoints>(stage_id.to_string(), tip_checkpoint).unwrap();
}
for segment in INDEX_PRUNE_SEGMENTS {
tx.put::<tables::PruneCheckpoints>(
segment,
PruneCheckpoint {
block_number: Some(24_500_000),
tx_number: None,
prune_mode: PruneMode::Full,
},
)
.unwrap();
}
// Sender recovery checkpoints should be preserved by reset.
tx.put::<tables::StageCheckpoints>("SenderRecovery".to_string(), tip_checkpoint)
.unwrap();
tx.put::<tables::PruneCheckpoints>(
PruneSegment::SenderRecovery,
PruneCheckpoint {
block_number: Some(24_500_000),
tx_number: None,
prune_mode: PruneMode::Full,
},
)
.unwrap();
tx.commit().unwrap();
}
// Reset
{
let tx = db.tx_mut().unwrap();
reset_index_stage_checkpoints_tx(&tx).unwrap();
tx.commit().unwrap();
}
// Verify stage checkpoints are at block 0
let tx = db.tx().unwrap();
for stage_id in INDEX_STAGE_IDS {
let checkpoint = tx
.get::<tables::StageCheckpoints>(stage_id.to_string())
.unwrap()
.expect("checkpoint should exist");
assert_eq!(checkpoint.block_number, 0, "stage {stage_id} should be reset to block 0");
}
// Verify prune checkpoints are deleted
for segment in INDEX_PRUNE_SEGMENTS {
assert!(
tx.get::<tables::PruneCheckpoints>(segment).unwrap().is_none(),
"prune checkpoint for {segment} should be deleted"
);
}
// Verify sender checkpoints are left untouched.
let sender_stage_checkpoint = tx
.get::<tables::StageCheckpoints>("SenderRecovery".to_string())
.unwrap()
.expect("sender checkpoint should exist");
assert_eq!(sender_stage_checkpoint.block_number, tip_checkpoint.block_number);
let sender_prune_checkpoint = tx
.get::<tables::PruneCheckpoints>(PruneSegment::SenderRecovery)
.unwrap()
.expect("sender prune checkpoint should exist");
assert_eq!(sender_prune_checkpoint.block_number, Some(tip_checkpoint.block_number));
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,232 +0,0 @@
use crate::download::manifest::generate_manifest;
use clap::Parser;
use eyre::{Result, WrapErr};
use reth_db::{mdbx::DatabaseArguments, open_db_read_only, tables, Database};
use reth_db_api::transaction::DbTx;
use reth_stages_types::StageId;
use reth_static_file_types::DEFAULT_BLOCKS_PER_STATIC_FILE;
use std::{path::PathBuf, time::Instant};
use tracing::{info, warn};
/// Generate modular chunk archives and a snapshot manifest from a source datadir.
///
/// Archive naming convention:
/// - Chunked: `{component}-{start}-{end}.tar.zst` (e.g. `transactions-0-499999.tar.zst`)
#[derive(Debug, Parser)]
pub struct SnapshotManifestCommand {
/// Source datadir containing static files.
#[arg(long, short = 'd')]
source_datadir: PathBuf,
/// Optional base URL where archives will be hosted.
#[arg(long)]
base_url: Option<String>,
/// Output directory where chunk archives and manifest.json are written.
#[arg(long, short = 'o')]
output_dir: PathBuf,
/// Block number this snapshot was taken at.
///
/// If omitted, this is inferred from the source datadir's `Finish` stage checkpoint.
#[arg(long)]
block: Option<u64>,
/// Chain ID.
#[arg(long, default_value = "1")]
chain_id: u64,
/// Blocks per archive file for chunked components.
///
/// If omitted, this is inferred from header static file ranges in the source datadir.
#[arg(long)]
blocks_per_file: Option<u64>,
}
impl SnapshotManifestCommand {
pub fn execute(self) -> Result<()> {
let block = match self.block {
Some(block) => block,
None => infer_snapshot_block(&self.source_datadir)?,
};
let blocks_per_file = match self.blocks_per_file {
Some(blocks_per_file) => blocks_per_file,
None => infer_blocks_per_file(&self.source_datadir)?,
};
info!(target: "reth::cli",
dir = ?self.source_datadir,
output = ?self.output_dir,
block,
blocks_per_file,
"Packaging modular snapshot archives"
);
let start = Instant::now();
let manifest = generate_manifest(
&self.source_datadir,
&self.output_dir,
self.base_url.as_deref(),
block,
self.chain_id,
blocks_per_file,
)?;
let num_components = manifest.components.len();
let json = serde_json::to_string_pretty(&manifest)?;
let output = self.output_dir.join("manifest.json");
reth_fs_util::write(&output, &json)?;
info!(target: "reth::cli",
path = ?output,
components = num_components,
block = manifest.block,
elapsed = ?start.elapsed(),
"Manifest written"
);
Ok(())
}
}
fn infer_snapshot_block(source_datadir: &std::path::Path) -> Result<u64> {
if let Ok(block) = infer_snapshot_block_from_db(source_datadir) {
return Ok(block);
}
let block = infer_snapshot_block_from_headers(source_datadir)?;
warn!(
target: "reth::cli",
block,
"Could not read Finish stage checkpoint from source DB, using header static-file tip"
);
Ok(block)
}
fn infer_snapshot_block_from_db(source_datadir: &std::path::Path) -> Result<u64> {
let candidates = [source_datadir.join("db"), source_datadir.to_path_buf()];
for db_path in candidates {
if !db_path.exists() {
continue;
}
let db = match open_db_read_only(&db_path, DatabaseArguments::default()) {
Ok(db) => db,
Err(_) => continue,
};
let tx = db.tx()?;
if let Some(checkpoint) = tx.get::<tables::StageCheckpoints>(StageId::Finish.to_string())? {
return Ok(checkpoint.block_number);
}
}
eyre::bail!(
"Could not infer --block from source DB (Finish checkpoint missing); pass --block manually"
)
}
fn infer_snapshot_block_from_headers(source_datadir: &std::path::Path) -> Result<u64> {
let max_end = header_ranges(source_datadir)?
.into_iter()
.map(|(_, end)| end)
.max()
.ok_or_else(|| eyre::eyre!("No header static files found to infer --block"))?;
Ok(max_end)
}
fn infer_blocks_per_file(source_datadir: &std::path::Path) -> Result<u64> {
let mut inferred = None;
for (start, end) in header_ranges(source_datadir)? {
let span = end.saturating_sub(start).saturating_add(1);
if span == 0 {
continue;
}
if let Some(existing) = inferred {
if existing != span {
eyre::bail!(
"Inconsistent header static file ranges; pass --blocks-per-file manually"
);
}
} else {
inferred = Some(span);
}
}
inferred.ok_or_else(|| {
eyre::eyre!(
"Could not infer --blocks-per-file from header static files; pass it manually (default is {DEFAULT_BLOCKS_PER_STATIC_FILE})"
)
})
}
fn header_ranges(source_datadir: &std::path::Path) -> Result<Vec<(u64, u64)>> {
let static_files_dir = source_datadir.join("static_files");
let static_files_dir =
if static_files_dir.exists() { static_files_dir } else { source_datadir.to_path_buf() };
let entries = std::fs::read_dir(&static_files_dir).wrap_err_with(|| {
format!("Failed to read static files directory: {}", static_files_dir.display())
})?;
let mut ranges = Vec::new();
for entry in entries {
let entry = entry?;
let file_name = entry.file_name();
let file_name = file_name.to_string_lossy();
if let Some(range) = parse_headers_range(&file_name) {
ranges.push(range);
}
}
Ok(ranges)
}
fn parse_headers_range(file_name: &str) -> Option<(u64, u64)> {
let remainder = file_name.strip_prefix("static_file_headers_")?;
let (start, end_with_suffix) = remainder.split_once('_')?;
let start = start.parse::<u64>().ok()?;
let end_digits: String = end_with_suffix.chars().take_while(|ch| ch.is_ascii_digit()).collect();
let end = end_digits.parse::<u64>().ok()?;
Some((start, end))
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::tempdir;
#[test]
fn parse_headers_range_works_with_suffixes() {
assert_eq!(parse_headers_range("static_file_headers_0_499999"), Some((0, 499_999)));
assert_eq!(
parse_headers_range("static_file_headers_500000_999999.jar"),
Some((500_000, 999_999))
);
assert_eq!(parse_headers_range("static_file_transactions_0_499999"), None);
}
#[test]
fn infer_blocks_per_file_from_header_ranges() {
let dir = tempdir().unwrap();
let sf = dir.path().join("static_files");
std::fs::create_dir_all(&sf).unwrap();
std::fs::write(sf.join("static_file_headers_0_499999"), []).unwrap();
std::fs::write(sf.join("static_file_headers_500000_999999.jar"), []).unwrap();
assert_eq!(infer_blocks_per_file(dir.path()).unwrap(), 500_000);
}
#[test]
fn infer_snapshot_block_from_headers_uses_max_end() {
let dir = tempdir().unwrap();
let sf = dir.path().join("static_files");
std::fs::create_dir_all(&sf).unwrap();
std::fs::write(sf.join("static_file_headers_0_499999"), []).unwrap();
std::fs::write(sf.join("static_file_headers_500000_999999"), []).unwrap();
assert_eq!(infer_snapshot_block_from_headers(dir.path()).unwrap(), 999_999);
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,437 +0,0 @@
use crate::download::{
manifest::{ComponentSelection, SnapshotComponentType, SnapshotManifest},
DownloadProgress, SelectionPreset,
};
use crossterm::{
event::{self, Event, KeyCode},
execute,
terminal::{disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen},
};
use ratatui::{
backend::CrosstermBackend,
layout::{Constraint, Direction, Layout},
style::{Color, Modifier, Style},
text::{Line, Span},
widgets::{Block, Borders, List, ListItem, ListState, Paragraph},
Frame, Terminal,
};
use std::{
collections::BTreeMap,
io,
time::{Duration, Instant},
};
/// Result of the interactive component selector.
pub struct SelectorOutput {
/// User-confirmed selections with per-component ranges.
pub selections: BTreeMap<SnapshotComponentType, ComponentSelection>,
/// Last preset action used in the TUI, if any.
pub preset: Option<SelectionPreset>,
}
/// All distance presets. Groups filter this to only valid options.
const DISTANCE_PRESETS: [ComponentSelection; 6] = [
ComponentSelection::None,
ComponentSelection::Distance(64),
ComponentSelection::Distance(10_064),
ComponentSelection::Distance(100_000),
ComponentSelection::Distance(1_000_000),
ComponentSelection::All,
];
/// Presets for components that require at least 64 blocks (receipts).
const RECEIPTS_PRESETS: [ComponentSelection; 5] = [
ComponentSelection::Distance(64),
ComponentSelection::Distance(10_064),
ComponentSelection::Distance(100_000),
ComponentSelection::Distance(1_000_000),
ComponentSelection::All,
];
/// Presets for components that require at least 10064 blocks (account/storage history).
const HISTORY_PRESETS: [ComponentSelection; 4] = [
ComponentSelection::Distance(10_064),
ComponentSelection::Distance(100_000),
ComponentSelection::Distance(1_000_000),
ComponentSelection::All,
];
/// A display group bundles one or more component types into a single TUI row.
struct DisplayGroup {
/// Display name shown in the TUI.
name: &'static str,
/// Underlying component types this group controls.
types: Vec<SnapshotComponentType>,
/// Whether this group is required and locked to All.
required: bool,
/// Valid presets for this group. Components with minimum distance requirements
/// exclude presets that would produce invalid prune configs.
presets: &'static [ComponentSelection],
}
/// Build the display groups from available components in the manifest.
fn build_groups(manifest: &SnapshotManifest) -> Vec<DisplayGroup> {
let has = |ty: SnapshotComponentType| manifest.component(ty).is_some();
let mut groups = Vec::new();
if has(SnapshotComponentType::State) {
groups.push(DisplayGroup {
name: "State (mdbx)",
types: vec![SnapshotComponentType::State],
required: true,
presets: &DISTANCE_PRESETS,
});
}
if has(SnapshotComponentType::Headers) {
groups.push(DisplayGroup {
name: "Headers",
types: vec![SnapshotComponentType::Headers],
required: true,
presets: &DISTANCE_PRESETS,
});
}
if has(SnapshotComponentType::Transactions) {
groups.push(DisplayGroup {
name: "Transactions",
types: vec![SnapshotComponentType::Transactions],
required: false,
presets: &HISTORY_PRESETS,
});
}
if has(SnapshotComponentType::Receipts) {
groups.push(DisplayGroup {
name: "Receipts",
types: vec![SnapshotComponentType::Receipts],
required: false,
presets: &RECEIPTS_PRESETS,
});
}
// Bundle account + storage changesets as "State History"
let has_acc = has(SnapshotComponentType::AccountChangesets);
let has_stor = has(SnapshotComponentType::StorageChangesets);
if has_acc || has_stor {
let mut types = Vec::new();
if has_acc {
types.push(SnapshotComponentType::AccountChangesets);
}
if has_stor {
types.push(SnapshotComponentType::StorageChangesets);
}
groups.push(DisplayGroup {
name: "State History",
types,
required: false,
presets: &HISTORY_PRESETS,
});
}
groups
}
struct SelectorApp {
manifest: SnapshotManifest,
full_preset: BTreeMap<SnapshotComponentType, ComponentSelection>,
/// Display groups shown in the TUI.
groups: Vec<DisplayGroup>,
/// Current selection for each group.
selections: Vec<ComponentSelection>,
/// Last preset action invoked by user.
preset: Option<SelectionPreset>,
/// Current cursor position.
cursor: usize,
/// List state for ratatui.
list_state: ListState,
}
impl SelectorApp {
fn new(
manifest: SnapshotManifest,
full_preset: BTreeMap<SnapshotComponentType, ComponentSelection>,
) -> Self {
let groups = build_groups(&manifest);
// Default to the minimal preset (matches --minimal prune config)
let selections = groups.iter().map(|g| g.types[0].minimal_selection()).collect();
let mut list_state = ListState::default();
list_state.select(Some(0));
Self {
manifest,
full_preset,
groups,
selections,
preset: Some(SelectionPreset::Minimal),
cursor: 0,
list_state,
}
}
fn cycle_right(&mut self) {
if let Some(group) = self.groups.get(self.cursor) {
if group.required {
return;
}
let presets = group.presets;
let current = self.selections[self.cursor];
let idx = presets.iter().position(|p| *p == current).unwrap_or(0);
self.selections[self.cursor] = presets[(idx + 1) % presets.len()];
self.preset = None;
}
}
fn cycle_left(&mut self) {
if let Some(group) = self.groups.get(self.cursor) {
if group.required {
return;
}
let presets = group.presets;
let current = self.selections[self.cursor];
let idx = presets.iter().position(|p| *p == current).unwrap_or(0);
self.selections[self.cursor] = presets[(idx + presets.len() - 1) % presets.len()];
self.preset = None;
}
}
fn select_all(&mut self) {
for sel in &mut self.selections {
*sel = ComponentSelection::All;
}
self.preset = Some(SelectionPreset::Archive);
}
fn select_minimal(&mut self) {
for (i, group) in self.groups.iter().enumerate() {
self.selections[i] = group.types[0].minimal_selection();
}
self.preset = Some(SelectionPreset::Minimal);
}
fn select_full(&mut self) {
for (i, group) in self.groups.iter().enumerate() {
let mut selection = group.types[0].minimal_selection();
for ty in &group.types {
if let Some(sel) = self.full_preset.get(ty).copied() {
selection = sel;
break;
}
}
self.selections[i] = selection;
}
self.preset = Some(SelectionPreset::Full);
}
fn move_up(&mut self) {
if self.cursor > 0 {
self.cursor -= 1;
} else {
self.cursor = self.groups.len().saturating_sub(1);
}
self.list_state.select(Some(self.cursor));
}
fn move_down(&mut self) {
if self.cursor < self.groups.len() - 1 {
self.cursor += 1;
} else {
self.cursor = 0;
}
self.list_state.select(Some(self.cursor));
}
/// Build the flat component→selection map from grouped selections.
fn selection_map(&self) -> BTreeMap<SnapshotComponentType, ComponentSelection> {
let mut map = BTreeMap::new();
for (group, sel) in self.groups.iter().zip(&self.selections) {
for ty in &group.types {
map.insert(*ty, *sel);
}
}
map
}
/// Size for a single group, summing all component types in the group.
fn group_size(&self, group_idx: usize) -> u64 {
let sel = self.selections[group_idx];
let distance = match sel {
ComponentSelection::None => return 0,
ComponentSelection::All => None,
ComponentSelection::Distance(d) => Some(d),
};
self.groups[group_idx]
.types
.iter()
.map(|ty| self.manifest.size_for_distance(*ty, distance))
.sum()
}
fn total_selected_size(&self) -> u64 {
(0..self.groups.len()).map(|i| self.group_size(i)).sum()
}
}
/// Runs the interactive component selector TUI.
pub fn run_selector(
manifest: SnapshotManifest,
full_preset: &BTreeMap<SnapshotComponentType, ComponentSelection>,
) -> eyre::Result<SelectorOutput> {
enable_raw_mode()?;
let mut stdout = io::stdout();
execute!(stdout, EnterAlternateScreen)?;
let backend = CrosstermBackend::new(stdout);
let mut terminal = Terminal::new(backend)?;
let mut app = SelectorApp::new(manifest, full_preset.clone());
let result = event_loop(&mut terminal, &mut app);
disable_raw_mode()?;
execute!(terminal.backend_mut(), LeaveAlternateScreen)?;
terminal.show_cursor()?;
result
}
fn event_loop(
terminal: &mut Terminal<CrosstermBackend<io::Stdout>>,
app: &mut SelectorApp,
) -> eyre::Result<SelectorOutput> {
let tick_rate = Duration::from_millis(100);
let mut last_tick = Instant::now();
loop {
terminal.draw(|f| render(f, app))?;
let timeout =
tick_rate.checked_sub(last_tick.elapsed()).unwrap_or_else(|| Duration::from_secs(0));
if crossterm::event::poll(timeout)? &&
let Event::Key(key) = event::read()? &&
key.kind == event::KeyEventKind::Press
{
match key.code {
KeyCode::Char('q') | KeyCode::Esc => {
eyre::bail!("Download cancelled by user");
}
KeyCode::Enter => {
return Ok(SelectorOutput {
selections: app.selection_map(),
preset: app.preset,
});
}
KeyCode::Right | KeyCode::Char('l') | KeyCode::Char(' ') => app.cycle_right(),
KeyCode::Left | KeyCode::Char('h') => app.cycle_left(),
KeyCode::Char('a') => app.select_all(),
KeyCode::Char('f') => app.select_full(),
KeyCode::Char('m') => app.select_minimal(),
KeyCode::Up | KeyCode::Char('k') => app.move_up(),
KeyCode::Down | KeyCode::Char('j') => app.move_down(),
_ => {}
}
}
if last_tick.elapsed() >= tick_rate {
last_tick = Instant::now();
}
}
}
fn format_selection(sel: &ComponentSelection) -> String {
match sel {
ComponentSelection::All => "All".to_string(),
ComponentSelection::Distance(d) => format!("Last {d} blocks"),
ComponentSelection::None => "None".to_string(),
}
}
fn render(f: &mut Frame<'_>, app: &mut SelectorApp) {
let chunks = Layout::default()
.direction(Direction::Vertical)
.constraints([
Constraint::Length(3), // Header
Constraint::Min(8), // Component list
Constraint::Length(3), // Footer
])
.split(f.area());
// Header
let block_info = if app.manifest.block > 0 {
format!(" (block {})", app.manifest.block)
} else {
String::new()
};
let header = Paragraph::new(format!(" Select snapshot components to download{}", block_info))
.style(Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD))
.block(Block::default().borders(Borders::ALL).title("reth download"));
f.render_widget(header, chunks[0]);
// Component list
let items: Vec<ListItem<'_>> = app
.groups
.iter()
.enumerate()
.map(|(i, group)| {
let sel = &app.selections[i];
let sel_str = format_selection(sel);
let size = app.group_size(i);
let size_str =
if size > 0 { DownloadProgress::format_size(size) } else { String::new() };
let required = if group.required { " (required)" } else { "" };
let at_max = *sel == *group.presets.last().unwrap_or(&ComponentSelection::All);
let at_min = *sel == group.presets[0];
let arrows = if group.required {
" "
} else if at_max {
""
} else if at_min {
""
} else {
"◂ ▸"
};
let style = if group.required {
Style::default().fg(Color::DarkGray)
} else if matches!(sel, ComponentSelection::None) {
Style::default().fg(Color::White)
} else {
Style::default().fg(Color::Green)
};
ListItem::new(Line::from(vec![
Span::styled(format!(" {:<22}", group.name), style),
Span::styled(
format!("{arrows} {:<12}", sel_str),
style.add_modifier(Modifier::BOLD),
),
Span::styled(format!("{:>10}", size_str), style.add_modifier(Modifier::DIM)),
Span::styled(required.to_string(), Style::default().fg(Color::DarkGray)),
]))
})
.collect();
let total_str = DownloadProgress::format_size(app.total_selected_size());
let list = List::new(items)
.block(
Block::default()
.borders(Borders::ALL)
.title(format!("Components — Total: {total_str}")),
)
.highlight_style(Style::default().add_modifier(Modifier::BOLD).bg(Color::DarkGray))
.highlight_symbol("");
f.render_stateful_widget(list, chunks[1], &mut app.list_state);
// Footer
let footer = Paragraph::new(
" [←/→] adjust [m] minimal [f] full [a] archive [Enter] confirm [Esc] cancel",
)
.style(Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD))
.block(Block::default().borders(Borders::ALL));
f.render_widget(footer, chunks[2]);
}

View File

@@ -19,12 +19,11 @@ use reth_node_api::BlockTy;
use reth_node_events::node::NodeEvent;
use reth_provider::{
providers::ProviderNodeTypes, BlockNumReader, HeaderProvider, ProviderError, ProviderFactory,
RocksDBProviderFactory, StageCheckpointReader,
StageCheckpointReader,
};
use reth_prune::PruneModes;
use reth_stages::{prelude::*, ControlFlow, Pipeline, StageId, StageSet};
use reth_static_file::StaticFileProducer;
use reth_storage_api::StorageSettingsCache;
use std::{path::Path, sync::Arc};
use tokio::sync::watch;
use tracing::{debug, error, info, warn};
@@ -109,11 +108,7 @@ where
let provider = provider_factory.provider()?;
let init_blocks = provider.tx_ref().entries::<tables::HeaderNumbers>()?;
let init_txns = if provider_factory.cached_storage_settings().storage_v2 {
provider_factory.rocksdb_provider().iter::<tables::TransactionHashNumbers>()?.count()
} else {
provider.tx_ref().entries::<tables::TransactionHashNumbers>()?
};
let init_txns = provider.tx_ref().entries::<tables::TransactionHashNumbers>()?;
drop(provider);
let mut total_decoded_blocks = 0;
@@ -220,12 +215,8 @@ where
let provider = provider_factory.provider()?;
let total_imported_blocks = provider.tx_ref().entries::<tables::HeaderNumbers>()? - init_blocks;
let current_txns = if provider_factory.cached_storage_settings().storage_v2 {
provider_factory.rocksdb_provider().iter::<tables::TransactionHashNumbers>()?.count()
} else {
provider.tx_ref().entries::<tables::TransactionHashNumbers>()?
};
let total_imported_txns = current_txns - init_txns;
let total_imported_txns =
provider.tx_ref().entries::<tables::TransactionHashNumbers>()? - init_txns;
let result = ImportResult {
total_decoded_blocks,

View File

@@ -47,7 +47,7 @@ pub struct InitStateCommand<C: ChainSpecParser> {
/// Specifies whether to initialize the state without relying on EVM historical data.
///
/// When enabled, and before inserting the state, it creates a dummy chain up to the last EVM
/// block specified. It then appends the first provided block.
/// block specified. It then, appends the first block provided block.
///
/// - **Note**: **Do not** import receipts and blocks beforehand, or this will fail or be
/// ignored.

View File

@@ -125,12 +125,12 @@ pub struct NodeCommand<C: ChainSpecParser, Ext: clap::Args + fmt::Debug = NoArgs
}
impl<C: ChainSpecParser> NodeCommand<C> {
/// Parses only the default CLI arguments
/// Parsers only the default CLI arguments
pub fn parse_args() -> Self {
Self::parse()
}
/// Parses only the default [`NodeCommand`] arguments from the given iterator
/// Parsers only the default [`NodeCommand`] arguments from the given iterator
pub fn try_parse_args_from<I, T>(itr: I) -> Result<Self, clap::error::Error>
where
I: IntoIterator<Item = T>,

View File

@@ -193,15 +193,11 @@ impl<C: ChainSpecParser> DownloadArgs<C> {
let default_secret_key_path = data_dir.p2p_secret();
let p2p_secret_key = self.network.secret_key(default_secret_key_path)?;
let rlpx_socket = (self.network.addr, self.network.port).into();
let boot_nodes = self
.network
.resolved_bootnodes()
.unwrap_or_else(|| self.chain.bootnodes().unwrap_or_default());
let boot_nodes = self.chain.bootnodes().unwrap_or_default();
let net =
NetworkConfigBuilder::<N::NetworkPrimitives>::new(p2p_secret_key, Runtime::test())
.peer_config(config.peers_config_with_basic_nodes_from_file(None))
.sessions_config(config.sessions)
.external_ip_resolver(self.network.nat.clone())
.network_id(self.network.network_id)
.boot_nodes(boot_nodes.clone())

View File

@@ -12,6 +12,7 @@ use reth_node_metrics::{
server::{MetricServer, MetricServerConfig},
version::VersionInfo,
};
#[cfg(all(unix, feature = "rocksdb"))]
use reth_provider::RocksDBProviderFactory;
use reth_prune::PrunerBuilder;
use reth_static_file::StaticFileProducer;
@@ -121,6 +122,7 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + EthereumHardforks>> PruneComma
}
// Flush and compact RocksDB to reclaim disk space after pruning
#[cfg(all(unix, feature = "rocksdb"))]
{
info!(target: "reth::cli", "Flushing and compacting RocksDB...");
provider_factory.rocksdb_provider().flush_and_compact()?;

View File

@@ -95,10 +95,6 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + Hardforks + EthereumHardforks>
}
};
if min_block > max_block {
eyre::bail!("--from ({min_block}) is beyond --to ({max_block}), nothing to re-execute");
}
let num_tasks = self.num_tasks.unwrap_or_else(|| {
std::thread::available_parallelism().map(|n| n.get() as u64).unwrap_or(10)
});

View File

@@ -127,14 +127,12 @@ fn unwind_and_copy<N: ProviderNodeTypes>(
AccountHashingStage {
clean_threshold: u64::MAX,
commit_threshold: u64::MAX,
commit_entries: u64::MAX,
etl_config: EtlConfig::default(),
}
.execute(&provider, execute_input)?;
StorageHashingStage {
clean_threshold: u64::MAX,
commit_threshold: u64::MAX,
commit_entries: u64::MAX,
etl_config: EtlConfig::default(),
}
.execute(&provider, execute_input)?;

View File

@@ -107,7 +107,7 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + Hardforks + EthereumHardforks>
Comp: CliNodeComponents<N>,
F: FnOnce(Arc<C::ChainSpec>) -> Comp,
{
// Quit early if the stage requires a commit and `--commit` is not provided.
// Quit early if the stages requires a commit and `--commit` is not provided.
if self.requires_commit() && !self.commit {
return Err(eyre::eyre!(
"The stage {} requires overwriting existing static files and must commit, but `--commit` was not provided. Please pass `--commit` and try again.",
@@ -282,22 +282,14 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + Hardforks + EthereumHardforks>
),
StageEnum::AccountHashing => (
Box::new(AccountHashingStage::new(
HashingConfig {
clean_threshold: 1,
commit_threshold: batch_size,
commit_entries: u64::MAX,
},
HashingConfig { clean_threshold: 1, commit_threshold: batch_size },
etl_config,
)),
None,
),
StageEnum::StorageHashing => (
Box::new(StorageHashingStage::new(
HashingConfig {
clean_threshold: 1,
commit_threshold: batch_size,
commit_entries: u64::MAX,
},
HashingConfig { clean_threshold: 1, commit_threshold: batch_size },
etl_config,
)),
None,

View File

@@ -71,12 +71,7 @@ impl CliRunner {
) -> Result<(), E>
where
F: Future<Output = Result<(), E>>,
E: Send
+ Sync
+ std::fmt::Display
+ From<std::io::Error>
+ From<reth_tasks::PanickedTaskError>
+ 'static,
E: Send + Sync + From<std::io::Error> + From<reth_tasks::PanickedTaskError> + 'static,
{
let (context, task_manager_handle) = cli_context(&self.runtime);
@@ -86,8 +81,8 @@ impl CliRunner {
run_until_ctrl_c(command(context)),
));
if let Err(err) = &command_res {
error!(target: "reth::cli", %err, "shutting down due to error");
if command_res.is_err() {
error!(target: "reth::cli", "shutting down due to error");
} else {
debug!(target: "reth::cli", "shutting down gracefully");
// after the command has finished or exit signal was received we shutdown the
@@ -110,12 +105,7 @@ impl CliRunner {
) -> Result<(), E>
where
F: Future<Output = Result<(), E>> + Send + 'static,
E: Send
+ Sync
+ std::fmt::Display
+ From<std::io::Error>
+ From<reth_tasks::PanickedTaskError>
+ 'static,
E: Send + Sync + From<std::io::Error> + From<reth_tasks::PanickedTaskError> + 'static,
{
let (context, task_manager_handle) = cli_context(&self.runtime);
@@ -132,8 +122,8 @@ impl CliRunner {
),
));
if let Err(err) = &command_res {
error!(target: "reth::cli", %err, "shutting down due to error");
if command_res.is_err() {
error!(target: "reth::cli", "shutting down due to error");
} else {
debug!(target: "reth::cli", "shutting down gracefully");
self.runtime.graceful_shutdown_with_timeout(self.config.graceful_shutdown_timeout);

View File

@@ -1,6 +1,6 @@
//! Configuration files.
use reth_network_types::{PeersConfig, SessionsConfig};
use reth_prune_types::{PruneModes, MINIMUM_UNWIND_SAFE_DISTANCE};
use reth_prune_types::PruneModes;
use reth_stages_types::ExecutionStageThresholds;
use reth_static_file_types::{StaticFileMap, StaticFileSegment};
use std::{
@@ -332,15 +332,11 @@ pub struct HashingConfig {
pub clean_threshold: u64,
/// The maximum number of entities to process before committing progress to the database.
pub commit_threshold: u64,
/// The maximum number of changeset entries to process before committing progress. The stage
/// commits after either `commit_threshold` blocks or `commit_entries` entries, whichever
/// comes first. This bounds memory usage when blocks contain many state changes.
pub commit_entries: u64,
}
impl Default for HashingConfig {
fn default() -> Self {
Self { clean_threshold: 500_000, commit_threshold: 100_000, commit_entries: 30_000_000 }
Self { clean_threshold: 500_000, commit_threshold: 100_000 }
}
}
@@ -540,24 +536,11 @@ pub struct PruneConfig {
/// Pruning configuration for every part of the data that can be pruned.
#[cfg_attr(feature = "serde", serde(alias = "parts"))]
pub segments: PruneModes,
/// Minimum distance from the tip required for pruning. Controls the safety margin for
/// reorgs and manual unwinds. Defaults to [`MINIMUM_UNWIND_SAFE_DISTANCE`].
#[cfg_attr(feature = "serde", serde(default = "default_minimum_pruning_distance"))]
pub minimum_pruning_distance: u64,
}
/// Returns the default minimum pruning distance.
const fn default_minimum_pruning_distance() -> u64 {
MINIMUM_UNWIND_SAFE_DISTANCE
}
impl Default for PruneConfig {
fn default() -> Self {
Self {
block_interval: DEFAULT_BLOCK_INTERVAL,
segments: PruneModes::default(),
minimum_pruning_distance: MINIMUM_UNWIND_SAFE_DISTANCE,
}
Self { block_interval: DEFAULT_BLOCK_INTERVAL, segments: PruneModes::default() }
}
}
@@ -590,7 +573,6 @@ impl PruneConfig {
bodies_history,
receipts_log_filter,
},
minimum_pruning_distance,
} = other;
// Merge block_interval, only update if it's the default interval
@@ -598,11 +580,6 @@ impl PruneConfig {
self.block_interval = block_interval;
}
// Merge minimum_pruning_distance, only update if it's the default
if self.minimum_pruning_distance == MINIMUM_UNWIND_SAFE_DISTANCE {
self.minimum_pruning_distance = minimum_pruning_distance;
}
// Merge the various segment prune modes
self.segments.sender_recovery = self.segments.sender_recovery.or(sender_recovery);
self.segments.transaction_lookup = self.segments.transaction_lookup.or(transaction_lookup);
@@ -642,9 +619,7 @@ mod tests {
use crate::PruneConfig;
use alloy_primitives::Address;
use reth_network_peers::TrustedPeer;
use reth_prune_types::{
PruneMode, PruneModes, ReceiptsLogPruneConfig, MINIMUM_UNWIND_SAFE_DISTANCE,
};
use reth_prune_types::{PruneMode, PruneModes, ReceiptsLogPruneConfig};
use std::{collections::BTreeMap, path::Path, str::FromStr, time::Duration};
fn with_tempdir(filename: &str, proc: fn(&std::path::Path)) {
@@ -1114,7 +1089,6 @@ receipts = { distance = 16384 }
fn test_prune_config_merge() {
let mut config1 = PruneConfig {
block_interval: 5,
minimum_pruning_distance: MINIMUM_UNWIND_SAFE_DISTANCE,
segments: PruneModes {
sender_recovery: Some(PruneMode::Full),
transaction_lookup: None,
@@ -1131,7 +1105,6 @@ receipts = { distance = 16384 }
let config2 = PruneConfig {
block_interval: 10,
minimum_pruning_distance: MINIMUM_UNWIND_SAFE_DISTANCE,
segments: PruneModes {
sender_recovery: Some(PruneMode::Distance(500)),
transaction_lookup: Some(PruneMode::Full),

View File

@@ -75,3 +75,8 @@ path = "tests/e2e-testsuite/main.rs"
[[test]]
name = "rocksdb"
path = "tests/rocksdb/main.rs"
required-features = ["rocksdb"]
[features]
rocksdb = ["reth-node-core/rocksdb", "reth-provider/rocksdb", "reth-cli-commands/rocksdb"]
edge = ["rocksdb"]

View File

@@ -96,11 +96,6 @@ where
self
}
/// Sets the pruning arguments for the test nodes.
pub fn with_pruning(self, pruning: reth_node_core::args::PruningArgs) -> Self {
self.with_node_config_modifier(move |config| config.with_pruning(pruning.clone()))
}
/// Enables v2 storage defaults (`--storage.v2`), routing tx hashes, history
/// indices, etc. to `RocksDB` and changesets/senders to static files.
pub fn with_storage_v2(self) -> Self {

View File

@@ -1,5 +1,7 @@
//! E2E tests for `RocksDB` provider functionality.
#![cfg(all(feature = "rocksdb", unix))]
use alloy_consensus::BlockHeader;
use alloy_primitives::B256;
use alloy_rpc_types_eth::{Transaction, TransactionReceipt};

View File

@@ -38,8 +38,8 @@ op-alloy-rpc-types-engine = { workspace = true, optional = true }
workspace = true
[features]
# op = [
# "dep:op-alloy-rpc-types-engine",
# "reth-payload-primitives/op",
# "reth-primitives-traits/op",
# ]
op = [
"dep:op-alloy-rpc-types-engine",
"reth-payload-primitives/op",
"reth-primitives-traits/op",
]

View File

@@ -39,7 +39,6 @@ thiserror.workspace = true
[features]
default = ["std"]
trie-debug = []
std = [
"reth-execution-types/std",
"reth-ethereum-primitives/std",

View File

@@ -26,15 +26,10 @@ pub const DEFAULT_RESERVED_CPU_CORES: usize = 1;
/// Depth 4 means we keep roughly 16^4 = 65536 potential branch paths at most.
pub const DEFAULT_SPARSE_TRIE_PRUNE_DEPTH: usize = 4;
/// Default LFU hot-slot capacity for sparse trie pruning.
/// Default maximum number of storage tries to keep after pruning.
///
/// Limits the number of `(address, slot)` pairs retained across prune cycles.
pub const DEFAULT_SPARSE_TRIE_MAX_HOT_SLOTS: usize = 1500;
/// Default LFU hot-account capacity for sparse trie pruning.
///
/// Limits the number of account addresses retained across prune cycles.
pub const DEFAULT_SPARSE_TRIE_MAX_HOT_ACCOUNTS: usize = 1000;
/// Storage tries beyond this limit are cleared (but allocations preserved).
pub const DEFAULT_SPARSE_TRIE_MAX_STORAGE_TRIES: usize = 100;
/// Default timeout for the state root task before spawning a sequential fallback.
pub const DEFAULT_STATE_ROOT_TASK_TIMEOUT: Duration = Duration::from_secs(1);
@@ -136,28 +131,15 @@ pub struct TreeConfig {
disable_cache_metrics: bool,
/// Depth for sparse trie pruning after state root computation.
sparse_trie_prune_depth: usize,
/// LFU hot-slot capacity: max `(address, slot)` pairs retained across prune cycles.
sparse_trie_max_hot_slots: usize,
/// LFU hot-account capacity: max account addresses retained across prune cycles.
sparse_trie_max_hot_accounts: usize,
/// When set, blocks whose total processing time (execution + state reads + state root +
/// DB commit) exceeds this duration trigger a structured `warn!` log with detailed timing,
/// state-operation counts, and cache hit-rate metrics. `Duration::ZERO` logs every block.
slow_block_threshold: Option<Duration>,
/// Maximum number of storage tries to retain after pruning.
sparse_trie_max_storage_tries: usize,
/// Whether to fully disable sparse trie cache pruning between blocks.
disable_sparse_trie_cache_pruning: bool,
/// Whether to use the arena-based sparse trie implementation.
enable_arena_sparse_trie: bool,
/// Timeout for the state root task before spawning a sequential fallback computation.
/// If `Some`, after waiting this duration for the state root task, a sequential state root
/// computation is spawned in parallel and whichever finishes first is used.
/// If `None`, the timeout fallback is disabled.
state_root_task_timeout: Option<Duration>,
/// Maximum random jitter applied before each proof computation (trie-debug only).
/// When set, each proof worker sleeps for a random duration up to this value
/// before starting a proof calculation.
#[cfg(feature = "trie-debug")]
proof_jitter: Option<Duration>,
}
impl Default for TreeConfig {
@@ -183,14 +165,9 @@ impl Default for TreeConfig {
allow_unwind_canonical_header: false,
disable_cache_metrics: false,
sparse_trie_prune_depth: DEFAULT_SPARSE_TRIE_PRUNE_DEPTH,
sparse_trie_max_hot_slots: DEFAULT_SPARSE_TRIE_MAX_HOT_SLOTS,
sparse_trie_max_hot_accounts: DEFAULT_SPARSE_TRIE_MAX_HOT_ACCOUNTS,
slow_block_threshold: None,
sparse_trie_max_storage_tries: DEFAULT_SPARSE_TRIE_MAX_STORAGE_TRIES,
disable_sparse_trie_cache_pruning: false,
enable_arena_sparse_trie: false,
state_root_task_timeout: Some(DEFAULT_STATE_ROOT_TASK_TIMEOUT),
#[cfg(feature = "trie-debug")]
proof_jitter: None,
}
}
}
@@ -219,9 +196,7 @@ impl TreeConfig {
allow_unwind_canonical_header: bool,
disable_cache_metrics: bool,
sparse_trie_prune_depth: usize,
sparse_trie_max_hot_slots: usize,
sparse_trie_max_hot_accounts: usize,
slow_block_threshold: Option<Duration>,
sparse_trie_max_storage_tries: usize,
state_root_task_timeout: Option<Duration>,
) -> Self {
Self {
@@ -245,14 +220,9 @@ impl TreeConfig {
allow_unwind_canonical_header,
disable_cache_metrics,
sparse_trie_prune_depth,
sparse_trie_max_hot_slots,
sparse_trie_max_hot_accounts,
slow_block_threshold,
sparse_trie_max_storage_tries,
disable_sparse_trie_cache_pruning: false,
enable_arena_sparse_trie: false,
state_root_task_timeout,
#[cfg(feature = "trie-debug")]
proof_jitter: None,
}
}
@@ -501,43 +471,14 @@ impl TreeConfig {
self
}
/// Returns the LFU hot-slot capacity for sparse trie pruning.
pub const fn sparse_trie_max_hot_slots(&self) -> usize {
self.sparse_trie_max_hot_slots
/// Returns the maximum number of storage tries to retain after pruning.
pub const fn sparse_trie_max_storage_tries(&self) -> usize {
self.sparse_trie_max_storage_tries
}
/// Setter for LFU hot-slot capacity.
pub const fn with_sparse_trie_max_hot_slots(mut self, max_hot_slots: usize) -> Self {
self.sparse_trie_max_hot_slots = max_hot_slots;
self
}
/// Returns the LFU hot-account capacity for sparse trie pruning.
pub const fn sparse_trie_max_hot_accounts(&self) -> usize {
self.sparse_trie_max_hot_accounts
}
/// Setter for LFU hot-account capacity.
pub const fn with_sparse_trie_max_hot_accounts(mut self, max_hot_accounts: usize) -> Self {
self.sparse_trie_max_hot_accounts = max_hot_accounts;
self
}
/// Returns the slow block threshold, if configured.
///
/// When `Some`, blocks whose total processing time exceeds this duration emit a structured
/// warning with timing, state-operation, and cache-hit-rate details. `Duration::ZERO` logs
/// every block.
pub const fn slow_block_threshold(&self) -> Option<Duration> {
self.slow_block_threshold
}
/// Setter for slow block threshold.
pub const fn with_slow_block_threshold(
mut self,
slow_block_threshold: Option<Duration>,
) -> Self {
self.slow_block_threshold = slow_block_threshold;
/// Setter for maximum storage tries to retain.
pub const fn with_sparse_trie_max_storage_tries(mut self, max_tries: usize) -> Self {
self.sparse_trie_max_storage_tries = max_tries;
self
}
@@ -552,17 +493,6 @@ impl TreeConfig {
self
}
/// Returns whether the arena-based sparse trie is enabled.
pub const fn enable_arena_sparse_trie(&self) -> bool {
self.enable_arena_sparse_trie
}
/// Setter for whether to enable the arena-based sparse trie.
pub const fn with_enable_arena_sparse_trie(mut self, value: bool) -> Self {
self.enable_arena_sparse_trie = value;
self
}
/// Returns the state root task timeout.
pub const fn state_root_task_timeout(&self) -> Option<Duration> {
self.state_root_task_timeout
@@ -573,17 +503,4 @@ impl TreeConfig {
self.state_root_task_timeout = timeout;
self
}
/// Returns the proof jitter duration, if configured (trie-debug only).
#[cfg(feature = "trie-debug")]
pub const fn proof_jitter(&self) -> Option<Duration> {
self.proof_jitter
}
/// Setter for proof jitter (trie-debug only).
#[cfg(feature = "trie-debug")]
pub const fn with_proof_jitter(mut self, proof_jitter: Option<Duration>) -> Self {
self.proof_jitter = proof_jitter;
self
}
}

View File

@@ -9,7 +9,7 @@ use core::{
fmt::{Display, Formatter, Result},
time::Duration,
};
use reth_chain_state::{ExecutedBlock, ExecutionTimingStats};
use reth_chain_state::ExecutedBlock;
use reth_ethereum_primitives::EthPrimitives;
use reth_primitives_traits::{NodePrimitives, SealedBlock, SealedHeader};
@@ -32,8 +32,6 @@ pub enum ConsensusEngineEvent<N: NodePrimitives = EthPrimitives> {
CanonicalChainCommitted(Box<SealedHeader<N::BlockHeader>>, Duration),
/// The consensus engine processed an invalid block.
InvalidBlock(Box<SealedBlock<N::Block>>),
/// A slow block was detected after persistence, with its timing statistics.
SlowBlock(SlowBlockInfo),
}
impl<N: NodePrimitives> ConsensusEngineEvent<N> {
@@ -75,25 +73,6 @@ where
Self::BlockReceived(num_hash) => {
write!(f, "BlockReceived({num_hash:?})")
}
Self::SlowBlock(info) => {
write!(
f,
"SlowBlock(block={}, total={:?})",
info.stats.block_number, info.total_duration
)
}
}
}
}
/// Information about a slow block detected after persistence.
#[derive(Clone, Debug)]
pub struct SlowBlockInfo {
/// The timing statistics for the slow block.
pub stats: Box<ExecutionTimingStats>,
/// The commit duration for the batch containing this block.
pub commit_duration: Duration,
/// The total duration (execution + `state_root` + commit).
/// Note: `state_read` is a subset of execution and is not added separately.
pub total_duration: Duration,
}

Some files were not shown because too many files have changed in this diff Show More