mirror of
https://github.com/paradigmxyz/reth.git
synced 2026-04-30 03:01:58 -04:00
Compare commits
1 Commits
eip8037
...
emma/fix-c
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f04db57d3a |
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: patch
|
||||
---
|
||||
|
||||
Fixed a bug in `merge_subtrie_updates` where source insertions did not cancel destination removals (and vice versa), causing inconsistent trie updates accumulated across multiple `root()` calls without intermediate `take_updates()`. Added a test covering the cross-cancellation behavior.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-tasks: patch
|
||||
---
|
||||
|
||||
Added panic handler to all rayon thread pools that logs panics via `tracing::error` instead of aborting the process.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-network-types: patch
|
||||
---
|
||||
|
||||
Increased default maximum concurrent outbound dials from 15 to 30.
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
reth-trie-common: minor
|
||||
reth-trie: minor
|
||||
---
|
||||
|
||||
Added `contains_range` method to `PrefixSet` for checking if any key falls within a half-open range. Added prefix set support to `ProofCalculator` via `with_prefix_set`, enabling stale cached hash invalidation and branch collapse detection when keys are inserted or removed; propagated storage prefix sets through `SyncAccountValueEncoder`.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: patch
|
||||
---
|
||||
|
||||
Refactored arena trie internals by adding a `BranchChildIdx::sibling()` helper, deduplicating `Index`/`NodeArena` type aliases, and replacing `is_empty()` with a `drop_root()` method. Fixed a bug where `cursor.pop()` was called before checking if the leaf was the root node, which could cause incorrect dirty-state propagation.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: minor
|
||||
---
|
||||
|
||||
Fixed a bug in `ArenaParallelSparseTrie` where subtrie updates that would completely empty a subtrie were incorrectly dispatched to parallel workers instead of being processed inline, preventing correct branch collapse detection when blinded siblings are present. Refactored the `SparseTrie` test suite to accept a `fn() -> T` factory instead of requiring `T: Default`, enabling a new `arena_parallel_sparse_trie_always_parallel` test variant that exercises all tests with parallelism thresholds set to 1. Added `test_branch_collapse_multi_empty_subtries_blinded_remaining` to cover the case where removing multiple revealed leaves empties their subtries and leaves a single blinded sibling requiring a proof.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-payload-builder: minor
|
||||
---
|
||||
|
||||
Added observability metrics for payload resolve latency and new payload job creation latency to the payload builder service.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: patch
|
||||
---
|
||||
|
||||
Fixed another branch collapse edge case where `check_subtrie_collapse_needs_proof` incorrectly compared removal count against total update count (including `Touched` entries), causing it to skip proof requests for blinded siblings and panic when the subtrie emptied. Added a regression test covering the removals + `Touched` + blinded sibling scenario.
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
reth-chain-state: minor
|
||||
reth-engine-primitives: minor
|
||||
reth-engine-tree: minor
|
||||
reth-node-core: minor
|
||||
reth-node-events: minor
|
||||
reth: patch
|
||||
---
|
||||
|
||||
Added configurable slow block logging (`--engine.slow-block-threshold`) that emits a structured `warn!` log with detailed timing, state-operation counts, and cache hit-rate metrics for blocks whose total processing time exceeds the threshold. Introduced `ExecutionTimingStats`, `CacheStats`, `StateProviderStats`, and `SlowBlockInfo` types to carry execution statistics from block validation through persistence, and refactored `PersistenceResult` to carry commit duration alongside the last persisted block.
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
reth-trie-db: minor
|
||||
reth-engine-tree: minor
|
||||
---
|
||||
|
||||
Added `PendingChangeset` and `PendingChangesetGuard` to `ChangesetCache` so concurrent readers wait for an in-progress computation instead of falling back to the expensive DB-based path. The guard automatically cancels the pending entry on drop (e.g. task panic), ensuring waiters always make progress.
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
reth-engine-primitives: minor
|
||||
reth-engine-tree: minor
|
||||
reth-node-core: minor
|
||||
reth-trie-parallel: minor
|
||||
---
|
||||
|
||||
Added `--engine.proof-jitter` CLI option behind the `trie-debug` feature flag. When set, each proof worker sleeps for a random duration up to the specified value before starting proof computation, useful for stress-testing timing-sensitive proof logic.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-rpc-convert: patch
|
||||
---
|
||||
|
||||
Updated `alloy-evm` dependency to git revision `9bc2dba` and adapted `TxEnvConverter` impl to the updated `TryIntoTxEnv` trait signature that now includes a `Spec` generic parameter.
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
reth-trie: patch
|
||||
reth-trie-sparse: patch
|
||||
---
|
||||
|
||||
Refactored test harness for sparse trie tests by extracting `TrieTestHarness` into a shared `reth-trie` test utility, replacing duplicated inline harness code across multiple test modules. Updated `proof_v2` return type to include an optional root hash, and converted `original_root` and `storage` from public fields to accessor methods.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie: patch
|
||||
---
|
||||
|
||||
Removed the local `increment_and_strip_trailing_zeros` function and `PATH_ALL_ZEROS` static in `proof_v2`, replacing them with the equivalent `Nibbles::next_without_prefix` and `Nibbles::is_zeroes` builtins. Also replaced manual `.get()` calls on `state_mask`/`hash_mask` with direct field access and switched to `Nibbles::unpack_array` over the unsafe `unpack_unchecked`.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-engine-tree: patch
|
||||
---
|
||||
|
||||
Added idle-time pre-computation of account trie upper hashes in the sparse trie payload processor when no pending proof results are available.
|
||||
@@ -1,7 +0,0 @@
|
||||
---
|
||||
reth-cli-commands: minor
|
||||
reth-node-core: minor
|
||||
reth: patch
|
||||
---
|
||||
|
||||
Made v2 storage the default for all new databases, deprecating the `--storage.v2` flag to a hidden no-op kept for backwards compatibility. Updated CLI reference docs to remove the now-hidden flag from all command help pages.
|
||||
@@ -1,7 +0,0 @@
|
||||
---
|
||||
reth-engine-tree: patch
|
||||
reth-trie-sparse: patch
|
||||
reth-tasks: patch
|
||||
---
|
||||
|
||||
Offloaded deallocation of expensive proof node buffers to a persistent background thread (`Runtime::spawn_drop`) to avoid blocking state root computation or lock-holding code.
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: patch
|
||||
reth-engine-tree: patch
|
||||
---
|
||||
|
||||
Removed the `skip_proof_node_filtering` flag, `revealed_account_paths`/`revealed_paths` tracking, and the `filter_revealed_v2_proof_nodes` function from the sparse trie implementation. Also removed the corresponding skipped-nodes metrics, simplifying the proof node reveal path to always pass nodes directly to the sparse trie without pre-filtering.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: minor
|
||||
---
|
||||
|
||||
Added a comprehensive generic `SparseTrie` test suite covering `set_root`, `reveal_nodes`, `update_leaves`, `root`, `take_updates`, `commit_updates`, `prune`, `wipe`/`clear`, `get_leaf_value`, `find_leaf`, `size_hint`, and integration lifecycle scenarios. Tests are stamped out for all concrete `SparseTrie` implementations via a macro.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-cli-commands: minor
|
||||
---
|
||||
|
||||
Added `reth_version` field to `SnapshotManifest` to record the Reth version that produced a snapshot. The field is optional and populated automatically during manifest generation.
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
reth-trie-common: minor
|
||||
reth-trie: minor
|
||||
reth-trie-parallel: minor
|
||||
reth-engine-tree: patch
|
||||
---
|
||||
|
||||
Moved `ProofV2Target`, `MultiProofTargetsV2`, and `ChunkedMultiProofTargetsV2` from `reth-trie-parallel::targets_v2` into a new `reth-trie-common::target_v2` module, making these types available at a lower level without pulling in the full parallel trie crate. Added a `multiproof_v2` method to `Proof` in `reth-trie` that generates a state multiproof using the V2 proof calculator with synchronous account value encoding.
|
||||
@@ -1,14 +0,0 @@
|
||||
---
|
||||
reth-engine-tree: patch
|
||||
reth-evm-ethereum: patch
|
||||
reth-evm: patch
|
||||
reth-primitives-traits: patch
|
||||
reth-revm: patch
|
||||
reth-rpc-eth-api: patch
|
||||
reth-rpc-eth-types: patch
|
||||
reth-provider: patch
|
||||
example-custom-evm: patch
|
||||
example-precompile-cache: patch
|
||||
---
|
||||
|
||||
Bumped revm to v35.0.0, revm-inspectors to 0.35.0, and alloy-evm to 0.29.0. Updated call sites throughout the codebase to align with the new APIs, including `ExecutionResult` field changes (`gas_used` → `gas.used()`/`gas.final_refunded()`), removal of `.without_state_clear()`, updated `EthPrecompiles::new(spec)` constructor, and updated `block_hashes.lowest()` access.
|
||||
@@ -1,9 +0,0 @@
|
||||
---
|
||||
reth-trie-sparse: minor
|
||||
reth-engine-primitives: minor
|
||||
reth-engine-tree: minor
|
||||
reth-node-core: minor
|
||||
reth-trie-common: patch
|
||||
---
|
||||
|
||||
Added an arena-based sparse trie implementation (`ArenaParallelSparseTrie`) using `slotmap` arena allocation for node storage, enabling parallel subtrie mutation without per-node hashing overhead. Added `ConfigurableSparseTrie` enum to switch between the arena and hash-map implementations, and a `--engine.enable-arena-sparse-trie` CLI flag to opt in at runtime.
|
||||
@@ -1,7 +0,0 @@
|
||||
---
|
||||
reth-trie: major
|
||||
reth-trie-db: major
|
||||
reth-provider: minor
|
||||
---
|
||||
|
||||
Added `MaskedTrieCursorFactory` and `MaskedTrieCursor` to handle prefix-set-based hash invalidation at the cursor layer, replacing the `DatabaseTrieWitness` trait abstraction. Removed `with_prefix_sets_mut` from `TrieWitness` and deleted `DatabaseTrieWitness` — callers should now wrap their cursor factory with `MaskedTrieCursorFactory` to apply prefix sets during witness/proof computation.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
reth-trie: patch
|
||||
---
|
||||
|
||||
Fixed a potential panic in `ProofCalculator` by clearing internal computation state (`branch_stack`, `child_stack`, `branch_path`, etc.) after errors, preventing stale state from causing `usize` underflow panics when the calculator is reused. Added a test verifying correct behavior after simulated mid-computation errors.
|
||||
@@ -20,11 +20,6 @@
|
||||
# include dist directory, where the reth binary is located after compilation
|
||||
!/dist
|
||||
|
||||
# include PGO build helper used by Dockerfile.depot
|
||||
!/.github
|
||||
!/.github/scripts
|
||||
!/.github/scripts/build_pgo_bolt.sh
|
||||
|
||||
# include licenses
|
||||
!LICENSE-*
|
||||
|
||||
|
||||
1
.github/actionlint.yaml
vendored
1
.github/actionlint.yaml
vendored
@@ -5,4 +5,3 @@ self-hosted-runner:
|
||||
- depot-ubuntu-latest-4
|
||||
- depot-ubuntu-latest-8
|
||||
- depot-ubuntu-latest-16
|
||||
- available
|
||||
|
||||
106
.github/scripts/bench-job-summary.js
vendored
106
.github/scripts/bench-job-summary.js
vendored
@@ -1,106 +0,0 @@
|
||||
// Generates a rich GitHub Actions job summary for reth-bench results.
|
||||
//
|
||||
// Reads from environment:
|
||||
// BENCH_WORK_DIR – Directory containing summary.json
|
||||
// BENCH_PR – PR number (may be empty)
|
||||
// BENCH_ACTOR – GitHub user who triggered the bench
|
||||
// BENCH_CORES – CPU core limit (0 = all)
|
||||
// BENCH_WARMUP_BLOCKS – Number of warmup blocks
|
||||
// BENCH_SAMPLY – 'true' if samply profiling was enabled
|
||||
// BENCH_ABBA – 'true' if ABBA interleaved order was used
|
||||
//
|
||||
// Usage from actions/github-script:
|
||||
// const jobSummary = require('./.github/scripts/bench-job-summary.js');
|
||||
// await jobSummary({ core, context, chartSha, grafanaUrl, runId });
|
||||
|
||||
const fs = require('fs');
|
||||
const { verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
|
||||
|
||||
module.exports = async function ({ core, context, chartSha, grafanaUrl, runId }) {
|
||||
let summary;
|
||||
try {
|
||||
summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
|
||||
} catch (e) {
|
||||
await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
|
||||
return;
|
||||
}
|
||||
|
||||
const repo = `${context.repo.owner}/${context.repo.repo}`;
|
||||
const prNumber = process.env.BENCH_PR;
|
||||
const actor = process.env.BENCH_ACTOR;
|
||||
const commitUrl = `https://github.com/${repo}/commit`;
|
||||
|
||||
const { emoji, label } = verdict(summary.changes);
|
||||
const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
|
||||
const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
|
||||
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
|
||||
|
||||
// Header & metadata
|
||||
const metaParts = [];
|
||||
if (prNumber) metaParts.push(`**[PR #${prNumber}](https://github.com/${repo}/pull/${prNumber})**`);
|
||||
metaParts.push(`triggered by @${actor}`);
|
||||
|
||||
let md = `# ${emoji} ${label}\n\n`;
|
||||
md += metaParts.join(' · ') + '\n\n';
|
||||
md += `**Baseline:** ${baselineLink}\n`;
|
||||
md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
|
||||
md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
|
||||
|
||||
// Main comparison table
|
||||
const rows = metricRows(summary);
|
||||
md += `| Metric | Baseline | Feature | Change |\n`;
|
||||
md += `|--------|----------|---------|--------|\n`;
|
||||
for (const r of rows) {
|
||||
md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
|
||||
}
|
||||
md += '\n';
|
||||
|
||||
// Wait time breakdown
|
||||
const wtRows = waitTimeRows(summary);
|
||||
if (wtRows.length > 0) {
|
||||
md += `### Wait Time Breakdown\n\n`;
|
||||
md += `| Metric | Baseline | Feature |\n`;
|
||||
md += `|--------|----------|--------|\n`;
|
||||
for (const r of wtRows) {
|
||||
md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
|
||||
}
|
||||
md += '\n';
|
||||
}
|
||||
|
||||
// Charts
|
||||
if (chartSha) {
|
||||
const prNum = prNumber || '0';
|
||||
const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/pr/${prNum}/${runId}`;
|
||||
const charts = [
|
||||
{ file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
|
||||
{ file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
|
||||
{ file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
|
||||
];
|
||||
md += `### Charts\n\n`;
|
||||
for (const chart of charts) {
|
||||
md += `<details><summary>${chart.label}</summary>\n\n`;
|
||||
md += `\n\n`;
|
||||
md += `</details>\n\n`;
|
||||
}
|
||||
}
|
||||
|
||||
// Samply profiles
|
||||
const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
|
||||
const samplyLinks = Object.entries(samplyUrls).map(([run, url]) => `- **${run}**: [Firefox Profiler](${url})`);
|
||||
if (samplyLinks.length > 0) {
|
||||
md += `### Samply Profiles\n\n${samplyLinks.join('\n')}\n\n`;
|
||||
}
|
||||
|
||||
// Grafana
|
||||
if (grafanaUrl) {
|
||||
md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
|
||||
}
|
||||
|
||||
// Node errors
|
||||
try {
|
||||
const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
|
||||
if (errors.trim()) md += '\n' + errors + '\n';
|
||||
} catch {}
|
||||
|
||||
await core.summary.addRaw(md).write();
|
||||
};
|
||||
276
.github/scripts/bench-metrics-proxy.py
vendored
276
.github/scripts/bench-metrics-proxy.py
vendored
@@ -1,276 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Prometheus metrics proxy that fetches from a local reth node and
|
||||
re-exposes with additional benchmark labels.
|
||||
|
||||
Reads labels from a JSON file (updated by local-reth-bench.sh between runs)
|
||||
and injects them into every Prometheus metric line.
|
||||
|
||||
Returns empty 200 when reth is not running (clean Grafana gaps).
|
||||
"""
|
||||
import argparse
|
||||
import ipaddress
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
from urllib.request import urlopen
|
||||
from urllib.error import URLError
|
||||
|
||||
|
||||
def read_labels(path):
|
||||
try:
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
except (FileNotFoundError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def inject_labels(metrics_bytes, label_str, label_names):
|
||||
"""Inject labels into Prometheus text format.
|
||||
|
||||
Operates on bytes and uses simple string ops instead of regex
|
||||
for speed on large payloads (reth exposes thousands of metrics).
|
||||
|
||||
Skips injecting into lines that already contain any of the label names
|
||||
to avoid duplicate labels (which Prometheus rejects).
|
||||
"""
|
||||
if not label_str:
|
||||
return metrics_bytes
|
||||
|
||||
label_bytes = label_str.encode("utf-8")
|
||||
# Pre-encode label names for fast duplicate detection
|
||||
label_name_bytes = [n.encode("utf-8") for n in label_names]
|
||||
out = []
|
||||
for line in metrics_bytes.split(b"\n"):
|
||||
# Skip comments and blank lines
|
||||
if line.startswith(b"#") or not line:
|
||||
out.append(line)
|
||||
continue
|
||||
|
||||
brace = line.find(b"{")
|
||||
space = line.find(b" ")
|
||||
|
||||
if space == -1:
|
||||
# Malformed, pass through
|
||||
out.append(line)
|
||||
elif brace != -1 and brace < space:
|
||||
# Has labels: metric{existing="val"} 123
|
||||
close = line.find(b"}", brace)
|
||||
if close == -1:
|
||||
out.append(line)
|
||||
continue
|
||||
|
||||
# Filter out labels that already exist in this line
|
||||
existing = line[brace + 1:close]
|
||||
inject = label_bytes
|
||||
if existing:
|
||||
for name in label_name_bytes:
|
||||
if name + b"=" in existing:
|
||||
# Rebuild inject string excluding this label
|
||||
inject = _remove_label(inject, name)
|
||||
if not inject:
|
||||
out.append(line)
|
||||
continue
|
||||
|
||||
if close == brace + 1:
|
||||
# Empty braces: metric{} 123
|
||||
out.append(line[:close] + inject + line[close:])
|
||||
else:
|
||||
out.append(line[:close] + b"," + inject + line[close:])
|
||||
else:
|
||||
# No labels: metric 123
|
||||
out.append(line[:space] + b"{" + label_bytes + b"}" + line[space:])
|
||||
|
||||
return b"\n".join(out)
|
||||
|
||||
|
||||
def _remove_label(label_bytes, name):
|
||||
"""Remove a single label (name=\"...\") from a comma-separated label string."""
|
||||
parts = []
|
||||
for part in label_bytes.split(b","):
|
||||
if not part.startswith(name + b"="):
|
||||
parts.append(part)
|
||||
return b",".join(parts)
|
||||
|
||||
|
||||
def build_label_str(labels):
|
||||
"""Pre-format the label injection string: key1="val1",key2="val2" """
|
||||
if not labels:
|
||||
return ""
|
||||
return ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
|
||||
|
||||
|
||||
def build_elapsed_gauge(labels):
|
||||
"""Build a bench_elapsed_seconds gauge from run_start_epoch in labels."""
|
||||
start = labels.get("run_start_epoch")
|
||||
if not start:
|
||||
return b""
|
||||
try:
|
||||
elapsed = time.time() - float(start)
|
||||
except (ValueError, TypeError):
|
||||
return b""
|
||||
# Build labels excluding internal keys
|
||||
display = {k: v for k, v in labels.items()
|
||||
if k not in ("run_start_epoch", "reference_epoch")}
|
||||
lstr = build_label_str(display)
|
||||
return (
|
||||
f"# HELP bench_elapsed_seconds Seconds since benchmark run started\n"
|
||||
f"# TYPE bench_elapsed_seconds gauge\n"
|
||||
f"bench_elapsed_seconds{{{lstr}}} {elapsed:.1f}\n"
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
def compute_timestamp_ms(labels):
|
||||
"""Compute a synthetic timestamp so all runs share a common time origin.
|
||||
|
||||
Returns the timestamp in milliseconds, or None if not enough info.
|
||||
Uses: reference_epoch + (now - run_start_epoch) → all runs overlay at
|
||||
the same Grafana time range.
|
||||
"""
|
||||
ref = labels.get("reference_epoch")
|
||||
start = labels.get("run_start_epoch")
|
||||
if not ref or not start:
|
||||
return None
|
||||
try:
|
||||
elapsed = time.time() - float(start)
|
||||
return int((float(ref) + elapsed) * 1000)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def inject_timestamps(metrics_bytes, timestamp_ms):
|
||||
"""Append a Prometheus timestamp (ms) to every data line.
|
||||
|
||||
Prometheus text format: metric{labels} value [timestamp_ms]
|
||||
Adding timestamps causes Prometheus to store all runs' samples
|
||||
at the same relative time, enabling natural overlay in Grafana.
|
||||
"""
|
||||
if timestamp_ms is None:
|
||||
return metrics_bytes
|
||||
|
||||
ts = str(timestamp_ms).encode("utf-8")
|
||||
out = []
|
||||
for line in metrics_bytes.split(b"\n"):
|
||||
if line.startswith(b"#") or not line:
|
||||
out.append(line)
|
||||
else:
|
||||
out.append(line + b" " + ts)
|
||||
return b"\n".join(out)
|
||||
|
||||
|
||||
class MetricsHandler(BaseHTTPRequestHandler):
|
||||
# Use HTTP/1.1 so Content-Length is respected and Prometheus
|
||||
# doesn't have to rely on connection close to detect end of body.
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def do_GET(self):
|
||||
src = self.client_address[0]
|
||||
try:
|
||||
resp = urlopen(self.server.upstream, timeout=2)
|
||||
metrics = resp.read()
|
||||
except (URLError, ConnectionError, OSError):
|
||||
# reth not running — return empty 200
|
||||
self._send(b"")
|
||||
#print(f" scrape from {src}: empty (reth not running)", flush=True)
|
||||
return
|
||||
|
||||
all_labels = read_labels(self.server.labels_file)
|
||||
# Internal keys — not injected as Prometheus labels
|
||||
internal = ("run_start_epoch", "reference_epoch")
|
||||
labels = {k: v for k, v in all_labels.items() if k not in internal}
|
||||
label_str = build_label_str(labels)
|
||||
label_names = sorted(labels.keys())
|
||||
|
||||
t0 = time.monotonic()
|
||||
result = inject_labels(metrics, label_str, label_names)
|
||||
result += build_elapsed_gauge(all_labels)
|
||||
ts_ms = compute_timestamp_ms(all_labels)
|
||||
result = inject_timestamps(result, ts_ms)
|
||||
dt = time.monotonic() - t0
|
||||
|
||||
self._send(result)
|
||||
print(f" scrape from {src}: {len(metrics)} -> {len(result)} bytes, "
|
||||
f"inject {dt*1000:.1f}ms", flush=True)
|
||||
|
||||
def _send(self, body):
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/plain; version=0.0.4")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.send_header("Connection", "close")
|
||||
self.end_headers()
|
||||
if body:
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass # suppress per-request logging
|
||||
|
||||
|
||||
def resolve_bind_address(subnet_cidr):
|
||||
"""Find the local IP address that belongs to the given subnet.
|
||||
|
||||
Uses ``ip -j addr show`` to enumerate interfaces and returns the first
|
||||
address that falls within *subnet_cidr* (e.g. ``10.10.0.0/24``).
|
||||
"""
|
||||
network = ipaddress.ip_network(subnet_cidr, strict=False)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ip", "-j", "addr", "show"],
|
||||
capture_output=True, text=True, check=True,
|
||||
)
|
||||
interfaces = json.loads(result.stdout)
|
||||
except (subprocess.CalledProcessError, FileNotFoundError, json.JSONDecodeError) as exc:
|
||||
print(f"Error: cannot enumerate interfaces: {exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
for iface in interfaces:
|
||||
for addr_info in iface.get("addr_info", []):
|
||||
try:
|
||||
addr = ipaddress.ip_address(addr_info["local"])
|
||||
except (KeyError, ValueError):
|
||||
continue
|
||||
if addr in network:
|
||||
return str(addr)
|
||||
|
||||
print(f"Error: no interface address found in subnet {subnet_cidr}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Prometheus metrics proxy with label injection")
|
||||
parser.add_argument("--labels", default="/tmp/bench-metrics-labels.json",
|
||||
help="Path to JSON file with labels to inject (default: /tmp/bench-metrics-labels.json)")
|
||||
parser.add_argument("--upstream", default="http://127.0.0.1:9100/",
|
||||
help="Upstream reth metrics URL (default: http://127.0.0.1:9100/)")
|
||||
|
||||
bind_group = parser.add_mutually_exclusive_group()
|
||||
bind_group.add_argument("--bind", default=None,
|
||||
help="Address to bind the proxy (default: 0.0.0.0)")
|
||||
bind_group.add_argument("--subnet", default=None,
|
||||
help="Auto-detect bind address from a local interface in this subnet (e.g. 10.10.0.0/24)")
|
||||
|
||||
parser.add_argument("--port", type=int, default=9090,
|
||||
help="Port to bind the proxy (default: 9090)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.subnet:
|
||||
bind_addr = resolve_bind_address(args.subnet)
|
||||
elif args.bind:
|
||||
bind_addr = args.bind
|
||||
else:
|
||||
bind_addr = "0.0.0.0"
|
||||
|
||||
server = HTTPServer((bind_addr, args.port), MetricsHandler)
|
||||
server.upstream = args.upstream
|
||||
server.labels_file = args.labels
|
||||
|
||||
print(f"bench-metrics-proxy listening on {bind_addr}:{args.port}")
|
||||
print(f" upstream: {args.upstream}")
|
||||
print(f" labels: {args.labels}")
|
||||
sys.stdout.flush()
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
40
.github/scripts/bench-reth-build.sh
vendored
40
.github/scripts/bench-reth-build.sh
vendored
@@ -22,22 +22,6 @@ MODE="$1"
|
||||
SOURCE_DIR="$2"
|
||||
COMMIT="$3"
|
||||
|
||||
# Tracy support: when BENCH_TRACY is "on" or "full", add Tracy cargo features
|
||||
# and frame pointers for accurate stack traces.
|
||||
EXTRA_FEATURES=""
|
||||
EXTRA_RUSTFLAGS=""
|
||||
if [ "${BENCH_TRACY:-off}" != "off" ]; then
|
||||
EXTRA_FEATURES="tracy,tracy-client/ondemand"
|
||||
EXTRA_RUSTFLAGS=" -C force-frame-pointers=yes"
|
||||
fi
|
||||
|
||||
# Cache suffix: hash of features+rustflags so different build configs get separate cache entries
|
||||
if [ -n "$EXTRA_FEATURES" ] || [ -n "$EXTRA_RUSTFLAGS" ]; then
|
||||
BUILD_SUFFIX="-$(echo "${EXTRA_FEATURES}${EXTRA_RUSTFLAGS}" | sha256sum | cut -c1-12)"
|
||||
else
|
||||
BUILD_SUFFIX=""
|
||||
fi
|
||||
|
||||
# Verify a cached reth binary was built from the expected commit.
|
||||
# `reth --version` outputs "Commit SHA: <full-sha>" on its own line.
|
||||
verify_binary() {
|
||||
@@ -58,7 +42,7 @@ verify_binary() {
|
||||
|
||||
case "$MODE" in
|
||||
baseline|main)
|
||||
BUCKET="minio/reth-binaries/${COMMIT}${BUILD_SUFFIX}"
|
||||
BUCKET="minio/reth-binaries/${COMMIT}"
|
||||
mkdir -p "${SOURCE_DIR}/target/profiling"
|
||||
|
||||
CACHE_VALID=false
|
||||
@@ -75,23 +59,14 @@ case "$MODE" in
|
||||
if [ "$CACHE_VALID" = false ]; then
|
||||
echo "Building baseline (${COMMIT}) from source..."
|
||||
cd "${SOURCE_DIR}"
|
||||
FEATURES_ARG=""
|
||||
WORKSPACE_ARG=""
|
||||
if [ -n "$EXTRA_FEATURES" ]; then
|
||||
# --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
|
||||
FEATURES_ARG="--features ${EXTRA_FEATURES}"
|
||||
WORKSPACE_ARG="--workspace"
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
|
||||
cargo build --profile profiling --bin reth $WORKSPACE_ARG $FEATURES_ARG
|
||||
cargo build --profile profiling --bin reth
|
||||
$MC cp target/profiling/reth "${BUCKET}/reth"
|
||||
fi
|
||||
;;
|
||||
|
||||
feature|branch)
|
||||
BRANCH_SHA="${4:-$COMMIT}"
|
||||
BUCKET="minio/reth-binaries/${BRANCH_SHA}${BUILD_SUFFIX}"
|
||||
BUCKET="minio/reth-binaries/${BRANCH_SHA}"
|
||||
|
||||
CACHE_VALID=false
|
||||
if $MC stat "${BUCKET}/reth" &>/dev/null && $MC stat "${BUCKET}/reth-bench" &>/dev/null; then
|
||||
@@ -110,14 +85,7 @@ case "$MODE" in
|
||||
echo "Building feature (${COMMIT}) from source..."
|
||||
cd "${SOURCE_DIR}"
|
||||
rustup show active-toolchain || rustup default stable
|
||||
if [ -n "$EXTRA_FEATURES" ]; then
|
||||
# Can't use `make profiling` when adding features; build explicitly
|
||||
# --workspace is needed for cross-package feature syntax (tracy-client/ondemand)
|
||||
RUSTFLAGS="-C target-cpu=native${EXTRA_RUSTFLAGS}" \
|
||||
cargo build --profile profiling --workspace --bin reth --features "${EXTRA_FEATURES}"
|
||||
else
|
||||
make profiling
|
||||
fi
|
||||
make profiling
|
||||
make install-reth-bench
|
||||
$MC cp target/profiling/reth "${BUCKET}/reth"
|
||||
$MC cp "$(which reth-bench)" "${BUCKET}/reth-bench"
|
||||
|
||||
581
.github/scripts/bench-reth-local.sh
vendored
581
.github/scripts/bench-reth-local.sh
vendored
@@ -1,581 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# local-reth-bench.sh — Run the reth Engine API benchmark locally.
|
||||
#
|
||||
# Replicates the CI bench.yml workflow (build, snapshot, system tuning,
|
||||
# interleaved B-F-F-B execution, summary, charts) without any GitHub
|
||||
# Actions glue (no PR comments, no artifact upload, no Slack).
|
||||
#
|
||||
# Usage:
|
||||
# local-reth-bench.sh <baseline-ref> <feature-ref> [options]
|
||||
#
|
||||
# Options:
|
||||
# --blocks N Number of blocks to benchmark (default: 500)
|
||||
# --warmup N Number of warmup blocks (default: 100)
|
||||
# --cores N Limit reth to N CPU cores, 0 = all available (default: 0)
|
||||
# --samply Enable samply profiling
|
||||
# --tracy MODE Tracy profiling: off, on, full (default: off)
|
||||
# --tracy-filter F Tracy tracing filter (default: debug)
|
||||
# --no-tune Skip system tuning (useful on dev machines / macOS)
|
||||
#
|
||||
# Requires: the reth repo at RETH_REPO (default: ~/reth)
|
||||
#
|
||||
# Dependencies (install before first run):
|
||||
# mc (MinIO client), schelk, cpupower, taskset, stdbuf, python3, curl,
|
||||
# make, uv, pzstd, jq, Rust toolchain (cargo/rustup)
|
||||
#
|
||||
# The script delegates to the existing bench-reth-*.sh scripts in the reth
|
||||
# repo for the actual build, snapshot, and run steps.
|
||||
set -euo pipefail
|
||||
|
||||
# ── PATH ──────────────────────────────────────────────────────────────
|
||||
# Ensure cargo and user-local bins (mc, uv) are visible
|
||||
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
|
||||
|
||||
# ── Defaults ──────────────────────────────────────────────────────────
|
||||
RETH_REPO="${RETH_REPO:-$HOME/reth}"
|
||||
BLOCKS=500
|
||||
WARMUP=100
|
||||
CORES=0
|
||||
SAMPLY=false
|
||||
TRACY="off"
|
||||
TRACY_FILTER="debug"
|
||||
TUNE=true
|
||||
BASELINE_REF=""
|
||||
FEATURE_REF=""
|
||||
|
||||
# ── Parse arguments ──────────────────────────────────────────────────
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "$0") <baseline-ref> <feature-ref> [options]
|
||||
|
||||
Options:
|
||||
--blocks N Number of blocks to benchmark (default: 500)
|
||||
--warmup N Number of warmup blocks (default: 100)
|
||||
--cores N Limit reth to N CPU cores (default: 0 = all)
|
||||
--samply Enable samply profiling
|
||||
--tracy MODE Tracy profiling: off, on, full (default: off)
|
||||
on = tracing only (lower overhead)
|
||||
full = tracing + CPU sampling (higher overhead)
|
||||
--tracy-filter F Tracy tracing filter (default: debug)
|
||||
--no-tune Skip system tuning
|
||||
EOF
|
||||
exit 1
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--blocks) BLOCKS="$2"; shift 2 ;;
|
||||
--warmup) WARMUP="$2"; shift 2 ;;
|
||||
--cores) CORES="$2"; shift 2 ;;
|
||||
--samply) SAMPLY=true; shift ;;
|
||||
--tracy) TRACY="$2"; shift 2 ;;
|
||||
--tracy-filter) TRACY_FILTER="$2"; shift 2 ;;
|
||||
--no-tune) TUNE=false; shift ;;
|
||||
--help|-h) usage ;;
|
||||
-*) echo "Unknown option: $1"; usage ;;
|
||||
*)
|
||||
if [ -z "$BASELINE_REF" ]; then
|
||||
BASELINE_REF="$1"
|
||||
elif [ -z "$FEATURE_REF" ]; then
|
||||
FEATURE_REF="$1"
|
||||
else
|
||||
echo "Unexpected argument: $1"; usage
|
||||
fi
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -z "$BASELINE_REF" ] || [ -z "$FEATURE_REF" ]; then
|
||||
echo "Error: both <baseline-ref> and <feature-ref> are required."
|
||||
usage
|
||||
fi
|
||||
|
||||
# Validate --tracy value
|
||||
case "$TRACY" in
|
||||
off|on|full) ;;
|
||||
*) echo "Error: --tracy must be off, on, or full (got: $TRACY)"; usage ;;
|
||||
esac
|
||||
|
||||
# Samply + tracy=full are mutually exclusive (both use perf sampling)
|
||||
if [ "$SAMPLY" = "true" ] && [ "$TRACY" = "full" ]; then
|
||||
echo "Warning: samply and tracy=full both use perf sampling; downgrading tracy to 'on'."
|
||||
TRACY="on"
|
||||
fi
|
||||
|
||||
# ── Check dependencies ───────────────────────────────────────────────
|
||||
missing=()
|
||||
for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq cargo; do
|
||||
command -v "$cmd" &>/dev/null || missing+=("$cmd")
|
||||
done
|
||||
if [ ${#missing[@]} -gt 0 ]; then
|
||||
echo "Error: missing required tools: ${missing[*]}"
|
||||
echo "See the CI 'Install dependencies' step in .github/workflows/bench.yml for install instructions."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$TRACY" != "off" ]; then
|
||||
if ! command -v tracy-capture &>/dev/null; then
|
||||
echo "Error: tracy-capture is required for --tracy $TRACY"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Ensure tools that run via sudo are in a sudo-visible path.
|
||||
# The bench scripts use `sudo schelk` / `sudo samply` but cargo installs
|
||||
# them to ~/.cargo/bin which sudo's secure_path doesn't include.
|
||||
for cmd in schelk samply; do
|
||||
if command -v "$cmd" &>/dev/null && ! sudo sh -c "command -v $cmd" &>/dev/null; then
|
||||
echo "Installing $cmd to /usr/local/bin (needed for sudo)..."
|
||||
sudo install "$(command -v "$cmd")" /usr/local/bin/
|
||||
fi
|
||||
done
|
||||
|
||||
if [ ! -d "$RETH_REPO/.git" ]; then
|
||||
echo "Error: RETH_REPO=$RETH_REPO is not a git repository."
|
||||
echo "Set RETH_REPO or clone reth to ~/reth"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Resolve paths ────────────────────────────────────────────────────
|
||||
SELF_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
SCRIPTS_DIR="${RETH_REPO}/.github/scripts"
|
||||
BENCH_WORK_DIR="${RETH_REPO}/../bench-work-$(date +%Y%m%d-%H%M%S)"
|
||||
BASELINE_SRC="${RETH_REPO}/../reth-baseline"
|
||||
FEATURE_SRC="${RETH_REPO}/../reth-feature"
|
||||
|
||||
mkdir -p "$BENCH_WORK_DIR"
|
||||
BENCH_WORK_DIR="$(cd "$BENCH_WORK_DIR" && pwd)"
|
||||
|
||||
# ── Global cleanup trap (restores system tuning on any exit) ─────────
|
||||
TUNING_APPLIED=false
|
||||
CSTATE_PID=
|
||||
METRICS_PROXY_PID=
|
||||
cleanup_global() {
|
||||
[ -n "$METRICS_PROXY_PID" ] && kill "$METRICS_PROXY_PID" 2>/dev/null || true
|
||||
if [ "$TUNING_APPLIED" = true ]; then
|
||||
echo
|
||||
echo "▸ Restoring system settings..."
|
||||
[ -n "$CSTATE_PID" ] && kill "$CSTATE_PID" 2>/dev/null || true
|
||||
sudo systemctl start irqbalance cron atd 2>/dev/null || true
|
||||
echo " System settings restored."
|
||||
fi
|
||||
}
|
||||
trap cleanup_global EXIT
|
||||
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo " reth local benchmark"
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo " Baseline ref : $BASELINE_REF"
|
||||
echo " Feature ref : $FEATURE_REF"
|
||||
echo " Blocks : $BLOCKS"
|
||||
echo " Warmup : $WARMUP"
|
||||
echo " Cores : $CORES"
|
||||
echo " Samply : $SAMPLY"
|
||||
echo " Tracy : $TRACY"
|
||||
echo " Tracy filter : $TRACY_FILTER"
|
||||
echo " System tune : $TUNE"
|
||||
echo " Work dir : $BENCH_WORK_DIR"
|
||||
echo " Reth repo : $RETH_REPO"
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo
|
||||
|
||||
# Enable sccache if available (matches CI's RUSTC_WRAPPER=sccache)
|
||||
if command -v sccache &>/dev/null; then
|
||||
export RUSTC_WRAPPER="sccache"
|
||||
fi
|
||||
|
||||
# Export env vars expected by the bench-reth-*.sh scripts
|
||||
export BENCH_BLOCKS="$BLOCKS"
|
||||
export BENCH_WARMUP_BLOCKS="$WARMUP"
|
||||
export BENCH_CORES="$CORES"
|
||||
export BENCH_SAMPLY="$SAMPLY"
|
||||
export BENCH_TRACY="$TRACY"
|
||||
export BENCH_TRACY_FILTER="$TRACY_FILTER"
|
||||
export BENCH_WORK_DIR
|
||||
export SCHELK_MOUNT="${SCHELK_MOUNT:-/reth-bench}"
|
||||
export BENCH_RPC_URL="${BENCH_RPC_URL:-https://ethereum.reth.rs/rpc}"
|
||||
export BENCH_METRICS_ADDR="127.0.0.1:9100"
|
||||
|
||||
# ── Step 1: Resolve refs to full SHAs ────────────────────────────────
|
||||
echo "▸ Resolving git refs..."
|
||||
cd "$RETH_REPO"
|
||||
|
||||
resolve_ref() {
|
||||
local ref="$1"
|
||||
git fetch origin "$ref" --quiet 2>/dev/null || true
|
||||
git rev-parse "$ref" 2>/dev/null \
|
||||
|| git rev-parse "origin/$ref" 2>/dev/null \
|
||||
|| { echo "Error: cannot resolve ref '$ref'"; exit 1; }
|
||||
}
|
||||
|
||||
BASELINE_SHA="$(resolve_ref "$BASELINE_REF")"
|
||||
FEATURE_SHA="$(resolve_ref "$FEATURE_REF")"
|
||||
echo " Baseline SHA : $BASELINE_SHA"
|
||||
echo " Feature SHA : $FEATURE_SHA"
|
||||
echo
|
||||
|
||||
# ── Step 2: Prepare source directories ───────────────────────────────
|
||||
echo "▸ Preparing source directories..."
|
||||
|
||||
prepare_source() {
|
||||
local src_dir="$1" ref="$2"
|
||||
if [ -d "$src_dir" ]; then
|
||||
git -C "$src_dir" fetch origin "$ref" 2>/dev/null || true
|
||||
else
|
||||
git clone --recurse-submodules "$RETH_REPO" "$src_dir"
|
||||
fi
|
||||
git -C "$src_dir" checkout "$ref" --force
|
||||
git -C "$src_dir" submodule update --init --recursive
|
||||
}
|
||||
|
||||
prepare_source "$BASELINE_SRC" "$BASELINE_SHA"
|
||||
prepare_source "$FEATURE_SRC" "$FEATURE_SHA"
|
||||
BASELINE_SRC="$(cd "$BASELINE_SRC" && pwd)"
|
||||
FEATURE_SRC="$(cd "$FEATURE_SRC" && pwd)"
|
||||
echo " Baseline src : $BASELINE_SRC"
|
||||
echo " Feature src : $FEATURE_SRC"
|
||||
echo
|
||||
|
||||
# ── Step 3: Check / download snapshot ────────────────────────────────
|
||||
echo "▸ Checking snapshot..."
|
||||
cd "$RETH_REPO"
|
||||
SNAPSHOT_NEEDED=false
|
||||
if ! "${SCRIPTS_DIR}/bench-reth-snapshot.sh" --check; then
|
||||
SNAPSHOT_NEEDED=true
|
||||
echo " Snapshot needs update."
|
||||
else
|
||||
echo " Snapshot is up-to-date."
|
||||
fi
|
||||
echo
|
||||
|
||||
# ── Step 4: Build binaries (+ snapshot download) in parallel ─────────
|
||||
echo "▸ Building binaries (parallel)..."
|
||||
cd "$RETH_REPO"
|
||||
|
||||
FAIL=0
|
||||
|
||||
"${SCRIPTS_DIR}/bench-reth-build.sh" baseline "$BASELINE_SRC" "$BASELINE_SHA" &
|
||||
PID_BASELINE=$!
|
||||
|
||||
"${SCRIPTS_DIR}/bench-reth-build.sh" feature "$FEATURE_SRC" "$FEATURE_SHA" &
|
||||
PID_FEATURE=$!
|
||||
|
||||
PID_SNAPSHOT=
|
||||
if [ "$SNAPSHOT_NEEDED" = "true" ]; then
|
||||
echo " Also downloading snapshot in parallel..."
|
||||
"${SCRIPTS_DIR}/bench-reth-snapshot.sh" &
|
||||
PID_SNAPSHOT=$!
|
||||
fi
|
||||
|
||||
wait $PID_BASELINE || FAIL=1
|
||||
wait $PID_FEATURE || FAIL=1
|
||||
[ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
|
||||
|
||||
if [ $FAIL -ne 0 ]; then
|
||||
echo "Error: one or more parallel tasks failed (builds / snapshot)"
|
||||
exit 1
|
||||
fi
|
||||
echo " Binaries built successfully."
|
||||
echo
|
||||
|
||||
# ── Step 5: System tuning (optional) ────────────────────────────────
|
||||
if [ "$TUNE" = "true" ]; then
|
||||
echo "▸ Applying system tuning..."
|
||||
|
||||
sudo cpupower frequency-set -g performance 2>/dev/null || true
|
||||
|
||||
# Disable turbo boost (Intel + AMD)
|
||||
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
|
||||
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
|
||||
|
||||
sudo swapoff -a 2>/dev/null || true
|
||||
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space 2>/dev/null || true
|
||||
|
||||
# Disable SMT (hyperthreading)
|
||||
for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
|
||||
[ -f "$cpu" ] || continue
|
||||
first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
|
||||
current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
|
||||
if [ "$current" != "$first" ]; then
|
||||
echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
echo " Online CPUs: $(nproc)"
|
||||
|
||||
# Disable transparent huge pages
|
||||
for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
|
||||
if [ -d "$p" ]; then
|
||||
echo never | sudo tee "$p/enabled" 2>/dev/null || true
|
||||
echo never | sudo tee "$p/defrag" 2>/dev/null || true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# Prevent deep C-states
|
||||
sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
|
||||
CSTATE_PID=$!
|
||||
|
||||
# Pin IRQs to core 0
|
||||
for irq in /proc/irq/*/smp_affinity_list; do
|
||||
echo 0 | sudo tee "$irq" 2>/dev/null || true
|
||||
done
|
||||
|
||||
# Stop noisy background services
|
||||
sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
|
||||
|
||||
TUNING_APPLIED=true
|
||||
|
||||
# Log environment for reproducibility (matches CI)
|
||||
echo " === Benchmark environment ==="
|
||||
echo " Kernel : $(uname -r)"
|
||||
lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA' | sed 's/^/ /'
|
||||
echo " Governor : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo unknown)"
|
||||
echo " Freq : $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null || echo unknown)"
|
||||
echo " THP : $(cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo unknown)"
|
||||
free -h | sed 's/^/ /'
|
||||
echo " System tuning applied."
|
||||
echo
|
||||
fi
|
||||
|
||||
# ── Step 5b: Tracefs mount (tracy=full only) ─────────────────────────
|
||||
if [ "$TRACY" = "full" ] && [ "$(uname)" = "Linux" ]; then
|
||||
echo "▸ Mounting tracefs for Tracy full mode..."
|
||||
sudo mount -t tracefs tracefs /sys/kernel/tracing -o mode=755 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ── Tracy upload & viewer helpers ────────────────────────────────────
|
||||
TRACY_VIEWER_BASE="${TRACY_VIEWER_BASE:-}"
|
||||
|
||||
tracy_viewer_url() {
|
||||
local profile_url="$1"
|
||||
if [ -z "$TRACY_VIEWER_BASE" ]; then
|
||||
echo ""
|
||||
return
|
||||
fi
|
||||
local encoded
|
||||
encoded=$(python3 -c "import urllib.parse, sys; print(urllib.parse.quote(sys.argv[1], safe=''))" "$profile_url")
|
||||
echo "${TRACY_VIEWER_BASE}?profile_url=${encoded}"
|
||||
}
|
||||
|
||||
upload_tracy() {
|
||||
local label="$1" output_dir="$2" sha="$3"
|
||||
local tracy_file="$output_dir/tracy-profile.tracy"
|
||||
|
||||
if [ ! -f "$tracy_file" ]; then
|
||||
echo " Tracy: no profile found, skipping upload."
|
||||
return
|
||||
fi
|
||||
|
||||
local timestamp short_sha remote_name bucket mc_alias
|
||||
timestamp=$(date +%Y%m%d-%H%M%S)
|
||||
short_sha="${sha:0:7}"
|
||||
remote_name="${label}-${short_sha}-${timestamp}.tracy"
|
||||
bucket="${TRACY_BUCKET:-tracy-profiles}"
|
||||
mc_alias="${MC_ALIAS:-minio}"
|
||||
local minio_base="${TRACY_MINIO_URL:-http://minio.minio.svc.cluster.local:9000}"
|
||||
|
||||
echo " Tracy: uploading profile..."
|
||||
if mc cp "$tracy_file" "${mc_alias}/${bucket}/${remote_name}"; then
|
||||
local url="${minio_base}/${bucket}/${remote_name}"
|
||||
echo "$url" > "$output_dir/tracy_url.txt"
|
||||
local viewer
|
||||
viewer=$(tracy_viewer_url "$url")
|
||||
if [ -n "$viewer" ]; then
|
||||
echo "$viewer" > "$output_dir/tracy_viewer_url.txt"
|
||||
echo " Tracy: uploaded → $viewer"
|
||||
else
|
||||
echo " Tracy: uploaded → $url"
|
||||
fi
|
||||
else
|
||||
echo " Tracy: upload failed (non-fatal)."
|
||||
fi
|
||||
|
||||
# Delete large profile to free disk
|
||||
rm -f "$tracy_file"
|
||||
}
|
||||
|
||||
# ── Step 6: Pre-flight cleanup ───────────────────────────────────────
|
||||
echo "▸ Pre-flight cleanup..."
|
||||
pkill -f bench-metrics-proxy 2>/dev/null || true
|
||||
sudo pkill -9 reth 2>/dev/null || true
|
||||
sleep 1
|
||||
if mountpoint -q "$SCHELK_MOUNT" 2>/dev/null; then
|
||||
sudo umount -l "$SCHELK_MOUNT" 2>/dev/null || true
|
||||
sudo schelk recover -y 2>/dev/null || true
|
||||
fi
|
||||
echo
|
||||
|
||||
# ── Step 7: Interleaved benchmark runs (B-F-F-B) ────────────────────
|
||||
# This ordering reduces systematic bias from thermal drift and cache warming.
|
||||
BASELINE_BIN="${BASELINE_SRC}/target/profiling/reth"
|
||||
FEATURE_BIN="${FEATURE_SRC}/target/profiling/reth"
|
||||
|
||||
# Start metrics proxy (reth → label injection → Prometheus)
|
||||
LABELS_FILE="/tmp/bench-metrics-labels.json"
|
||||
echo '{}' > "$LABELS_FILE"
|
||||
METRICS_SUBNET="${METRICS_SUBNET:-10.10.0.0/24}"
|
||||
METRICS_PORT="${METRICS_PORT:-9090}"
|
||||
python3 "${SELF_DIR}/bench-metrics-proxy.py" \
|
||||
--labels "$LABELS_FILE" \
|
||||
--upstream "http://${BENCH_METRICS_ADDR}/" \
|
||||
--subnet "$METRICS_SUBNET" \
|
||||
--port "$METRICS_PORT" &
|
||||
METRICS_PROXY_PID=$!
|
||||
echo "▸ Metrics proxy started (PID $METRICS_PROXY_PID) on subnet ${METRICS_SUBNET}, port ${METRICS_PORT}"
|
||||
|
||||
# Unique benchmark ID: local-<timestamp> for local runs, ci-<run_id> for CI
|
||||
BENCH_ID="local-$(basename "$BENCH_WORK_DIR" | sed 's/bench-work-//')"
|
||||
# Reference epoch: shared time origin so all runs overlay in Grafana.
|
||||
# The proxy maps each run's elapsed time onto this common origin.
|
||||
BENCH_REFERENCE_EPOCH=$(date +%s)
|
||||
|
||||
write_labels() {
|
||||
local run_label="$1" run_type="$2" ref="$3" sha="$4"
|
||||
LAST_RUN_START=$(date +%s)
|
||||
cat > "$LABELS_FILE" <<-EOF
|
||||
{"benchmark_run":"${run_label}","run_type":"${run_type}","git_ref":"${ref}","bench_sha":"${sha}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
EOF
|
||||
}
|
||||
|
||||
run_bench() {
|
||||
local label="$1" binary="$2" output_dir="$3"
|
||||
echo "▸ Running benchmark: ${label}..."
|
||||
cd "$RETH_REPO"
|
||||
if command -v taskset &>/dev/null; then
|
||||
taskset -c 0 "${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
|
||||
else
|
||||
"${SCRIPTS_DIR}/bench-reth-run.sh" "$label" "$binary" "$output_dir"
|
||||
fi
|
||||
echo " ✓ ${label} complete."
|
||||
echo
|
||||
}
|
||||
|
||||
write_labels "baseline-1" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
|
||||
run_bench "baseline-1" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-1"
|
||||
|
||||
write_labels "feature-1" "feature" "$FEATURE_REF" "$FEATURE_SHA"
|
||||
run_bench "feature-1" "$FEATURE_BIN" "$BENCH_WORK_DIR/feature-1"
|
||||
|
||||
write_labels "feature-2" "feature" "$FEATURE_REF" "$FEATURE_SHA"
|
||||
run_bench "feature-2" "$FEATURE_BIN" "$BENCH_WORK_DIR/feature-2"
|
||||
|
||||
write_labels "baseline-2" "baseline" "$BASELINE_REF" "$BASELINE_SHA"
|
||||
run_bench "baseline-2" "$BASELINE_BIN" "$BENCH_WORK_DIR/baseline-2"
|
||||
|
||||
# ── Compute Grafana URL ──────────────────────────────────────────────
|
||||
GRAFANA_BASE_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr"
|
||||
GRAFANA_DATASOURCE="ef57fux92e9z4e"
|
||||
LAST_RUN_DURATION=$(( $(date +%s) - LAST_RUN_START ))
|
||||
FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
|
||||
TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
|
||||
GRAFANA_URL="${GRAFANA_BASE_URL}?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=${GRAFANA_DATASOURCE}&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
|
||||
|
||||
# ── Step 8: Scan logs for errors ─────────────────────────────────────
|
||||
echo "▸ Scanning logs for errors..."
|
||||
ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
|
||||
found_errors=false
|
||||
for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
|
||||
LOG="$BENCH_WORK_DIR/$run_dir/node.log"
|
||||
[ -f "$LOG" ] || continue
|
||||
panics=$(grep -c -E 'panicked at' "$LOG" 2>/dev/null || true)
|
||||
errors=$(grep -c ' ERROR ' "$LOG" 2>/dev/null || true)
|
||||
if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
|
||||
if [ "$found_errors" = false ]; then
|
||||
printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
|
||||
found_errors=true
|
||||
fi
|
||||
printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' \
|
||||
"$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
|
||||
if [ "$panics" -gt 0 ]; then
|
||||
printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
|
||||
grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
|
||||
printf '```\n' >> "$ERRORS_FILE"
|
||||
fi
|
||||
if [ "$errors" -gt 0 ]; then
|
||||
printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
|
||||
grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
|
||||
printf '```\n' >> "$ERRORS_FILE"
|
||||
fi
|
||||
printf '\n</details>\n\n' >> "$ERRORS_FILE"
|
||||
fi
|
||||
done
|
||||
if [ "$found_errors" = true ]; then
|
||||
echo " ⚠ Errors found — see $ERRORS_FILE"
|
||||
else
|
||||
echo " No errors found."
|
||||
fi
|
||||
echo
|
||||
|
||||
# ── Step 9: Parse results ───────────────────────────────────────────
|
||||
echo "▸ Parsing results..."
|
||||
cd "$RETH_REPO"
|
||||
|
||||
SUMMARY_ARGS=(
|
||||
--output-summary "$BENCH_WORK_DIR/summary.json"
|
||||
--output-markdown "$BENCH_WORK_DIR/comment.md"
|
||||
--repo "paradigmxyz/reth"
|
||||
--baseline-ref "$BASELINE_SHA"
|
||||
--baseline-name "$BASELINE_REF"
|
||||
--feature-name "$FEATURE_REF"
|
||||
--feature-ref "$FEATURE_SHA"
|
||||
--baseline-csv "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
--feature-csv "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
--gas-csv "$BENCH_WORK_DIR/feature-1/total_gas.csv"
|
||||
--grafana-url "$GRAFANA_URL"
|
||||
)
|
||||
|
||||
python3 "${SCRIPTS_DIR}/bench-reth-summary.py" "${SUMMARY_ARGS[@]}"
|
||||
echo
|
||||
|
||||
# ── Step 10: Generate charts ─────────────────────────────────────────
|
||||
echo "▸ Generating charts..."
|
||||
CHART_ARGS=(
|
||||
--output-dir "$BENCH_WORK_DIR/charts"
|
||||
--feature "$BENCH_WORK_DIR/feature-1/combined_latency.csv" "$BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
--baseline "$BENCH_WORK_DIR/baseline-1/combined_latency.csv" "$BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
--baseline-name "$BASELINE_REF"
|
||||
--feature-name "$FEATURE_REF"
|
||||
)
|
||||
|
||||
if python3 -c "import matplotlib" 2>/dev/null; then
|
||||
python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
|
||||
elif command -v uv &>/dev/null; then
|
||||
uv run --with matplotlib python3 "${SCRIPTS_DIR}/bench-reth-charts.py" "${CHART_ARGS[@]}"
|
||||
else
|
||||
echo " Warning: matplotlib not available, skipping chart generation."
|
||||
fi
|
||||
echo
|
||||
|
||||
# ── Step 11: Upload Tracy profiles ────────────────────────────────────
|
||||
if [ "$TRACY" != "off" ]; then
|
||||
echo "▸ Uploading Tracy profiles..."
|
||||
upload_tracy "baseline-1" "$BENCH_WORK_DIR/baseline-1" "$BASELINE_SHA"
|
||||
upload_tracy "feature-1" "$BENCH_WORK_DIR/feature-1" "$FEATURE_SHA"
|
||||
upload_tracy "feature-2" "$BENCH_WORK_DIR/feature-2" "$FEATURE_SHA"
|
||||
upload_tracy "baseline-2" "$BENCH_WORK_DIR/baseline-2" "$BASELINE_SHA"
|
||||
echo
|
||||
fi
|
||||
|
||||
# ── Done (system restore happens via EXIT trap) ─────────────────────
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo " Benchmark complete!"
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
echo " Results : $BENCH_WORK_DIR/summary.json"
|
||||
echo " Markdown : $BENCH_WORK_DIR/comment.md"
|
||||
echo " Charts : $BENCH_WORK_DIR/charts/"
|
||||
if [ -f "$ERRORS_FILE" ]; then
|
||||
echo " Errors : $ERRORS_FILE"
|
||||
fi
|
||||
echo " Grafana : $GRAFANA_URL"
|
||||
if [ "$TRACY" != "off" ]; then
|
||||
echo " ─── Tracy Profiles ───"
|
||||
for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
|
||||
url_file="$BENCH_WORK_DIR/$run_dir/tracy_viewer_url.txt"
|
||||
if [ -f "$url_file" ]; then
|
||||
echo " $run_dir : $(cat "$url_file")"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
echo "═══════════════════════════════════════════════════════════"
|
||||
163
.github/scripts/bench-reth-run.sh
vendored
163
.github/scripts/bench-reth-run.sh
vendored
@@ -6,13 +6,6 @@
|
||||
# Usage: bench-reth-run.sh <label> <binary> <output-dir>
|
||||
#
|
||||
# Required env: SCHELK_MOUNT, BENCH_RPC_URL, BENCH_BLOCKS, BENCH_WARMUP_BLOCKS
|
||||
# Optional env: BENCH_BIG_BLOCKS (true/false), BENCH_WORK_DIR (for big blocks path)
|
||||
# BENCH_RETH_NEW_PAYLOAD (true/false, default true)
|
||||
# BENCH_WAIT_TIME (duration like 500ms, default empty)
|
||||
# BENCH_BASELINE_ARGS (extra reth node args for baseline runs)
|
||||
# BENCH_FEATURE_ARGS (extra reth node args for feature runs)
|
||||
# BENCH_OTLP_TRACES_ENDPOINT (OTLP HTTP endpoint for traces, e.g. https://host/insert/opentelemetry/v1/traces)
|
||||
# BENCH_OTLP_LOGS_ENDPOINT (OTLP HTTP endpoint for logs, e.g. https://host/insert/opentelemetry/v1/logs)
|
||||
set -euo pipefail
|
||||
|
||||
LABEL="$1"
|
||||
@@ -24,24 +17,6 @@ LOG="${OUTPUT_DIR}/node.log"
|
||||
|
||||
cleanup() {
|
||||
kill "$TAIL_PID" 2>/dev/null || true
|
||||
# Stop tracy-capture first (SIGINT makes it disconnect and flush to disk)
|
||||
# Must happen before killing reth, otherwise reth keeps streaming data.
|
||||
if [ -n "${TRACY_PID:-}" ] && kill -0 "$TRACY_PID" 2>/dev/null; then
|
||||
echo "Stopping tracy-capture..."
|
||||
kill -INT "$TRACY_PID" 2>/dev/null || true
|
||||
for i in $(seq 1 30); do
|
||||
kill -0 "$TRACY_PID" 2>/dev/null || break
|
||||
if [ $((i % 10)) -eq 0 ]; then
|
||||
echo "Waiting for tracy-capture to finish writing... (${i}s)"
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
if kill -0 "$TRACY_PID" 2>/dev/null; then
|
||||
echo "tracy-capture still running after 30s, killing..."
|
||||
kill -9 "$TRACY_PID" 2>/dev/null || true
|
||||
fi
|
||||
wait "$TRACY_PID" 2>/dev/null || true
|
||||
fi
|
||||
if [ -n "${RETH_PID:-}" ] && sudo kill -0 "$RETH_PID" 2>/dev/null; then
|
||||
if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
|
||||
# Send SIGINT to the inner reth process by exact name (not -f which
|
||||
@@ -78,14 +53,8 @@ cleanup() {
|
||||
fi
|
||||
}
|
||||
TAIL_PID=
|
||||
TRACY_PID=
|
||||
trap cleanup EXIT
|
||||
|
||||
# Clean up stale schelk state from a previous cancelled run.
|
||||
# If schelk thinks it's still mounted (e.g. a cancelled run skipped cleanup),
|
||||
# recover first to reset state.
|
||||
sudo schelk recover -y -k || true
|
||||
|
||||
# Mount
|
||||
sudo schelk mount -y
|
||||
sync
|
||||
@@ -104,8 +73,6 @@ if [ "${BENCH_CORES:-0}" -gt 0 ] && [ "$BENCH_CORES" -lt "$MAX_RETH" ]; then
|
||||
fi
|
||||
RETH_CPUS="1-${MAX_RETH}"
|
||||
|
||||
BIG_BLOCKS="${BENCH_BIG_BLOCKS:-false}"
|
||||
|
||||
RETH_ARGS=(
|
||||
node
|
||||
--datadir "$DATADIR"
|
||||
@@ -120,68 +87,16 @@ RETH_ARGS=(
|
||||
--no-persist-peers
|
||||
)
|
||||
|
||||
# Big blocks mode requires the testing API and skip-invalid-transactions
|
||||
if [ "$BIG_BLOCKS" = "true" ]; then
|
||||
RETH_ARGS+=(--http.api eth,net,web3,reth,testing --testing.skip-invalid-transactions)
|
||||
fi
|
||||
|
||||
# Append per-label extra node args (baseline or feature)
|
||||
EXTRA_NODE_ARGS=""
|
||||
case "$LABEL" in
|
||||
baseline*) EXTRA_NODE_ARGS="${BENCH_BASELINE_ARGS:-}" ;;
|
||||
feature*) EXTRA_NODE_ARGS="${BENCH_FEATURE_ARGS:-}" ;;
|
||||
esac
|
||||
if [ -n "$EXTRA_NODE_ARGS" ]; then
|
||||
# Word-split the string into individual args
|
||||
# shellcheck disable=SC2206
|
||||
RETH_ARGS+=($EXTRA_NODE_ARGS)
|
||||
fi
|
||||
|
||||
if [ -n "${BENCH_METRICS_ADDR:-}" ]; then
|
||||
RETH_ARGS+=(--metrics "$BENCH_METRICS_ADDR")
|
||||
fi
|
||||
|
||||
# OTLP traces and logs export
|
||||
if [ -n "${BENCH_OTLP_TRACES_ENDPOINT:-}" ]; then
|
||||
RETH_ARGS+=(--tracing-otlp="${BENCH_OTLP_TRACES_ENDPOINT}" --tracing-otlp.service-name=reth-bench)
|
||||
fi
|
||||
if [ -n "${BENCH_OTLP_LOGS_ENDPOINT:-}" ]; then
|
||||
RETH_ARGS+=(--logs-otlp="${BENCH_OTLP_LOGS_ENDPOINT}" --logs-otlp.filter=debug)
|
||||
fi
|
||||
|
||||
# Tracy profiling: add --log.tracy flags and set environment
|
||||
if [ "${BENCH_TRACY:-off}" != "off" ]; then
|
||||
RETH_ARGS+=(--log.tracy --log.tracy.filter "${BENCH_TRACY_FILTER:-debug}")
|
||||
if [ "${BENCH_TRACY}" = "on" ]; then
|
||||
export TRACY_NO_SYS_TRACE=1
|
||||
elif [ "${BENCH_TRACY}" = "full" ]; then
|
||||
export TRACY_SAMPLING_HZ="${BENCH_TRACY_SAMPLING_HZ:-1}"
|
||||
fi
|
||||
fi
|
||||
|
||||
SUDO_ENV=()
|
||||
if [ -n "${OTEL_RESOURCE_ATTRIBUTES:-}" ]; then
|
||||
SUDO_ENV+=("OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES}")
|
||||
SUDO_ENV+=("OTEL_BSP_MAX_QUEUE_SIZE=65536" "OTEL_BLRP_MAX_QUEUE_SIZE=65536")
|
||||
fi
|
||||
|
||||
# Limit reth memory to 95% of available RAM to prevent OOM kills
|
||||
TOTAL_MEM_KB=$(awk '/^MemTotal:/ {print $2}' /proc/meminfo)
|
||||
MEM_LIMIT=$(( TOTAL_MEM_KB * 95 / 100 * 1024 ))
|
||||
echo "Memory limit: $(( MEM_LIMIT / 1024 / 1024 ))MB (95% of $(( TOTAL_MEM_KB / 1024 ))MB)"
|
||||
|
||||
if [ "${BENCH_SAMPLY:-false}" = "true" ]; then
|
||||
RETH_ARGS+=(--log.samply)
|
||||
SAMPLY="$(which samply)"
|
||||
sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
|
||||
env "${SUDO_ENV[@]}" nice -n -20 \
|
||||
sudo taskset -c "$RETH_CPUS" nice -n -20 \
|
||||
"$SAMPLY" record --save-only --presymbolicate --rate 10000 \
|
||||
--output "$OUTPUT_DIR/samply-profile.json.gz" \
|
||||
-- "$BINARY" "${RETH_ARGS[@]}" \
|
||||
> "$LOG" 2>&1 &
|
||||
else
|
||||
sudo systemd-run --scope -p MemoryMax="$MEM_LIMIT" -p AllowedCPUs="$RETH_CPUS" \
|
||||
env "${SUDO_ENV[@]}" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
|
||||
sudo taskset -c "$RETH_CPUS" nice -n -20 "$BINARY" "${RETH_ARGS[@]}" \
|
||||
> "$LOG" 2>&1 &
|
||||
fi
|
||||
|
||||
@@ -209,65 +124,21 @@ done
|
||||
# files are not root-owned (avoids EACCES on next checkout).
|
||||
BENCH_NICE="sudo nice -n -20 sudo -u $(id -un)"
|
||||
|
||||
# Build optional flags
|
||||
EXTRA_BENCH_ARGS=()
|
||||
if [ "${BENCH_RETH_NEW_PAYLOAD:-true}" != "false" ]; then
|
||||
EXTRA_BENCH_ARGS+=(--reth-new-payload)
|
||||
fi
|
||||
if [ -n "${BENCH_WAIT_TIME:-}" ]; then
|
||||
EXTRA_BENCH_ARGS+=(--wait-time "$BENCH_WAIT_TIME")
|
||||
fi
|
||||
# Warmup
|
||||
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
|
||||
--rpc-url "$BENCH_RPC_URL" \
|
||||
--engine-rpc-url http://127.0.0.1:8551 \
|
||||
--jwt-secret "$DATADIR/jwt.hex" \
|
||||
--advance "${BENCH_WARMUP_BLOCKS:-50}" \
|
||||
--reth-new-payload 2>&1 | sed -u "s/^/[bench] /"
|
||||
|
||||
if [ "$BIG_BLOCKS" = "true" ]; then
|
||||
# Big blocks mode: replay pre-generated payloads with gas ramp
|
||||
BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
|
||||
# Count gas ramp blocks for reporting
|
||||
GAS_RAMP_COUNT=$(find "$BIG_BLOCKS_DIR/gas-ramp-dir" -name '*.json' | wc -l)
|
||||
echo "$GAS_RAMP_COUNT" > "$OUTPUT_DIR/gas_ramp_blocks.txt"
|
||||
echo "Gas ramp blocks: $GAS_RAMP_COUNT"
|
||||
|
||||
# Start tracy-capture so profile only covers the benchmark
|
||||
if [ "${BENCH_TRACY:-off}" != "off" ]; then
|
||||
echo "Starting tracy-capture..."
|
||||
tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
|
||||
TRACY_PID=$!
|
||||
sleep 0.5 # give tracy-capture time to connect
|
||||
fi
|
||||
|
||||
echo "Running big blocks benchmark (replay-payloads)..."
|
||||
$BENCH_NICE "$RETH_BENCH" replay-payloads \
|
||||
"${EXTRA_BENCH_ARGS[@]}" \
|
||||
--gas-ramp-dir "$BIG_BLOCKS_DIR/gas-ramp-dir" \
|
||||
--payload-dir "$BIG_BLOCKS_DIR/payloads" \
|
||||
--engine-rpc-url http://127.0.0.1:8551 \
|
||||
--jwt-secret "$DATADIR/jwt.hex" \
|
||||
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
|
||||
else
|
||||
# Standard mode: warmup + new-payload-fcu
|
||||
# Warmup
|
||||
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
|
||||
--rpc-url "$BENCH_RPC_URL" \
|
||||
--engine-rpc-url http://127.0.0.1:8551 \
|
||||
--jwt-secret "$DATADIR/jwt.hex" \
|
||||
--advance "${BENCH_WARMUP_BLOCKS:-50}" \
|
||||
"${EXTRA_BENCH_ARGS[@]}" 2>&1 | sed -u "s/^/[bench] /"
|
||||
|
||||
# Start tracy-capture after warmup so profile only covers the benchmark
|
||||
if [ "${BENCH_TRACY:-off}" != "off" ]; then
|
||||
echo "Starting tracy-capture..."
|
||||
tracy-capture -f -o "$OUTPUT_DIR/tracy-profile.tracy" &
|
||||
TRACY_PID=$!
|
||||
sleep 0.5 # give tracy-capture time to connect
|
||||
fi
|
||||
|
||||
# Benchmark
|
||||
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
|
||||
--rpc-url "$BENCH_RPC_URL" \
|
||||
--engine-rpc-url http://127.0.0.1:8551 \
|
||||
--jwt-secret "$DATADIR/jwt.hex" \
|
||||
--advance "$BENCH_BLOCKS" \
|
||||
"${EXTRA_BENCH_ARGS[@]}" \
|
||||
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
|
||||
fi
|
||||
# Benchmark
|
||||
$BENCH_NICE "$RETH_BENCH" new-payload-fcu \
|
||||
--rpc-url "$BENCH_RPC_URL" \
|
||||
--engine-rpc-url http://127.0.0.1:8551 \
|
||||
--jwt-secret "$DATADIR/jwt.hex" \
|
||||
--advance "$BENCH_BLOCKS" \
|
||||
--reth-new-payload \
|
||||
--output "$OUTPUT_DIR" 2>&1 | sed -u "s/^/[bench] /"
|
||||
|
||||
# cleanup runs via trap
|
||||
|
||||
47
.github/scripts/bench-reth-summary.py
vendored
47
.github/scripts/bench-reth-summary.py
vendored
@@ -107,10 +107,6 @@ def compute_stats(combined: list[dict]) -> dict:
|
||||
mgas_s_values.append(r["gas_used"] / lat_s / 1_000_000)
|
||||
mean_mgas_s = sum(mgas_s_values) / len(mgas_s_values) if mgas_s_values else 0
|
||||
|
||||
total_latencies_ms = [r["total_latency_us"] / 1_000 for r in combined]
|
||||
wall_clock_s = sum(total_latencies_ms) / 1_000
|
||||
mean_total_lat_ms = sum(total_latencies_ms) / n
|
||||
|
||||
return {
|
||||
"n": n,
|
||||
"mean_ms": mean_lat,
|
||||
@@ -119,8 +115,6 @@ def compute_stats(combined: list[dict]) -> dict:
|
||||
"p90_ms": percentile(sorted_lat, 90),
|
||||
"p99_ms": percentile(sorted_lat, 99),
|
||||
"mean_mgas_s": mean_mgas_s,
|
||||
"wall_clock_s": wall_clock_s,
|
||||
"mean_total_lat_ms": mean_total_lat_ms,
|
||||
}
|
||||
|
||||
|
||||
@@ -145,14 +139,13 @@ def compute_wait_stats(combined: list[dict], field: str) -> dict:
|
||||
|
||||
def _paired_data(
|
||||
baseline: list[dict], feature: list[dict]
|
||||
) -> tuple[list[tuple[float, float]], list[float], list[float], list[float]]:
|
||||
) -> tuple[list[tuple[float, float]], list[float], list[float]]:
|
||||
"""Match blocks and return paired latencies and per-block diffs.
|
||||
|
||||
Returns:
|
||||
pairs: list of (baseline_ms, feature_ms) tuples
|
||||
lat_diffs_ms: list of feature − baseline latency diffs in ms
|
||||
mgas_diffs: list of feature − baseline Mgas/s diffs
|
||||
total_lat_diffs_ms: list of feature − baseline total latency diffs in ms
|
||||
"""
|
||||
baseline_by_block = {r["block_number"]: r for r in baseline}
|
||||
feature_by_block = {r["block_number"]: r for r in feature}
|
||||
@@ -161,7 +154,6 @@ def _paired_data(
|
||||
pairs = []
|
||||
lat_diffs_ms = []
|
||||
mgas_diffs = []
|
||||
total_lat_diffs_ms = []
|
||||
for bn in common_blocks:
|
||||
b = baseline_by_block[bn]
|
||||
f = feature_by_block[bn]
|
||||
@@ -176,10 +168,7 @@ def _paired_data(
|
||||
f["gas_used"] / f_lat_s / 1_000_000
|
||||
- b["gas_used"] / b_lat_s / 1_000_000
|
||||
)
|
||||
total_lat_diffs_ms.append(
|
||||
f["total_latency_us"] / 1_000 - b["total_latency_us"] / 1_000
|
||||
)
|
||||
return pairs, lat_diffs_ms, mgas_diffs, total_lat_diffs_ms
|
||||
return pairs, lat_diffs_ms, mgas_diffs
|
||||
|
||||
|
||||
def compute_paired_stats(
|
||||
@@ -194,14 +183,12 @@ def compute_paired_stats(
|
||||
all_pairs = []
|
||||
all_lat_diffs = []
|
||||
all_mgas_diffs = []
|
||||
all_total_lat_diffs = []
|
||||
blocks_per_pair = []
|
||||
for baseline, feature in zip(baseline_runs, feature_runs):
|
||||
pairs, lat_diffs, mgas_diffs, total_lat_diffs = _paired_data(baseline, feature)
|
||||
pairs, lat_diffs, mgas_diffs = _paired_data(baseline, feature)
|
||||
all_pairs.extend(pairs)
|
||||
all_lat_diffs.extend(lat_diffs)
|
||||
all_mgas_diffs.extend(mgas_diffs)
|
||||
all_total_lat_diffs.extend(total_lat_diffs)
|
||||
blocks_per_pair.append(len(pairs))
|
||||
|
||||
if not all_lat_diffs:
|
||||
@@ -240,11 +227,6 @@ def compute_paired_stats(
|
||||
mgas_se = std_mgas_diff / math.sqrt(len(all_mgas_diffs)) if all_mgas_diffs else 0.0
|
||||
mgas_ci = T_CRITICAL * mgas_se
|
||||
|
||||
mean_total_diff = sum(all_total_lat_diffs) / len(all_total_lat_diffs) if all_total_lat_diffs else 0.0
|
||||
std_total_diff = stddev(all_total_lat_diffs, mean_total_diff) if len(all_total_lat_diffs) > 1 else 0.0
|
||||
total_se = std_total_diff / math.sqrt(len(all_total_lat_diffs)) if all_total_lat_diffs else 0.0
|
||||
wall_clock_ci_ms = T_CRITICAL * total_se
|
||||
|
||||
return {
|
||||
"n": n,
|
||||
"mean_diff_ms": mean_diff,
|
||||
@@ -257,7 +239,6 @@ def compute_paired_stats(
|
||||
"p99_ci_ms": (p99_boot[hi] - p99_boot[lo]) / 2,
|
||||
"mean_mgas_diff": mean_mgas_diff,
|
||||
"mgas_ci": mgas_ci,
|
||||
"wall_clock_ci_ms": wall_clock_ci_ms,
|
||||
"blocks": max(blocks_per_pair),
|
||||
}
|
||||
|
||||
@@ -286,10 +267,6 @@ def fmt_mgas(v: float) -> str:
|
||||
return f"{v:.2f}"
|
||||
|
||||
|
||||
def fmt_s(v: float) -> str:
|
||||
return f"{v:.2f}s"
|
||||
|
||||
|
||||
def significance(pct: float, ci_pct: float, lower_is_better: bool) -> str:
|
||||
"""Return significance label: 'good', 'bad', or 'neutral'."""
|
||||
significant = abs(pct) > ci_pct
|
||||
@@ -327,7 +304,6 @@ def compute_changes(
|
||||
("p90", "p90_ms", "p90_ci_ms", "p90_ms", True),
|
||||
("p99", "p99_ms", "p99_ci_ms", "p99_ms", True),
|
||||
("mgas_s", "mean_mgas_s", "mgas_ci", "mean_mgas_s", False),
|
||||
("wall_clock", "wall_clock_s", "wall_clock_ci_ms", "mean_total_lat_ms", True),
|
||||
]
|
||||
changes = {}
|
||||
for name, stat_key, ci_key, base_key, lower_is_better in metrics:
|
||||
@@ -350,7 +326,6 @@ def generate_comparison_table(
|
||||
baseline_name: str,
|
||||
feature_name: str,
|
||||
feature_sha: str,
|
||||
big_blocks: bool = False,
|
||||
) -> str:
|
||||
"""Generate a markdown comparison table between baseline and feature."""
|
||||
n = paired["blocks"]
|
||||
@@ -360,7 +335,6 @@ def generate_comparison_table(
|
||||
|
||||
mean_pct = pct(run1["mean_ms"], run2["mean_ms"])
|
||||
gas_pct = pct(run1["mean_mgas_s"], run2["mean_mgas_s"])
|
||||
wall_pct = pct(run1["wall_clock_s"], run2["wall_clock_s"])
|
||||
|
||||
p50_pct = pct(run1["p50_ms"], run2["p50_ms"])
|
||||
p90_pct = pct(run1["p90_ms"], run2["p90_ms"])
|
||||
@@ -374,7 +348,6 @@ def generate_comparison_table(
|
||||
# CI as a percentage of baseline mean
|
||||
lat_ci_pct = paired["ci_ms"] / run1["mean_ms"] * 100.0 if run1["mean_ms"] > 0 else 0.0
|
||||
mgas_ci_pct = paired["mgas_ci"] / run1["mean_mgas_s"] * 100.0 if run1["mean_mgas_s"] > 0 else 0.0
|
||||
wall_ci_pct = paired["wall_clock_ci_ms"] / run1["mean_total_lat_ms"] * 100.0 if run1["mean_total_lat_ms"] > 0 else 0.0
|
||||
|
||||
base_url = f"https://github.com/{repo}/commit"
|
||||
baseline_label = f"[`{baseline_name}`]({base_url}/{baseline_ref})"
|
||||
@@ -389,9 +362,8 @@ def generate_comparison_table(
|
||||
f"| P90 | {fmt_ms(run1['p90_ms'])} | {fmt_ms(run2['p90_ms'])} | {change_str(p90_pct, p90_ci_pct, lower_is_better=True)} |",
|
||||
f"| P99 | {fmt_ms(run1['p99_ms'])} | {fmt_ms(run2['p99_ms'])} | {change_str(p99_pct, p99_ci_pct, lower_is_better=True)} |",
|
||||
f"| Mgas/s | {fmt_mgas(run1['mean_mgas_s'])} | {fmt_mgas(run2['mean_mgas_s'])} | {change_str(gas_pct, mgas_ci_pct, lower_is_better=False)} |",
|
||||
f"| Wall Clock | {fmt_s(run1['wall_clock_s'])} | {fmt_s(run2['wall_clock_s'])} | {change_str(wall_pct, wall_ci_pct, lower_is_better=True)} |",
|
||||
"",
|
||||
f"*{n} {'big blocks' if big_blocks else 'blocks'}*",
|
||||
f"*{n} blocks*",
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
@@ -422,7 +394,6 @@ def generate_markdown(
|
||||
summary: dict, comparison_table: str,
|
||||
wait_time_tables: list[str] | None = None,
|
||||
behind_baseline: int = 0, repo: str = "", baseline_ref: str = "", baseline_name: str = "",
|
||||
grafana_url: str | None = None,
|
||||
) -> str:
|
||||
"""Generate a markdown comment body."""
|
||||
lines = ["## Benchmark Results", ""]
|
||||
@@ -442,9 +413,6 @@ def generate_markdown(
|
||||
lines.append(table)
|
||||
lines.append("")
|
||||
lines.append("</details>")
|
||||
if grafana_url:
|
||||
lines.append("")
|
||||
lines.append(f"**[Grafana Dashboard]({grafana_url})**")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
@@ -471,9 +439,6 @@ def main():
|
||||
parser.add_argument("--feature-name", "--branch-name", default=None, help="Feature branch name")
|
||||
parser.add_argument("--feature-ref", "--branch-sha", "--feature-sha", default=None, help="Feature commit SHA")
|
||||
parser.add_argument("--behind-baseline", "--behind-main", type=int, default=0, help="Commits behind baseline")
|
||||
parser.add_argument("--big-blocks", action="store_true", default=False, help="Big blocks mode")
|
||||
parser.add_argument("--gas-ramp-blocks", type=int, default=0, help="Number of gas ramp blocks (big blocks mode)")
|
||||
parser.add_argument("--grafana-url", default=None, help="Grafana dashboard URL for this benchmark run")
|
||||
args = parser.parse_args()
|
||||
|
||||
if len(args.baseline_csv) != len(args.feature_csv):
|
||||
@@ -522,7 +487,6 @@ def main():
|
||||
baseline_name=baseline_name,
|
||||
feature_name=feature_name,
|
||||
feature_sha=feature_sha,
|
||||
big_blocks=args.big_blocks,
|
||||
)
|
||||
print(f"Generated comparison ({paired_stats['n']} paired blocks, "
|
||||
f"mean diff {paired_stats['mean_diff_ms']:+.3f}ms ± {paired_stats['ci_ms']:.3f}ms)")
|
||||
@@ -553,8 +517,6 @@ def main():
|
||||
|
||||
summary = {
|
||||
"blocks": paired_stats["blocks"],
|
||||
"big_blocks": args.big_blocks,
|
||||
"gas_ramp_blocks": args.gas_ramp_blocks,
|
||||
"baseline": {
|
||||
"name": baseline_name,
|
||||
"ref": baseline_ref,
|
||||
@@ -580,7 +542,6 @@ def main():
|
||||
repo=args.repo,
|
||||
baseline_ref=baseline_ref,
|
||||
baseline_name=baseline_name,
|
||||
grafana_url=args.grafana_url,
|
||||
)
|
||||
|
||||
with open(args.output_markdown, "w") as f:
|
||||
|
||||
132
.github/scripts/bench-scheduled-refs.sh
vendored
132
.github/scripts/bench-scheduled-refs.sh
vendored
@@ -1,132 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Resolves baseline and feature refs for nightly regression benchmark runs.
|
||||
#
|
||||
# Queries the latest successful scheduled docker.yml run via GitHub API
|
||||
# to find the commit that built the nightly Docker image. Compares with
|
||||
# the last successful feature ref (from GH Actions cache) to determine
|
||||
# baseline, detect staleness, and decide whether to skip.
|
||||
#
|
||||
# Usage: bench-nightly-refs.sh [--force]
|
||||
#
|
||||
# Outputs (via GITHUB_OUTPUT):
|
||||
# baseline-ref — commit SHA for baseline
|
||||
# feature-ref — commit SHA for feature (current nightly)
|
||||
# should-skip — "true" if no new nightly since last run
|
||||
# is-stale — "true" if latest nightly build is >24h old
|
||||
# stale-age-hours — age of the nightly build in hours (only if stale)
|
||||
# nightly-created — ISO timestamp of the nightly build
|
||||
#
|
||||
# Reads:
|
||||
# .nightly-state/last-feature-ref (from GH Actions cache, may not exist)
|
||||
#
|
||||
# Requires: gh (GitHub CLI), jq, date
|
||||
set -euo pipefail
|
||||
|
||||
FORCE="${1:-false}"
|
||||
REPO="${GITHUB_REPOSITORY:-paradigmxyz/reth}"
|
||||
|
||||
# --- Step 1: Query latest successful scheduled docker.yml run ---
|
||||
echo "::group::Querying latest nightly docker build"
|
||||
|
||||
RUNS_JSON=$(gh run list \
|
||||
-R "$REPO" \
|
||||
--workflow=docker.yml \
|
||||
--event=schedule \
|
||||
--status=completed \
|
||||
--limit 5 \
|
||||
--json headSha,createdAt,conclusion)
|
||||
|
||||
# Find the most recent successful run
|
||||
LATEST=$(echo "$RUNS_JSON" | jq -r '[.[] | select(.conclusion == "success")] | first // empty')
|
||||
|
||||
if [ -z "$LATEST" ]; then
|
||||
echo "::error::No successful scheduled docker.yml run found in the last 5 runs"
|
||||
echo "Runs found: $RUNS_JSON"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
FEATURE_REF=$(echo "$LATEST" | jq -r '.headSha')
|
||||
CREATED_AT=$(echo "$LATEST" | jq -r '.createdAt')
|
||||
echo "Latest nightly commit: $FEATURE_REF"
|
||||
echo "Built at: $CREATED_AT"
|
||||
echo "::endgroup::"
|
||||
|
||||
# --- Step 2: Staleness check ---
|
||||
echo "::group::Checking staleness"
|
||||
NOW_EPOCH=$(date +%s)
|
||||
# Handle both GNU date (-d) and BSD date (-j -f) for cross-platform compat
|
||||
CREATED_EPOCH=$(date -d "$CREATED_AT" +%s 2>/dev/null || \
|
||||
date -j -f "%Y-%m-%dT%H:%M:%SZ" "$CREATED_AT" +%s 2>/dev/null || \
|
||||
date -j -f "%Y-%m-%dT%T%z" "$CREATED_AT" +%s 2>/dev/null || \
|
||||
{ echo "::error::Cannot parse date: $CREATED_AT"; exit 1; })
|
||||
|
||||
AGE_SECONDS=$(( NOW_EPOCH - CREATED_EPOCH ))
|
||||
AGE_HOURS=$(( AGE_SECONDS / 3600 ))
|
||||
IS_STALE="false"
|
||||
|
||||
if [ "$AGE_HOURS" -gt 24 ]; then
|
||||
IS_STALE="true"
|
||||
echo "::warning::STALE NIGHTLY: Build is ${AGE_HOURS}h old (>24h threshold)"
|
||||
echo "This indicates the nightly docker build failed — no new image was produced"
|
||||
else
|
||||
echo "Nightly build age: ${AGE_HOURS}h (within 24h threshold)"
|
||||
fi
|
||||
echo "::endgroup::"
|
||||
|
||||
# --- Step 3: Read last successful feature ref from cache ---
|
||||
echo "::group::Reading cached state"
|
||||
LAST_FEATURE_REF=""
|
||||
STATE_FILE=".nightly-state/last-feature-ref"
|
||||
if [ -f "$STATE_FILE" ]; then
|
||||
LAST_FEATURE_REF=$(tr -d '[:space:]' < "$STATE_FILE")
|
||||
echo "Previous feature ref: $LAST_FEATURE_REF"
|
||||
else
|
||||
echo "No cached state found (first run)"
|
||||
fi
|
||||
echo "::endgroup::"
|
||||
|
||||
# --- Step 4: Determine baseline and skip logic ---
|
||||
echo "::group::Resolving refs"
|
||||
SHOULD_SKIP="false"
|
||||
BASELINE_REF="$FEATURE_REF" # default for first run
|
||||
|
||||
if [ "$IS_STALE" = "true" ]; then
|
||||
# Stale = error path, don't skip (will alert and fail downstream)
|
||||
SHOULD_SKIP="false"
|
||||
BASELINE_REF="${LAST_FEATURE_REF:-$FEATURE_REF}"
|
||||
echo "Stale nightly detected — will alert and fail"
|
||||
elif [ -z "$LAST_FEATURE_REF" ]; then
|
||||
# First run: baseline = feature (self-comparison to establish baseline)
|
||||
BASELINE_REF="$FEATURE_REF"
|
||||
echo "First run — will benchmark nightly against itself to establish baseline"
|
||||
elif [ "$LAST_FEATURE_REF" = "$FEATURE_REF" ]; then
|
||||
# No new nightly since last successful run
|
||||
if [ "$FORCE" = "true" ] || [ "$FORCE" = "--force" ]; then
|
||||
echo "No new nightly, but force=true — running anyway"
|
||||
BASELINE_REF="$LAST_FEATURE_REF"
|
||||
else
|
||||
SHOULD_SKIP="true"
|
||||
echo "No new nightly since last run — will skip"
|
||||
fi
|
||||
else
|
||||
# Normal case: new nightly available
|
||||
BASELINE_REF="$LAST_FEATURE_REF"
|
||||
echo "New nightly detected"
|
||||
fi
|
||||
|
||||
echo "Baseline: $BASELINE_REF"
|
||||
echo "Feature: $FEATURE_REF"
|
||||
echo "Skip: $SHOULD_SKIP"
|
||||
echo "Stale: $IS_STALE"
|
||||
echo "::endgroup::"
|
||||
|
||||
# --- Step 5: Write outputs ---
|
||||
{
|
||||
echo "baseline-ref=$BASELINE_REF"
|
||||
echo "feature-ref=$FEATURE_REF"
|
||||
echo "should-skip=$SHOULD_SKIP"
|
||||
echo "is-stale=$IS_STALE"
|
||||
echo "stale-age-hours=$AGE_HOURS"
|
||||
echo "nightly-created=$CREATED_AT"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
106
.github/scripts/bench-slack-notify.js
vendored
106
.github/scripts/bench-slack-notify.js
vendored
@@ -7,8 +7,6 @@
|
||||
// BENCH_PR – PR number (may be empty)
|
||||
// BENCH_ACTOR – GitHub user who triggered the bench
|
||||
// BENCH_JOB_URL – URL to the Actions job page
|
||||
// BENCH_BASELINE_ARGS – Extra CLI args for the baseline reth node
|
||||
// BENCH_FEATURE_ARGS – Extra CLI args for the feature reth node
|
||||
// BENCH_SAMPLY – 'true' if samply profiling was enabled
|
||||
//
|
||||
// Usage from actions/github-script:
|
||||
@@ -18,7 +16,6 @@
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { fmtChange, fmtMs, verdict, loadSamplyUrls, blocksLabel, metricRows, waitTimeRows } = require('./bench-utils');
|
||||
|
||||
const SLACK_API = 'https://slack.com/api/chat.postMessage';
|
||||
|
||||
@@ -62,17 +59,40 @@ function cell(text) {
|
||||
return { type: 'raw_text', text: s || ' ' };
|
||||
}
|
||||
|
||||
// Slack shortcodes for verdict (Block Kit header doesn't support unicode emoji)
|
||||
const SLACK_VERDICT = {
|
||||
'⚠️': ':warning:',
|
||||
'❌': ':x:',
|
||||
'✅': ':white_check_mark:',
|
||||
'⚪': ':white_circle:',
|
||||
};
|
||||
|
||||
function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, repo, samplyUrls }) {
|
||||
const { emoji, label } = verdict(summary.changes);
|
||||
const headerEmoji = SLACK_VERDICT[emoji] || emoji;
|
||||
const b = summary.baseline.stats;
|
||||
const f = summary.feature.stats;
|
||||
const c = summary.changes;
|
||||
|
||||
const sigEmoji = { good: '\u2705', bad: '\u274c', neutral: '\u26aa' };
|
||||
|
||||
function fmtMs(v) { return v.toFixed(2) + 'ms'; }
|
||||
function fmtMgas(v) { return v.toFixed(2); }
|
||||
function fmtChange(ch) {
|
||||
if (!ch.pct && !ch.ci_pct) return ' ';
|
||||
const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
|
||||
const ciStr = ch.ci_pct ? ` (\u00b1${ch.ci_pct.toFixed(2)}%)` : '';
|
||||
return `${pctStr}${ciStr} ${sigEmoji[ch.sig]}`;
|
||||
}
|
||||
|
||||
// Overall result for header
|
||||
const vals = Object.values(c);
|
||||
const hasBad = vals.some(v => v.sig === 'bad');
|
||||
const hasGood = vals.some(v => v.sig === 'good');
|
||||
let headerEmoji, headerResult;
|
||||
if (hasBad && hasGood) {
|
||||
headerEmoji = ':warning:';
|
||||
headerResult = 'Mixed Results';
|
||||
} else if (hasBad) {
|
||||
headerEmoji = ':x:';
|
||||
headerResult = 'Regression';
|
||||
} else if (hasGood) {
|
||||
headerEmoji = ':white_check_mark:';
|
||||
headerResult = 'Improvement';
|
||||
} else {
|
||||
headerEmoji = ':white_circle:';
|
||||
headerResult = 'No Difference';
|
||||
}
|
||||
|
||||
const prUrl = prNumber ? `https://github.com/${repo}/pull/${prNumber}` : '';
|
||||
const commitUrl = `https://github.com/${repo}/commit`;
|
||||
@@ -97,15 +117,15 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
|
||||
if (fl1) featureLine += ` | <${fl1}|Samply 1>`;
|
||||
if (fl2) featureLine += ` | <${fl2}|Samply 2>`;
|
||||
|
||||
const countsLine = blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | ');
|
||||
const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
|
||||
const cores = process.env.BENCH_CORES || '0';
|
||||
const countsParts = [];
|
||||
if (warmup) countsParts.push(`*Warmup:* ${warmup}`);
|
||||
countsParts.push(`*Blocks:* ${summary.blocks}`);
|
||||
if (cores !== '0') countsParts.push(`*Cores:* ${cores}`);
|
||||
const countsLine = countsParts.join(' | ');
|
||||
|
||||
const baselineArgs = process.env.BENCH_BASELINE_ARGS || '';
|
||||
const featureArgs = process.env.BENCH_FEATURE_ARGS || '';
|
||||
const argsLines = [];
|
||||
if (baselineArgs) argsLines.push(`*Baseline Args:* \`${baselineArgs}\``);
|
||||
if (featureArgs) argsLines.push(`*Feature Args:* \`${featureArgs}\``);
|
||||
|
||||
const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, ...argsLines, countsLine].join('\n');
|
||||
const sectionText = [metaParts.join(' | '), '', baselineLine, featureLine, countsLine].join('\n');
|
||||
|
||||
// Action buttons
|
||||
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
|
||||
@@ -124,17 +144,10 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
|
||||
},
|
||||
];
|
||||
|
||||
// Build table rows from shared metricRows
|
||||
const rows = metricRows(summary);
|
||||
const tableRows = [
|
||||
[cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
|
||||
...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
|
||||
];
|
||||
|
||||
const blocks = [
|
||||
{
|
||||
type: 'header',
|
||||
text: { type: 'plain_text', text: `${headerEmoji} ${label}`, emoji: true },
|
||||
text: { type: 'plain_text', text: `${headerEmoji} ${headerResult}`, emoji: true },
|
||||
},
|
||||
{
|
||||
type: 'section',
|
||||
@@ -148,7 +161,15 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
|
||||
{ align: 'right' },
|
||||
{ align: 'right' },
|
||||
],
|
||||
rows: tableRows,
|
||||
rows: [
|
||||
[cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
|
||||
[cell('Mean'), cell(fmtMs(b.mean_ms)), cell(fmtMs(f.mean_ms)), cell(fmtChange(c.mean))],
|
||||
[cell('StdDev'), cell(fmtMs(b.stddev_ms)), cell(fmtMs(f.stddev_ms)), cell(' ')],
|
||||
[cell('P50'), cell(fmtMs(b.p50_ms)), cell(fmtMs(f.p50_ms)), cell(fmtChange(c.p50))],
|
||||
[cell('P90'), cell(fmtMs(b.p90_ms)), cell(fmtMs(f.p90_ms)), cell(fmtChange(c.p90))],
|
||||
[cell('P99'), cell(fmtMs(b.p99_ms)), cell(fmtMs(f.p99_ms)), cell(fmtChange(c.p99))],
|
||||
[cell('Mgas/s'), cell(fmtMgas(b.mean_mgas_s)), cell(fmtMgas(f.mean_mgas_s)), cell(fmtChange(c.mgas_s))],
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'actions',
|
||||
@@ -158,12 +179,16 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
|
||||
|
||||
// Wait times as a separate table block (sent as threaded reply due to Slack one-table limit)
|
||||
const threadBlocks = [];
|
||||
const wtRows = waitTimeRows(summary);
|
||||
if (wtRows.length > 0) {
|
||||
const waitTableRows = [
|
||||
const waitTimes = summary.wait_times || {};
|
||||
const waitKeys = Object.keys(waitTimes);
|
||||
if (waitKeys.length > 0) {
|
||||
const waitRows = [
|
||||
[cell('Wait Time'), cell('Baseline'), cell('Feature')],
|
||||
...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
|
||||
];
|
||||
for (const key of waitKeys) {
|
||||
const wt = waitTimes[key];
|
||||
waitRows.push([cell(wt.title), cell(fmtMs(wt.baseline.mean_ms)), cell(fmtMs(wt.feature.mean_ms))]);
|
||||
}
|
||||
threadBlocks.push({
|
||||
type: 'table',
|
||||
column_settings: [
|
||||
@@ -171,7 +196,7 @@ function buildSuccessBlocks({ summary, prNumber, actor, actorSlackId, jobUrl, re
|
||||
{ align: 'right' },
|
||||
{ align: 'right' },
|
||||
],
|
||||
rows: waitTableRows,
|
||||
rows: waitRows,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -233,7 +258,16 @@ async function success({ core, context }) {
|
||||
const jobUrl = process.env.BENCH_JOB_URL ||
|
||||
`${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
|
||||
|
||||
const samplyUrls = loadSamplyUrls(process.env.BENCH_WORK_DIR);
|
||||
// Load samply profile URLs (files exist when samply profiling was enabled)
|
||||
const samplyUrls = {};
|
||||
for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
|
||||
try {
|
||||
const url = fs.readFileSync(
|
||||
path.join(process.env.BENCH_WORK_DIR, run, 'samply-profile-url.txt'), 'utf8'
|
||||
).trim();
|
||||
if (url) samplyUrls[run] = url;
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const slackUsers = loadSlackUsers(process.env.GITHUB_WORKSPACE || '.');
|
||||
const actorSlackId = slackUsers[actor];
|
||||
|
||||
13
.github/scripts/bench-slack-users.json
vendored
13
.github/scripts/bench-slack-users.json
vendored
@@ -9,16 +9,5 @@
|
||||
"gakonst": "U092SEPDM40",
|
||||
"Rjected": "U09F6SCKRGT",
|
||||
"DaniPopes": "U09FAT8EK2A",
|
||||
"emmajam": "U0A34UN92HW",
|
||||
"onbjerg": "U09FB0UK5AA",
|
||||
"fgimenez": "U09G3GP7CSU",
|
||||
"rakita": "U09FB3Z2M7Y",
|
||||
"jxom": "U09F72MG083",
|
||||
"tmm": "U0AD0U8E88N",
|
||||
"pepyakin": "U0A7HKMGEHJ",
|
||||
"grandizzy": "U09F8DBDDRT",
|
||||
"SuperFluffy": "U095BKHB2Q4",
|
||||
"kamsz": "U0A2563UBRD",
|
||||
"zerosnacks": "U09FARPMN74",
|
||||
"samczsun": "U096R14E4H3"
|
||||
"emmajam": "U0A34UN92HW"
|
||||
}
|
||||
|
||||
96
.github/scripts/bench-utils.js
vendored
96
.github/scripts/bench-utils.js
vendored
@@ -1,96 +0,0 @@
|
||||
// Shared utilities for reth-bench result rendering.
|
||||
//
|
||||
// Used by bench-job-summary.js and bench-slack-notify.js.
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const SIG_EMOJI = { good: '✅', bad: '❌', neutral: '⚪' };
|
||||
|
||||
function fmtMs(v) { return v.toFixed(2) + 'ms'; }
|
||||
function fmtMgas(v) { return v.toFixed(2); }
|
||||
function fmtS(v) { return v.toFixed(2) + 's'; }
|
||||
|
||||
function fmtChange(ch) {
|
||||
if (!ch || (!ch.pct && !ch.ci_pct)) return '';
|
||||
const pctStr = `${ch.pct >= 0 ? '+' : ''}${ch.pct.toFixed(2)}%`;
|
||||
const ciStr = ch.ci_pct ? ` (±${ch.ci_pct.toFixed(2)}%)` : '';
|
||||
return `${pctStr}${ciStr} ${SIG_EMOJI[ch.sig]}`;
|
||||
}
|
||||
|
||||
function verdict(changes) {
|
||||
const vals = Object.values(changes);
|
||||
const hasBad = vals.some(v => v.sig === 'bad');
|
||||
const hasGood = vals.some(v => v.sig === 'good');
|
||||
if (hasBad && hasGood) return { emoji: '⚠️', label: 'Mixed Results' };
|
||||
if (hasBad) return { emoji: '❌', label: 'Regression' };
|
||||
if (hasGood) return { emoji: '✅', label: 'Improvement' };
|
||||
return { emoji: '⚪', label: 'No Difference' };
|
||||
}
|
||||
|
||||
function loadSamplyUrls(workDir) {
|
||||
const urls = {};
|
||||
for (const run of ['baseline-1', 'baseline-2', 'feature-1', 'feature-2']) {
|
||||
try {
|
||||
const url = fs.readFileSync(path.join(workDir, run, 'samply-profile-url.txt'), 'utf8').trim();
|
||||
if (url) urls[run] = url;
|
||||
} catch {}
|
||||
}
|
||||
return urls;
|
||||
}
|
||||
|
||||
function blocksLabel(summary) {
|
||||
const parts = [];
|
||||
if (summary.big_blocks) {
|
||||
if (summary.gas_ramp_blocks) parts.push({ key: 'Gas Ramp', value: summary.gas_ramp_blocks });
|
||||
parts.push({ key: 'Big Blocks', value: summary.blocks });
|
||||
} else {
|
||||
const warmup = summary.warmup_blocks || process.env.BENCH_WARMUP_BLOCKS || '';
|
||||
if (warmup) parts.push({ key: 'Warmup', value: warmup });
|
||||
parts.push({ key: 'Blocks', value: summary.blocks });
|
||||
}
|
||||
const cores = process.env.BENCH_CORES || '0';
|
||||
if (cores !== '0') parts.push({ key: 'Cores', value: cores });
|
||||
return parts;
|
||||
}
|
||||
|
||||
// The 7 metric rows shared by all renderers.
|
||||
// Returns an array of { label, baseline, feature, change } objects.
|
||||
function metricRows(summary) {
|
||||
const b = summary.baseline.stats;
|
||||
const f = summary.feature.stats;
|
||||
const c = summary.changes;
|
||||
return [
|
||||
{ label: 'Mean', baseline: fmtMs(b.mean_ms), feature: fmtMs(f.mean_ms), change: fmtChange(c.mean) },
|
||||
{ label: 'StdDev', baseline: fmtMs(b.stddev_ms), feature: fmtMs(f.stddev_ms), change: '' },
|
||||
{ label: 'P50', baseline: fmtMs(b.p50_ms), feature: fmtMs(f.p50_ms), change: fmtChange(c.p50) },
|
||||
{ label: 'P90', baseline: fmtMs(b.p90_ms), feature: fmtMs(f.p90_ms), change: fmtChange(c.p90) },
|
||||
{ label: 'P99', baseline: fmtMs(b.p99_ms), feature: fmtMs(f.p99_ms), change: fmtChange(c.p99) },
|
||||
{ label: 'Mgas/s', baseline: fmtMgas(b.mean_mgas_s), feature: fmtMgas(f.mean_mgas_s), change: fmtChange(c.mgas_s) },
|
||||
{ label: 'Wall Clock', baseline: fmtS(b.wall_clock_s), feature: fmtS(f.wall_clock_s), change: fmtChange(c.wall_clock) },
|
||||
];
|
||||
}
|
||||
|
||||
// Wait time rows: one row per metric showing mean values.
|
||||
function waitTimeRows(summary) {
|
||||
const waitTimes = summary.wait_times || {};
|
||||
const rows = [];
|
||||
for (const key of Object.keys(waitTimes)) {
|
||||
const wt = waitTimes[key];
|
||||
rows.push({ title: wt.title, baseline: fmtMs(wt.baseline.mean_ms), feature: fmtMs(wt.feature.mean_ms) });
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
SIG_EMOJI,
|
||||
fmtMs,
|
||||
fmtMgas,
|
||||
fmtS,
|
||||
fmtChange,
|
||||
verdict,
|
||||
loadSamplyUrls,
|
||||
blocksLabel,
|
||||
metricRows,
|
||||
waitTimeRows,
|
||||
};
|
||||
414
.github/scripts/build_pgo_bolt.sh
vendored
414
.github/scripts/build_pgo_bolt.sh
vendored
@@ -1,414 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Full PGO+BOLT optimized build for reth using real reth-bench workloads.
|
||||
#
|
||||
# Phases:
|
||||
# 1. Build PGO-instrumented reth, run reth-bench → collect PGO profiles
|
||||
# 2. Build BOLT-instrumented reth (with PGO), run reth-bench → collect BOLT profiles
|
||||
# 3. Build final PGO+BOLT optimized binary
|
||||
#
|
||||
# Required environment variables:
|
||||
# DATADIR - Path to reth datadir (must already contain chain data)
|
||||
# RPC_URL - Source RPC URL for reth-bench to fetch payloads from
|
||||
#
|
||||
# Optional environment variables:
|
||||
# PGO_BLOCKS - Number of blocks for PGO profiling (default: 20)
|
||||
# BOLT_BLOCKS - Number of blocks for BOLT profiling (default: 20)
|
||||
# SKIP_BOLT - Temporarily skip BOLT phases (default: false)
|
||||
# STRIP_SYMBOLS - Strip debug symbols from output binary (default: true)
|
||||
# COLLECT_PGO_ONLY - Stop after producing merged.profdata (default: false)
|
||||
# PGO_PROFDATA - Path to pre-collected merged.profdata (optional)
|
||||
# PROFILE - Cargo profile (default: maxperf-symbols)
|
||||
# FEATURES - Cargo features (default: jemalloc,asm-keccak,min-debug-logs)
|
||||
# TARGET - Target triple (default: auto-detected)
|
||||
# EXTRA_RUSTFLAGS - Additional RUSTFLAGS (e.g. -C target-cpu=x86-64-v3)
|
||||
#
|
||||
# Output:
|
||||
# target/$PROFILE_DIR/reth — final optimized binary
|
||||
set -euo pipefail
|
||||
|
||||
gha_section_start() {
|
||||
local title="$1"
|
||||
if [ -n "${GITHUB_ACTIONS:-}" ]; then
|
||||
echo "::group::$title"
|
||||
else
|
||||
echo ""
|
||||
echo "=== $title ==="
|
||||
fi
|
||||
}
|
||||
|
||||
gha_section_end() {
|
||||
if [ -n "${GITHUB_ACTIONS:-}" ]; then
|
||||
echo "::endgroup::"
|
||||
fi
|
||||
}
|
||||
|
||||
cd "$(dirname "$0")/../.."
|
||||
|
||||
# ── Configuration ──────────────────────────────────────────────────────────────
|
||||
PGO_BLOCKS="${PGO_BLOCKS:-20}"
|
||||
BOLT_BLOCKS="${BOLT_BLOCKS:-20}"
|
||||
SKIP_BOLT="${SKIP_BOLT:-false}"
|
||||
STRIP_SYMBOLS="${STRIP_SYMBOLS:-true}"
|
||||
COLLECT_PGO_ONLY="${COLLECT_PGO_ONLY:-false}"
|
||||
PROFILE="${PROFILE:-maxperf-symbols}"
|
||||
FEATURES="${FEATURES:-jemalloc,asm-keccak,min-debug-logs}"
|
||||
TARGET="${TARGET:-$(rustc -Vv | grep host | cut -d' ' -f2)}"
|
||||
BASE_RUSTFLAGS="${RUSTFLAGS:-}"
|
||||
EXTRA_RUSTFLAGS="${EXTRA_RUSTFLAGS:-}"
|
||||
COMBINED_RUSTFLAGS="$BASE_RUSTFLAGS $EXTRA_RUSTFLAGS"
|
||||
PGO_PROFDATA="${PGO_PROFDATA:-}"
|
||||
DATADIR="${DATADIR:-}"
|
||||
RPC_URL="${RPC_URL:-}"
|
||||
|
||||
SKIP_BOLT_BOOL=false
|
||||
if [[ "${SKIP_BOLT,,}" == "true" || "$SKIP_BOLT" == "1" ]]; then
|
||||
SKIP_BOLT_BOOL=true
|
||||
fi
|
||||
|
||||
STRIP_SYMBOLS_BOOL=false
|
||||
if [[ "${STRIP_SYMBOLS,,}" == "true" || "$STRIP_SYMBOLS" == "1" ]]; then
|
||||
STRIP_SYMBOLS_BOOL=true
|
||||
fi
|
||||
|
||||
COLLECT_PGO_ONLY_BOOL=false
|
||||
if [[ "${COLLECT_PGO_ONLY,,}" == "true" || "$COLLECT_PGO_ONLY" == "1" ]]; then
|
||||
COLLECT_PGO_ONLY_BOOL=true
|
||||
fi
|
||||
|
||||
USE_PRECOLLECTED_PGO=false
|
||||
if [ -n "$PGO_PROFDATA" ]; then
|
||||
if [ ! -f "$PGO_PROFDATA" ]; then
|
||||
echo "error: PGO_PROFDATA points to a missing file: $PGO_PROFDATA"
|
||||
exit 1
|
||||
fi
|
||||
USE_PRECOLLECTED_PGO=true
|
||||
fi
|
||||
|
||||
NEEDS_BENCH_WORKLOAD=true
|
||||
if [ "$USE_PRECOLLECTED_PGO" = true ] && [ "$SKIP_BOLT_BOOL" = true ]; then
|
||||
NEEDS_BENCH_WORKLOAD=false
|
||||
fi
|
||||
|
||||
if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
|
||||
: "${DATADIR:?DATADIR must be set to the reth data directory}"
|
||||
: "${RPC_URL:?RPC_URL must be set}"
|
||||
fi
|
||||
|
||||
if [[ "$PROFILE" == dev ]]; then
|
||||
PROFILE_DIR=debug
|
||||
else
|
||||
PROFILE_DIR=$PROFILE
|
||||
fi
|
||||
|
||||
MANIFEST_PATH="bin/reth"
|
||||
|
||||
LLVM_VERSION=$(rustc -Vv | grep -oP 'LLVM version: \K\d+')
|
||||
PGO_DIR="$PWD/target/pgo-profiles"
|
||||
BOLT_DIR="$PWD/target/bolt-profiles"
|
||||
CARGO_ARGS=(--profile "$PROFILE" --features "$FEATURES" --manifest-path "$MANIFEST_PATH/Cargo.toml" --bin "reth" --locked)
|
||||
|
||||
# Enable debug symbols for BOLT (requires symbols to reorder code).
|
||||
# Strip them at the end.
|
||||
PROFILE_UPPER=$(echo "$PROFILE" | tr '[:lower:]-' '[:upper:]_')
|
||||
export "CARGO_PROFILE_${PROFILE_UPPER}_STRIP=debuginfo"
|
||||
|
||||
gha_section_start "Full PGO+BOLT Build"
|
||||
echo "Binary: reth"
|
||||
echo "Manifest: $MANIFEST_PATH"
|
||||
echo "Target: $TARGET"
|
||||
echo "Profile: $PROFILE"
|
||||
echo "Features: $FEATURES"
|
||||
echo "LLVM: $LLVM_VERSION"
|
||||
echo "PGO blocks: $PGO_BLOCKS"
|
||||
echo "BOLT blocks: $BOLT_BLOCKS"
|
||||
echo "Skip BOLT: $SKIP_BOLT"
|
||||
echo "Strip symbols: $STRIP_SYMBOLS"
|
||||
echo "Collect only: $COLLECT_PGO_ONLY"
|
||||
echo "PGO profdata: ${PGO_PROFDATA:-<collect with reth-bench>}"
|
||||
echo "RUSTFLAGS: ${BASE_RUSTFLAGS:-<unset>}"
|
||||
echo "EXTRA_RUSTFLAGS: ${EXTRA_RUSTFLAGS:-<unset>}"
|
||||
if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
|
||||
echo "Datadir: $DATADIR"
|
||||
echo "RPC URL: $RPC_URL"
|
||||
else
|
||||
echo "Datadir: <not required>"
|
||||
echo "RPC URL: <not required>"
|
||||
fi
|
||||
gha_section_end
|
||||
|
||||
# ── Prerequisites ──────────────────────────────────────────────────────────────
|
||||
gha_section_start "Installing prerequisites"
|
||||
rustup component add llvm-tools-preview
|
||||
|
||||
LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
|
||||
if [ -z "$LLVM_PROFDATA" ]; then
|
||||
echo "error: llvm-profdata not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
install_bolt() {
|
||||
if command -v llvm-bolt &>/dev/null; then
|
||||
echo "BOLT already installed"
|
||||
return
|
||||
fi
|
||||
echo "Installing BOLT from apt.llvm.org..."
|
||||
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc >/dev/null
|
||||
CODENAME=$(lsb_release -cs)
|
||||
echo "deb http://apt.llvm.org/$CODENAME/ llvm-toolchain-$CODENAME-$LLVM_VERSION main" | sudo tee /etc/apt/sources.list.d/llvm.list >/dev/null
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y -qq "bolt-$LLVM_VERSION"
|
||||
sudo ln -sf "/usr/bin/llvm-bolt-$LLVM_VERSION" /usr/local/bin/llvm-bolt
|
||||
sudo ln -sf "/usr/bin/merge-fdata-$LLVM_VERSION" /usr/local/bin/merge-fdata
|
||||
}
|
||||
if [ "$SKIP_BOLT_BOOL" = true ]; then
|
||||
echo "Skipping BOLT installation (SKIP_BOLT=$SKIP_BOLT)"
|
||||
else
|
||||
install_bolt
|
||||
fi
|
||||
gha_section_end
|
||||
|
||||
if [ "$NEEDS_BENCH_WORKLOAD" = true ]; then
|
||||
# Build reth-bench once (non-instrumented) — reused for both phases.
|
||||
gha_section_start "Building reth-bench"
|
||||
RUSTFLAGS="$COMBINED_RUSTFLAGS" \
|
||||
cargo build --profile "$PROFILE" --features "$FEATURES" \
|
||||
--manifest-path bin/reth-bench/Cargo.toml --bin reth-bench --locked
|
||||
RETH_BENCH_BIN="$(find target -name reth-bench -type f -executable | head -1)"
|
||||
echo "reth-bench: $RETH_BENCH_BIN"
|
||||
gha_section_end
|
||||
else
|
||||
gha_section_start "Building reth-bench"
|
||||
echo "Skipping reth-bench build (pre-collected PGO with SKIP_BOLT=true)"
|
||||
gha_section_end
|
||||
fi
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────────────
|
||||
RETH_PID=
|
||||
cleanup() {
|
||||
if [ -n "${RETH_PID:-}" ] && kill -0 "$RETH_PID" 2>/dev/null; then
|
||||
echo "Stopping reth (pid $RETH_PID)..."
|
||||
sudo kill "$RETH_PID" 2>/dev/null || true
|
||||
for i in $(seq 1 60); do
|
||||
sudo kill -0 "$RETH_PID" 2>/dev/null || break
|
||||
if [ $((i % 10)) -eq 0 ]; then
|
||||
echo " waiting... (${i}s)"
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
sudo kill -9 "$RETH_PID" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# Start reth, wait for RPC, run reth-bench, then stop reth.
|
||||
# Arguments: $1 = reth binary path, $2 = number of blocks, $3 = log label
|
||||
run_bench_workload() {
|
||||
local reth_bin="$1" blocks="$2" label="$3"
|
||||
local http_port=8545 authrpc_port=8551
|
||||
|
||||
echo "--- Starting reth ($label) ---"
|
||||
sudo "$reth_bin" node \
|
||||
--datadir "$DATADIR" \
|
||||
--log.file.directory "/tmp/reth-${label}-logs" \
|
||||
--engine.accept-execution-requests-hash \
|
||||
--http --http.port "$http_port" \
|
||||
--authrpc.port "$authrpc_port" \
|
||||
--disable-discovery --no-persist-peers \
|
||||
> "/tmp/reth-${label}.log" 2>&1 &
|
||||
RETH_PID=$!
|
||||
|
||||
echo "Waiting for reth RPC..."
|
||||
for i in $(seq 1 120); do
|
||||
if curl -sf "http://127.0.0.1:$http_port" -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
|
||||
> /dev/null 2>&1; then
|
||||
echo "reth is ready after ${i}s"
|
||||
break
|
||||
fi
|
||||
if [ "$i" -eq 120 ]; then
|
||||
echo "error: reth failed to start within 120s"
|
||||
cat "/tmp/reth-${label}.log"
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo "Running reth-bench ($blocks blocks)..."
|
||||
"$RETH_BENCH_BIN" new-payload-fcu \
|
||||
--rpc-url "$RPC_URL" \
|
||||
--engine-rpc-url "http://127.0.0.1:$authrpc_port" \
|
||||
--jwt-secret "$DATADIR/jwt.hex" \
|
||||
--advance "$blocks" \
|
||||
--reth-new-payload 2>&1 | sed -u "s/^/[$label] /"
|
||||
|
||||
echo "Stopping reth ($label)..."
|
||||
sudo kill "$RETH_PID" 2>/dev/null || true
|
||||
for i in $(seq 1 60); do
|
||||
sudo kill -0 "$RETH_PID" 2>/dev/null || break
|
||||
sleep 1
|
||||
done
|
||||
sudo kill -9 "$RETH_PID" 2>/dev/null || true
|
||||
RETH_PID=
|
||||
}
|
||||
|
||||
publish_binary() {
|
||||
local source_bin="$1"
|
||||
for out in "target/$TARGET/$PROFILE_DIR" "target/$PROFILE_DIR"; do
|
||||
local destination="$out/reth"
|
||||
mkdir -p "$out"
|
||||
# Skip copying when source and destination resolve to the same inode.
|
||||
if [ -e "$destination" ] && [ "$source_bin" -ef "$destination" ]; then
|
||||
continue
|
||||
fi
|
||||
cp "$source_bin" "$destination"
|
||||
done
|
||||
}
|
||||
|
||||
if [ "$USE_PRECOLLECTED_PGO" = true ]; then
|
||||
gha_section_start "Phase 1: Using Pre-Collected PGO Profile"
|
||||
rm -rf "$PGO_DIR"
|
||||
mkdir -p "$PGO_DIR"
|
||||
cp "$PGO_PROFDATA" "$PGO_DIR/merged.profdata"
|
||||
echo "Using pre-collected profile: $PGO_PROFDATA"
|
||||
echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
|
||||
gha_section_end
|
||||
else
|
||||
# ── Phase 1: PGO profile collection ───────────────────────────────────────
|
||||
gha_section_start "Phase 1: PGO Profile Collection"
|
||||
|
||||
rm -rf "$PGO_DIR"
|
||||
mkdir -p "$PGO_DIR"
|
||||
|
||||
echo "Building PGO-instrumented binary..."
|
||||
RUSTFLAGS="-Cprofile-generate=$PGO_DIR -Crelocation-model=pic $COMBINED_RUSTFLAGS" \
|
||||
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
|
||||
|
||||
PGO_RETH_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
|
||||
echo "Instrumented binary: $PGO_RETH_BIN ($(ls -lh "$PGO_RETH_BIN" | awk '{print $5}'))"
|
||||
|
||||
run_bench_workload "$PGO_RETH_BIN" "$PGO_BLOCKS" "pgo"
|
||||
|
||||
# Fix ownership if reth ran as root.
|
||||
sudo chown -R "$(id -un):$(id -gn)" "$PGO_DIR" 2>/dev/null || true
|
||||
|
||||
# Merge PGO profiles.
|
||||
echo "Merging PGO profiles..."
|
||||
PROFRAW_COUNT=$(find "$PGO_DIR" -name '*.profraw' | wc -l)
|
||||
echo "Found $PROFRAW_COUNT .profraw files"
|
||||
if [ "$PROFRAW_COUNT" -eq 0 ]; then
|
||||
echo "error: no .profraw files — instrumented binary did not produce profiles"
|
||||
exit 1
|
||||
fi
|
||||
"$LLVM_PROFDATA" merge -o "$PGO_DIR/merged.profdata" "$PGO_DIR"/*.profraw
|
||||
echo "PGO profile: $PGO_DIR/merged.profdata ($(ls -lh "$PGO_DIR/merged.profdata" | awk '{print $5}'))"
|
||||
gha_section_end
|
||||
fi
|
||||
|
||||
if [ "$COLLECT_PGO_ONLY_BOOL" = true ]; then
|
||||
gha_section_start "PGO Collection Complete"
|
||||
echo "COLLECT_PGO_ONLY=true, skipping PGO/BOLT optimized binary build"
|
||||
echo "Profile: $PGO_DIR/merged.profdata"
|
||||
gha_section_end
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$SKIP_BOLT_BOOL" = true ]; then
|
||||
gha_section_start "BOLT Phase Skipped"
|
||||
echo "SKIP_BOLT=$SKIP_BOLT, building PGO-only binary"
|
||||
echo "Building PGO-optimized binary..."
|
||||
RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata $COMBINED_RUSTFLAGS" \
|
||||
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
|
||||
|
||||
BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
|
||||
if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
|
||||
echo "Stripping debug symbols..."
|
||||
strip "$BUILT_BIN"
|
||||
else
|
||||
echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
|
||||
fi
|
||||
publish_binary "$BUILT_BIN"
|
||||
gha_section_end
|
||||
else
|
||||
# ── Phase 2: BOLT profile collection (with PGO) ──────────────────────────
|
||||
gha_section_start "Phase 2: BOLT Profile Collection (with PGO)"
|
||||
|
||||
rm -rf "$BOLT_DIR"
|
||||
mkdir -p "$BOLT_DIR"
|
||||
|
||||
echo "Building BOLT-instrumented binary with PGO..."
|
||||
# --emit-relocs preserves relocation entries in the binary, required by llvm-bolt -instrument
|
||||
RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
|
||||
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
|
||||
|
||||
# Instrument with BOLT
|
||||
BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
|
||||
BOLT_INSTRUMENTED_BIN="$BUILT_BIN-bolt-instrumented"
|
||||
|
||||
echo "Instrumenting binary with BOLT..."
|
||||
# --skip-funcs: skip compiler-generated drop_in_place functions that BOLT can't handle
|
||||
# as split functions in relocation mode (triggered by --emit-relocs)
|
||||
llvm-bolt "$BUILT_BIN" \
|
||||
-instrument \
|
||||
--instrumentation-file-append-pid \
|
||||
--instrumentation-file="$BOLT_DIR/prof" \
|
||||
--skip-funcs='.*drop_in_place.*' \
|
||||
-o "$BOLT_INSTRUMENTED_BIN"
|
||||
echo "BOLT-instrumented binary: $BOLT_INSTRUMENTED_BIN ($(ls -lh "$BOLT_INSTRUMENTED_BIN" | awk '{print $5}'))"
|
||||
|
||||
run_bench_workload "$BOLT_INSTRUMENTED_BIN" "$BOLT_BLOCKS" "bolt"
|
||||
|
||||
# Fix ownership for BOLT profiles
|
||||
sudo chown -R "$(id -un):$(id -gn)" "$BOLT_DIR" 2>/dev/null || true
|
||||
|
||||
# Merge BOLT profiles
|
||||
echo "Merging BOLT profiles..."
|
||||
FDATA_COUNT=$(find "$BOLT_DIR" -name '*.fdata' | wc -l)
|
||||
echo "Found $FDATA_COUNT .fdata files"
|
||||
if [ "$FDATA_COUNT" -eq 0 ]; then
|
||||
echo "error: no .fdata files — BOLT-instrumented binary did not produce profiles"
|
||||
exit 1
|
||||
fi
|
||||
merge-fdata "$BOLT_DIR"/*.fdata > "$BOLT_DIR/merged.fdata"
|
||||
echo "BOLT profile: $BOLT_DIR/merged.fdata ($(ls -lh "$BOLT_DIR/merged.fdata" | awk '{print $5}'))"
|
||||
gha_section_end
|
||||
|
||||
# ── Phase 3: Final optimized build ───────────────────────────────────────
|
||||
gha_section_start "Phase 3: Final PGO+BOLT Optimized Build"
|
||||
|
||||
echo "Building PGO-optimized binary..."
|
||||
# --emit-relocs preserves relocation entries in the binary, required by llvm-bolt for code reordering
|
||||
RUSTFLAGS="-Cprofile-use=$PGO_DIR/merged.profdata -Clink-arg=-Wl,--emit-relocs $COMBINED_RUSTFLAGS" \
|
||||
cargo build "${CARGO_ARGS[@]}" --target "$TARGET"
|
||||
|
||||
BUILT_BIN="$PWD/target/$TARGET/$PROFILE_DIR/reth"
|
||||
OPTIMIZED_BIN="$BUILT_BIN-bolt-optimized"
|
||||
|
||||
echo "Optimizing with BOLT..."
|
||||
llvm-bolt "$BUILT_BIN" \
|
||||
-o "$OPTIMIZED_BIN" \
|
||||
--data "$BOLT_DIR/merged.fdata" \
|
||||
-reorder-blocks=ext-tsp \
|
||||
-reorder-functions=cdsort \
|
||||
-split-functions \
|
||||
-split-all-cold \
|
||||
-dyno-stats \
|
||||
-icf=1 \
|
||||
-use-gnu-stack \
|
||||
--skip-funcs='.*drop_in_place.*'
|
||||
|
||||
if [ "$STRIP_SYMBOLS_BOOL" = true ]; then
|
||||
echo "Stripping debug symbols..."
|
||||
strip "$OPTIMIZED_BIN"
|
||||
else
|
||||
echo "Skipping strip (STRIP_SYMBOLS=$STRIP_SYMBOLS)"
|
||||
fi
|
||||
publish_binary "$OPTIMIZED_BIN"
|
||||
gha_section_end
|
||||
fi
|
||||
|
||||
gha_section_start "Build Complete"
|
||||
ls -lh "target/$PROFILE_DIR/reth"
|
||||
echo "Output: target/$PROFILE_DIR/reth"
|
||||
gha_section_end
|
||||
23
.github/scripts/hive/expected_failures.yaml
vendored
23
.github/scripts/hive/expected_failures.yaml
vendored
@@ -16,11 +16,28 @@ rpc-compat:
|
||||
# syncing mode, the test expects syncing to be false on start
|
||||
- eth_syncing/check-syncing (reth)
|
||||
|
||||
engine-withdrawals: [ ]
|
||||
# no fix due to https://github.com/paradigmxyz/reth/issues/8732
|
||||
engine-withdrawals:
|
||||
- Withdrawals Fork On Genesis (Paris) (reth)
|
||||
- Withdrawals Fork on Block 1 (Paris) (reth)
|
||||
- Withdrawals Fork on Block 2 (Paris) (reth)
|
||||
- Withdrawals Fork on Block 3 (Paris) (reth)
|
||||
- Withdraw to a single account (Paris) (reth)
|
||||
- Withdraw to two accounts (Paris) (reth)
|
||||
- Withdraw many accounts (Paris) (reth)
|
||||
- Withdraw zero amount (Paris) (reth)
|
||||
- Empty Withdrawals (Paris) (reth)
|
||||
- Corrupted Block Hash Payload (INVALID) (Paris) (reth)
|
||||
- Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org (Paris) (reth)
|
||||
|
||||
engine-api: [ ]
|
||||
|
||||
engine-cancun: [ ]
|
||||
# no fix due to https://github.com/paradigmxyz/reth/issues/8732
|
||||
engine-cancun:
|
||||
- Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True (Cancun) (reth)
|
||||
# the test fails with older versions of the code for which it passed before, probably related to changes
|
||||
# in hive or its dependencies
|
||||
- Blob Transaction Ordering, Multiple Clients (Cancun) (reth)
|
||||
|
||||
sync: [ ]
|
||||
|
||||
@@ -32,7 +49,7 @@ engine-auth: [ ]
|
||||
# The test artificially creates an empty account with storage, then tests EIP-7610's behavior.
|
||||
# On mainnet, ~25 such accounts exist as contract addresses (derived from keccak(prefix, caller,
|
||||
# nonce/salt), not from public keys). No private key exists for contract addresses. To trigger
|
||||
# this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible.
|
||||
# this with EIP-7702, you'd need to recover a private key from one of the already deployed contract addresses - mathematically impossible.
|
||||
#
|
||||
# tests/paris/eip7610_create_collision/test_initcollision.py::test_init_collision_*
|
||||
# Requires hash collision on create2 address to target already deployed accounts with storage.
|
||||
|
||||
865
.github/workflows/bench-scheduled.yml
vendored
865
.github/workflows/bench-scheduled.yml
vendored
@@ -1,865 +0,0 @@
|
||||
# Nightly regression benchmark.
|
||||
#
|
||||
# Compares the previous nightly build against the current nightly build to
|
||||
# detect performance regressions. Runs daily after docker.yml produces a new
|
||||
# nightly image at 01:00 UTC.
|
||||
#
|
||||
# State is persisted between runs via GitHub Actions cache: each successful
|
||||
# run saves the feature commit SHA so the next run knows what to compare against.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "30 5 * * *" # 06:30 UTC daily
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
force:
|
||||
description: "Force run even if no new nightly (bypass skip logic)"
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
no_slack:
|
||||
description: "Suppress Slack notifications"
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
RUSTC_WRAPPER: "sccache"
|
||||
|
||||
name: bench-scheduled
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
|
||||
jobs:
|
||||
# ---------------------------------------------------------------------------
|
||||
# Job 1: Resolve nightly refs, check staleness, manage state
|
||||
# ---------------------------------------------------------------------------
|
||||
resolve-refs:
|
||||
name: resolve-refs
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
baseline-ref: ${{ steps.refs.outputs.baseline-ref }}
|
||||
feature-ref: ${{ steps.refs.outputs.feature-ref }}
|
||||
should-skip: ${{ steps.refs.outputs.should-skip }}
|
||||
is-stale: ${{ steps.refs.outputs.is-stale }}
|
||||
stale-age-hours: ${{ steps.refs.outputs.stale-age-hours }}
|
||||
nightly-created: ${{ steps.refs.outputs.nightly-created }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
sparse-checkout: .github/scripts
|
||||
sparse-checkout-cone-mode: true
|
||||
|
||||
- name: Restore nightly state
|
||||
id: state-cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: .nightly-state
|
||||
key: bench-scheduled-state-dummy
|
||||
restore-keys: |
|
||||
bench-scheduled-state-
|
||||
|
||||
- name: Resolve nightly refs
|
||||
id: refs
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
run: |
|
||||
FORCE="${{ inputs.force || 'false' }}"
|
||||
.github/scripts/bench-scheduled-refs.sh "$FORCE"
|
||||
|
||||
- name: Alert on stale nightly
|
||||
if: steps.refs.outputs.is-stale == 'true'
|
||||
uses: actions/github-script@v8
|
||||
env:
|
||||
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
|
||||
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
|
||||
with:
|
||||
script: |
|
||||
const token = process.env.SLACK_BENCH_BOT_TOKEN;
|
||||
const channel = process.env.SLACK_BENCH_CHANNEL;
|
||||
if (!token || !channel) {
|
||||
core.warning('Slack credentials not set, skipping stale nightly alert');
|
||||
return;
|
||||
}
|
||||
|
||||
const ageHours = '${{ steps.refs.outputs.stale-age-hours }}';
|
||||
const created = '${{ steps.refs.outputs.nightly-created }}';
|
||||
const featureRef = '${{ steps.refs.outputs.feature-ref }}';
|
||||
const shortSha = featureRef.slice(0, 8);
|
||||
const repo = '${{ github.repository }}';
|
||||
const runUrl = `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
|
||||
|
||||
const blocks = [
|
||||
{
|
||||
type: 'header',
|
||||
text: { type: 'plain_text', text: ':rotating_light: Nightly Regression: nightly build is stale', emoji: true },
|
||||
},
|
||||
{
|
||||
type: 'section',
|
||||
text: {
|
||||
type: 'mrkdwn',
|
||||
text: [
|
||||
'*Nightly regression did not run* — nightly build is stale',
|
||||
'',
|
||||
`The latest nightly image was built from a commit that is *${ageHours}h old* (threshold: 24h).`,
|
||||
`This means today's nightly docker build likely failed and no new image was produced.`,
|
||||
'',
|
||||
`Stale commit: \`${shortSha}\` (built at ${created})`,
|
||||
'',
|
||||
'*Action required:* Check the <https://github.com/' + repo + '/actions/workflows/docker.yml|docker.yml> workflow for failures.',
|
||||
].join('\n'),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: 'actions',
|
||||
elements: [
|
||||
{
|
||||
type: 'button',
|
||||
text: { type: 'plain_text', text: 'View Run :github:', emoji: true },
|
||||
url: runUrl,
|
||||
action_id: 'ci_button',
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
const resp = await fetch('https://slack.com/api/chat.postMessage', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Authorization': `Bearer ${token}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
channel,
|
||||
blocks,
|
||||
text: 'Nightly regression: nightly build is stale',
|
||||
unfurl_links: false,
|
||||
}),
|
||||
});
|
||||
const data = await resp.json();
|
||||
if (!data.ok) {
|
||||
core.warning(`Slack API error: ${JSON.stringify(data)}`);
|
||||
}
|
||||
|
||||
- name: Fail on stale nightly
|
||||
if: steps.refs.outputs.is-stale == 'true'
|
||||
run: |
|
||||
echo "::error::Nightly build is stale (>24h old). Aborting."
|
||||
exit 1
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Job 2: Run the benchmark
|
||||
# ---------------------------------------------------------------------------
|
||||
bench-scheduled:
|
||||
needs: resolve-refs
|
||||
if: |
|
||||
needs.resolve-refs.outputs.should-skip != 'true' &&
|
||||
needs.resolve-refs.outputs.is-stale != 'true'
|
||||
name: bench-scheduled
|
||||
runs-on: [self-hosted, Linux, X64, available]
|
||||
timeout-minutes: 120
|
||||
env:
|
||||
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
|
||||
SCHELK_MOUNT: /reth-bench
|
||||
BENCH_WORK_DIR: ${{ github.workspace }}/bench-work
|
||||
BENCH_PR: ""
|
||||
BENCH_ACTOR: "nightly-regression"
|
||||
BENCH_BLOCKS: "2000"
|
||||
BENCH_WARMUP_BLOCKS: "500"
|
||||
BENCH_SAMPLY: "false"
|
||||
BENCH_CORES: "0"
|
||||
BENCH_BIG_BLOCKS: "false"
|
||||
BENCH_RETH_NEW_PAYLOAD: "true"
|
||||
BENCH_WAIT_TIME: ""
|
||||
BENCH_BASELINE_ARGS: ""
|
||||
BENCH_FEATURE_ARGS: ""
|
||||
BENCH_ABBA: "true"
|
||||
BENCH_COMMENT_ID: ""
|
||||
BENCH_NO_SLACK: ${{ github.event_name == 'workflow_dispatch' && inputs.no_slack == true && 'true' || 'false' }}
|
||||
BENCH_METRICS_ADDR: "127.0.0.1:9100"
|
||||
BENCH_OTLP_TRACES_ENDPOINT: ${{ secrets.BENCH_OTLP_TRACES_ENDPOINT }}
|
||||
BENCH_OTLP_LOGS_ENDPOINT: ${{ secrets.BENCH_OTLP_LOGS_ENDPOINT }}
|
||||
BASELINE_REF: ${{ needs.resolve-refs.outputs.baseline-ref }}
|
||||
FEATURE_REF: ${{ needs.resolve-refs.outputs.feature-ref }}
|
||||
steps:
|
||||
- name: Clean up previous bench-work
|
||||
run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
fetch-depth: 0
|
||||
ref: ${{ needs.resolve-refs.outputs.feature-ref }}
|
||||
|
||||
- name: Resolve job URL
|
||||
id: job-url
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const { data: jobs } = await github.rest.actions.listJobsForWorkflowRun({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
run_id: context.runId,
|
||||
});
|
||||
const job = jobs.jobs.find(j => j.name === 'bench-scheduled');
|
||||
const jobUrl = job ? job.html_url : `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
|
||||
core.exportVariable('BENCH_JOB_URL', jobUrl);
|
||||
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: mozilla-actions/sccache-action@v0.0.9
|
||||
continue-on-error: true
|
||||
|
||||
- name: Install dependencies
|
||||
env:
|
||||
DEREK_TOKEN: ${{ secrets.DEREK_TOKEN }}
|
||||
run: |
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
|
||||
# apt packages
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y --no-install-recommends \
|
||||
python3 make jq zstd curl dmsetup \
|
||||
linux-tools-"$(uname -r)" || \
|
||||
sudo apt-get install -y --no-install-recommends linux-tools-generic
|
||||
|
||||
# mc (MinIO client)
|
||||
if ! command -v mc &>/dev/null; then
|
||||
curl -sSfL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$HOME/.local/bin/mc"
|
||||
chmod +x "$HOME/.local/bin/mc"
|
||||
fi
|
||||
|
||||
# uv (Python package manager)
|
||||
if ! command -v uv &>/dev/null; then
|
||||
curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$HOME/.local/bin" sh
|
||||
fi
|
||||
|
||||
# Configure git auth for private repos
|
||||
git config --global url."https://x-access-token:${DEREK_TOKEN}@github.com/".insteadOf "https://github.com/"
|
||||
|
||||
# thin-provisioning-tools (era_invalidate, required by schelk)
|
||||
if ! command -v era_invalidate &>/dev/null; then
|
||||
git clone --depth 1 https://github.com/jthornber/thin-provisioning-tools /tmp/tpt
|
||||
sudo make -C /tmp/tpt install
|
||||
rm -rf /tmp/tpt
|
||||
fi
|
||||
|
||||
# schelk (snapshot rollback tool, invoked via sudo)
|
||||
if ! sudo sh -c 'command -v schelk' &>/dev/null; then
|
||||
cargo install --git https://github.com/tempoxyz/schelk --locked
|
||||
sudo install "$HOME/.cargo/bin/schelk" /usr/local/bin/
|
||||
fi
|
||||
|
||||
- name: Check dependencies
|
||||
run: |
|
||||
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
||||
missing=()
|
||||
for cmd in mc schelk cpupower taskset stdbuf python3 curl make uv pzstd jq; do
|
||||
command -v "$cmd" &>/dev/null || missing+=("$cmd")
|
||||
done
|
||||
if [ ${#missing[@]} -gt 0 ]; then
|
||||
echo "::error::Missing required tools: ${missing[*]}"
|
||||
exit 1
|
||||
fi
|
||||
echo "All dependencies found"
|
||||
|
||||
- name: Resolve display names
|
||||
id: refs
|
||||
run: |
|
||||
BASELINE_SHORT=$(echo "$BASELINE_REF" | cut -c1-8)
|
||||
FEATURE_SHORT=$(echo "$FEATURE_REF" | cut -c1-8)
|
||||
echo "baseline-name=nightly-${BASELINE_SHORT}" >> "$GITHUB_OUTPUT"
|
||||
echo "feature-name=nightly-${FEATURE_SHORT}" >> "$GITHUB_OUTPUT"
|
||||
echo "baseline-ref=$BASELINE_REF" >> "$GITHUB_OUTPUT"
|
||||
echo "feature-ref=$FEATURE_REF" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Check if snapshot needs update
|
||||
id: snapshot-check
|
||||
run: |
|
||||
if .github/scripts/bench-reth-snapshot.sh --check; then
|
||||
echo "needed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "needed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Prepare source dirs
|
||||
run: |
|
||||
if [ -d ../reth-baseline ]; then
|
||||
git -C ../reth-baseline fetch origin "$BASELINE_REF"
|
||||
else
|
||||
git clone . ../reth-baseline
|
||||
fi
|
||||
git -C ../reth-baseline checkout "$BASELINE_REF"
|
||||
|
||||
if [ -d ../reth-feature ]; then
|
||||
git -C ../reth-feature fetch origin "$FEATURE_REF"
|
||||
else
|
||||
git clone . ../reth-feature
|
||||
fi
|
||||
git -C ../reth-feature checkout "$FEATURE_REF"
|
||||
|
||||
- name: Build binaries and download snapshot in parallel
|
||||
id: build
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
BENCH_REPO: ${{ github.repository }}
|
||||
SNAPSHOT_NEEDED: ${{ steps.snapshot-check.outputs.needed }}
|
||||
run: |
|
||||
BASELINE_DIR="$(cd ../reth-baseline && pwd)"
|
||||
FEATURE_DIR="$(cd ../reth-feature && pwd)"
|
||||
|
||||
.github/scripts/bench-reth-build.sh baseline "${BASELINE_DIR}" "$BASELINE_REF" &
|
||||
PID_BASELINE=$!
|
||||
.github/scripts/bench-reth-build.sh feature "${FEATURE_DIR}" "$FEATURE_REF" &
|
||||
PID_FEATURE=$!
|
||||
|
||||
PID_SNAPSHOT=
|
||||
if [ "$SNAPSHOT_NEEDED" = "true" ]; then
|
||||
.github/scripts/bench-reth-snapshot.sh &
|
||||
PID_SNAPSHOT=$!
|
||||
fi
|
||||
|
||||
FAIL=0
|
||||
wait $PID_BASELINE || FAIL=1
|
||||
wait $PID_FEATURE || FAIL=1
|
||||
[ -n "$PID_SNAPSHOT" ] && { wait $PID_SNAPSHOT || FAIL=1; }
|
||||
if [ $FAIL -ne 0 ]; then
|
||||
echo "::error::One or more parallel tasks failed (builds / snapshot download)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# System tuning for reproducible benchmarks
|
||||
- name: System setup
|
||||
run: |
|
||||
sudo cpupower frequency-set -g performance || true
|
||||
# Disable turbo boost (Intel and AMD paths)
|
||||
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo 2>/dev/null || true
|
||||
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 2>/dev/null || true
|
||||
sudo swapoff -a || true
|
||||
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space || true
|
||||
# Disable SMT (hyperthreading)
|
||||
for cpu in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
|
||||
first=$(cut -d, -f1 < "$cpu" | cut -d- -f1)
|
||||
current=$(echo "$cpu" | grep -o 'cpu[0-9]*' | grep -o '[0-9]*')
|
||||
if [ "$current" != "$first" ]; then
|
||||
echo 0 | sudo tee "/sys/devices/system/cpu/cpu${current}/online" || true
|
||||
fi
|
||||
done
|
||||
echo "Online CPUs: $(nproc)"
|
||||
# Disable transparent huge pages
|
||||
for p in /sys/kernel/mm/transparent_hugepage /sys/kernel/mm/transparent_hugepages; do
|
||||
[ -d "$p" ] && echo never | sudo tee "$p/enabled" && echo never | sudo tee "$p/defrag" && break
|
||||
done || true
|
||||
# Prevent deep C-states
|
||||
sudo sh -c 'exec 3<>/dev/cpu_dma_latency; echo -ne "\x00\x00\x00\x00" >&3; sleep infinity' &
|
||||
# Move all IRQs to core 0
|
||||
for irq in /proc/irq/*/smp_affinity_list; do
|
||||
echo 0 | sudo tee "$irq" 2>/dev/null || true
|
||||
done
|
||||
# Stop noisy background services
|
||||
sudo systemctl stop irqbalance cron atd unattended-upgrades snapd 2>/dev/null || true
|
||||
echo "=== Benchmark environment ==="
|
||||
uname -r
|
||||
lscpu | grep -E 'Model name|CPU\(s\)|MHz|NUMA'
|
||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
|
||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
|
||||
cat /sys/kernel/mm/transparent_hugepage/enabled 2>/dev/null || cat /sys/kernel/mm/transparent_hugepages/enabled 2>/dev/null || echo "THP: unknown"
|
||||
free -h
|
||||
|
||||
- name: Pre-flight cleanup
|
||||
run: |
|
||||
sudo pkill -9 reth || true
|
||||
sleep 1
|
||||
if mountpoint -q "$SCHELK_MOUNT"; then
|
||||
sudo umount -l "$SCHELK_MOUNT" || true
|
||||
sudo schelk recover -y || true
|
||||
fi
|
||||
rm -rf "$BENCH_WORK_DIR"
|
||||
mkdir -p "$BENCH_WORK_DIR"
|
||||
|
||||
- name: Start metrics proxy
|
||||
run: |
|
||||
BENCH_ID="nightly-${{ github.run_id }}"
|
||||
BENCH_REFERENCE_EPOCH=$(date +%s)
|
||||
echo "BENCH_ID=${BENCH_ID}" >> "$GITHUB_ENV"
|
||||
echo "BENCH_REFERENCE_EPOCH=${BENCH_REFERENCE_EPOCH}" >> "$GITHUB_ENV"
|
||||
|
||||
LABELS_FILE="/tmp/bench-metrics-labels.json"
|
||||
echo '{}' > "$LABELS_FILE"
|
||||
echo "BENCH_LABELS_FILE=${LABELS_FILE}" >> "$GITHUB_ENV"
|
||||
|
||||
python3 .github/scripts/bench-metrics-proxy.py \
|
||||
--labels "$LABELS_FILE" \
|
||||
--upstream "http://${BENCH_METRICS_ADDR}/" \
|
||||
--subnet 10.10.0.0/24 \
|
||||
--port 9090 &
|
||||
PROXY_PID=$!
|
||||
echo "BENCH_METRICS_PROXY_PID=${PROXY_PID}" >> "$GITHUB_ENV"
|
||||
echo "Metrics proxy started (PID $PROXY_PID)"
|
||||
|
||||
# Interleaved run order (B-F-F-B) to reduce systematic bias
|
||||
- name: "Run benchmark: baseline (1/2)"
|
||||
id: run-baseline-1
|
||||
run: |
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
|
||||
|
||||
- name: "Run benchmark: feature (1/2)"
|
||||
id: run-feature-1
|
||||
run: |
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
|
||||
|
||||
- name: "Run benchmark: feature (2/2)"
|
||||
id: run-feature-2
|
||||
run: |
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
|
||||
|
||||
- name: "Run benchmark: baseline (2/2)"
|
||||
id: run-baseline-2
|
||||
run: |
|
||||
LAST_RUN_START=$(date +%s)
|
||||
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
|
||||
|
||||
- name: Stop metrics proxy & generate Grafana URL
|
||||
id: metrics
|
||||
if: "!cancelled()"
|
||||
run: |
|
||||
kill "$BENCH_METRICS_PROXY_PID" 2>/dev/null || true
|
||||
|
||||
LAST_RUN_DURATION=$(( $(date +%s) - BENCH_LAST_RUN_START ))
|
||||
FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
|
||||
TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
|
||||
GRAFANA_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=ef57fux92e9z4e&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
|
||||
echo "grafana-url=${GRAFANA_URL}" >> "$GITHUB_OUTPUT"
|
||||
echo "Grafana URL: ${GRAFANA_URL}"
|
||||
|
||||
- name: Scan logs for errors
|
||||
if: "!cancelled()"
|
||||
run: |
|
||||
ERRORS_FILE="$BENCH_WORK_DIR/errors.md"
|
||||
found=false
|
||||
for run_dir in baseline-1 feature-1 feature-2 baseline-2; do
|
||||
LOG="$BENCH_WORK_DIR/$run_dir/node.log"
|
||||
if [ ! -f "$LOG" ]; then continue; fi
|
||||
|
||||
panics=$(grep -c -E 'panicked at' "$LOG" || true)
|
||||
errors=$(grep -c ' ERROR ' "$LOG" || true)
|
||||
|
||||
if [ "$panics" -gt 0 ] || [ "$errors" -gt 0 ]; then
|
||||
if [ "$found" = false ]; then
|
||||
printf '### ⚠️ Node Errors\n\n' >> "$ERRORS_FILE"
|
||||
found=true
|
||||
fi
|
||||
printf '<details><summary><b>%s</b>: %d panic(s), %d error(s)</summary>\n\n' "$run_dir" "$panics" "$errors" >> "$ERRORS_FILE"
|
||||
if [ "$panics" -gt 0 ]; then
|
||||
printf '**Panics:**\n```\n' >> "$ERRORS_FILE"
|
||||
grep -E 'panicked at' "$LOG" | head -10 >> "$ERRORS_FILE"
|
||||
printf '```\n' >> "$ERRORS_FILE"
|
||||
fi
|
||||
if [ "$errors" -gt 0 ]; then
|
||||
printf '**Errors (first 20):**\n```\n' >> "$ERRORS_FILE"
|
||||
grep ' ERROR ' "$LOG" | head -20 >> "$ERRORS_FILE"
|
||||
printf '```\n' >> "$ERRORS_FILE"
|
||||
fi
|
||||
printf '\n</details>\n\n' >> "$ERRORS_FILE"
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Parse results
|
||||
id: results
|
||||
if: success()
|
||||
env:
|
||||
BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
|
||||
FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
|
||||
run: |
|
||||
SUMMARY_ARGS="--output-summary $BENCH_WORK_DIR/summary.json"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --output-markdown $BENCH_WORK_DIR/comment.md"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --repo ${{ github.repository }}"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-ref ${BASELINE_REF}"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-name ${BASELINE_NAME}"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-name ${FEATURE_NAME}"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-ref ${FEATURE_REF}"
|
||||
|
||||
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
|
||||
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
|
||||
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BASELINE_CSVS"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $FEATURE_CSVS"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --gas-csv $BENCH_WORK_DIR/feature-1/total_gas.csv"
|
||||
|
||||
GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
|
||||
if [ -n "$GRAFANA_URL" ]; then
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --grafana-url $GRAFANA_URL"
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
python3 .github/scripts/bench-reth-summary.py $SUMMARY_ARGS
|
||||
|
||||
- name: Generate charts
|
||||
if: success()
|
||||
env:
|
||||
BASELINE_NAME: ${{ steps.refs.outputs.baseline-name }}
|
||||
FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
|
||||
run: |
|
||||
CHART_ARGS="--output-dir $BENCH_WORK_DIR/charts"
|
||||
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
|
||||
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
|
||||
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
CHART_ARGS="$CHART_ARGS --feature $FEATURE_CSVS"
|
||||
CHART_ARGS="$CHART_ARGS --baseline $BASELINE_CSVS"
|
||||
CHART_ARGS="$CHART_ARGS --baseline-name ${BASELINE_NAME}"
|
||||
CHART_ARGS="$CHART_ARGS --feature-name ${FEATURE_NAME}"
|
||||
# shellcheck disable=SC2086
|
||||
uv run --with matplotlib python3 .github/scripts/bench-reth-charts.py $CHART_ARGS
|
||||
|
||||
- name: Upload results
|
||||
if: "!cancelled()"
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: bench-scheduled-results
|
||||
path: ${{ env.BENCH_WORK_DIR }}
|
||||
|
||||
- name: Push charts
|
||||
id: push-charts
|
||||
if: success()
|
||||
run: |
|
||||
RUN_ID=${{ github.run_id }}
|
||||
CHART_DIR="nightly/${RUN_ID}"
|
||||
CHARTS_REPO="https://x-access-token:${{ secrets.DEREK_TOKEN }}@github.com/decofe/reth-bench-charts.git"
|
||||
|
||||
TMP_DIR=$(mktemp -d)
|
||||
if git clone --depth 1 "${CHARTS_REPO}" "${TMP_DIR}" 2>/dev/null; then
|
||||
true
|
||||
else
|
||||
git init "${TMP_DIR}"
|
||||
git -C "${TMP_DIR}" remote add origin "${CHARTS_REPO}"
|
||||
fi
|
||||
|
||||
mkdir -p "${TMP_DIR}/${CHART_DIR}"
|
||||
cp "$BENCH_WORK_DIR"/charts/*.png "${TMP_DIR}/${CHART_DIR}/"
|
||||
git -C "${TMP_DIR}" add "${CHART_DIR}"
|
||||
git -C "${TMP_DIR}" -c user.name="github-actions" -c user.email="github-actions@github.com" \
|
||||
commit -m "nightly bench charts for run ${RUN_ID}"
|
||||
git -C "${TMP_DIR}" push origin HEAD:main
|
||||
echo "sha=$(git -C "${TMP_DIR}" rev-parse HEAD)" >> "$GITHUB_OUTPUT"
|
||||
rm -rf "${TMP_DIR}"
|
||||
|
||||
- name: Write job summary
|
||||
if: success()
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const { verdict, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
|
||||
|
||||
let summary;
|
||||
try {
|
||||
summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
|
||||
} catch (e) {
|
||||
await core.summary.addRaw('⚠️ Benchmark completed but failed to load summary.').write();
|
||||
return;
|
||||
}
|
||||
|
||||
const repo = `${context.repo.owner}/${context.repo.repo}`;
|
||||
const commitUrl = `https://github.com/${repo}/commit`;
|
||||
const { emoji, label } = verdict(summary.changes);
|
||||
const baselineLink = `[\`${summary.baseline.name}\`](${commitUrl}/${summary.baseline.ref})`;
|
||||
const featureLink = `[\`${summary.feature.name}\`](${commitUrl}/${summary.feature.ref})`;
|
||||
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
|
||||
|
||||
let md = `# ${emoji} Nightly Regression: ${label}\n\n`;
|
||||
md += `**Baseline:** ${baselineLink}\n`;
|
||||
md += `**Feature:** ${featureLink} ([diff](${diffUrl}))\n`;
|
||||
md += blocksLabel(summary).map(p => `**${p.key}:** ${p.value}`).join(' · ') + '\n\n';
|
||||
|
||||
const rows = metricRows(summary);
|
||||
md += `| Metric | Baseline | Feature | Change |\n`;
|
||||
md += `|--------|----------|---------|--------|\n`;
|
||||
for (const r of rows) {
|
||||
md += `| ${r.label} | ${r.baseline} | ${r.feature} | ${r.change} |\n`;
|
||||
}
|
||||
md += '\n';
|
||||
|
||||
const wtRows = waitTimeRows(summary);
|
||||
if (wtRows.length > 0) {
|
||||
md += `### Wait Time Breakdown\n\n`;
|
||||
md += `| Metric | Baseline | Feature |\n`;
|
||||
md += `|--------|----------|--------|\n`;
|
||||
for (const r of wtRows) {
|
||||
md += `| ${r.title} | ${r.baseline} | ${r.feature} |\n`;
|
||||
}
|
||||
md += '\n';
|
||||
}
|
||||
|
||||
// Charts
|
||||
const chartSha = '${{ steps.push-charts.outputs.sha }}';
|
||||
if (chartSha) {
|
||||
const runId = '${{ github.run_id }}';
|
||||
const baseUrl = `https://raw.githubusercontent.com/decofe/reth-bench-charts/${chartSha}/nightly/${runId}`;
|
||||
const charts = [
|
||||
{ file: 'latency_throughput.png', label: 'Latency, Throughput & Diff' },
|
||||
{ file: 'wait_breakdown.png', label: 'Wait Time Breakdown' },
|
||||
{ file: 'gas_vs_latency.png', label: 'Gas vs Latency' },
|
||||
];
|
||||
md += `### Charts\n\n`;
|
||||
for (const chart of charts) {
|
||||
md += `<details><summary>${chart.label}</summary>\n\n`;
|
||||
md += `\n\n`;
|
||||
md += `</details>\n\n`;
|
||||
}
|
||||
}
|
||||
|
||||
const grafanaUrl = '${{ steps.metrics.outputs.grafana-url }}';
|
||||
if (grafanaUrl) {
|
||||
md += `### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n\n`;
|
||||
}
|
||||
|
||||
try {
|
||||
const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
|
||||
if (errors.trim()) md += '\n' + errors + '\n';
|
||||
} catch {}
|
||||
|
||||
await core.summary.addRaw(md).write();
|
||||
|
||||
- name: Send Slack notification (success)
|
||||
if: success() && env.BENCH_NO_SLACK != 'true'
|
||||
uses: actions/github-script@v8
|
||||
env:
|
||||
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
|
||||
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const { verdict, fmtChange, fmtMs, metricRows, waitTimeRows, blocksLabel } = require('./.github/scripts/bench-utils');
|
||||
|
||||
const token = process.env.SLACK_BENCH_BOT_TOKEN;
|
||||
const channel = process.env.SLACK_BENCH_CHANNEL;
|
||||
if (!token || !channel) {
|
||||
core.info('Slack credentials not set, skipping notification');
|
||||
return;
|
||||
}
|
||||
|
||||
let summary;
|
||||
try {
|
||||
summary = JSON.parse(fs.readFileSync(process.env.BENCH_WORK_DIR + '/summary.json', 'utf8'));
|
||||
} catch (e) {
|
||||
core.warning('Could not read summary.json for Slack notification');
|
||||
return;
|
||||
}
|
||||
|
||||
// Only notify on significant changes (regression OR improvement)
|
||||
const changes = summary.changes || {};
|
||||
const hasSignificant = Object.values(changes).some(c => c.sig === 'good' || c.sig === 'bad');
|
||||
if (!hasSignificant) {
|
||||
core.info('No significant changes detected, skipping nightly Slack notification');
|
||||
return;
|
||||
}
|
||||
|
||||
const SLACK_VERDICT = {
|
||||
'⚠️': ':warning:',
|
||||
'❌': ':x:',
|
||||
'✅': ':white_check_mark:',
|
||||
'⚪': ':white_circle:',
|
||||
};
|
||||
|
||||
const repo = `${context.repo.owner}/${context.repo.repo}`;
|
||||
const { emoji, label } = verdict(changes);
|
||||
const headerEmoji = SLACK_VERDICT[emoji] || emoji;
|
||||
const commitUrl = `https://github.com/${repo}/commit`;
|
||||
const baselineLink = `<${commitUrl}/${summary.baseline.ref}|${summary.baseline.name}>`;
|
||||
const featureLink = `<${commitUrl}/${summary.feature.ref}|${summary.feature.name}>`;
|
||||
const diffUrl = `https://github.com/${repo}/compare/${summary.baseline.ref}...${summary.feature.ref}`;
|
||||
const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
|
||||
|
||||
function cell(text) { return { type: 'raw_text', text: String(text) || ' ' }; }
|
||||
|
||||
const sectionText = [
|
||||
'*Nightly Regression*',
|
||||
'',
|
||||
`*Baseline:* ${baselineLink}`,
|
||||
`*Feature:* ${featureLink}`,
|
||||
blocksLabel(summary).map(p => `*${p.key}:* ${p.value}`).join(' | '),
|
||||
].join('\n');
|
||||
|
||||
const rows = metricRows(summary);
|
||||
const tableRows = [
|
||||
[cell('Metric'), cell('Baseline'), cell('Feature'), cell('Change')],
|
||||
...rows.map(r => [cell(r.label), cell(r.baseline), cell(r.feature), cell(r.change || ' ')]),
|
||||
];
|
||||
|
||||
const blocks = [
|
||||
{
|
||||
type: 'header',
|
||||
text: { type: 'plain_text', text: `${headerEmoji} Nightly: ${label}`, emoji: true },
|
||||
},
|
||||
{
|
||||
type: 'section',
|
||||
text: { type: 'mrkdwn', text: sectionText },
|
||||
},
|
||||
{
|
||||
type: 'table',
|
||||
column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }, { align: 'right' }],
|
||||
rows: tableRows,
|
||||
},
|
||||
{
|
||||
type: 'actions',
|
||||
elements: [
|
||||
{
|
||||
type: 'button',
|
||||
text: { type: 'plain_text', text: 'CI :github:', emoji: true },
|
||||
url: jobUrl,
|
||||
action_id: 'ci_button',
|
||||
},
|
||||
{
|
||||
type: 'button',
|
||||
text: { type: 'plain_text', text: 'Diff :github:', emoji: true },
|
||||
url: diffUrl,
|
||||
action_id: 'diff_button',
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
const text = `Nightly Regression: ${summary.baseline.name} vs ${summary.feature.name}`;
|
||||
const resp = await fetch('https://slack.com/api/chat.postMessage', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Authorization': `Bearer ${token}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({ channel, blocks, text, unfurl_links: false }),
|
||||
});
|
||||
const data = await resp.json();
|
||||
if (!data.ok) {
|
||||
core.warning(`Slack API error: ${JSON.stringify(data)}`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Post wait time breakdown as threaded reply
|
||||
const wtRows = waitTimeRows(summary);
|
||||
if (data.ts && wtRows.length > 0) {
|
||||
const waitTableRows = [
|
||||
[cell('Wait Time'), cell('Baseline'), cell('Feature')],
|
||||
...wtRows.map(r => [cell(r.title), cell(r.baseline), cell(r.feature)]),
|
||||
];
|
||||
await fetch('https://slack.com/api/chat.postMessage', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Authorization': `Bearer ${token}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
channel,
|
||||
thread_ts: data.ts,
|
||||
blocks: [{
|
||||
type: 'table',
|
||||
column_settings: [{ align: 'left' }, { align: 'right' }, { align: 'right' }],
|
||||
rows: waitTableRows,
|
||||
}],
|
||||
text: 'Wait time breakdown',
|
||||
unfurl_links: false,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
- name: Send Slack notification (failure)
|
||||
if: failure()
|
||||
uses: actions/github-script@v8
|
||||
env:
|
||||
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
|
||||
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
|
||||
with:
|
||||
script: |
|
||||
const token = process.env.SLACK_BENCH_BOT_TOKEN;
|
||||
const channel = process.env.SLACK_BENCH_CHANNEL;
|
||||
if (!token || !channel) return;
|
||||
|
||||
const steps_status = [
|
||||
['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
|
||||
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
|
||||
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
|
||||
['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
|
||||
['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
|
||||
];
|
||||
const failed = steps_status.find(([, o]) => o === 'failure');
|
||||
const failedStep = failed ? failed[0] : 'unknown step';
|
||||
|
||||
const repo = `${context.repo.owner}/${context.repo.repo}`;
|
||||
const jobUrl = process.env.BENCH_JOB_URL || `${context.serverUrl}/${repo}/actions/runs/${context.runId}`;
|
||||
|
||||
const blocks = [
|
||||
{
|
||||
type: 'header',
|
||||
text: { type: 'plain_text', text: ':rotating_light: Nightly Bench Failed', emoji: true },
|
||||
},
|
||||
{
|
||||
type: 'section',
|
||||
text: { type: 'mrkdwn', text: `*Nightly regression* failed while *${failedStep}*` },
|
||||
},
|
||||
{
|
||||
type: 'actions',
|
||||
elements: [{
|
||||
type: 'button',
|
||||
text: { type: 'plain_text', text: 'View Logs :github:', emoji: true },
|
||||
url: jobUrl,
|
||||
action_id: 'ci_button',
|
||||
}],
|
||||
},
|
||||
];
|
||||
|
||||
await fetch('https://slack.com/api/chat.postMessage', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Authorization': `Bearer ${token}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
channel,
|
||||
blocks,
|
||||
text: `Nightly bench failed while ${failedStep}`,
|
||||
unfurl_links: false,
|
||||
}),
|
||||
});
|
||||
|
||||
- name: Restore system settings
|
||||
if: always()
|
||||
run: |
|
||||
sudo systemctl start irqbalance cron atd 2>/dev/null || true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Job 3: Save state on success
|
||||
# ---------------------------------------------------------------------------
|
||||
save-state:
|
||||
needs: [resolve-refs, bench-scheduled]
|
||||
if: success()
|
||||
name: save-state
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Write state file
|
||||
run: |
|
||||
mkdir -p .nightly-state
|
||||
echo "${{ needs.resolve-refs.outputs.feature-ref }}" > .nightly-state/last-feature-ref
|
||||
|
||||
- name: Save nightly state
|
||||
uses: actions/cache/save@v4
|
||||
with:
|
||||
path: .nightly-state
|
||||
key: bench-scheduled-state-${{ needs.resolve-refs.outputs.feature-ref }}
|
||||
419
.github/workflows/bench.yml
vendored
419
.github/workflows/bench.yml
vendored
@@ -7,12 +7,16 @@
|
||||
# same block range (snapshot recovered between runs) to compare performance.
|
||||
|
||||
on:
|
||||
# TODO: Disabled temporarily for https://github.com/CodSpeedHQ/runner/issues/55
|
||||
# merge_group:
|
||||
push:
|
||||
branches: [main]
|
||||
issue_comment:
|
||||
types: [created]
|
||||
types: [created, edited]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
blocks:
|
||||
description: "Number of blocks to benchmark (or 'big' for big blocks mode)"
|
||||
description: "Number of blocks to benchmark"
|
||||
required: false
|
||||
default: "500"
|
||||
type: string
|
||||
@@ -31,51 +35,16 @@ on:
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
wait_time:
|
||||
description: "Fixed wait time between blocks (e.g. 500ms, 1s)"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
baseline_args:
|
||||
description: "Extra CLI args for the baseline reth node"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
feature_args:
|
||||
description: "Extra CLI args for the feature reth node"
|
||||
required: false
|
||||
default: ""
|
||||
type: string
|
||||
samply:
|
||||
description: "Enable samply profiling"
|
||||
required: false
|
||||
default: "false"
|
||||
type: boolean
|
||||
reth_newPayload:
|
||||
description: "Use reth_newPayload RPC (server-side timing)"
|
||||
required: false
|
||||
default: "true"
|
||||
type: boolean
|
||||
cores:
|
||||
description: "Limit reth to N CPU cores (0 = all available)"
|
||||
required: false
|
||||
default: "0"
|
||||
type: string
|
||||
no_slack:
|
||||
description: "Suppress Slack notifications for benchmark results"
|
||||
required: false
|
||||
default: "true"
|
||||
type: boolean
|
||||
abba:
|
||||
description: "Run ABBA (BFFB) interleaved order; false = single AB pass"
|
||||
required: false
|
||||
default: "true"
|
||||
type: boolean
|
||||
otlp:
|
||||
description: "Export OTLP traces and logs"
|
||||
required: false
|
||||
default: "true"
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
@@ -91,6 +60,47 @@ permissions:
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
codspeed:
|
||||
if: github.event_name == 'push'
|
||||
continue-on-error: true
|
||||
runs-on: depot-ubuntu-latest
|
||||
concurrency:
|
||||
group: bench-codspeed-${{ github.head_ref || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
strategy:
|
||||
matrix:
|
||||
partition: [1, 2]
|
||||
total_partitions: [2]
|
||||
include:
|
||||
- partition: 1
|
||||
crates: "-p reth-primitives -p reth-trie-common -p reth-trie-sparse"
|
||||
- partition: 2
|
||||
crates: "-p reth-trie"
|
||||
name: codspeed (${{ matrix.partition }}/${{ matrix.total_partitions }})
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
ref: ${{ github.event_name == 'issue_comment' && format('refs/pull/{0}/merge', github.event.issue.number) || '' }}
|
||||
- uses: rui314/setup-mold@v1
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: mozilla-actions/sccache-action@v0.0.9
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
cache-on-failure: true
|
||||
- name: Install cargo-codspeed
|
||||
uses: taiki-e/install-action@v2
|
||||
with:
|
||||
tool: cargo-codspeed
|
||||
- name: Build the benchmark target(s)
|
||||
run: cargo codspeed build --profile profiling --features test-utils ${{ matrix.crates }}
|
||||
- name: Run the benchmarks
|
||||
uses: CodSpeedHQ/action@v4
|
||||
with:
|
||||
run: cargo codspeed run ${{ matrix.crates }}
|
||||
mode: instrumentation
|
||||
token: ${{ secrets.CODSPEED_TOKEN }}
|
||||
|
||||
reth-bench-ack:
|
||||
if: |
|
||||
(github.event_name == 'issue_comment' && github.event.issue.pull_request && (startsWith(github.event.comment.body, '@decofe bench') || startsWith(github.event.comment.body, 'derek bench'))) ||
|
||||
@@ -107,15 +117,7 @@ jobs:
|
||||
baseline-name: ${{ steps.args.outputs.baseline-name }}
|
||||
feature-name: ${{ steps.args.outputs.feature-name }}
|
||||
samply: ${{ steps.args.outputs.samply }}
|
||||
no-slack: ${{ steps.args.outputs.no-slack }}
|
||||
cores: ${{ steps.args.outputs.cores }}
|
||||
big-blocks: ${{ steps.args.outputs.big-blocks }}
|
||||
reth-new-payload: ${{ steps.args.outputs.reth-new-payload }}
|
||||
wait-time: ${{ steps.args.outputs.wait-time }}
|
||||
baseline-args: ${{ steps.args.outputs.baseline-args }}
|
||||
feature-args: ${{ steps.args.outputs.feature-args }}
|
||||
abba: ${{ steps.args.outputs.abba }}
|
||||
otlp: ${{ steps.args.outputs.otlp }}
|
||||
comment-id: ${{ steps.ack.outputs.comment-id }}
|
||||
steps:
|
||||
- name: Check org membership
|
||||
@@ -143,7 +145,7 @@ jobs:
|
||||
with:
|
||||
github-token: ${{ secrets.DEREK_PAT }}
|
||||
script: |
|
||||
let pr, actor, blocks, warmup, baseline, feature, samply, cores, bigBlocks;
|
||||
let pr, actor, blocks, warmup, baseline, feature, samply, cores;
|
||||
|
||||
if (context.eventName === 'workflow_dispatch') {
|
||||
actor = '${{ github.actor }}';
|
||||
@@ -152,15 +154,7 @@ jobs:
|
||||
baseline = '${{ github.event.inputs.baseline }}';
|
||||
feature = '${{ github.event.inputs.feature }}';
|
||||
samply = '${{ github.event.inputs.samply }}' === 'true' ? 'true' : 'false';
|
||||
var noSlack = '${{ github.event.inputs.no_slack }}' !== 'false' ? 'true' : 'false';
|
||||
cores = '${{ github.event.inputs.cores }}' || '0';
|
||||
bigBlocks = blocks === 'big' ? 'true' : 'false';
|
||||
var rethNewPayload = '${{ github.event.inputs.reth_newPayload }}' !== 'false' ? 'true' : 'false';
|
||||
var abba = '${{ github.event.inputs.abba }}' !== 'false' ? 'true' : 'false';
|
||||
var otlp = '${{ github.event.inputs.otlp }}' !== 'false' ? 'true' : 'false';
|
||||
var waitTime = '${{ github.event.inputs.wait_time }}' || '';
|
||||
var baselineNodeArgs = '${{ github.event.inputs.baseline_args }}' || '';
|
||||
var featureNodeArgs = '${{ github.event.inputs.feature_args }}' || '';
|
||||
|
||||
// Find PR for the selected branch
|
||||
const branch = '${{ github.ref_name }}';
|
||||
@@ -180,75 +174,37 @@ jobs:
|
||||
actor = context.payload.comment.user.login;
|
||||
|
||||
const body = context.payload.comment.body.trim();
|
||||
const intArgs = new Set(['warmup', 'cores']);
|
||||
const intOrKeywordArgs = new Map([['blocks', new Set(['big'])]]);
|
||||
const intArgs = new Set(['blocks', 'warmup', 'cores']);
|
||||
const refArgs = new Set(['baseline', 'feature']);
|
||||
const boolArgs = new Set(['samply', 'no-slack']);
|
||||
const boolDefaultTrue = new Set(['reth_newPayload', 'abba', 'otlp']);
|
||||
const durationArgs = new Set(['wait-time']);
|
||||
const stringArgs = new Set(['baseline-args', 'feature-args']);
|
||||
const defaults = { blocks: '500', warmup: '100', baseline: '', feature: '', samply: 'false', 'no-slack': 'false', cores: '0', reth_newPayload: 'true', abba: 'true', otlp: 'true', 'wait-time': '', 'baseline-args': '', 'feature-args': '' };
|
||||
const boolArgs = new Set(['samply']);
|
||||
const defaults = { blocks: '500', warmup: '100', baseline: '', feature: '', samply: 'false', cores: '0' };
|
||||
const unknown = [];
|
||||
const invalid = [];
|
||||
const args = body.replace(/^(?:@decofe|derek) bench\s*/, '');
|
||||
// Parse args, handling quoted values like key="value with spaces"
|
||||
const parts = [];
|
||||
const argRegex = /(\S+?="[^"]*"|\S+?='[^']*'|\S+)/g;
|
||||
let m;
|
||||
while ((m = argRegex.exec(args)) !== null) parts.push(m[1]);
|
||||
for (const part of parts) {
|
||||
for (const part of args.split(/\s+/).filter(Boolean)) {
|
||||
const eq = part.indexOf('=');
|
||||
if (eq === -1) {
|
||||
if (boolArgs.has(part)) {
|
||||
defaults[part] = 'true';
|
||||
} else if (boolDefaultTrue.has(part)) {
|
||||
defaults[part] = 'true';
|
||||
} else {
|
||||
unknown.push(part);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const key = part.slice(0, eq);
|
||||
let value = part.slice(eq + 1);
|
||||
// Strip surrounding quotes
|
||||
if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
|
||||
value = value.slice(1, -1);
|
||||
}
|
||||
if (boolDefaultTrue.has(key)) {
|
||||
if (value === 'true' || value === 'false') {
|
||||
defaults[key] = value;
|
||||
} else {
|
||||
invalid.push(`\`${key}=${value}\` (must be true or false)`);
|
||||
}
|
||||
} else if (durationArgs.has(key)) {
|
||||
if (/^\d+(ms|s|m)$/.test(value)) {
|
||||
defaults[key] = value;
|
||||
} else {
|
||||
invalid.push(`\`${key}=${value}\` (must be a duration like 500ms, 1s, 2m)`);
|
||||
}
|
||||
} else if (intArgs.has(key)) {
|
||||
const value = part.slice(eq + 1);
|
||||
if (intArgs.has(key)) {
|
||||
if (!/^\d+$/.test(value)) {
|
||||
invalid.push(`\`${key}=${value}\` (must be a positive integer)`);
|
||||
} else {
|
||||
defaults[key] = value;
|
||||
}
|
||||
} else if (intOrKeywordArgs.has(key)) {
|
||||
const keywords = intOrKeywordArgs.get(key);
|
||||
if (keywords.has(value)) {
|
||||
defaults[key] = value;
|
||||
} else if (/^\d+$/.test(value)) {
|
||||
defaults[key] = value;
|
||||
} else {
|
||||
invalid.push(`\`${key}=${value}\` (must be a positive integer or one of: ${[...keywords].join(', ')})`);
|
||||
}
|
||||
} else if (refArgs.has(key)) {
|
||||
if (!value) {
|
||||
invalid.push(`\`${key}=\` (must be a git ref)`);
|
||||
} else {
|
||||
defaults[key] = value;
|
||||
}
|
||||
} else if (stringArgs.has(key)) {
|
||||
defaults[key] = value;
|
||||
} else {
|
||||
unknown.push(key);
|
||||
}
|
||||
@@ -257,7 +213,7 @@ jobs:
|
||||
if (unknown.length) errors.push(`Unknown argument(s): \`${unknown.join('`, `')}\``);
|
||||
if (invalid.length) errors.push(`Invalid value(s): ${invalid.join(', ')}`);
|
||||
if (errors.length) {
|
||||
const msg = `❌ **Invalid bench command**\n\n${errors.join('\n')}\n\n**Usage:** \`@decofe bench [blocks=N|big] [warmup=N] [baseline=REF] [feature=REF] [samply] [no-slack] [cores=N] [reth_newPayload=true|false] [abba=true|false] [otlp=true|false] [wait-time=DURATION] [baseline-args="..."] [feature-args="..."]\``;
|
||||
const msg = `❌ **Invalid bench command**\n\n${errors.join('\n')}\n\n**Usage:** \`@decofe bench [blocks=N] [warmup=N] [baseline=REF] [feature=REF] [samply] [cores=N]\``;
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
@@ -272,15 +228,7 @@ jobs:
|
||||
baseline = defaults.baseline;
|
||||
feature = defaults.feature;
|
||||
samply = defaults.samply;
|
||||
var noSlack = defaults['no-slack'];
|
||||
cores = defaults.cores;
|
||||
bigBlocks = blocks === 'big' ? 'true' : 'false';
|
||||
var rethNewPayload = defaults.reth_newPayload;
|
||||
var abba = defaults.abba;
|
||||
var otlp = defaults.otlp;
|
||||
var waitTime = defaults['wait-time'];
|
||||
var baselineNodeArgs = defaults['baseline-args'];
|
||||
var featureNodeArgs = defaults['feature-args'];
|
||||
}
|
||||
|
||||
// Resolve display names for baseline/feature
|
||||
@@ -308,15 +256,7 @@ jobs:
|
||||
core.setOutput('baseline-name', baselineName);
|
||||
core.setOutput('feature-name', featureName);
|
||||
core.setOutput('samply', samply);
|
||||
core.setOutput('no-slack', noSlack);
|
||||
core.setOutput('cores', cores);
|
||||
core.setOutput('big-blocks', bigBlocks);
|
||||
core.setOutput('reth-new-payload', rethNewPayload);
|
||||
core.setOutput('wait-time', waitTime);
|
||||
core.setOutput('baseline-args', baselineNodeArgs);
|
||||
core.setOutput('feature-args', featureNodeArgs);
|
||||
core.setOutput('abba', abba);
|
||||
core.setOutput('otlp', otlp);
|
||||
|
||||
- name: Acknowledge request
|
||||
id: ack
|
||||
@@ -356,6 +296,7 @@ jobs:
|
||||
});
|
||||
allRuns.push(...r);
|
||||
}
|
||||
// Only count runs that trigger reth-bench (not push-triggered codspeed runs)
|
||||
const benchRuns = allRuns.filter(r => r.event === 'issue_comment' || r.event === 'workflow_dispatch');
|
||||
const thisRun = benchRuns.find(r => r.id === context.runId);
|
||||
const thisCreatedAt = thisRun ? new Date(thisRun.created_at) : new Date();
|
||||
@@ -374,26 +315,10 @@ jobs:
|
||||
const baseline = '${{ steps.args.outputs.baseline-name }}';
|
||||
const feature = '${{ steps.args.outputs.feature-name }}';
|
||||
const samply = '${{ steps.args.outputs.samply }}' === 'true';
|
||||
const noSlack = '${{ steps.args.outputs.no-slack }}' === 'true';
|
||||
const bigBlocks = '${{ steps.args.outputs.big-blocks }}' === 'true';
|
||||
const samplyNote = samply ? ', samply: `enabled`' : '';
|
||||
const noSlackNote = noSlack ? ', no-slack' : '';
|
||||
const cores = '${{ steps.args.outputs.cores }}';
|
||||
const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
|
||||
const rethNP = '${{ steps.args.outputs.reth-new-payload }}' !== 'false';
|
||||
const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
|
||||
const abbaEnabled = '${{ steps.args.outputs.abba }}' !== 'false';
|
||||
const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
|
||||
const otlpEnabled = '${{ steps.args.outputs.otlp }}' !== 'false';
|
||||
const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
|
||||
const waitTimeVal = '${{ steps.args.outputs.wait-time }}';
|
||||
const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
|
||||
const baselineArgsVal = '${{ steps.args.outputs.baseline-args }}';
|
||||
const baselineArgsNote = baselineArgsVal ? `, baseline-args: \`${baselineArgsVal}\`` : '';
|
||||
const featureArgsVal = '${{ steps.args.outputs.feature-args }}';
|
||||
const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
|
||||
const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
|
||||
const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
|
||||
const config = `**Config:** ${blocks} blocks, ${warmup} warmup blocks, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${coresNote}`;
|
||||
|
||||
const { data: comment } = await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
@@ -418,26 +343,10 @@ jobs:
|
||||
const baseline = '${{ steps.args.outputs.baseline-name }}';
|
||||
const feature = '${{ steps.args.outputs.feature-name }}';
|
||||
const samply = '${{ steps.args.outputs.samply }}' === 'true';
|
||||
const noSlack = '${{ steps.args.outputs.no-slack }}' === 'true';
|
||||
const bigBlocks = '${{ steps.args.outputs.big-blocks }}' === 'true';
|
||||
const samplyNote = samply ? ', samply: `enabled`' : '';
|
||||
const noSlackNote = noSlack ? ', no-slack' : '';
|
||||
const cores = '${{ steps.args.outputs.cores }}';
|
||||
const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
|
||||
const rethNP = '${{ steps.args.outputs.reth-new-payload }}' !== 'false';
|
||||
const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
|
||||
const abbaEnabled = '${{ steps.args.outputs.abba }}' !== 'false';
|
||||
const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
|
||||
const otlpEnabled = '${{ steps.args.outputs.otlp }}' !== 'false';
|
||||
const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
|
||||
const waitTimeVal = '${{ steps.args.outputs.wait-time }}';
|
||||
const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
|
||||
const baselineArgsVal = '${{ steps.args.outputs.baseline-args }}';
|
||||
const baselineArgsNote = baselineArgsVal ? `, baseline-args: \`${baselineArgsVal}\`` : '';
|
||||
const featureArgsVal = '${{ steps.args.outputs.feature-args }}';
|
||||
const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
|
||||
const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
|
||||
const config = `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`;
|
||||
const config = `**Config:** ${blocks} blocks, ${warmup} warmup blocks, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${coresNote}`;
|
||||
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
|
||||
|
||||
const numRunners = parseInt(process.env.BENCH_RUNNERS) || 1;
|
||||
@@ -489,7 +398,7 @@ jobs:
|
||||
reth-bench:
|
||||
needs: reth-bench-ack
|
||||
name: reth-bench
|
||||
runs-on: [self-hosted, Linux, X64, available]
|
||||
runs-on: [self-hosted, Linux, X64]
|
||||
timeout-minutes: 120
|
||||
env:
|
||||
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
|
||||
@@ -501,22 +410,8 @@ jobs:
|
||||
BENCH_WARMUP_BLOCKS: ${{ needs.reth-bench-ack.outputs.warmup }}
|
||||
BENCH_SAMPLY: ${{ needs.reth-bench-ack.outputs.samply }}
|
||||
BENCH_CORES: ${{ needs.reth-bench-ack.outputs.cores }}
|
||||
BENCH_BIG_BLOCKS: ${{ needs.reth-bench-ack.outputs.big-blocks }}
|
||||
BENCH_RETH_NEW_PAYLOAD: ${{ needs.reth-bench-ack.outputs.reth-new-payload }}
|
||||
BENCH_WAIT_TIME: ${{ needs.reth-bench-ack.outputs.wait-time }}
|
||||
BENCH_BASELINE_ARGS: ${{ needs.reth-bench-ack.outputs.baseline-args }}
|
||||
BENCH_FEATURE_ARGS: ${{ needs.reth-bench-ack.outputs.feature-args }}
|
||||
BENCH_ABBA: ${{ needs.reth-bench-ack.outputs.abba }}
|
||||
BENCH_OTLP: ${{ needs.reth-bench-ack.outputs.otlp }}
|
||||
BENCH_COMMENT_ID: ${{ needs.reth-bench-ack.outputs.comment-id }}
|
||||
BENCH_NO_SLACK: ${{ needs.reth-bench-ack.outputs.no-slack }}
|
||||
BENCH_METRICS_ADDR: "127.0.0.1:9100"
|
||||
BENCH_OTLP_TRACES_ENDPOINT: ${{ needs.reth-bench-ack.outputs.otlp != 'false' && secrets.BENCH_OTLP_TRACES_ENDPOINT || '' }}
|
||||
BENCH_OTLP_LOGS_ENDPOINT: ${{ needs.reth-bench-ack.outputs.otlp != 'false' && secrets.BENCH_OTLP_LOGS_ENDPOINT || '' }}
|
||||
steps:
|
||||
- name: Clean up previous bench-work
|
||||
run: sudo rm -rf "$BENCH_WORK_DIR" 2>/dev/null || true
|
||||
|
||||
- name: Resolve checkout ref
|
||||
id: checkout-ref
|
||||
uses: actions/github-script@v8
|
||||
@@ -531,11 +426,13 @@ jobs:
|
||||
repo: context.repo.repo,
|
||||
pull_number: parseInt(process.env.BENCH_PR),
|
||||
});
|
||||
// Always use head SHA — the merge ref (refs/pull/N/merge) may not
|
||||
// exist if the PR has conflicts, was force-pushed, or was
|
||||
// merged/closed between this step and checkout.
|
||||
core.info(`PR #${process.env.BENCH_PR} (${pr.state}), using head SHA ${pr.head.sha}`);
|
||||
core.setOutput('ref', pr.head.sha);
|
||||
// For closed/merged PRs, the merge ref doesn't exist — use head SHA
|
||||
if (pr.state !== 'open') {
|
||||
core.info(`PR #${process.env.BENCH_PR} is ${pr.state}, using head SHA ${pr.head.sha}`);
|
||||
core.setOutput('ref', pr.head.sha);
|
||||
} else {
|
||||
core.setOutput('ref', `refs/pull/${process.env.BENCH_PR}/merge`);
|
||||
}
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
@@ -563,26 +460,10 @@ jobs:
|
||||
const baseline = '${{ needs.reth-bench-ack.outputs.baseline-name }}';
|
||||
const feature = '${{ needs.reth-bench-ack.outputs.feature-name }}';
|
||||
const samply = process.env.BENCH_SAMPLY === 'true';
|
||||
const noSlack = process.env.BENCH_NO_SLACK === 'true';
|
||||
const bigBlocks = process.env.BENCH_BIG_BLOCKS === 'true';
|
||||
const samplyNote = samply ? ', samply: `enabled`' : '';
|
||||
const noSlackNote = noSlack ? ', no-slack' : '';
|
||||
const cores = process.env.BENCH_CORES || '0';
|
||||
const coresNote = cores && cores !== '0' ? `, cores: \`${cores}\`` : '';
|
||||
const rethNP = (process.env.BENCH_RETH_NEW_PAYLOAD || 'true') !== 'false';
|
||||
const rethNPNote = !rethNP ? ', reth_newPayload: `disabled`' : '';
|
||||
const abbaEnabled = (process.env.BENCH_ABBA || 'true') !== 'false';
|
||||
const abbaNote = !abbaEnabled ? ', abba: `disabled`' : '';
|
||||
const otlpEnabled = (process.env.BENCH_OTLP || 'true') !== 'false';
|
||||
const otlpNote = !otlpEnabled ? ', otlp: `disabled`' : '';
|
||||
const waitTimeVal = process.env.BENCH_WAIT_TIME || '';
|
||||
const waitTimeNote = waitTimeVal ? `, wait-time: \`${waitTimeVal}\`` : '';
|
||||
const baselineArgsVal = process.env.BENCH_BASELINE_ARGS || '';
|
||||
const baselineArgsNote = baselineArgsVal ? `, baseline-args: \`${baselineArgsVal}\`` : '';
|
||||
const featureArgsVal = process.env.BENCH_FEATURE_ARGS || '';
|
||||
const featureArgsNote = featureArgsVal ? `, feature-args: \`${featureArgsVal}\`` : '';
|
||||
const blocksDesc = bigBlocks ? 'blocks: `big`' : `${blocks} blocks, ${warmup} warmup blocks`;
|
||||
core.exportVariable('BENCH_CONFIG', `**Config:** ${blocksDesc}, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${noSlackNote}${coresNote}${rethNPNote}${abbaNote}${otlpNote}${waitTimeNote}${baselineArgsNote}${featureArgsNote}`);
|
||||
core.exportVariable('BENCH_CONFIG', `**Config:** ${blocks} blocks, ${warmup} warmup blocks, baseline: \`${baseline}\`, feature: \`${feature}\`${samplyNote}${coresNote}`);
|
||||
|
||||
const { buildBody } = require('./.github/scripts/bench-update-status.js');
|
||||
await github.rest.issues.updateComment({
|
||||
@@ -842,45 +723,6 @@ jobs:
|
||||
rm -rf "$BENCH_WORK_DIR"
|
||||
mkdir -p "$BENCH_WORK_DIR"
|
||||
|
||||
- name: Download big blocks
|
||||
if: env.BENCH_BIG_BLOCKS == 'true'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
MC="mc --config-dir /home/ubuntu/.mc"
|
||||
BUCKET="minio/reth-snapshots/reth-1-minimal-nightly-previous-big-blocks.tar.zst"
|
||||
BIG_BLOCKS_DIR="${BENCH_WORK_DIR}/big-blocks"
|
||||
rm -rf "$BIG_BLOCKS_DIR"; mkdir -p "$BIG_BLOCKS_DIR"
|
||||
echo "Downloading big blocks from $BUCKET..."
|
||||
$MC cat "$BUCKET" | pzstd -d -p 6 | tar -xf - -C "$BIG_BLOCKS_DIR"
|
||||
echo "Big blocks downloaded to $BIG_BLOCKS_DIR"
|
||||
# Verify expected directory structure
|
||||
if [ ! -d "$BIG_BLOCKS_DIR/gas-ramp-dir" ] || [ ! -d "$BIG_BLOCKS_DIR/payloads" ]; then
|
||||
echo "::error::Big blocks archive missing expected gas-ramp-dir/ or payloads/ directories"
|
||||
ls -laR "$BIG_BLOCKS_DIR"
|
||||
exit 1
|
||||
fi
|
||||
echo "Payload files: $(find "$BIG_BLOCKS_DIR/payloads" -name '*.json' | wc -l)"
|
||||
|
||||
- name: Start metrics proxy
|
||||
run: |
|
||||
BENCH_ID="ci-${{ github.run_id }}"
|
||||
BENCH_REFERENCE_EPOCH=$(date +%s)
|
||||
echo "BENCH_ID=${BENCH_ID}" >> "$GITHUB_ENV"
|
||||
echo "BENCH_REFERENCE_EPOCH=${BENCH_REFERENCE_EPOCH}" >> "$GITHUB_ENV"
|
||||
|
||||
LABELS_FILE="/tmp/bench-metrics-labels.json"
|
||||
echo '{}' > "$LABELS_FILE"
|
||||
echo "BENCH_LABELS_FILE=${LABELS_FILE}" >> "$GITHUB_ENV"
|
||||
|
||||
python3 .github/scripts/bench-metrics-proxy.py \
|
||||
--labels "$LABELS_FILE" \
|
||||
--upstream "http://${BENCH_METRICS_ADDR}/" \
|
||||
--subnet 10.10.0.0/24 \
|
||||
--port 9090 &
|
||||
PROXY_PID=$!
|
||||
echo "BENCH_METRICS_PROXY_PID=${PROXY_PID}" >> "$GITHUB_ENV"
|
||||
echo "Metrics proxy started (PID $PROXY_PID)"
|
||||
|
||||
- name: Update status (running benchmarks)
|
||||
if: success() && env.BENCH_COMMENT_ID
|
||||
uses: actions/github-script@v8
|
||||
@@ -894,64 +736,19 @@ jobs:
|
||||
# thermal drift and cache warming.
|
||||
- name: "Run benchmark: baseline (1/2)"
|
||||
id: run-baseline-1
|
||||
env:
|
||||
BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
|
||||
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-1,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
|
||||
run: |
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"baseline-1","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
|
||||
run: taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-1"
|
||||
|
||||
- name: "Run benchmark: feature (1/2)"
|
||||
id: run-feature-1
|
||||
env:
|
||||
FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
|
||||
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-1,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
|
||||
run: |
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"feature-1","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
|
||||
run: taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-1"
|
||||
|
||||
- name: "Run benchmark: feature (2/2)"
|
||||
if: env.BENCH_ABBA != 'false'
|
||||
id: run-feature-2
|
||||
env:
|
||||
FEATURE_REF: ${{ steps.refs.outputs.feature-ref }}
|
||||
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=feature-2,run_type=feature,git_ref=${{ steps.refs.outputs.feature-ref }}"
|
||||
run: |
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"feature-2","run_type":"feature","git_ref":"${FEATURE_REF}","bench_sha":"${FEATURE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"$(date +%s)","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
|
||||
run: taskset -c 0 .github/scripts/bench-reth-run.sh feature ../reth-feature/target/profiling/reth "$BENCH_WORK_DIR/feature-2"
|
||||
|
||||
- name: "Run benchmark: baseline (2/2)"
|
||||
if: env.BENCH_ABBA != 'false'
|
||||
id: run-baseline-2
|
||||
env:
|
||||
BASELINE_REF: ${{ steps.refs.outputs.baseline-ref }}
|
||||
OTEL_RESOURCE_ATTRIBUTES: "benchmark_id=${{ env.BENCH_ID }},benchmark_run=baseline-2,run_type=baseline,git_ref=${{ steps.refs.outputs.baseline-ref }}"
|
||||
run: |
|
||||
LAST_RUN_START=$(date +%s)
|
||||
echo "BENCH_LAST_RUN_START=${LAST_RUN_START}" >> "$GITHUB_ENV"
|
||||
cat > "$BENCH_LABELS_FILE" <<LABELS
|
||||
{"benchmark_run":"baseline-2","run_type":"baseline","git_ref":"${BASELINE_REF}","bench_sha":"${BASELINE_REF}","benchmark_id":"${BENCH_ID}","run_start_epoch":"${LAST_RUN_START}","reference_epoch":"${BENCH_REFERENCE_EPOCH}"}
|
||||
LABELS
|
||||
taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
|
||||
|
||||
- name: Stop metrics proxy & generate Grafana URL
|
||||
id: metrics
|
||||
if: "!cancelled()"
|
||||
run: |
|
||||
kill "$BENCH_METRICS_PROXY_PID" 2>/dev/null || true
|
||||
|
||||
LAST_RUN_DURATION=$(( $(date +%s) - BENCH_LAST_RUN_START ))
|
||||
FROM_MS=$(( BENCH_REFERENCE_EPOCH * 1000 ))
|
||||
TO_MS=$(( (BENCH_REFERENCE_EPOCH + LAST_RUN_DURATION) * 1000 ))
|
||||
GRAFANA_URL="https://tempoxyz.grafana.net/d/reth-bench-ghr/reth-bench-ghr?orgId=1&from=${FROM_MS}&to=${TO_MS}&timezone=browser&var-datasource=ef57fux92e9z4e&var-job=reth-bench&var-benchmark_id=${BENCH_ID}&var-benchmark_run=\$__all"
|
||||
echo "grafana-url=${GRAFANA_URL}" >> "$GITHUB_OUTPUT"
|
||||
echo "Grafana URL: ${GRAFANA_URL}"
|
||||
run: taskset -c 0 .github/scripts/bench-reth-run.sh baseline ../reth-baseline/target/profiling/reth "$BENCH_WORK_DIR/baseline-2"
|
||||
|
||||
- name: Scan logs for errors
|
||||
if: "!cancelled()"
|
||||
@@ -1053,30 +850,12 @@ jobs:
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-name ${BASELINE_NAME}"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-name ${FEATURE_NAME}"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-ref ${FEATURE_REF}"
|
||||
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
|
||||
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
|
||||
if [ "${BENCH_ABBA:-true}" = "true" ]; then
|
||||
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
fi
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BASELINE_CSVS"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $FEATURE_CSVS"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --baseline-csv $BENCH_WORK_DIR/baseline-1/combined_latency.csv $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --feature-csv $BENCH_WORK_DIR/feature-1/combined_latency.csv $BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --gas-csv $BENCH_WORK_DIR/feature-1/total_gas.csv"
|
||||
if [ "$BEHIND_BASELINE" -gt 0 ]; then
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --behind-baseline $BEHIND_BASELINE"
|
||||
fi
|
||||
if [ "${BENCH_BIG_BLOCKS:-false}" = "true" ]; then
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --big-blocks"
|
||||
# Read gas ramp blocks count from first baseline run (same for all runs)
|
||||
GAS_RAMP_FILE="$BENCH_WORK_DIR/baseline-1/gas_ramp_blocks.txt"
|
||||
if [ -f "$GAS_RAMP_FILE" ]; then
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --gas-ramp-blocks $(cat "$GAS_RAMP_FILE" | tr -d '[:space:]')"
|
||||
fi
|
||||
fi
|
||||
GRAFANA_URL='${{ steps.metrics.outputs.grafana-url }}'
|
||||
if [ -n "$GRAFANA_URL" ]; then
|
||||
SUMMARY_ARGS="$SUMMARY_ARGS --grafana-url $GRAFANA_URL"
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
python3 .github/scripts/bench-reth-summary.py $SUMMARY_ARGS
|
||||
|
||||
@@ -1087,14 +866,8 @@ jobs:
|
||||
FEATURE_NAME: ${{ steps.refs.outputs.feature-name }}
|
||||
run: |
|
||||
CHART_ARGS="--output-dir $BENCH_WORK_DIR/charts"
|
||||
FEATURE_CSVS="$BENCH_WORK_DIR/feature-1/combined_latency.csv"
|
||||
BASELINE_CSVS="$BENCH_WORK_DIR/baseline-1/combined_latency.csv"
|
||||
if [ "${BENCH_ABBA:-true}" = "true" ]; then
|
||||
FEATURE_CSVS="$FEATURE_CSVS $BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
BASELINE_CSVS="$BASELINE_CSVS $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
fi
|
||||
CHART_ARGS="$CHART_ARGS --feature $FEATURE_CSVS"
|
||||
CHART_ARGS="$CHART_ARGS --baseline $BASELINE_CSVS"
|
||||
CHART_ARGS="$CHART_ARGS --feature $BENCH_WORK_DIR/feature-1/combined_latency.csv $BENCH_WORK_DIR/feature-2/combined_latency.csv"
|
||||
CHART_ARGS="$CHART_ARGS --baseline $BENCH_WORK_DIR/baseline-1/combined_latency.csv $BENCH_WORK_DIR/baseline-2/combined_latency.csv"
|
||||
CHART_ARGS="$CHART_ARGS --baseline-name ${BASELINE_NAME}"
|
||||
CHART_ARGS="$CHART_ARGS --feature-name ${FEATURE_NAME}"
|
||||
# shellcheck disable=SC2086
|
||||
@@ -1102,7 +875,7 @@ jobs:
|
||||
|
||||
- name: Upload results
|
||||
if: "!cancelled()"
|
||||
uses: actions/upload-artifact@v7
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: bench-reth-results
|
||||
path: ${{ env.BENCH_WORK_DIR }}
|
||||
@@ -1134,7 +907,7 @@ jobs:
|
||||
rm -rf "${TMP_DIR}"
|
||||
|
||||
- name: Compare & comment
|
||||
if: success() && env.BENCH_COMMENT_ID
|
||||
if: success()
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.DEREK_PAT }}
|
||||
@@ -1170,8 +943,7 @@ jobs:
|
||||
|
||||
// Samply profile links (URLs point directly to Firefox Profiler)
|
||||
if (process.env.BENCH_SAMPLY === 'true') {
|
||||
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
|
||||
const runs = abba ? ['baseline-1', 'feature-1', 'feature-2', 'baseline-2'] : ['baseline-1', 'feature-1'];
|
||||
const runs = ['baseline-1', 'feature-1', 'feature-2', 'baseline-2'];
|
||||
const links = [];
|
||||
for (const run of runs) {
|
||||
try {
|
||||
@@ -1186,12 +958,6 @@ jobs:
|
||||
}
|
||||
}
|
||||
|
||||
// Grafana dashboard link
|
||||
const grafanaUrl = '${{ steps.metrics.outputs.grafana-url }}';
|
||||
if (grafanaUrl) {
|
||||
comment += `\n\n### Grafana Dashboard\n\n[View real-time metrics](${grafanaUrl})\n`;
|
||||
}
|
||||
|
||||
// Node errors (panics / ERROR logs)
|
||||
try {
|
||||
const errors = fs.readFileSync(process.env.BENCH_WORK_DIR + '/errors.md', 'utf8');
|
||||
@@ -1211,24 +977,13 @@ jobs:
|
||||
comment_id: parseInt(ackCommentId),
|
||||
body,
|
||||
});
|
||||
} else {
|
||||
// No PR — write results to job summary
|
||||
await core.summary.addRaw(body).write();
|
||||
}
|
||||
|
||||
- name: Write job summary
|
||||
if: success()
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const jobSummary = require('./.github/scripts/bench-job-summary.js');
|
||||
await jobSummary({
|
||||
core,
|
||||
context,
|
||||
chartSha: '${{ steps.push-charts.outputs.sha }}',
|
||||
grafanaUrl: '${{ steps.metrics.outputs.grafana-url }}',
|
||||
runId: '${{ github.run_id }}',
|
||||
});
|
||||
|
||||
- name: Send Slack notification (success)
|
||||
if: success() && env.BENCH_NO_SLACK != 'true'
|
||||
if: success()
|
||||
uses: actions/github-script@v8
|
||||
env:
|
||||
SLACK_BENCH_BOT_TOKEN: ${{ secrets.SLACK_BENCH_BOT_TOKEN }}
|
||||
@@ -1244,13 +999,12 @@ jobs:
|
||||
with:
|
||||
github-token: ${{ secrets.DEREK_PAT }}
|
||||
script: |
|
||||
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
|
||||
const steps_status = [
|
||||
['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
|
||||
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
|
||||
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
|
||||
...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
|
||||
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
|
||||
['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
|
||||
['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
|
||||
];
|
||||
const failed = steps_status.find(([, o]) => o === 'failure');
|
||||
const failedStep = failed ? failed[0] : 'unknown step';
|
||||
@@ -1279,13 +1033,12 @@ jobs:
|
||||
SLACK_BENCH_CHANNEL: ${{ secrets.SLACK_BENCH_CHANNEL }}
|
||||
with:
|
||||
script: |
|
||||
const abba = (process.env.BENCH_ABBA || 'true') !== 'false';
|
||||
const steps_status = [
|
||||
['building binaries${{ steps.snapshot-check.outputs.needed == 'true' && ' & downloading snapshot' || '' }}', '${{ steps.build.outcome }}'],
|
||||
['running baseline benchmark (1/2)', '${{ steps.run-baseline-1.outcome }}'],
|
||||
['running feature benchmark (1/2)', '${{ steps.run-feature-1.outcome }}'],
|
||||
...(abba ? [['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}']] : []),
|
||||
...(abba ? [['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}']] : []),
|
||||
['running feature benchmark (2/2)', '${{ steps.run-feature-2.outcome }}'],
|
||||
['running baseline benchmark (2/2)', '${{ steps.run-baseline-2.outcome }}'],
|
||||
];
|
||||
const failed = steps_status.find(([, o]) => o === 'failure');
|
||||
const failedStep = failed ? failed[0] : 'unknown step';
|
||||
|
||||
4
.github/workflows/docker-test.yml
vendored
4
.github/workflows/docker-test.yml
vendored
@@ -6,7 +6,7 @@ on:
|
||||
hive_target:
|
||||
required: true
|
||||
type: string
|
||||
description: "Docker bake target to build (e.g. hive)"
|
||||
description: "Docker bake target to build (e.g. hive-stable, hive-edge)"
|
||||
artifact_name:
|
||||
required: false
|
||||
type: string
|
||||
@@ -76,7 +76,7 @@ jobs:
|
||||
*.dockerfile=Dockerfile
|
||||
|
||||
- name: Upload reth image
|
||||
uses: actions/upload-artifact@v7
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: ${{ inputs.artifact_name }}
|
||||
path: ./artifacts
|
||||
|
||||
49
.github/workflows/docker.yml
vendored
49
.github/workflows/docker.yml
vendored
@@ -28,30 +28,12 @@ on:
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
pgo:
|
||||
description: "Enable PGO profiling"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
pgo_blocks:
|
||||
description: "Number of blocks to execute for PGO profiling"
|
||||
required: false
|
||||
type: string
|
||||
default: "20"
|
||||
|
||||
jobs:
|
||||
collect-pgo-profile:
|
||||
if: github.repository == 'paradigmxyz/reth' && github.event_name == 'workflow_dispatch' && inputs.pgo
|
||||
uses: ./.github/workflows/pgo-profile.yml
|
||||
with:
|
||||
pgo_blocks: ${{ inputs.pgo_blocks || '20' }}
|
||||
secrets: inherit
|
||||
|
||||
build:
|
||||
if: github.repository == 'paradigmxyz/reth' && !failure() && !cancelled()
|
||||
if: github.repository == 'paradigmxyz/reth'
|
||||
name: Build Docker images
|
||||
runs-on: ubuntu-24.04
|
||||
needs: collect-pgo-profile
|
||||
permissions:
|
||||
packages: write
|
||||
contents: read
|
||||
@@ -63,7 +45,7 @@ jobs:
|
||||
uses: depot/setup-action@v1
|
||||
|
||||
- name: Log in to GHCR
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
@@ -76,30 +58,6 @@ jobs:
|
||||
echo "describe=$(git describe --always --tags)" >> "$GITHUB_OUTPUT"
|
||||
echo "dirty=false" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Download pre-collected PGO profile
|
||||
if: ${{ github.event_name == 'workflow_dispatch' && inputs.pgo }}
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: pgo-profdata
|
||||
path: dist
|
||||
|
||||
- name: Configure PGO build args
|
||||
id: pgo
|
||||
run: |
|
||||
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]] && [[ "${{ inputs.pgo }}" == "true" ]]; then
|
||||
if [ ! -f dist/merged.profdata ]; then
|
||||
echo "::error::Expected dist/merged.profdata from collect-pgo-profile job"
|
||||
exit 1
|
||||
fi
|
||||
echo "use_pgo_bolt=true" >> "$GITHUB_OUTPUT"
|
||||
echo "pgo_profdata=dist/merged.profdata" >> "$GITHUB_OUTPUT"
|
||||
echo "Using pre-collected PGO profile from collect-pgo-profile job"
|
||||
else
|
||||
echo "use_pgo_bolt=false" >> "$GITHUB_OUTPUT"
|
||||
echo "pgo_profdata=" >> "$GITHUB_OUTPUT"
|
||||
echo "PGO disabled"
|
||||
fi
|
||||
|
||||
- name: Determine build parameters
|
||||
id: params
|
||||
run: |
|
||||
@@ -149,9 +107,6 @@ jobs:
|
||||
push: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }}
|
||||
set: |
|
||||
${{ steps.params.outputs.ethereum_set }}
|
||||
*.args.USE_PGO_BOLT=${{ steps.pgo.outputs.use_pgo_bolt }}
|
||||
*.args.PGO_PROFDATA=${{ steps.pgo.outputs.pgo_profdata }}
|
||||
*.args.STRIP_SYMBOLS=false
|
||||
|
||||
- name: Verify image architectures
|
||||
env:
|
||||
|
||||
2
.github/workflows/e2e.yml
vendored
2
.github/workflows/e2e.yml
vendored
@@ -63,6 +63,6 @@ jobs:
|
||||
run: |
|
||||
cargo nextest run \
|
||||
--no-fail-fast \
|
||||
--locked \
|
||||
--locked --features "edge" \
|
||||
-p reth-e2e-test-utils \
|
||||
-E 'binary(rocksdb)'
|
||||
|
||||
27
.github/workflows/hive.yml
vendored
27
.github/workflows/hive.yml
vendored
@@ -15,11 +15,18 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build-reth:
|
||||
build-reth-stable:
|
||||
uses: ./.github/workflows/docker-test.yml
|
||||
with:
|
||||
hive_target: hive
|
||||
artifact_name: "reth"
|
||||
hive_target: hive-stable
|
||||
artifact_name: "reth-stable"
|
||||
secrets: inherit
|
||||
|
||||
build-reth-edge:
|
||||
uses: ./.github/workflows/docker-test.yml
|
||||
with:
|
||||
hive_target: hive-edge
|
||||
artifact_name: "reth-edge"
|
||||
secrets: inherit
|
||||
|
||||
prepare-hive:
|
||||
@@ -68,7 +75,7 @@ jobs:
|
||||
chmod +x hive
|
||||
|
||||
- name: Upload hive assets
|
||||
uses: actions/upload-artifact@v7
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: hive_assets
|
||||
path: ./hive_assets
|
||||
@@ -77,6 +84,7 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
storage: [stable, edge]
|
||||
# ethereum/rpc to be deprecated:
|
||||
# https://github.com/ethereum/hive/pull/1117
|
||||
scenario:
|
||||
@@ -176,9 +184,10 @@ jobs:
|
||||
- sim: ethereum/eels/consume-rlp
|
||||
limit: .*tests/paris.*
|
||||
needs:
|
||||
- build-reth
|
||||
- build-reth-stable
|
||||
- build-reth-edge
|
||||
- prepare-hive
|
||||
name: ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
|
||||
name: ${{ matrix.storage }} / ${{ matrix.scenario.sim }}${{ matrix.scenario.limit && format(' - {0}', matrix.scenario.limit) }}
|
||||
# Use larger runners for eels tests to avoid OOM runner crashes
|
||||
runs-on: ${{ github.repository == 'paradigmxyz/reth' && (contains(matrix.scenario.sim, 'eels') && 'depot-ubuntu-latest-8' || 'depot-ubuntu-latest-4') || 'ubuntu-latest' }}
|
||||
permissions:
|
||||
@@ -189,15 +198,15 @@ jobs:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Download hive assets
|
||||
uses: actions/download-artifact@v8
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: hive_assets
|
||||
path: /tmp
|
||||
|
||||
- name: Download reth image
|
||||
uses: actions/download-artifact@v8
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: reth
|
||||
name: reth-${{ matrix.storage }}
|
||||
path: /tmp
|
||||
|
||||
- name: Load Docker images
|
||||
|
||||
5
.github/workflows/integration.yml
vendored
5
.github/workflows/integration.yml
vendored
@@ -22,7 +22,7 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
name: test / ${{ matrix.network }}
|
||||
name: test / ${{ matrix.network }} / ${{ matrix.storage }}
|
||||
if: github.event_name != 'schedule'
|
||||
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
|
||||
env:
|
||||
@@ -30,6 +30,7 @@ jobs:
|
||||
strategy:
|
||||
matrix:
|
||||
network: ["ethereum"]
|
||||
storage: ["stable", "edge"]
|
||||
timeout-minutes: 60
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
@@ -46,7 +47,7 @@ jobs:
|
||||
run: |
|
||||
cargo nextest run \
|
||||
--no-fail-fast \
|
||||
--locked --features "asm-keccak ${{ matrix.network }}" \
|
||||
--locked --features "asm-keccak ${{ matrix.network }} ${{ matrix.storage == 'edge' && 'edge' || '' }}" \
|
||||
--workspace --exclude ef-tests \
|
||||
-E "kind(test) and not binary(e2e_testsuite)"
|
||||
|
||||
|
||||
2
.github/workflows/kurtosis.yml
vendored
2
.github/workflows/kurtosis.yml
vendored
@@ -40,7 +40,7 @@ jobs:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Download reth image
|
||||
uses: actions/download-artifact@v8
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: artifacts
|
||||
path: /tmp
|
||||
|
||||
107
.github/workflows/pgo-profile.yml
vendored
107
.github/workflows/pgo-profile.yml
vendored
@@ -1,107 +0,0 @@
|
||||
name: pgo-profile
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
pgo_blocks:
|
||||
description: "Number of blocks to execute for PGO profiling"
|
||||
required: false
|
||||
type: string
|
||||
default: "20"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pgo_blocks:
|
||||
description: "Number of blocks to execute for PGO profiling"
|
||||
required: false
|
||||
type: string
|
||||
default: "20"
|
||||
|
||||
jobs:
|
||||
collect:
|
||||
name: collect PGO profiles
|
||||
runs-on: [self-hosted, Linux, X64]
|
||||
timeout-minutes: 180
|
||||
env:
|
||||
SCHELK_MOUNT: /reth-bench
|
||||
BENCH_RPC_URL: https://ethereum.reth.rs/rpc
|
||||
RUSTC_WRAPPER: "sccache"
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
- uses: rui314/setup-mold@v1
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
target: x86_64-unknown-linux-gnu
|
||||
- uses: mozilla-actions/sccache-action@v0.0.9
|
||||
continue-on-error: true
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
cache-on-failure: true
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y --no-install-recommends \
|
||||
dmsetup lsb-release wget linux-tools-"$(uname -r)" || \
|
||||
sudo apt-get install -y --no-install-recommends linux-tools-generic
|
||||
|
||||
- name: Download snapshot if needed
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
BENCH_REPO: ${{ github.repository }}
|
||||
run: |
|
||||
if ! .github/scripts/bench-reth-snapshot.sh --check; then
|
||||
echo "Snapshot outdated or missing, downloading..."
|
||||
.github/scripts/bench-reth-snapshot.sh
|
||||
fi
|
||||
|
||||
- name: Mount snapshot
|
||||
run: |
|
||||
sudo pkill -9 reth || true
|
||||
sleep 1
|
||||
if mountpoint -q "$SCHELK_MOUNT"; then
|
||||
sudo umount -l "$SCHELK_MOUNT" || true
|
||||
sudo schelk recover -y || true
|
||||
fi
|
||||
sudo schelk mount -y
|
||||
sync
|
||||
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
|
||||
|
||||
- name: Collect PGO profile
|
||||
run: |
|
||||
DATADIR="$SCHELK_MOUNT/datadir" \
|
||||
RPC_URL="$BENCH_RPC_URL" \
|
||||
PGO_BLOCKS="${{ inputs.pgo_blocks || '20' }}" \
|
||||
BOLT_BLOCKS="${{ inputs.pgo_blocks || '20' }}" \
|
||||
COLLECT_PGO_ONLY=true \
|
||||
SKIP_BOLT=true \
|
||||
PROFILE=maxperf-symbols \
|
||||
FEATURES="jemalloc,asm-keccak,min-debug-logs" \
|
||||
TARGET=x86_64-unknown-linux-gnu \
|
||||
EXTRA_RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq" \
|
||||
.github/scripts/build_pgo_bolt.sh
|
||||
|
||||
- name: Show PGO profile stats
|
||||
run: |
|
||||
LLVM_PROFDATA=$(find "$(rustc --print sysroot)" -name llvm-profdata -type f | head -1)
|
||||
if [ -z "$LLVM_PROFDATA" ]; then
|
||||
echo "::error::llvm-profdata not found in rust toolchain"
|
||||
exit 1
|
||||
fi
|
||||
"$LLVM_PROFDATA" show --detailed-summary --topn=20 target/pgo-profiles/merged.profdata
|
||||
|
||||
- name: Upload PGO profile
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: pgo-profdata
|
||||
path: target/pgo-profiles/merged.profdata
|
||||
retention-days: 1
|
||||
|
||||
- name: Recover snapshot
|
||||
if: always()
|
||||
run: |
|
||||
if mountpoint -q "$SCHELK_MOUNT"; then
|
||||
sudo umount -l "$SCHELK_MOUNT" || true
|
||||
sudo schelk recover -y || true
|
||||
fi
|
||||
30
.github/workflows/pr-audit.yml
vendored
30
.github/workflows/pr-audit.yml
vendored
@@ -1,30 +0,0 @@
|
||||
name: Pull request audit
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event.label.name == 'cyclops'
|
||||
steps:
|
||||
- name: Publish event
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "${{ secrets.EVENTS_KEY }}" > ${{ runner.temp }}/key
|
||||
echo "${{ secrets.EVENTS_CERT }}" > ${{ runner.temp }}/cert
|
||||
|
||||
curl -sf -o /dev/null -X POST ${{ secrets.EVENTS_ARGS }} \
|
||||
-H "Content-Type: application/json" \
|
||||
--key ${{ runner.temp }}/key \
|
||||
--cert ${{ runner.temp }}/cert \
|
||||
-d '{
|
||||
"repository": "${{ github.repository }}",
|
||||
"event": "pr_audit",
|
||||
"data": {
|
||||
"pr_number": ${{ github.event.pull_request.number }},
|
||||
"sha": "${{ github.event.pull_request.head.sha }}"
|
||||
}
|
||||
}'
|
||||
2
.github/workflows/release-reproducible.yml
vendored
2
.github/workflows/release-reproducible.yml
vendored
@@ -52,7 +52,7 @@ jobs:
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Log in to GitHub Container Registry
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
|
||||
118
.github/workflows/release.yml
vendored
118
.github/workflows/release.yml
vendored
@@ -13,14 +13,6 @@ on:
|
||||
description: "Enable dry run mode (builds artifacts but skips uploads and release creation)"
|
||||
type: boolean
|
||||
default: false
|
||||
pgo:
|
||||
description: "Enable PGO profiling"
|
||||
type: boolean
|
||||
default: false
|
||||
pgo_blocks:
|
||||
description: "Number of blocks to execute for PGO profiling on self-hosted runner"
|
||||
type: string
|
||||
default: "20"
|
||||
|
||||
env:
|
||||
REPO_NAME: ${{ github.repository_owner }}/reth
|
||||
@@ -45,7 +37,7 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Extract version
|
||||
run: echo "VERSION=${GITHUB_REF_NAME//\//-}" >> $GITHUB_OUTPUT
|
||||
run: echo "VERSION=${GITHUB_REF_NAME}" >> $GITHUB_OUTPUT
|
||||
id: extract_version
|
||||
outputs:
|
||||
VERSION: ${{ steps.extract_version.outputs.VERSION }}
|
||||
@@ -77,22 +69,22 @@ jobs:
|
||||
fail-fast: true
|
||||
matrix:
|
||||
configs:
|
||||
- target: x86_64-unknown-linux-gnu
|
||||
os: ubuntu-24.04
|
||||
profile: maxperf
|
||||
allow_fail: false
|
||||
- target: aarch64-unknown-linux-gnu
|
||||
os: ubuntu-24.04-arm
|
||||
profile: maxperf
|
||||
allow_fail: false
|
||||
rustflags: ""
|
||||
native: true
|
||||
- target: x86_64-apple-darwin
|
||||
os: macos-14
|
||||
profile: maxperf
|
||||
allow_fail: false
|
||||
rustflags: "-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"
|
||||
- target: aarch64-apple-darwin
|
||||
os: macos-14
|
||||
profile: maxperf
|
||||
allow_fail: false
|
||||
rustflags: ""
|
||||
build:
|
||||
- command: build
|
||||
binary: reth
|
||||
@@ -104,7 +96,6 @@ jobs:
|
||||
target: ${{ matrix.configs.target }}
|
||||
- uses: mozilla-actions/sccache-action@v0.0.9
|
||||
- name: Install cross main
|
||||
if: ${{ !matrix.configs.native }}
|
||||
id: cross_main
|
||||
run: |
|
||||
cargo install cross --locked --git https://github.com/cross-rs/cross
|
||||
@@ -119,12 +110,7 @@ jobs:
|
||||
echo "MACOSX_DEPLOYMENT_TARGET=$(xcrun -sdk macosx --show-sdk-platform-version)" >> $GITHUB_ENV
|
||||
|
||||
- name: Build Reth
|
||||
run: |
|
||||
if [ "${{ matrix.configs.native }}" = "true" ]; then
|
||||
make PROFILE=${{ matrix.configs.profile }} EXTRA_RUSTFLAGS="${{ matrix.configs.rustflags }}" ${{ matrix.build.command }}-native-${{ matrix.configs.target }}
|
||||
else
|
||||
make PROFILE=${{ matrix.configs.profile }} EXTRA_RUSTFLAGS="${{ matrix.configs.rustflags }}" ${{ matrix.build.command }}-${{ matrix.configs.target }}
|
||||
fi
|
||||
run: make PROFILE=${{ matrix.configs.profile }} ${{ matrix.build.command }}-${{ matrix.configs.target }}
|
||||
- name: Move binary
|
||||
run: |
|
||||
mkdir artifacts
|
||||
@@ -145,105 +131,23 @@ jobs:
|
||||
|
||||
- name: Upload artifact
|
||||
if: ${{ github.event.inputs.dry_run != 'true' }}
|
||||
uses: actions/upload-artifact@v7
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
|
||||
path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz
|
||||
|
||||
- name: Upload signature
|
||||
if: ${{ github.event.inputs.dry_run != 'true' }}
|
||||
uses: actions/upload-artifact@v7
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
|
||||
path: ${{ matrix.build.binary }}-${{ needs.extract-version.outputs.VERSION }}-${{ matrix.configs.target }}.tar.gz.asc
|
||||
|
||||
collect-pgo-profile:
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.pgo
|
||||
uses: ./.github/workflows/pgo-profile.yml
|
||||
with:
|
||||
pgo_blocks: ${{ inputs.pgo_blocks || '20' }}
|
||||
secrets: inherit
|
||||
|
||||
build-pgo:
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.pgo
|
||||
name: build release (x86_64-linux PGO+BOLT)
|
||||
runs-on: [self-hosted, Linux, X64]
|
||||
needs: [extract-version, collect-pgo-profile]
|
||||
timeout-minutes: 120
|
||||
env:
|
||||
RUSTC_WRAPPER: "sccache"
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
- uses: rui314/setup-mold@v1
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
target: x86_64-unknown-linux-gnu
|
||||
- uses: mozilla-actions/sccache-action@v0.0.9
|
||||
continue-on-error: true
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
cache-on-failure: true
|
||||
|
||||
- name: Download pre-collected PGO profile
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: pgo-profdata
|
||||
path: dist
|
||||
|
||||
- name: Verify PGO profile artifact
|
||||
run: |
|
||||
test -f dist/merged.profdata
|
||||
ls -lh dist/merged.profdata
|
||||
|
||||
- name: Build Reth with PGO+BOLT
|
||||
run: |
|
||||
SKIP_BOLT=true \
|
||||
PGO_PROFDATA="$PWD/dist/merged.profdata" \
|
||||
PROFILE=maxperf-symbols \
|
||||
FEATURES="jemalloc,asm-keccak,min-debug-logs" \
|
||||
TARGET=x86_64-unknown-linux-gnu \
|
||||
EXTRA_RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq" \
|
||||
.github/scripts/build_pgo_bolt.sh
|
||||
|
||||
- name: Move binary
|
||||
run: |
|
||||
mkdir artifacts
|
||||
mv target/maxperf-symbols/reth ./artifacts
|
||||
|
||||
- name: Configure GPG and create artifacts
|
||||
env:
|
||||
GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }}
|
||||
GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
|
||||
run: |
|
||||
export GPG_TTY=$(tty)
|
||||
echo -n "$GPG_SIGNING_KEY" | base64 --decode | gpg --batch --import
|
||||
cd artifacts
|
||||
tar -czf reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz reth*
|
||||
echo "$GPG_PASSPHRASE" | gpg --passphrase-fd 0 --pinentry-mode loopback --batch -ab reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
|
||||
mv *tar.gz* ..
|
||||
shell: bash
|
||||
|
||||
- name: Upload artifact
|
||||
if: ${{ github.event.inputs.dry_run != 'true' }}
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
|
||||
path: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz
|
||||
|
||||
- name: Upload signature
|
||||
if: ${{ github.event.inputs.dry_run != 'true' }}
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz.asc
|
||||
path: reth-${{ needs.extract-version.outputs.VERSION }}-x86_64-unknown-linux-gnu.tar.gz.asc
|
||||
|
||||
draft-release:
|
||||
name: draft release
|
||||
runs-on: ubuntu-latest
|
||||
needs: [build, build-pgo, extract-version]
|
||||
if: ${{ !failure() && !cancelled() && github.event.inputs.dry_run != 'true' }}
|
||||
needs: [build, extract-version]
|
||||
if: ${{ github.event.inputs.dry_run != 'true' }}
|
||||
env:
|
||||
VERSION: ${{ needs.extract-version.outputs.VERSION }}
|
||||
permissions:
|
||||
@@ -256,7 +160,7 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: Download artifacts
|
||||
uses: actions/download-artifact@v8
|
||||
uses: actions/download-artifact@v7
|
||||
- name: Generate full changelog
|
||||
id: changelog
|
||||
run: |
|
||||
|
||||
6
.github/workflows/reproducible-build.yml
vendored
6
.github/workflows/reproducible-build.yml
vendored
@@ -43,7 +43,7 @@ jobs:
|
||||
echo "Binaries SHA256 on ${{ matrix.machine }}: $(cat checksum.sha256)"
|
||||
|
||||
- name: Upload the hash
|
||||
uses: actions/upload-artifact@v7
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: checksum-${{ matrix.machine }}
|
||||
path: |
|
||||
@@ -56,12 +56,12 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Download artifacts from machine-1
|
||||
uses: actions/download-artifact@v8
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: checksum-machine-1
|
||||
path: machine-1/
|
||||
- name: Download artifacts from machine-2
|
||||
uses: actions/download-artifact@v8
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: checksum-machine-2
|
||||
path: machine-2/
|
||||
|
||||
15
.github/workflows/stage.yml
vendored
15
.github/workflows/stage.yml
vendored
@@ -51,12 +51,15 @@ jobs:
|
||||
- name: Run execution stage
|
||||
run: |
|
||||
reth stage run execution --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
|
||||
# NOTE: account-hashing, storage-hashing, and hashing stages are omitted.
|
||||
# With storage v2 (now default), these stages are no-ops because the
|
||||
# execution stage writes directly to HashedAccounts/HashedStorages.
|
||||
# Running them here is harmful: `stage run` unwinds before executing,
|
||||
# and the unwind reverts the hashed state that execution wrote, but
|
||||
# the no-op execute never restores it — causing merkle to fail.
|
||||
- name: Run account-hashing stage
|
||||
run: |
|
||||
reth stage run account-hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
|
||||
- name: Run storage hashing stage
|
||||
run: |
|
||||
reth stage run storage-hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
|
||||
- name: Run hashing stage
|
||||
run: |
|
||||
reth stage run hashing --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
|
||||
- name: Run merkle stage
|
||||
run: |
|
||||
reth stage run merkle --from ${{ env.FROM_BLOCK }} --to ${{ env.TO_BLOCK }} --commit --checkpoints
|
||||
|
||||
8
.github/workflows/unit.yml
vendored
8
.github/workflows/unit.yml
vendored
@@ -19,13 +19,15 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
name: test / ${{ matrix.type }}
|
||||
name: test / ${{ matrix.type }} / ${{ matrix.storage }}
|
||||
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
|
||||
env:
|
||||
RUST_BACKTRACE: 1
|
||||
EDGE_FEATURES: ${{ matrix.storage == 'edge' && 'edge' || '' }}
|
||||
strategy:
|
||||
matrix:
|
||||
type: [ethereum]
|
||||
storage: [stable, edge]
|
||||
include:
|
||||
- type: ethereum
|
||||
features: asm-keccak ethereum
|
||||
@@ -48,14 +50,14 @@ jobs:
|
||||
run: |
|
||||
cargo nextest run \
|
||||
--no-fail-fast \
|
||||
--features "${{ matrix.features }}" --locked \
|
||||
--features "${{ matrix.features }} $EDGE_FEATURES" --locked \
|
||||
${{ matrix.exclude_args }} --workspace \
|
||||
--exclude ef-tests --no-tests=warn \
|
||||
-E "!kind(test) and not binary(e2e_testsuite)"
|
||||
|
||||
state:
|
||||
name: Ethereum state tests
|
||||
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-8' || 'ubuntu-latest' }}
|
||||
runs-on: ${{ github.repository == 'paradigmxyz/reth' && 'depot-ubuntu-latest-4' || 'ubuntu-latest' }}
|
||||
env:
|
||||
RUST_LOG: info,sync=error
|
||||
RUST_BACKTRACE: 1
|
||||
|
||||
972
Cargo.lock
generated
972
Cargo.lock
generated
File diff suppressed because it is too large
Load Diff
107
Cargo.toml
107
Cargo.toml
@@ -1,5 +1,5 @@
|
||||
[workspace.package]
|
||||
version = "1.11.3"
|
||||
version = "1.11.1"
|
||||
edition = "2024"
|
||||
rust-version = "1.93"
|
||||
license = "MIT OR Apache-2.0"
|
||||
@@ -170,7 +170,6 @@ rust.rust_2018_idioms = { level = "deny", priority = -1 }
|
||||
rust.unreachable_pub = "warn"
|
||||
rust.unused_must_use = "deny"
|
||||
rust.rust_2024_incompatible_pat = "warn"
|
||||
rust.unexpected_cfgs = { level = "warn", check-cfg = ['cfg(tokio_unstable)'] }
|
||||
rustdoc.all = "warn"
|
||||
# rust.unnameable-types = "warn"
|
||||
|
||||
@@ -399,9 +398,7 @@ reth-payload-builder-primitives = { path = "crates/payload/builder-primitives" }
|
||||
reth-payload-primitives = { path = "crates/payload/primitives" }
|
||||
reth-payload-validator = { path = "crates/payload/validator" }
|
||||
reth-payload-util = { path = "crates/payload/util" }
|
||||
reth-primitives = { path = "crates/primitives", default-features = false, features = [
|
||||
"__internal",
|
||||
] }
|
||||
reth-primitives = { path = "crates/primitives", default-features = false, features = ["__internal"] }
|
||||
reth-primitives-traits = { path = "crates/primitives-traits", default-features = false }
|
||||
reth-provider = { path = "crates/storage/provider" }
|
||||
reth-prune = { path = "crates/prune/prune" }
|
||||
@@ -439,29 +436,26 @@ reth-trie-sparse = { path = "crates/trie/sparse", default-features = false }
|
||||
reth-zstd-compressors = { path = "crates/storage/zstd-compressors", default-features = false }
|
||||
|
||||
# revm
|
||||
revm = { version = "36.0.0", default-features = false }
|
||||
revm-bytecode = { version = "9.0.0", default-features = false }
|
||||
revm-database = { version = "12.0.0", default-features = false }
|
||||
revm-state = { version = "10.0.0", default-features = false }
|
||||
revm-primitives = { version = "22.1.0", default-features = false }
|
||||
revm-interpreter = { version = "34.0.0", default-features = false }
|
||||
revm-database-interface = { version = "10.0.0", default-features = false }
|
||||
revm-inspectors = "0.36.1"
|
||||
revm = { version = "34.0.0", default-features = false }
|
||||
revm-bytecode = { version = "8.0.0", default-features = false }
|
||||
revm-database = { version = "10.0.0", default-features = false }
|
||||
revm-state = { version = "9.0.0", default-features = false }
|
||||
revm-primitives = { version = "22.0.0", default-features = false }
|
||||
revm-interpreter = { version = "32.0.0", default-features = false }
|
||||
revm-database-interface = { version = "9.0.0", default-features = false }
|
||||
op-revm = { version = "15.0.0", default-features = false }
|
||||
revm-inspectors = "0.34.2"
|
||||
|
||||
# eth
|
||||
alloy-dyn-abi = "1.5.6"
|
||||
alloy-primitives = { version = "1.5.6", default-features = false, features = [
|
||||
"map-foldhash",
|
||||
] }
|
||||
alloy-primitives = { version = "1.5.6", default-features = false, features = ["map-foldhash"] }
|
||||
alloy-sol-types = { version = "1.5.6", default-features = false }
|
||||
|
||||
alloy-chains = { version = "0.2.5", default-features = false }
|
||||
alloy-eip2124 = { version = "0.2.0", default-features = false }
|
||||
alloy-eip7928 = { version = "0.3.0", default-features = false }
|
||||
alloy-evm = { version = "0.29.2", default-features = false }
|
||||
alloy-rlp = { version = "0.3.13", default-features = false, features = [
|
||||
"core-net",
|
||||
] }
|
||||
alloy-evm = { version = "0.27.2", default-features = false }
|
||||
alloy-rlp = { version = "0.3.13", default-features = false, features = ["core-net"] }
|
||||
alloy-trie = { version = "0.9.4", default-features = false }
|
||||
|
||||
alloy-hardforks = "0.4.5"
|
||||
@@ -473,15 +467,10 @@ alloy-genesis = { version = "1.7.3", default-features = false }
|
||||
alloy-json-rpc = { version = "1.7.3", default-features = false }
|
||||
alloy-network = { version = "1.7.3", default-features = false }
|
||||
alloy-network-primitives = { version = "1.7.3", default-features = false }
|
||||
alloy-provider = { version = "1.7.3", features = [
|
||||
"reqwest",
|
||||
"debug-api",
|
||||
], default-features = false }
|
||||
alloy-provider = { version = "1.7.3", features = ["reqwest", "debug-api"], default-features = false }
|
||||
alloy-pubsub = { version = "1.7.3", default-features = false }
|
||||
alloy-rpc-client = { version = "1.7.3", default-features = false }
|
||||
alloy-rpc-types = { version = "1.7.3", features = [
|
||||
"eth",
|
||||
], default-features = false }
|
||||
alloy-rpc-types = { version = "1.7.3", features = ["eth"], default-features = false }
|
||||
alloy-rpc-types-admin = { version = "1.7.3", default-features = false }
|
||||
alloy-rpc-types-anvil = { version = "1.7.3", default-features = false }
|
||||
alloy-rpc-types-beacon = { version = "1.7.3", default-features = false }
|
||||
@@ -495,31 +484,30 @@ alloy-serde = { version = "1.7.3", default-features = false }
|
||||
alloy-signer = { version = "1.7.3", default-features = false }
|
||||
alloy-signer-local = { version = "1.7.3", default-features = false }
|
||||
alloy-transport = { version = "1.7.3" }
|
||||
alloy-transport-http = { version = "1.7.3", features = [
|
||||
"reqwest-rustls-tls",
|
||||
], default-features = false }
|
||||
alloy-transport-http = { version = "1.7.3", features = ["reqwest-rustls-tls"], default-features = false }
|
||||
alloy-transport-ipc = { version = "1.7.3", default-features = false }
|
||||
alloy-transport-ws = { version = "1.7.3", default-features = false }
|
||||
|
||||
# op
|
||||
op-alloy-rpc-types = { version = "0.24.0", default-features = false }
|
||||
op-alloy-rpc-types-engine = { version = "0.24.0", default-features = false }
|
||||
op-alloy-consensus = { version = "0.24.0", default-features = false }
|
||||
alloy-op-evm = { version = "0.27.2", default-features = false }
|
||||
alloy-op-hardforks = "0.4.4"
|
||||
op-alloy-rpc-types = { version = "0.23.1", default-features = false }
|
||||
op-alloy-rpc-types-engine = { version = "0.23.1", default-features = false }
|
||||
op-alloy-network = { version = "0.23.1", default-features = false }
|
||||
op-alloy-consensus = { version = "0.23.1", default-features = false }
|
||||
op-alloy-rpc-jsonrpsee = { version = "0.23.1", default-features = false }
|
||||
op-alloy-flz = { version = "0.13.1", default-features = false }
|
||||
|
||||
# misc
|
||||
either = { version = "1.15.0", default-features = false }
|
||||
arrayvec = { version = "0.7.6", default-features = false }
|
||||
aquamarine = "0.6"
|
||||
auto_impl = "1"
|
||||
backon = { version = "1.2", default-features = false, features = [
|
||||
"std-blocking-sleep",
|
||||
"tokio-sleep",
|
||||
] }
|
||||
backon = { version = "1.2", default-features = false, features = ["std-blocking-sleep", "tokio-sleep"] }
|
||||
bincode = "1.3"
|
||||
bitflags = "2.4"
|
||||
boyer-moore-magiclen = "0.2.16"
|
||||
bytes = { version = "1.11.1", default-features = false }
|
||||
blake3 = "1.8"
|
||||
brotli = "8"
|
||||
cfg-if = "1.0"
|
||||
clap = "4"
|
||||
@@ -535,16 +523,11 @@ humantime = "2.1"
|
||||
humantime-serde = "1.1"
|
||||
itertools = { version = "0.14", default-features = false }
|
||||
linked_hash_set = "0.1"
|
||||
libc = "0.2"
|
||||
lz4 = "1.28.1"
|
||||
modular-bitfield = "0.13.1"
|
||||
notify = { version = "8.0.0", default-features = false, features = [
|
||||
"macos_fsevent",
|
||||
] }
|
||||
notify = { version = "8.0.0", default-features = false, features = ["macos_fsevent"] }
|
||||
nybbles = { version = "0.4.8", default-features = false }
|
||||
once_cell = { version = "1.19", default-features = false, features = [
|
||||
"critical-section",
|
||||
] }
|
||||
once_cell = { version = "1.19", default-features = false, features = ["critical-section"] }
|
||||
parking_lot = "0.12"
|
||||
quanta = "0.12"
|
||||
paste = "1.0"
|
||||
@@ -558,16 +541,13 @@ serde_json = { version = "1.0", default-features = false, features = ["alloc"] }
|
||||
serde_with = { version = "3", default-features = false, features = ["macros"] }
|
||||
sha2 = { version = "0.10", default-features = false }
|
||||
shlex = "1.3"
|
||||
slotmap = "1"
|
||||
smallvec = "1"
|
||||
strum = { version = "0.27", default-features = false }
|
||||
strum_macros = "0.27"
|
||||
syn = "2.0"
|
||||
thiserror = { version = "2.0.0", default-features = false }
|
||||
tar = "0.4.44"
|
||||
tracing = { version = "0.1.0", default-features = false, features = [
|
||||
"attributes",
|
||||
] }
|
||||
tracing = { version = "0.1.0", default-features = false, features = ["attributes"] }
|
||||
tracing-appender = "0.2"
|
||||
url = { version = "2.3", default-features = false }
|
||||
zstd = "0.13"
|
||||
@@ -605,11 +585,7 @@ futures-util = { version = "0.3", default-features = false }
|
||||
hyper = "1.3"
|
||||
hyper-util = "0.1.5"
|
||||
pin-project = "1.0.12"
|
||||
reqwest = { version = "0.12", default-features = false, features = [
|
||||
"rustls-tls",
|
||||
"rustls-tls-native-roots",
|
||||
"stream",
|
||||
] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "rustls-tls-native-roots", "stream"] }
|
||||
tracing-futures = "0.2"
|
||||
tower = "0.5"
|
||||
tower-http = "0.6"
|
||||
@@ -634,10 +610,7 @@ proptest-arbitrary-interop = "0.1.0"
|
||||
# crypto
|
||||
enr = { version = "0.13", default-features = false }
|
||||
k256 = { version = "0.13", default-features = false, features = ["ecdsa"] }
|
||||
secp256k1 = { version = "0.30", default-features = false, features = [
|
||||
"global-context",
|
||||
"recovery",
|
||||
] }
|
||||
secp256k1 = { version = "0.30", default-features = false, features = ["global-context", "recovery"] }
|
||||
# rand 8 for secp256k1
|
||||
rand_08 = { package = "rand", version = "0.8" }
|
||||
|
||||
@@ -692,7 +665,6 @@ cipher = "0.4.3"
|
||||
comfy-table = "7.0"
|
||||
concat-kdf = "0.1.0"
|
||||
crossbeam-channel = "0.5.13"
|
||||
crossbeam-queue = "0.3"
|
||||
crossbeam-utils = "0.8"
|
||||
crossterm = "0.29.0"
|
||||
csv = "1.3.0"
|
||||
@@ -769,8 +741,10 @@ ipnet = "2.11"
|
||||
# alloy-transport-ws = { git = "https://github.com/alloy-rs/alloy", rev = "3049f232fbb44d1909883e154eb38ec5962f53a3" }
|
||||
|
||||
# op-alloy-consensus = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
|
||||
# op-alloy-network = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
|
||||
# op-alloy-rpc-types = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
|
||||
# op-alloy-rpc-types-engine = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
|
||||
# op-alloy-rpc-jsonrpsee = { git = "https://github.com/alloy-rs/op-alloy", rev = "a79d6fc" }
|
||||
#
|
||||
# revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "1207e33" }
|
||||
#
|
||||
@@ -780,15 +754,10 @@ ipnet = "2.11"
|
||||
# jsonrpsee-http-client = { git = "https://github.com/paradigmxyz/jsonrpsee", branch = "matt/make-rpc-service-pub" }
|
||||
# jsonrpsee-types = { git = "https://github.com/paradigmxyz/jsonrpsee", branch = "matt/make-rpc-service-pub" }
|
||||
|
||||
alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "b0eb7e6" }
|
||||
# alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "df124c0" }
|
||||
# alloy-op-evm = { git = "https://github.com/alloy-rs/evm", rev = "df124c0" }
|
||||
|
||||
revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "24becc3" }
|
||||
# revm-inspectors = { git = "https://github.com/paradigmxyz/revm-inspectors", rev = "3020ea8" }
|
||||
|
||||
# revm from rakita/state-gas branch
|
||||
revm = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
revm-bytecode = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
revm-database = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
revm-state = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
revm-primitives = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
revm-interpreter = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
revm-database-interface = { git = "https://github.com/bluealloy/revm", rev = "712dac7a" }
|
||||
# alloy-evm = { git = "https://github.com/alloy-rs/evm", rev = "072c248" }
|
||||
# alloy-op-evm = { git = "https://github.com/alloy-rs/evm", rev = "072c248" }
|
||||
|
||||
@@ -1,10 +1,8 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
# Dockerfile for reth, optimized for Depot builds
|
||||
# Supports PGO+BOLT optimization for maximum performance
|
||||
# Usage:
|
||||
# reth: --build-arg BINARY=reth
|
||||
# PGO+BOLT: --build-arg USE_PGO_BOLT=true (Linux x86_64/aarch64 only)
|
||||
|
||||
FROM rust:1.93 AS builder
|
||||
WORKDIR /app
|
||||
@@ -45,18 +43,6 @@ ENV VERGEN_GIT_SHA=$VERGEN_GIT_SHA
|
||||
ENV VERGEN_GIT_DESCRIBE=$VERGEN_GIT_DESCRIBE
|
||||
ENV VERGEN_GIT_DIRTY=$VERGEN_GIT_DIRTY
|
||||
|
||||
# Enable PGO+BOLT optimization (Linux only)
|
||||
ARG USE_PGO_BOLT=false
|
||||
ENV USE_PGO_BOLT=$USE_PGO_BOLT
|
||||
|
||||
# Optional path to a pre-collected merged.profdata file in build context.
|
||||
ARG PGO_PROFDATA=""
|
||||
ENV PGO_PROFDATA=$PGO_PROFDATA
|
||||
|
||||
# Whether to strip debug symbols from PGO-built binaries.
|
||||
ARG STRIP_SYMBOLS=true
|
||||
ENV STRIP_SYMBOLS=$STRIP_SYMBOLS
|
||||
|
||||
# Build application
|
||||
# Platform-specific RUSTFLAGS: amd64 uses x86-64-v3 (Haswell+) with pclmulqdq for rocksdb
|
||||
ARG TARGETPLATFORM
|
||||
@@ -67,21 +53,12 @@ RUN --mount=type=secret,id=DEPOT_TOKEN,env=SCCACHE_WEBDAV_TOKEN \
|
||||
--mount=type=cache,target=$SCCACHE_DIR,sharing=shared \
|
||||
export RUSTC_WRAPPER=sccache SCCACHE_WEBDAV_ENDPOINT=https://cache.depot.dev SCCACHE_DIR=/sccache && \
|
||||
sccache --start-server && \
|
||||
if [ "$USE_PGO_BOLT" = "true" ] && [ "$TARGETPLATFORM" = "linux/amd64" ] && [ -n "$PGO_PROFDATA" ] && [ -f "$PGO_PROFDATA" ]; then \
|
||||
apt-get update && apt-get install -y -qq lsb-release wget sudo && \
|
||||
BINARY="$BINARY" PROFILE="$BUILD_PROFILE" FEATURES="$FEATURES" SKIP_BOLT=true STRIP_SYMBOLS="$STRIP_SYMBOLS" PGO_PROFDATA="$PGO_PROFDATA" \
|
||||
./.github/scripts/build_pgo_bolt.sh; \
|
||||
else \
|
||||
if [ "$USE_PGO_BOLT" = "true" ]; then \
|
||||
echo "PGO requested but pre-collected profile missing at '${PGO_PROFDATA:-<unset>}' - falling back to non-PGO build"; \
|
||||
fi; \
|
||||
if [ -n "$RUSTFLAGS" ]; then \
|
||||
export RUSTFLAGS="$RUSTFLAGS"; \
|
||||
elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
|
||||
export RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"; \
|
||||
fi && \
|
||||
cargo build --profile $BUILD_PROFILE --features "$FEATURES" --locked --bin $BINARY --manifest-path $MANIFEST_PATH/Cargo.toml; \
|
||||
if [ -n "$RUSTFLAGS" ]; then \
|
||||
export RUSTFLAGS="$RUSTFLAGS"; \
|
||||
elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \
|
||||
export RUSTFLAGS="-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq"; \
|
||||
fi && \
|
||||
cargo build --profile $BUILD_PROFILE --features "$FEATURES" --locked --bin $BINARY --manifest-path $MANIFEST_PATH/Cargo.toml && \
|
||||
sccache --show-stats
|
||||
|
||||
# Copy binary to a known location (ARG not resolved in COPY)
|
||||
|
||||
8
Makefile
8
Makefile
@@ -17,9 +17,6 @@ FEATURES ?=
|
||||
# Cargo profile for builds. Default is for local builds, CI uses an override.
|
||||
PROFILE ?= release
|
||||
|
||||
# Extra RUSTFLAGS to append to build targets (e.g., "-C target-cpu=x86-64-v3")
|
||||
EXTRA_RUSTFLAGS ?=
|
||||
|
||||
# Extra flags for Cargo
|
||||
CARGO_INSTALL_EXTRA_FLAGS ?=
|
||||
|
||||
@@ -77,7 +74,7 @@ build-debug: ## Build the reth binary into `target/debug` directory.
|
||||
cargo build --bin reth --features "$(FEATURES)"
|
||||
# Builds the reth binary natively.
|
||||
build-native-%:
|
||||
$(if $(EXTRA_RUSTFLAGS),RUSTFLAGS="$(EXTRA_RUSTFLAGS)") cargo build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
|
||||
cargo build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
|
||||
|
||||
# The following commands use `cross` to build a cross-compile.
|
||||
#
|
||||
@@ -95,12 +92,11 @@ build-native-%:
|
||||
# on other systems. JEMALLOC_SYS_WITH_LG_PAGE=16 tells jemalloc to use 64-KiB
|
||||
# pages. See: https://github.com/paradigmxyz/reth/issues/6742
|
||||
build-aarch64-unknown-linux-gnu: export JEMALLOC_SYS_WITH_LG_PAGE=16
|
||||
build-native-aarch64-unknown-linux-gnu: export JEMALLOC_SYS_WITH_LG_PAGE=16
|
||||
|
||||
# Note: The additional rustc compiler flags are for intrinsics needed by MDBX.
|
||||
# See: https://github.com/cross-rs/cross/wiki/FAQ#undefined-reference-with-build-std
|
||||
build-%:
|
||||
RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc $(EXTRA_RUSTFLAGS)" \
|
||||
RUSTFLAGS="-C link-arg=-lgcc -Clink-arg=-static-libgcc" \
|
||||
cross build --bin reth --target $* --features "$(FEATURES)" --profile "$(PROFILE)"
|
||||
|
||||
# Unfortunately we can't easily use cross to build for Darwin because of licensing issues.
|
||||
|
||||
@@ -7,7 +7,7 @@ use alloy_primitives::address;
|
||||
use alloy_provider::{network::AnyNetwork, Provider, RootProvider};
|
||||
use alloy_rpc_client::ClientBuilder;
|
||||
use alloy_rpc_types_engine::JwtSecret;
|
||||
use alloy_transport::layers::{RateLimitRetryPolicy, RetryBackoffLayer};
|
||||
use alloy_transport::layers::RetryBackoffLayer;
|
||||
use reqwest::Url;
|
||||
use reth_node_core::args::BenchmarkArgs;
|
||||
use tracing::info;
|
||||
@@ -53,15 +53,9 @@ impl BenchContext {
|
||||
}
|
||||
}
|
||||
|
||||
// set up alloy client for blocks, retrying on 429/503 (default) and 502
|
||||
let retry_policy =
|
||||
RateLimitRetryPolicy::default().or(|err: &alloy_transport::TransportError| -> bool {
|
||||
err.as_transport_err()
|
||||
.and_then(|t| t.as_http_error())
|
||||
.is_some_and(|e| e.status == 502)
|
||||
});
|
||||
// set up alloy client for blocks
|
||||
let client = ClientBuilder::default()
|
||||
.layer(RetryBackoffLayer::new_with_policy(10, 800, u64::MAX, retry_policy))
|
||||
.layer(RetryBackoffLayer::new(10, 800, u64::MAX))
|
||||
.http(rpc_url.parse()?);
|
||||
let block_provider = RootProvider::<AnyNetwork>::new(client);
|
||||
|
||||
@@ -94,11 +88,9 @@ impl BenchContext {
|
||||
|
||||
// Computes the block range for the benchmark.
|
||||
//
|
||||
// - If `--advance` is provided, fetches the latest block from the engine and sets:
|
||||
// - If `--advance` is provided, fetches the latest block and sets:
|
||||
// - `from = head + 1`
|
||||
// - `to = head + advance`
|
||||
// - If only `--to` is provided, fetches the latest block from the engine and sets:
|
||||
// - `from = head`
|
||||
// - Otherwise, uses the values from `--from` and `--to`.
|
||||
let (from, to) = if let Some(advance) = bench_args.advance {
|
||||
if advance == 0 {
|
||||
@@ -111,14 +103,6 @@ impl BenchContext {
|
||||
.ok_or_else(|| eyre::eyre!("Failed to fetch latest block for --advance"))?;
|
||||
let head_number = head_block.header.number;
|
||||
(Some(head_number), Some(head_number + advance))
|
||||
} else if bench_args.from.is_none() && bench_args.to.is_some() {
|
||||
let head_block = auth_provider
|
||||
.get_block_by_number(BlockNumberOrTag::Latest)
|
||||
.await?
|
||||
.ok_or_else(|| eyre::eyre!("Failed to fetch latest block from engine"))?;
|
||||
let head_number = head_block.header.number;
|
||||
info!(target: "reth-bench", "No --from provided, derived from engine head: {}", head_number);
|
||||
(Some(head_number), bench_args.to)
|
||||
} else {
|
||||
(bench_args.from, bench_args.to)
|
||||
};
|
||||
@@ -130,7 +114,7 @@ impl BenchContext {
|
||||
.full()
|
||||
.await?
|
||||
.ok_or_else(|| eyre::eyre!("Failed to fetch latest block from RPC"))?;
|
||||
let mut benchmark_mode = BenchMode::new(from, to, latest_block.into_inner().number());
|
||||
let mut benchmark_mode = BenchMode::new(from, to, latest_block.into_inner().number())?;
|
||||
|
||||
let first_block = match benchmark_mode {
|
||||
BenchMode::Continuous(start) => {
|
||||
|
||||
@@ -24,7 +24,7 @@ pub(crate) struct GasRampPayloadFile {
|
||||
/// Engine API version (1-5).
|
||||
///
|
||||
/// `None` indicates that `reth_newPayload` should be used.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) version: Option<u8>,
|
||||
/// The block hash for FCU.
|
||||
pub(crate) block_hash: B256,
|
||||
@@ -114,27 +114,16 @@ impl CombinedResult {
|
||||
|
||||
impl std::fmt::Display for CombinedResult {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
let np = &self.new_payload_result;
|
||||
write!(
|
||||
f,
|
||||
"Block {} processed at {:.4} Ggas/s, used {} total gas. Combined: {:.4} Ggas/s. fcu: {:?}, newPayload: {:?}",
|
||||
self.block_number,
|
||||
np.gas_per_second() / GIGAGAS as f64,
|
||||
np.gas_used,
|
||||
self.new_payload_result.gas_per_second() / GIGAGAS as f64,
|
||||
self.new_payload_result.gas_used,
|
||||
self.combined_gas_per_second() / GIGAGAS as f64,
|
||||
self.fcu_latency,
|
||||
np.latency,
|
||||
)?;
|
||||
if !np.execution_cache_wait.is_zero() {
|
||||
write!(f, ", execution cache wait: {:?}", np.execution_cache_wait)?;
|
||||
}
|
||||
if !np.sparse_trie_wait.is_zero() {
|
||||
write!(f, ", trie cache wait: {:?}", np.sparse_trie_wait)?;
|
||||
}
|
||||
if let Some(d) = np.persistence_wait {
|
||||
write!(f, ", persistence wait: {d:?}")?;
|
||||
}
|
||||
Ok(())
|
||||
self.new_payload_result.latency
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -292,6 +292,10 @@ impl Command {
|
||||
|
||||
info!(target: "reth-bench", gas_ramp_payload = i + 1, "Gas ramp payload executed successfully");
|
||||
|
||||
if let Some(w) = &mut waiter {
|
||||
w.on_block(payload.block_number).await?;
|
||||
}
|
||||
|
||||
parent_hash = payload.file.block_hash;
|
||||
}
|
||||
|
||||
|
||||
@@ -34,16 +34,17 @@ impl BenchMode {
|
||||
}
|
||||
|
||||
/// Create a [`BenchMode`] from optional `from` and `to` fields.
|
||||
///
|
||||
/// If only `--to` is provided, `from` is derived as `latest_block + 1`.
|
||||
pub const fn new(from: Option<u64>, to: Option<u64>, latest_block: u64) -> Self {
|
||||
pub fn new(from: Option<u64>, to: Option<u64>, latest_block: u64) -> Result<Self, eyre::Error> {
|
||||
// If neither `--from` nor `--to` are provided, we will run the benchmark continuously,
|
||||
// starting at the latest block.
|
||||
match (from, to) {
|
||||
(Some(from), Some(to)) => Self::Range(from..=to),
|
||||
(None, None) => Self::Continuous(latest_block),
|
||||
(Some(start), None) => Self::Continuous(start),
|
||||
(None, Some(to)) => Self::Range(latest_block + 1..=to),
|
||||
(Some(from), Some(to)) => Ok(Self::Range(from..=to)),
|
||||
(None, None) => Ok(Self::Continuous(latest_block)),
|
||||
(Some(start), None) => Ok(Self::Continuous(start)),
|
||||
_ => {
|
||||
// both or neither are allowed, everything else is ambiguous
|
||||
Err(eyre::eyre!("`from` and `to` must be provided together, or not at all."))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -235,8 +235,8 @@ pub(crate) fn payload_to_new_payload(
|
||||
))?,
|
||||
)
|
||||
} else {
|
||||
// Preserve the original RequestsOrHash payload for engine_newPayloadV4.
|
||||
let requests = prague.requests.clone();
|
||||
// Extract actual Requests from RequestsOrHash
|
||||
let requests = prague.requests.requests_hash();
|
||||
(
|
||||
version,
|
||||
serde_json::to_value((
|
||||
|
||||
@@ -89,6 +89,7 @@ default = [
|
||||
"keccak-cache-global",
|
||||
"asm-keccak",
|
||||
"min-debug-logs",
|
||||
"rocksdb",
|
||||
]
|
||||
|
||||
otlp = [
|
||||
@@ -124,7 +125,6 @@ jemalloc = [
|
||||
"reth-node-core/jemalloc",
|
||||
"reth-node-metrics/jemalloc",
|
||||
"reth-ethereum-cli/jemalloc",
|
||||
"reth-provider/jemalloc",
|
||||
]
|
||||
jemalloc-prof = [
|
||||
"reth-cli-util/jemalloc",
|
||||
@@ -191,6 +191,8 @@ min-trace-logs = [
|
||||
]
|
||||
|
||||
trie-debug = ["reth-node-builder/trie-debug", "reth-node-core/trie-debug"]
|
||||
rocksdb = ["reth-ethereum-cli/rocksdb", "reth-node-core/rocksdb"]
|
||||
edge = ["rocksdb"]
|
||||
|
||||
[[bin]]
|
||||
name = "reth"
|
||||
|
||||
@@ -56,6 +56,7 @@ alloy-signer.workspace = true
|
||||
alloy-signer-local.workspace = true
|
||||
rand.workspace = true
|
||||
revm-state.workspace = true
|
||||
criterion.workspace = true
|
||||
|
||||
[features]
|
||||
serde = [
|
||||
@@ -85,3 +86,8 @@ test-utils = [
|
||||
"reth-ethereum-primitives/test-utils",
|
||||
]
|
||||
rayon = ["dep:rayon"]
|
||||
|
||||
[[bench]]
|
||||
name = "canonical_hashes_range"
|
||||
harness = false
|
||||
required-features = ["test-utils"]
|
||||
|
||||
96
crates/chain-state/benches/canonical_hashes_range.rs
Normal file
96
crates/chain-state/benches/canonical_hashes_range.rs
Normal file
@@ -0,0 +1,96 @@
|
||||
#![allow(missing_docs)]
|
||||
|
||||
use criterion::{black_box, criterion_group, criterion_main, Criterion};
|
||||
use reth_chain_state::{
|
||||
test_utils::TestBlockBuilder, ExecutedBlock, MemoryOverlayStateProviderRef,
|
||||
};
|
||||
use reth_ethereum_primitives::EthPrimitives;
|
||||
use reth_storage_api::{noop::NoopProvider, BlockHashReader};
|
||||
|
||||
criterion_group!(benches, bench_canonical_hashes_range);
|
||||
criterion_main!(benches);
|
||||
|
||||
fn bench_canonical_hashes_range(c: &mut Criterion) {
|
||||
let mut group = c.benchmark_group("canonical_hashes_range");
|
||||
|
||||
let scenarios = [("small", 10), ("medium", 100), ("large", 1000)];
|
||||
|
||||
for (name, num_blocks) in scenarios {
|
||||
group.bench_function(format!("{}_blocks_{}", name, num_blocks), |b| {
|
||||
let (provider, blocks) = setup_provider_with_blocks(num_blocks);
|
||||
let start_block = blocks[0].recovered_block().number;
|
||||
let end_block = blocks[num_blocks / 2].recovered_block().number;
|
||||
|
||||
b.iter(|| {
|
||||
black_box(
|
||||
provider
|
||||
.canonical_hashes_range(black_box(start_block), black_box(end_block))
|
||||
.unwrap(),
|
||||
)
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
let (provider, blocks) = setup_provider_with_blocks(500);
|
||||
let base_block = blocks[100].recovered_block().number;
|
||||
|
||||
let range_sizes = [1, 10, 50, 100, 250];
|
||||
for range_size in range_sizes {
|
||||
group.bench_function(format!("range_size_{}", range_size), |b| {
|
||||
let end_block = base_block + range_size;
|
||||
|
||||
b.iter(|| {
|
||||
black_box(
|
||||
provider
|
||||
.canonical_hashes_range(black_box(base_block), black_box(end_block))
|
||||
.unwrap(),
|
||||
)
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
// Benchmark edge cases
|
||||
group.bench_function("no_in_memory_matches", |b| {
|
||||
let (provider, blocks) = setup_provider_with_blocks(100);
|
||||
let first_block = blocks[0].recovered_block().number;
|
||||
let start_block = first_block - 50;
|
||||
let end_block = first_block - 10;
|
||||
|
||||
b.iter(|| {
|
||||
black_box(
|
||||
provider
|
||||
.canonical_hashes_range(black_box(start_block), black_box(end_block))
|
||||
.unwrap(),
|
||||
)
|
||||
})
|
||||
});
|
||||
|
||||
group.bench_function("all_in_memory_matches", |b| {
|
||||
let (provider, blocks) = setup_provider_with_blocks(100);
|
||||
let first_block = blocks[0].recovered_block().number;
|
||||
let last_block = blocks[blocks.len() - 1].recovered_block().number;
|
||||
|
||||
b.iter(|| {
|
||||
black_box(
|
||||
provider
|
||||
.canonical_hashes_range(black_box(first_block), black_box(last_block + 1))
|
||||
.unwrap(),
|
||||
)
|
||||
})
|
||||
});
|
||||
|
||||
group.finish();
|
||||
}
|
||||
|
||||
fn setup_provider_with_blocks(
|
||||
num_blocks: usize,
|
||||
) -> (MemoryOverlayStateProviderRef<'static, EthPrimitives>, Vec<ExecutedBlock<EthPrimitives>>) {
|
||||
let mut builder = TestBlockBuilder::<EthPrimitives>::default();
|
||||
|
||||
let blocks: Vec<_> = builder.get_executed_blocks(1000..1000 + num_blocks as u64).collect();
|
||||
|
||||
let historical = Box::new(NoopProvider::default());
|
||||
let provider = MemoryOverlayStateProviderRef::new(historical, blocks.clone());
|
||||
|
||||
(provider, blocks)
|
||||
}
|
||||
@@ -9,7 +9,7 @@ use std::{
|
||||
fmt,
|
||||
sync::{Arc, LazyLock},
|
||||
};
|
||||
use tracing::{debug_span, instrument};
|
||||
use tracing::instrument;
|
||||
|
||||
/// Shared handle to asynchronously populated trie data.
|
||||
///
|
||||
@@ -163,8 +163,6 @@ impl DeferredTrieData {
|
||||
anchor_hash: B256,
|
||||
ancestors: &[Self],
|
||||
) -> ComputedTrieData {
|
||||
let _span = debug_span!(target: "engine::tree::deferred_trie", "sort_inputs").entered();
|
||||
|
||||
#[cfg(feature = "rayon")]
|
||||
let (sorted_hashed_state, sorted_trie_updates) = rayon::join(
|
||||
|| match Arc::try_unwrap(hashed_state) {
|
||||
@@ -189,10 +187,6 @@ impl DeferredTrieData {
|
||||
},
|
||||
);
|
||||
|
||||
drop(_span);
|
||||
|
||||
let _span = debug_span!(target: "engine::tree::deferred_trie", "build_overlay").entered();
|
||||
|
||||
// Reuse parent's overlay if available and anchors match.
|
||||
// We can only reuse the parent's overlay if it was built on top of the same
|
||||
// persisted anchor. If the anchor has changed (e.g., due to persistence),
|
||||
@@ -211,9 +205,6 @@ impl DeferredTrieData {
|
||||
Arc::clone(&trie_input.state),
|
||||
Default::default(), // prefix_sets are per-block, not cumulative
|
||||
);
|
||||
let _span =
|
||||
debug_span!(target: "engine::tree::deferred_trie", "extend_overlay")
|
||||
.entered();
|
||||
// Only trigger COW clone if there's actually data to add.
|
||||
#[cfg(feature = "rayon")]
|
||||
{
|
||||
@@ -281,7 +272,6 @@ impl DeferredTrieData {
|
||||
sorted_hashed_state: &HashedPostStateSorted,
|
||||
sorted_trie_updates: &TrieUpdatesSorted,
|
||||
) -> TrieInputSorted {
|
||||
let _span = debug_span!(target: "engine::tree::deferred_trie", "merge_ancestors", num_ancestors = ancestors.len()).entered();
|
||||
let mut overlay = TrieInputSorted::default();
|
||||
|
||||
let state_mut = Arc::make_mut(&mut overlay.state);
|
||||
|
||||
@@ -1,69 +0,0 @@
|
||||
//! Execution timing statistics for detailed block logging.
|
||||
//!
|
||||
//! This module provides types for collecting and passing execution timing statistics
|
||||
//! through the block processing pipeline, enabling unified detailed block logging after
|
||||
//! database commit.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use alloy_primitives::B256;
|
||||
|
||||
/// Statistics collected during block execution for cross-client performance analysis.
|
||||
///
|
||||
/// These statistics are populated during block validation and carried through to
|
||||
/// persistence, where they are used to emit a single unified log entry that includes
|
||||
/// complete timing information (including commit time).
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct ExecutionTimingStats {
|
||||
/// Block number
|
||||
pub block_number: u64,
|
||||
/// Block hash
|
||||
pub block_hash: B256,
|
||||
/// Total gas used by the block
|
||||
pub gas_used: u64,
|
||||
/// Number of transactions in the block
|
||||
pub tx_count: usize,
|
||||
/// Time spent executing transactions (includes state reads)
|
||||
pub execution_duration: Duration,
|
||||
/// Time spent fetching state during execution (subset of `execution_duration`, includes cache
|
||||
/// hits)
|
||||
pub state_read_duration: Duration,
|
||||
/// Time spent computing state root hash
|
||||
pub state_hash_duration: Duration,
|
||||
/// Number of accounts read during execution
|
||||
pub accounts_read: usize,
|
||||
/// Number of storage slots read (SLOAD operations)
|
||||
pub storage_read: usize,
|
||||
/// Number of code reads (EXTCODE* operations)
|
||||
pub code_read: usize,
|
||||
/// Total bytes of code read
|
||||
pub code_bytes_read: usize,
|
||||
/// Number of accounts changed (balance/nonce updates)
|
||||
pub accounts_changed: usize,
|
||||
/// Number of accounts deleted (SELFDESTRUCT)
|
||||
pub accounts_deleted: usize,
|
||||
/// Number of storage slots changed (SSTORE operations)
|
||||
pub storage_slots_changed: usize,
|
||||
/// Number of storage slots deleted (set to zero)
|
||||
pub storage_slots_deleted: usize,
|
||||
/// Number of bytecodes created/changed (contract deployments)
|
||||
pub bytecodes_changed: usize,
|
||||
/// Total bytes of code written
|
||||
pub code_bytes_written: usize,
|
||||
/// Number of EIP-7702 delegations set
|
||||
pub eip7702_delegations_set: usize,
|
||||
/// Number of EIP-7702 delegations cleared
|
||||
pub eip7702_delegations_cleared: usize,
|
||||
/// Account cache hits
|
||||
pub account_cache_hits: usize,
|
||||
/// Account cache misses
|
||||
pub account_cache_misses: usize,
|
||||
/// Storage cache hits
|
||||
pub storage_cache_hits: usize,
|
||||
/// Storage cache misses
|
||||
pub storage_cache_misses: usize,
|
||||
/// Code cache hits
|
||||
pub code_cache_hits: usize,
|
||||
/// Code cache misses
|
||||
pub code_cache_misses: usize,
|
||||
}
|
||||
@@ -992,7 +992,7 @@ impl<N: NodePrimitives<SignedTx: SignedTransaction>> NewCanonicalChain<N> {
|
||||
///
|
||||
/// Returns the new tip for [`Self::Reorg`] and [`Self::Commit`] variants which commit at least
|
||||
/// 1 new block.
|
||||
pub fn tip(&self) -> &RecoveredBlock<N::Block> {
|
||||
pub fn tip(&self) -> &SealedBlock<N::Block> {
|
||||
match self {
|
||||
Self::Commit { new } | Self::Reorg { new, .. } => {
|
||||
new.last().expect("non empty blocks").recovered_block()
|
||||
|
||||
@@ -8,9 +8,6 @@
|
||||
#![cfg_attr(not(test), warn(unused_crate_dependencies))]
|
||||
#![cfg_attr(docsrs, feature(doc_cfg))]
|
||||
|
||||
mod execution_stats;
|
||||
pub use execution_stats::ExecutionTimingStats;
|
||||
|
||||
mod in_memory;
|
||||
pub use in_memory::*;
|
||||
|
||||
|
||||
@@ -36,6 +36,15 @@ pub use spec::{
|
||||
DepositContract, ForkBaseFeeParams, DEV, HOLESKY, HOODI, MAINNET, SEPOLIA,
|
||||
};
|
||||
|
||||
use reth_primitives_traits::sync::OnceLock;
|
||||
|
||||
/// Simple utility to create a thread-safe sync cell with a value set.
|
||||
pub fn once_cell_set<T>(value: T) -> OnceLock<T> {
|
||||
let once = OnceLock::new();
|
||||
let _ = once.set(value);
|
||||
once
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -21,7 +21,7 @@ use crate::chainspec::ChainSpecParser;
|
||||
///
|
||||
/// This trait is supposed to be implemented by the main struct of the CLI.
|
||||
///
|
||||
/// It provides commonly used functionality for running commands and information about the CLI, such
|
||||
/// It provides commonly used functionality for running commands and information about the CL, such
|
||||
/// as the name and version.
|
||||
pub trait RethCli: Sized {
|
||||
/// The associated `ChainSpecParser` type
|
||||
|
||||
@@ -46,7 +46,7 @@ reth-prune.workspace = true
|
||||
reth-prune-types.workspace = true
|
||||
reth-revm.workspace = true
|
||||
reth-stages.workspace = true
|
||||
reth-stages-types.workspace = true
|
||||
reth-stages-types = { workspace = true, optional = true }
|
||||
reth-static-file-types = { workspace = true, features = ["clap"] }
|
||||
reth-static-file.workspace = true
|
||||
reth-tasks.workspace = true
|
||||
@@ -87,8 +87,6 @@ tokio-stream.workspace = true
|
||||
reqwest.workspace = true
|
||||
url.workspace = true
|
||||
metrics.workspace = true
|
||||
blake3.workspace = true
|
||||
rayon.workspace = true
|
||||
|
||||
# io
|
||||
fdlimit.workspace = true
|
||||
@@ -110,6 +108,7 @@ reth-provider = { workspace = true, features = ["test-utils"] }
|
||||
tempfile.workspace = true
|
||||
|
||||
[features]
|
||||
default = []
|
||||
arbitrary = [
|
||||
"dep:proptest",
|
||||
"dep:arbitrary",
|
||||
@@ -128,9 +127,12 @@ arbitrary = [
|
||||
"reth-trie-common?/test-utils",
|
||||
"reth-codecs/arbitrary",
|
||||
"reth-prune-types/arbitrary",
|
||||
"reth-stages-types/arbitrary",
|
||||
"reth-stages-types?/arbitrary",
|
||||
"reth-trie-common?/arbitrary",
|
||||
"alloy-consensus/arbitrary",
|
||||
"reth-primitives-traits/arbitrary",
|
||||
"reth-ethereum-primitives/arbitrary",
|
||||
]
|
||||
|
||||
rocksdb = ["reth-db-common/rocksdb", "reth-stages/rocksdb", "reth-provider/rocksdb", "reth-prune/rocksdb"]
|
||||
edge = ["rocksdb"]
|
||||
|
||||
@@ -73,12 +73,17 @@ pub struct EnvironmentArgs<C: ChainSpecParser> {
|
||||
}
|
||||
|
||||
impl<C: ChainSpecParser> EnvironmentArgs<C> {
|
||||
/// Returns the storage settings for new database initialization.
|
||||
/// Returns the effective storage settings derived from `--storage.v2`.
|
||||
///
|
||||
/// Always returns [`StorageSettings::v2()`] — v2 is the default for all new
|
||||
/// databases. Existing databases use the settings persisted in their metadata.
|
||||
/// The base storage mode is determined by `--storage.v2`:
|
||||
/// - When `--storage.v2` is set: uses [`StorageSettings::v2()`] defaults
|
||||
/// - Otherwise: uses [`StorageSettings::base()`] defaults
|
||||
pub fn storage_settings(&self) -> StorageSettings {
|
||||
StorageSettings::v2()
|
||||
if self.storage.v2 {
|
||||
StorageSettings::v2()
|
||||
} else {
|
||||
StorageSettings::base()
|
||||
}
|
||||
}
|
||||
|
||||
/// Initializes environment according to [`AccessRights`] and returns an instance of
|
||||
@@ -187,6 +192,7 @@ impl<C: ChainSpecParser> EnvironmentArgs<C> {
|
||||
where
|
||||
C: ChainSpecParser<ChainSpec = N::ChainSpec>,
|
||||
{
|
||||
let prune_modes = config.prune.segments.clone();
|
||||
let factory = ProviderFactory::<NodeTypesWithDBAdapter<N, DatabaseEnv>>::new(
|
||||
db,
|
||||
self.chain.clone(),
|
||||
@@ -194,8 +200,7 @@ impl<C: ChainSpecParser> EnvironmentArgs<C> {
|
||||
rocksdb_provider,
|
||||
runtime,
|
||||
)?
|
||||
.with_prune_modes(config.prune.segments.clone())
|
||||
.with_minimum_pruning_distance(config.prune.minimum_pruning_distance);
|
||||
.with_prune_modes(prune_modes.clone());
|
||||
|
||||
// Check for consistency between database and static files.
|
||||
if !access.is_read_only_inconsistent() &&
|
||||
@@ -229,13 +234,10 @@ impl<C: ChainSpecParser> EnvironmentArgs<C> {
|
||||
NoopBodiesDownloader::default(),
|
||||
NoopEvmConfig::<N::Evm>::default(),
|
||||
config.stages.clone(),
|
||||
config.prune.segments.clone(),
|
||||
prune_modes.clone(),
|
||||
None,
|
||||
))
|
||||
.build(
|
||||
factory.clone(),
|
||||
StaticFileProducer::new(factory.clone(), config.prune.segments.clone()),
|
||||
);
|
||||
.build(factory.clone(), StaticFileProducer::new(factory.clone(), prune_modes));
|
||||
|
||||
// Move all applicable data from database to static files.
|
||||
pipeline.move_to_static_files()?;
|
||||
|
||||
@@ -21,6 +21,7 @@ use std::{
|
||||
};
|
||||
use tracing::{info, warn};
|
||||
|
||||
#[cfg(all(unix, feature = "rocksdb"))]
|
||||
mod rocksdb;
|
||||
|
||||
/// Interval for logging progress during checksum computation.
|
||||
@@ -72,6 +73,7 @@ enum Subcommand {
|
||||
limit: Option<usize>,
|
||||
},
|
||||
/// Calculates the checksum of a RocksDB table
|
||||
#[cfg(all(unix, feature = "rocksdb"))]
|
||||
Rocksdb {
|
||||
/// The RocksDB table
|
||||
#[arg(value_enum)]
|
||||
@@ -98,6 +100,7 @@ impl Command {
|
||||
Subcommand::StaticFile { segment, start_block, end_block, limit } => {
|
||||
checksum_static_file(tool, segment, start_block, end_block, limit)?;
|
||||
}
|
||||
#[cfg(all(unix, feature = "rocksdb"))]
|
||||
Subcommand::Rocksdb { table, limit } => {
|
||||
rocksdb::checksum_rocksdb(tool, table, limit)?;
|
||||
}
|
||||
|
||||
@@ -102,14 +102,14 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + EthereumHardforks>> Command<C>
|
||||
let static_files_path = data_dir.static_files();
|
||||
let exex_wal_path = data_dir.exex_wal();
|
||||
|
||||
// ensure the provided datadir exists
|
||||
// ensure the provided datadir exist
|
||||
eyre::ensure!(
|
||||
data_dir.data_dir().is_dir(),
|
||||
"Datadir does not exist: {:?}",
|
||||
data_dir.data_dir()
|
||||
);
|
||||
|
||||
// ensure the provided database exists
|
||||
// ensure the provided database exist
|
||||
eyre::ensure!(db_path.is_dir(), "Database does not exist: {:?}", db_path);
|
||||
|
||||
match self.command {
|
||||
|
||||
@@ -9,7 +9,7 @@ use reth_db_api::{
|
||||
};
|
||||
use reth_db_common::DbTool;
|
||||
use reth_node_builder::NodeTypesWithDB;
|
||||
use reth_provider::{providers::ProviderNodeTypes, StaticFileProviderFactory};
|
||||
use reth_provider::providers::ProviderNodeTypes;
|
||||
use reth_storage_api::{BlockNumReader, StateProvider, StorageSettingsCache};
|
||||
use reth_tasks::spawn_scoped_os_thread;
|
||||
use std::{
|
||||
@@ -17,9 +17,9 @@ use std::{
|
||||
thread,
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
use tracing::info;
|
||||
use tracing::{error, info};
|
||||
|
||||
/// Log progress every 30 seconds
|
||||
/// Log progress every 5 seconds
|
||||
const LOG_INTERVAL: Duration = Duration::from_secs(30);
|
||||
|
||||
/// The arguments for the `reth db state` command
|
||||
@@ -152,11 +152,17 @@ impl Command {
|
||||
let mut storage_keys = BTreeSet::new();
|
||||
|
||||
if history_in_rocksdb {
|
||||
self.collect_staticfile_storage_keys(tool, address, &mut storage_keys)?;
|
||||
} else {
|
||||
self.collect_mdbx_storage_keys_parallel(tool, address, &mut storage_keys)?;
|
||||
error!(
|
||||
target: "reth::cli",
|
||||
"Historical storage queries with RocksDB backend are not yet supported. \
|
||||
Use MDBX for storage history or query current state without --block."
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Collect keys from MDBX StorageChangeSets using parallel scanning
|
||||
self.collect_mdbx_storage_keys_parallel(tool, address, &mut storage_keys)?;
|
||||
|
||||
info!(
|
||||
target: "reth::cli",
|
||||
address = %address,
|
||||
@@ -201,63 +207,6 @@ impl Command {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Collects storage keys from static file StorageChangeSets (storage_v2).
|
||||
fn collect_staticfile_storage_keys<N: NodeTypesWithDB + ProviderNodeTypes>(
|
||||
&self,
|
||||
tool: &DbTool<N>,
|
||||
address: Address,
|
||||
keys: &mut BTreeSet<B256>,
|
||||
) -> eyre::Result<()> {
|
||||
let tip = tool.provider_factory.provider()?.best_block_number()?;
|
||||
|
||||
if tip == 0 {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
info!(
|
||||
target: "reth::cli",
|
||||
address = %address,
|
||||
tip,
|
||||
"Scanning static file storage changesets"
|
||||
);
|
||||
|
||||
let static_file_provider = tool.provider_factory.static_file_provider();
|
||||
let walker = static_file_provider.walk_storage_changeset_range(0..=tip);
|
||||
|
||||
let mut total_scanned = 0usize;
|
||||
let mut last_log = Instant::now();
|
||||
|
||||
for changeset_result in walker {
|
||||
let (block_addr, storage_entry) = changeset_result?;
|
||||
total_scanned += 1;
|
||||
|
||||
if block_addr.address() == address {
|
||||
keys.insert(storage_entry.key);
|
||||
}
|
||||
|
||||
if last_log.elapsed() >= LOG_INTERVAL {
|
||||
info!(
|
||||
target: "reth::cli",
|
||||
address = %address,
|
||||
entries_scanned = total_scanned,
|
||||
unique_keys = keys.len(),
|
||||
"Scanning static file storage changesets"
|
||||
);
|
||||
last_log = Instant::now();
|
||||
}
|
||||
}
|
||||
|
||||
info!(
|
||||
target: "reth::cli",
|
||||
address = %address,
|
||||
total_entries = total_scanned,
|
||||
unique_keys = keys.len(),
|
||||
"Finished static file storage changeset scan"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Collects storage keys from MDBX StorageChangeSets using parallel block range scanning.
|
||||
fn collect_mdbx_storage_keys_parallel<N: NodeTypesWithDB + ProviderNodeTypes>(
|
||||
&self,
|
||||
|
||||
630
crates/cli/commands/src/download.rs
Normal file
630
crates/cli/commands/src/download.rs
Normal file
@@ -0,0 +1,630 @@
|
||||
use crate::common::EnvironmentArgs;
|
||||
use clap::Parser;
|
||||
use eyre::Result;
|
||||
use lz4::Decoder;
|
||||
use reqwest::{blocking::Client as BlockingClient, header::RANGE, Client, StatusCode};
|
||||
use reth_chainspec::{EthChainSpec, EthereumHardforks};
|
||||
use reth_cli::chainspec::ChainSpecParser;
|
||||
use reth_fs_util as fs;
|
||||
use std::{
|
||||
borrow::Cow,
|
||||
fs::OpenOptions,
|
||||
io::{self, BufWriter, Read, Write},
|
||||
path::{Path, PathBuf},
|
||||
sync::{Arc, OnceLock},
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
use tar::Archive;
|
||||
use tokio::task;
|
||||
use tracing::info;
|
||||
use url::Url;
|
||||
use zstd::stream::read::Decoder as ZstdDecoder;
|
||||
|
||||
const BYTE_UNITS: [&str; 4] = ["B", "KB", "MB", "GB"];
|
||||
const MERKLE_BASE_URL: &str = "https://downloads.merkle.io";
|
||||
const EXTENSION_TAR_LZ4: &str = ".tar.lz4";
|
||||
const EXTENSION_TAR_ZSTD: &str = ".tar.zst";
|
||||
|
||||
/// Global static download defaults
|
||||
static DOWNLOAD_DEFAULTS: OnceLock<DownloadDefaults> = OnceLock::new();
|
||||
|
||||
/// Download configuration defaults
|
||||
///
|
||||
/// Global defaults can be set via [`DownloadDefaults::try_init`].
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct DownloadDefaults {
|
||||
/// List of available snapshot sources
|
||||
pub available_snapshots: Vec<Cow<'static, str>>,
|
||||
/// Default base URL for snapshots
|
||||
pub default_base_url: Cow<'static, str>,
|
||||
/// Default base URL for chain-aware snapshots.
|
||||
///
|
||||
/// When set, the chain ID is appended to form the full URL: `{base_url}/{chain_id}`.
|
||||
/// For example, given a base URL of `https://snapshots.example.com` and chain ID `1`,
|
||||
/// the resulting URL would be `https://snapshots.example.com/1`.
|
||||
///
|
||||
/// Falls back to [`default_base_url`](Self::default_base_url) when `None`.
|
||||
pub default_chain_aware_base_url: Option<Cow<'static, str>>,
|
||||
/// Optional custom long help text that overrides the generated help
|
||||
pub long_help: Option<String>,
|
||||
}
|
||||
|
||||
impl DownloadDefaults {
|
||||
/// Initialize the global download defaults with this configuration
|
||||
pub fn try_init(self) -> Result<(), Self> {
|
||||
DOWNLOAD_DEFAULTS.set(self)
|
||||
}
|
||||
|
||||
/// Get a reference to the global download defaults
|
||||
pub fn get_global() -> &'static DownloadDefaults {
|
||||
DOWNLOAD_DEFAULTS.get_or_init(DownloadDefaults::default_download_defaults)
|
||||
}
|
||||
|
||||
/// Default download configuration with defaults from merkle.io and publicnode
|
||||
pub fn default_download_defaults() -> Self {
|
||||
Self {
|
||||
available_snapshots: vec![
|
||||
Cow::Borrowed("https://www.merkle.io/snapshots (default, mainnet archive)"),
|
||||
Cow::Borrowed("https://publicnode.com/snapshots (full nodes & testnets)"),
|
||||
],
|
||||
default_base_url: Cow::Borrowed(MERKLE_BASE_URL),
|
||||
default_chain_aware_base_url: None,
|
||||
long_help: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Generates the long help text for the download URL argument using these defaults.
|
||||
///
|
||||
/// If a custom long_help is set, it will be returned. Otherwise, help text is generated
|
||||
/// from the available_snapshots list.
|
||||
pub fn long_help(&self) -> String {
|
||||
if let Some(ref custom_help) = self.long_help {
|
||||
return custom_help.clone();
|
||||
}
|
||||
|
||||
let mut help = String::from(
|
||||
"Specify a snapshot URL or let the command propose a default one.\n\nAvailable snapshot sources:\n",
|
||||
);
|
||||
|
||||
for source in &self.available_snapshots {
|
||||
help.push_str("- ");
|
||||
help.push_str(source);
|
||||
help.push('\n');
|
||||
}
|
||||
|
||||
help.push_str(
|
||||
"\nIf no URL is provided, the latest archive snapshot for the selected chain\nwill be proposed for download from ",
|
||||
);
|
||||
help.push_str(
|
||||
self.default_chain_aware_base_url.as_deref().unwrap_or(&self.default_base_url),
|
||||
);
|
||||
help.push_str(
|
||||
".\n\nLocal file:// URLs are also supported for extracting snapshots from disk.",
|
||||
);
|
||||
help
|
||||
}
|
||||
|
||||
/// Add a snapshot source to the list
|
||||
pub fn with_snapshot(mut self, source: impl Into<Cow<'static, str>>) -> Self {
|
||||
self.available_snapshots.push(source.into());
|
||||
self
|
||||
}
|
||||
|
||||
/// Replace all snapshot sources
|
||||
pub fn with_snapshots(mut self, sources: Vec<Cow<'static, str>>) -> Self {
|
||||
self.available_snapshots = sources;
|
||||
self
|
||||
}
|
||||
|
||||
/// Set the default base URL, e.g. `https://downloads.merkle.io`.
|
||||
pub fn with_base_url(mut self, url: impl Into<Cow<'static, str>>) -> Self {
|
||||
self.default_base_url = url.into();
|
||||
self
|
||||
}
|
||||
|
||||
/// Set the default chain-aware base URL.
|
||||
pub fn with_chain_aware_base_url(mut self, url: impl Into<Cow<'static, str>>) -> Self {
|
||||
self.default_chain_aware_base_url = Some(url.into());
|
||||
self
|
||||
}
|
||||
|
||||
/// Builder: Set custom long help text, overriding the generated help
|
||||
pub fn with_long_help(mut self, help: impl Into<String>) -> Self {
|
||||
self.long_help = Some(help.into());
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for DownloadDefaults {
|
||||
fn default() -> Self {
|
||||
Self::default_download_defaults()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Parser)]
|
||||
pub struct DownloadCommand<C: ChainSpecParser> {
|
||||
#[command(flatten)]
|
||||
env: EnvironmentArgs<C>,
|
||||
|
||||
/// Custom URL to download the snapshot from
|
||||
#[arg(long, short, long_help = DownloadDefaults::get_global().long_help())]
|
||||
url: Option<String>,
|
||||
}
|
||||
|
||||
impl<C: ChainSpecParser<ChainSpec: EthChainSpec + EthereumHardforks>> DownloadCommand<C> {
|
||||
pub async fn execute<N>(self) -> Result<()> {
|
||||
let data_dir = self.env.datadir.resolve_datadir(self.env.chain.chain());
|
||||
fs::create_dir_all(&data_dir)?;
|
||||
|
||||
let url = match self.url {
|
||||
Some(url) => url,
|
||||
None => {
|
||||
let url = get_latest_snapshot_url(self.env.chain.chain().id()).await?;
|
||||
info!(target: "reth::cli", "Using default snapshot URL: {}", url);
|
||||
url
|
||||
}
|
||||
};
|
||||
|
||||
info!(target: "reth::cli",
|
||||
chain = %self.env.chain.chain(),
|
||||
dir = ?data_dir.data_dir(),
|
||||
url = %url,
|
||||
"Starting snapshot download and extraction"
|
||||
);
|
||||
|
||||
stream_and_extract(&url, data_dir.data_dir()).await?;
|
||||
info!(target: "reth::cli", "Snapshot downloaded and extracted successfully");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl<C: ChainSpecParser> DownloadCommand<C> {
|
||||
/// Returns the underlying chain being used to run this command
|
||||
pub fn chain_spec(&self) -> Option<&Arc<C::ChainSpec>> {
|
||||
Some(&self.env.chain)
|
||||
}
|
||||
}
|
||||
|
||||
// Monitor process status and display progress every 100ms
|
||||
// to avoid overwhelming stdout
|
||||
struct DownloadProgress {
|
||||
downloaded: u64,
|
||||
total_size: u64,
|
||||
last_displayed: Instant,
|
||||
started_at: Instant,
|
||||
}
|
||||
|
||||
impl DownloadProgress {
|
||||
/// Creates new progress tracker with given total size
|
||||
fn new(total_size: u64) -> Self {
|
||||
let now = Instant::now();
|
||||
Self { downloaded: 0, total_size, last_displayed: now, started_at: now }
|
||||
}
|
||||
|
||||
/// Converts bytes to human readable format (B, KB, MB, GB)
|
||||
fn format_size(size: u64) -> String {
|
||||
let mut size = size as f64;
|
||||
let mut unit_index = 0;
|
||||
|
||||
while size >= 1024.0 && unit_index < BYTE_UNITS.len() - 1 {
|
||||
size /= 1024.0;
|
||||
unit_index += 1;
|
||||
}
|
||||
|
||||
format!("{:.2} {}", size, BYTE_UNITS[unit_index])
|
||||
}
|
||||
|
||||
/// Format duration as human readable string
|
||||
fn format_duration(duration: Duration) -> String {
|
||||
let secs = duration.as_secs();
|
||||
if secs < 60 {
|
||||
format!("{secs}s")
|
||||
} else if secs < 3600 {
|
||||
format!("{}m {}s", secs / 60, secs % 60)
|
||||
} else {
|
||||
format!("{}h {}m", secs / 3600, (secs % 3600) / 60)
|
||||
}
|
||||
}
|
||||
|
||||
/// Updates progress bar
|
||||
fn update(&mut self, chunk_size: u64) -> Result<()> {
|
||||
self.downloaded += chunk_size;
|
||||
|
||||
// Only update display at most 10 times per second for efficiency
|
||||
if self.last_displayed.elapsed() >= Duration::from_millis(100) {
|
||||
let formatted_downloaded = Self::format_size(self.downloaded);
|
||||
let formatted_total = Self::format_size(self.total_size);
|
||||
let progress = (self.downloaded as f64 / self.total_size as f64) * 100.0;
|
||||
|
||||
// Calculate ETA based on current speed
|
||||
let elapsed = self.started_at.elapsed();
|
||||
let eta = if self.downloaded > 0 {
|
||||
let remaining = self.total_size.saturating_sub(self.downloaded);
|
||||
let speed = self.downloaded as f64 / elapsed.as_secs_f64();
|
||||
if speed > 0.0 {
|
||||
Duration::from_secs_f64(remaining as f64 / speed)
|
||||
} else {
|
||||
Duration::ZERO
|
||||
}
|
||||
} else {
|
||||
Duration::ZERO
|
||||
};
|
||||
let eta_str = Self::format_duration(eta);
|
||||
|
||||
// Pad with spaces to clear any previous longer line
|
||||
print!(
|
||||
"\rDownloading and extracting... {progress:.2}% ({formatted_downloaded} / {formatted_total}) ETA: {eta_str} ",
|
||||
);
|
||||
io::stdout().flush()?;
|
||||
self.last_displayed = Instant::now();
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Adapter to track progress while reading
|
||||
struct ProgressReader<R> {
|
||||
reader: R,
|
||||
progress: DownloadProgress,
|
||||
}
|
||||
|
||||
impl<R: Read> ProgressReader<R> {
|
||||
fn new(reader: R, total_size: u64) -> Self {
|
||||
Self { reader, progress: DownloadProgress::new(total_size) }
|
||||
}
|
||||
}
|
||||
|
||||
impl<R: Read> Read for ProgressReader<R> {
|
||||
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
|
||||
let bytes = self.reader.read(buf)?;
|
||||
if bytes > 0 &&
|
||||
let Err(e) = self.progress.update(bytes as u64)
|
||||
{
|
||||
return Err(io::Error::other(e));
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
}
|
||||
|
||||
/// Supported compression formats for snapshots
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
enum CompressionFormat {
|
||||
Lz4,
|
||||
Zstd,
|
||||
}
|
||||
|
||||
impl CompressionFormat {
|
||||
/// Detect compression format from file extension
|
||||
fn from_url(url: &str) -> Result<Self> {
|
||||
let path =
|
||||
Url::parse(url).map(|u| u.path().to_string()).unwrap_or_else(|_| url.to_string());
|
||||
|
||||
if path.ends_with(EXTENSION_TAR_LZ4) {
|
||||
Ok(Self::Lz4)
|
||||
} else if path.ends_with(EXTENSION_TAR_ZSTD) {
|
||||
Ok(Self::Zstd)
|
||||
} else {
|
||||
Err(eyre::eyre!(
|
||||
"Unsupported file format. Expected .tar.lz4 or .tar.zst, got: {}",
|
||||
path
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Extracts a compressed tar archive to the target directory with progress tracking.
|
||||
fn extract_archive<R: Read>(
|
||||
reader: R,
|
||||
total_size: u64,
|
||||
format: CompressionFormat,
|
||||
target_dir: &Path,
|
||||
) -> Result<()> {
|
||||
let progress_reader = ProgressReader::new(reader, total_size);
|
||||
|
||||
match format {
|
||||
CompressionFormat::Lz4 => {
|
||||
let decoder = Decoder::new(progress_reader)?;
|
||||
Archive::new(decoder).unpack(target_dir)?;
|
||||
}
|
||||
CompressionFormat::Zstd => {
|
||||
let decoder = ZstdDecoder::new(progress_reader)?;
|
||||
Archive::new(decoder).unpack(target_dir)?;
|
||||
}
|
||||
}
|
||||
|
||||
info!(target: "reth::cli", "Extraction complete.");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Extracts a snapshot from a local file.
|
||||
fn extract_from_file(path: &Path, format: CompressionFormat, target_dir: &Path) -> Result<()> {
|
||||
let file = std::fs::File::open(path)?;
|
||||
let total_size = file.metadata()?.len();
|
||||
extract_archive(file, total_size, format, target_dir)
|
||||
}
|
||||
|
||||
const MAX_DOWNLOAD_RETRIES: u32 = 10;
|
||||
const RETRY_BACKOFF_SECS: u64 = 5;
|
||||
|
||||
/// Wrapper that tracks download progress while writing data.
|
||||
/// Used with [`io::copy`] to display progress during downloads.
|
||||
struct ProgressWriter<W> {
|
||||
inner: W,
|
||||
progress: DownloadProgress,
|
||||
}
|
||||
|
||||
impl<W: Write> Write for ProgressWriter<W> {
|
||||
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
|
||||
let n = self.inner.write(buf)?;
|
||||
let _ = self.progress.update(n as u64);
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> io::Result<()> {
|
||||
self.inner.flush()
|
||||
}
|
||||
}
|
||||
|
||||
/// Downloads a file with resume support using HTTP Range requests.
|
||||
/// Automatically retries on failure, resuming from where it left off.
|
||||
/// Returns the path to the downloaded file and its total size.
|
||||
fn resumable_download(url: &str, target_dir: &Path) -> Result<(PathBuf, u64)> {
|
||||
let file_name = Url::parse(url)
|
||||
.ok()
|
||||
.and_then(|u| u.path_segments()?.next_back().map(|s| s.to_string()))
|
||||
.unwrap_or_else(|| "snapshot.tar".to_string());
|
||||
|
||||
let final_path = target_dir.join(&file_name);
|
||||
let part_path = target_dir.join(format!("{file_name}.part"));
|
||||
|
||||
let client = BlockingClient::builder().timeout(Duration::from_secs(30)).build()?;
|
||||
|
||||
let mut total_size: Option<u64> = None;
|
||||
let mut last_error: Option<eyre::Error> = None;
|
||||
|
||||
let finalize_download = |size: u64| -> Result<(PathBuf, u64)> {
|
||||
fs::rename(&part_path, &final_path)?;
|
||||
info!(target: "reth::cli", "Download complete: {}", final_path.display());
|
||||
Ok((final_path.clone(), size))
|
||||
};
|
||||
|
||||
for attempt in 1..=MAX_DOWNLOAD_RETRIES {
|
||||
let existing_size = fs::metadata(&part_path).map(|m| m.len()).unwrap_or(0);
|
||||
|
||||
if let Some(total) = total_size &&
|
||||
existing_size >= total
|
||||
{
|
||||
return finalize_download(total);
|
||||
}
|
||||
|
||||
if attempt > 1 {
|
||||
info!(target: "reth::cli",
|
||||
"Retry attempt {}/{} - resuming from {} bytes",
|
||||
attempt, MAX_DOWNLOAD_RETRIES, existing_size
|
||||
);
|
||||
}
|
||||
|
||||
let mut request = client.get(url);
|
||||
if existing_size > 0 {
|
||||
request = request.header(RANGE, format!("bytes={existing_size}-"));
|
||||
if attempt == 1 {
|
||||
info!(target: "reth::cli", "Resuming download from {} bytes", existing_size);
|
||||
}
|
||||
}
|
||||
|
||||
let response = match request.send().and_then(|r| r.error_for_status()) {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
last_error = Some(e.into());
|
||||
if attempt < MAX_DOWNLOAD_RETRIES {
|
||||
info!(target: "reth::cli",
|
||||
"Download failed, retrying in {} seconds...", RETRY_BACKOFF_SECS
|
||||
);
|
||||
std::thread::sleep(Duration::from_secs(RETRY_BACKOFF_SECS));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
let is_partial = response.status() == StatusCode::PARTIAL_CONTENT;
|
||||
|
||||
let size = if is_partial {
|
||||
response
|
||||
.headers()
|
||||
.get("Content-Range")
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|v| v.split('/').next_back())
|
||||
.and_then(|v| v.parse().ok())
|
||||
} else {
|
||||
response.content_length()
|
||||
};
|
||||
|
||||
if total_size.is_none() {
|
||||
total_size = size;
|
||||
}
|
||||
|
||||
let current_total = total_size.ok_or_else(|| {
|
||||
eyre::eyre!("Server did not provide Content-Length or Content-Range header")
|
||||
})?;
|
||||
|
||||
let file = if is_partial && existing_size > 0 {
|
||||
OpenOptions::new()
|
||||
.append(true)
|
||||
.open(&part_path)
|
||||
.map_err(|e| fs::FsPathError::open(e, &part_path))?
|
||||
} else {
|
||||
fs::create_file(&part_path)?
|
||||
};
|
||||
|
||||
let start_offset = if is_partial { existing_size } else { 0 };
|
||||
let mut progress = DownloadProgress::new(current_total);
|
||||
progress.downloaded = start_offset;
|
||||
|
||||
let mut writer = ProgressWriter { inner: BufWriter::new(file), progress };
|
||||
let mut reader = response;
|
||||
|
||||
let copy_result = io::copy(&mut reader, &mut writer);
|
||||
let flush_result = writer.inner.flush();
|
||||
println!();
|
||||
|
||||
if let Err(e) = copy_result.and(flush_result) {
|
||||
last_error = Some(e.into());
|
||||
if attempt < MAX_DOWNLOAD_RETRIES {
|
||||
info!(target: "reth::cli",
|
||||
"Download interrupted, retrying in {} seconds...", RETRY_BACKOFF_SECS
|
||||
);
|
||||
std::thread::sleep(Duration::from_secs(RETRY_BACKOFF_SECS));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
return finalize_download(current_total);
|
||||
}
|
||||
|
||||
Err(last_error
|
||||
.unwrap_or_else(|| eyre::eyre!("Download failed after {} attempts", MAX_DOWNLOAD_RETRIES)))
|
||||
}
|
||||
|
||||
/// Fetches the snapshot from a remote URL with resume support, then extracts it.
|
||||
fn download_and_extract(url: &str, format: CompressionFormat, target_dir: &Path) -> Result<()> {
|
||||
let (downloaded_path, total_size) = resumable_download(url, target_dir)?;
|
||||
|
||||
info!(target: "reth::cli", "Extracting snapshot...");
|
||||
let file = fs::open(&downloaded_path)?;
|
||||
extract_archive(file, total_size, format, target_dir)?;
|
||||
|
||||
fs::remove_file(&downloaded_path)?;
|
||||
info!(target: "reth::cli", "Removed downloaded archive");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Downloads and extracts a snapshot, blocking until finished.
|
||||
///
|
||||
/// Supports both `file://` URLs for local files and HTTP(S) URLs for remote downloads.
|
||||
fn blocking_download_and_extract(url: &str, target_dir: &Path) -> Result<()> {
|
||||
let format = CompressionFormat::from_url(url)?;
|
||||
|
||||
if let Ok(parsed_url) = Url::parse(url) &&
|
||||
parsed_url.scheme() == "file"
|
||||
{
|
||||
let file_path = parsed_url
|
||||
.to_file_path()
|
||||
.map_err(|_| eyre::eyre!("Invalid file:// URL path: {}", url))?;
|
||||
extract_from_file(&file_path, format, target_dir)
|
||||
} else {
|
||||
download_and_extract(url, format, target_dir)
|
||||
}
|
||||
}
|
||||
|
||||
async fn stream_and_extract(url: &str, target_dir: &Path) -> Result<()> {
|
||||
let target_dir = target_dir.to_path_buf();
|
||||
let url = url.to_string();
|
||||
task::spawn_blocking(move || blocking_download_and_extract(&url, &target_dir)).await??;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// Builds default URL for latest mainnet archive snapshot using configured defaults
|
||||
async fn get_latest_snapshot_url(chain_id: u64) -> Result<String> {
|
||||
let defaults = DownloadDefaults::get_global();
|
||||
let base_url = match &defaults.default_chain_aware_base_url {
|
||||
Some(url) => format!("{url}/{chain_id}"),
|
||||
None => defaults.default_base_url.to_string(),
|
||||
};
|
||||
let latest_url = format!("{base_url}/latest.txt");
|
||||
let filename = Client::new()
|
||||
.get(latest_url)
|
||||
.send()
|
||||
.await?
|
||||
.error_for_status()?
|
||||
.text()
|
||||
.await?
|
||||
.trim()
|
||||
.to_string();
|
||||
|
||||
Ok(format!("{base_url}/{filename}"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_download_defaults_builder() {
|
||||
let defaults = DownloadDefaults::default()
|
||||
.with_snapshot("https://example.com/snapshots (example)")
|
||||
.with_base_url("https://example.com");
|
||||
|
||||
assert_eq!(defaults.default_base_url, "https://example.com");
|
||||
assert_eq!(defaults.available_snapshots.len(), 3); // 2 defaults + 1 added
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_download_defaults_replace_snapshots() {
|
||||
let defaults = DownloadDefaults::default().with_snapshots(vec![
|
||||
Cow::Borrowed("https://custom1.com"),
|
||||
Cow::Borrowed("https://custom2.com"),
|
||||
]);
|
||||
|
||||
assert_eq!(defaults.available_snapshots.len(), 2);
|
||||
assert_eq!(defaults.available_snapshots[0], "https://custom1.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_long_help_generation() {
|
||||
let defaults = DownloadDefaults::default();
|
||||
let help = defaults.long_help();
|
||||
|
||||
assert!(help.contains("Available snapshot sources:"));
|
||||
assert!(help.contains("merkle.io"));
|
||||
assert!(help.contains("publicnode.com"));
|
||||
assert!(help.contains("file://"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_long_help_override() {
|
||||
let custom_help = "This is custom help text for downloading snapshots.";
|
||||
let defaults = DownloadDefaults::default().with_long_help(custom_help);
|
||||
|
||||
let help = defaults.long_help();
|
||||
assert_eq!(help, custom_help);
|
||||
assert!(!help.contains("Available snapshot sources:"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_builder_chaining() {
|
||||
let defaults = DownloadDefaults::default()
|
||||
.with_base_url("https://custom.example.com")
|
||||
.with_snapshot("https://snapshot1.com")
|
||||
.with_snapshot("https://snapshot2.com")
|
||||
.with_long_help("Custom help for snapshots");
|
||||
|
||||
assert_eq!(defaults.default_base_url, "https://custom.example.com");
|
||||
assert_eq!(defaults.available_snapshots.len(), 4); // 2 defaults + 2 added
|
||||
assert_eq!(defaults.long_help, Some("Custom help for snapshots".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compression_format_detection() {
|
||||
assert!(matches!(
|
||||
CompressionFormat::from_url("https://example.com/snapshot.tar.lz4"),
|
||||
Ok(CompressionFormat::Lz4)
|
||||
));
|
||||
assert!(matches!(
|
||||
CompressionFormat::from_url("https://example.com/snapshot.tar.zst"),
|
||||
Ok(CompressionFormat::Zstd)
|
||||
));
|
||||
assert!(matches!(
|
||||
CompressionFormat::from_url("file:///path/to/snapshot.tar.lz4"),
|
||||
Ok(CompressionFormat::Lz4)
|
||||
));
|
||||
assert!(matches!(
|
||||
CompressionFormat::from_url("file:///path/to/snapshot.tar.zst"),
|
||||
Ok(CompressionFormat::Zstd)
|
||||
));
|
||||
assert!(CompressionFormat::from_url("https://example.com/snapshot.tar.gz").is_err());
|
||||
}
|
||||
}
|
||||
@@ -1,610 +0,0 @@
|
||||
use crate::download::{
|
||||
manifest::{ComponentManifest, ComponentSelection, SnapshotComponentType, SnapshotManifest},
|
||||
SelectionPreset,
|
||||
};
|
||||
use reth_chainspec::{EthereumHardfork, EthereumHardforks};
|
||||
use reth_config::config::{BlocksPerFileConfig, Config, PruneConfig, StaticFilesConfig};
|
||||
use reth_db::tables;
|
||||
use reth_db_api::transaction::{DbTx, DbTxMut};
|
||||
use reth_node_core::args::DefaultPruningValues;
|
||||
use reth_prune_types::{PruneCheckpoint, PruneMode, PruneSegment};
|
||||
use reth_stages_types::StageCheckpoint;
|
||||
use std::{collections::BTreeMap, path::Path};
|
||||
use tracing::info;
|
||||
|
||||
/// Minimum blocks to keep for receipts, matching `--minimal` prune settings.
|
||||
const MINIMUM_RECEIPTS_DISTANCE: u64 = 64;
|
||||
|
||||
/// Minimum blocks to keep for history/bodies, matching `--minimal` prune settings
|
||||
/// (`MINIMUM_UNWIND_SAFE_DISTANCE`).
|
||||
const MINIMUM_HISTORY_DISTANCE: u64 = 10064;
|
||||
|
||||
/// Writes a [`Config`] as TOML to `<data_dir>/reth.toml`.
|
||||
///
|
||||
/// If the file already exists, it is not overwritten. Returns `true` if the file was written.
|
||||
pub fn write_config(config: &Config, data_dir: &Path) -> eyre::Result<bool> {
|
||||
let config_path = data_dir.join("reth.toml");
|
||||
|
||||
if config_path.exists() {
|
||||
info!(target: "reth::cli",
|
||||
path = ?config_path,
|
||||
"reth.toml already exists, skipping config generation"
|
||||
);
|
||||
return Ok(false);
|
||||
}
|
||||
|
||||
let toml_str = toml::to_string_pretty(config)?;
|
||||
reth_fs_util::write(&config_path, toml_str)?;
|
||||
|
||||
info!(target: "reth::cli",
|
||||
path = ?config_path,
|
||||
"Generated reth.toml based on downloaded components"
|
||||
);
|
||||
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Writes prune checkpoints to the provided write transaction.
|
||||
pub(crate) fn write_prune_checkpoints_tx<Tx>(
|
||||
tx: &Tx,
|
||||
config: &Config,
|
||||
snapshot_block: u64,
|
||||
) -> eyre::Result<()>
|
||||
where
|
||||
Tx: DbTx + DbTxMut,
|
||||
{
|
||||
let segments = &config.prune.segments;
|
||||
|
||||
// Collect (segment, mode) pairs for all configured prune segments
|
||||
let checkpoints: Vec<(PruneSegment, PruneMode)> = [
|
||||
(PruneSegment::SenderRecovery, segments.sender_recovery),
|
||||
(PruneSegment::TransactionLookup, segments.transaction_lookup),
|
||||
(PruneSegment::Receipts, segments.receipts),
|
||||
(PruneSegment::AccountHistory, segments.account_history),
|
||||
(PruneSegment::StorageHistory, segments.storage_history),
|
||||
(PruneSegment::Bodies, segments.bodies_history),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(segment, mode)| mode.map(|m| (segment, m)))
|
||||
.collect();
|
||||
|
||||
if checkpoints.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Look up the last tx number for the snapshot block from BlockBodyIndices
|
||||
let tx_number =
|
||||
tx.get::<tables::BlockBodyIndices>(snapshot_block)?.map(|indices| indices.last_tx_num());
|
||||
|
||||
for (segment, prune_mode) in &checkpoints {
|
||||
let checkpoint = PruneCheckpoint {
|
||||
block_number: Some(snapshot_block),
|
||||
tx_number,
|
||||
prune_mode: *prune_mode,
|
||||
};
|
||||
|
||||
tx.put::<tables::PruneCheckpoints>(*segment, checkpoint)?;
|
||||
|
||||
info!(target: "reth::cli",
|
||||
segment = %segment,
|
||||
block = snapshot_block,
|
||||
tx = ?tx_number,
|
||||
mode = ?prune_mode,
|
||||
"Set prune checkpoint"
|
||||
);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Stage IDs for index stages whose output is stored in RocksDB and is never
|
||||
/// distributed in snapshots.
|
||||
const INDEX_STAGE_IDS: [&str; 3] =
|
||||
["TransactionLookup", "IndexAccountHistory", "IndexStorageHistory"];
|
||||
|
||||
/// Prune segments that correspond to the index stages.
|
||||
const INDEX_PRUNE_SEGMENTS: [PruneSegment; 3] =
|
||||
[PruneSegment::TransactionLookup, PruneSegment::AccountHistory, PruneSegment::StorageHistory];
|
||||
|
||||
/// Resets stage and prune checkpoints for stages whose output is not included
|
||||
/// in the snapshot inside an existing write transaction.
|
||||
///
|
||||
/// A snapshot's mdbx comes from a fully synced node, so it has stage checkpoints
|
||||
/// at the tip for `TransactionLookup`, `IndexAccountHistory`, and
|
||||
/// `IndexStorageHistory`. Since we don't distribute the rocksdb indices those
|
||||
/// stages produced, we must reset their checkpoints to block 0. Otherwise the
|
||||
/// pipeline would see "already done" and skip rebuilding entirely.
|
||||
///
|
||||
/// We intentionally do not reset `SenderRecovery`: sender static files are
|
||||
/// distributed for archive downloads, and non-archive downloads rely on the
|
||||
/// configured prune checkpoints for this segment.
|
||||
pub(crate) fn reset_index_stage_checkpoints_tx<Tx>(tx: &Tx) -> eyre::Result<()>
|
||||
where
|
||||
Tx: DbTx + DbTxMut,
|
||||
{
|
||||
for stage_id in INDEX_STAGE_IDS {
|
||||
tx.put::<tables::StageCheckpoints>(stage_id.to_string(), StageCheckpoint::default())?;
|
||||
|
||||
// Also clear any stage-specific progress data
|
||||
tx.delete::<tables::StageCheckpointProgresses>(stage_id.to_string(), None)?;
|
||||
|
||||
info!(target: "reth::cli", stage = stage_id, "Reset stage checkpoint to block 0");
|
||||
}
|
||||
|
||||
// Clear corresponding prune checkpoints so the pruner doesn't inherit
|
||||
// state from the source node
|
||||
for segment in INDEX_PRUNE_SEGMENTS {
|
||||
tx.delete::<tables::PruneCheckpoints>(segment, None)?;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Generates a [`Config`] from per-component range selections.
|
||||
///
|
||||
/// When all data components are selected as `All`, no pruning is configured (archive node).
|
||||
/// Otherwise, `--minimal` style pruning is applied for missing/partial components.
|
||||
pub(crate) fn config_for_selections(
|
||||
selections: &BTreeMap<SnapshotComponentType, ComponentSelection>,
|
||||
manifest: &SnapshotManifest,
|
||||
preset: Option<SelectionPreset>,
|
||||
chain_spec: Option<&impl EthereumHardforks>,
|
||||
) -> Config {
|
||||
let selection_for = |ty| selections.get(&ty).copied().unwrap_or(ComponentSelection::None);
|
||||
|
||||
let tx_sel = selection_for(SnapshotComponentType::Transactions);
|
||||
let senders_sel = selection_for(SnapshotComponentType::TransactionSenders);
|
||||
let receipt_sel = selection_for(SnapshotComponentType::Receipts);
|
||||
let account_cs_sel = selection_for(SnapshotComponentType::AccountChangesets);
|
||||
let storage_cs_sel = selection_for(SnapshotComponentType::StorageChangesets);
|
||||
|
||||
// Archive node — all data components present, no pruning
|
||||
let is_archive = [tx_sel, senders_sel, receipt_sel, account_cs_sel, storage_cs_sel]
|
||||
.iter()
|
||||
.all(|s| *s == ComponentSelection::All);
|
||||
|
||||
// Extract blocks_per_file from manifest for all component types
|
||||
let blocks_per_file = |ty: SnapshotComponentType| -> Option<u64> {
|
||||
match manifest.component(ty)? {
|
||||
ComponentManifest::Chunked(c) => Some(c.blocks_per_file),
|
||||
ComponentManifest::Single(_) => None,
|
||||
}
|
||||
};
|
||||
let static_files = StaticFilesConfig {
|
||||
blocks_per_file: BlocksPerFileConfig {
|
||||
headers: blocks_per_file(SnapshotComponentType::Headers),
|
||||
transactions: blocks_per_file(SnapshotComponentType::Transactions),
|
||||
receipts: blocks_per_file(SnapshotComponentType::Receipts),
|
||||
transaction_senders: blocks_per_file(SnapshotComponentType::TransactionSenders),
|
||||
account_change_sets: blocks_per_file(SnapshotComponentType::AccountChangesets),
|
||||
storage_change_sets: blocks_per_file(SnapshotComponentType::StorageChangesets),
|
||||
},
|
||||
};
|
||||
|
||||
if is_archive || matches!(preset, Some(SelectionPreset::Archive)) {
|
||||
return Config { static_files, ..Default::default() };
|
||||
}
|
||||
|
||||
if matches!(preset, Some(SelectionPreset::Full)) {
|
||||
let defaults = DefaultPruningValues::get_global();
|
||||
let mut segments = defaults.full_prune_modes.clone();
|
||||
|
||||
if defaults.full_bodies_history_use_pre_merge {
|
||||
segments.bodies_history = chain_spec.and_then(|chain_spec| {
|
||||
chain_spec
|
||||
.ethereum_fork_activation(EthereumHardfork::Paris)
|
||||
.block_number()
|
||||
.map(PruneMode::Before)
|
||||
});
|
||||
}
|
||||
|
||||
return Config {
|
||||
prune: PruneConfig { segments, ..Default::default() },
|
||||
static_files,
|
||||
..Default::default()
|
||||
};
|
||||
}
|
||||
|
||||
let mut config = Config::default();
|
||||
let mut prune = PruneConfig::default();
|
||||
|
||||
if senders_sel != ComponentSelection::All {
|
||||
prune.segments.sender_recovery = Some(PruneMode::Full);
|
||||
}
|
||||
prune.segments.transaction_lookup = Some(PruneMode::Full);
|
||||
|
||||
if let Some(mode) = selection_to_prune_mode(tx_sel, Some(MINIMUM_HISTORY_DISTANCE)) {
|
||||
prune.segments.bodies_history = Some(mode);
|
||||
}
|
||||
|
||||
if let Some(mode) = selection_to_prune_mode(receipt_sel, Some(MINIMUM_RECEIPTS_DISTANCE)) {
|
||||
prune.segments.receipts = Some(mode);
|
||||
}
|
||||
|
||||
if let Some(mode) = selection_to_prune_mode(account_cs_sel, Some(MINIMUM_HISTORY_DISTANCE)) {
|
||||
prune.segments.account_history = Some(mode);
|
||||
}
|
||||
|
||||
if let Some(mode) = selection_to_prune_mode(storage_cs_sel, Some(MINIMUM_HISTORY_DISTANCE)) {
|
||||
prune.segments.storage_history = Some(mode);
|
||||
}
|
||||
|
||||
config.prune = prune;
|
||||
config.static_files = static_files;
|
||||
config
|
||||
}
|
||||
|
||||
/// Converts a [`ComponentSelection`] to an optional [`PruneMode`].
|
||||
///
|
||||
/// `min_distance` enforces the minimum blocks required for this segment.
|
||||
/// When set, `None` and distances below the minimum are clamped to it
|
||||
/// instead of producing `PruneMode::Full` which reth would reject.
|
||||
fn selection_to_prune_mode(
|
||||
sel: ComponentSelection,
|
||||
min_distance: Option<u64>,
|
||||
) -> Option<PruneMode> {
|
||||
match sel {
|
||||
ComponentSelection::All => None,
|
||||
ComponentSelection::Distance(d) => {
|
||||
Some(PruneMode::Distance(min_distance.map_or(d, |min| d.max(min))))
|
||||
}
|
||||
ComponentSelection::None => Some(min_distance.map_or(PruneMode::Full, PruneMode::Distance)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Human-readable prune config summary.
|
||||
pub(crate) fn describe_prune_config(config: &Config) -> Vec<String> {
|
||||
let segments = &config.prune.segments;
|
||||
|
||||
[
|
||||
("sender_recovery", segments.sender_recovery),
|
||||
("transaction_lookup", segments.transaction_lookup),
|
||||
("bodies_history", segments.bodies_history),
|
||||
("receipts", segments.receipts),
|
||||
("account_history", segments.account_history),
|
||||
("storage_history", segments.storage_history),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(name, mode)| mode.map(|m| format!("{name}={}", format_mode(&m))))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn format_mode(mode: &PruneMode) -> String {
|
||||
match mode {
|
||||
PruneMode::Full => "\"full\"".to_string(),
|
||||
PruneMode::Distance(d) => format!("{{ distance = {d} }}"),
|
||||
PruneMode::Before(b) => format!("{{ before = {b} }}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use reth_db::Database;
|
||||
|
||||
/// Empty manifest for tests that only care about prune config.
|
||||
fn empty_manifest() -> SnapshotManifest {
|
||||
SnapshotManifest {
|
||||
block: 0,
|
||||
chain_id: 1,
|
||||
storage_version: 2,
|
||||
timestamp: 0,
|
||||
base_url: None,
|
||||
reth_version: None,
|
||||
components: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_prune_checkpoints_sets_all_segments() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db = reth_db::init_db(dir.path(), reth_db::mdbx::DatabaseArguments::default()).unwrap();
|
||||
|
||||
let mut selections = BTreeMap::new();
|
||||
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
|
||||
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
let snapshot_block = 21_000_000;
|
||||
|
||||
{
|
||||
let tx = db.tx_mut().unwrap();
|
||||
write_prune_checkpoints_tx(&tx, &config, snapshot_block).unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
|
||||
// Verify all expected segments have checkpoints
|
||||
let tx = db.tx().unwrap();
|
||||
for segment in [
|
||||
PruneSegment::SenderRecovery,
|
||||
PruneSegment::TransactionLookup,
|
||||
PruneSegment::Receipts,
|
||||
PruneSegment::AccountHistory,
|
||||
PruneSegment::StorageHistory,
|
||||
PruneSegment::Bodies,
|
||||
] {
|
||||
let checkpoint = tx
|
||||
.get::<tables::PruneCheckpoints>(segment)
|
||||
.unwrap()
|
||||
.unwrap_or_else(|| panic!("expected checkpoint for {segment}"));
|
||||
assert_eq!(checkpoint.block_number, Some(snapshot_block));
|
||||
// No BlockBodyIndices in empty DB, so tx_number should be None
|
||||
assert_eq!(checkpoint.tx_number, None);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_prune_checkpoints_archive_no_checkpoints() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db = reth_db::init_db(dir.path(), reth_db::mdbx::DatabaseArguments::default()).unwrap();
|
||||
|
||||
// Archive node — no pruning configured, so no checkpoints written
|
||||
let mut selections = BTreeMap::new();
|
||||
for ty in SnapshotComponentType::ALL {
|
||||
selections.insert(ty, ComponentSelection::All);
|
||||
}
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
|
||||
{
|
||||
let tx = db.tx_mut().unwrap();
|
||||
write_prune_checkpoints_tx(&tx, &config, 21_000_000).unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
|
||||
let tx = db.tx().unwrap();
|
||||
for segment in [PruneSegment::SenderRecovery, PruneSegment::TransactionLookup] {
|
||||
assert!(
|
||||
tx.get::<tables::PruneCheckpoints>(segment).unwrap().is_none(),
|
||||
"expected no checkpoint for {segment} on archive node"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selections_all_no_pruning() {
|
||||
let mut selections = BTreeMap::new();
|
||||
for ty in SnapshotComponentType::ALL {
|
||||
selections.insert(ty, ComponentSelection::All);
|
||||
}
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
// Archive node — nothing pruned
|
||||
assert_eq!(config.prune.segments.transaction_lookup, None);
|
||||
assert_eq!(config.prune.segments.sender_recovery, None);
|
||||
assert_eq!(config.prune.segments.bodies_history, None);
|
||||
assert_eq!(config.prune.segments.receipts, None);
|
||||
assert_eq!(config.prune.segments.account_history, None);
|
||||
assert_eq!(config.prune.segments.storage_history, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selections_none_clamps_to_minimum_distance() {
|
||||
let mut selections = BTreeMap::new();
|
||||
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
|
||||
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
assert_eq!(config.prune.segments.transaction_lookup, Some(PruneMode::Full));
|
||||
assert_eq!(config.prune.segments.sender_recovery, Some(PruneMode::Full));
|
||||
// All segments clamped to their minimum distances
|
||||
assert_eq!(
|
||||
config.prune.segments.bodies_history,
|
||||
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
|
||||
);
|
||||
assert_eq!(
|
||||
config.prune.segments.receipts,
|
||||
Some(PruneMode::Distance(MINIMUM_RECEIPTS_DISTANCE))
|
||||
);
|
||||
assert_eq!(
|
||||
config.prune.segments.account_history,
|
||||
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
|
||||
);
|
||||
assert_eq!(
|
||||
config.prune.segments.storage_history,
|
||||
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selections_distance_maps_bodies_history() {
|
||||
let mut selections = BTreeMap::new();
|
||||
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
|
||||
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
|
||||
selections
|
||||
.insert(SnapshotComponentType::Transactions, ComponentSelection::Distance(10_064));
|
||||
selections.insert(SnapshotComponentType::Receipts, ComponentSelection::None);
|
||||
selections
|
||||
.insert(SnapshotComponentType::AccountChangesets, ComponentSelection::Distance(10_064));
|
||||
selections
|
||||
.insert(SnapshotComponentType::StorageChangesets, ComponentSelection::Distance(10_064));
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
|
||||
assert_eq!(config.prune.segments.transaction_lookup, Some(PruneMode::Full));
|
||||
assert_eq!(config.prune.segments.sender_recovery, Some(PruneMode::Full));
|
||||
// Bodies follows tx selection
|
||||
assert_eq!(config.prune.segments.bodies_history, Some(PruneMode::Distance(10_064)));
|
||||
assert_eq!(
|
||||
config.prune.segments.receipts,
|
||||
Some(PruneMode::Distance(MINIMUM_RECEIPTS_DISTANCE))
|
||||
);
|
||||
assert_eq!(config.prune.segments.account_history, Some(PruneMode::Distance(10_064)));
|
||||
assert_eq!(config.prune.segments.storage_history, Some(PruneMode::Distance(10_064)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn full_preset_matches_default_full_prune_config() {
|
||||
let mut selections = BTreeMap::new();
|
||||
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
|
||||
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
|
||||
selections
|
||||
.insert(SnapshotComponentType::Transactions, ComponentSelection::Distance(500_000));
|
||||
selections.insert(SnapshotComponentType::Receipts, ComponentSelection::Distance(10_064));
|
||||
|
||||
let chain_spec = reth_chainspec::MAINNET.clone();
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
Some(SelectionPreset::Full),
|
||||
Some(chain_spec.as_ref()),
|
||||
);
|
||||
|
||||
assert_eq!(config.prune.segments.sender_recovery, Some(PruneMode::Full));
|
||||
assert_eq!(config.prune.segments.transaction_lookup, None);
|
||||
assert_eq!(
|
||||
config.prune.segments.receipts,
|
||||
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
|
||||
);
|
||||
assert_eq!(
|
||||
config.prune.segments.account_history,
|
||||
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
|
||||
);
|
||||
assert_eq!(
|
||||
config.prune.segments.storage_history,
|
||||
Some(PruneMode::Distance(MINIMUM_HISTORY_DISTANCE))
|
||||
);
|
||||
|
||||
let paris_block = chain_spec
|
||||
.ethereum_fork_activation(EthereumHardfork::Paris)
|
||||
.block_number()
|
||||
.expect("mainnet Paris block should be known");
|
||||
assert_eq!(config.prune.segments.bodies_history, Some(PruneMode::Before(paris_block)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn describe_selections_all_no_pruning() {
|
||||
let mut selections = BTreeMap::new();
|
||||
for ty in SnapshotComponentType::ALL {
|
||||
selections.insert(ty, ComponentSelection::All);
|
||||
}
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
let desc = describe_prune_config(&config);
|
||||
// Archive node — no prune segments described
|
||||
assert!(desc.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn describe_selections_with_distances() {
|
||||
let mut selections = BTreeMap::new();
|
||||
selections.insert(SnapshotComponentType::State, ComponentSelection::All);
|
||||
selections.insert(SnapshotComponentType::Headers, ComponentSelection::All);
|
||||
selections
|
||||
.insert(SnapshotComponentType::Transactions, ComponentSelection::Distance(10_064));
|
||||
selections.insert(SnapshotComponentType::Receipts, ComponentSelection::None);
|
||||
let config = config_for_selections(
|
||||
&selections,
|
||||
&empty_manifest(),
|
||||
None,
|
||||
None::<&reth_chainspec::ChainSpec>,
|
||||
);
|
||||
let desc = describe_prune_config(&config);
|
||||
assert!(desc.contains(&"sender_recovery=\"full\"".to_string()));
|
||||
// Bodies follows tx selection
|
||||
assert!(desc.contains(&"bodies_history={ distance = 10064 }".to_string()));
|
||||
assert!(desc.contains(&"receipts={ distance = 64 }".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reset_index_stage_checkpoints_clears_only_rocksdb_index_stages() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db = reth_db::init_db(dir.path(), reth_db::mdbx::DatabaseArguments::default()).unwrap();
|
||||
|
||||
// Simulate a fully synced node: set stage checkpoints at tip
|
||||
let tip_checkpoint = StageCheckpoint::new(24_500_000);
|
||||
{
|
||||
let tx = db.tx_mut().unwrap();
|
||||
for stage_id in INDEX_STAGE_IDS {
|
||||
tx.put::<tables::StageCheckpoints>(stage_id.to_string(), tip_checkpoint).unwrap();
|
||||
}
|
||||
for segment in INDEX_PRUNE_SEGMENTS {
|
||||
tx.put::<tables::PruneCheckpoints>(
|
||||
segment,
|
||||
PruneCheckpoint {
|
||||
block_number: Some(24_500_000),
|
||||
tx_number: None,
|
||||
prune_mode: PruneMode::Full,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
// Sender recovery checkpoints should be preserved by reset.
|
||||
tx.put::<tables::StageCheckpoints>("SenderRecovery".to_string(), tip_checkpoint)
|
||||
.unwrap();
|
||||
tx.put::<tables::PruneCheckpoints>(
|
||||
PruneSegment::SenderRecovery,
|
||||
PruneCheckpoint {
|
||||
block_number: Some(24_500_000),
|
||||
tx_number: None,
|
||||
prune_mode: PruneMode::Full,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
|
||||
// Reset
|
||||
{
|
||||
let tx = db.tx_mut().unwrap();
|
||||
reset_index_stage_checkpoints_tx(&tx).unwrap();
|
||||
tx.commit().unwrap();
|
||||
}
|
||||
|
||||
// Verify stage checkpoints are at block 0
|
||||
let tx = db.tx().unwrap();
|
||||
for stage_id in INDEX_STAGE_IDS {
|
||||
let checkpoint = tx
|
||||
.get::<tables::StageCheckpoints>(stage_id.to_string())
|
||||
.unwrap()
|
||||
.expect("checkpoint should exist");
|
||||
assert_eq!(checkpoint.block_number, 0, "stage {stage_id} should be reset to block 0");
|
||||
}
|
||||
|
||||
// Verify prune checkpoints are deleted
|
||||
for segment in INDEX_PRUNE_SEGMENTS {
|
||||
assert!(
|
||||
tx.get::<tables::PruneCheckpoints>(segment).unwrap().is_none(),
|
||||
"prune checkpoint for {segment} should be deleted"
|
||||
);
|
||||
}
|
||||
|
||||
// Verify sender checkpoints are left untouched.
|
||||
let sender_stage_checkpoint = tx
|
||||
.get::<tables::StageCheckpoints>("SenderRecovery".to_string())
|
||||
.unwrap()
|
||||
.expect("sender checkpoint should exist");
|
||||
assert_eq!(sender_stage_checkpoint.block_number, tip_checkpoint.block_number);
|
||||
|
||||
let sender_prune_checkpoint = tx
|
||||
.get::<tables::PruneCheckpoints>(PruneSegment::SenderRecovery)
|
||||
.unwrap()
|
||||
.expect("sender prune checkpoint should exist");
|
||||
assert_eq!(sender_prune_checkpoint.block_number, Some(tip_checkpoint.block_number));
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,232 +0,0 @@
|
||||
use crate::download::manifest::generate_manifest;
|
||||
use clap::Parser;
|
||||
use eyre::{Result, WrapErr};
|
||||
use reth_db::{mdbx::DatabaseArguments, open_db_read_only, tables, Database};
|
||||
use reth_db_api::transaction::DbTx;
|
||||
use reth_stages_types::StageId;
|
||||
use reth_static_file_types::DEFAULT_BLOCKS_PER_STATIC_FILE;
|
||||
use std::{path::PathBuf, time::Instant};
|
||||
use tracing::{info, warn};
|
||||
|
||||
/// Generate modular chunk archives and a snapshot manifest from a source datadir.
|
||||
///
|
||||
/// Archive naming convention:
|
||||
/// - Chunked: `{component}-{start}-{end}.tar.zst` (e.g. `transactions-0-499999.tar.zst`)
|
||||
#[derive(Debug, Parser)]
|
||||
pub struct SnapshotManifestCommand {
|
||||
/// Source datadir containing static files.
|
||||
#[arg(long, short = 'd')]
|
||||
source_datadir: PathBuf,
|
||||
|
||||
/// Optional base URL where archives will be hosted.
|
||||
#[arg(long)]
|
||||
base_url: Option<String>,
|
||||
|
||||
/// Output directory where chunk archives and manifest.json are written.
|
||||
#[arg(long, short = 'o')]
|
||||
output_dir: PathBuf,
|
||||
|
||||
/// Block number this snapshot was taken at.
|
||||
///
|
||||
/// If omitted, this is inferred from the source datadir's `Finish` stage checkpoint.
|
||||
#[arg(long)]
|
||||
block: Option<u64>,
|
||||
|
||||
/// Chain ID.
|
||||
#[arg(long, default_value = "1")]
|
||||
chain_id: u64,
|
||||
|
||||
/// Blocks per archive file for chunked components.
|
||||
///
|
||||
/// If omitted, this is inferred from header static file ranges in the source datadir.
|
||||
#[arg(long)]
|
||||
blocks_per_file: Option<u64>,
|
||||
}
|
||||
|
||||
impl SnapshotManifestCommand {
|
||||
pub fn execute(self) -> Result<()> {
|
||||
let block = match self.block {
|
||||
Some(block) => block,
|
||||
None => infer_snapshot_block(&self.source_datadir)?,
|
||||
};
|
||||
let blocks_per_file = match self.blocks_per_file {
|
||||
Some(blocks_per_file) => blocks_per_file,
|
||||
None => infer_blocks_per_file(&self.source_datadir)?,
|
||||
};
|
||||
|
||||
info!(target: "reth::cli",
|
||||
dir = ?self.source_datadir,
|
||||
output = ?self.output_dir,
|
||||
block,
|
||||
blocks_per_file,
|
||||
"Packaging modular snapshot archives"
|
||||
);
|
||||
let start = Instant::now();
|
||||
let manifest = generate_manifest(
|
||||
&self.source_datadir,
|
||||
&self.output_dir,
|
||||
self.base_url.as_deref(),
|
||||
block,
|
||||
self.chain_id,
|
||||
blocks_per_file,
|
||||
)?;
|
||||
|
||||
let num_components = manifest.components.len();
|
||||
let json = serde_json::to_string_pretty(&manifest)?;
|
||||
let output = self.output_dir.join("manifest.json");
|
||||
reth_fs_util::write(&output, &json)?;
|
||||
info!(target: "reth::cli",
|
||||
path = ?output,
|
||||
components = num_components,
|
||||
block = manifest.block,
|
||||
elapsed = ?start.elapsed(),
|
||||
"Manifest written"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn infer_snapshot_block(source_datadir: &std::path::Path) -> Result<u64> {
|
||||
if let Ok(block) = infer_snapshot_block_from_db(source_datadir) {
|
||||
return Ok(block);
|
||||
}
|
||||
|
||||
let block = infer_snapshot_block_from_headers(source_datadir)?;
|
||||
warn!(
|
||||
target: "reth::cli",
|
||||
block,
|
||||
"Could not read Finish stage checkpoint from source DB, using header static-file tip"
|
||||
);
|
||||
Ok(block)
|
||||
}
|
||||
|
||||
fn infer_snapshot_block_from_db(source_datadir: &std::path::Path) -> Result<u64> {
|
||||
let candidates = [source_datadir.join("db"), source_datadir.to_path_buf()];
|
||||
|
||||
for db_path in candidates {
|
||||
if !db_path.exists() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let db = match open_db_read_only(&db_path, DatabaseArguments::default()) {
|
||||
Ok(db) => db,
|
||||
Err(_) => continue,
|
||||
};
|
||||
|
||||
let tx = db.tx()?;
|
||||
if let Some(checkpoint) = tx.get::<tables::StageCheckpoints>(StageId::Finish.to_string())? {
|
||||
return Ok(checkpoint.block_number);
|
||||
}
|
||||
}
|
||||
|
||||
eyre::bail!(
|
||||
"Could not infer --block from source DB (Finish checkpoint missing); pass --block manually"
|
||||
)
|
||||
}
|
||||
|
||||
fn infer_snapshot_block_from_headers(source_datadir: &std::path::Path) -> Result<u64> {
|
||||
let max_end = header_ranges(source_datadir)?
|
||||
.into_iter()
|
||||
.map(|(_, end)| end)
|
||||
.max()
|
||||
.ok_or_else(|| eyre::eyre!("No header static files found to infer --block"))?;
|
||||
Ok(max_end)
|
||||
}
|
||||
|
||||
fn infer_blocks_per_file(source_datadir: &std::path::Path) -> Result<u64> {
|
||||
let mut inferred = None;
|
||||
for (start, end) in header_ranges(source_datadir)? {
|
||||
let span = end.saturating_sub(start).saturating_add(1);
|
||||
if span == 0 {
|
||||
continue;
|
||||
}
|
||||
|
||||
if let Some(existing) = inferred {
|
||||
if existing != span {
|
||||
eyre::bail!(
|
||||
"Inconsistent header static file ranges; pass --blocks-per-file manually"
|
||||
);
|
||||
}
|
||||
} else {
|
||||
inferred = Some(span);
|
||||
}
|
||||
}
|
||||
|
||||
inferred.ok_or_else(|| {
|
||||
eyre::eyre!(
|
||||
"Could not infer --blocks-per-file from header static files; pass it manually (default is {DEFAULT_BLOCKS_PER_STATIC_FILE})"
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn header_ranges(source_datadir: &std::path::Path) -> Result<Vec<(u64, u64)>> {
|
||||
let static_files_dir = source_datadir.join("static_files");
|
||||
let static_files_dir =
|
||||
if static_files_dir.exists() { static_files_dir } else { source_datadir.to_path_buf() };
|
||||
|
||||
let entries = std::fs::read_dir(&static_files_dir).wrap_err_with(|| {
|
||||
format!("Failed to read static files directory: {}", static_files_dir.display())
|
||||
})?;
|
||||
|
||||
let mut ranges = Vec::new();
|
||||
for entry in entries {
|
||||
let entry = entry?;
|
||||
let file_name = entry.file_name();
|
||||
let file_name = file_name.to_string_lossy();
|
||||
if let Some(range) = parse_headers_range(&file_name) {
|
||||
ranges.push(range);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(ranges)
|
||||
}
|
||||
|
||||
fn parse_headers_range(file_name: &str) -> Option<(u64, u64)> {
|
||||
let remainder = file_name.strip_prefix("static_file_headers_")?;
|
||||
let (start, end_with_suffix) = remainder.split_once('_')?;
|
||||
|
||||
let start = start.parse::<u64>().ok()?;
|
||||
let end_digits: String = end_with_suffix.chars().take_while(|ch| ch.is_ascii_digit()).collect();
|
||||
let end = end_digits.parse::<u64>().ok()?;
|
||||
|
||||
Some((start, end))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tempfile::tempdir;
|
||||
|
||||
#[test]
|
||||
fn parse_headers_range_works_with_suffixes() {
|
||||
assert_eq!(parse_headers_range("static_file_headers_0_499999"), Some((0, 499_999)));
|
||||
assert_eq!(
|
||||
parse_headers_range("static_file_headers_500000_999999.jar"),
|
||||
Some((500_000, 999_999))
|
||||
);
|
||||
assert_eq!(parse_headers_range("static_file_transactions_0_499999"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn infer_blocks_per_file_from_header_ranges() {
|
||||
let dir = tempdir().unwrap();
|
||||
let sf = dir.path().join("static_files");
|
||||
std::fs::create_dir_all(&sf).unwrap();
|
||||
std::fs::write(sf.join("static_file_headers_0_499999"), []).unwrap();
|
||||
std::fs::write(sf.join("static_file_headers_500000_999999.jar"), []).unwrap();
|
||||
|
||||
assert_eq!(infer_blocks_per_file(dir.path()).unwrap(), 500_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn infer_snapshot_block_from_headers_uses_max_end() {
|
||||
let dir = tempdir().unwrap();
|
||||
let sf = dir.path().join("static_files");
|
||||
std::fs::create_dir_all(&sf).unwrap();
|
||||
std::fs::write(sf.join("static_file_headers_0_499999"), []).unwrap();
|
||||
std::fs::write(sf.join("static_file_headers_500000_999999"), []).unwrap();
|
||||
|
||||
assert_eq!(infer_snapshot_block_from_headers(dir.path()).unwrap(), 999_999);
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,437 +0,0 @@
|
||||
use crate::download::{
|
||||
manifest::{ComponentSelection, SnapshotComponentType, SnapshotManifest},
|
||||
DownloadProgress, SelectionPreset,
|
||||
};
|
||||
use crossterm::{
|
||||
event::{self, Event, KeyCode},
|
||||
execute,
|
||||
terminal::{disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen},
|
||||
};
|
||||
use ratatui::{
|
||||
backend::CrosstermBackend,
|
||||
layout::{Constraint, Direction, Layout},
|
||||
style::{Color, Modifier, Style},
|
||||
text::{Line, Span},
|
||||
widgets::{Block, Borders, List, ListItem, ListState, Paragraph},
|
||||
Frame, Terminal,
|
||||
};
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
io,
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
|
||||
/// Result of the interactive component selector.
|
||||
pub struct SelectorOutput {
|
||||
/// User-confirmed selections with per-component ranges.
|
||||
pub selections: BTreeMap<SnapshotComponentType, ComponentSelection>,
|
||||
/// Last preset action used in the TUI, if any.
|
||||
pub preset: Option<SelectionPreset>,
|
||||
}
|
||||
|
||||
/// All distance presets. Groups filter this to only valid options.
|
||||
const DISTANCE_PRESETS: [ComponentSelection; 6] = [
|
||||
ComponentSelection::None,
|
||||
ComponentSelection::Distance(64),
|
||||
ComponentSelection::Distance(10_064),
|
||||
ComponentSelection::Distance(100_000),
|
||||
ComponentSelection::Distance(1_000_000),
|
||||
ComponentSelection::All,
|
||||
];
|
||||
|
||||
/// Presets for components that require at least 64 blocks (receipts).
|
||||
const RECEIPTS_PRESETS: [ComponentSelection; 5] = [
|
||||
ComponentSelection::Distance(64),
|
||||
ComponentSelection::Distance(10_064),
|
||||
ComponentSelection::Distance(100_000),
|
||||
ComponentSelection::Distance(1_000_000),
|
||||
ComponentSelection::All,
|
||||
];
|
||||
|
||||
/// Presets for components that require at least 10064 blocks (account/storage history).
|
||||
const HISTORY_PRESETS: [ComponentSelection; 4] = [
|
||||
ComponentSelection::Distance(10_064),
|
||||
ComponentSelection::Distance(100_000),
|
||||
ComponentSelection::Distance(1_000_000),
|
||||
ComponentSelection::All,
|
||||
];
|
||||
|
||||
/// A display group bundles one or more component types into a single TUI row.
|
||||
struct DisplayGroup {
|
||||
/// Display name shown in the TUI.
|
||||
name: &'static str,
|
||||
/// Underlying component types this group controls.
|
||||
types: Vec<SnapshotComponentType>,
|
||||
/// Whether this group is required and locked to All.
|
||||
required: bool,
|
||||
/// Valid presets for this group. Components with minimum distance requirements
|
||||
/// exclude presets that would produce invalid prune configs.
|
||||
presets: &'static [ComponentSelection],
|
||||
}
|
||||
|
||||
/// Build the display groups from available components in the manifest.
|
||||
fn build_groups(manifest: &SnapshotManifest) -> Vec<DisplayGroup> {
|
||||
let has = |ty: SnapshotComponentType| manifest.component(ty).is_some();
|
||||
|
||||
let mut groups = Vec::new();
|
||||
|
||||
if has(SnapshotComponentType::State) {
|
||||
groups.push(DisplayGroup {
|
||||
name: "State (mdbx)",
|
||||
types: vec![SnapshotComponentType::State],
|
||||
required: true,
|
||||
presets: &DISTANCE_PRESETS,
|
||||
});
|
||||
}
|
||||
|
||||
if has(SnapshotComponentType::Headers) {
|
||||
groups.push(DisplayGroup {
|
||||
name: "Headers",
|
||||
types: vec![SnapshotComponentType::Headers],
|
||||
required: true,
|
||||
presets: &DISTANCE_PRESETS,
|
||||
});
|
||||
}
|
||||
|
||||
if has(SnapshotComponentType::Transactions) {
|
||||
groups.push(DisplayGroup {
|
||||
name: "Transactions",
|
||||
types: vec![SnapshotComponentType::Transactions],
|
||||
required: false,
|
||||
presets: &HISTORY_PRESETS,
|
||||
});
|
||||
}
|
||||
|
||||
if has(SnapshotComponentType::Receipts) {
|
||||
groups.push(DisplayGroup {
|
||||
name: "Receipts",
|
||||
types: vec![SnapshotComponentType::Receipts],
|
||||
required: false,
|
||||
presets: &RECEIPTS_PRESETS,
|
||||
});
|
||||
}
|
||||
|
||||
// Bundle account + storage changesets as "State History"
|
||||
let has_acc = has(SnapshotComponentType::AccountChangesets);
|
||||
let has_stor = has(SnapshotComponentType::StorageChangesets);
|
||||
if has_acc || has_stor {
|
||||
let mut types = Vec::new();
|
||||
if has_acc {
|
||||
types.push(SnapshotComponentType::AccountChangesets);
|
||||
}
|
||||
if has_stor {
|
||||
types.push(SnapshotComponentType::StorageChangesets);
|
||||
}
|
||||
groups.push(DisplayGroup {
|
||||
name: "State History",
|
||||
types,
|
||||
required: false,
|
||||
presets: &HISTORY_PRESETS,
|
||||
});
|
||||
}
|
||||
|
||||
groups
|
||||
}
|
||||
|
||||
struct SelectorApp {
|
||||
manifest: SnapshotManifest,
|
||||
full_preset: BTreeMap<SnapshotComponentType, ComponentSelection>,
|
||||
/// Display groups shown in the TUI.
|
||||
groups: Vec<DisplayGroup>,
|
||||
/// Current selection for each group.
|
||||
selections: Vec<ComponentSelection>,
|
||||
/// Last preset action invoked by user.
|
||||
preset: Option<SelectionPreset>,
|
||||
/// Current cursor position.
|
||||
cursor: usize,
|
||||
/// List state for ratatui.
|
||||
list_state: ListState,
|
||||
}
|
||||
|
||||
impl SelectorApp {
|
||||
fn new(
|
||||
manifest: SnapshotManifest,
|
||||
full_preset: BTreeMap<SnapshotComponentType, ComponentSelection>,
|
||||
) -> Self {
|
||||
let groups = build_groups(&manifest);
|
||||
|
||||
// Default to the minimal preset (matches --minimal prune config)
|
||||
let selections = groups.iter().map(|g| g.types[0].minimal_selection()).collect();
|
||||
|
||||
let mut list_state = ListState::default();
|
||||
list_state.select(Some(0));
|
||||
|
||||
Self {
|
||||
manifest,
|
||||
full_preset,
|
||||
groups,
|
||||
selections,
|
||||
preset: Some(SelectionPreset::Minimal),
|
||||
cursor: 0,
|
||||
list_state,
|
||||
}
|
||||
}
|
||||
|
||||
fn cycle_right(&mut self) {
|
||||
if let Some(group) = self.groups.get(self.cursor) {
|
||||
if group.required {
|
||||
return;
|
||||
}
|
||||
let presets = group.presets;
|
||||
let current = self.selections[self.cursor];
|
||||
let idx = presets.iter().position(|p| *p == current).unwrap_or(0);
|
||||
self.selections[self.cursor] = presets[(idx + 1) % presets.len()];
|
||||
self.preset = None;
|
||||
}
|
||||
}
|
||||
|
||||
fn cycle_left(&mut self) {
|
||||
if let Some(group) = self.groups.get(self.cursor) {
|
||||
if group.required {
|
||||
return;
|
||||
}
|
||||
let presets = group.presets;
|
||||
let current = self.selections[self.cursor];
|
||||
let idx = presets.iter().position(|p| *p == current).unwrap_or(0);
|
||||
self.selections[self.cursor] = presets[(idx + presets.len() - 1) % presets.len()];
|
||||
self.preset = None;
|
||||
}
|
||||
}
|
||||
|
||||
fn select_all(&mut self) {
|
||||
for sel in &mut self.selections {
|
||||
*sel = ComponentSelection::All;
|
||||
}
|
||||
self.preset = Some(SelectionPreset::Archive);
|
||||
}
|
||||
|
||||
fn select_minimal(&mut self) {
|
||||
for (i, group) in self.groups.iter().enumerate() {
|
||||
self.selections[i] = group.types[0].minimal_selection();
|
||||
}
|
||||
self.preset = Some(SelectionPreset::Minimal);
|
||||
}
|
||||
|
||||
fn select_full(&mut self) {
|
||||
for (i, group) in self.groups.iter().enumerate() {
|
||||
let mut selection = group.types[0].minimal_selection();
|
||||
for ty in &group.types {
|
||||
if let Some(sel) = self.full_preset.get(ty).copied() {
|
||||
selection = sel;
|
||||
break;
|
||||
}
|
||||
}
|
||||
self.selections[i] = selection;
|
||||
}
|
||||
self.preset = Some(SelectionPreset::Full);
|
||||
}
|
||||
|
||||
fn move_up(&mut self) {
|
||||
if self.cursor > 0 {
|
||||
self.cursor -= 1;
|
||||
} else {
|
||||
self.cursor = self.groups.len().saturating_sub(1);
|
||||
}
|
||||
self.list_state.select(Some(self.cursor));
|
||||
}
|
||||
|
||||
fn move_down(&mut self) {
|
||||
if self.cursor < self.groups.len() - 1 {
|
||||
self.cursor += 1;
|
||||
} else {
|
||||
self.cursor = 0;
|
||||
}
|
||||
self.list_state.select(Some(self.cursor));
|
||||
}
|
||||
|
||||
/// Build the flat component→selection map from grouped selections.
|
||||
fn selection_map(&self) -> BTreeMap<SnapshotComponentType, ComponentSelection> {
|
||||
let mut map = BTreeMap::new();
|
||||
for (group, sel) in self.groups.iter().zip(&self.selections) {
|
||||
for ty in &group.types {
|
||||
map.insert(*ty, *sel);
|
||||
}
|
||||
}
|
||||
map
|
||||
}
|
||||
|
||||
/// Size for a single group, summing all component types in the group.
|
||||
fn group_size(&self, group_idx: usize) -> u64 {
|
||||
let sel = self.selections[group_idx];
|
||||
let distance = match sel {
|
||||
ComponentSelection::None => return 0,
|
||||
ComponentSelection::All => None,
|
||||
ComponentSelection::Distance(d) => Some(d),
|
||||
};
|
||||
self.groups[group_idx]
|
||||
.types
|
||||
.iter()
|
||||
.map(|ty| self.manifest.size_for_distance(*ty, distance))
|
||||
.sum()
|
||||
}
|
||||
|
||||
fn total_selected_size(&self) -> u64 {
|
||||
(0..self.groups.len()).map(|i| self.group_size(i)).sum()
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs the interactive component selector TUI.
|
||||
pub fn run_selector(
|
||||
manifest: SnapshotManifest,
|
||||
full_preset: &BTreeMap<SnapshotComponentType, ComponentSelection>,
|
||||
) -> eyre::Result<SelectorOutput> {
|
||||
enable_raw_mode()?;
|
||||
let mut stdout = io::stdout();
|
||||
execute!(stdout, EnterAlternateScreen)?;
|
||||
let backend = CrosstermBackend::new(stdout);
|
||||
let mut terminal = Terminal::new(backend)?;
|
||||
|
||||
let mut app = SelectorApp::new(manifest, full_preset.clone());
|
||||
let result = event_loop(&mut terminal, &mut app);
|
||||
|
||||
disable_raw_mode()?;
|
||||
execute!(terminal.backend_mut(), LeaveAlternateScreen)?;
|
||||
terminal.show_cursor()?;
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
fn event_loop(
|
||||
terminal: &mut Terminal<CrosstermBackend<io::Stdout>>,
|
||||
app: &mut SelectorApp,
|
||||
) -> eyre::Result<SelectorOutput> {
|
||||
let tick_rate = Duration::from_millis(100);
|
||||
let mut last_tick = Instant::now();
|
||||
|
||||
loop {
|
||||
terminal.draw(|f| render(f, app))?;
|
||||
|
||||
let timeout =
|
||||
tick_rate.checked_sub(last_tick.elapsed()).unwrap_or_else(|| Duration::from_secs(0));
|
||||
|
||||
if crossterm::event::poll(timeout)? &&
|
||||
let Event::Key(key) = event::read()? &&
|
||||
key.kind == event::KeyEventKind::Press
|
||||
{
|
||||
match key.code {
|
||||
KeyCode::Char('q') | KeyCode::Esc => {
|
||||
eyre::bail!("Download cancelled by user");
|
||||
}
|
||||
KeyCode::Enter => {
|
||||
return Ok(SelectorOutput {
|
||||
selections: app.selection_map(),
|
||||
preset: app.preset,
|
||||
});
|
||||
}
|
||||
KeyCode::Right | KeyCode::Char('l') | KeyCode::Char(' ') => app.cycle_right(),
|
||||
KeyCode::Left | KeyCode::Char('h') => app.cycle_left(),
|
||||
KeyCode::Char('a') => app.select_all(),
|
||||
KeyCode::Char('f') => app.select_full(),
|
||||
KeyCode::Char('m') => app.select_minimal(),
|
||||
KeyCode::Up | KeyCode::Char('k') => app.move_up(),
|
||||
KeyCode::Down | KeyCode::Char('j') => app.move_down(),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
if last_tick.elapsed() >= tick_rate {
|
||||
last_tick = Instant::now();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn format_selection(sel: &ComponentSelection) -> String {
|
||||
match sel {
|
||||
ComponentSelection::All => "All".to_string(),
|
||||
ComponentSelection::Distance(d) => format!("Last {d} blocks"),
|
||||
ComponentSelection::None => "None".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn render(f: &mut Frame<'_>, app: &mut SelectorApp) {
|
||||
let chunks = Layout::default()
|
||||
.direction(Direction::Vertical)
|
||||
.constraints([
|
||||
Constraint::Length(3), // Header
|
||||
Constraint::Min(8), // Component list
|
||||
Constraint::Length(3), // Footer
|
||||
])
|
||||
.split(f.area());
|
||||
|
||||
// Header
|
||||
let block_info = if app.manifest.block > 0 {
|
||||
format!(" (block {})", app.manifest.block)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
let header = Paragraph::new(format!(" Select snapshot components to download{}", block_info))
|
||||
.style(Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD))
|
||||
.block(Block::default().borders(Borders::ALL).title("reth download"));
|
||||
f.render_widget(header, chunks[0]);
|
||||
|
||||
// Component list
|
||||
let items: Vec<ListItem<'_>> = app
|
||||
.groups
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, group)| {
|
||||
let sel = &app.selections[i];
|
||||
let sel_str = format_selection(sel);
|
||||
|
||||
let size = app.group_size(i);
|
||||
let size_str =
|
||||
if size > 0 { DownloadProgress::format_size(size) } else { String::new() };
|
||||
|
||||
let required = if group.required { " (required)" } else { "" };
|
||||
|
||||
let at_max = *sel == *group.presets.last().unwrap_or(&ComponentSelection::All);
|
||||
let at_min = *sel == group.presets[0];
|
||||
let arrows = if group.required {
|
||||
" "
|
||||
} else if at_max {
|
||||
"◂ "
|
||||
} else if at_min {
|
||||
" ▸"
|
||||
} else {
|
||||
"◂ ▸"
|
||||
};
|
||||
|
||||
let style = if group.required {
|
||||
Style::default().fg(Color::DarkGray)
|
||||
} else if matches!(sel, ComponentSelection::None) {
|
||||
Style::default().fg(Color::White)
|
||||
} else {
|
||||
Style::default().fg(Color::Green)
|
||||
};
|
||||
|
||||
ListItem::new(Line::from(vec![
|
||||
Span::styled(format!(" {:<22}", group.name), style),
|
||||
Span::styled(
|
||||
format!("{arrows} {:<12}", sel_str),
|
||||
style.add_modifier(Modifier::BOLD),
|
||||
),
|
||||
Span::styled(format!("{:>10}", size_str), style.add_modifier(Modifier::DIM)),
|
||||
Span::styled(required.to_string(), Style::default().fg(Color::DarkGray)),
|
||||
]))
|
||||
})
|
||||
.collect();
|
||||
|
||||
let total_str = DownloadProgress::format_size(app.total_selected_size());
|
||||
let list = List::new(items)
|
||||
.block(
|
||||
Block::default()
|
||||
.borders(Borders::ALL)
|
||||
.title(format!("Components — Total: {total_str}")),
|
||||
)
|
||||
.highlight_style(Style::default().add_modifier(Modifier::BOLD).bg(Color::DarkGray))
|
||||
.highlight_symbol("▸ ");
|
||||
f.render_stateful_widget(list, chunks[1], &mut app.list_state);
|
||||
|
||||
// Footer
|
||||
let footer = Paragraph::new(
|
||||
" [←/→] adjust [m] minimal [f] full [a] archive [Enter] confirm [Esc] cancel",
|
||||
)
|
||||
.style(Style::default().fg(Color::Cyan).add_modifier(Modifier::BOLD))
|
||||
.block(Block::default().borders(Borders::ALL));
|
||||
f.render_widget(footer, chunks[2]);
|
||||
}
|
||||
@@ -19,12 +19,11 @@ use reth_node_api::BlockTy;
|
||||
use reth_node_events::node::NodeEvent;
|
||||
use reth_provider::{
|
||||
providers::ProviderNodeTypes, BlockNumReader, HeaderProvider, ProviderError, ProviderFactory,
|
||||
RocksDBProviderFactory, StageCheckpointReader,
|
||||
StageCheckpointReader,
|
||||
};
|
||||
use reth_prune::PruneModes;
|
||||
use reth_stages::{prelude::*, ControlFlow, Pipeline, StageId, StageSet};
|
||||
use reth_static_file::StaticFileProducer;
|
||||
use reth_storage_api::StorageSettingsCache;
|
||||
use std::{path::Path, sync::Arc};
|
||||
use tokio::sync::watch;
|
||||
use tracing::{debug, error, info, warn};
|
||||
@@ -109,11 +108,7 @@ where
|
||||
|
||||
let provider = provider_factory.provider()?;
|
||||
let init_blocks = provider.tx_ref().entries::<tables::HeaderNumbers>()?;
|
||||
let init_txns = if provider_factory.cached_storage_settings().storage_v2 {
|
||||
provider_factory.rocksdb_provider().iter::<tables::TransactionHashNumbers>()?.count()
|
||||
} else {
|
||||
provider.tx_ref().entries::<tables::TransactionHashNumbers>()?
|
||||
};
|
||||
let init_txns = provider.tx_ref().entries::<tables::TransactionHashNumbers>()?;
|
||||
drop(provider);
|
||||
|
||||
let mut total_decoded_blocks = 0;
|
||||
@@ -220,12 +215,8 @@ where
|
||||
|
||||
let provider = provider_factory.provider()?;
|
||||
let total_imported_blocks = provider.tx_ref().entries::<tables::HeaderNumbers>()? - init_blocks;
|
||||
let current_txns = if provider_factory.cached_storage_settings().storage_v2 {
|
||||
provider_factory.rocksdb_provider().iter::<tables::TransactionHashNumbers>()?.count()
|
||||
} else {
|
||||
provider.tx_ref().entries::<tables::TransactionHashNumbers>()?
|
||||
};
|
||||
let total_imported_txns = current_txns - init_txns;
|
||||
let total_imported_txns =
|
||||
provider.tx_ref().entries::<tables::TransactionHashNumbers>()? - init_txns;
|
||||
|
||||
let result = ImportResult {
|
||||
total_decoded_blocks,
|
||||
|
||||
@@ -47,7 +47,7 @@ pub struct InitStateCommand<C: ChainSpecParser> {
|
||||
/// Specifies whether to initialize the state without relying on EVM historical data.
|
||||
///
|
||||
/// When enabled, and before inserting the state, it creates a dummy chain up to the last EVM
|
||||
/// block specified. It then appends the first provided block.
|
||||
/// block specified. It then, appends the first block provided block.
|
||||
///
|
||||
/// - **Note**: **Do not** import receipts and blocks beforehand, or this will fail or be
|
||||
/// ignored.
|
||||
|
||||
@@ -125,12 +125,12 @@ pub struct NodeCommand<C: ChainSpecParser, Ext: clap::Args + fmt::Debug = NoArgs
|
||||
}
|
||||
|
||||
impl<C: ChainSpecParser> NodeCommand<C> {
|
||||
/// Parses only the default CLI arguments
|
||||
/// Parsers only the default CLI arguments
|
||||
pub fn parse_args() -> Self {
|
||||
Self::parse()
|
||||
}
|
||||
|
||||
/// Parses only the default [`NodeCommand`] arguments from the given iterator
|
||||
/// Parsers only the default [`NodeCommand`] arguments from the given iterator
|
||||
pub fn try_parse_args_from<I, T>(itr: I) -> Result<Self, clap::error::Error>
|
||||
where
|
||||
I: IntoIterator<Item = T>,
|
||||
|
||||
@@ -193,15 +193,11 @@ impl<C: ChainSpecParser> DownloadArgs<C> {
|
||||
let default_secret_key_path = data_dir.p2p_secret();
|
||||
let p2p_secret_key = self.network.secret_key(default_secret_key_path)?;
|
||||
let rlpx_socket = (self.network.addr, self.network.port).into();
|
||||
let boot_nodes = self
|
||||
.network
|
||||
.resolved_bootnodes()
|
||||
.unwrap_or_else(|| self.chain.bootnodes().unwrap_or_default());
|
||||
let boot_nodes = self.chain.bootnodes().unwrap_or_default();
|
||||
|
||||
let net =
|
||||
NetworkConfigBuilder::<N::NetworkPrimitives>::new(p2p_secret_key, Runtime::test())
|
||||
.peer_config(config.peers_config_with_basic_nodes_from_file(None))
|
||||
.sessions_config(config.sessions)
|
||||
.external_ip_resolver(self.network.nat.clone())
|
||||
.network_id(self.network.network_id)
|
||||
.boot_nodes(boot_nodes.clone())
|
||||
|
||||
@@ -12,6 +12,7 @@ use reth_node_metrics::{
|
||||
server::{MetricServer, MetricServerConfig},
|
||||
version::VersionInfo,
|
||||
};
|
||||
#[cfg(all(unix, feature = "rocksdb"))]
|
||||
use reth_provider::RocksDBProviderFactory;
|
||||
use reth_prune::PrunerBuilder;
|
||||
use reth_static_file::StaticFileProducer;
|
||||
@@ -121,6 +122,7 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + EthereumHardforks>> PruneComma
|
||||
}
|
||||
|
||||
// Flush and compact RocksDB to reclaim disk space after pruning
|
||||
#[cfg(all(unix, feature = "rocksdb"))]
|
||||
{
|
||||
info!(target: "reth::cli", "Flushing and compacting RocksDB...");
|
||||
provider_factory.rocksdb_provider().flush_and_compact()?;
|
||||
|
||||
@@ -95,10 +95,6 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + Hardforks + EthereumHardforks>
|
||||
}
|
||||
};
|
||||
|
||||
if min_block > max_block {
|
||||
eyre::bail!("--from ({min_block}) is beyond --to ({max_block}), nothing to re-execute");
|
||||
}
|
||||
|
||||
let num_tasks = self.num_tasks.unwrap_or_else(|| {
|
||||
std::thread::available_parallelism().map(|n| n.get() as u64).unwrap_or(10)
|
||||
});
|
||||
|
||||
@@ -127,14 +127,12 @@ fn unwind_and_copy<N: ProviderNodeTypes>(
|
||||
AccountHashingStage {
|
||||
clean_threshold: u64::MAX,
|
||||
commit_threshold: u64::MAX,
|
||||
commit_entries: u64::MAX,
|
||||
etl_config: EtlConfig::default(),
|
||||
}
|
||||
.execute(&provider, execute_input)?;
|
||||
StorageHashingStage {
|
||||
clean_threshold: u64::MAX,
|
||||
commit_threshold: u64::MAX,
|
||||
commit_entries: u64::MAX,
|
||||
etl_config: EtlConfig::default(),
|
||||
}
|
||||
.execute(&provider, execute_input)?;
|
||||
|
||||
@@ -107,7 +107,7 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + Hardforks + EthereumHardforks>
|
||||
Comp: CliNodeComponents<N>,
|
||||
F: FnOnce(Arc<C::ChainSpec>) -> Comp,
|
||||
{
|
||||
// Quit early if the stage requires a commit and `--commit` is not provided.
|
||||
// Quit early if the stages requires a commit and `--commit` is not provided.
|
||||
if self.requires_commit() && !self.commit {
|
||||
return Err(eyre::eyre!(
|
||||
"The stage {} requires overwriting existing static files and must commit, but `--commit` was not provided. Please pass `--commit` and try again.",
|
||||
@@ -282,22 +282,14 @@ impl<C: ChainSpecParser<ChainSpec: EthChainSpec + Hardforks + EthereumHardforks>
|
||||
),
|
||||
StageEnum::AccountHashing => (
|
||||
Box::new(AccountHashingStage::new(
|
||||
HashingConfig {
|
||||
clean_threshold: 1,
|
||||
commit_threshold: batch_size,
|
||||
commit_entries: u64::MAX,
|
||||
},
|
||||
HashingConfig { clean_threshold: 1, commit_threshold: batch_size },
|
||||
etl_config,
|
||||
)),
|
||||
None,
|
||||
),
|
||||
StageEnum::StorageHashing => (
|
||||
Box::new(StorageHashingStage::new(
|
||||
HashingConfig {
|
||||
clean_threshold: 1,
|
||||
commit_threshold: batch_size,
|
||||
commit_entries: u64::MAX,
|
||||
},
|
||||
HashingConfig { clean_threshold: 1, commit_threshold: batch_size },
|
||||
etl_config,
|
||||
)),
|
||||
None,
|
||||
|
||||
@@ -71,12 +71,7 @@ impl CliRunner {
|
||||
) -> Result<(), E>
|
||||
where
|
||||
F: Future<Output = Result<(), E>>,
|
||||
E: Send
|
||||
+ Sync
|
||||
+ std::fmt::Display
|
||||
+ From<std::io::Error>
|
||||
+ From<reth_tasks::PanickedTaskError>
|
||||
+ 'static,
|
||||
E: Send + Sync + From<std::io::Error> + From<reth_tasks::PanickedTaskError> + 'static,
|
||||
{
|
||||
let (context, task_manager_handle) = cli_context(&self.runtime);
|
||||
|
||||
@@ -86,8 +81,8 @@ impl CliRunner {
|
||||
run_until_ctrl_c(command(context)),
|
||||
));
|
||||
|
||||
if let Err(err) = &command_res {
|
||||
error!(target: "reth::cli", %err, "shutting down due to error");
|
||||
if command_res.is_err() {
|
||||
error!(target: "reth::cli", "shutting down due to error");
|
||||
} else {
|
||||
debug!(target: "reth::cli", "shutting down gracefully");
|
||||
// after the command has finished or exit signal was received we shutdown the
|
||||
@@ -110,12 +105,7 @@ impl CliRunner {
|
||||
) -> Result<(), E>
|
||||
where
|
||||
F: Future<Output = Result<(), E>> + Send + 'static,
|
||||
E: Send
|
||||
+ Sync
|
||||
+ std::fmt::Display
|
||||
+ From<std::io::Error>
|
||||
+ From<reth_tasks::PanickedTaskError>
|
||||
+ 'static,
|
||||
E: Send + Sync + From<std::io::Error> + From<reth_tasks::PanickedTaskError> + 'static,
|
||||
{
|
||||
let (context, task_manager_handle) = cli_context(&self.runtime);
|
||||
|
||||
@@ -132,8 +122,8 @@ impl CliRunner {
|
||||
),
|
||||
));
|
||||
|
||||
if let Err(err) = &command_res {
|
||||
error!(target: "reth::cli", %err, "shutting down due to error");
|
||||
if command_res.is_err() {
|
||||
error!(target: "reth::cli", "shutting down due to error");
|
||||
} else {
|
||||
debug!(target: "reth::cli", "shutting down gracefully");
|
||||
self.runtime.graceful_shutdown_with_timeout(self.config.graceful_shutdown_timeout);
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
//! Configuration files.
|
||||
use reth_network_types::{PeersConfig, SessionsConfig};
|
||||
use reth_prune_types::{PruneModes, MINIMUM_UNWIND_SAFE_DISTANCE};
|
||||
use reth_prune_types::PruneModes;
|
||||
use reth_stages_types::ExecutionStageThresholds;
|
||||
use reth_static_file_types::{StaticFileMap, StaticFileSegment};
|
||||
use std::{
|
||||
@@ -332,15 +332,11 @@ pub struct HashingConfig {
|
||||
pub clean_threshold: u64,
|
||||
/// The maximum number of entities to process before committing progress to the database.
|
||||
pub commit_threshold: u64,
|
||||
/// The maximum number of changeset entries to process before committing progress. The stage
|
||||
/// commits after either `commit_threshold` blocks or `commit_entries` entries, whichever
|
||||
/// comes first. This bounds memory usage when blocks contain many state changes.
|
||||
pub commit_entries: u64,
|
||||
}
|
||||
|
||||
impl Default for HashingConfig {
|
||||
fn default() -> Self {
|
||||
Self { clean_threshold: 500_000, commit_threshold: 100_000, commit_entries: 30_000_000 }
|
||||
Self { clean_threshold: 500_000, commit_threshold: 100_000 }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -540,24 +536,11 @@ pub struct PruneConfig {
|
||||
/// Pruning configuration for every part of the data that can be pruned.
|
||||
#[cfg_attr(feature = "serde", serde(alias = "parts"))]
|
||||
pub segments: PruneModes,
|
||||
/// Minimum distance from the tip required for pruning. Controls the safety margin for
|
||||
/// reorgs and manual unwinds. Defaults to [`MINIMUM_UNWIND_SAFE_DISTANCE`].
|
||||
#[cfg_attr(feature = "serde", serde(default = "default_minimum_pruning_distance"))]
|
||||
pub minimum_pruning_distance: u64,
|
||||
}
|
||||
|
||||
/// Returns the default minimum pruning distance.
|
||||
const fn default_minimum_pruning_distance() -> u64 {
|
||||
MINIMUM_UNWIND_SAFE_DISTANCE
|
||||
}
|
||||
|
||||
impl Default for PruneConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
block_interval: DEFAULT_BLOCK_INTERVAL,
|
||||
segments: PruneModes::default(),
|
||||
minimum_pruning_distance: MINIMUM_UNWIND_SAFE_DISTANCE,
|
||||
}
|
||||
Self { block_interval: DEFAULT_BLOCK_INTERVAL, segments: PruneModes::default() }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -590,7 +573,6 @@ impl PruneConfig {
|
||||
bodies_history,
|
||||
receipts_log_filter,
|
||||
},
|
||||
minimum_pruning_distance,
|
||||
} = other;
|
||||
|
||||
// Merge block_interval, only update if it's the default interval
|
||||
@@ -598,11 +580,6 @@ impl PruneConfig {
|
||||
self.block_interval = block_interval;
|
||||
}
|
||||
|
||||
// Merge minimum_pruning_distance, only update if it's the default
|
||||
if self.minimum_pruning_distance == MINIMUM_UNWIND_SAFE_DISTANCE {
|
||||
self.minimum_pruning_distance = minimum_pruning_distance;
|
||||
}
|
||||
|
||||
// Merge the various segment prune modes
|
||||
self.segments.sender_recovery = self.segments.sender_recovery.or(sender_recovery);
|
||||
self.segments.transaction_lookup = self.segments.transaction_lookup.or(transaction_lookup);
|
||||
@@ -642,9 +619,7 @@ mod tests {
|
||||
use crate::PruneConfig;
|
||||
use alloy_primitives::Address;
|
||||
use reth_network_peers::TrustedPeer;
|
||||
use reth_prune_types::{
|
||||
PruneMode, PruneModes, ReceiptsLogPruneConfig, MINIMUM_UNWIND_SAFE_DISTANCE,
|
||||
};
|
||||
use reth_prune_types::{PruneMode, PruneModes, ReceiptsLogPruneConfig};
|
||||
use std::{collections::BTreeMap, path::Path, str::FromStr, time::Duration};
|
||||
|
||||
fn with_tempdir(filename: &str, proc: fn(&std::path::Path)) {
|
||||
@@ -1114,7 +1089,6 @@ receipts = { distance = 16384 }
|
||||
fn test_prune_config_merge() {
|
||||
let mut config1 = PruneConfig {
|
||||
block_interval: 5,
|
||||
minimum_pruning_distance: MINIMUM_UNWIND_SAFE_DISTANCE,
|
||||
segments: PruneModes {
|
||||
sender_recovery: Some(PruneMode::Full),
|
||||
transaction_lookup: None,
|
||||
@@ -1131,7 +1105,6 @@ receipts = { distance = 16384 }
|
||||
|
||||
let config2 = PruneConfig {
|
||||
block_interval: 10,
|
||||
minimum_pruning_distance: MINIMUM_UNWIND_SAFE_DISTANCE,
|
||||
segments: PruneModes {
|
||||
sender_recovery: Some(PruneMode::Distance(500)),
|
||||
transaction_lookup: Some(PruneMode::Full),
|
||||
|
||||
@@ -75,3 +75,8 @@ path = "tests/e2e-testsuite/main.rs"
|
||||
[[test]]
|
||||
name = "rocksdb"
|
||||
path = "tests/rocksdb/main.rs"
|
||||
required-features = ["rocksdb"]
|
||||
|
||||
[features]
|
||||
rocksdb = ["reth-node-core/rocksdb", "reth-provider/rocksdb", "reth-cli-commands/rocksdb"]
|
||||
edge = ["rocksdb"]
|
||||
|
||||
@@ -96,11 +96,6 @@ where
|
||||
self
|
||||
}
|
||||
|
||||
/// Sets the pruning arguments for the test nodes.
|
||||
pub fn with_pruning(self, pruning: reth_node_core::args::PruningArgs) -> Self {
|
||||
self.with_node_config_modifier(move |config| config.with_pruning(pruning.clone()))
|
||||
}
|
||||
|
||||
/// Enables v2 storage defaults (`--storage.v2`), routing tx hashes, history
|
||||
/// indices, etc. to `RocksDB` and changesets/senders to static files.
|
||||
pub fn with_storage_v2(self) -> Self {
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
//! E2E tests for `RocksDB` provider functionality.
|
||||
|
||||
#![cfg(all(feature = "rocksdb", unix))]
|
||||
|
||||
use alloy_consensus::BlockHeader;
|
||||
use alloy_primitives::B256;
|
||||
use alloy_rpc_types_eth::{Transaction, TransactionReceipt};
|
||||
|
||||
@@ -38,8 +38,8 @@ op-alloy-rpc-types-engine = { workspace = true, optional = true }
|
||||
workspace = true
|
||||
|
||||
[features]
|
||||
# op = [
|
||||
# "dep:op-alloy-rpc-types-engine",
|
||||
# "reth-payload-primitives/op",
|
||||
# "reth-primitives-traits/op",
|
||||
# ]
|
||||
op = [
|
||||
"dep:op-alloy-rpc-types-engine",
|
||||
"reth-payload-primitives/op",
|
||||
"reth-primitives-traits/op",
|
||||
]
|
||||
|
||||
@@ -39,7 +39,6 @@ thiserror.workspace = true
|
||||
|
||||
[features]
|
||||
default = ["std"]
|
||||
trie-debug = []
|
||||
std = [
|
||||
"reth-execution-types/std",
|
||||
"reth-ethereum-primitives/std",
|
||||
|
||||
@@ -26,15 +26,10 @@ pub const DEFAULT_RESERVED_CPU_CORES: usize = 1;
|
||||
/// Depth 4 means we keep roughly 16^4 = 65536 potential branch paths at most.
|
||||
pub const DEFAULT_SPARSE_TRIE_PRUNE_DEPTH: usize = 4;
|
||||
|
||||
/// Default LFU hot-slot capacity for sparse trie pruning.
|
||||
/// Default maximum number of storage tries to keep after pruning.
|
||||
///
|
||||
/// Limits the number of `(address, slot)` pairs retained across prune cycles.
|
||||
pub const DEFAULT_SPARSE_TRIE_MAX_HOT_SLOTS: usize = 1500;
|
||||
|
||||
/// Default LFU hot-account capacity for sparse trie pruning.
|
||||
///
|
||||
/// Limits the number of account addresses retained across prune cycles.
|
||||
pub const DEFAULT_SPARSE_TRIE_MAX_HOT_ACCOUNTS: usize = 1000;
|
||||
/// Storage tries beyond this limit are cleared (but allocations preserved).
|
||||
pub const DEFAULT_SPARSE_TRIE_MAX_STORAGE_TRIES: usize = 100;
|
||||
|
||||
/// Default timeout for the state root task before spawning a sequential fallback.
|
||||
pub const DEFAULT_STATE_ROOT_TASK_TIMEOUT: Duration = Duration::from_secs(1);
|
||||
@@ -136,28 +131,15 @@ pub struct TreeConfig {
|
||||
disable_cache_metrics: bool,
|
||||
/// Depth for sparse trie pruning after state root computation.
|
||||
sparse_trie_prune_depth: usize,
|
||||
/// LFU hot-slot capacity: max `(address, slot)` pairs retained across prune cycles.
|
||||
sparse_trie_max_hot_slots: usize,
|
||||
/// LFU hot-account capacity: max account addresses retained across prune cycles.
|
||||
sparse_trie_max_hot_accounts: usize,
|
||||
/// When set, blocks whose total processing time (execution + state reads + state root +
|
||||
/// DB commit) exceeds this duration trigger a structured `warn!` log with detailed timing,
|
||||
/// state-operation counts, and cache hit-rate metrics. `Duration::ZERO` logs every block.
|
||||
slow_block_threshold: Option<Duration>,
|
||||
/// Maximum number of storage tries to retain after pruning.
|
||||
sparse_trie_max_storage_tries: usize,
|
||||
/// Whether to fully disable sparse trie cache pruning between blocks.
|
||||
disable_sparse_trie_cache_pruning: bool,
|
||||
/// Whether to use the arena-based sparse trie implementation.
|
||||
enable_arena_sparse_trie: bool,
|
||||
/// Timeout for the state root task before spawning a sequential fallback computation.
|
||||
/// If `Some`, after waiting this duration for the state root task, a sequential state root
|
||||
/// computation is spawned in parallel and whichever finishes first is used.
|
||||
/// If `None`, the timeout fallback is disabled.
|
||||
state_root_task_timeout: Option<Duration>,
|
||||
/// Maximum random jitter applied before each proof computation (trie-debug only).
|
||||
/// When set, each proof worker sleeps for a random duration up to this value
|
||||
/// before starting a proof calculation.
|
||||
#[cfg(feature = "trie-debug")]
|
||||
proof_jitter: Option<Duration>,
|
||||
}
|
||||
|
||||
impl Default for TreeConfig {
|
||||
@@ -183,14 +165,9 @@ impl Default for TreeConfig {
|
||||
allow_unwind_canonical_header: false,
|
||||
disable_cache_metrics: false,
|
||||
sparse_trie_prune_depth: DEFAULT_SPARSE_TRIE_PRUNE_DEPTH,
|
||||
sparse_trie_max_hot_slots: DEFAULT_SPARSE_TRIE_MAX_HOT_SLOTS,
|
||||
sparse_trie_max_hot_accounts: DEFAULT_SPARSE_TRIE_MAX_HOT_ACCOUNTS,
|
||||
slow_block_threshold: None,
|
||||
sparse_trie_max_storage_tries: DEFAULT_SPARSE_TRIE_MAX_STORAGE_TRIES,
|
||||
disable_sparse_trie_cache_pruning: false,
|
||||
enable_arena_sparse_trie: false,
|
||||
state_root_task_timeout: Some(DEFAULT_STATE_ROOT_TASK_TIMEOUT),
|
||||
#[cfg(feature = "trie-debug")]
|
||||
proof_jitter: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -219,9 +196,7 @@ impl TreeConfig {
|
||||
allow_unwind_canonical_header: bool,
|
||||
disable_cache_metrics: bool,
|
||||
sparse_trie_prune_depth: usize,
|
||||
sparse_trie_max_hot_slots: usize,
|
||||
sparse_trie_max_hot_accounts: usize,
|
||||
slow_block_threshold: Option<Duration>,
|
||||
sparse_trie_max_storage_tries: usize,
|
||||
state_root_task_timeout: Option<Duration>,
|
||||
) -> Self {
|
||||
Self {
|
||||
@@ -245,14 +220,9 @@ impl TreeConfig {
|
||||
allow_unwind_canonical_header,
|
||||
disable_cache_metrics,
|
||||
sparse_trie_prune_depth,
|
||||
sparse_trie_max_hot_slots,
|
||||
sparse_trie_max_hot_accounts,
|
||||
slow_block_threshold,
|
||||
sparse_trie_max_storage_tries,
|
||||
disable_sparse_trie_cache_pruning: false,
|
||||
enable_arena_sparse_trie: false,
|
||||
state_root_task_timeout,
|
||||
#[cfg(feature = "trie-debug")]
|
||||
proof_jitter: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -501,43 +471,14 @@ impl TreeConfig {
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns the LFU hot-slot capacity for sparse trie pruning.
|
||||
pub const fn sparse_trie_max_hot_slots(&self) -> usize {
|
||||
self.sparse_trie_max_hot_slots
|
||||
/// Returns the maximum number of storage tries to retain after pruning.
|
||||
pub const fn sparse_trie_max_storage_tries(&self) -> usize {
|
||||
self.sparse_trie_max_storage_tries
|
||||
}
|
||||
|
||||
/// Setter for LFU hot-slot capacity.
|
||||
pub const fn with_sparse_trie_max_hot_slots(mut self, max_hot_slots: usize) -> Self {
|
||||
self.sparse_trie_max_hot_slots = max_hot_slots;
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns the LFU hot-account capacity for sparse trie pruning.
|
||||
pub const fn sparse_trie_max_hot_accounts(&self) -> usize {
|
||||
self.sparse_trie_max_hot_accounts
|
||||
}
|
||||
|
||||
/// Setter for LFU hot-account capacity.
|
||||
pub const fn with_sparse_trie_max_hot_accounts(mut self, max_hot_accounts: usize) -> Self {
|
||||
self.sparse_trie_max_hot_accounts = max_hot_accounts;
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns the slow block threshold, if configured.
|
||||
///
|
||||
/// When `Some`, blocks whose total processing time exceeds this duration emit a structured
|
||||
/// warning with timing, state-operation, and cache-hit-rate details. `Duration::ZERO` logs
|
||||
/// every block.
|
||||
pub const fn slow_block_threshold(&self) -> Option<Duration> {
|
||||
self.slow_block_threshold
|
||||
}
|
||||
|
||||
/// Setter for slow block threshold.
|
||||
pub const fn with_slow_block_threshold(
|
||||
mut self,
|
||||
slow_block_threshold: Option<Duration>,
|
||||
) -> Self {
|
||||
self.slow_block_threshold = slow_block_threshold;
|
||||
/// Setter for maximum storage tries to retain.
|
||||
pub const fn with_sparse_trie_max_storage_tries(mut self, max_tries: usize) -> Self {
|
||||
self.sparse_trie_max_storage_tries = max_tries;
|
||||
self
|
||||
}
|
||||
|
||||
@@ -552,17 +493,6 @@ impl TreeConfig {
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns whether the arena-based sparse trie is enabled.
|
||||
pub const fn enable_arena_sparse_trie(&self) -> bool {
|
||||
self.enable_arena_sparse_trie
|
||||
}
|
||||
|
||||
/// Setter for whether to enable the arena-based sparse trie.
|
||||
pub const fn with_enable_arena_sparse_trie(mut self, value: bool) -> Self {
|
||||
self.enable_arena_sparse_trie = value;
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns the state root task timeout.
|
||||
pub const fn state_root_task_timeout(&self) -> Option<Duration> {
|
||||
self.state_root_task_timeout
|
||||
@@ -573,17 +503,4 @@ impl TreeConfig {
|
||||
self.state_root_task_timeout = timeout;
|
||||
self
|
||||
}
|
||||
|
||||
/// Returns the proof jitter duration, if configured (trie-debug only).
|
||||
#[cfg(feature = "trie-debug")]
|
||||
pub const fn proof_jitter(&self) -> Option<Duration> {
|
||||
self.proof_jitter
|
||||
}
|
||||
|
||||
/// Setter for proof jitter (trie-debug only).
|
||||
#[cfg(feature = "trie-debug")]
|
||||
pub const fn with_proof_jitter(mut self, proof_jitter: Option<Duration>) -> Self {
|
||||
self.proof_jitter = proof_jitter;
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@ use core::{
|
||||
fmt::{Display, Formatter, Result},
|
||||
time::Duration,
|
||||
};
|
||||
use reth_chain_state::{ExecutedBlock, ExecutionTimingStats};
|
||||
use reth_chain_state::ExecutedBlock;
|
||||
use reth_ethereum_primitives::EthPrimitives;
|
||||
use reth_primitives_traits::{NodePrimitives, SealedBlock, SealedHeader};
|
||||
|
||||
@@ -32,8 +32,6 @@ pub enum ConsensusEngineEvent<N: NodePrimitives = EthPrimitives> {
|
||||
CanonicalChainCommitted(Box<SealedHeader<N::BlockHeader>>, Duration),
|
||||
/// The consensus engine processed an invalid block.
|
||||
InvalidBlock(Box<SealedBlock<N::Block>>),
|
||||
/// A slow block was detected after persistence, with its timing statistics.
|
||||
SlowBlock(SlowBlockInfo),
|
||||
}
|
||||
|
||||
impl<N: NodePrimitives> ConsensusEngineEvent<N> {
|
||||
@@ -75,25 +73,6 @@ where
|
||||
Self::BlockReceived(num_hash) => {
|
||||
write!(f, "BlockReceived({num_hash:?})")
|
||||
}
|
||||
Self::SlowBlock(info) => {
|
||||
write!(
|
||||
f,
|
||||
"SlowBlock(block={}, total={:?})",
|
||||
info.stats.block_number, info.total_duration
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Information about a slow block detected after persistence.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct SlowBlockInfo {
|
||||
/// The timing statistics for the slow block.
|
||||
pub stats: Box<ExecutionTimingStats>,
|
||||
/// The commit duration for the batch containing this block.
|
||||
pub commit_duration: Duration,
|
||||
/// The total duration (execution + `state_root` + commit).
|
||||
/// Note: `state_read` is a subset of execution and is not added separately.
|
||||
pub total_duration: Duration,
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user