Implements a new verification tool to check that backfill successfully
retrieved blobs and data columns from the beacon chain. The tool:
- Auto-detects network configuration by querying beacon node genesis info
- Calculates blob retention window based on MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUEST
- Verifies blobs via /eth/v1/beacon/blob_sidecars/{slot} API
- Verifies data columns (post-Fulu) via /eth/v1/debug/beacon/data_column_sidecars/{slot}
- Provides configurable verbosity with progress updates every 1000 slots
- Generates comprehensive statistics summary
Usage:
prysmctl backfill verify --beacon-node-host=http://localhost:3500
prysmctl backfill verify --beacon-node-host=http://localhost:3500 --verbose
Supports Hoodi testnet and mainnet with automatic config detection.
Fix backfill verify tool
Adds TestWithBlobRetentionEpochPreserved which verifies that the
--blob-retention-epochs flag value is preserved through Start()
initialization. The test detects if the bug at line 264 (using
syncNeeds{} instead of s.syncNeeds) regresses.
<!-- Thanks for sending a PR! Before submitting:
1. If this is your first PR, check out our contribution guide here
https://docs.prylabs.network/docs/contribute/contribution-guidelines
You will then need to sign our Contributor License Agreement (CLA),
which will show up as a comment from a bot in this pull request after
you open it. We cannot review code without a signed CLA.
2. Please file an associated tracking issue if this pull request is
non-trivial and requires context for our team to understand. All
features and most bug fixes should have
an associated issue with a design discussed and decided upon. Small bug
fixes and documentation improvements don't need issues.
3. New features and bug fixes must have tests. Documentation may need to
be updated. If you're unsure what to update, send the PR, and we'll
discuss
in review.
4. Note that PRs updating dependencies and new Go versions are not
accepted.
Please file an issue instead.
5. A changelog entry is required for user facing issues.
-->
**What type of PR is this?**
Bug fix
**What does this PR do? Why is it needed?**
Allows for starting e2e tests from electra or a specific fork of
interest again. doesn't fix missing execution requests tests, nishant
reverted it.
**Which issues(s) does this PR fix?**
Fixes #
**Other notes for review**
**Acknowledgements**
- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description to this PR with sufficient context for
reviewers to understand this PR.
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
The slot parameter in blobCacheEntry.filter was unused and redundant.
All slot/epoch-sensitive checks happen before filter
(commitmentsToCheck), and disk availability is handled via
BlobStorageSummary (epoch-aware).
Changes:
- Drop slot from blobCacheEntry.filter signature.
- Update call sites in availability_blobs.go and blob_cache_test.go.
Mirrors the data_column_cache.filter API (which does not take slot),
reduces API noise, and removes dead code without changing behavior.
<!-- Thanks for sending a PR! Before submitting:
1. If this is your first PR, check out our contribution guide here
https://docs.prylabs.network/docs/contribute/contribution-guidelines
You will then need to sign our Contributor License Agreement (CLA),
which will show up as a comment from a bot in this pull request after
you open it. We cannot review code without a signed CLA.
2. Please file an associated tracking issue if this pull request is
non-trivial and requires context for our team to understand. All
features and most bug fixes should have
an associated issue with a design discussed and decided upon. Small bug
fixes and documentation improvements don't need issues.
3. New features and bug fixes must have tests. Documentation may need to
be updated. If you're unsure what to update, send the PR, and we'll
discuss
in review.
4. Note that PRs updating dependencies and new Go versions are not
accepted.
Please file an issue instead.
5. A changelog entry is required for user facing issues.
-->
**What type of PR is this?**
Feature
**What does this PR do? Why is it needed?**
| Feature | Semi-Supernode | Supernode |
| ----------------------- | ------------------------- |
------------------------ |
| **Custody Groups** | 64 | 128 |
| **Data Columns** | 64 | 128 |
| **Storage** | ~50% | ~100% |
| **Blob Reconstruction** | Yes (via Reed-Solomon) | No reconstruction
needed |
| **Flag** | `--semi-supernode` | `--supernode` |
| **Can serve all blobs** | Yes (with reconstruction) | Yes (directly) |
**note** if your validator total effective balance results in more
custody than the semi-supernode it will override those those
requirements.
cgc=64 from @nalepae
Pro:
- We are useful to the network
- Less disconnection likelihood
- Straight forward to implement
Con:
- We cannot revert to a full node
- We have to serve incoming RPC requests corresponding to 64 columns
Tested the following using this kurtosis setup
```
participants:
# Super-nodes
- el_type: geth
el_image: ethpandaops/geth:master
cl_type: prysm
vc_image: gcr.io/offchainlabs/prysm/validator:latest
cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
count: 2
cl_extra_params:
- --supernode
vc_extra_params:
- --verbosity=debug
# Full-nodes
- el_type: geth
el_image: ethpandaops/geth:master
cl_type: prysm
vc_image: gcr.io/offchainlabs/prysm/validator:latest
cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
count: 2
validator_count: 1
cl_extra_params:
- --semi-supernode
vc_extra_params:
- --verbosity=debug
additional_services:
- dora
- spamoor
spamoor_params:
image: ethpandaops/spamoor:master
max_mem: 4000
spammers:
- scenario: eoatx
config:
throughput: 200
- scenario: blobs
config:
throughput: 20
network_params:
fulu_fork_epoch: 0
withdrawal_type: "0x02"
preset: mainnet
global_log_level: debug
```
```
curl -H "Accept: application/json" http://127.0.0.1:32961/eth/v1/node/identity
{"data":{"peer_id":"16Uiu2HAm7xzhnGwea8gkcxRSC6fzUkvryP6d9HdWNkoeTkj6RSqw","enr":"enr:-Ni4QIH5u2NQz17_pTe9DcCfUyG8TidDJJjIeBpJRRm4ACQzGBpCJdyUP9eGZzwwZ2HS1TnB9ACxFMQ5LP5njnMDLm-GAZqZEXjih2F0dG5ldHOIAAAAAAAwAACDY2djQIRldGgykLZy_whwAAA4__________-CaWSCdjSCaXCErBAAE4NuZmSEAAAAAIRxdWljgjLIiXNlY3AyNTZrMaECulJrXpSOBmCsQWcGYzQsst7r3-Owlc9iZbEcJTDkB6qIc3luY25ldHMFg3RjcIIyyIN1ZHCCLuA","p2p_addresses":["/ip4/172.16.0.19/tcp/13000/p2p/16Uiu2HAm7xzhnGwea8gkcxRSC6fzUkvryP6d9HdWNkoeTkj6RSqw","/ip4/172.16.0.19/udp/13000/quic-v1/p2p/16Uiu2HAm7xzhnGwea8gkcxRSC6fzUkvryP6d9HdWNkoeTkj6RSqw"],"discovery_addresses":["/ip4/172.16.0.19/udp/12000/p2p/16Uiu2HAm7xzhnGwea8gkcxRSC6fzUkvryP6d9HdWNkoeTkj6RSqw"],"metadata":{"seq_number":"3","attnets":"0x0000000000300000","syncnets":"0x05","custody_group_count":"64"}}}
```
```
curl -s http://127.0.0.1:32961/eth/v1/debug/beacon/data_column_sidecars/head | jq '.data | length'
64
```
```
curl -X 'GET' \
'http://127.0.0.1:32961/eth/v1/beacon/blobs/head' \
-H 'accept: application/json'
```
**Which issues(s) does this PR fix?**
Fixes #
**Other notes for review**
**Acknowledgements**
- [x] I have read [CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description to this PR with sufficient context for reviewers to understand this PR.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: james-prysm <jhe@offchainlabs.com>
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
**What type of PR is this?**
Bug fix
**What does this PR do? Why is it needed?**
I am seeing massive metrics cardinality on error cases.
Example:
```
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682952",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682953",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682954",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682955",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682956",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682957",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682958",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682959",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682960",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682961",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682962",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682966",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682967",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682968",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682969",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682970",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682971",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682972",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682973",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682974",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682975",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682976",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682977",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682978",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682980",method="GET"} 2
http_error_count{code="Not Found",endpoint="/eth/v1/beacon/blob_sidecars/1682983",method="GET"} 2
```
Now it looks like this:
```
# TYPE http_error_count counter
http_error_count{code="Not Found",endpoint="beacon.GetBlockV2",method="GET"} 606
http_error_count{code="Not Found",endpoint="blob.Blobs",method="GET"} 4304
```
**Which issues(s) does this PR fix?**
**Other notes for review**
Other uses of http metrics use the endpoint name rather than the request
URL.
**Acknowledgements**
- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description to this PR with sufficient context for
reviewers to understand this PR.
This change ensures FlagOptions in cmd/beacon-chain/execution/options.go
appends only one endpoint option depending on whether a JWT secret is
present. Previously the code always appended WithHttpEndpoint and then
conditionally appended WithHttpEndpointAndJWTSecret which overwrote the
first option, adding unnecessary allocations and cognitive overhead.
Since WithHttpEndpointAndJWTSecret fully configures the endpoint,
including URL and Bearer auth needed by the Engine API, the initial
WithHttpEndpoint is redundant when a JWT is supplied. The refactor
preserves behavior while simplifying option composition and avoiding
redundant state churn.
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
**What type of PR is this?**
Other
**What does this PR do? Why is it needed?**
When unmarshaling a block with fastssz, if the target block's
`ExecutionRequests` field is nil, it will not get populated
```
if b.ExecutionRequests == nil {
b.ExecutionRequests = new(v1.ExecutionRequests)
}
if err = b.ExecutionRequests.UnmarshalSSZ(buf); err != nil {
return err
}
```
This is true for other fields and that's why we initialize them in our
gossip data maps. There is no bug at the moment because even if
execution requests are nil, we initialize them in
`consensus-types/blocks/proto.go`
```
er := pb.ExecutionRequests
if er == nil {
er = &enginev1.ExecutionRequests{}
}
```
However, since we initialize other fields in the data map, it's safer to
do it for execution requests too, to avoid a bug in case the code in
`consensus-types/blocks/proto.go` changes in the future.
**Acknowledgements**
- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description to this PR with sufficient context for
reviewers to understand this PR.
**What type of PR is this?**
Feature
**What does this PR do? Why is it needed?**
This PR adds integrates state-diff into `HasState()`.
One thing to note: we are assuming that, for a given block root, that
has either a state summary or a block in db, and also falls in the state
diff tree, then there must exist a state. This function could return
true, even when there is no actual state saved due to any error.
But this is fine, because we have that assumption throughout the whole
state diff feature.
**Review after #16033 is merged**
**What type of PR is this?**
Other
**What does this PR do? Why is it needed?**
This PR refactors the code to find the corresponding slot of a given
block root using state summary or the block itself, into its own
function `SlotByBlockRoot(ctx, blockroot)`.
Note that there exists a function `slotByBlockRoot(ctx context.Context,
tx *bolt.Tx, blockRoot []byte)` immediately below the new function. Also
note that this function has two drawbacks, which led to creation of the
new function:
- the old function requires a boltdb tx, which is not necessarily
available to the caller.
- the old function does NOT make use of the state summary cache.
edit:
- the old function also uses the state bucket to retrieve the state and
it's slot. this is not something we want in the state diff feature,
since there is no state bucket.
**What type of PR is this?**
Feature
**What does this PR do? Why is it needed?**
This PR integrates the state diff path into the `State()` function from
`db/kv`, which allows reading of states using the state diff db, when
the `EnableStateDiff` flag is enabled.
**Notes for reviewers:**
Files `kv/state_diff_test.go` and `config/features/config.go` only
contain renamings:
- `kv/state_diff_test.go`: rename `setDefaultExponents()` to
`setDefaultStateDiffExponents()` to be less vague.
- `config/features/config.go`: rename `enableStateDiff` to
`EnableStateDiff` to make it public.
<!-- Thanks for sending a PR! Before submitting:
1. If this is your first PR, check out our contribution guide here
https://docs.prylabs.network/docs/contribute/contribution-guidelines
You will then need to sign our Contributor License Agreement (CLA),
which will show up as a comment from a bot in this pull request after
you open it. We cannot review code without a signed CLA.
2. Please file an associated tracking issue if this pull request is
non-trivial and requires context for our team to understand. All
features and most bug fixes should have
an associated issue with a design discussed and decided upon. Small bug
fixes and documentation improvements don't need issues.
3. New features and bug fixes must have tests. Documentation may need to
be updated. If you're unsure what to update, send the PR, and we'll
discuss
in review.
4. Note that PRs updating dependencies and new Go versions are not
accepted.
Please file an issue instead.
5. A changelog entry is required for user facing issues.
-->
**What type of PR is this?**
Bug fix
**What does this PR do? Why is it needed?**
Previously, JWT secrets longer than 256 bits could cause client
compatibility issues. For example, Prysm would accept longer secrets
while Geth strictly requires exactly 32 bytes, causing Geth startup
failures when using the same secret file.
This change enforces the Engine API specification requirement that JWT
secrets must be exactly 256 bits (32 bytes), ensuring consistent
behavior across different client implementations.
**Which issues(s) does this PR fix?**
Fixes #
**Other notes for review**
**Acknowledgements**
- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description to this PR with sufficient context for
reviewers to understand this PR.
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
