Compare commits

...

82 Commits

Author SHA1 Message Date
terence tsao
f26f197c02 Log atts bit indices 2025-03-02 16:27:21 -08:00
Potuz
a9dc6a1dbb missing save 2025-03-02 16:48:38 -03:00
Nishant Das
713fd33eb5 Fix Bugs In Sync From Head (#15006)
* Fix Bugs

* Remove log
2025-03-02 16:48:16 -03:00
nisdas
5f56507bee Merge branch 'sync_from_head' of https://github.com/prysmaticlabs/geth-sharding into hackSync 2025-03-01 07:39:52 +08:00
rkapka
07f0d5ee72 delete on error 2025-02-28 18:16:30 +01:00
rkapka
23bbf2380f Ignore errors from hasSeenBit and don't pack unaggregated attestations 2025-02-28 18:11:47 +01:00
Potuz
9e96de033b Add feature flag to start from any beacon block in db
The new feature flag called --sync-from takes a string that can take
values:

- `head` or
- a 0x-prefixed hex encoded beacon block root.

The beacon block root or the head block root has to be known in db and
has to be a descendant of the current justified checkpoint.
2025-02-28 10:47:43 -03:00
Potuz
b6d1866deb Split out forkchoice setup from service start (#14997) 2025-02-27 20:38:29 +00:00
james-prysm
e56f489d06 add log for committee index on electra attesation (#14993)
* adding log in the case of debugging

* gaz
2025-02-27 18:11:27 +00:00
Nishant Das
bf62afb27c Fix Gossip Validation of Electra Attester Slashings (#14985)
* Fix Attester Slashing Validation In Electra

* Changelog
2025-02-27 08:50:24 +00:00
nisdas
fd41691178 Revert Fallback Sync 2025-02-27 12:11:12 +08:00
nisdas
5916c6e625 Fix Attester Slashing Validation In Electra 2025-02-26 15:09:59 +08:00
terence tsao
0312cb223a Move blockRoot == badHoleskyRoot earlier 2025-02-25 13:53:24 -08:00
terence tsao
68cf7a59f2 Update BUILD.bazel 2025-02-25 12:46:40 -08:00
Kasey Kirkham
537ddb1a24 reject bad root in receive block(batch) 2025-02-25 14:44:18 -06:00
Kasey Kirkham
7afaa6994b hardcode root to remove cache eviction possibility 2025-02-25 14:37:50 -06:00
terence tsao
defb3ab87b Revert terence's downscore change 2025-02-25 11:45:56 -08:00
terence tsao
81dce25c98 Gazelle 2025-02-25 11:41:28 -08:00
Kasey Kirkham
5c409b90bc hack to downscore peers with bad holesky root 2025-02-25 13:40:33 -06:00
terence tsao
aa47661602 Downscore 2025-02-25 11:29:53 -08:00
potuz
7259a2b983 blacklist on batches too 2025-02-25 16:16:53 -03:00
terence tsao
e9f511ac00 Update bootnodes 2025-02-25 08:39:17 -08:00
potuz
34a68715b8 blacklist bad block 2025-02-25 11:40:35 -03:00
Bastin
8369056027 SSZ Support for LC finality and optimistic update APIs (#14836)
* bundle handlers test

* ssz support for optimistic and finality updates APIs

* changelog PR link

* delete helpers

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-02-25 13:09:28 +00:00
nisdas
c33e0575ab do not resync 2025-02-25 17:32:16 +08:00
Preston Van Loon
09499a732f gosec: Fix violations of G301 (#14980)
* gosec: Fix violations of G301

* Changelog fragment
2025-02-24 15:13:53 +00:00
Preston Van Loon
2ee015452c Use go-cmp and protocmp for assertion diff printing (#14978)
Fix assertion tests after switching to cmp

Changelog fragment
2025-02-23 21:06:07 +00:00
terence
ffc1bf8bbe Add more verbosity to validate range message (#14975) 2025-02-21 19:54:40 +00:00
Preston Van Loon
014dbd5c3a Lint: Fix violations of non-constant format string in call (#14974)
* Fix violations of non-constant format string in call

* Changelog fragment
2025-02-21 19:46:19 +00:00
Preston Van Loon
9bceaa59d2 tracing: Add otel tracer transport to all http clients (#14972)
* Fixed otelhttp client setups.

Note: This may not be the best solution as the http client is defined in many places. There should be a canoncial http client with the proper setup.

* Changelog fragment

* go mod tidy and gazelle
2025-02-21 19:39:53 +00:00
Preston Van Loon
832ebb3f39 Lint: Fix violations of S1009: should omit nil check; len() for nil slices is defined as zero (#14973)
* Fix violations of S1009: should omit nil check; len() for nil slices is defined as zero

* Changelog fragment
2025-02-21 19:39:33 +00:00
james-prysm
8345c271cc Code Cleanup: payload conversions and fixing gocognit ignores (#14953)
* created new converstions_execution files moving the from and to consensus functions for execution related items, also created a placeholder for block_execution with the intent of adding execution types there

* moving execution types from block.go to block_execution.go

* migrating more types and logic

* adding all the to consensus functions for payloads

* changelog

* linting

* updating unit tests for conversions

* fixing linting

* forgot to fix test

* updating name based on feedback
2025-02-21 16:12:01 +00:00
terence
56208aa84d Add more verbosity to fork digest mismatch (#14968) 2025-02-21 03:36:31 +00:00
james-prysm
b866a2c744 setting rest endpoints as deprecated for electra (#14967) 2025-02-20 18:20:51 +00:00
Nishant Das
a77234e637 Test Execution Deposit Requests in E2E (#14964)
* Test Deposit Requests

* Remove extra epochs

* Clean up Panic

* Fix Slashing Config

* Fix Slashing Test

---------

Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
2025-02-20 14:54:45 +00:00
james-prysm
e0e7354708 improving the error messages for execution request deserialization (#14962)
* improving the error messages for execution request deserialization

* changelog
2025-02-20 14:31:02 +00:00
james-prysm
0f86a16915 builder: api calls should have appropriate headers (#14961)
* adding correct headers when posting for validator registration on builder api

* changelog
2025-02-20 14:27:14 +00:00
Radosław Kapka
972c22b02f SingleAttestation support in the monitor service (#14965)
* `SingleAttestation` support in the monitor service

* changelog <3
2025-02-20 11:26:51 +00:00
Manu NALEPA
93c27340e4 Tracked validator TTL (#14957)
* `TrackedValidatorsCache`: Implement a 1-hour TTL by uding `go-cache`.

* `TrackedValidatorsCache`: Add the `ItemCount` method.

* `TrackedValidatorsCache`: Add the `Indices` method.

* Add changelog.

* `TrackedValidatorsCache`: Add prometheus metrics.

* Update beacon-chain/cache/tracked_validators.go

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>

---------

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
2025-02-19 18:04:13 +00:00
Manu NALEPA
c3edb32558 ServiceRegistry.StartAll: Remove redundant log. (#14958) 2025-02-19 17:12:32 +00:00
Sammy Rosso
3baaa732df Add get pending partial withdrawals (#14949)
* add pending partial withdrawals endpoint

* changelog

* missing new line

* fix changelog

* removing unneeded header

* using generic instead of redundant functions

---------

Co-authored-by: james-prysm <james@prysmaticlabs.com>
2025-02-19 12:34:48 +00:00
Nishant Das
8ceb7e76ea Log execution requests (#14956) 2025-02-19 10:19:04 +00:00
terence
4d5dddd302 Add request hash to header for builder: executable data to block (#14955)
* Add request hash to header for builder: executable data to block

* go fmt
2025-02-19 05:18:18 +00:00
Sammy Rosso
55efccb07f Add get pending deposits endpoint (#14941)
* Add GetPendingDeposits endpoint

* add comment

* add changelog

* gaz

* Radek' review

* move JSON object params

* gaz

* Radek' nits xD

* James' review
2025-02-18 16:16:20 +00:00
Radosław Kapka
961d8e1481 Don't use MaxCover for Electra on-chain aggregates (#14925)
* Don't use MaxCover for Electra on-chain aggregates

* changelog <3
2025-02-18 14:44:18 +00:00
Nishant Das
d396a9931e Add in Multiclient E2E For Electra (#14946)
* Add in Multiclient E2E

* Fix Execution Engine

* Update testing/endtoend/endtoend_test.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update testing/endtoend/endtoend_test.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-02-18 10:32:58 +00:00
james-prysm
e3f8f121f4 web3signer electra for e2e (#14936)
* changes needed to support web3signer running on electra for e2e

* updating web3signer version and fixing missed configs and test alignment
2025-02-18 01:36:19 +00:00
fuyangpengqi
80f29e9eda refactor: use a more straightforward return value (#14942)
Signed-off-by: fuyangpengqi <995764973@qq.com>
2025-02-17 19:18:15 +00:00
Nishant Das
8995d8133a Fix Deposit Activation Evaluator (#14938)
* Fix evaluator

* fix deposit activation
2025-02-17 13:43:23 +00:00
Preston Van Loon
31044206b8 tracing: Replace deprecated jaeger exporter with otelhttp exporter (#14928)
* Update go.opentelemetry.io/otel to v1.34.0

* Update otel exporter to replace deprecated jaeger exporter

* Changelog

* Use WithEndpointURL

* Clarify potential breaking change
2025-02-15 17:26:57 +00:00
Manu NALEPA
3a1702e56f Fixed the bazel run //:gazelle command in DEPENDENCIES.md. (#14934) 2025-02-14 15:00:10 +00:00
Nishant Das
501ec74a48 Fix Deposit Evaluator in E2E (#14933)
* fix evaluator in electra

* remove function

* Fix evaluator

---------

Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
2025-02-14 14:59:51 +00:00
Potuz
c248fe0bb3 Add logs for RPC handlers registered/removed at forks (#14932) 2025-02-14 13:01:01 +00:00
Manu NALEPA
215fbcb2e4 Remove Fulu block and state. (#14905)
* Remove Fulu block and state.

* Add missing tests.

* Alias `ProtobufBeaconStateFulu` to `ProtobufBeaconStateElectra`
2025-02-14 10:48:24 +00:00
kasey
e39f44b529 fix path parsing bug on windows (#14931)
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2025-02-14 06:10:58 +00:00
Nishant Das
9eff6ae476 Update Blst to v3.14.0 (#14921)
* updateBlst

* changelog
2025-02-13 14:31:19 +00:00
Nishant Das
3eec5a5cb6 Fix Engine Capabilites Check (#14924)
* Fix Capabilities Check

* Changelog
2025-02-13 13:37:05 +00:00
Preston Van Loon
66878deb2c Update changelog for v5.3.0 release (#14918)
* Prysm v5.3.0 changelog update

* Add v5.3.0 preamble

* Remove experimental feature from suggestions

* Changelog fragment
2025-02-12 22:36:34 +00:00
james-prysm
0b6e1711e4 Electra e2e minimal (updates geth to 1.15.0) (#14842)
* wip electra e2e

* add Deneb state to `validatorsParticipating`

* Run bazel

* add Electra state to `validatorsParticipating`

* fixing some e2e issues

* more evaluator fixes and changelog

* adding in special condition to pass electra epoch participation

* fixing typo

* missed updating forks for e2e tests

* reverting change current release fork

* missed updating e2e config for test

* updating to latest devnet 5 to fix unit tests

* go mod tidy

* fixing branch, temporary will need to update geth version later

* update to goethereum v1.15.0

* changing changelog to reflect update in geth dependency

* fixing test failures

* adding fix for range request limit during transition period between forks

* enabling validator rest for Electra

* rolling back error message

* adding fixed change logs

* fixing dependencies based on nishant's comments, deps.bzl should be updated not workspace

* partially reverting incorrect change

* removing fixes from change log, handled in separate prs

* removing comment

* updating update fraction field to the corrected spec value from prague

* Update testing/endtoend/evaluators/fork.go

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>

---------

Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
2025-02-12 15:58:06 +00:00
james-prysm
15025837bb fix: gocognit on publish block and fixing publish blinded block header check (#14913)
* refactored code and added in checks for blinded endpoints

* changelog

* cleaning up some comments and error messages

* fixing linting

* adding clarifying comment
2025-02-11 21:34:37 +00:00
Radosław Kapka
0229a2055e Rename files in beacon-chain/operations/slashings (#14904)
* pool

* service

* changelog <3
2025-02-11 16:13:23 +00:00
terence
eb9af15c7a Add blobs by range electra test (#14912) 2025-02-11 15:34:44 +00:00
james-prysm
0584746815 Dynamic max blobs config (#14911)
* fixing max config helpers to use dynamic values instead of static ones

* changelog
2025-02-11 15:04:22 +00:00
Nishant Das
8c4ea850ba Fix Blobs By Range RPC Handler (#14910)
* Add tests for TestSendBlobsByRangeRequest. Currently not working with sequential blob validation.

* Copy Root First

* Allow Test For Maximum Amount of Blobs

* Fails with the Same error

* Fix Last Test Assertion

* Add in Fix

* Changelog

---------

Co-authored-by: Preston Van Loon <preston@pvl.dev>
2025-02-11 14:11:12 +00:00
Nishant Das
4b43f13e65 Fix Blob Reconstruction (#14909)
* Fix Mutating Blob Mask

* Changelog

* Typo
2025-02-11 13:44:00 +00:00
james-prysm
26d35474e9 fix: /eth/v2/beacon/blocks post api to handle electra and fulu blocks correctly (#14897)
* adding fix and changelog

* adding no lint gocognit for now

* fixing linting

* Update beacon-chain/rpc/eth/beacon/handlers.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update beacon-chain/rpc/eth/beacon/handlers.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update beacon-chain/rpc/eth/beacon/handlers.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* updating based on kasey's suggestions

* preston's comments

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-02-10 23:50:09 +00:00
terence
9fbe3564df Update spec tests to v1.5.0-beta.2 (#14901) 2025-02-10 15:12:57 +00:00
terence
bed5547890 Add pectra testnet dates (#14884) 2025-02-10 15:09:42 +00:00
Nishant Das
47922fe7d8 Remove Unused assignment (#14906)
* Remove unused boolean assignment

* Changelog

* Remove debug line
2025-02-10 15:01:23 +00:00
Radosław Kapka
dcd25d1d97 Add missing config values from the spec (#14903)
* Add missing config values from the spec

* remove placeholders

* add some more values
2025-02-10 14:17:13 +00:00
terence
81a2a17c5f Fix electra state to safe share references on pending fields when append (#14895)
* Fix electra state to safe share references on pending fields when append

* Feedback
2025-02-08 03:04:02 +00:00
Rupam Dey
6b3f1de19d change lc flag name from enable-lightclient to enable-light-client (#14887)
* change flag name from `enable-lightclient` to `enable-light-client`

* changelog
2025-02-07 17:35:12 +00:00
Bastin
7c17af2a41 bundle handlers test (#14834)
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-02-07 16:00:22 +00:00
Nishant Das
ecf5a368d7 Update it (#14890) 2025-02-07 08:31:36 +00:00
Jun Song
557c5be433 Prune pending deposits from the deposit cache post-Electra (#14829)
* Add metrics for pruned proofs & pending deposits

* Add PruneAllProofs & PruneAllPendingDeposits

* Add simple unit tests

* Add DepositPruner interface

* Add pruning logic at post finalization task

* Move pruner logic into new file(deposit_pruner.go)

Rationale:
As deposit_fetcher.go contains all pruning logics, it would be better to separate its interest into fetcher/inserter/pruner.

* Gofmt

* Add reference link for deprecating eth1 polling

* Add changelog

* Apply reviews from nisdas and james

* add pre and post deposit request tests

* nishant's comment

---------

Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>
2025-02-07 04:31:01 +00:00
Radosław Kapka
49405c3afd Notify about attestations from the pending att queue (#14862)
* Notify about attestations from the pending att queue

* changelog <3

* fix tests

* adding to existing tests to track appropriate event feed sends

---------

Co-authored-by: james-prysm <james@prysmaticlabs.com>
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
2025-02-06 22:07:42 +00:00
Nishant Das
3439122629 Set New Blob Limits For Electra (#14883)
* Set New Blob Limits For Electra

* Add Changelog

* Bump up blob limit
2025-02-06 16:53:39 +00:00
Potuz
f6e5da6723 Do not error on overflow when converting slashings (#14882) 2025-02-05 21:01:27 +00:00
kasey
842f241cb9 Reduce size of api/client import graph (#14871)
* relocate DownloadFinalizedData from api to sync

* unexpected go mod changes

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2025-02-05 20:40:13 +00:00
kasey
41daac1b04 Organize blobs on disk by epoch (#14023)
* organize blob directories by period and epoch

* changelog

* remove Indices and replace with Summary

* old PR feedback

* log to advise about the speed of blob migration

* rename level->layer (hoping term is more clear)

* assert path in tests for increased legibility

* lint

* lint

* remove test covering a newly impossible error

* improve feedback from flag validation failure

* Try to clean dangling dirs epoch->flat migration

* lint

* Preston feedback

* try all layouts and short-circuit if base not found

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2025-02-05 20:09:38 +00:00
Potuz
2a7fc84044 Fix startup log for config file values (#14865) 2025-02-05 16:01:25 +00:00
Rupam Dey
44ff0b1a14 add missing Electra tests for light client (#14783)
* add Electra tests for finality update

* override beacon config

* add Electra tests to

* fix setupTestElectra

* changelog

* cleanup test config

* Update beacon-chain/core/light-client/lightclient_test.go

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>

* changelog

* move config to top

---------

Co-authored-by: Bastin <bastin.m@proton.me>
Co-authored-by: Bastin <43618253+Inspector-Butters@users.noreply.github.com>
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-02-05 15:16:53 +00:00
351 changed files with 12175 additions and 12096 deletions

View File

@@ -4,6 +4,141 @@ All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
## [v5.3.0](https://github.com/prysmaticlabs/prysm/compare/v5.2.0...v5.3.0) - 2025-02-12
This release includes support for Pectra activation in the [Holesky](https://github.com/eth-clients/holesky) and [Sepolia](https://github.com/eth-clients/sepolia) testnets! The release contains many fixes for Electra that have been found in rigorous testing through devnets in the last few months.
For mainnet, we have a few nice features for you to try:
- [PR #14023](https://github.com/prysmaticlabs/prysm/pull/14023) introduces a new file layout structure for storing blobs. Rather than storing all blob root directories in one parent directory, blob root directories are organized in subdirectories by epoch. This should vastly decrease the blob cache warmup time when Prysm is starting. Try this feature with `--blob-storage-layout=by-epoch`.
Updating to this release is **required** for Holesky and Sepolia operators and it is **recommended** for mainnet users as there are a few bug fixes that apply to deneb logic.
### Added
- Added an error field to log `Finished building block`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14696)
- Implemented a new `EmptyExecutionPayloadHeader` function. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14713)
- Added proper gas limit check for header from the builder. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14707)
- `Finished building block`: Display error only if not nil. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14722)
- Added light client feature flag check to RPC handlers. [PR](https://github.com/prysmaticlabs/prysm/pull/14736). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14782)
- Added support to update target and max blob count to different values per hard fork config. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14678)
- Log before blob filesystem cache warm-up. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14735)
- New design for the attestation pool. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14324)
- Add field param placeholder for Electra blob target and max to pass spec tests. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14733)
- Light client: Add better error handling. [PR](https://github.com/prysmaticlabs/prysm/pull/14749). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14782)
- Add EIP-7691: Blob throughput increase. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14750)
- Trace IDONTWANT Messages in Pubsub. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14778)
- Add Fulu fork boilerplate. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14771)
- DB optimization for saving light client bootstraps (save unique sync committees only). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14782)
- Separate type for unaggregated network attestations. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14659)
- Remote signer electra fork support. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14477)
- Add Electra test case to rewards API. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14816)
- Update `proto_test.go` to Electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14817)
- Update slasher service to Electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14812)
- Builder API endpoint to support Electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14344)
- Added protoc toolchains with a version of v25.3. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14818)
- Add test cases for the eth_lightclient_bootstrap API SSZ support. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14824)
- Handle `AttesterSlashingElectra` everywhere in the codebase. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14823)
- Add Beacon DB pruning service to prune historical data older than MIN_EPOCHS_FOR_BLOCK_REQUESTS (roughly equivalent to the weak subjectivity period). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14687)
- Nil consolidation request check for core processing. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14851)
- Updated blob sidecar api endpoint for Electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14852)
- Slashing pool service to convert slashings from Phase0 to Electra at the fork. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14844)
- check to stop eth1 voting after electra and eth1 deposits stop. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14835)
- WARN log message on node startup advising of the upcoming deprecation of the --enable-historical-state-representation feature flag. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14856)
- Beacon API event support for `SingleAttestation` and `SignedAggregateAttestationAndProofElectra`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14855)
- Added Electra tests for `TestLightClient_NewLightClientOptimisticUpdateFromBeaconState` and `TestLightClient_NewLightClientFinalityUpdateFromBeaconState`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14783)
- New option to select an alternate blob storage layout. Rather than a flat directory with a subdir for each block root, a multi-level scheme is used to organize blobs by epoch/slot/root, enabling leaner syscalls, indexing and pruning. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14023)
- Send pending att queue's attestations through the notification feed. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14862)
- Prune all pending deposits and proofs in post-Electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14829)
- Add Pectra testnet dates. (Sepolia and Holesky). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14884)
### Changed
- Process light client finality updates only for new finalized epochs instead of doing it for every block. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14713)
- Refactor subnets subscriptions. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14711)
- Refactor RPC handlers subscriptions. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14732)
- Go deps upgrade, from `ioutil` to `io`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14737)
- Move successfully registered validator(s) on builder log to debug. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14735)
- Update some test files to use `crypto/rand` instead of `math/rand`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14747)
- Re-organize the content of the `*.proto` files (No functional change). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14755)
- SSZ files generation: Remove the `// Hash: ...` header.[[PR]](https://github.com/prysmaticlabs/prysm/pull/14760)
- Updated Electra spec definition for `process_epoch`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14768)
- Update our `go-libp2p-pubsub` dependency. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14770)
- Re-organize the content of files to ease the creation of a new fork boilerplate. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14761)
- Updated spec definition electra `process_registry_updates`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14767)
- Fixed Metadata errors for peers connected via QUIC. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14776)
- Updated spec definitions for `process_slashings` in godocs. Simplified `ProcessSlashings` API. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14766)
- Update spec tests to v1.5.0-beta.0. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14788)
- Process light client finality updates only for new finalized epochs instead of doing it for every block. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14718)
- Update blobs by rpc topics from V2 to V1. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14785)
- Updated geth to 1.14~. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14351)
- E2e tests start from bellatrix. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14351)
- Version pinning unclog after making some ux improvements. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14802)
- Remove helpers to check for execution/compounding withdrawal credentials and expose them as methods. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14808)
- Refactor `2006-01-02 15:04:05` to `time.DateTime`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14792)
- Updated Prysm to Go v1.23.5. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14818)
- Updated Bazel version to v7.4.1. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14818)
- Updated rules_go to v0.46.0. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14818)
- Updated golang.org/x/tools to be compatible with v1.23.5. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14818)
- CI now requires proto files to be properly formatted with clang-format. [[PR](https://github.com/prysmaticlabs/prysm/pull/14831)]. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14831)
- Improved test coverage of beacon-chain/core/electra/churn.go. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14837)
- Update electra spec test to beta1. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14841)
- Move deposit request nil check to apply all. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14849)
- Do not mark blocks as invalid on context deadlines during state transition. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14838)
- Update electra core processing to not mark block bad if execution request error. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14826)
- Dependency: Updated go-ethereum to v1.14.13. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14872)
- improving readability on proposer settings loader. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14868)
- Removes existing validator.processSlot span and adds validator.processSlot span to slotCtx. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14874)
- DownloadFinalizedData has moved from the api/client package to beacon-chain/sync/checkpoint. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14871)
- Updated Blob-Batch-Limit to increase to 192 for electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14883)
- Updated Blob-Batch-Limit-Burst-Factor to increase to 3. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14883)
- Changed the derived batch limit when serving blobs. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14883)
- Updated go-libp2p-pubsub to v0.13.0. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14890)
- Rename light client flag from `enable-lightclient` to `enable-light-client`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14887)
- Update electra spec test to beta2. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14901)
### Removed
- Cleanup ProcessSlashings method to remove unnecessary argument. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14762)
- Remove `/proto/eth/v2` directory. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14765)
- Remove `/memsize/` pprof endpoint as it will no longer be supported in go 1.23. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14351)
- Clean `TestCanUpgrade*` tests. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14791)
- Remove `Copy()` from the `ReadOnlyBeaconBlock` interface. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14811)
- Removed a tracing span on signature requests. These requests usually took less than 5 nanoseconds and are generally not worth tracing. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14864)
### Fixed
- Added check to prevent nil pointer deference or out of bounds array access when validating the BLSToExecutionChange on an impossibly nil validator. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14705)
- EIP-7691: Ensure new blobs subnets are subscribed on epoch in advance. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14759)
- Fix kzg commitment inclusion proof depth minimal value. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14787)
- Replace exampleIP to `96.7.129.13`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14795)
- Fixed a p2p test to reliably return a static IP through DNS resolution. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14800)
- `ToBlinded`: Use Fulu struct for Fulu (instead of Electra). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14797)
- fix panic with type cast on pbgenericblock(). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14801)
- Prysmctl generate genesis state: fix truncation of ExtraData to 32 bytes to satisfy SSZ marshaling. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14803)
- added conditional evaluators to fix scenario e2e tests. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14798)
- Use `SingleAttestation` for Fulu in p2p attestation map. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14809)
- `UpgradeToFulu`: Respect the specification. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14821)
- `nodeFilter`: Implement `filterPeerForBlobSubnet` to avoid error logs. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14822)
- Fixed deposit packing for post-Electra: early return if EIP-6110 is applied. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14697)
- Fix batch process new pending deposits by getting validators from state. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14827)
- Fix handling unfound block at slot. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14852)
- Fixed incorrect attester slashing length check. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14833)
- Fix monitor service for Electra. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14853)
- add more nil checks on ToConsensus functions for added safety. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14867)
- Fix electra state to safe share references on pending fields when append. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14895)
- Add missing config values from the spec. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14903)
- We remove the unused `rebuildTrie` assignments for fields which do not use them. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14906)
- fix block api endpoint to handle blocks with the same structure but on different forks (i.e. fulu and electra). [[PR]](https://github.com/prysmaticlabs/prysm/pull/14897)
- We change how we track blob indexes during their reconstruction from the EL to prevent. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14909)
- We now use the correct maximum value when serving blobs for electra blocks. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14910)
### Security
- go version upgrade to 1.22.10 for CVE CVE-2024-34156. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14729)
- Update golang.org/x/crypto to v0.31.0 to address CVE-2024-45337. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14777)
- Update golang.org/x/net to v0.33.0 to address CVE-2024-45338. [[PR]](https://github.com/prysmaticlabs/prysm/pull/14780)
## [v5.2.0](https://github.com/prysmaticlabs/prysm/compare/v5.1.2...v5.2.0)
Updating to this release is highly recommended, especially for users running v5.1.1 or v5.1.2.
@@ -2987,4 +3122,4 @@ There are no security updates in this release.
# Older than v2.0.0
For changelog history for releases older than v2.0.0, please refer to https://github.com/prysmaticlabs/prysm/releases
For changelog history for releases older than v2.0.0, please refer to https://github.com/prysmaticlabs/prysm/releases

View File

@@ -55,7 +55,7 @@ bazel build //beacon-chain --config=release
## Adding / updating dependencies
1. Add your dependency as you would with go modules. I.e. `go get ...`
1. Run `bazel run //:gazelle -- update-repos -from_file=go.mod` to update the bazel managed dependencies.
1. Run `bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=deps.bzl%prysm_deps -prune=true` to update the bazel managed dependencies.
Example:

View File

@@ -255,7 +255,7 @@ filegroup(
url = "https://github.com/ethereum/EIPs/archive/5480440fe51742ed23342b68cf106cefd427e39d.tar.gz",
)
consensus_spec_version = "v1.5.0-beta.1"
consensus_spec_version = "v1.5.0-beta.2"
bls_test_version = "v0.1.1"
@@ -271,7 +271,7 @@ filegroup(
visibility = ["//visibility:public"],
)
""",
integrity = "sha256-R6r60geCfEjMaB1Ag3svaMFXFIgaJvkTJhfKsf76rFE=",
integrity = "sha256-X/bMxbKg1clo2aFEjBoeuFq/U+BF1eQopgRP/7nI3Qg=",
url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/general.tar.gz" % consensus_spec_version,
)
@@ -287,7 +287,7 @@ filegroup(
visibility = ["//visibility:public"],
)
""",
integrity = "sha256-2Pem2gMHxW/6bBhZ2BaqkQruQSd/dTS3WMaMQO8rZ/o=",
integrity = "sha256-WSxdri5OJGuNApW+odKle5UzToDyEOx+F3lMiqamJAg=",
url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/minimal.tar.gz" % consensus_spec_version,
)
@@ -303,7 +303,7 @@ filegroup(
visibility = ["//visibility:public"],
)
""",
integrity = "sha256-5yP05JTV1MhcUZ2kSh+T+kXjG+uW3A5877veC5c1mD4=",
integrity = "sha256-LYE8l3y/zSt4YVrehrJ3ralqtgeYNildiIp+HR6+xAI=",
url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/mainnet.tar.gz" % consensus_spec_version,
)
@@ -318,7 +318,7 @@ filegroup(
visibility = ["//visibility:public"],
)
""",
integrity = "sha256-O6Rg6h19T0RsJs0sBDZ9O1k4LnCJ/gu2ilHijFBVfME=",
integrity = "sha256-jvZQ90qcJMTOqMsPO7sgeEVQmewZTHcz7LVDkNqwTFQ=",
strip_prefix = "consensus-specs-" + consensus_spec_version[1:],
url = "https://github.com/ethereum/consensus-specs/archive/refs/tags/%s.tar.gz" % consensus_spec_version,
)

View File

@@ -3,7 +3,6 @@ load("@prysm//tools/go:def.bzl", "go_library", "go_test")
go_library(
name = "go_default_library",
srcs = [
"checkpoint.go",
"client.go",
"doc.go",
"health.go",
@@ -16,28 +15,19 @@ go_library(
"//api/client/beacon/iface:go_default_library",
"//api/server:go_default_library",
"//api/server/structs:go_default_library",
"//beacon-chain/core/helpers:go_default_library",
"//beacon-chain/state:go_default_library",
"//consensus-types/interfaces:go_default_library",
"//consensus-types/primitives:go_default_library",
"//encoding/bytesutil:go_default_library",
"//encoding/ssz/detect:go_default_library",
"//io/file:go_default_library",
"//network/forks:go_default_library",
"//proto/prysm/v1alpha1:go_default_library",
"//runtime/version:go_default_library",
"//time/slots:go_default_library",
"@com_github_ethereum_go_ethereum//common/hexutil:go_default_library",
"@com_github_pkg_errors//:go_default_library",
"@com_github_sirupsen_logrus//:go_default_library",
"@org_golang_x_mod//semver:go_default_library",
],
)
go_test(
name = "go_default_test",
srcs = [
"checkpoint_test.go",
"client_test.go",
"health_test.go",
],
@@ -45,19 +35,7 @@ go_test(
deps = [
"//api/client:go_default_library",
"//api/client/beacon/testing:go_default_library",
"//beacon-chain/state:go_default_library",
"//config/params:go_default_library",
"//consensus-types/blocks:go_default_library",
"//consensus-types/blocks/testing:go_default_library",
"//consensus-types/primitives:go_default_library",
"//encoding/ssz/detect:go_default_library",
"//network/forks:go_default_library",
"//proto/prysm/v1alpha1:go_default_library",
"//runtime/version:go_default_library",
"//testing/require:go_default_library",
"//testing/util:go_default_library",
"//time/slots:go_default_library",
"@com_github_pkg_errors//:go_default_library",
"@org_uber_go_mock//gomock:go_default_library",
],
)

View File

@@ -1,276 +0,0 @@
package beacon
import (
"context"
"fmt"
"path"
"github.com/ethereum/go-ethereum/common/hexutil"
"github.com/pkg/errors"
base "github.com/prysmaticlabs/prysm/v5/api/client"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/helpers"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
"github.com/prysmaticlabs/prysm/v5/encoding/ssz/detect"
"github.com/prysmaticlabs/prysm/v5/io/file"
"github.com/prysmaticlabs/prysm/v5/runtime/version"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/sirupsen/logrus"
"golang.org/x/mod/semver"
)
var errCheckpointBlockMismatch = errors.New("mismatch between checkpoint sync state and block")
// OriginData represents the BeaconState and ReadOnlySignedBeaconBlock necessary to start an empty Beacon Node
// using Checkpoint Sync.
type OriginData struct {
sb []byte
bb []byte
st state.BeaconState
b interfaces.ReadOnlySignedBeaconBlock
vu *detect.VersionedUnmarshaler
br [32]byte
sr [32]byte
}
// SaveBlock saves the downloaded block to a unique file in the given path.
// For readability and collision avoidance, the file name includes: type, config name, slot and root
func (o *OriginData) SaveBlock(dir string) (string, error) {
blockPath := path.Join(dir, fname("block", o.vu, o.b.Block().Slot(), o.br))
return blockPath, file.WriteFile(blockPath, o.BlockBytes())
}
// SaveState saves the downloaded state to a unique file in the given path.
// For readability and collision avoidance, the file name includes: type, config name, slot and root
func (o *OriginData) SaveState(dir string) (string, error) {
statePath := path.Join(dir, fname("state", o.vu, o.st.Slot(), o.sr))
return statePath, file.WriteFile(statePath, o.StateBytes())
}
// StateBytes returns the ssz-encoded bytes of the downloaded BeaconState value.
func (o *OriginData) StateBytes() []byte {
return o.sb
}
// BlockBytes returns the ssz-encoded bytes of the downloaded ReadOnlySignedBeaconBlock value.
func (o *OriginData) BlockBytes() []byte {
return o.bb
}
func fname(prefix string, vu *detect.VersionedUnmarshaler, slot primitives.Slot, root [32]byte) string {
return fmt.Sprintf("%s_%s_%s_%d-%#x.ssz", prefix, vu.Config.ConfigName, version.String(vu.Fork), slot, root)
}
// DownloadFinalizedData downloads the most recently finalized state, and the block most recently applied to that state.
// This pair can be used to initialize a new beacon node via checkpoint sync.
func DownloadFinalizedData(ctx context.Context, client *Client) (*OriginData, error) {
sb, err := client.GetState(ctx, IdFinalized)
if err != nil {
return nil, err
}
vu, err := detect.FromState(sb)
if err != nil {
return nil, errors.Wrap(err, "error detecting chain config for finalized state")
}
log.WithFields(logrus.Fields{
"name": vu.Config.ConfigName,
"fork": version.String(vu.Fork),
}).Info("Detected supported config in remote finalized state")
s, err := vu.UnmarshalBeaconState(sb)
if err != nil {
return nil, errors.Wrap(err, "error unmarshaling finalized state to correct version")
}
slot := s.LatestBlockHeader().Slot
bb, err := client.GetBlock(ctx, IdFromSlot(slot))
if err != nil {
return nil, errors.Wrapf(err, "error requesting block by slot = %d", slot)
}
b, err := vu.UnmarshalBeaconBlock(bb)
if err != nil {
return nil, errors.Wrap(err, "unable to unmarshal block to a supported type using the detected fork schedule")
}
br, err := b.Block().HashTreeRoot()
if err != nil {
return nil, errors.Wrap(err, "error computing hash_tree_root of retrieved block")
}
bodyRoot, err := b.Block().Body().HashTreeRoot()
if err != nil {
return nil, errors.Wrap(err, "error computing hash_tree_root of retrieved block body")
}
sbr := bytesutil.ToBytes32(s.LatestBlockHeader().BodyRoot)
if sbr != bodyRoot {
return nil, errors.Wrapf(errCheckpointBlockMismatch, "state body root = %#x, block body root = %#x", sbr, bodyRoot)
}
sr, err := s.HashTreeRoot(ctx)
if err != nil {
return nil, errors.Wrapf(err, "failed to compute htr for finalized state at slot=%d", s.Slot())
}
log.
WithField("blockSlot", b.Block().Slot()).
WithField("stateSlot", s.Slot()).
WithField("stateRoot", hexutil.Encode(sr[:])).
WithField("blockRoot", hexutil.Encode(br[:])).
Info("Downloaded checkpoint sync state and block.")
return &OriginData{
st: s,
b: b,
sb: sb,
bb: bb,
vu: vu,
br: br,
sr: sr,
}, nil
}
// WeakSubjectivityData represents the state root, block root and epoch of the BeaconState + ReadOnlySignedBeaconBlock
// that falls at the beginning of the current weak subjectivity period. These values can be used to construct
// a weak subjectivity checkpoint beacon node flag to be used for validation.
type WeakSubjectivityData struct {
BlockRoot [32]byte
StateRoot [32]byte
Epoch primitives.Epoch
}
// CheckpointString returns the standard string representation of a Checkpoint.
// The format is a hex-encoded block root, followed by the epoch of the block, separated by a colon. For example:
// "0x1c35540cac127315fabb6bf29181f2ae0de1a3fc909d2e76ba771e61312cc49a:74888"
func (wsd *WeakSubjectivityData) CheckpointString() string {
return fmt.Sprintf("%#x:%d", wsd.BlockRoot, wsd.Epoch)
}
// ComputeWeakSubjectivityCheckpoint attempts to use the prysm weak_subjectivity api
// to obtain the current weak_subjectivity checkpoint.
// For non-prysm nodes, the same computation will be performed with extra steps,
// using the head state downloaded from the beacon node api.
func ComputeWeakSubjectivityCheckpoint(ctx context.Context, client *Client) (*WeakSubjectivityData, error) {
ws, err := client.GetWeakSubjectivity(ctx)
if err != nil {
// a 404/405 is expected if querying an endpoint that doesn't support the weak subjectivity checkpoint api
if !errors.Is(err, base.ErrNotOK) {
return nil, errors.Wrap(err, "unexpected API response for prysm-only weak subjectivity checkpoint API")
}
// fall back to vanilla Beacon Node API method
return computeBackwardsCompatible(ctx, client)
}
log.Printf("server weak subjectivity checkpoint response - epoch=%d, block_root=%#x, state_root=%#x", ws.Epoch, ws.BlockRoot, ws.StateRoot)
return ws, nil
}
const (
prysmMinimumVersion = "v2.0.7"
prysmImplementationName = "Prysm"
)
// errUnsupportedPrysmCheckpointVersion indicates remote beacon node can't be used for checkpoint retrieval.
var errUnsupportedPrysmCheckpointVersion = errors.New("node does not meet minimum version requirements for checkpoint retrieval")
// for older endpoints or clients that do not support the weak_subjectivity api method
// we gather the necessary data for a checkpoint sync by:
// - inspecting the remote server's head state and computing the weak subjectivity epoch locally
// - requesting the state at the first slot of the epoch
// - using hash_tree_root(state.latest_block_header) to compute the block the state integrates
// - requesting that block by its root
func computeBackwardsCompatible(ctx context.Context, client *Client) (*WeakSubjectivityData, error) {
log.Print("falling back to generic checkpoint derivation, weak_subjectivity API not supported by server")
nv, err := client.GetNodeVersion(ctx)
if err != nil {
return nil, errors.Wrap(err, "unable to proceed with fallback method without confirming node version")
}
if nv.implementation == prysmImplementationName && semver.Compare(nv.semver, prysmMinimumVersion) < 0 {
return nil, errors.Wrapf(errUnsupportedPrysmCheckpointVersion, "%s < minimum (%s)", nv.semver, prysmMinimumVersion)
}
epoch, err := getWeakSubjectivityEpochFromHead(ctx, client)
if err != nil {
return nil, errors.Wrap(err, "error computing weak subjectivity epoch via head state inspection")
}
// use first slot of the epoch for the state slot
slot, err := slots.EpochStart(epoch)
if err != nil {
return nil, errors.Wrapf(err, "error computing first slot of epoch=%d", epoch)
}
log.Printf("requesting checkpoint state at slot %d", slot)
// get the state at the first slot of the epoch
sb, err := client.GetState(ctx, IdFromSlot(slot))
if err != nil {
return nil, errors.Wrapf(err, "failed to request state by slot from api, slot=%d", slot)
}
// ConfigFork is used to unmarshal the BeaconState so we can read the block root in latest_block_header
vu, err := detect.FromState(sb)
if err != nil {
return nil, errors.Wrap(err, "error detecting chain config for beacon state")
}
log.Printf("detected supported config in checkpoint state, name=%s, fork=%s", vu.Config.ConfigName, version.String(vu.Fork))
s, err := vu.UnmarshalBeaconState(sb)
if err != nil {
return nil, errors.Wrap(err, "error using detected config fork to unmarshal state bytes")
}
// compute state and block roots
sr, err := s.HashTreeRoot(ctx)
if err != nil {
return nil, errors.Wrap(err, "error computing hash_tree_root of state")
}
h := s.LatestBlockHeader()
h.StateRoot = sr[:]
br, err := h.HashTreeRoot()
if err != nil {
return nil, errors.Wrap(err, "error while computing block root using state data")
}
bb, err := client.GetBlock(ctx, IdFromRoot(br))
if err != nil {
return nil, errors.Wrapf(err, "error requesting block by root = %d", br)
}
b, err := vu.UnmarshalBeaconBlock(bb)
if err != nil {
return nil, errors.Wrap(err, "unable to unmarshal block to a supported type using the detected fork schedule")
}
br, err = b.Block().HashTreeRoot()
if err != nil {
return nil, errors.Wrap(err, "error computing hash_tree_root for block obtained via root")
}
return &WeakSubjectivityData{
Epoch: epoch,
BlockRoot: br,
StateRoot: sr,
}, nil
}
// this method downloads the head state, which can be used to find the correct chain config
// and use prysm's helper methods to compute the latest weak subjectivity epoch.
func getWeakSubjectivityEpochFromHead(ctx context.Context, client *Client) (primitives.Epoch, error) {
headBytes, err := client.GetState(ctx, IdHead)
if err != nil {
return 0, err
}
vu, err := detect.FromState(headBytes)
if err != nil {
return 0, errors.Wrap(err, "error detecting chain config for beacon state")
}
log.Printf("detected supported config in remote head state, name=%s, fork=%s", vu.Config.ConfigName, version.String(vu.Fork))
headState, err := vu.UnmarshalBeaconState(headBytes)
if err != nil {
return 0, errors.Wrap(err, "error unmarshaling state to correct version")
}
epoch, err := helpers.LatestWeakSubjectivityEpoch(ctx, headState, vu.Config)
if err != nil {
return 0, errors.Wrap(err, "error computing the weak subjectivity epoch from head state")
}
log.Printf("(computed client-side) weak subjectivity epoch = %d", epoch)
return epoch, nil
}

View File

@@ -29,12 +29,13 @@ const (
getSignedBlockPath = "/eth/v2/beacon/blocks"
getBlockRootPath = "/eth/v1/beacon/blocks/{{.Id}}/root"
getForkForStatePath = "/eth/v1/beacon/states/{{.Id}}/fork"
getWeakSubjectivityPath = "/prysm/v1/beacon/weak_subjectivity"
getForkSchedulePath = "/eth/v1/config/fork_schedule"
getConfigSpecPath = "/eth/v1/config/spec"
getStatePath = "/eth/v2/debug/beacon/states"
getNodeVersionPath = "/eth/v1/node/version"
changeBLStoExecutionPath = "/eth/v1/beacon/pool/bls_to_execution_changes"
GetNodeVersionPath = "/eth/v1/node/version"
GetWeakSubjectivityPath = "/prysm/v1/beacon/weak_subjectivity"
)
// StateOrBlockId represents the block_id / state_id parameters that several of the Eth Beacon API methods accept.
@@ -80,7 +81,8 @@ func idTemplate(ts string) func(StateOrBlockId) string {
return f
}
func renderGetBlockPath(id StateOrBlockId) string {
// RenderGetBlockPath formats a block id into a path for the GetBlock API endpoint.
func RenderGetBlockPath(id StateOrBlockId) string {
return path.Join(getSignedBlockPath, string(id))
}
@@ -104,7 +106,7 @@ func NewClient(host string, opts ...client.ClientOpt) (*Client, error) {
// for the named identifiers.
// The return value contains the ssz-encoded bytes.
func (c *Client) GetBlock(ctx context.Context, blockId StateOrBlockId) ([]byte, error) {
blockPath := renderGetBlockPath(blockId)
blockPath := RenderGetBlockPath(blockId)
b, err := c.Get(ctx, blockPath, client.WithSSZEncoding())
if err != nil {
return nil, errors.Wrapf(err, "error requesting state by id = %s", blockId)
@@ -195,6 +197,10 @@ type NodeVersion struct {
systemInfo string
}
func (nv *NodeVersion) SetImplementation(impl string) {
nv.implementation = impl
}
var versionRE = regexp.MustCompile(`^(\w+)/(v\d+\.\d+\.\d+[-a-zA-Z0-9]*)\s*/?(.*)$`)
func parseNodeVersion(v string) (*NodeVersion, error) {
@@ -212,7 +218,7 @@ func parseNodeVersion(v string) (*NodeVersion, error) {
// GetNodeVersion requests that the beacon node identify information about its implementation in a format
// similar to a HTTP User-Agent field. ex: Lighthouse/v0.1.5 (Linux x86_64)
func (c *Client) GetNodeVersion(ctx context.Context) (*NodeVersion, error) {
b, err := c.Get(ctx, getNodeVersionPath)
b, err := c.Get(ctx, GetNodeVersionPath)
if err != nil {
return nil, errors.Wrap(err, "error requesting node version")
}
@@ -228,7 +234,8 @@ func (c *Client) GetNodeVersion(ctx context.Context) (*NodeVersion, error) {
return parseNodeVersion(d.Data.Version)
}
func renderGetStatePath(id StateOrBlockId) string {
// RenderGetStatePath formats a state id into a path for the GetState API endpoint.
func RenderGetStatePath(id StateOrBlockId) string {
return path.Join(getStatePath, string(id))
}
@@ -246,13 +253,29 @@ func (c *Client) GetState(ctx context.Context, stateId StateOrBlockId) ([]byte,
return b, nil
}
// WeakSubjectivityData represents the state root, block root and epoch of the BeaconState + ReadOnlySignedBeaconBlock
// that falls at the beginning of the current weak subjectivity period. These values can be used to construct
// a weak subjectivity checkpoint beacon node flag to be used for validation.
type WeakSubjectivityData struct {
BlockRoot [32]byte
StateRoot [32]byte
Epoch primitives.Epoch
}
// CheckpointString returns the standard string representation of a Checkpoint.
// The format is a hex-encoded block root, followed by the epoch of the block, separated by a colon. For example:
// "0x1c35540cac127315fabb6bf29181f2ae0de1a3fc909d2e76ba771e61312cc49a:74888"
func (wsd *WeakSubjectivityData) CheckpointString() string {
return fmt.Sprintf("%#x:%d", wsd.BlockRoot, wsd.Epoch)
}
// GetWeakSubjectivity calls a proposed API endpoint that is unique to prysm
// This api method does the following:
// - computes weak subjectivity epoch
// - finds the highest non-skipped block preceding the epoch
// - returns the htr of the found block and returns this + the value of state_root from the block
func (c *Client) GetWeakSubjectivity(ctx context.Context) (*WeakSubjectivityData, error) {
body, err := c.Get(ctx, getWeakSubjectivityPath)
body, err := c.Get(ctx, GetWeakSubjectivityPath)
if err != nil {
return nil, err
}

View File

@@ -97,31 +97,31 @@ func TestValidHostname(t *testing.T) {
{
name: "hostname with port",
hostArg: "mydomain.org:3500",
path: getNodeVersionPath,
path: GetNodeVersionPath,
joined: "http://mydomain.org:3500/eth/v1/node/version",
},
{
name: "https scheme, hostname with port",
hostArg: "https://mydomain.org:3500",
path: getNodeVersionPath,
path: GetNodeVersionPath,
joined: "https://mydomain.org:3500/eth/v1/node/version",
},
{
name: "http scheme, hostname without port",
hostArg: "http://mydomain.org",
path: getNodeVersionPath,
path: GetNodeVersionPath,
joined: "http://mydomain.org/eth/v1/node/version",
},
{
name: "http scheme, trailing slash, hostname without port",
hostArg: "http://mydomain.org/",
path: getNodeVersionPath,
path: GetNodeVersionPath,
joined: "http://mydomain.org/eth/v1/node/version",
},
{
name: "http scheme, hostname with basic auth creds and no port",
hostArg: "http://username:pass@mydomain.org/",
path: getNodeVersionPath,
path: GetNodeVersionPath,
joined: "http://username:pass@mydomain.org/eth/v1/node/version",
},
}

View File

@@ -33,6 +33,7 @@ go_library(
"@com_github_pkg_errors//:go_default_library",
"@com_github_prysmaticlabs_fastssz//:go_default_library",
"@com_github_sirupsen_logrus//:go_default_library",
"@io_opentelemetry_go_contrib_instrumentation_net_http_otelhttp//:go_default_library",
"@org_golang_google_protobuf//proto:go_default_library",
],
)
@@ -46,6 +47,7 @@ go_test(
data = glob(["testdata/**"]),
embed = [":go_default_library"],
deps = [
"//api:go_default_library",
"//api/server/structs:go_default_library",
"//config/fieldparams:go_default_library",
"//config/params:go_default_library",

View File

@@ -25,6 +25,7 @@ import (
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/runtime/version"
log "github.com/sirupsen/logrus"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
const (
@@ -108,7 +109,7 @@ func NewClient(host string, opts ...ClientOpt) (*Client, error) {
return nil, err
}
c := &Client{
hc: &http.Client{},
hc: &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)},
baseURL: u,
}
for _, o := range opts {
@@ -154,6 +155,10 @@ func (c *Client) do(ctx context.Context, method string, path string, body io.Rea
if err != nil {
return
}
if method == http.MethodPost {
req.Header.Set("Content-Type", api.JsonMediaType)
}
req.Header.Set("Accept", api.JsonMediaType)
req.Header.Add("User-Agent", version.BuildData())
for _, o := range opts {
o(req)

View File

@@ -12,6 +12,7 @@ import (
"testing"
"github.com/prysmaticlabs/go-bitfield"
"github.com/prysmaticlabs/prysm/v5/api"
"github.com/prysmaticlabs/prysm/v5/api/server/structs"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
@@ -89,6 +90,8 @@ func TestClient_RegisterValidator(t *testing.T) {
expectedPath := "/eth/v1/builder/validators"
hc := &http.Client{
Transport: roundtrip(func(r *http.Request) (*http.Response, error) {
require.Equal(t, api.JsonMediaType, r.Header.Get("Content-Type"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Accept"))
body, err := io.ReadAll(r.Body)
defer func() {
require.NoError(t, r.Body.Close())
@@ -364,8 +367,8 @@ func TestSubmitBlindedBlock(t *testing.T) {
Transport: roundtrip(func(r *http.Request) (*http.Response, error) {
require.Equal(t, postBlindedBeaconBlockPath, r.URL.Path)
require.Equal(t, "bellatrix", r.Header.Get("Eth-Consensus-Version"))
require.Equal(t, "application/json", r.Header.Get("Content-Type"))
require.Equal(t, "application/json", r.Header.Get("Accept"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Content-Type"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Accept"))
return &http.Response{
StatusCode: http.StatusOK,
Body: io.NopCloser(bytes.NewBufferString(testExampleExecutionPayload)),
@@ -392,8 +395,8 @@ func TestSubmitBlindedBlock(t *testing.T) {
Transport: roundtrip(func(r *http.Request) (*http.Response, error) {
require.Equal(t, postBlindedBeaconBlockPath, r.URL.Path)
require.Equal(t, "capella", r.Header.Get("Eth-Consensus-Version"))
require.Equal(t, "application/json", r.Header.Get("Content-Type"))
require.Equal(t, "application/json", r.Header.Get("Accept"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Content-Type"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Accept"))
return &http.Response{
StatusCode: http.StatusOK,
Body: io.NopCloser(bytes.NewBufferString(testExampleExecutionPayloadCapella)),
@@ -423,8 +426,8 @@ func TestSubmitBlindedBlock(t *testing.T) {
Transport: roundtrip(func(r *http.Request) (*http.Response, error) {
require.Equal(t, postBlindedBeaconBlockPath, r.URL.Path)
require.Equal(t, "deneb", r.Header.Get("Eth-Consensus-Version"))
require.Equal(t, "application/json", r.Header.Get("Content-Type"))
require.Equal(t, "application/json", r.Header.Get("Accept"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Content-Type"))
require.Equal(t, api.JsonMediaType, r.Header.Get("Accept"))
var req structs.SignedBlindedBeaconBlockDeneb
err := json.NewDecoder(r.Body).Decode(&req)
require.NoError(t, err)

View File

@@ -4,9 +4,11 @@ go_library(
name = "go_default_library",
srcs = [
"block.go",
"block_execution.go",
"conversions.go",
"conversions_blob.go",
"conversions_block.go",
"conversions_block_execution.go",
"conversions_lightclient.go",
"conversions_state.go",
"endpoints_beacon.go",
@@ -47,10 +49,16 @@ go_library(
go_test(
name = "go_default_test",
srcs = ["conversions_test.go"],
srcs = [
"conversions_block_execution_test.go",
"conversions_test.go",
],
embed = [":go_default_library"],
deps = [
"//proto/engine/v1:go_default_library",
"//proto/prysm/v1alpha1:go_default_library",
"//testing/require:go_default_library",
"@com_github_ethereum_go_ethereum//common:go_default_library",
"@com_github_ethereum_go_ethereum//common/hexutil:go_default_library",
],
)

View File

@@ -186,40 +186,6 @@ type BlindedBeaconBlockBodyBellatrix struct {
ExecutionPayloadHeader *ExecutionPayloadHeader `json:"execution_payload_header"`
}
type ExecutionPayload struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
Transactions []string `json:"transactions"`
}
type ExecutionPayloadHeader struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
TransactionsRoot string `json:"transactions_root"`
}
// ----------------------------------------------------------------------------
// Capella
// ----------------------------------------------------------------------------
@@ -298,42 +264,6 @@ type BlindedBeaconBlockBodyCapella struct {
BLSToExecutionChanges []*SignedBLSToExecutionChange `json:"bls_to_execution_changes"`
}
type ExecutionPayloadCapella struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
Transactions []string `json:"transactions"`
Withdrawals []*Withdrawal `json:"withdrawals"`
}
type ExecutionPayloadHeaderCapella struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
TransactionsRoot string `json:"transactions_root"`
WithdrawalsRoot string `json:"withdrawals_root"`
}
// ----------------------------------------------------------------------------
// Deneb
// ----------------------------------------------------------------------------
@@ -426,46 +356,6 @@ type BlindedBeaconBlockBodyDeneb struct {
BlobKzgCommitments []string `json:"blob_kzg_commitments"`
}
type ExecutionPayloadDeneb struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
Transactions []string `json:"transactions"`
Withdrawals []*Withdrawal `json:"withdrawals"`
BlobGasUsed string `json:"blob_gas_used"`
ExcessBlobGas string `json:"excess_blob_gas"`
}
type ExecutionPayloadHeaderDeneb struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
TransactionsRoot string `json:"transactions_root"`
WithdrawalsRoot string `json:"withdrawals_root"`
BlobGasUsed string `json:"blob_gas_used"`
ExcessBlobGas string `json:"excess_blob_gas"`
}
// ----------------------------------------------------------------------------
// Electra
// ----------------------------------------------------------------------------
@@ -560,14 +450,6 @@ type BlindedBeaconBlockBodyElectra struct {
ExecutionRequests *ExecutionRequests `json:"execution_requests"`
}
type (
ExecutionRequests struct {
Deposits []*DepositRequest `json:"deposits"`
Withdrawals []*WithdrawalRequest `json:"withdrawals"`
Consolidations []*ConsolidationRequest `json:"consolidations"`
}
)
// ----------------------------------------------------------------------------
// Fulu
// ----------------------------------------------------------------------------
@@ -579,14 +461,14 @@ type SignedBeaconBlockContentsFulu struct {
}
type BeaconBlockContentsFulu struct {
Block *BeaconBlockFulu `json:"block"`
KzgProofs []string `json:"kzg_proofs"`
Blobs []string `json:"blobs"`
Block *BeaconBlockElectra `json:"block"`
KzgProofs []string `json:"kzg_proofs"`
Blobs []string `json:"blobs"`
}
type SignedBeaconBlockFulu struct {
Message *BeaconBlockFulu `json:"message"`
Signature string `json:"signature"`
Message *BeaconBlockElectra `json:"message"`
Signature string `json:"signature"`
}
var _ SignedMessageJsoner = &SignedBeaconBlockFulu{}
@@ -599,36 +481,12 @@ func (s *SignedBeaconBlockFulu) SigString() string {
return s.Signature
}
type BeaconBlockFulu struct {
Slot string `json:"slot"`
ProposerIndex string `json:"proposer_index"`
ParentRoot string `json:"parent_root"`
StateRoot string `json:"state_root"`
Body *BeaconBlockBodyFulu `json:"body"`
}
type BeaconBlockBodyFulu struct {
RandaoReveal string `json:"randao_reveal"`
Eth1Data *Eth1Data `json:"eth1_data"`
Graffiti string `json:"graffiti"`
ProposerSlashings []*ProposerSlashing `json:"proposer_slashings"`
AttesterSlashings []*AttesterSlashingElectra `json:"attester_slashings"`
Attestations []*AttestationElectra `json:"attestations"`
Deposits []*Deposit `json:"deposits"`
VoluntaryExits []*SignedVoluntaryExit `json:"voluntary_exits"`
SyncAggregate *SyncAggregate `json:"sync_aggregate"`
ExecutionPayload *ExecutionPayloadDeneb `json:"execution_payload"`
BLSToExecutionChanges []*SignedBLSToExecutionChange `json:"bls_to_execution_changes"`
BlobKzgCommitments []string `json:"blob_kzg_commitments"`
ExecutionRequests *ExecutionRequests `json:"execution_requests"`
}
type BlindedBeaconBlockFulu struct {
Slot string `json:"slot"`
ProposerIndex string `json:"proposer_index"`
ParentRoot string `json:"parent_root"`
StateRoot string `json:"state_root"`
Body *BlindedBeaconBlockBodyFulu `json:"body"`
Slot string `json:"slot"`
ProposerIndex string `json:"proposer_index"`
ParentRoot string `json:"parent_root"`
StateRoot string `json:"state_root"`
Body *BlindedBeaconBlockBodyElectra `json:"body"`
}
type SignedBlindedBeaconBlockFulu struct {
@@ -645,19 +503,3 @@ func (s *SignedBlindedBeaconBlockFulu) MessageRawJson() ([]byte, error) {
func (s *SignedBlindedBeaconBlockFulu) SigString() string {
return s.Signature
}
type BlindedBeaconBlockBodyFulu struct {
RandaoReveal string `json:"randao_reveal"`
Eth1Data *Eth1Data `json:"eth1_data"`
Graffiti string `json:"graffiti"`
ProposerSlashings []*ProposerSlashing `json:"proposer_slashings"`
AttesterSlashings []*AttesterSlashingElectra `json:"attester_slashings"`
Attestations []*AttestationElectra `json:"attestations"`
Deposits []*Deposit `json:"deposits"`
VoluntaryExits []*SignedVoluntaryExit `json:"voluntary_exits"`
SyncAggregate *SyncAggregate `json:"sync_aggregate"`
ExecutionPayloadHeader *ExecutionPayloadHeaderDeneb `json:"execution_payload_header"`
BLSToExecutionChanges []*SignedBLSToExecutionChange `json:"bls_to_execution_changes"`
BlobKzgCommitments []string `json:"blob_kzg_commitments"`
ExecutionRequests *ExecutionRequests `json:"execution_requests"`
}

View File

@@ -0,0 +1,157 @@
package structs
// ----------------------------------------------------------------------------
// Bellatrix
// ----------------------------------------------------------------------------
type ExecutionPayload struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
Transactions []string `json:"transactions"`
}
type ExecutionPayloadHeader struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
TransactionsRoot string `json:"transactions_root"`
}
// ----------------------------------------------------------------------------
// Capella
// ----------------------------------------------------------------------------
type ExecutionPayloadCapella struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
Transactions []string `json:"transactions"`
Withdrawals []*Withdrawal `json:"withdrawals"`
}
type ExecutionPayloadHeaderCapella struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
TransactionsRoot string `json:"transactions_root"`
WithdrawalsRoot string `json:"withdrawals_root"`
}
// ----------------------------------------------------------------------------
// Deneb
// ----------------------------------------------------------------------------
type ExecutionPayloadDeneb struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
Transactions []string `json:"transactions"`
Withdrawals []*Withdrawal `json:"withdrawals"`
BlobGasUsed string `json:"blob_gas_used"`
ExcessBlobGas string `json:"excess_blob_gas"`
}
type ExecutionPayloadHeaderDeneb struct {
ParentHash string `json:"parent_hash"`
FeeRecipient string `json:"fee_recipient"`
StateRoot string `json:"state_root"`
ReceiptsRoot string `json:"receipts_root"`
LogsBloom string `json:"logs_bloom"`
PrevRandao string `json:"prev_randao"`
BlockNumber string `json:"block_number"`
GasLimit string `json:"gas_limit"`
GasUsed string `json:"gas_used"`
Timestamp string `json:"timestamp"`
ExtraData string `json:"extra_data"`
BaseFeePerGas string `json:"base_fee_per_gas"`
BlockHash string `json:"block_hash"`
TransactionsRoot string `json:"transactions_root"`
WithdrawalsRoot string `json:"withdrawals_root"`
BlobGasUsed string `json:"blob_gas_used"`
ExcessBlobGas string `json:"excess_blob_gas"`
}
// ----------------------------------------------------------------------------
// Electra
// ----------------------------------------------------------------------------
type ExecutionRequests struct {
Deposits []*DepositRequest `json:"deposits"`
Withdrawals []*WithdrawalRequest `json:"withdrawals"`
Consolidations []*ConsolidationRequest `json:"consolidations"`
}
type DepositRequest struct {
Pubkey string `json:"pubkey"`
WithdrawalCredentials string `json:"withdrawal_credentials"`
Amount string `json:"amount"`
Signature string `json:"signature"`
Index string `json:"index"`
}
type WithdrawalRequest struct {
SourceAddress string `json:"source_address"`
ValidatorPubkey string `json:"validator_pubkey"`
Amount string `json:"amount"`
}
type ConsolidationRequest struct {
SourceAddress string `json:"source_address"`
SourcePubkey string `json:"source_pubkey"`
TargetPubkey string `json:"target_pubkey"`
}
// ----------------------------------------------------------------------------
// Fulu
// ----------------------------------------------------------------------------

View File

@@ -889,126 +889,6 @@ func WithdrawalFromConsensus(w *enginev1.Withdrawal) *Withdrawal {
}
}
func WithdrawalRequestsFromConsensus(ws []*enginev1.WithdrawalRequest) []*WithdrawalRequest {
result := make([]*WithdrawalRequest, len(ws))
for i, w := range ws {
result[i] = WithdrawalRequestFromConsensus(w)
}
return result
}
func WithdrawalRequestFromConsensus(w *enginev1.WithdrawalRequest) *WithdrawalRequest {
return &WithdrawalRequest{
SourceAddress: hexutil.Encode(w.SourceAddress),
ValidatorPubkey: hexutil.Encode(w.ValidatorPubkey),
Amount: fmt.Sprintf("%d", w.Amount),
}
}
func (w *WithdrawalRequest) ToConsensus() (*enginev1.WithdrawalRequest, error) {
src, err := bytesutil.DecodeHexWithLength(w.SourceAddress, common.AddressLength)
if err != nil {
return nil, server.NewDecodeError(err, "SourceAddress")
}
pubkey, err := bytesutil.DecodeHexWithLength(w.ValidatorPubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "ValidatorPubkey")
}
amount, err := strconv.ParseUint(w.Amount, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "Amount")
}
return &enginev1.WithdrawalRequest{
SourceAddress: src,
ValidatorPubkey: pubkey,
Amount: amount,
}, nil
}
func ConsolidationRequestsFromConsensus(cs []*enginev1.ConsolidationRequest) []*ConsolidationRequest {
result := make([]*ConsolidationRequest, len(cs))
for i, c := range cs {
result[i] = ConsolidationRequestFromConsensus(c)
}
return result
}
func ConsolidationRequestFromConsensus(c *enginev1.ConsolidationRequest) *ConsolidationRequest {
return &ConsolidationRequest{
SourceAddress: hexutil.Encode(c.SourceAddress),
SourcePubkey: hexutil.Encode(c.SourcePubkey),
TargetPubkey: hexutil.Encode(c.TargetPubkey),
}
}
func (c *ConsolidationRequest) ToConsensus() (*enginev1.ConsolidationRequest, error) {
srcAddress, err := bytesutil.DecodeHexWithLength(c.SourceAddress, common.AddressLength)
if err != nil {
return nil, server.NewDecodeError(err, "SourceAddress")
}
srcPubkey, err := bytesutil.DecodeHexWithLength(c.SourcePubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "SourcePubkey")
}
targetPubkey, err := bytesutil.DecodeHexWithLength(c.TargetPubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "TargetPubkey")
}
return &enginev1.ConsolidationRequest{
SourceAddress: srcAddress,
SourcePubkey: srcPubkey,
TargetPubkey: targetPubkey,
}, nil
}
func DepositRequestsFromConsensus(ds []*enginev1.DepositRequest) []*DepositRequest {
result := make([]*DepositRequest, len(ds))
for i, d := range ds {
result[i] = DepositRequestFromConsensus(d)
}
return result
}
func DepositRequestFromConsensus(d *enginev1.DepositRequest) *DepositRequest {
return &DepositRequest{
Pubkey: hexutil.Encode(d.Pubkey),
WithdrawalCredentials: hexutil.Encode(d.WithdrawalCredentials),
Amount: fmt.Sprintf("%d", d.Amount),
Signature: hexutil.Encode(d.Signature),
Index: fmt.Sprintf("%d", d.Index),
}
}
func (d *DepositRequest) ToConsensus() (*enginev1.DepositRequest, error) {
pubkey, err := bytesutil.DecodeHexWithLength(d.Pubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "Pubkey")
}
withdrawalCredentials, err := bytesutil.DecodeHexWithLength(d.WithdrawalCredentials, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "WithdrawalCredentials")
}
amount, err := strconv.ParseUint(d.Amount, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "Amount")
}
sig, err := bytesutil.DecodeHexWithLength(d.Signature, fieldparams.BLSSignatureLength)
if err != nil {
return nil, server.NewDecodeError(err, "Signature")
}
index, err := strconv.ParseUint(d.Index, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "Index")
}
return &enginev1.DepositRequest{
Pubkey: pubkey,
WithdrawalCredentials: withdrawalCredentials,
Amount: amount,
Signature: sig,
Index: index,
}, nil
}
func ProposerSlashingsToConsensus(src []*ProposerSlashing) ([]*eth.ProposerSlashing, error) {
if src == nil {
return nil, server.NewDecodeError(errNilValue, "ProposerSlashings")

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,973 @@
package structs
import (
"fmt"
"strconv"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/hexutil"
"github.com/prysmaticlabs/prysm/v5/api/server"
fieldparams "github.com/prysmaticlabs/prysm/v5/config/fieldparams"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/container/slice"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
enginev1 "github.com/prysmaticlabs/prysm/v5/proto/engine/v1"
)
// ----------------------------------------------------------------------------
// Bellatrix
// ----------------------------------------------------------------------------
func ExecutionPayloadFromConsensus(payload *enginev1.ExecutionPayload) (*ExecutionPayload, error) {
baseFeePerGas, err := sszBytesToUint256String(payload.BaseFeePerGas)
if err != nil {
return nil, err
}
transactions := make([]string, len(payload.Transactions))
for i, tx := range payload.Transactions {
transactions[i] = hexutil.Encode(tx)
}
return &ExecutionPayload{
ParentHash: hexutil.Encode(payload.ParentHash),
FeeRecipient: hexutil.Encode(payload.FeeRecipient),
StateRoot: hexutil.Encode(payload.StateRoot),
ReceiptsRoot: hexutil.Encode(payload.ReceiptsRoot),
LogsBloom: hexutil.Encode(payload.LogsBloom),
PrevRandao: hexutil.Encode(payload.PrevRandao),
BlockNumber: fmt.Sprintf("%d", payload.BlockNumber),
GasLimit: fmt.Sprintf("%d", payload.GasLimit),
GasUsed: fmt.Sprintf("%d", payload.GasUsed),
Timestamp: fmt.Sprintf("%d", payload.Timestamp),
ExtraData: hexutil.Encode(payload.ExtraData),
BaseFeePerGas: baseFeePerGas,
BlockHash: hexutil.Encode(payload.BlockHash),
Transactions: transactions,
}, nil
}
func (e *ExecutionPayload) ToConsensus() (*enginev1.ExecutionPayload, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionPayload")
}
payloadParentHash, err := bytesutil.DecodeHexWithLength(e.ParentHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ParentHash")
}
payloadFeeRecipient, err := bytesutil.DecodeHexWithLength(e.FeeRecipient, fieldparams.FeeRecipientLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.FeeRecipient")
}
payloadStateRoot, err := bytesutil.DecodeHexWithLength(e.StateRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.StateRoot")
}
payloadReceiptsRoot, err := bytesutil.DecodeHexWithLength(e.ReceiptsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ReceiptsRoot")
}
payloadLogsBloom, err := bytesutil.DecodeHexWithLength(e.LogsBloom, fieldparams.LogsBloomLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.LogsBloom")
}
payloadPrevRandao, err := bytesutil.DecodeHexWithLength(e.PrevRandao, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.PrevRandao")
}
payloadBlockNumber, err := strconv.ParseUint(e.BlockNumber, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlockNumber")
}
payloadGasLimit, err := strconv.ParseUint(e.GasLimit, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.GasLimit")
}
payloadGasUsed, err := strconv.ParseUint(e.GasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.GasUsed")
}
payloadTimestamp, err := strconv.ParseUint(e.Timestamp, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Timestamp")
}
payloadExtraData, err := bytesutil.DecodeHexWithMaxLength(e.ExtraData, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ExtraData")
}
payloadBaseFeePerGas, err := bytesutil.Uint256ToSSZBytes(e.BaseFeePerGas)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BaseFeePerGas")
}
payloadBlockHash, err := bytesutil.DecodeHexWithLength(e.BlockHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlockHash")
}
err = slice.VerifyMaxLength(e.Transactions, fieldparams.MaxTxsPerPayloadLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Transactions")
}
payloadTxs := make([][]byte, len(e.Transactions))
for i, tx := range e.Transactions {
payloadTxs[i], err = bytesutil.DecodeHexWithMaxLength(tx, fieldparams.MaxBytesPerTxLength)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Transactions[%d]", i))
}
}
return &enginev1.ExecutionPayload{
ParentHash: payloadParentHash,
FeeRecipient: payloadFeeRecipient,
StateRoot: payloadStateRoot,
ReceiptsRoot: payloadReceiptsRoot,
LogsBloom: payloadLogsBloom,
PrevRandao: payloadPrevRandao,
BlockNumber: payloadBlockNumber,
GasLimit: payloadGasLimit,
GasUsed: payloadGasUsed,
Timestamp: payloadTimestamp,
ExtraData: payloadExtraData,
BaseFeePerGas: payloadBaseFeePerGas,
BlockHash: payloadBlockHash,
Transactions: payloadTxs,
}, nil
}
func ExecutionPayloadHeaderFromConsensus(payload *enginev1.ExecutionPayloadHeader) (*ExecutionPayloadHeader, error) {
baseFeePerGas, err := sszBytesToUint256String(payload.BaseFeePerGas)
if err != nil {
return nil, err
}
return &ExecutionPayloadHeader{
ParentHash: hexutil.Encode(payload.ParentHash),
FeeRecipient: hexutil.Encode(payload.FeeRecipient),
StateRoot: hexutil.Encode(payload.StateRoot),
ReceiptsRoot: hexutil.Encode(payload.ReceiptsRoot),
LogsBloom: hexutil.Encode(payload.LogsBloom),
PrevRandao: hexutil.Encode(payload.PrevRandao),
BlockNumber: fmt.Sprintf("%d", payload.BlockNumber),
GasLimit: fmt.Sprintf("%d", payload.GasLimit),
GasUsed: fmt.Sprintf("%d", payload.GasUsed),
Timestamp: fmt.Sprintf("%d", payload.Timestamp),
ExtraData: hexutil.Encode(payload.ExtraData),
BaseFeePerGas: baseFeePerGas,
BlockHash: hexutil.Encode(payload.BlockHash),
TransactionsRoot: hexutil.Encode(payload.TransactionsRoot),
}, nil
}
func (e *ExecutionPayloadHeader) ToConsensus() (*enginev1.ExecutionPayloadHeader, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionPayloadHeader")
}
payloadParentHash, err := bytesutil.DecodeHexWithLength(e.ParentHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ParentHash")
}
payloadFeeRecipient, err := bytesutil.DecodeHexWithLength(e.FeeRecipient, fieldparams.FeeRecipientLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.FeeRecipient")
}
payloadStateRoot, err := bytesutil.DecodeHexWithLength(e.StateRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.StateRoot")
}
payloadReceiptsRoot, err := bytesutil.DecodeHexWithLength(e.ReceiptsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ReceiptsRoot")
}
payloadLogsBloom, err := bytesutil.DecodeHexWithLength(e.LogsBloom, fieldparams.LogsBloomLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.LogsBloom")
}
payloadPrevRandao, err := bytesutil.DecodeHexWithLength(e.PrevRandao, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.PrevRandao")
}
payloadBlockNumber, err := strconv.ParseUint(e.BlockNumber, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BlockNumber")
}
payloadGasLimit, err := strconv.ParseUint(e.GasLimit, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.GasLimit")
}
payloadGasUsed, err := strconv.ParseUint(e.GasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.GasUsed")
}
payloadTimestamp, err := strconv.ParseUint(e.Timestamp, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.Timestamp")
}
payloadExtraData, err := bytesutil.DecodeHexWithMaxLength(e.ExtraData, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ExtraData")
}
payloadBaseFeePerGas, err := bytesutil.Uint256ToSSZBytes(e.BaseFeePerGas)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BaseFeePerGas")
}
payloadBlockHash, err := bytesutil.DecodeHexWithLength(e.BlockHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BlockHash")
}
payloadTxsRoot, err := bytesutil.DecodeHexWithLength(e.TransactionsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.TransactionsRoot")
}
return &enginev1.ExecutionPayloadHeader{
ParentHash: payloadParentHash,
FeeRecipient: payloadFeeRecipient,
StateRoot: payloadStateRoot,
ReceiptsRoot: payloadReceiptsRoot,
LogsBloom: payloadLogsBloom,
PrevRandao: payloadPrevRandao,
BlockNumber: payloadBlockNumber,
GasLimit: payloadGasLimit,
GasUsed: payloadGasUsed,
Timestamp: payloadTimestamp,
ExtraData: payloadExtraData,
BaseFeePerGas: payloadBaseFeePerGas,
BlockHash: payloadBlockHash,
TransactionsRoot: payloadTxsRoot,
}, nil
}
// ----------------------------------------------------------------------------
// Capella
// ----------------------------------------------------------------------------
func ExecutionPayloadCapellaFromConsensus(payload *enginev1.ExecutionPayloadCapella) (*ExecutionPayloadCapella, error) {
baseFeePerGas, err := sszBytesToUint256String(payload.BaseFeePerGas)
if err != nil {
return nil, err
}
transactions := make([]string, len(payload.Transactions))
for i, tx := range payload.Transactions {
transactions[i] = hexutil.Encode(tx)
}
return &ExecutionPayloadCapella{
ParentHash: hexutil.Encode(payload.ParentHash),
FeeRecipient: hexutil.Encode(payload.FeeRecipient),
StateRoot: hexutil.Encode(payload.StateRoot),
ReceiptsRoot: hexutil.Encode(payload.ReceiptsRoot),
LogsBloom: hexutil.Encode(payload.LogsBloom),
PrevRandao: hexutil.Encode(payload.PrevRandao),
BlockNumber: fmt.Sprintf("%d", payload.BlockNumber),
GasLimit: fmt.Sprintf("%d", payload.GasLimit),
GasUsed: fmt.Sprintf("%d", payload.GasUsed),
Timestamp: fmt.Sprintf("%d", payload.Timestamp),
ExtraData: hexutil.Encode(payload.ExtraData),
BaseFeePerGas: baseFeePerGas,
BlockHash: hexutil.Encode(payload.BlockHash),
Transactions: transactions,
Withdrawals: WithdrawalsFromConsensus(payload.Withdrawals),
}, nil
}
func (e *ExecutionPayloadCapella) ToConsensus() (*enginev1.ExecutionPayloadCapella, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionPayload")
}
payloadParentHash, err := bytesutil.DecodeHexWithLength(e.ParentHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ParentHash")
}
payloadFeeRecipient, err := bytesutil.DecodeHexWithLength(e.FeeRecipient, fieldparams.FeeRecipientLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.FeeRecipient")
}
payloadStateRoot, err := bytesutil.DecodeHexWithLength(e.StateRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.StateRoot")
}
payloadReceiptsRoot, err := bytesutil.DecodeHexWithLength(e.ReceiptsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ReceiptsRoot")
}
payloadLogsBloom, err := bytesutil.DecodeHexWithLength(e.LogsBloom, fieldparams.LogsBloomLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.LogsBloom")
}
payloadPrevRandao, err := bytesutil.DecodeHexWithLength(e.PrevRandao, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.PrevRandao")
}
payloadBlockNumber, err := strconv.ParseUint(e.BlockNumber, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlockNumber")
}
payloadGasLimit, err := strconv.ParseUint(e.GasLimit, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.GasLimit")
}
payloadGasUsed, err := strconv.ParseUint(e.GasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.GasUsed")
}
payloadTimestamp, err := strconv.ParseUint(e.Timestamp, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Timestamp")
}
payloadExtraData, err := bytesutil.DecodeHexWithMaxLength(e.ExtraData, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ExtraData")
}
payloadBaseFeePerGas, err := bytesutil.Uint256ToSSZBytes(e.BaseFeePerGas)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BaseFeePerGas")
}
payloadBlockHash, err := bytesutil.DecodeHexWithLength(e.BlockHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlockHash")
}
err = slice.VerifyMaxLength(e.Transactions, fieldparams.MaxTxsPerPayloadLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Transactions")
}
payloadTxs := make([][]byte, len(e.Transactions))
for i, tx := range e.Transactions {
payloadTxs[i], err = bytesutil.DecodeHexWithMaxLength(tx, fieldparams.MaxBytesPerTxLength)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Transactions[%d]", i))
}
}
err = slice.VerifyMaxLength(e.Withdrawals, fieldparams.MaxWithdrawalsPerPayload)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Withdrawals")
}
withdrawals := make([]*enginev1.Withdrawal, len(e.Withdrawals))
for i, w := range e.Withdrawals {
withdrawalIndex, err := strconv.ParseUint(w.WithdrawalIndex, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].WithdrawalIndex", i))
}
validatorIndex, err := strconv.ParseUint(w.ValidatorIndex, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].ValidatorIndex", i))
}
address, err := bytesutil.DecodeHexWithLength(w.ExecutionAddress, common.AddressLength)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].ExecutionAddress", i))
}
amount, err := strconv.ParseUint(w.Amount, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].Amount", i))
}
withdrawals[i] = &enginev1.Withdrawal{
Index: withdrawalIndex,
ValidatorIndex: primitives.ValidatorIndex(validatorIndex),
Address: address,
Amount: amount,
}
}
return &enginev1.ExecutionPayloadCapella{
ParentHash: payloadParentHash,
FeeRecipient: payloadFeeRecipient,
StateRoot: payloadStateRoot,
ReceiptsRoot: payloadReceiptsRoot,
LogsBloom: payloadLogsBloom,
PrevRandao: payloadPrevRandao,
BlockNumber: payloadBlockNumber,
GasLimit: payloadGasLimit,
GasUsed: payloadGasUsed,
Timestamp: payloadTimestamp,
ExtraData: payloadExtraData,
BaseFeePerGas: payloadBaseFeePerGas,
BlockHash: payloadBlockHash,
Transactions: payloadTxs,
Withdrawals: withdrawals,
}, nil
}
func ExecutionPayloadHeaderCapellaFromConsensus(payload *enginev1.ExecutionPayloadHeaderCapella) (*ExecutionPayloadHeaderCapella, error) {
baseFeePerGas, err := sszBytesToUint256String(payload.BaseFeePerGas)
if err != nil {
return nil, err
}
return &ExecutionPayloadHeaderCapella{
ParentHash: hexutil.Encode(payload.ParentHash),
FeeRecipient: hexutil.Encode(payload.FeeRecipient),
StateRoot: hexutil.Encode(payload.StateRoot),
ReceiptsRoot: hexutil.Encode(payload.ReceiptsRoot),
LogsBloom: hexutil.Encode(payload.LogsBloom),
PrevRandao: hexutil.Encode(payload.PrevRandao),
BlockNumber: fmt.Sprintf("%d", payload.BlockNumber),
GasLimit: fmt.Sprintf("%d", payload.GasLimit),
GasUsed: fmt.Sprintf("%d", payload.GasUsed),
Timestamp: fmt.Sprintf("%d", payload.Timestamp),
ExtraData: hexutil.Encode(payload.ExtraData),
BaseFeePerGas: baseFeePerGas,
BlockHash: hexutil.Encode(payload.BlockHash),
TransactionsRoot: hexutil.Encode(payload.TransactionsRoot),
WithdrawalsRoot: hexutil.Encode(payload.WithdrawalsRoot),
}, nil
}
func (e *ExecutionPayloadHeaderCapella) ToConsensus() (*enginev1.ExecutionPayloadHeaderCapella, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionPayloadHeader")
}
payloadParentHash, err := bytesutil.DecodeHexWithLength(e.ParentHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ParentHash")
}
payloadFeeRecipient, err := bytesutil.DecodeHexWithLength(e.FeeRecipient, fieldparams.FeeRecipientLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.FeeRecipient")
}
payloadStateRoot, err := bytesutil.DecodeHexWithLength(e.StateRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.StateRoot")
}
payloadReceiptsRoot, err := bytesutil.DecodeHexWithLength(e.ReceiptsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ReceiptsRoot")
}
payloadLogsBloom, err := bytesutil.DecodeHexWithLength(e.LogsBloom, fieldparams.LogsBloomLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.LogsBloom")
}
payloadPrevRandao, err := bytesutil.DecodeHexWithLength(e.PrevRandao, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.PrevRandao")
}
payloadBlockNumber, err := strconv.ParseUint(e.BlockNumber, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BlockNumber")
}
payloadGasLimit, err := strconv.ParseUint(e.GasLimit, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.GasLimit")
}
payloadGasUsed, err := strconv.ParseUint(e.GasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.GasUsed")
}
payloadTimestamp, err := strconv.ParseUint(e.Timestamp, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.Timestamp")
}
payloadExtraData, err := bytesutil.DecodeHexWithMaxLength(e.ExtraData, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ExtraData")
}
payloadBaseFeePerGas, err := bytesutil.Uint256ToSSZBytes(e.BaseFeePerGas)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BaseFeePerGas")
}
payloadBlockHash, err := bytesutil.DecodeHexWithMaxLength(e.BlockHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BlockHash")
}
payloadTxsRoot, err := bytesutil.DecodeHexWithMaxLength(e.TransactionsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.TransactionsRoot")
}
payloadWithdrawalsRoot, err := bytesutil.DecodeHexWithMaxLength(e.WithdrawalsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.WithdrawalsRoot")
}
return &enginev1.ExecutionPayloadHeaderCapella{
ParentHash: payloadParentHash,
FeeRecipient: payloadFeeRecipient,
StateRoot: payloadStateRoot,
ReceiptsRoot: payloadReceiptsRoot,
LogsBloom: payloadLogsBloom,
PrevRandao: payloadPrevRandao,
BlockNumber: payloadBlockNumber,
GasLimit: payloadGasLimit,
GasUsed: payloadGasUsed,
Timestamp: payloadTimestamp,
ExtraData: payloadExtraData,
BaseFeePerGas: payloadBaseFeePerGas,
BlockHash: payloadBlockHash,
TransactionsRoot: payloadTxsRoot,
WithdrawalsRoot: payloadWithdrawalsRoot,
}, nil
}
// ----------------------------------------------------------------------------
// Deneb
// ----------------------------------------------------------------------------
func ExecutionPayloadDenebFromConsensus(payload *enginev1.ExecutionPayloadDeneb) (*ExecutionPayloadDeneb, error) {
baseFeePerGas, err := sszBytesToUint256String(payload.BaseFeePerGas)
if err != nil {
return nil, err
}
transactions := make([]string, len(payload.Transactions))
for i, tx := range payload.Transactions {
transactions[i] = hexutil.Encode(tx)
}
return &ExecutionPayloadDeneb{
ParentHash: hexutil.Encode(payload.ParentHash),
FeeRecipient: hexutil.Encode(payload.FeeRecipient),
StateRoot: hexutil.Encode(payload.StateRoot),
ReceiptsRoot: hexutil.Encode(payload.ReceiptsRoot),
LogsBloom: hexutil.Encode(payload.LogsBloom),
PrevRandao: hexutil.Encode(payload.PrevRandao),
BlockNumber: fmt.Sprintf("%d", payload.BlockNumber),
GasLimit: fmt.Sprintf("%d", payload.GasLimit),
GasUsed: fmt.Sprintf("%d", payload.GasUsed),
Timestamp: fmt.Sprintf("%d", payload.Timestamp),
ExtraData: hexutil.Encode(payload.ExtraData),
BaseFeePerGas: baseFeePerGas,
BlockHash: hexutil.Encode(payload.BlockHash),
Transactions: transactions,
Withdrawals: WithdrawalsFromConsensus(payload.Withdrawals),
BlobGasUsed: fmt.Sprintf("%d", payload.BlobGasUsed),
ExcessBlobGas: fmt.Sprintf("%d", payload.ExcessBlobGas),
}, nil
}
func (e *ExecutionPayloadDeneb) ToConsensus() (*enginev1.ExecutionPayloadDeneb, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionPayload")
}
payloadParentHash, err := bytesutil.DecodeHexWithLength(e.ParentHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ParentHash")
}
payloadFeeRecipient, err := bytesutil.DecodeHexWithLength(e.FeeRecipient, fieldparams.FeeRecipientLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.FeeRecipient")
}
payloadStateRoot, err := bytesutil.DecodeHexWithLength(e.StateRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.StateRoot")
}
payloadReceiptsRoot, err := bytesutil.DecodeHexWithLength(e.ReceiptsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ReceiptsRoot")
}
payloadLogsBloom, err := bytesutil.DecodeHexWithLength(e.LogsBloom, fieldparams.LogsBloomLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.LogsBloom")
}
payloadPrevRandao, err := bytesutil.DecodeHexWithLength(e.PrevRandao, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.PrevRandao")
}
payloadBlockNumber, err := strconv.ParseUint(e.BlockNumber, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlockNumber")
}
payloadGasLimit, err := strconv.ParseUint(e.GasLimit, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.GasLimit")
}
payloadGasUsed, err := strconv.ParseUint(e.GasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.GasUsed")
}
payloadTimestamp, err := strconv.ParseUint(e.Timestamp, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.Timestamp")
}
payloadExtraData, err := bytesutil.DecodeHexWithMaxLength(e.ExtraData, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ExtraData")
}
payloadBaseFeePerGas, err := bytesutil.Uint256ToSSZBytes(e.BaseFeePerGas)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BaseFeePerGas")
}
payloadBlockHash, err := bytesutil.DecodeHexWithLength(e.BlockHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlockHash")
}
err = slice.VerifyMaxLength(e.Transactions, fieldparams.MaxTxsPerPayloadLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Transactions")
}
txs := make([][]byte, len(e.Transactions))
for i, tx := range e.Transactions {
txs[i], err = bytesutil.DecodeHexWithMaxLength(tx, fieldparams.MaxBytesPerTxLength)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Transactions[%d]", i))
}
}
err = slice.VerifyMaxLength(e.Withdrawals, fieldparams.MaxWithdrawalsPerPayload)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.Withdrawals")
}
withdrawals := make([]*enginev1.Withdrawal, len(e.Withdrawals))
for i, w := range e.Withdrawals {
withdrawalIndex, err := strconv.ParseUint(w.WithdrawalIndex, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].WithdrawalIndex", i))
}
validatorIndex, err := strconv.ParseUint(w.ValidatorIndex, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].ValidatorIndex", i))
}
address, err := bytesutil.DecodeHexWithLength(w.ExecutionAddress, common.AddressLength)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].ExecutionAddress", i))
}
amount, err := strconv.ParseUint(w.Amount, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionPayload.Withdrawals[%d].Amount", i))
}
withdrawals[i] = &enginev1.Withdrawal{
Index: withdrawalIndex,
ValidatorIndex: primitives.ValidatorIndex(validatorIndex),
Address: address,
Amount: amount,
}
}
payloadBlobGasUsed, err := strconv.ParseUint(e.BlobGasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlobGasUsed")
}
payloadExcessBlobGas, err := strconv.ParseUint(e.ExcessBlobGas, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ExcessBlobGas")
}
return &enginev1.ExecutionPayloadDeneb{
ParentHash: payloadParentHash,
FeeRecipient: payloadFeeRecipient,
StateRoot: payloadStateRoot,
ReceiptsRoot: payloadReceiptsRoot,
LogsBloom: payloadLogsBloom,
PrevRandao: payloadPrevRandao,
BlockNumber: payloadBlockNumber,
GasLimit: payloadGasLimit,
GasUsed: payloadGasUsed,
Timestamp: payloadTimestamp,
ExtraData: payloadExtraData,
BaseFeePerGas: payloadBaseFeePerGas,
BlockHash: payloadBlockHash,
Transactions: txs,
Withdrawals: withdrawals,
BlobGasUsed: payloadBlobGasUsed,
ExcessBlobGas: payloadExcessBlobGas,
}, nil
}
func ExecutionPayloadHeaderDenebFromConsensus(payload *enginev1.ExecutionPayloadHeaderDeneb) (*ExecutionPayloadHeaderDeneb, error) {
baseFeePerGas, err := sszBytesToUint256String(payload.BaseFeePerGas)
if err != nil {
return nil, err
}
return &ExecutionPayloadHeaderDeneb{
ParentHash: hexutil.Encode(payload.ParentHash),
FeeRecipient: hexutil.Encode(payload.FeeRecipient),
StateRoot: hexutil.Encode(payload.StateRoot),
ReceiptsRoot: hexutil.Encode(payload.ReceiptsRoot),
LogsBloom: hexutil.Encode(payload.LogsBloom),
PrevRandao: hexutil.Encode(payload.PrevRandao),
BlockNumber: fmt.Sprintf("%d", payload.BlockNumber),
GasLimit: fmt.Sprintf("%d", payload.GasLimit),
GasUsed: fmt.Sprintf("%d", payload.GasUsed),
Timestamp: fmt.Sprintf("%d", payload.Timestamp),
ExtraData: hexutil.Encode(payload.ExtraData),
BaseFeePerGas: baseFeePerGas,
BlockHash: hexutil.Encode(payload.BlockHash),
TransactionsRoot: hexutil.Encode(payload.TransactionsRoot),
WithdrawalsRoot: hexutil.Encode(payload.WithdrawalsRoot),
BlobGasUsed: fmt.Sprintf("%d", payload.BlobGasUsed),
ExcessBlobGas: fmt.Sprintf("%d", payload.ExcessBlobGas),
}, nil
}
func (e *ExecutionPayloadHeaderDeneb) ToConsensus() (*enginev1.ExecutionPayloadHeaderDeneb, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionPayloadHeader")
}
payloadParentHash, err := bytesutil.DecodeHexWithLength(e.ParentHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ParentHash")
}
payloadFeeRecipient, err := bytesutil.DecodeHexWithLength(e.FeeRecipient, fieldparams.FeeRecipientLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.FeeRecipient")
}
payloadStateRoot, err := bytesutil.DecodeHexWithLength(e.StateRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.StateRoot")
}
payloadReceiptsRoot, err := bytesutil.DecodeHexWithLength(e.ReceiptsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ReceiptsRoot")
}
payloadLogsBloom, err := bytesutil.DecodeHexWithLength(e.LogsBloom, fieldparams.LogsBloomLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.LogsBloom")
}
payloadPrevRandao, err := bytesutil.DecodeHexWithLength(e.PrevRandao, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.PrevRandao")
}
payloadBlockNumber, err := strconv.ParseUint(e.BlockNumber, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BlockNumber")
}
payloadGasLimit, err := strconv.ParseUint(e.GasLimit, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.GasLimit")
}
payloadGasUsed, err := strconv.ParseUint(e.GasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.GasUsed")
}
payloadTimestamp, err := strconv.ParseUint(e.Timestamp, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.Timestamp")
}
payloadExtraData, err := bytesutil.DecodeHexWithMaxLength(e.ExtraData, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.ExtraData")
}
payloadBaseFeePerGas, err := bytesutil.Uint256ToSSZBytes(e.BaseFeePerGas)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BaseFeePerGas")
}
payloadBlockHash, err := bytesutil.DecodeHexWithLength(e.BlockHash, common.HashLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.BlockHash")
}
payloadTxsRoot, err := bytesutil.DecodeHexWithLength(e.TransactionsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.TransactionsRoot")
}
payloadWithdrawalsRoot, err := bytesutil.DecodeHexWithLength(e.WithdrawalsRoot, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayloadHeader.WithdrawalsRoot")
}
payloadBlobGasUsed, err := strconv.ParseUint(e.BlobGasUsed, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.BlobGasUsed")
}
payloadExcessBlobGas, err := strconv.ParseUint(e.ExcessBlobGas, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "ExecutionPayload.ExcessBlobGas")
}
return &enginev1.ExecutionPayloadHeaderDeneb{
ParentHash: payloadParentHash,
FeeRecipient: payloadFeeRecipient,
StateRoot: payloadStateRoot,
ReceiptsRoot: payloadReceiptsRoot,
LogsBloom: payloadLogsBloom,
PrevRandao: payloadPrevRandao,
BlockNumber: payloadBlockNumber,
GasLimit: payloadGasLimit,
GasUsed: payloadGasUsed,
Timestamp: payloadTimestamp,
ExtraData: payloadExtraData,
BaseFeePerGas: payloadBaseFeePerGas,
BlockHash: payloadBlockHash,
TransactionsRoot: payloadTxsRoot,
WithdrawalsRoot: payloadWithdrawalsRoot,
BlobGasUsed: payloadBlobGasUsed,
ExcessBlobGas: payloadExcessBlobGas,
}, nil
}
// ----------------------------------------------------------------------------
// Electra
// ----------------------------------------------------------------------------
var (
ExecutionPayloadElectraFromConsensus = ExecutionPayloadDenebFromConsensus
ExecutionPayloadHeaderElectraFromConsensus = ExecutionPayloadHeaderDenebFromConsensus
)
func WithdrawalRequestsFromConsensus(ws []*enginev1.WithdrawalRequest) []*WithdrawalRequest {
result := make([]*WithdrawalRequest, len(ws))
for i, w := range ws {
result[i] = WithdrawalRequestFromConsensus(w)
}
return result
}
func WithdrawalRequestFromConsensus(w *enginev1.WithdrawalRequest) *WithdrawalRequest {
return &WithdrawalRequest{
SourceAddress: hexutil.Encode(w.SourceAddress),
ValidatorPubkey: hexutil.Encode(w.ValidatorPubkey),
Amount: fmt.Sprintf("%d", w.Amount),
}
}
func (w *WithdrawalRequest) ToConsensus() (*enginev1.WithdrawalRequest, error) {
if w == nil {
return nil, server.NewDecodeError(errNilValue, "WithdrawalRequest")
}
src, err := bytesutil.DecodeHexWithLength(w.SourceAddress, common.AddressLength)
if err != nil {
return nil, server.NewDecodeError(err, "SourceAddress")
}
pubkey, err := bytesutil.DecodeHexWithLength(w.ValidatorPubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "ValidatorPubkey")
}
amount, err := strconv.ParseUint(w.Amount, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "Amount")
}
return &enginev1.WithdrawalRequest{
SourceAddress: src,
ValidatorPubkey: pubkey,
Amount: amount,
}, nil
}
func ConsolidationRequestsFromConsensus(cs []*enginev1.ConsolidationRequest) []*ConsolidationRequest {
result := make([]*ConsolidationRequest, len(cs))
for i, c := range cs {
result[i] = ConsolidationRequestFromConsensus(c)
}
return result
}
func ConsolidationRequestFromConsensus(c *enginev1.ConsolidationRequest) *ConsolidationRequest {
return &ConsolidationRequest{
SourceAddress: hexutil.Encode(c.SourceAddress),
SourcePubkey: hexutil.Encode(c.SourcePubkey),
TargetPubkey: hexutil.Encode(c.TargetPubkey),
}
}
func (c *ConsolidationRequest) ToConsensus() (*enginev1.ConsolidationRequest, error) {
if c == nil {
return nil, server.NewDecodeError(errNilValue, "ConsolidationRequest")
}
srcAddress, err := bytesutil.DecodeHexWithLength(c.SourceAddress, common.AddressLength)
if err != nil {
return nil, server.NewDecodeError(err, "SourceAddress")
}
srcPubkey, err := bytesutil.DecodeHexWithLength(c.SourcePubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "SourcePubkey")
}
targetPubkey, err := bytesutil.DecodeHexWithLength(c.TargetPubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "TargetPubkey")
}
return &enginev1.ConsolidationRequest{
SourceAddress: srcAddress,
SourcePubkey: srcPubkey,
TargetPubkey: targetPubkey,
}, nil
}
func DepositRequestsFromConsensus(ds []*enginev1.DepositRequest) []*DepositRequest {
result := make([]*DepositRequest, len(ds))
for i, d := range ds {
result[i] = DepositRequestFromConsensus(d)
}
return result
}
func DepositRequestFromConsensus(d *enginev1.DepositRequest) *DepositRequest {
return &DepositRequest{
Pubkey: hexutil.Encode(d.Pubkey),
WithdrawalCredentials: hexutil.Encode(d.WithdrawalCredentials),
Amount: fmt.Sprintf("%d", d.Amount),
Signature: hexutil.Encode(d.Signature),
Index: fmt.Sprintf("%d", d.Index),
}
}
func (d *DepositRequest) ToConsensus() (*enginev1.DepositRequest, error) {
if d == nil {
return nil, server.NewDecodeError(errNilValue, "DepositRequest")
}
pubkey, err := bytesutil.DecodeHexWithLength(d.Pubkey, fieldparams.BLSPubkeyLength)
if err != nil {
return nil, server.NewDecodeError(err, "Pubkey")
}
withdrawalCredentials, err := bytesutil.DecodeHexWithLength(d.WithdrawalCredentials, fieldparams.RootLength)
if err != nil {
return nil, server.NewDecodeError(err, "WithdrawalCredentials")
}
amount, err := strconv.ParseUint(d.Amount, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "Amount")
}
sig, err := bytesutil.DecodeHexWithLength(d.Signature, fieldparams.BLSSignatureLength)
if err != nil {
return nil, server.NewDecodeError(err, "Signature")
}
index, err := strconv.ParseUint(d.Index, 10, 64)
if err != nil {
return nil, server.NewDecodeError(err, "Index")
}
return &enginev1.DepositRequest{
Pubkey: pubkey,
WithdrawalCredentials: withdrawalCredentials,
Amount: amount,
Signature: sig,
Index: index,
}, nil
}
func ExecutionRequestsFromConsensus(er *enginev1.ExecutionRequests) *ExecutionRequests {
return &ExecutionRequests{
Deposits: DepositRequestsFromConsensus(er.Deposits),
Withdrawals: WithdrawalRequestsFromConsensus(er.Withdrawals),
Consolidations: ConsolidationRequestsFromConsensus(er.Consolidations),
}
}
func (e *ExecutionRequests) ToConsensus() (*enginev1.ExecutionRequests, error) {
if e == nil {
return nil, server.NewDecodeError(errNilValue, "ExecutionRequests")
}
var err error
if err = slice.VerifyMaxLength(e.Deposits, params.BeaconConfig().MaxDepositRequestsPerPayload); err != nil {
return nil, err
}
depositRequests := make([]*enginev1.DepositRequest, len(e.Deposits))
for i, d := range e.Deposits {
depositRequests[i], err = d.ToConsensus()
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionRequests.Deposits[%d]", i))
}
}
if err = slice.VerifyMaxLength(e.Withdrawals, params.BeaconConfig().MaxWithdrawalRequestsPerPayload); err != nil {
return nil, err
}
withdrawalRequests := make([]*enginev1.WithdrawalRequest, len(e.Withdrawals))
for i, w := range e.Withdrawals {
withdrawalRequests[i], err = w.ToConsensus()
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionRequests.Withdrawals[%d]", i))
}
}
if err = slice.VerifyMaxLength(e.Consolidations, params.BeaconConfig().MaxConsolidationsRequestsPerPayload); err != nil {
return nil, err
}
consolidationRequests := make([]*enginev1.ConsolidationRequest, len(e.Consolidations))
for i, c := range e.Consolidations {
consolidationRequests[i], err = c.ToConsensus()
if err != nil {
return nil, server.NewDecodeError(err, fmt.Sprintf("ExecutionRequests.Consolidations[%d]", i))
}
}
return &enginev1.ExecutionRequests{
Deposits: depositRequests,
Withdrawals: withdrawalRequests,
Consolidations: consolidationRequests,
}, nil
}
// ----------------------------------------------------------------------------
// Fulu
// ----------------------------------------------------------------------------
var (
ExecutionPayloadFuluFromConsensus = ExecutionPayloadDenebFromConsensus
ExecutionPayloadHeaderFuluFromConsensus = ExecutionPayloadHeaderDenebFromConsensus
BeaconBlockFuluFromConsensus = BeaconBlockElectraFromConsensus
)

View File

@@ -0,0 +1,563 @@
package structs
import (
"fmt"
"testing"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/hexutil"
enginev1 "github.com/prysmaticlabs/prysm/v5/proto/engine/v1"
"github.com/prysmaticlabs/prysm/v5/testing/require"
)
func fillByteSlice(sliceLength int, value byte) []byte {
bytes := make([]byte, sliceLength)
for index := range bytes {
bytes[index] = value
}
return bytes
}
// TestExecutionPayloadFromConsensus_HappyPath checks the
// ExecutionPayloadFromConsensus function under normal conditions.
func TestExecutionPayloadFromConsensus_HappyPath(t *testing.T) {
consensusPayload := &enginev1.ExecutionPayload{
ParentHash: fillByteSlice(common.HashLength, 0xaa),
FeeRecipient: fillByteSlice(20, 0xbb),
StateRoot: fillByteSlice(32, 0xcc),
ReceiptsRoot: fillByteSlice(32, 0xdd),
LogsBloom: fillByteSlice(256, 0xee),
PrevRandao: fillByteSlice(32, 0xff),
BlockNumber: 12345,
GasLimit: 15000000,
GasUsed: 8000000,
Timestamp: 1680000000,
ExtraData: fillByteSlice(8, 0x11),
BaseFeePerGas: fillByteSlice(32, 0x01),
BlockHash: fillByteSlice(common.HashLength, 0x22),
Transactions: [][]byte{
fillByteSlice(10, 0x33),
fillByteSlice(10, 0x44),
},
}
result, err := ExecutionPayloadFromConsensus(consensusPayload)
require.NoError(t, err)
require.NotNil(t, result)
require.Equal(t, hexutil.Encode(consensusPayload.ParentHash), result.ParentHash)
require.Equal(t, hexutil.Encode(consensusPayload.FeeRecipient), result.FeeRecipient)
require.Equal(t, hexutil.Encode(consensusPayload.StateRoot), result.StateRoot)
require.Equal(t, hexutil.Encode(consensusPayload.ReceiptsRoot), result.ReceiptsRoot)
require.Equal(t, fmt.Sprintf("%d", consensusPayload.BlockNumber), result.BlockNumber)
}
// TestExecutionPayload_ToConsensus_HappyPath checks the
// (*ExecutionPayload).ToConsensus function under normal conditions.
func TestExecutionPayload_ToConsensus_HappyPath(t *testing.T) {
payload := &ExecutionPayload{
ParentHash: hexutil.Encode(fillByteSlice(common.HashLength, 0xaa)),
FeeRecipient: hexutil.Encode(fillByteSlice(20, 0xbb)),
StateRoot: hexutil.Encode(fillByteSlice(32, 0xcc)),
ReceiptsRoot: hexutil.Encode(fillByteSlice(32, 0xdd)),
LogsBloom: hexutil.Encode(fillByteSlice(256, 0xee)),
PrevRandao: hexutil.Encode(fillByteSlice(32, 0xff)),
BlockNumber: "12345",
GasLimit: "15000000",
GasUsed: "8000000",
Timestamp: "1680000000",
ExtraData: "0x11111111",
BaseFeePerGas: "1234",
BlockHash: hexutil.Encode(fillByteSlice(common.HashLength, 0x22)),
Transactions: []string{
hexutil.Encode(fillByteSlice(10, 0x33)),
hexutil.Encode(fillByteSlice(10, 0x44)),
},
}
result, err := payload.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, result.ParentHash, fillByteSlice(common.HashLength, 0xaa))
require.DeepEqual(t, result.FeeRecipient, fillByteSlice(20, 0xbb))
require.DeepEqual(t, result.StateRoot, fillByteSlice(32, 0xcc))
}
// TestExecutionPayloadHeaderFromConsensus_HappyPath checks the
// ExecutionPayloadHeaderFromConsensus function under normal conditions.
func TestExecutionPayloadHeaderFromConsensus_HappyPath(t *testing.T) {
consensusHeader := &enginev1.ExecutionPayloadHeader{
ParentHash: fillByteSlice(common.HashLength, 0xaa),
FeeRecipient: fillByteSlice(20, 0xbb),
StateRoot: fillByteSlice(32, 0xcc),
ReceiptsRoot: fillByteSlice(32, 0xdd),
LogsBloom: fillByteSlice(256, 0xee),
PrevRandao: fillByteSlice(32, 0xff),
BlockNumber: 9999,
GasLimit: 5000000,
GasUsed: 2500000,
Timestamp: 1111111111,
ExtraData: fillByteSlice(4, 0x12),
BaseFeePerGas: fillByteSlice(32, 0x34),
BlockHash: fillByteSlice(common.HashLength, 0x56),
TransactionsRoot: fillByteSlice(32, 0x78),
}
result, err := ExecutionPayloadHeaderFromConsensus(consensusHeader)
require.NoError(t, err)
require.NotNil(t, result)
require.Equal(t, hexutil.Encode(consensusHeader.ParentHash), result.ParentHash)
require.Equal(t, fmt.Sprintf("%d", consensusHeader.BlockNumber), result.BlockNumber)
}
// TestExecutionPayloadHeader_ToConsensus_HappyPath checks the
// (*ExecutionPayloadHeader).ToConsensus function under normal conditions.
func TestExecutionPayloadHeader_ToConsensus_HappyPath(t *testing.T) {
header := &ExecutionPayloadHeader{
ParentHash: hexutil.Encode(fillByteSlice(common.HashLength, 0xaa)),
FeeRecipient: hexutil.Encode(fillByteSlice(20, 0xbb)),
StateRoot: hexutil.Encode(fillByteSlice(32, 0xcc)),
ReceiptsRoot: hexutil.Encode(fillByteSlice(32, 0xdd)),
LogsBloom: hexutil.Encode(fillByteSlice(256, 0xee)),
PrevRandao: hexutil.Encode(fillByteSlice(32, 0xff)),
BlockNumber: "9999",
GasLimit: "5000000",
GasUsed: "2500000",
Timestamp: "1111111111",
ExtraData: "0x1234abcd",
BaseFeePerGas: "1234",
BlockHash: hexutil.Encode(fillByteSlice(common.HashLength, 0x56)),
TransactionsRoot: hexutil.Encode(fillByteSlice(32, 0x78)),
}
result, err := header.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, hexutil.Encode(result.ParentHash), header.ParentHash)
require.DeepEqual(t, hexutil.Encode(result.FeeRecipient), header.FeeRecipient)
require.DeepEqual(t, hexutil.Encode(result.StateRoot), header.StateRoot)
}
// TestExecutionPayloadCapellaFromConsensus_HappyPath checks the
// ExecutionPayloadCapellaFromConsensus function under normal conditions.
func TestExecutionPayloadCapellaFromConsensus_HappyPath(t *testing.T) {
capellaPayload := &enginev1.ExecutionPayloadCapella{
ParentHash: fillByteSlice(common.HashLength, 0xaa),
FeeRecipient: fillByteSlice(20, 0xbb),
StateRoot: fillByteSlice(32, 0xcc),
ReceiptsRoot: fillByteSlice(32, 0xdd),
LogsBloom: fillByteSlice(256, 0xee),
PrevRandao: fillByteSlice(32, 0xff),
BlockNumber: 123,
GasLimit: 9876543,
GasUsed: 1234567,
Timestamp: 5555555,
ExtraData: fillByteSlice(6, 0x11),
BaseFeePerGas: fillByteSlice(32, 0x22),
BlockHash: fillByteSlice(common.HashLength, 0x33),
Transactions: [][]byte{
fillByteSlice(5, 0x44),
},
Withdrawals: []*enginev1.Withdrawal{
{
Index: 1,
ValidatorIndex: 2,
Address: fillByteSlice(20, 0xaa),
Amount: 100,
},
},
}
result, err := ExecutionPayloadCapellaFromConsensus(capellaPayload)
require.NoError(t, err)
require.NotNil(t, result)
require.Equal(t, hexutil.Encode(capellaPayload.ParentHash), result.ParentHash)
require.Equal(t, len(capellaPayload.Transactions), len(result.Transactions))
require.Equal(t, len(capellaPayload.Withdrawals), len(result.Withdrawals))
}
// TestExecutionPayloadCapella_ToConsensus_HappyPath checks the
// (*ExecutionPayloadCapella).ToConsensus function under normal conditions.
func TestExecutionPayloadCapella_ToConsensus_HappyPath(t *testing.T) {
capella := &ExecutionPayloadCapella{
ParentHash: hexutil.Encode(fillByteSlice(common.HashLength, 0xaa)),
FeeRecipient: hexutil.Encode(fillByteSlice(20, 0xbb)),
StateRoot: hexutil.Encode(fillByteSlice(32, 0xcc)),
ReceiptsRoot: hexutil.Encode(fillByteSlice(32, 0xdd)),
LogsBloom: hexutil.Encode(fillByteSlice(256, 0xee)),
PrevRandao: hexutil.Encode(fillByteSlice(32, 0xff)),
BlockNumber: "123",
GasLimit: "9876543",
GasUsed: "1234567",
Timestamp: "5555555",
ExtraData: hexutil.Encode(fillByteSlice(6, 0x11)),
BaseFeePerGas: "1234",
BlockHash: hexutil.Encode(fillByteSlice(common.HashLength, 0x33)),
Transactions: []string{
hexutil.Encode(fillByteSlice(5, 0x44)),
},
Withdrawals: []*Withdrawal{
{
WithdrawalIndex: "1",
ValidatorIndex: "2",
ExecutionAddress: hexutil.Encode(fillByteSlice(20, 0xaa)),
Amount: "100",
},
},
}
result, err := capella.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, hexutil.Encode(result.ParentHash), capella.ParentHash)
require.DeepEqual(t, hexutil.Encode(result.FeeRecipient), capella.FeeRecipient)
require.DeepEqual(t, hexutil.Encode(result.StateRoot), capella.StateRoot)
}
// TestExecutionPayloadDenebFromConsensus_HappyPath checks the
// ExecutionPayloadDenebFromConsensus function under normal conditions.
func TestExecutionPayloadDenebFromConsensus_HappyPath(t *testing.T) {
denebPayload := &enginev1.ExecutionPayloadDeneb{
ParentHash: fillByteSlice(common.HashLength, 0xaa),
FeeRecipient: fillByteSlice(20, 0xbb),
StateRoot: fillByteSlice(32, 0xcc),
ReceiptsRoot: fillByteSlice(32, 0xdd),
LogsBloom: fillByteSlice(256, 0xee),
PrevRandao: fillByteSlice(32, 0xff),
BlockNumber: 999,
GasLimit: 2222222,
GasUsed: 1111111,
Timestamp: 666666,
ExtraData: fillByteSlice(6, 0x11),
BaseFeePerGas: fillByteSlice(32, 0x22),
BlockHash: fillByteSlice(common.HashLength, 0x33),
Transactions: [][]byte{
fillByteSlice(5, 0x44),
},
Withdrawals: []*enginev1.Withdrawal{
{
Index: 1,
ValidatorIndex: 2,
Address: fillByteSlice(20, 0xaa),
Amount: 100,
},
},
BlobGasUsed: 1234,
ExcessBlobGas: 5678,
}
result, err := ExecutionPayloadDenebFromConsensus(denebPayload)
require.NoError(t, err)
require.Equal(t, hexutil.Encode(denebPayload.ParentHash), result.ParentHash)
require.Equal(t, len(denebPayload.Transactions), len(result.Transactions))
require.Equal(t, len(denebPayload.Withdrawals), len(result.Withdrawals))
require.Equal(t, "1234", result.BlobGasUsed)
require.Equal(t, fmt.Sprintf("%d", denebPayload.BlockNumber), result.BlockNumber)
}
// TestExecutionPayloadDeneb_ToConsensus_HappyPath checks the
// (*ExecutionPayloadDeneb).ToConsensus function under normal conditions.
func TestExecutionPayloadDeneb_ToConsensus_HappyPath(t *testing.T) {
deneb := &ExecutionPayloadDeneb{
ParentHash: hexutil.Encode(fillByteSlice(common.HashLength, 0xaa)),
FeeRecipient: hexutil.Encode(fillByteSlice(20, 0xbb)),
StateRoot: hexutil.Encode(fillByteSlice(32, 0xcc)),
ReceiptsRoot: hexutil.Encode(fillByteSlice(32, 0xdd)),
LogsBloom: hexutil.Encode(fillByteSlice(256, 0xee)),
PrevRandao: hexutil.Encode(fillByteSlice(32, 0xff)),
BlockNumber: "999",
GasLimit: "2222222",
GasUsed: "1111111",
Timestamp: "666666",
ExtraData: hexutil.Encode(fillByteSlice(6, 0x11)),
BaseFeePerGas: "1234",
BlockHash: hexutil.Encode(fillByteSlice(common.HashLength, 0x33)),
Transactions: []string{
hexutil.Encode(fillByteSlice(5, 0x44)),
},
Withdrawals: []*Withdrawal{
{
WithdrawalIndex: "1",
ValidatorIndex: "2",
ExecutionAddress: hexutil.Encode(fillByteSlice(20, 0xaa)),
Amount: "100",
},
},
BlobGasUsed: "1234",
ExcessBlobGas: "5678",
}
result, err := deneb.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, hexutil.Encode(result.ParentHash), deneb.ParentHash)
require.DeepEqual(t, hexutil.Encode(result.FeeRecipient), deneb.FeeRecipient)
require.Equal(t, result.BlockNumber, uint64(999))
}
func TestExecutionPayloadHeaderCapellaFromConsensus_HappyPath(t *testing.T) {
capellaHeader := &enginev1.ExecutionPayloadHeaderCapella{
ParentHash: fillByteSlice(common.HashLength, 0xaa),
FeeRecipient: fillByteSlice(20, 0xbb),
StateRoot: fillByteSlice(32, 0xcc),
ReceiptsRoot: fillByteSlice(32, 0xdd),
LogsBloom: fillByteSlice(256, 0xee),
PrevRandao: fillByteSlice(32, 0xff),
BlockNumber: 555,
GasLimit: 1111111,
GasUsed: 222222,
Timestamp: 3333333333,
ExtraData: fillByteSlice(4, 0x12),
BaseFeePerGas: fillByteSlice(32, 0x34),
BlockHash: fillByteSlice(common.HashLength, 0x56),
TransactionsRoot: fillByteSlice(32, 0x78),
WithdrawalsRoot: fillByteSlice(32, 0x99),
}
result, err := ExecutionPayloadHeaderCapellaFromConsensus(capellaHeader)
require.NoError(t, err)
require.Equal(t, hexutil.Encode(capellaHeader.ParentHash), result.ParentHash)
require.DeepEqual(t, hexutil.Encode(capellaHeader.WithdrawalsRoot), result.WithdrawalsRoot)
}
func TestExecutionPayloadHeaderCapella_ToConsensus_HappyPath(t *testing.T) {
header := &ExecutionPayloadHeaderCapella{
ParentHash: hexutil.Encode(fillByteSlice(common.HashLength, 0xaa)),
FeeRecipient: hexutil.Encode(fillByteSlice(20, 0xbb)),
StateRoot: hexutil.Encode(fillByteSlice(32, 0xcc)),
ReceiptsRoot: hexutil.Encode(fillByteSlice(32, 0xdd)),
LogsBloom: hexutil.Encode(fillByteSlice(256, 0xee)),
PrevRandao: hexutil.Encode(fillByteSlice(32, 0xff)),
BlockNumber: "555",
GasLimit: "1111111",
GasUsed: "222222",
Timestamp: "3333333333",
ExtraData: "0x1234abcd",
BaseFeePerGas: "1234",
BlockHash: hexutil.Encode(fillByteSlice(common.HashLength, 0x56)),
TransactionsRoot: hexutil.Encode(fillByteSlice(32, 0x78)),
WithdrawalsRoot: hexutil.Encode(fillByteSlice(32, 0x99)),
}
result, err := header.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, hexutil.Encode(result.ParentHash), header.ParentHash)
require.DeepEqual(t, hexutil.Encode(result.FeeRecipient), header.FeeRecipient)
require.DeepEqual(t, hexutil.Encode(result.StateRoot), header.StateRoot)
require.DeepEqual(t, hexutil.Encode(result.ReceiptsRoot), header.ReceiptsRoot)
require.DeepEqual(t, hexutil.Encode(result.WithdrawalsRoot), header.WithdrawalsRoot)
}
func TestExecutionPayloadHeaderDenebFromConsensus_HappyPath(t *testing.T) {
denebHeader := &enginev1.ExecutionPayloadHeaderDeneb{
ParentHash: fillByteSlice(common.HashLength, 0xaa),
FeeRecipient: fillByteSlice(20, 0xbb),
StateRoot: fillByteSlice(32, 0xcc),
ReceiptsRoot: fillByteSlice(32, 0xdd),
LogsBloom: fillByteSlice(256, 0xee),
PrevRandao: fillByteSlice(32, 0xff),
BlockNumber: 999,
GasLimit: 5000000,
GasUsed: 2500000,
Timestamp: 4444444444,
ExtraData: fillByteSlice(4, 0x12),
BaseFeePerGas: fillByteSlice(32, 0x34),
BlockHash: fillByteSlice(common.HashLength, 0x56),
TransactionsRoot: fillByteSlice(32, 0x78),
WithdrawalsRoot: fillByteSlice(32, 0x99),
BlobGasUsed: 1234,
ExcessBlobGas: 5678,
}
result, err := ExecutionPayloadHeaderDenebFromConsensus(denebHeader)
require.NoError(t, err)
require.Equal(t, hexutil.Encode(denebHeader.ParentHash), result.ParentHash)
require.DeepEqual(t, hexutil.Encode(denebHeader.FeeRecipient), result.FeeRecipient)
require.DeepEqual(t, hexutil.Encode(denebHeader.StateRoot), result.StateRoot)
require.DeepEqual(t, fmt.Sprintf("%d", denebHeader.BlobGasUsed), result.BlobGasUsed)
}
func TestExecutionPayloadHeaderDeneb_ToConsensus_HappyPath(t *testing.T) {
header := &ExecutionPayloadHeaderDeneb{
ParentHash: hexutil.Encode(fillByteSlice(common.HashLength, 0xaa)),
FeeRecipient: hexutil.Encode(fillByteSlice(20, 0xbb)),
StateRoot: hexutil.Encode(fillByteSlice(32, 0xcc)),
ReceiptsRoot: hexutil.Encode(fillByteSlice(32, 0xdd)),
LogsBloom: hexutil.Encode(fillByteSlice(256, 0xee)),
PrevRandao: hexutil.Encode(fillByteSlice(32, 0xff)),
BlockNumber: "999",
GasLimit: "5000000",
GasUsed: "2500000",
Timestamp: "4444444444",
ExtraData: "0x1234abcd",
BaseFeePerGas: "1234",
BlockHash: hexutil.Encode(fillByteSlice(common.HashLength, 0x56)),
TransactionsRoot: hexutil.Encode(fillByteSlice(32, 0x78)),
WithdrawalsRoot: hexutil.Encode(fillByteSlice(32, 0x99)),
BlobGasUsed: "1234",
ExcessBlobGas: "5678",
}
result, err := header.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, hexutil.Encode(result.ParentHash), header.ParentHash)
require.DeepEqual(t, result.BlobGasUsed, uint64(1234))
require.DeepEqual(t, result.ExcessBlobGas, uint64(5678))
require.DeepEqual(t, result.BlockNumber, uint64(999))
}
func TestWithdrawalRequestsFromConsensus_HappyPath(t *testing.T) {
consensusRequests := []*enginev1.WithdrawalRequest{
{
SourceAddress: fillByteSlice(20, 0xbb),
ValidatorPubkey: fillByteSlice(48, 0xbb),
Amount: 12345,
},
{
SourceAddress: fillByteSlice(20, 0xcc),
ValidatorPubkey: fillByteSlice(48, 0xcc),
Amount: 54321,
},
}
result := WithdrawalRequestsFromConsensus(consensusRequests)
require.DeepEqual(t, len(result), len(consensusRequests))
require.DeepEqual(t, result[0].Amount, fmt.Sprintf("%d", consensusRequests[0].Amount))
}
func TestWithdrawalRequestFromConsensus_HappyPath(t *testing.T) {
req := &enginev1.WithdrawalRequest{
SourceAddress: fillByteSlice(20, 0xbb),
ValidatorPubkey: fillByteSlice(48, 0xbb),
Amount: 42,
}
result := WithdrawalRequestFromConsensus(req)
require.NotNil(t, result)
require.DeepEqual(t, result.SourceAddress, hexutil.Encode(fillByteSlice(20, 0xbb)))
}
func TestWithdrawalRequest_ToConsensus_HappyPath(t *testing.T) {
withdrawalReq := &WithdrawalRequest{
SourceAddress: hexutil.Encode(fillByteSlice(20, 111)),
ValidatorPubkey: hexutil.Encode(fillByteSlice(48, 123)),
Amount: "12345",
}
result, err := withdrawalReq.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, result.Amount, uint64(12345))
}
func TestConsolidationRequestsFromConsensus_HappyPath(t *testing.T) {
consensusRequests := []*enginev1.ConsolidationRequest{
{
SourceAddress: fillByteSlice(20, 111),
SourcePubkey: fillByteSlice(48, 112),
TargetPubkey: fillByteSlice(48, 113),
},
}
result := ConsolidationRequestsFromConsensus(consensusRequests)
require.DeepEqual(t, len(result), len(consensusRequests))
require.DeepEqual(t, result[0].SourceAddress, "0x6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f")
}
func TestDepositRequestsFromConsensus_HappyPath(t *testing.T) {
ds := []*enginev1.DepositRequest{
{
Pubkey: fillByteSlice(48, 0xbb),
WithdrawalCredentials: fillByteSlice(32, 0xdd),
Amount: 98765,
Signature: fillByteSlice(96, 0xff),
Index: 111,
},
}
result := DepositRequestsFromConsensus(ds)
require.DeepEqual(t, len(result), len(ds))
require.DeepEqual(t, result[0].Amount, "98765")
}
func TestDepositRequest_ToConsensus_HappyPath(t *testing.T) {
req := &DepositRequest{
Pubkey: hexutil.Encode(fillByteSlice(48, 0xbb)),
WithdrawalCredentials: hexutil.Encode(fillByteSlice(32, 0xaa)),
Amount: "123",
Signature: hexutil.Encode(fillByteSlice(96, 0xdd)),
Index: "456",
}
result, err := req.ToConsensus()
require.NoError(t, err)
require.DeepEqual(t, result.Amount, uint64(123))
require.DeepEqual(t, result.Signature, fillByteSlice(96, 0xdd))
}
func TestExecutionRequestsFromConsensus_HappyPath(t *testing.T) {
er := &enginev1.ExecutionRequests{
Deposits: []*enginev1.DepositRequest{
{
Pubkey: fillByteSlice(48, 0xba),
WithdrawalCredentials: fillByteSlice(32, 0xaa),
Amount: 33,
Signature: fillByteSlice(96, 0xff),
Index: 44,
},
},
Withdrawals: []*enginev1.WithdrawalRequest{
{
SourceAddress: fillByteSlice(20, 0xaa),
ValidatorPubkey: fillByteSlice(48, 0xba),
Amount: 555,
},
},
Consolidations: []*enginev1.ConsolidationRequest{
{
SourceAddress: fillByteSlice(20, 0xdd),
SourcePubkey: fillByteSlice(48, 0xdd),
TargetPubkey: fillByteSlice(48, 0xcc),
},
},
}
result := ExecutionRequestsFromConsensus(er)
require.NotNil(t, result)
require.Equal(t, 1, len(result.Deposits))
require.Equal(t, "33", result.Deposits[0].Amount)
require.Equal(t, 1, len(result.Withdrawals))
require.Equal(t, "555", result.Withdrawals[0].Amount)
require.Equal(t, 1, len(result.Consolidations))
require.Equal(t, "0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", result.Consolidations[0].TargetPubkey)
}
func TestExecutionRequests_ToConsensus_HappyPath(t *testing.T) {
execReq := &ExecutionRequests{
Deposits: []*DepositRequest{
{
Pubkey: hexutil.Encode(fillByteSlice(48, 0xbb)),
WithdrawalCredentials: hexutil.Encode(fillByteSlice(32, 0xaa)),
Amount: "33",
Signature: hexutil.Encode(fillByteSlice(96, 0xff)),
Index: "44",
},
},
Withdrawals: []*WithdrawalRequest{
{
SourceAddress: hexutil.Encode(fillByteSlice(20, 0xdd)),
ValidatorPubkey: hexutil.Encode(fillByteSlice(48, 0xbb)),
Amount: "555",
},
},
Consolidations: []*ConsolidationRequest{
{
SourceAddress: hexutil.Encode(fillByteSlice(20, 0xcc)),
SourcePubkey: hexutil.Encode(fillByteSlice(48, 0xbb)),
TargetPubkey: hexutil.Encode(fillByteSlice(48, 0xcc)),
},
},
}
result, err := execReq.ToConsensus()
require.NoError(t, err)
require.Equal(t, 1, len(result.Deposits))
require.Equal(t, uint64(33), result.Deposits[0].Amount)
require.Equal(t, 1, len(result.Withdrawals))
require.Equal(t, uint64(555), result.Withdrawals[0].Amount)
require.Equal(t, 1, len(result.Consolidations))
require.DeepEqual(t, fillByteSlice(48, 0xcc), result.Consolidations[0].TargetPubkey)
}

View File

@@ -250,3 +250,17 @@ type ChainHead struct {
PreviousJustifiedBlockRoot string `json:"previous_justified_block_root"`
OptimisticStatus bool `json:"optimistic_status"`
}
type GetPendingDepositsResponse struct {
Version string `json:"version"`
ExecutionOptimistic bool `json:"execution_optimistic"`
Finalized bool `json:"finalized"`
Data []*PendingDeposit `json:"data"`
}
type GetPendingPartialWithdrawalsResponse struct {
Version string `json:"version"`
ExecutionOptimistic bool `json:"execution_optimistic"`
Finalized bool `json:"finalized"`
Data []*PendingPartialWithdrawal `json:"data"`
}

View File

@@ -244,26 +244,6 @@ type Withdrawal struct {
Amount string `json:"amount"`
}
type DepositRequest struct {
Pubkey string `json:"pubkey"`
WithdrawalCredentials string `json:"withdrawal_credentials"`
Amount string `json:"amount"`
Signature string `json:"signature"`
Index string `json:"index"`
}
type WithdrawalRequest struct {
SourceAddress string `json:"source_address"`
ValidatorPubkey string `json:"validator_pubkey"`
Amount string `json:"amount"`
}
type ConsolidationRequest struct {
SourceAddress string `json:"source_address"`
SourcePubkey string `json:"source_pubkey"`
TargetPubkey string `json:"target_pubkey"`
}
type PendingDeposit struct {
Pubkey string `json:"pubkey"`
WithdrawalCredentials string `json:"withdrawal_credentials"`

View File

@@ -125,7 +125,7 @@ func getChan(key string) chan byte {
// Return a new string with unique elements.
func unique(arr []string) []string {
if arr == nil || len(arr) <= 1 {
if len(arr) <= 1 {
return arr
}

View File

@@ -12,6 +12,7 @@ go_library(
"forkchoice_update_execution.go",
"head.go",
"head_sync_committee_info.go",
"holeskyhack.go",
"init_sync_process_block.go",
"log.go",
"merge_ascii_art.go",
@@ -26,6 +27,7 @@ go_library(
"receive_blob.go",
"receive_block.go",
"service.go",
"setup_forchoice.go",
"tracked_proposer.go",
"weak_subjectivity_checks.go",
],

View File

@@ -0,0 +1,21 @@
package blockchain
import (
"encoding/hex"
"github.com/pkg/errors"
)
var errHoleskyForbiddenRoot = errors.New("refusing to process forbidden holesky block")
// hack to prevent bad holesky block importation
var badHoleskyRoot [32]byte
func init() {
hexStr := "2db899881ed8546476d0b92c6aa9110bea9a4cd0dbeb5519eb0ea69575f1f359"
bytes, err := hex.DecodeString(hexStr)
if err != nil {
panic(err)
}
badHoleskyRoot = [32]byte(bytes)
}

View File

@@ -69,6 +69,22 @@ func logStateTransitionData(b interfaces.ReadOnlyBeaconBlock) error {
log = log.WithField("kzgCommitmentCount", len(kzgs))
}
}
if b.Version() >= version.Electra {
eReqs, err := b.Body().ExecutionRequests()
if err != nil {
log.WithError(err).Error("Failed to get execution requests")
} else {
if len(eReqs.Deposits) > 0 {
log = log.WithField("depositRequestCount", len(eReqs.Deposits))
}
if len(eReqs.Consolidations) > 0 {
log = log.WithField("consolidationRequestCount", len(eReqs.Consolidations))
}
if len(eReqs.Withdrawals) > 0 {
log = log.WithField("withdrawalRequestCount", len(eReqs.Withdrawals))
}
}
}
log.Info("Finished applying state transition")
return nil
}

View File

@@ -173,6 +173,9 @@ func (s *Service) onBlockBatch(ctx context.Context, blks []consensusblocks.ROBlo
var set *bls.SignatureBatch
boundaries := make(map[[32]byte]state.BeaconState)
for i, b := range blks {
if b.Root() == badHoleskyRoot {
return errHoleskyForbiddenRoot
}
v, h, err := getStateVersionAndPayload(preState)
if err != nil {
return err
@@ -512,17 +515,11 @@ func missingIndices(bs *filesystem.BlobStorage, root [32]byte, expected [][]byte
if len(expected) > maxBlobsPerBlock {
return nil, errMaxBlobsExceeded
}
indices, err := bs.Indices(root, slot)
if err != nil {
return nil, err
}
indices := bs.Summary(root)
missing := make(map[uint64]struct{}, len(expected))
for i := range expected {
ui := uint64(i)
if len(expected[i]) > 0 {
if !indices[i] {
missing[ui] = struct{}{}
}
if len(expected[i]) > 0 && !indices.HasIndex(uint64(i)) {
missing[uint64(i)] = struct{}{}
}
}
return missing, nil

View File

@@ -7,6 +7,7 @@ import (
"strings"
"time"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/helpers"
lightclient "github.com/prysmaticlabs/prysm/v5/beacon-chain/core/light-client"
"github.com/ethereum/go-ethereum/common"
@@ -552,7 +553,8 @@ func (s *Service) fillInForkChoiceMissingBlocks(ctx context.Context, signed inte
// inserts finalized deposits into our finalized deposit trie, needs to be
// called in the background
func (s *Service) insertFinalizedDeposits(ctx context.Context, fRoot [32]byte) {
// Post-Electra: prunes all proofs and pending deposits in the cache
func (s *Service) insertFinalizedDepositsAndPrune(ctx context.Context, fRoot [32]byte) {
ctx, span := trace.StartSpan(ctx, "blockChain.insertFinalizedDeposits")
defer span.End()
startTime := time.Now()
@@ -563,6 +565,16 @@ func (s *Service) insertFinalizedDeposits(ctx context.Context, fRoot [32]byte) {
log.WithError(err).Error("could not fetch finalized state")
return
}
// Check if we should prune all pending deposits.
// In post-Electra(after the legacy deposit mechanism is deprecated),
// we can prune all pending deposits in the deposit cache.
// See: https://eips.ethereum.org/EIPS/eip-6110#eth1data-poll-deprecation
if helpers.DepositRequestsStarted(finalizedState) {
s.pruneAllPendingDepositsAndProofs(ctx)
return
}
// We update the cache up to the last deposit index in the finalized block's state.
// We can be confident that these deposits will be included in some block
// because the Eth1 follow distance makes such long-range reorgs extremely unlikely.
@@ -591,6 +603,12 @@ func (s *Service) insertFinalizedDeposits(ctx context.Context, fRoot [32]byte) {
log.WithField("duration", time.Since(startTime).String()).Debugf("Finalized deposit insertion completed at index %d", finalizedEth1DepIdx)
}
// pruneAllPendingDepositsAndProofs prunes all proofs and pending deposits in the cache.
func (s *Service) pruneAllPendingDepositsAndProofs(ctx context.Context) {
s.cfg.DepositCache.PruneAllPendingDeposits(ctx)
s.cfg.DepositCache.PruneAllProofs(ctx)
}
// This ensures that the input root defaults to using genesis root instead of zero hashes. This is needed for handling
// fork choice justification routine.
func (s *Service) ensureRootNotZeros(root [32]byte) [32]byte {

View File

@@ -723,7 +723,7 @@ func TestInsertFinalizedDeposits(t *testing.T) {
Signature: zeroSig[:],
}, Proof: [][]byte{root}}, 100+i, int64(i), bytesutil.ToBytes32(root)))
}
service.insertFinalizedDeposits(ctx, [32]byte{'m', 'o', 'c', 'k'})
service.insertFinalizedDepositsAndPrune(ctx, [32]byte{'m', 'o', 'c', 'k'})
fDeposits, err := depositCache.FinalizedDeposits(ctx)
require.NoError(t, err)
assert.Equal(t, 7, int(fDeposits.MerkleTrieIndex()), "Finalized deposits not inserted correctly")
@@ -759,7 +759,7 @@ func TestInsertFinalizedDeposits_PrunePendingDeposits(t *testing.T) {
Signature: zeroSig[:],
}, Proof: [][]byte{root}}, 100+i, int64(i), bytesutil.ToBytes32(root))
}
service.insertFinalizedDeposits(ctx, [32]byte{'m', 'o', 'c', 'k'})
service.insertFinalizedDepositsAndPrune(ctx, [32]byte{'m', 'o', 'c', 'k'})
fDeposits, err := depositCache.FinalizedDeposits(ctx)
require.NoError(t, err)
assert.Equal(t, 7, int(fDeposits.MerkleTrieIndex()), "Finalized deposits not inserted correctly")
@@ -799,7 +799,7 @@ func TestInsertFinalizedDeposits_MultipleFinalizedRoutines(t *testing.T) {
}
// Insert 3 deposits before hand.
require.NoError(t, depositCache.InsertFinalizedDeposits(ctx, 2, [32]byte{}, 0))
service.insertFinalizedDeposits(ctx, [32]byte{'m', 'o', 'c', 'k'})
service.insertFinalizedDepositsAndPrune(ctx, [32]byte{'m', 'o', 'c', 'k'})
fDeposits, err := depositCache.FinalizedDeposits(ctx)
require.NoError(t, err)
assert.Equal(t, 5, int(fDeposits.MerkleTrieIndex()), "Finalized deposits not inserted correctly")
@@ -810,7 +810,7 @@ func TestInsertFinalizedDeposits_MultipleFinalizedRoutines(t *testing.T) {
}
// Insert New Finalized State with higher deposit count.
service.insertFinalizedDeposits(ctx, [32]byte{'m', 'o', 'c', 'k', '2'})
service.insertFinalizedDepositsAndPrune(ctx, [32]byte{'m', 'o', 'c', 'k', '2'})
fDeposits, err = depositCache.FinalizedDeposits(ctx)
require.NoError(t, err)
assert.Equal(t, 12, int(fDeposits.MerkleTrieIndex()), "Finalized deposits not inserted correctly")
@@ -2039,7 +2039,7 @@ func TestNoViableHead_Reboot(t *testing.T) {
require.Equal(t, genesisRoot, bytesutil.ToBytes32(headRoot))
optimistic, err := service.IsOptimistic(ctx)
require.NoError(t, err)
require.Equal(t, false, optimistic)
require.Equal(t, true, optimistic)
// Check that the node's justified checkpoint does not agree with the
// last valid state's justified checkpoint
@@ -2297,7 +2297,7 @@ func TestMissingIndices(t *testing.T) {
for _, c := range cases {
bm, bs := filesystem.NewEphemeralBlobStorageWithMocker(t)
t.Run(c.name, func(t *testing.T) {
require.NoError(t, bm.CreateFakeIndices(c.root, c.present...))
require.NoError(t, bm.CreateFakeIndices(c.root, 0, c.present...))
missing, err := missingIndices(bs, c.root, c.expected, 0)
if c.err != nil {
require.ErrorIs(t, err, c.err)

View File

@@ -65,6 +65,11 @@ type SlashingReceiver interface {
func (s *Service) ReceiveBlock(ctx context.Context, block interfaces.ReadOnlySignedBeaconBlock, blockRoot [32]byte, avs das.AvailabilityStore) error {
ctx, span := trace.StartSpan(ctx, "blockChain.ReceiveBlock")
defer span.End()
if blockRoot == badHoleskyRoot {
return errHoleskyForbiddenRoot
}
// Return early if the block has been synced
if s.InForkchoice(blockRoot) {
log.WithField("blockRoot", fmt.Sprintf("%#x", blockRoot)).Debug("Ignoring already synced block")
@@ -279,9 +284,10 @@ func (s *Service) executePostFinalizationTasks(ctx context.Context, finalizedSta
go func() {
s.sendNewFinalizedEvent(ctx, finalizedState)
}()
depCtx, cancel := context.WithTimeout(context.Background(), depositDeadline)
go func() {
s.insertFinalizedDeposits(depCtx, finalized.Root)
s.insertFinalizedDepositsAndPrune(depCtx, finalized.Root)
cancel()
}()
}

View File

@@ -455,41 +455,81 @@ func Test_executePostFinalizationTasks(t *testing.T) {
Root: headRoot[:],
}))
require.NoError(t, headState.SetGenesisValidatorsRoot(params.BeaconConfig().ZeroHash[:]))
t.Run("pre deposit request", func(t *testing.T) {
require.NoError(t, headState.SetEth1DepositIndex(1))
s, tr := minimalTestService(t, WithFinalizedStateAtStartUp(headState))
ctx, beaconDB, stateGen := tr.ctx, tr.db, tr.sg
s, tr := minimalTestService(t, WithFinalizedStateAtStartUp(headState))
ctx, beaconDB, stateGen := tr.ctx, tr.db, tr.sg
require.NoError(t, beaconDB.SaveGenesisBlockRoot(ctx, genesisRoot))
util.SaveBlock(t, ctx, beaconDB, genesis)
require.NoError(t, beaconDB.SaveState(ctx, headState, headRoot))
require.NoError(t, beaconDB.SaveState(ctx, headState, genesisRoot))
util.SaveBlock(t, ctx, beaconDB, headBlock)
require.NoError(t, beaconDB.SaveFinalizedCheckpoint(ctx, &ethpb.Checkpoint{Epoch: slots.ToEpoch(finalizedSlot), Root: headRoot[:]}))
require.NoError(t, beaconDB.SaveGenesisBlockRoot(ctx, genesisRoot))
util.SaveBlock(t, ctx, beaconDB, genesis)
require.NoError(t, beaconDB.SaveState(ctx, headState, headRoot))
require.NoError(t, beaconDB.SaveState(ctx, headState, genesisRoot))
util.SaveBlock(t, ctx, beaconDB, headBlock)
require.NoError(t, beaconDB.SaveFinalizedCheckpoint(ctx, &ethpb.Checkpoint{Epoch: slots.ToEpoch(finalizedSlot), Root: headRoot[:]}))
require.NoError(t, err)
require.NoError(t, stateGen.SaveState(ctx, headRoot, headState))
require.NoError(t, beaconDB.SaveLastValidatedCheckpoint(ctx, &ethpb.Checkpoint{Epoch: slots.ToEpoch(finalizedSlot), Root: headRoot[:]}))
require.NoError(t, err)
require.NoError(t, stateGen.SaveState(ctx, headRoot, headState))
require.NoError(t, beaconDB.SaveLastValidatedCheckpoint(ctx, &ethpb.Checkpoint{Epoch: slots.ToEpoch(finalizedSlot), Root: headRoot[:]}))
notifier := &blockchainTesting.MockStateNotifier{RecordEvents: true}
s.cfg.StateNotifier = notifier
s.executePostFinalizationTasks(s.ctx, headState)
notifier := &blockchainTesting.MockStateNotifier{RecordEvents: true}
s.cfg.StateNotifier = notifier
s.executePostFinalizationTasks(s.ctx, headState)
time.Sleep(1 * time.Second) // sleep for a second because event is in a separate go routine
require.Equal(t, 1, len(notifier.ReceivedEvents()))
e := notifier.ReceivedEvents()[0]
assert.Equal(t, statefeed.FinalizedCheckpoint, int(e.Type))
fc, ok := e.Data.(*ethpbv1.EventFinalizedCheckpoint)
require.Equal(t, true, ok, "event has wrong data type")
assert.Equal(t, primitives.Epoch(123), fc.Epoch)
assert.DeepEqual(t, headRoot[:], fc.Block)
assert.DeepEqual(t, finalizedStRoot[:], fc.State)
assert.Equal(t, false, fc.ExecutionOptimistic)
time.Sleep(1 * time.Second) // sleep for a second because event is in a separate go routine
require.Equal(t, 1, len(notifier.ReceivedEvents()))
e := notifier.ReceivedEvents()[0]
assert.Equal(t, statefeed.FinalizedCheckpoint, int(e.Type))
fc, ok := e.Data.(*ethpbv1.EventFinalizedCheckpoint)
require.Equal(t, true, ok, "event has wrong data type")
assert.Equal(t, primitives.Epoch(123), fc.Epoch)
assert.DeepEqual(t, headRoot[:], fc.Block)
assert.DeepEqual(t, finalizedStRoot[:], fc.State)
assert.Equal(t, false, fc.ExecutionOptimistic)
// check the cache
index, ok := headState.ValidatorIndexByPubkey(bytesutil.ToBytes48(key))
require.Equal(t, true, ok)
require.Equal(t, primitives.ValidatorIndex(0), index) // first index
// check the cache
index, ok := headState.ValidatorIndexByPubkey(bytesutil.ToBytes48(key))
require.Equal(t, true, ok)
require.Equal(t, primitives.ValidatorIndex(0), index) // first index
// check deposit
require.LogsContain(t, logHook, "Finalized deposit insertion completed at index")
})
t.Run("deposit requests started", func(t *testing.T) {
require.NoError(t, headState.SetEth1DepositIndex(1))
require.NoError(t, headState.SetDepositRequestsStartIndex(1))
s, tr := minimalTestService(t, WithFinalizedStateAtStartUp(headState))
ctx, beaconDB, stateGen := tr.ctx, tr.db, tr.sg
require.NoError(t, beaconDB.SaveGenesisBlockRoot(ctx, genesisRoot))
util.SaveBlock(t, ctx, beaconDB, genesis)
require.NoError(t, beaconDB.SaveState(ctx, headState, headRoot))
require.NoError(t, beaconDB.SaveState(ctx, headState, genesisRoot))
util.SaveBlock(t, ctx, beaconDB, headBlock)
require.NoError(t, beaconDB.SaveFinalizedCheckpoint(ctx, &ethpb.Checkpoint{Epoch: slots.ToEpoch(finalizedSlot), Root: headRoot[:]}))
require.NoError(t, err)
require.NoError(t, stateGen.SaveState(ctx, headRoot, headState))
require.NoError(t, beaconDB.SaveLastValidatedCheckpoint(ctx, &ethpb.Checkpoint{Epoch: slots.ToEpoch(finalizedSlot), Root: headRoot[:]}))
notifier := &blockchainTesting.MockStateNotifier{RecordEvents: true}
s.cfg.StateNotifier = notifier
s.executePostFinalizationTasks(s.ctx, headState)
time.Sleep(1 * time.Second) // sleep for a second because event is in a separate go routine
require.Equal(t, 1, len(notifier.ReceivedEvents()))
e := notifier.ReceivedEvents()[0]
assert.Equal(t, statefeed.FinalizedCheckpoint, int(e.Type))
fc, ok := e.Data.(*ethpbv1.EventFinalizedCheckpoint)
require.Equal(t, true, ok, "event has wrong data type")
assert.Equal(t, primitives.Epoch(123), fc.Epoch)
assert.DeepEqual(t, headRoot[:], fc.Block)
assert.DeepEqual(t, finalizedStRoot[:], fc.State)
assert.Equal(t, false, fc.ExecutionOptimistic)
// check the cache
index, ok := headState.ValidatorIndexByPubkey(bytesutil.ToBytes48(key))
require.Equal(t, true, ok)
require.Equal(t, primitives.ValidatorIndex(0), index) // first index
})
// check deposit
require.LogsContain(t, logHook, "Finalized deposit insertion completed at index")
}

View File

@@ -3,7 +3,6 @@
package blockchain
import (
"bytes"
"context"
"fmt"
"runtime"
@@ -23,7 +22,6 @@ import (
"github.com/prysmaticlabs/prysm/v5/beacon-chain/db/filesystem"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/execution"
f "github.com/prysmaticlabs/prysm/v5/beacon-chain/forkchoice"
forkchoicetypes "github.com/prysmaticlabs/prysm/v5/beacon-chain/forkchoice/types"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/operations/attestations"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/operations/blstoexec"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/operations/slashings"
@@ -32,7 +30,6 @@ import (
"github.com/prysmaticlabs/prysm/v5/beacon-chain/startup"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state/stategen"
"github.com/prysmaticlabs/prysm/v5/config/features"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
@@ -42,6 +39,7 @@ import (
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
prysmTime "github.com/prysmaticlabs/prysm/v5/time"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/sirupsen/logrus"
)
// Service represents a service that handles the internal
@@ -269,69 +267,18 @@ func (s *Service) StartFromSavedState(saved state.BeaconState) error {
return err
}
s.originBlockRoot = originRoot
if err := s.initializeHeadFromDB(s.ctx); err != nil {
return errors.Wrap(err, "could not set up chain info")
st, err := s.cfg.StateGen.Resume(s.ctx, s.cfg.FinalizedStateAtStartUp)
if err != nil {
return errors.Wrap(err, "could not get finalized state from db")
}
spawnCountdownIfPreGenesis(s.ctx, s.genesisTime, s.cfg.BeaconDB)
justified, err := s.cfg.BeaconDB.JustifiedCheckpoint(s.ctx)
if err != nil {
return errors.Wrap(err, "could not get justified checkpoint")
}
if justified == nil {
return errNilJustifiedCheckpoint
}
finalized, err := s.cfg.BeaconDB.FinalizedCheckpoint(s.ctx)
if err != nil {
return errors.Wrap(err, "could not get finalized checkpoint")
}
if finalized == nil {
return errNilFinalizedCheckpoint
}
fRoot := s.ensureRootNotZeros(bytesutil.ToBytes32(finalized.Root))
s.cfg.ForkChoiceStore.Lock()
defer s.cfg.ForkChoiceStore.Unlock()
if err := s.cfg.ForkChoiceStore.UpdateJustifiedCheckpoint(s.ctx, &forkchoicetypes.Checkpoint{Epoch: justified.Epoch,
Root: bytesutil.ToBytes32(justified.Root)}); err != nil {
return errors.Wrap(err, "could not update forkchoice's justified checkpoint")
}
if err := s.cfg.ForkChoiceStore.UpdateFinalizedCheckpoint(&forkchoicetypes.Checkpoint{Epoch: finalized.Epoch,
Root: bytesutil.ToBytes32(finalized.Root)}); err != nil {
return errors.Wrap(err, "could not update forkchoice's finalized checkpoint")
}
s.cfg.ForkChoiceStore.SetGenesisTime(uint64(s.genesisTime.Unix()))
st, err := s.cfg.StateGen.StateByRoot(s.ctx, fRoot)
if err != nil {
return errors.Wrap(err, "could not get finalized checkpoint state")
}
finalizedBlock, err := s.cfg.BeaconDB.Block(s.ctx, fRoot)
if err != nil {
return errors.Wrap(err, "could not get finalized checkpoint block")
}
roblock, err := blocks.NewROBlockWithRoot(finalizedBlock, fRoot)
if err != nil {
return err
}
if err := s.cfg.ForkChoiceStore.InsertNode(s.ctx, st, roblock); err != nil {
return errors.Wrap(err, "could not insert finalized block to forkchoice")
}
if !features.Get().EnableStartOptimistic {
lastValidatedCheckpoint, err := s.cfg.BeaconDB.LastValidatedCheckpoint(s.ctx)
if err != nil {
return errors.Wrap(err, "could not get last validated checkpoint")
}
if bytes.Equal(finalized.Root, lastValidatedCheckpoint.Root) {
if err := s.cfg.ForkChoiceStore.SetOptimisticToValid(s.ctx, fRoot); err != nil {
return errors.Wrap(err, "could not set finalized block as validated")
}
}
if err := s.setupForkchoice(st); err != nil {
return errors.Wrap(err, "could not set up forkchoice")
}
// not attempting to save initial sync blocks here, because there shouldn't be any until
// after the statefeed.Initialized event is fired (below)
if err := s.wsVerifier.VerifyWeakSubjectivity(s.ctx, finalized.Epoch); err != nil {
cp := s.FinalizedCheckpt()
if err := s.wsVerifier.VerifyWeakSubjectivity(s.ctx, cp.Epoch); err != nil {
// Exit run time if the node failed to verify weak subjectivity checkpoint.
return errors.Wrap(err, "could not verify initial checkpoint provided for chain sync")
}
@@ -340,7 +287,6 @@ func (s *Service) StartFromSavedState(saved state.BeaconState) error {
if err := s.clockSetter.SetClock(startup.NewClock(s.genesisTime, vr)); err != nil {
return errors.Wrap(err, "failed to initialize blockchain service")
}
return nil
}
@@ -370,46 +316,40 @@ func (s *Service) originRootFromSavedState(ctx context.Context) ([32]byte, error
return genesisBlkRoot, nil
}
// initializeHeadFromDB uses the finalized checkpoint and head block found in the database to set the current head.
// initializeHeadFromDB uses the finalized checkpoint and head block root from forkchoice to set the current head.
// Note that this may block until stategen replays blocks between the finalized and head blocks
// if the head sync flag was specified and the gap between the finalized and head blocks is at least 128 epochs long.
func (s *Service) initializeHeadFromDB(ctx context.Context) error {
finalized, err := s.cfg.BeaconDB.FinalizedCheckpoint(ctx)
if err != nil {
return errors.Wrap(err, "could not get finalized checkpoint from db")
}
if finalized == nil {
// This should never happen. At chain start, the finalized checkpoint
// would be the genesis state and block.
return errors.New("no finalized epoch in the database")
}
finalizedRoot := s.ensureRootNotZeros(bytesutil.ToBytes32(finalized.Root))
var finalizedState state.BeaconState
finalizedState, err = s.cfg.StateGen.Resume(ctx, s.cfg.FinalizedStateAtStartUp)
if err != nil {
return errors.Wrap(err, "could not get finalized state from db")
}
if finalizedState == nil || finalizedState.IsNil() {
func (s *Service) initializeHead(ctx context.Context, st state.BeaconState) error {
cp := s.FinalizedCheckpt()
fRoot := s.ensureRootNotZeros([32]byte(cp.Root))
if st == nil || st.IsNil() {
return errors.New("finalized state can't be nil")
}
finalizedBlock, err := s.getBlock(ctx, finalizedRoot)
s.cfg.ForkChoiceStore.RLock()
root := s.cfg.ForkChoiceStore.HighestReceivedBlockRoot()
s.cfg.ForkChoiceStore.RUnlock()
blk, err := s.cfg.BeaconDB.Block(ctx, root)
if err != nil {
return errors.Wrap(err, "could not get finalized block")
return errors.Wrap(err, "could not get head block")
}
if err := s.setHead(&head{
finalizedRoot,
finalizedBlock,
finalizedState,
finalizedBlock.Block().Slot(),
if root != fRoot {
st, err = s.cfg.StateGen.StateByRoot(ctx, root)
if err != nil {
return errors.Wrap(err, "could not get head state")
}
}
log.WithFields(logrus.Fields{
"root": fmt.Sprintf("%#x", root),
"slot": blk.Block().Slot(),
}).Info("Initialized head block from DB")
return errors.Wrap(s.setHead(&head{
root,
blk,
st,
blk.Block().Slot(),
false,
}); err != nil {
return errors.Wrap(err, "could not set head")
}
return nil
}), "could not set head")
}
func (s *Service) startFromExecutionChain() error {

View File

@@ -0,0 +1,187 @@
package blockchain
import (
"bytes"
"context"
"fmt"
"github.com/pkg/errors"
forkchoicetypes "github.com/prysmaticlabs/prysm/v5/beacon-chain/forkchoice/types"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/config/features"
"github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
"github.com/prysmaticlabs/prysm/v5/time/slots"
)
func (s *Service) setupForkchoice(st state.BeaconState) error {
if err := s.setupForkchoiceCheckpoints(); err != nil {
return errors.Wrap(err, "could not set up forkchoice checkpoints")
}
if err := s.setupForkchoiceTree(st); err != nil {
return errors.Wrap(err, "could not set up forkchoice root")
}
if err := s.initializeHead(s.ctx, st); err != nil {
return errors.Wrap(err, "could not initialize head from db")
}
return nil
}
func (s *Service) startupHeadRoot() [32]byte {
headStr := features.Get().ForceHead
cp := s.FinalizedCheckpt()
fRoot := s.ensureRootNotZeros([32]byte(cp.Root))
if headStr == "" {
return fRoot
}
if headStr == "head" {
root, err := s.cfg.BeaconDB.HeadBlockRoot()
if err != nil {
log.WithError(err).Error("could not get head block root, starting with finalized block as head")
return fRoot
}
log.Infof("Using Head root of %#x", root)
return root
}
root, err := bytesutil.DecodeHexWithLength(headStr, 32)
if err != nil {
log.WithError(err).Error("could not parse head root, starting with finalized block as head")
return fRoot
}
return [32]byte(root)
}
func (s *Service) setupForkchoiceTree(st state.BeaconState) error {
headRoot := s.startupHeadRoot()
cp := s.FinalizedCheckpt()
fRoot := s.ensureRootNotZeros([32]byte(cp.Root))
if err := s.setupForkchoiceRoot(st); err != nil {
return errors.Wrap(err, "could not set up forkchoice root")
}
fBlk, err := s.cfg.BeaconDB.Block(s.ctx, fRoot)
if err != nil {
return errors.Wrap(err, "could not get finalized block")
}
if err := s.setHead(&head{
fRoot,
fBlk,
st,
fBlk.Block().Slot(),
false,
}); err != nil {
return errors.Wrap(err, "could not set head")
}
if headRoot == fRoot {
return nil
}
blk, err := s.cfg.BeaconDB.Block(s.ctx, headRoot)
if err != nil {
log.WithError(err).Error("could not get head block, starting with finalized block as head")
return nil
}
if slots.ToEpoch(blk.Block().Slot()) < cp.Epoch {
log.WithField("headRoot", fmt.Sprintf("%#x", headRoot)).Error("head block is older than finalized block, starting with finalized block as head")
return nil
}
chain, err := s.buildForkchoiceChain(s.ctx, blk)
if err != nil {
log.WithError(err).Error("could not build forkchoice chain, starting with finalized block as head")
return nil
}
s.cfg.ForkChoiceStore.Lock()
defer s.cfg.ForkChoiceStore.Unlock()
return s.cfg.ForkChoiceStore.InsertChain(s.ctx, chain)
}
func (s *Service) buildForkchoiceChain(ctx context.Context, head interfaces.ReadOnlySignedBeaconBlock) ([]*forkchoicetypes.BlockAndCheckpoints, error) {
chain := []*forkchoicetypes.BlockAndCheckpoints{}
cp := s.FinalizedCheckpt()
fRoot := s.ensureRootNotZeros([32]byte(cp.Root))
jp := s.CurrentJustifiedCheckpt()
root, err := head.Block().HashTreeRoot()
if err != nil {
return nil, errors.Wrap(err, "could not get head block root")
}
for {
roblock, err := blocks.NewROBlockWithRoot(head, root)
if err != nil {
return nil, err
}
// This chain sets the justified checkpoint for every block, including some that are older than jp.
// This should be however safe for forkchoice at startup.
chain = append(chain, &forkchoicetypes.BlockAndCheckpoints{Block: roblock, JustifiedCheckpoint: jp, FinalizedCheckpoint: cp})
root = head.Block().ParentRoot()
if root == fRoot {
break
}
head, err = s.cfg.BeaconDB.Block(s.ctx, root)
if err != nil {
return nil, errors.Wrap(err, "could not get block")
}
if slots.ToEpoch(head.Block().Slot()) < cp.Epoch {
return nil, errors.New("head block is not a descendant of the finalized checkpoint")
}
}
return chain, nil
}
func (s *Service) setupForkchoiceRoot(st state.BeaconState) error {
cp := s.FinalizedCheckpt()
fRoot := s.ensureRootNotZeros([32]byte(cp.Root))
finalizedBlock, err := s.cfg.BeaconDB.Block(s.ctx, fRoot)
if err != nil {
return errors.Wrap(err, "could not get finalized checkpoint block")
}
roblock, err := blocks.NewROBlockWithRoot(finalizedBlock, fRoot)
if err != nil {
return err
}
if err := s.cfg.ForkChoiceStore.InsertNode(s.ctx, st, roblock); err != nil {
return errors.Wrap(err, "could not insert finalized block to forkchoice")
}
if !features.Get().EnableStartOptimistic {
lastValidatedCheckpoint, err := s.cfg.BeaconDB.LastValidatedCheckpoint(s.ctx)
if err != nil {
return errors.Wrap(err, "could not get last validated checkpoint")
}
if bytes.Equal(fRoot[:], lastValidatedCheckpoint.Root) {
if err := s.cfg.ForkChoiceStore.SetOptimisticToValid(s.ctx, fRoot); err != nil {
return errors.Wrap(err, "could not set finalized block as validated")
}
}
}
return nil
}
func (s *Service) setupForkchoiceCheckpoints() error {
justified, err := s.cfg.BeaconDB.JustifiedCheckpoint(s.ctx)
if err != nil {
return errors.Wrap(err, "could not get justified checkpoint")
}
if justified == nil {
return errNilJustifiedCheckpoint
}
finalized, err := s.cfg.BeaconDB.FinalizedCheckpoint(s.ctx)
if err != nil {
return errors.Wrap(err, "could not get finalized checkpoint")
}
if finalized == nil {
return errNilFinalizedCheckpoint
}
fRoot := s.ensureRootNotZeros(bytesutil.ToBytes32(finalized.Root))
s.cfg.ForkChoiceStore.Lock()
defer s.cfg.ForkChoiceStore.Unlock()
if err := s.cfg.ForkChoiceStore.UpdateJustifiedCheckpoint(s.ctx, &forkchoicetypes.Checkpoint{Epoch: justified.Epoch,
Root: bytesutil.ToBytes32(justified.Root)}); err != nil {
return errors.Wrap(err, "could not update forkchoice's justified checkpoint")
}
if err := s.cfg.ForkChoiceStore.UpdateFinalizedCheckpoint(&forkchoicetypes.Checkpoint{Epoch: finalized.Epoch,
Root: fRoot}); err != nil {
return errors.Wrap(err, "could not update forkchoice's finalized checkpoint")
}
s.cfg.ForkChoiceStore.SetGenesisTime(uint64(s.genesisTime.Unix()))
return nil
}

View File

@@ -84,6 +84,7 @@ go_test(
"sync_committee_head_state_test.go",
"sync_committee_test.go",
"sync_subnet_ids_test.go",
"tracked_validators_test.go",
],
embed = [":go_default_library"],
deps = [

View File

@@ -5,6 +5,7 @@ go_library(
srcs = [
"deposit_fetcher.go",
"deposit_inserter.go",
"deposit_pruner.go",
"deposit_tree.go",
"deposit_tree_snapshot.go",
"merkle_tree.go",
@@ -35,6 +36,7 @@ go_test(
srcs = [
"deposit_cache_test.go",
"deposit_fetcher_test.go",
"deposit_pruner_test.go",
"deposit_tree_snapshot_test.go",
"merkle_tree_test.go",
"spec_test.go",

View File

@@ -903,189 +903,6 @@ func TestMin(t *testing.T) {
}
func TestPruneProofs_Ok(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 1))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.NotNil(t, dc.deposits[2].Deposit.Proof)
assert.NotNil(t, dc.deposits[3].Deposit.Proof)
}
func TestPruneProofs_SomeAlreadyPruned(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: nil, Data: &ethpb.Deposit_Data{
PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: nil, Data: &ethpb.Deposit_Data{
PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}}, index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(), Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 2))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
}
func TestPruneProofs_PruneAllWhenDepositIndexTooBig(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 99))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[3].Deposit.Proof)
}
func TestPruneProofs_CorrectlyHandleLastIndex(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 4))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[3].Deposit.Proof)
}
func TestDepositMap_WorksCorrectly(t *testing.T) {
dc, err := New()
require.NoError(t, err)

View File

@@ -178,52 +178,6 @@ func (c *Cache) NonFinalizedDeposits(ctx context.Context, lastFinalizedIndex int
return deposits
}
// PruneProofs removes proofs from all deposits whose index is equal or less than untilDepositIndex.
func (c *Cache) PruneProofs(ctx context.Context, untilDepositIndex int64) error {
_, span := trace.StartSpan(ctx, "Cache.PruneProofs")
defer span.End()
c.depositsLock.Lock()
defer c.depositsLock.Unlock()
if untilDepositIndex >= int64(len(c.deposits)) {
untilDepositIndex = int64(len(c.deposits) - 1)
}
for i := untilDepositIndex; i >= 0; i-- {
// Finding a nil proof means that all proofs up to this deposit have been already pruned.
if c.deposits[i].Deposit.Proof == nil {
break
}
c.deposits[i].Deposit.Proof = nil
}
return nil
}
// PrunePendingDeposits removes any deposit which is older than the given deposit merkle tree index.
func (c *Cache) PrunePendingDeposits(ctx context.Context, merkleTreeIndex int64) {
_, span := trace.StartSpan(ctx, "Cache.PrunePendingDeposits")
defer span.End()
if merkleTreeIndex == 0 {
log.Debug("Ignoring 0 deposit removal")
return
}
c.depositsLock.Lock()
defer c.depositsLock.Unlock()
cleanDeposits := make([]*ethpb.DepositContainer, 0, len(c.pendingDeposits))
for _, dp := range c.pendingDeposits {
if dp.Index >= merkleTreeIndex {
cleanDeposits = append(cleanDeposits, dp)
}
}
c.pendingDeposits = cleanDeposits
pendingDepositsCount.Set(float64(len(c.pendingDeposits)))
}
// InsertPendingDeposit into the database. If deposit or block number are nil
// then this method does nothing.
func (c *Cache) InsertPendingDeposit(ctx context.Context, d *ethpb.Deposit, blockNum uint64, index int64, depositRoot [32]byte) {

View File

@@ -44,67 +44,3 @@ func TestPendingDeposits_OK(t *testing.T) {
all := dc.PendingDeposits(context.Background(), nil)
assert.Equal(t, len(dc.pendingDeposits), len(all), "PendingDeposits(ctx, nil) did not return all deposits")
}
func TestPrunePendingDeposits_ZeroMerkleIndex(t *testing.T) {
dc := Cache{}
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PrunePendingDeposits(context.Background(), 0)
expected := []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
assert.DeepEqual(t, expected, dc.pendingDeposits)
}
func TestPrunePendingDeposits_OK(t *testing.T) {
dc := Cache{}
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PrunePendingDeposits(context.Background(), 6)
expected := []*ethpb.DepositContainer{
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
assert.DeepEqual(t, expected, dc.pendingDeposits)
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PrunePendingDeposits(context.Background(), 10)
expected = []*ethpb.DepositContainer{
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
assert.DeepEqual(t, expected, dc.pendingDeposits)
}

View File

@@ -0,0 +1,88 @@
package depositsnapshot
import (
"context"
"github.com/prysmaticlabs/prysm/v5/monitoring/tracing/trace"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
)
// PruneProofs removes proofs from all deposits whose index is equal or less than untilDepositIndex.
func (c *Cache) PruneProofs(ctx context.Context, untilDepositIndex int64) error {
_, span := trace.StartSpan(ctx, "Cache.PruneProofs")
defer span.End()
c.depositsLock.Lock()
defer c.depositsLock.Unlock()
if untilDepositIndex >= int64(len(c.deposits)) {
untilDepositIndex = int64(len(c.deposits) - 1)
}
for i := untilDepositIndex; i >= 0; i-- {
// Finding a nil proof means that all proofs up to this deposit have been already pruned.
if c.deposits[i].Deposit.Proof == nil {
break
}
c.deposits[i].Deposit.Proof = nil
}
return nil
}
// PruneAllProofs removes proofs from all deposits.
// As EIP-6110 applies and the legacy deposit mechanism is deprecated,
// proofs in deposit snapshot are no longer needed.
// See: https://eips.ethereum.org/EIPS/eip-6110#eth1data-poll-deprecation
func (c *Cache) PruneAllProofs(ctx context.Context) {
_, span := trace.StartSpan(ctx, "Cache.PruneAllProofs")
defer span.End()
c.depositsLock.Lock()
defer c.depositsLock.Unlock()
for i := len(c.deposits) - 1; i >= 0; i-- {
if c.deposits[i].Deposit.Proof == nil {
break
}
c.deposits[i].Deposit.Proof = nil
}
}
// PrunePendingDeposits removes any deposit which is older than the given deposit merkle tree index.
func (c *Cache) PrunePendingDeposits(ctx context.Context, merkleTreeIndex int64) {
_, span := trace.StartSpan(ctx, "Cache.PrunePendingDeposits")
defer span.End()
if merkleTreeIndex == 0 {
log.Debug("Ignoring 0 deposit removal")
return
}
c.depositsLock.Lock()
defer c.depositsLock.Unlock()
cleanDeposits := make([]*ethpb.DepositContainer, 0, len(c.pendingDeposits))
for _, dp := range c.pendingDeposits {
if dp.Index >= merkleTreeIndex {
cleanDeposits = append(cleanDeposits, dp)
}
}
c.pendingDeposits = cleanDeposits
pendingDepositsCount.Set(float64(len(c.pendingDeposits)))
}
// PruneAllPendingDeposits removes all pending deposits from the cache.
// As EIP-6110 applies and the legacy deposit mechanism is deprecated,
// pending deposits in deposit snapshot are no longer needed.
// See: https://eips.ethereum.org/EIPS/eip-6110#eth1data-poll-deprecation
func (c *Cache) PruneAllPendingDeposits(ctx context.Context) {
_, span := trace.StartSpan(ctx, "Cache.PruneAllPendingDeposits")
defer span.End()
c.depositsLock.Lock()
defer c.depositsLock.Unlock()
c.pendingDeposits = make([]*ethpb.DepositContainer, 0)
pendingDepositsCount.Set(float64(0))
}

View File

@@ -0,0 +1,323 @@
package depositsnapshot
import (
"context"
"testing"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/testing/assert"
"github.com/prysmaticlabs/prysm/v5/testing/require"
)
func TestPrunePendingDeposits_ZeroMerkleIndex(t *testing.T) {
dc := Cache{}
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PrunePendingDeposits(context.Background(), 0)
expected := []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
assert.DeepEqual(t, expected, dc.pendingDeposits)
}
func TestPrunePendingDeposits_OK(t *testing.T) {
dc := Cache{}
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PrunePendingDeposits(context.Background(), 6)
expected := []*ethpb.DepositContainer{
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
assert.DeepEqual(t, expected, dc.pendingDeposits)
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PrunePendingDeposits(context.Background(), 10)
expected = []*ethpb.DepositContainer{
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
assert.DeepEqual(t, expected, dc.pendingDeposits)
}
func TestPruneAllPendingDeposits(t *testing.T) {
dc := Cache{}
dc.pendingDeposits = []*ethpb.DepositContainer{
{Eth1BlockHeight: 2, Index: 2},
{Eth1BlockHeight: 4, Index: 4},
{Eth1BlockHeight: 6, Index: 6},
{Eth1BlockHeight: 8, Index: 8},
{Eth1BlockHeight: 10, Index: 10},
{Eth1BlockHeight: 12, Index: 12},
}
dc.PruneAllPendingDeposits(context.Background())
expected := []*ethpb.DepositContainer{}
assert.DeepEqual(t, expected, dc.pendingDeposits)
}
func TestPruneProofs_Ok(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 1))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.NotNil(t, dc.deposits[2].Deposit.Proof)
assert.NotNil(t, dc.deposits[3].Deposit.Proof)
}
func TestPruneProofs_SomeAlreadyPruned(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: nil, Data: &ethpb.Deposit_Data{
PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: nil, Data: &ethpb.Deposit_Data{
PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}}, index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(), Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 2))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
}
func TestPruneProofs_PruneAllWhenDepositIndexTooBig(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 99))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[3].Deposit.Proof)
}
func TestPruneProofs_CorrectlyHandleLastIndex(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
require.NoError(t, dc.PruneProofs(context.Background(), 4))
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[3].Deposit.Proof)
}
func TestPruneAllProofs(t *testing.T) {
dc, err := New()
require.NoError(t, err)
deposits := []struct {
blkNum uint64
deposit *ethpb.Deposit
index int64
}{
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk0"), 48)}},
index: 0,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk1"), 48)}},
index: 1,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk2"), 48)}},
index: 2,
},
{
blkNum: 0,
deposit: &ethpb.Deposit{Proof: makeDepositProof(),
Data: &ethpb.Deposit_Data{PublicKey: bytesutil.PadTo([]byte("pk3"), 48)}},
index: 3,
},
}
for _, ins := range deposits {
assert.NoError(t, dc.InsertDeposit(context.Background(), ins.deposit, ins.blkNum, ins.index, [32]byte{}))
}
dc.PruneAllProofs(context.Background())
assert.DeepEqual(t, [][]byte(nil), dc.deposits[0].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[1].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[2].Deposit.Proof)
assert.DeepEqual(t, [][]byte(nil), dc.deposits[3].Deposit.Proof)
}

View File

@@ -12,6 +12,7 @@ import (
type DepositCache interface {
DepositFetcher
DepositInserter
DepositPruner
}
// DepositFetcher defines a struct which can retrieve deposit information from a store.
@@ -23,8 +24,6 @@ type DepositFetcher interface {
InsertPendingDeposit(ctx context.Context, d *ethpb.Deposit, blockNum uint64, index int64, depositRoot [32]byte)
PendingDeposits(ctx context.Context, untilBlk *big.Int) []*ethpb.Deposit
PendingContainers(ctx context.Context, untilBlk *big.Int) []*ethpb.DepositContainer
PrunePendingDeposits(ctx context.Context, merkleTreeIndex int64)
PruneProofs(ctx context.Context, untilDepositIndex int64) error
FinalizedFetcher
}
@@ -42,6 +41,14 @@ type FinalizedFetcher interface {
NonFinalizedDeposits(ctx context.Context, lastFinalizedIndex int64, untilBlk *big.Int) []*ethpb.Deposit
}
// DepositPruner is an interface for pruning deposits and proofs.
type DepositPruner interface {
PrunePendingDeposits(ctx context.Context, merkleTreeIndex int64)
PruneAllPendingDeposits(ctx context.Context)
PruneProofs(ctx context.Context, untilDepositIndex int64) error
PruneAllProofs(ctx context.Context)
}
// FinalizedDeposits defines a method to access a merkle tree containing deposits and their indexes.
type FinalizedDeposits interface {
Deposits() MerkleTree

View File

@@ -1,49 +1,139 @@
package cache
import (
"sync"
"strconv"
"time"
"github.com/patrickmn/go-cache"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/sirupsen/logrus"
)
type TrackedValidator struct {
Active bool
FeeRecipient primitives.ExecutionAddress
Index primitives.ValidatorIndex
}
const (
defaultExpiration = 1 * time.Hour
cleanupInterval = 15 * time.Minute
)
type TrackedValidatorsCache struct {
sync.Mutex
trackedValidators map[primitives.ValidatorIndex]TrackedValidator
}
type (
TrackedValidator struct {
Active bool
FeeRecipient primitives.ExecutionAddress
Index primitives.ValidatorIndex
}
TrackedValidatorsCache struct {
trackedValidators cache.Cache
}
)
var (
// Metrics.
trackedValidatorsCacheMiss = promauto.NewCounter(prometheus.CounterOpts{
Name: "tracked_validators_cache_miss",
Help: "The number of tracked validators requests that are not present in the cache.",
})
trackedValidatorsCacheTotal = promauto.NewCounter(prometheus.CounterOpts{
Name: "tracked_validators_cache_total",
Help: "The total number of tracked validators requests in the cache.",
})
trackedValidatorsCacheCount = promauto.NewGauge(prometheus.GaugeOpts{
Name: "tracked_validators_cache_count",
Help: "The number of tracked validators in the cache.",
})
)
// NewTrackedValidatorsCache creates a new cache for tracking validators.
func NewTrackedValidatorsCache() *TrackedValidatorsCache {
return &TrackedValidatorsCache{
trackedValidators: make(map[primitives.ValidatorIndex]TrackedValidator),
trackedValidators: *cache.New(defaultExpiration, cleanupInterval),
}
}
// Validator retrieves a tracked validator from the cache (if present).
func (t *TrackedValidatorsCache) Validator(index primitives.ValidatorIndex) (TrackedValidator, bool) {
t.Lock()
defer t.Unlock()
val, ok := t.trackedValidators[index]
return val, ok
trackedValidatorsCacheTotal.Inc()
key := toCacheKey(index)
item, ok := t.trackedValidators.Get(key)
if !ok {
trackedValidatorsCacheMiss.Inc()
return TrackedValidator{}, false
}
val, ok := item.(TrackedValidator)
if !ok {
logrus.Errorf("Failed to cast tracked validator from cache, got unexpected item type %T", item)
return TrackedValidator{}, false
}
return val, true
}
// Set adds a tracked validator to the cache.
func (t *TrackedValidatorsCache) Set(val TrackedValidator) {
t.Lock()
defer t.Unlock()
t.trackedValidators[val.Index] = val
key := toCacheKey(val.Index)
t.trackedValidators.Set(key, val, cache.DefaultExpiration)
}
// Delete removes a tracked validator from the cache.
func (t *TrackedValidatorsCache) Prune() {
t.Lock()
defer t.Unlock()
t.trackedValidators = make(map[primitives.ValidatorIndex]TrackedValidator)
t.trackedValidators.Flush()
trackedValidatorsCacheCount.Set(0)
}
// Validating returns true if there are at least one tracked validators in the cache.
func (t *TrackedValidatorsCache) Validating() bool {
t.Lock()
defer t.Unlock()
return len(t.trackedValidators) > 0
count := t.trackedValidators.ItemCount()
trackedValidatorsCacheCount.Set(float64(count))
return count > 0
}
// ItemCount returns the number of tracked validators in the cache.
func (t *TrackedValidatorsCache) ItemCount() int {
count := t.trackedValidators.ItemCount()
trackedValidatorsCacheCount.Set(float64(count))
return count
}
// Indices returns a map of validator indices that are being tracked.
func (t *TrackedValidatorsCache) Indices() map[primitives.ValidatorIndex]bool {
items := t.trackedValidators.Items()
count := len(items)
trackedValidatorsCacheCount.Set(float64(count))
indices := make(map[primitives.ValidatorIndex]bool, count)
for cacheKey := range items {
index, err := fromCacheKey(cacheKey)
if err != nil {
logrus.WithError(err).Error("Failed to get validator index from cache key")
continue
}
indices[index] = true
}
return indices
}
// toCacheKey creates a cache key from the validator index.
func toCacheKey(validatorIndex primitives.ValidatorIndex) string {
return strconv.FormatUint(uint64(validatorIndex), 10)
}
// fromCacheKey gets the validator index from the cache key.
func fromCacheKey(key string) (primitives.ValidatorIndex, error) {
validatorIndex, err := strconv.ParseUint(key, 10, 64)
if err != nil {
return 0, errors.Wrapf(err, "parse Uint: %s", key)
}
return primitives.ValidatorIndex(validatorIndex), nil
}

View File

@@ -0,0 +1,79 @@
package cache
import (
"testing"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/testing/require"
)
func mapEqual(a, b map[primitives.ValidatorIndex]bool) bool {
if len(a) != len(b) {
return false
}
for k, v := range a {
if b[k] != v {
return false
}
}
return true
}
func TestTrackedValidatorsCache(t *testing.T) {
vc := NewTrackedValidatorsCache()
// No validators in cache.
require.Equal(t, 0, vc.ItemCount())
require.Equal(t, false, vc.Validating())
require.Equal(t, 0, len(vc.Indices()))
_, ok := vc.Validator(41)
require.Equal(t, false, ok)
// Add some validators (one twice).
v42Expected := TrackedValidator{Active: true, FeeRecipient: [20]byte{1}, Index: 42}
v43Expected := TrackedValidator{Active: false, FeeRecipient: [20]byte{2}, Index: 43}
vc.Set(v42Expected)
vc.Set(v43Expected)
vc.Set(v42Expected)
// Check if they are in the cache.
v42Actual, ok := vc.Validator(42)
require.Equal(t, true, ok)
require.Equal(t, v42Expected, v42Actual)
v43Actual, ok := vc.Validator(43)
require.Equal(t, true, ok)
require.Equal(t, v43Expected, v43Actual)
expected := map[primitives.ValidatorIndex]bool{42: true, 43: true}
actual := vc.Indices()
require.Equal(t, true, mapEqual(expected, actual))
// Check the item count and if the cache is validating.
require.Equal(t, 2, vc.ItemCount())
require.Equal(t, true, vc.Validating())
// Check if a non-existing validator is in the cache.
_, ok = vc.Validator(41)
require.Equal(t, false, ok)
// Prune the cache and test it.
vc.Prune()
_, ok = vc.Validator(41)
require.Equal(t, false, ok)
_, ok = vc.Validator(42)
require.Equal(t, false, ok)
_, ok = vc.Validator(43)
require.Equal(t, false, ok)
require.Equal(t, 0, vc.ItemCount())
require.Equal(t, false, vc.Validating())
require.Equal(t, 0, len(vc.Indices()))
}

View File

@@ -177,9 +177,9 @@ func TestComputeConsolidationEpochAndUpdateChurn(t *testing.T) {
require.NoError(t, err)
return s
}(t),
consolidationBalance: helpers.ConsolidationChurnLimit(32000000000000000)+1,
consolidationBalance: helpers.ConsolidationChurnLimit(32000000000000000) + 1,
expectedEpoch: 18, // Flows into another epoch.
expectedConsolidationBalanceToConsume: helpers.ConsolidationChurnLimit(32000000000000000)-1,
expectedConsolidationBalanceToConsume: helpers.ConsolidationChurnLimit(32000000000000000) - 1,
},
}

View File

@@ -102,7 +102,7 @@ func UpgradeToFulu(beaconState state.BeaconState) (state.BeaconState, error) {
return nil, err
}
s := &ethpb.BeaconStateFulu{
s := &ethpb.BeaconStateElectra{
GenesisTime: beaconState.GenesisTime(),
GenesisValidatorsRoot: beaconState.GenesisValidatorsRoot(),
Slot: beaconState.Slot(),

View File

@@ -22,6 +22,15 @@ import (
)
func TestLightClient_NewLightClientOptimisticUpdateFromBeaconState(t *testing.T) {
params.SetupTestConfigCleanup(t)
cfg := params.BeaconConfig()
cfg.AltairForkEpoch = 1
cfg.BellatrixForkEpoch = 2
cfg.CapellaForkEpoch = 3
cfg.DenebForkEpoch = 4
cfg.ElectraForkEpoch = 5
params.OverrideBeaconConfig(cfg)
t.Run("Altair", func(t *testing.T) {
l := util.NewTestLightClient(t).SetupTestAltair()
@@ -59,9 +68,31 @@ func TestLightClient_NewLightClientOptimisticUpdateFromBeaconState(t *testing.T)
l.CheckSyncAggregate(update.SyncAggregate())
l.CheckAttestedHeader(update.AttestedHeader())
})
t.Run("Electra", func(t *testing.T) {
l := util.NewTestLightClient(t).SetupTestElectra(false)
update, err := lightClient.NewLightClientOptimisticUpdateFromBeaconState(l.Ctx, l.State.Slot(), l.State, l.Block, l.AttestedState, l.AttestedBlock)
require.NoError(t, err)
require.NotNil(t, update, "update is nil")
require.Equal(t, l.Block.Block().Slot(), update.SignatureSlot(), "Signature slot is not equal")
l.CheckSyncAggregate(update.SyncAggregate())
l.CheckAttestedHeader(update.AttestedHeader())
})
}
func TestLightClient_NewLightClientFinalityUpdateFromBeaconState(t *testing.T) {
params.SetupTestConfigCleanup(t)
cfg := params.BeaconConfig()
cfg.AltairForkEpoch = 1
cfg.BellatrixForkEpoch = 2
cfg.CapellaForkEpoch = 3
cfg.DenebForkEpoch = 4
cfg.ElectraForkEpoch = 5
params.OverrideBeaconConfig(cfg)
t.Run("Altair", func(t *testing.T) {
l := util.NewTestLightClient(t).SetupTestAltair()
@@ -356,6 +387,157 @@ func TestLightClient_NewLightClientFinalityUpdateFromBeaconState(t *testing.T) {
require.DeepSSZEqual(t, execution, updateExecution.Proto(), "Finalized Block Execution is not equal")
})
})
t.Run("Electra", func(t *testing.T) {
t.Run("FinalizedBlock Not Nil", func(t *testing.T) {
l := util.NewTestLightClient(t).SetupTestElectra(false)
update, err := lightClient.NewLightClientFinalityUpdateFromBeaconState(l.Ctx, l.State.Slot(), l.State, l.Block, l.AttestedState, l.AttestedBlock, l.FinalizedBlock)
require.NoError(t, err)
require.NotNil(t, update, "update is nil")
require.Equal(t, l.Block.Block().Slot(), update.SignatureSlot(), "Signature slot is not equal")
l.CheckSyncAggregate(update.SyncAggregate())
l.CheckAttestedHeader(update.AttestedHeader())
//zeroHash := params.BeaconConfig().ZeroHash[:]
finalizedBlockHeader, err := l.FinalizedBlock.Header()
require.NoError(t, err)
require.NotNil(t, update.FinalizedHeader(), "Finalized header is nil")
updateFinalizedHeaderBeacon := update.FinalizedHeader().Beacon()
require.Equal(t, finalizedBlockHeader.Header.Slot, updateFinalizedHeaderBeacon.Slot, "Finalized header slot is not equal")
require.Equal(t, finalizedBlockHeader.Header.ProposerIndex, updateFinalizedHeaderBeacon.ProposerIndex, "Finalized header proposer index is not equal")
require.DeepSSZEqual(t, finalizedBlockHeader.Header.ParentRoot, updateFinalizedHeaderBeacon.ParentRoot, "Finalized header parent root is not equal")
require.DeepSSZEqual(t, finalizedBlockHeader.Header.StateRoot, updateFinalizedHeaderBeacon.StateRoot, "Finalized header state root is not equal")
require.DeepSSZEqual(t, finalizedBlockHeader.Header.BodyRoot, updateFinalizedHeaderBeacon.BodyRoot, "Finalized header body root is not equal")
fb, err := update.FinalityBranchElectra()
require.NoError(t, err)
proof, err := l.AttestedState.FinalizedRootProof(l.Ctx)
require.NoError(t, err)
for i, leaf := range fb {
require.DeepSSZEqual(t, proof[i], leaf[:], "Leaf is not equal")
}
// Check Execution BlockHash
payloadInterface, err := l.FinalizedBlock.Block().Body().Execution()
require.NoError(t, err)
transactionsRoot, err := payloadInterface.TransactionsRoot()
if errors.Is(err, consensustypes.ErrUnsupportedField) {
transactions, err := payloadInterface.Transactions()
require.NoError(t, err)
transactionsRootArray, err := ssz.TransactionsRoot(transactions)
require.NoError(t, err)
transactionsRoot = transactionsRootArray[:]
} else {
require.NoError(t, err)
}
withdrawalsRoot, err := payloadInterface.WithdrawalsRoot()
if errors.Is(err, consensustypes.ErrUnsupportedField) {
withdrawals, err := payloadInterface.Withdrawals()
require.NoError(t, err)
withdrawalsRootArray, err := ssz.WithdrawalSliceRoot(withdrawals, fieldparams.MaxWithdrawalsPerPayload)
require.NoError(t, err)
withdrawalsRoot = withdrawalsRootArray[:]
} else {
require.NoError(t, err)
}
execution := &v11.ExecutionPayloadHeaderDeneb{
ParentHash: payloadInterface.ParentHash(),
FeeRecipient: payloadInterface.FeeRecipient(),
StateRoot: payloadInterface.StateRoot(),
ReceiptsRoot: payloadInterface.ReceiptsRoot(),
LogsBloom: payloadInterface.LogsBloom(),
PrevRandao: payloadInterface.PrevRandao(),
BlockNumber: payloadInterface.BlockNumber(),
GasLimit: payloadInterface.GasLimit(),
GasUsed: payloadInterface.GasUsed(),
Timestamp: payloadInterface.Timestamp(),
ExtraData: payloadInterface.ExtraData(),
BaseFeePerGas: payloadInterface.BaseFeePerGas(),
BlockHash: payloadInterface.BlockHash(),
TransactionsRoot: transactionsRoot,
WithdrawalsRoot: withdrawalsRoot,
}
updateExecution, err := update.FinalizedHeader().Execution()
require.NoError(t, err)
require.DeepSSZEqual(t, execution, updateExecution.Proto(), "Finalized Block Execution is not equal")
})
t.Run("FinalizedBlock In Previous Fork", func(t *testing.T) {
l := util.NewTestLightClient(t).SetupTestElectraFinalizedBlockDeneb(false)
update, err := lightClient.NewLightClientFinalityUpdateFromBeaconState(l.Ctx, l.State.Slot(), l.State, l.Block, l.AttestedState, l.AttestedBlock, l.FinalizedBlock)
require.NoError(t, err)
require.NotNil(t, update, "update is nil")
require.Equal(t, l.Block.Block().Slot(), update.SignatureSlot(), "Signature slot is not equal")
l.CheckSyncAggregate(update.SyncAggregate())
l.CheckAttestedHeader(update.AttestedHeader())
finalizedBlockHeader, err := l.FinalizedBlock.Header()
require.NoError(t, err)
require.NotNil(t, update.FinalizedHeader(), "Finalized header is nil")
updateFinalizedHeaderBeacon := update.FinalizedHeader().Beacon()
require.Equal(t, finalizedBlockHeader.Header.Slot, updateFinalizedHeaderBeacon.Slot, "Finalized header slot is not equal")
require.Equal(t, finalizedBlockHeader.Header.ProposerIndex, updateFinalizedHeaderBeacon.ProposerIndex, "Finalized header proposer index is not equal")
require.DeepSSZEqual(t, finalizedBlockHeader.Header.ParentRoot, updateFinalizedHeaderBeacon.ParentRoot, "Finalized header parent root is not equal")
require.DeepSSZEqual(t, finalizedBlockHeader.Header.StateRoot, updateFinalizedHeaderBeacon.StateRoot, "Finalized header state root is not equal")
require.DeepSSZEqual(t, finalizedBlockHeader.Header.BodyRoot, updateFinalizedHeaderBeacon.BodyRoot, "Finalized header body root is not equal")
fb, err := update.FinalityBranchElectra()
require.NoError(t, err)
proof, err := l.AttestedState.FinalizedRootProof(l.Ctx)
require.NoError(t, err)
for i, leaf := range fb {
require.DeepSSZEqual(t, proof[i], leaf[:], "Leaf is not equal")
}
// Check Execution BlockHash
payloadInterface, err := l.FinalizedBlock.Block().Body().Execution()
require.NoError(t, err)
transactionsRoot, err := payloadInterface.TransactionsRoot()
if errors.Is(err, consensustypes.ErrUnsupportedField) {
transactions, err := payloadInterface.Transactions()
require.NoError(t, err)
transactionsRootArray, err := ssz.TransactionsRoot(transactions)
require.NoError(t, err)
transactionsRoot = transactionsRootArray[:]
} else {
require.NoError(t, err)
}
withdrawalsRoot, err := payloadInterface.WithdrawalsRoot()
if errors.Is(err, consensustypes.ErrUnsupportedField) {
withdrawals, err := payloadInterface.Withdrawals()
require.NoError(t, err)
withdrawalsRootArray, err := ssz.WithdrawalSliceRoot(withdrawals, fieldparams.MaxWithdrawalsPerPayload)
require.NoError(t, err)
withdrawalsRoot = withdrawalsRootArray[:]
} else {
require.NoError(t, err)
}
execution := &v11.ExecutionPayloadHeaderDeneb{
ParentHash: payloadInterface.ParentHash(),
FeeRecipient: payloadInterface.FeeRecipient(),
StateRoot: payloadInterface.StateRoot(),
ReceiptsRoot: payloadInterface.ReceiptsRoot(),
LogsBloom: payloadInterface.LogsBloom(),
PrevRandao: payloadInterface.PrevRandao(),
BlockNumber: payloadInterface.BlockNumber(),
GasLimit: payloadInterface.GasLimit(),
GasUsed: payloadInterface.GasUsed(),
Timestamp: payloadInterface.Timestamp(),
ExtraData: payloadInterface.ExtraData(),
BaseFeePerGas: payloadInterface.BaseFeePerGas(),
BlockHash: payloadInterface.BlockHash(),
TransactionsRoot: transactionsRoot,
WithdrawalsRoot: withdrawalsRoot,
}
updateExecution, err := update.FinalizedHeader().Execution()
require.NoError(t, err)
require.DeepSSZEqual(t, execution, updateExecution.Proto(), "Finalized Block Execution is not equal")
})
})
}
func TestLightClient_BlockToLightClientHeader(t *testing.T) {

View File

@@ -94,14 +94,7 @@ func (s *LazilyPersistentStore) IsDataAvailable(ctx context.Context, current pri
entry := s.cache.ensure(key)
defer s.cache.delete(key)
root := b.Root()
sumz, err := s.store.WaitForSummarizer(ctx)
if err != nil {
log.WithField("root", fmt.Sprintf("%#x", b.Root())).
WithError(err).
Debug("Failed to receive BlobStorageSummarizer within IsDataAvailable")
} else {
entry.setDiskSummary(sumz.Summary(root))
}
entry.setDiskSummary(s.store.Summary(root))
// Verify we have all the expected sidecars, and fail fast if any are missing or inconsistent.
// We don't try to salvage problematic batches because this indicates a misbehaving peer and we'd rather

View File

@@ -5,6 +5,10 @@ go_library(
srcs = [
"blob.go",
"cache.go",
"iteration.go",
"layout.go",
"layout_by_epoch.go",
"layout_flat.go",
"log.go",
"metrics.go",
"mock.go",
@@ -13,6 +17,7 @@ go_library(
importpath = "github.com/prysmaticlabs/prysm/v5/beacon-chain/db/filesystem",
visibility = ["//visibility:public"],
deps = [
"//beacon-chain/db:go_default_library",
"//beacon-chain/verification:go_default_library",
"//config/fieldparams:go_default_library",
"//config/params:go_default_library",
@@ -20,7 +25,6 @@ go_library(
"//consensus-types/primitives:go_default_library",
"//encoding/bytesutil:go_default_library",
"//io/file:go_default_library",
"//proto/prysm/v1alpha1:go_default_library",
"//runtime/logging:go_default_library",
"//time/slots:go_default_library",
"@com_github_ethereum_go_ethereum//common/hexutil:go_default_library",
@@ -37,10 +41,14 @@ go_test(
srcs = [
"blob_test.go",
"cache_test.go",
"iteration_test.go",
"layout_test.go",
"migration_test.go",
"pruner_test.go",
],
embed = [":go_default_library"],
deps = [
"//beacon-chain/db:go_default_library",
"//beacon-chain/verification:go_default_library",
"//config/params:go_default_library",
"//consensus-types/primitives:go_default_library",
@@ -48,6 +56,7 @@ go_test(
"//proto/prysm/v1alpha1:go_default_library",
"//testing/require:go_default_library",
"//testing/util:go_default_library",
"//time/slots:go_default_library",
"@com_github_prysmaticlabs_fastssz//:go_default_library",
"@com_github_spf13_afero//:go_default_library",
],

View File

@@ -1,42 +1,31 @@
package filesystem
import (
"context"
"fmt"
"math"
"os"
"path"
"strconv"
"strings"
"time"
"github.com/ethereum/go-ethereum/common/hexutil"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/verification"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
"github.com/prysmaticlabs/prysm/v5/io/file"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/runtime/logging"
"github.com/sirupsen/logrus"
"github.com/spf13/afero"
)
func directoryPermissions() os.FileMode {
return params.BeaconIoConfig().ReadWriteExecutePermissions
}
var (
errIndexOutOfBounds = errors.New("blob index in file name >= DeprecatedMaxBlobsPerBlock")
errEmptyBlobWritten = errors.New("zero bytes written to disk when saving blob sidecar")
errIndexOutOfBounds = errors.New("blob index in file name >= MAX_BLOBS_PER_BLOCK")
errSidecarEmptySSZData = errors.New("sidecar marshalled to an empty ssz byte slice")
errNoBasePath = errors.New("BlobStorage base path not specified in init")
errInvalidRootString = errors.New("Could not parse hex string as a [32]byte")
)
const (
sszExt = "ssz"
partExt = "part"
directoryPermissions = 0700
)
// BlobStorageOption is a functional option for configuring a BlobStorage.
@@ -66,6 +55,23 @@ func WithSaveFsync(fsync bool) BlobStorageOption {
}
}
// WithFs allows the afero.Fs implementation to be customized. Used by tests
// to substitute an in-memory filesystem.
func WithFs(fs afero.Fs) BlobStorageOption {
return func(b *BlobStorage) error {
b.fs = fs
return nil
}
}
// WithLayout enables the user to specify which layout scheme to use, dictating how blob files are stored on disk.
func WithLayout(name string) BlobStorageOption {
return func(b *BlobStorage) error {
b.layoutName = name
return nil
}
}
// NewBlobStorage creates a new instance of the BlobStorage object. Note that the implementation of BlobStorage may
// attempt to hold a file lock to guarantee exclusive control of the blob storage directory, so this should only be
// initialized once per beacon node.
@@ -76,19 +82,27 @@ func NewBlobStorage(opts ...BlobStorageOption) (*BlobStorage, error) {
return nil, errors.Wrap(err, "failed to create blob storage")
}
}
if b.base == "" {
return nil, errNoBasePath
// Allow tests to set up a different fs using WithFs.
if b.fs == nil {
if b.base == "" {
return nil, errNoBasePath
}
b.base = path.Clean(b.base)
if err := file.MkdirAll(b.base); err != nil {
return nil, errors.Wrapf(err, "failed to create blob storage at %s", b.base)
}
b.fs = afero.NewBasePathFs(afero.NewOsFs(), b.base)
}
b.base = path.Clean(b.base)
if err := file.MkdirAll(b.base); err != nil {
return nil, errors.Wrapf(err, "failed to create blob storage at %s", b.base)
b.cache = newBlobStorageCache()
pruner := newBlobPruner(b.retentionEpochs)
if b.layoutName == "" {
b.layoutName = LayoutNameFlat
}
b.fs = afero.NewBasePathFs(afero.NewOsFs(), b.base)
pruner, err := newBlobPruner(b.fs, b.retentionEpochs)
layout, err := newLayout(b.layoutName, b.fs, b.cache, pruner)
if err != nil {
return nil, err
}
b.pruner = pruner
b.layout = layout
return b, nil
}
@@ -96,47 +110,103 @@ func NewBlobStorage(opts ...BlobStorageOption) (*BlobStorage, error) {
type BlobStorage struct {
base string
retentionEpochs primitives.Epoch
layoutName string
fsync bool
fs afero.Fs
pruner *blobPruner
layout fsLayout
cache *blobStorageSummaryCache
}
// WarmCache runs the prune routine with an expiration of slot of 0, so nothing will be pruned, but the pruner's cache
// will be populated at node startup, avoiding a costly cold prune (~4s in syscalls) during syncing.
func (bs *BlobStorage) WarmCache() {
if bs.pruner == nil {
return
}
go func() {
start := time.Now()
start := time.Now()
if bs.layoutName == LayoutNameFlat {
log.Info("Blob filesystem cache warm-up started. This may take a few minutes.")
if err := bs.pruner.warmCache(); err != nil {
log.WithError(err).Error("Error encountered while warming up blob pruner cache")
}
log.WithField("elapsed", time.Since(start)).Info("Blob filesystem cache warm-up complete")
}()
} else {
log.Info("Blob filesystem cache warm-up started.")
}
if err := warmCache(bs.layout, bs.cache); err != nil {
log.WithError(err).Error("Error encountered while warming up blob filesystem cache.")
}
if err := bs.migrateLayouts(); err != nil {
log.WithError(err).Error("Error encountered while migrating blob storage.")
}
log.WithField("elapsed", time.Since(start)).Info("Blob filesystem cache warm-up complete.")
}
// ErrBlobStorageSummarizerUnavailable is a sentinel error returned when there is no pruner/cache available.
// This should be used by code that optionally uses the summarizer to optimize rpc requests. Being able to
// fallback when there is no summarizer allows client code to avoid test complexity where the summarizer doesn't matter.
var ErrBlobStorageSummarizerUnavailable = errors.New("BlobStorage not initialized with a pruner or cache")
// WaitForSummarizer blocks until the BlobStorageSummarizer is ready to use.
// BlobStorageSummarizer is not ready immediately on node startup because it needs to sample the blob filesystem to
// determine which blobs are available.
func (bs *BlobStorage) WaitForSummarizer(ctx context.Context) (BlobStorageSummarizer, error) {
if bs == nil || bs.pruner == nil {
return nil, ErrBlobStorageSummarizerUnavailable
// If any blob storage directories are found for layouts besides the configured layout, migrate them.
func (bs *BlobStorage) migrateLayouts() error {
for _, name := range LayoutNames {
if name == bs.layoutName {
continue
}
from, err := newLayout(name, bs.fs, bs.cache, nil)
if err != nil {
return err
}
if err := migrateLayout(bs.fs, from, bs.layout, bs.cache); err != nil {
if errors.Is(err, errLayoutNotDetected) {
continue
}
return errors.Wrapf(err, "failed to migrate layout from %s to %s", name, bs.layoutName)
}
}
return bs.pruner.waitForCache(ctx)
return nil
}
func (bs *BlobStorage) writePart(sidecar blocks.VerifiedROBlob) (ppath string, err error) {
ident := identForSidecar(sidecar)
sidecarData, err := sidecar.MarshalSSZ()
if err != nil {
return "", errors.Wrap(err, "failed to serialize sidecar data")
}
if len(sidecarData) == 0 {
return "", errSidecarEmptySSZData
}
if err := bs.fs.MkdirAll(bs.layout.dir(ident), directoryPermissions()); err != nil {
return "", err
}
ppath = bs.layout.partPath(ident, fmt.Sprintf("%p", sidecarData))
// Create a partial file and write the serialized data to it.
partialFile, err := bs.fs.Create(ppath)
if err != nil {
return "", errors.Wrap(err, "failed to create partial file")
}
defer func() {
cerr := partialFile.Close()
// The close error is probably less important than any existing error, so only overwrite nil err.
if cerr != nil && err == nil {
err = cerr
}
}()
n, err := partialFile.Write(sidecarData)
if err != nil {
return ppath, errors.Wrap(err, "failed to write to partial file")
}
if bs.fsync {
if err := partialFile.Sync(); err != nil {
return ppath, err
}
}
if n != len(sidecarData) {
return ppath, fmt.Errorf("failed to write the full bytes of sidecarData, wrote only %d of %d bytes", n, len(sidecarData))
}
return ppath, nil
}
// Save saves blobs given a list of sidecars.
func (bs *BlobStorage) Save(sidecar blocks.VerifiedROBlob) error {
startTime := time.Now()
fname := namerForSidecar(sidecar)
sszPath := fname.path()
ident := identForSidecar(sidecar)
sszPath := bs.layout.sszPath(ident)
exists, err := afero.Exists(bs.fs, sszPath)
if err != nil {
return err
@@ -145,78 +215,36 @@ func (bs *BlobStorage) Save(sidecar blocks.VerifiedROBlob) error {
log.WithFields(logging.BlobFields(sidecar.ROBlob)).Debug("Ignoring a duplicate blob sidecar save attempt")
return nil
}
if bs.pruner != nil {
if err := bs.pruner.notify(sidecar.BlockRoot(), sidecar.Slot(), sidecar.Index); err != nil {
return errors.Wrapf(err, "problem maintaining pruning cache/metrics for sidecar with root=%#x", sidecar.BlockRoot())
}
}
// Serialize the ethpb.BlobSidecar to binary data using SSZ.
sidecarData, err := sidecar.MarshalSSZ()
if err != nil {
return errors.Wrap(err, "failed to serialize sidecar data")
} else if len(sidecarData) == 0 {
return errSidecarEmptySSZData
}
if err := bs.fs.MkdirAll(fname.dir(), directoryPermissions); err != nil {
return err
}
partPath := fname.partPath(fmt.Sprintf("%p", sidecarData))
partialMoved := false
partPath, err := bs.writePart(sidecar)
// Ensure the partial file is deleted.
defer func() {
if partialMoved {
if partialMoved || partPath == "" {
return
}
// It's expected to error if the save is successful.
err = bs.fs.Remove(partPath)
err := bs.fs.Remove(partPath)
if err == nil {
log.WithFields(logrus.Fields{
"partPath": partPath,
}).Debugf("Removed partial file")
}
}()
// Create a partial file and write the serialized data to it.
partialFile, err := bs.fs.Create(partPath)
if err != nil {
return errors.Wrap(err, "failed to create partial file")
}
n, err := partialFile.Write(sidecarData)
if err != nil {
closeErr := partialFile.Close()
if closeErr != nil {
return closeErr
}
return errors.Wrap(err, "failed to write to partial file")
}
if bs.fsync {
if err := partialFile.Sync(); err != nil {
return err
}
}
if err := partialFile.Close(); err != nil {
return err
}
if n != len(sidecarData) {
return fmt.Errorf("failed to write the full bytes of sidecarData, wrote only %d of %d bytes", n, len(sidecarData))
}
if n == 0 {
return errEmptyBlobWritten
}
// Atomically rename the partial file to its final name.
err = bs.fs.Rename(partPath, sszPath)
if err != nil {
return errors.Wrap(err, "failed to rename partial file to final name")
}
partialMoved = true
if err := bs.layout.notify(ident); err != nil {
return errors.Wrapf(err, "problem maintaining pruning cache/metrics for sidecar with root=%#x", sidecar.BlockRoot())
}
blobsWrittenCounter.Inc()
blobSaveLatency.Observe(float64(time.Since(startTime).Milliseconds()))
@@ -228,70 +256,30 @@ func (bs *BlobStorage) Save(sidecar blocks.VerifiedROBlob) error {
// value is always a VerifiedROBlob.
func (bs *BlobStorage) Get(root [32]byte, idx uint64) (blocks.VerifiedROBlob, error) {
startTime := time.Now()
expected := blobNamer{root: root, index: idx}
encoded, err := afero.ReadFile(bs.fs, expected.path())
var v blocks.VerifiedROBlob
ident, err := bs.layout.ident(root, idx)
if err != nil {
return v, err
}
s := &ethpb.BlobSidecar{}
if err := s.UnmarshalSSZ(encoded); err != nil {
return v, err
}
ro, err := blocks.NewROBlobWithRoot(s, root)
if err != nil {
return blocks.VerifiedROBlob{}, err
return verification.VerifiedROBlobError(err)
}
defer func() {
blobFetchLatency.Observe(float64(time.Since(startTime).Milliseconds()))
}()
return verification.BlobSidecarNoop(ro)
return verification.VerifiedROBlobFromDisk(bs.fs, root, bs.layout.sszPath(ident))
}
// Remove removes all blobs for a given root.
func (bs *BlobStorage) Remove(root [32]byte) error {
rootDir := blobNamer{root: root}.dir()
return bs.fs.RemoveAll(rootDir)
dirIdent, err := bs.layout.dirIdent(root)
if err != nil {
return err
}
_, err = bs.layout.remove(dirIdent)
return err
}
// Indices generates a bitmap representing which BlobSidecar.Index values are present on disk for a given root.
// This value can be compared to the commitments observed in a block to determine which indices need to be found
// on the network to confirm data availability.
func (bs *BlobStorage) Indices(root [32]byte, s primitives.Slot) ([]bool, error) {
maxBlobsPerBlock := params.BeaconConfig().MaxBlobsPerBlock(s)
mask := make([]bool, maxBlobsPerBlock)
rootDir := blobNamer{root: root}.dir()
entries, err := afero.ReadDir(bs.fs, rootDir)
if err != nil {
if os.IsNotExist(err) {
return mask, nil
}
return mask, err
}
for i := range entries {
if entries[i].IsDir() {
continue
}
name := entries[i].Name()
if !strings.HasSuffix(name, sszExt) {
continue
}
parts := strings.Split(name, ".")
if len(parts) != 2 {
continue
}
u, err := strconv.ParseUint(parts[0], 10, 64)
if err != nil {
return mask, errors.Wrapf(err, "unexpected directory entry breaks listing, %s", parts[0])
}
if u >= uint64(maxBlobsPerBlock) {
return mask, errIndexOutOfBounds
}
mask[u] = true
}
return mask, nil
// Summary returns the BlobStorageSummary from the layout.
// Internally, this is a cached representation of the directory listing for the given root.
func (bs *BlobStorage) Summary(root [32]byte) BlobStorageSummary {
return bs.layout.summary(root)
}
// Clear deletes all files on the filesystem.
@@ -316,36 +304,3 @@ func (bs *BlobStorage) WithinRetentionPeriod(requested, current primitives.Epoch
}
return requested+bs.retentionEpochs >= current
}
type blobNamer struct {
root [32]byte
index uint64
}
func namerForSidecar(sc blocks.VerifiedROBlob) blobNamer {
return blobNamer{root: sc.BlockRoot(), index: sc.Index}
}
func (p blobNamer) dir() string {
return rootString(p.root)
}
func (p blobNamer) partPath(entropy string) string {
return path.Join(p.dir(), fmt.Sprintf("%s-%d.%s", entropy, p.index, partExt))
}
func (p blobNamer) path() string {
return path.Join(p.dir(), fmt.Sprintf("%d.%s", p.index, sszExt))
}
func rootString(root [32]byte) string {
return fmt.Sprintf("%#x", root)
}
func stringToRoot(str string) ([32]byte, error) {
slice, err := hexutil.Decode(str)
if err != nil {
return [32]byte{}, errors.Wrapf(errInvalidRootString, "input=%s", str)
}
return bytesutil.ToBytes32(slice), nil
}

View File

@@ -9,26 +9,26 @@ import (
"testing"
ssz "github.com/prysmaticlabs/fastssz"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/db"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/verification"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/testing/require"
"github.com/prysmaticlabs/prysm/v5/testing/util"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/spf13/afero"
)
func TestBlobStorage_SaveBlobData(t *testing.T) {
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, 1, params.BeaconConfig().MaxBlobsPerBlock(1))
testSidecars, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
testSidecars := verification.FakeVerifySliceForTest(t, sidecars)
t.Run("no error for duplicate", func(t *testing.T) {
fs, bs := NewEphemeralBlobStorageWithFs(t)
fs, bs := NewEphemeralBlobStorageAndFs(t)
existingSidecar := testSidecars[0]
blobPath := namerForSidecar(existingSidecar).path()
blobPath := bs.layout.sszPath(identForSidecar(existingSidecar))
// Serialize the existing BlobSidecar to binary data.
existingSidecarData, err := ssz.MarshalSSZ(existingSidecar)
require.NoError(t, err)
@@ -56,8 +56,8 @@ func TestBlobStorage_SaveBlobData(t *testing.T) {
require.NoError(t, bs.Save(sc))
actualSc, err := bs.Get(sc.BlockRoot(), sc.Index)
require.NoError(t, err)
expectedIdx := []bool{false, false, true, false, false, false}
actualIdx, err := bs.Indices(actualSc.BlockRoot(), 100)
expectedIdx := blobIndexMask{false, false, true, false, false, false}
actualIdx := bs.Summary(actualSc.BlockRoot()).mask
require.NoError(t, err)
require.DeepEqual(t, expectedIdx, actualIdx)
})
@@ -85,7 +85,7 @@ func TestBlobStorage_SaveBlobData(t *testing.T) {
require.NoError(t, bs.Remove(expected.BlockRoot()))
_, err = bs.Get(expected.BlockRoot(), expected.Index)
require.ErrorContains(t, "file does not exist", err)
require.Equal(t, true, db.IsNotFound(err))
})
t.Run("clear", func(t *testing.T) {
@@ -126,16 +126,14 @@ func TestBlobStorage_SaveBlobData(t *testing.T) {
})
}
// pollUntil polls a condition function until it returns true or a timeout is reached.
func TestBlobIndicesBounds(t *testing.T) {
fs, bs := NewEphemeralBlobStorageWithFs(t)
fs := afero.NewMemMapFs()
root := [32]byte{}
okIdx := uint64(params.BeaconConfig().MaxBlobsPerBlock(0)) - 1
writeFakeSSZ(t, fs, root, okIdx)
indices, err := bs.Indices(root, 100)
require.NoError(t, err)
writeFakeSSZ(t, fs, root, 0, okIdx)
bs := NewWarmedEphemeralBlobStorageUsingFs(t, fs, WithLayout(LayoutNameByEpoch))
indices := bs.Summary(root).mask
expected := make([]bool, params.BeaconConfig().MaxBlobsPerBlock(0))
expected[okIdx] = true
for i := range expected {
@@ -143,102 +141,23 @@ func TestBlobIndicesBounds(t *testing.T) {
}
oobIdx := uint64(params.BeaconConfig().MaxBlobsPerBlock(0))
writeFakeSSZ(t, fs, root, oobIdx)
_, err = bs.Indices(root, 100)
require.ErrorIs(t, err, errIndexOutOfBounds)
writeFakeSSZ(t, fs, root, 0, oobIdx)
// This now fails at cache warmup time.
require.ErrorIs(t, warmCache(bs.layout, bs.cache), errIndexOutOfBounds)
}
func writeFakeSSZ(t *testing.T, fs afero.Fs, root [32]byte, idx uint64) {
namer := blobNamer{root: root, index: idx}
require.NoError(t, fs.MkdirAll(namer.dir(), 0700))
fh, err := fs.Create(namer.path())
func writeFakeSSZ(t *testing.T, fs afero.Fs, root [32]byte, slot primitives.Slot, idx uint64) {
epoch := slots.ToEpoch(slot)
namer := newBlobIdent(root, epoch, idx)
layout := periodicEpochLayout{}
require.NoError(t, fs.MkdirAll(layout.dir(namer), 0700))
fh, err := fs.Create(layout.sszPath(namer))
require.NoError(t, err)
_, err = fh.Write([]byte("derp"))
require.NoError(t, err)
require.NoError(t, fh.Close())
}
func TestBlobStoragePrune(t *testing.T) {
currentSlot := primitives.Slot(200000)
fs, bs := NewEphemeralBlobStorageWithFs(t)
t.Run("PruneOne", func(t *testing.T) {
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, 300, params.BeaconConfig().MaxBlobsPerBlock(0))
testSidecars, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
for _, sidecar := range testSidecars {
require.NoError(t, bs.Save(sidecar))
}
require.NoError(t, bs.pruner.prune(currentSlot-bs.pruner.windowSize))
remainingFolders, err := afero.ReadDir(fs, ".")
require.NoError(t, err)
require.Equal(t, 0, len(remainingFolders))
})
t.Run("Prune dangling blob", func(t *testing.T) {
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, 299, params.BeaconConfig().MaxBlobsPerBlock(0))
testSidecars, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
for _, sidecar := range testSidecars[4:] {
require.NoError(t, bs.Save(sidecar))
}
require.NoError(t, bs.pruner.prune(currentSlot-bs.pruner.windowSize))
remainingFolders, err := afero.ReadDir(fs, ".")
require.NoError(t, err)
require.Equal(t, 0, len(remainingFolders))
})
t.Run("PruneMany", func(t *testing.T) {
blockQty := 10
slot := primitives.Slot(1)
for j := 0; j <= blockQty; j++ {
root := bytesutil.ToBytes32(bytesutil.ToBytes(uint64(slot), 32))
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, root, slot, params.BeaconConfig().MaxBlobsPerBlock(0))
testSidecars, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
require.NoError(t, bs.Save(testSidecars[0]))
slot += 10000
}
require.NoError(t, bs.pruner.prune(currentSlot-bs.pruner.windowSize))
remainingFolders, err := afero.ReadDir(fs, ".")
require.NoError(t, err)
require.Equal(t, 4, len(remainingFolders))
})
}
func BenchmarkPruning(b *testing.B) {
var t *testing.T
_, bs := NewEphemeralBlobStorageWithFs(t)
blockQty := 10000
currentSlot := primitives.Slot(150000)
slot := primitives.Slot(0)
for j := 0; j <= blockQty; j++ {
root := bytesutil.ToBytes32(bytesutil.ToBytes(uint64(slot), 32))
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, root, slot, params.BeaconConfig().MaxBlobsPerBlock(0))
testSidecars, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
require.NoError(t, bs.Save(testSidecars[0]))
slot += 100
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
err := bs.pruner.prune(currentSlot)
require.NoError(b, err)
}
}
func TestNewBlobStorage(t *testing.T) {
_, err := NewBlobStorage()
require.ErrorIs(t, err, errNoBasePath)
@@ -292,3 +211,13 @@ func TestConfig_WithinRetentionPeriod(t *testing.T) {
require.Equal(t, true, storage.WithinRetentionPeriod(1, 1))
})
}
func TestLayoutNames(t *testing.T) {
badLayoutName := "bad"
for _, name := range LayoutNames {
_, err := newLayout(name, nil, nil, nil)
require.NoError(t, err)
}
_, err := newLayout(badLayoutName, nil, nil, nil)
require.ErrorIs(t, err, errInvalidLayoutName)
}

View File

@@ -1,8 +1,10 @@
package filesystem
import (
"fmt"
"sync"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/db"
fieldparams "github.com/prysmaticlabs/prysm/v5/config/fieldparams"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
@@ -13,17 +15,12 @@ type blobIndexMask []bool
// BlobStorageSummary represents cached information about the BlobSidecars on disk for each root the cache knows about.
type BlobStorageSummary struct {
slot primitives.Slot
mask blobIndexMask
epoch primitives.Epoch
mask blobIndexMask
}
// HasIndex returns true if the BlobSidecar at the given index is available in the filesystem.
func (s BlobStorageSummary) HasIndex(idx uint64) bool {
// Protect from panic, but assume callers are sophisticated enough to not need an error telling them they have an invalid idx.
maxBlobsPerBlock := params.BeaconConfig().MaxBlobsPerBlock(s.slot)
if idx >= uint64(maxBlobsPerBlock) {
return false
}
if idx >= uint64(len(s.mask)) {
return false
}
@@ -32,10 +29,6 @@ func (s BlobStorageSummary) HasIndex(idx uint64) bool {
// AllAvailable returns true if we have all blobs for all indices from 0 to count-1.
func (s BlobStorageSummary) AllAvailable(count int) bool {
maxBlobsPerBlock := params.BeaconConfig().MaxBlobsPerBlock(s.slot)
if count > maxBlobsPerBlock {
return false
}
if count > len(s.mask) {
return false
}
@@ -47,83 +40,121 @@ func (s BlobStorageSummary) AllAvailable(count int) bool {
return true
}
func (s BlobStorageSummary) MaxBlobsForEpoch() uint64 {
return uint64(params.BeaconConfig().MaxBlobsPerBlockAtEpoch(s.epoch))
}
// NewBlobStorageSummary creates a new BlobStorageSummary for a given epoch and mask.
func NewBlobStorageSummary(epoch primitives.Epoch, mask []bool) (BlobStorageSummary, error) {
c := params.BeaconConfig().MaxBlobsPerBlockAtEpoch(epoch)
if len(mask) != c {
return BlobStorageSummary{}, fmt.Errorf("mask length %d does not match expected %d for epoch %d", len(mask), c, epoch)
}
return BlobStorageSummary{
epoch: epoch,
mask: mask,
}, nil
}
// BlobStorageSummarizer can be used to receive a summary of metadata about blobs on disk for a given root.
// The BlobStorageSummary can be used to check which indices (if any) are available for a given block by root.
type BlobStorageSummarizer interface {
Summary(root [32]byte) BlobStorageSummary
}
type blobStorageCache struct {
type blobStorageSummaryCache struct {
mu sync.RWMutex
nBlobs float64
cache map[[32]byte]BlobStorageSummary
}
var _ BlobStorageSummarizer = &blobStorageCache{}
var _ BlobStorageSummarizer = &blobStorageSummaryCache{}
func newBlobStorageCache() *blobStorageCache {
return &blobStorageCache{
cache: make(map[[32]byte]BlobStorageSummary, params.BeaconConfig().MinEpochsForBlobsSidecarsRequest*fieldparams.SlotsPerEpoch),
func newBlobStorageCache() *blobStorageSummaryCache {
return &blobStorageSummaryCache{
cache: make(map[[32]byte]BlobStorageSummary),
}
}
// Summary returns the BlobStorageSummary for `root`. The BlobStorageSummary can be used to check for the presence of
// BlobSidecars based on Index.
func (s *blobStorageCache) Summary(root [32]byte) BlobStorageSummary {
func (s *blobStorageSummaryCache) Summary(root [32]byte) BlobStorageSummary {
s.mu.RLock()
defer s.mu.RUnlock()
return s.cache[root]
}
func (s *blobStorageCache) ensure(key [32]byte, slot primitives.Slot, idx uint64) error {
maxBlobsPerBlock := params.BeaconConfig().MaxBlobsPerBlock(slot)
if idx >= uint64(maxBlobsPerBlock) {
func (s *blobStorageSummaryCache) ensure(ident blobIdent) error {
maxBlobsPerBlock := params.BeaconConfig().MaxBlobsPerBlockAtEpoch(ident.epoch)
if ident.index >= uint64(maxBlobsPerBlock) {
return errIndexOutOfBounds
}
s.mu.Lock()
defer s.mu.Unlock()
v := s.cache[key]
v.slot = slot
v := s.cache[ident.root]
v.epoch = ident.epoch
if v.mask == nil {
v.mask = make(blobIndexMask, maxBlobsPerBlock)
}
if !v.mask[idx] {
if !v.mask[ident.index] {
s.updateMetrics(1)
}
v.mask[idx] = true
s.cache[key] = v
v.mask[ident.index] = true
s.cache[ident.root] = v
return nil
}
func (s *blobStorageCache) slot(key [32]byte) (primitives.Slot, bool) {
func (s *blobStorageSummaryCache) get(key [32]byte) (BlobStorageSummary, bool) {
s.mu.RLock()
defer s.mu.RUnlock()
v, ok := s.cache[key]
if !ok {
return 0, false
}
return v.slot, ok
return v, ok
}
func (s *blobStorageCache) evict(key [32]byte) {
var deleted float64
func (s *blobStorageSummaryCache) identForIdx(key [32]byte, idx uint64) (blobIdent, error) {
v, ok := s.get(key)
if !ok || !v.HasIndex(idx) {
return blobIdent{}, db.ErrNotFound
}
return blobIdent{
root: key,
index: idx,
epoch: v.epoch,
}, nil
}
func (s *blobStorageSummaryCache) identForRoot(key [32]byte) (blobIdent, error) {
v, ok := s.get(key)
if !ok {
return blobIdent{}, db.ErrNotFound
}
return blobIdent{
root: key,
epoch: v.epoch,
}, nil
}
func (s *blobStorageSummaryCache) evict(key [32]byte) int {
deleted := 0
s.mu.Lock()
defer s.mu.Unlock()
v, ok := s.cache[key]
if ok {
for i := range v.mask {
if v.mask[i] {
deleted += 1
}
if !ok {
return 0
}
for i := range v.mask {
if v.mask[i] {
deleted += 1
}
}
delete(s.cache, key)
s.mu.Unlock()
if deleted > 0 {
s.updateMetrics(-deleted)
s.updateMetrics(-float64(deleted))
}
return deleted
}
func (s *blobStorageCache) updateMetrics(delta float64) {
func (s *blobStorageSummaryCache) updateMetrics(delta float64) {
s.nBlobs += delta
blobDiskCount.Set(s.nBlobs)
blobDiskSize.Set(s.nBlobs * fieldparams.BlobSidecarSize)

View File

@@ -53,7 +53,7 @@ func TestSlotByRoot_Summary(t *testing.T) {
for _, c := range cases {
if c.expected != nil {
key := bytesutil.ToBytes32([]byte(c.name))
sc.cache[key] = BlobStorageSummary{slot: 0, mask: c.expected}
sc.cache[key] = BlobStorageSummary{epoch: 0, mask: c.expected}
}
}
for _, c := range cases {

View File

@@ -0,0 +1,237 @@
package filesystem
import (
"fmt"
"io"
"path/filepath"
"strconv"
"strings"
"github.com/ethereum/go-ethereum/common/hexutil"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
"github.com/sirupsen/logrus"
"github.com/spf13/afero"
)
var errIdentFailure = errors.New("failed to determine blob metadata, ignoring all sub-paths.")
type identificationError struct {
err error
path string
ident blobIdent
}
func (ide *identificationError) Error() string {
return fmt.Sprintf("%s path=%s, err=%s", errIdentFailure.Error(), ide.path, ide.err.Error())
}
func (ide *identificationError) Unwrap() error {
return ide.err
}
func (*identificationError) Is(err error) bool {
return err == errIdentFailure
}
func (ide *identificationError) LogFields() logrus.Fields {
fields := ide.ident.logFields()
fields["path"] = ide.path
return fields
}
func newIdentificationError(path string, ident blobIdent, err error) *identificationError {
return &identificationError{path: path, ident: ident, err: err}
}
func listDir(fs afero.Fs, dir string) ([]string, error) {
top, err := fs.Open(dir)
if err != nil {
return nil, errors.Wrap(err, "failed to open directory descriptor")
}
defer func() {
if err := top.Close(); err != nil {
log.WithError(err).Errorf("Could not close file %s", dir)
}
}()
// re the -1 param: "If n <= 0, Readdirnames returns all the names from the directory in a single slice"
dirs, err := top.Readdirnames(-1)
if err != nil {
return nil, errors.Wrap(err, "failed to read directory listing")
}
return dirs, nil
}
// identPopulator is a function that sets values in the blobIdent for a given layer of the filesystem layout.
type identPopulator func(blobIdent, string) (blobIdent, error)
// layoutLayer represents a layer of the nested directory scheme. Each layer is defined by a filter function that
// ensures any entries at that layer of the scheme are named in a valid way, and a populateIdent function that
// parses the directory name into a blobIdent object, used for iterating across the layout in a layout-independent way.
type layoutLayer struct {
populateIdent identPopulator
filter func(string) bool
}
// identIterator moves through the filesystem in order to yield blobIdents.
// layoutLayers (in the 'layers' field) allows a filesystem layout to control how the
// layout is traversed. A layoutLayer can filter out entries from the directory listing
// via the filter function, and populate fields in the blobIdent via the populateIdent function.
// The blobIdent is populated from an empty value at the root, accumulating values for its fields at each layer.
// The fully populated blobIdent is returned when the iterator reaches the leaf layer.
type identIterator struct {
fs afero.Fs
path string
child *identIterator
ident blobIdent
// layoutLayers are the heart of how the layout defines the nesting of the components of the path.
// Each layer of the layout represents a different layer of the directory layout hierarchy,
// from the relative root at the zero index to the blob files at the end.
layers []layoutLayer
entries []string
offset int
eof bool
}
// atEOF can be used to peek at the iterator to see if it's already finished. This is useful for the migration code to check
// if there are any entries in the directory indicated by the migration.
func (iter *identIterator) atEOF() bool {
return iter.eof
}
// next is the only method that a user of the identIterator needs to call.
// identIterator will yield blobIdents in a breadth-first fashion,
// returning an empty blobIdent and io.EOF once all branches have been traversed.
func (iter *identIterator) next() (blobIdent, error) {
if iter.eof {
return blobIdent{}, io.EOF
}
if iter.child != nil {
next, err := iter.child.next()
if err == nil {
return next, nil
}
if !errors.Is(err, io.EOF) {
return blobIdent{}, err
}
}
return iter.advanceChild()
}
// advanceChild is used to move to the next directory at each layer of the tree, either when
// the nodes are first being initialized at a layer, or when a sub-branch has been exhausted.
func (iter *identIterator) advanceChild() (blobIdent, error) {
defer func() {
iter.offset += 1
}()
for i := iter.offset; i < len(iter.entries); i++ {
iter.offset = i
nextPath := filepath.Join(iter.path, iter.entries[iter.offset])
nextLayer := iter.layers[0]
if !nextLayer.filter(nextPath) {
continue
}
ident, err := nextLayer.populateIdent(iter.ident, nextPath)
if err != nil {
return ident, newIdentificationError(nextPath, ident, err)
}
// if we're at the leaf layer , we can return the updated ident.
if len(iter.layers) == 1 {
return ident, nil
}
entries, err := listDir(iter.fs, nextPath)
if err != nil {
return blobIdent{}, err
}
if len(entries) == 0 {
continue
}
iter.child = &identIterator{
fs: iter.fs,
path: nextPath,
ident: ident,
layers: iter.layers[1:],
entries: entries,
}
return iter.child.next()
}
return blobIdent{}, io.EOF
}
func populateNoop(namer blobIdent, _ string) (blobIdent, error) {
return namer, nil
}
func populateRoot(namer blobIdent, dir string) (blobIdent, error) {
root, err := rootFromPath(dir)
if err != nil {
return namer, err
}
namer.root = root
return namer, nil
}
func populateIndex(namer blobIdent, fname string) (blobIdent, error) {
idx, err := idxFromPath(fname)
if err != nil {
return namer, err
}
namer.index = idx
return namer, nil
}
func rootFromPath(p string) ([32]byte, error) {
subdir := filepath.Base(p)
root, err := stringToRoot(subdir)
if err != nil {
return root, errors.Wrapf(err, "invalid directory, could not parse subdir as root %s", p)
}
return root, nil
}
func idxFromPath(p string) (uint64, error) {
p = filepath.Base(p)
if !isSszFile(p) {
return 0, errors.Wrap(errNotBlobSSZ, "does not have .ssz extension")
}
parts := strings.Split(p, ".")
if len(parts) != 2 {
return 0, errors.Wrap(errNotBlobSSZ, "unexpected filename structure (want <index>.ssz)")
}
idx, err := strconv.ParseUint(parts[0], 10, 64)
if err != nil {
return 0, err
}
return idx, nil
}
func filterNoop(_ string) bool {
return true
}
func isRootDir(p string) bool {
dir := filepath.Base(p)
return len(dir) == rootStringLen && strings.HasPrefix(dir, "0x")
}
func isSszFile(s string) bool {
return filepath.Ext(s) == "."+sszExt
}
func rootToString(root [32]byte) string {
return fmt.Sprintf("%#x", root)
}
func stringToRoot(str string) ([32]byte, error) {
if len(str) != rootStringLen {
return [32]byte{}, errors.Wrapf(errInvalidRootString, "incorrect len for input=%s", str)
}
slice, err := hexutil.Decode(str)
if err != nil {
return [32]byte{}, errors.Wrapf(errInvalidRootString, "input=%s", str)
}
return bytesutil.ToBytes32(slice), nil
}

View File

@@ -0,0 +1,304 @@
package filesystem
import (
"bytes"
"fmt"
"io"
"math"
"os"
"path"
"sort"
"testing"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/verification"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/testing/require"
"github.com/prysmaticlabs/prysm/v5/testing/util"
"github.com/spf13/afero"
)
func TestRootFromDir(t *testing.T) {
cases := []struct {
name string
dir string
err error
root [32]byte
}{
{
name: "happy path",
dir: "0xffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5cb",
root: [32]byte{255, 255, 135, 94, 29, 152, 92, 92, 203, 33, 72, 148, 152, 63, 36, 40,
237, 178, 113, 240, 248, 123, 104, 186, 112, 16, 228, 169, 157, 243, 181, 203},
},
{
name: "too short",
dir: "0xffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5c",
err: errInvalidRootString,
},
{
name: "too log",
dir: "0xffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5cbb",
err: errInvalidRootString,
},
{
name: "missing prefix",
dir: "ffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5cb",
err: errInvalidRootString,
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
root, err := stringToRoot(c.dir)
if c.err != nil {
require.ErrorIs(t, err, c.err)
return
}
require.NoError(t, err)
require.Equal(t, c.root, root)
})
}
}
func TestSlotFromFile(t *testing.T) {
cases := []struct {
slot primitives.Slot
}{
{slot: 0},
{slot: 2},
{slot: 1123581321},
{slot: math.MaxUint64},
}
for _, c := range cases {
t.Run(fmt.Sprintf("slot %d", c.slot), func(t *testing.T) {
fs, bs := NewEphemeralBlobStorageAndFs(t)
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, c.slot, 1)
sc := verification.FakeVerifyForTest(t, sidecars[0])
require.NoError(t, bs.Save(sc))
namer := identForSidecar(sc)
sszPath := bs.layout.sszPath(namer)
slot, err := slotFromFile(sszPath, fs)
require.NoError(t, err)
require.Equal(t, c.slot, slot)
})
}
}
type dirFiles struct {
name string
isDir bool
children []dirFiles
}
func (df dirFiles) reify(t *testing.T, fs afero.Fs, base string) {
fullPath := path.Join(base, df.name)
if df.isDir {
if df.name != "" {
require.NoError(t, fs.Mkdir(fullPath, directoryPermissions()))
}
for _, c := range df.children {
c.reify(t, fs, fullPath)
}
} else {
fp, err := fs.Create(fullPath)
require.NoError(t, err)
_, err = fp.WriteString("derp")
require.NoError(t, err)
}
}
func (df dirFiles) childNames() []string {
cn := make([]string, len(df.children))
for i := range df.children {
cn[i] = df.children[i].name
}
return cn
}
func TestListDir(t *testing.T) {
fs := afero.NewMemMapFs()
rootStrs := []string{
"0x0023dc5d063c7c1b37016bb54963c6ff4bfe5dfdf6dac29e7ceeb2b8fa81ed7a",
"0xff30526cd634a5af3a09cc9bff67f33a621fc5b975750bb4432f74df077554b4",
"0x23f5f795aaeb78c01fadaf3d06da2e99bd4b3622ae4dfea61b05b7d9adb119c2",
}
// parent directory
tree := dirFiles{isDir: true}
// break out each subdir for easier assertions
notABlob := dirFiles{name: "notABlob", isDir: true}
childlessBlob := dirFiles{name: rootStrs[0], isDir: true}
blobWithSsz := dirFiles{name: rootStrs[1], isDir: true,
children: []dirFiles{{name: "1.ssz"}, {name: "2.ssz"}},
}
blobWithSszAndTmp := dirFiles{name: rootStrs[2], isDir: true,
children: []dirFiles{{name: "5.ssz"}, {name: "0.part"}}}
tree.children = append(tree.children,
notABlob, childlessBlob, blobWithSsz, blobWithSszAndTmp)
topChildren := make([]string, len(tree.children))
for i := range tree.children {
topChildren[i] = tree.children[i].name
}
var filter = func(entries []string, filt func(string) bool) []string {
filtered := make([]string, 0, len(entries))
for i := range entries {
if filt(entries[i]) {
filtered = append(filtered, entries[i])
}
}
return filtered
}
tree.reify(t, fs, "")
cases := []struct {
name string
dirPath string
expected []string
filter func(string) bool
err error
}{
{
name: "non-existent",
dirPath: "derp",
expected: []string{},
err: os.ErrNotExist,
},
{
name: "empty",
dirPath: childlessBlob.name,
expected: []string{},
},
{
name: "top",
dirPath: ".",
expected: topChildren,
},
{
name: "custom filter: only notABlob",
dirPath: ".",
expected: []string{notABlob.name},
filter: func(s string) bool {
return s == notABlob.name
},
},
{
name: "root filter",
dirPath: ".",
expected: []string{childlessBlob.name, blobWithSsz.name, blobWithSszAndTmp.name},
filter: isRootDir,
},
{
name: "ssz filter",
dirPath: blobWithSsz.name,
expected: blobWithSsz.childNames(),
filter: isSszFile,
},
{
name: "ssz mixed filter",
dirPath: blobWithSszAndTmp.name,
expected: []string{"5.ssz"},
filter: isSszFile,
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
result, err := listDir(fs, c.dirPath)
if c.filter != nil {
result = filter(result, c.filter)
}
if c.err != nil {
require.ErrorIs(t, err, c.err)
require.Equal(t, 0, len(result))
} else {
require.NoError(t, err)
sort.Strings(c.expected)
sort.Strings(result)
require.DeepEqual(t, c.expected, result)
}
})
}
}
func TestSlotFromBlob(t *testing.T) {
cases := []struct {
slot primitives.Slot
}{
{slot: 0},
{slot: 2},
{slot: 1123581321},
{slot: math.MaxUint64},
}
for _, c := range cases {
t.Run(fmt.Sprintf("slot %d", c.slot), func(t *testing.T) {
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, c.slot, 1)
sc := sidecars[0]
enc, err := sc.MarshalSSZ()
require.NoError(t, err)
slot, err := slotFromBlob(bytes.NewReader(enc))
require.NoError(t, err)
require.Equal(t, c.slot, slot)
})
}
}
func TestIterationComplete(t *testing.T) {
targets := []migrationTestTarget{
{
ident: ezIdent(t, "0x0125e54c64c925018c9296965a5b622d9f5ab626c10917860dcfb6aa09a0a00b", 1234, 0),
path: "by-epoch/0/1234/0x0125e54c64c925018c9296965a5b622d9f5ab626c10917860dcfb6aa09a0a00b/0.ssz",
},
{
ident: ezIdent(t, "0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86", 5330, 0),
slotOffset: 31,
path: "by-epoch/1/5330/0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86/0.ssz",
},
{
ident: ezIdent(t, "0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86", 5330, 1),
slotOffset: 31,
path: "by-epoch/1/5330/0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86/1.ssz",
},
{
ident: ezIdent(t, "0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c", 16777216, 0),
slotOffset: 16,
path: "by-epoch/4096/16777216/0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c/0.ssz",
},
{
ident: ezIdent(t, "0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c", 16777216, 1),
slotOffset: 16,
path: "by-epoch/4096/16777216/0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c/1.ssz",
},
{
ident: ezIdent(t, "0x42eabe3d2c125410cd226de6f2825fb7575ab896c3f52e43de1fa29e4c809aba", 16777217, 0),
slotOffset: 16,
path: "by-epoch/4096/16777217/0x42eabe3d2c125410cd226de6f2825fb7575ab896c3f52e43de1fa29e4c809aba/0.ssz",
},
{
ident: ezIdent(t, "0x666cea5034e22bd3b849cb33914cad59afd88ee08e4d5bc0e997411c945fbc1d", 11235, 1),
path: "by-epoch/2/11235/0x666cea5034e22bd3b849cb33914cad59afd88ee08e4d5bc0e997411c945fbc1d/1.ssz",
},
}
fs := afero.NewMemMapFs()
cache := newBlobStorageCache()
byEpoch, err := newLayout(LayoutNameByEpoch, fs, cache, nil)
require.NoError(t, err)
for _, tar := range targets {
setupTestBlobFile(t, tar.ident, tar.slotOffset, fs, byEpoch)
}
iter, err := byEpoch.iterateIdents(0)
require.NoError(t, err)
nIdents := 0
for ident, err := iter.next(); err != io.EOF; ident, err = iter.next() {
require.NoError(t, err)
nIdents++
require.NoError(t, cache.ensure(ident))
}
require.Equal(t, len(targets), nIdents)
for _, tar := range targets {
entry, ok := cache.get(tar.ident.root)
require.Equal(t, true, ok)
require.Equal(t, tar.ident.epoch, entry.epoch)
require.Equal(t, true, entry.HasIndex(tar.ident.index))
require.Equal(t, tar.path, byEpoch.sszPath(tar.ident))
}
}

View File

@@ -0,0 +1,291 @@
package filesystem
import (
"fmt"
"io"
"path/filepath"
"strings"
"time"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/sirupsen/logrus"
"github.com/spf13/afero"
)
const (
// Full root in directory will be 66 chars, eg:
// >>> len('0x0002fb4db510b8618b04dc82d023793739c26346a8b02eb73482e24b0fec0555') == 66
rootStringLen = 66
sszExt = "ssz"
partExt = "part"
periodicEpochBaseDir = "by-epoch"
)
const (
LayoutNameFlat = "flat"
LayoutNameByEpoch = "by-epoch"
)
var LayoutNames = []string{LayoutNameFlat, LayoutNameByEpoch}
var (
errMigrationFailure = errors.New("unable to migrate blob directory between old and new layout")
errCacheWarmFailed = errors.New("failed to warm blob filesystem cache")
errPruneFailed = errors.New("failed to prune root")
errInvalidRootString = errors.New("Could not parse hex string as a [32]byte")
errInvalidDirectoryLayout = errors.New("Could not parse blob directory path")
errInvalidLayoutName = errors.New("unknown layout name")
errLayoutNotDetected = errors.New("given layout not observed in the blob filesystem tree")
)
type blobIdent struct {
root [32]byte
epoch primitives.Epoch
index uint64
}
func newBlobIdent(root [32]byte, epoch primitives.Epoch, index uint64) blobIdent {
return blobIdent{root: root, epoch: epoch, index: index}
}
func identForSidecar(sc blocks.VerifiedROBlob) blobIdent {
return newBlobIdent(sc.BlockRoot(), slots.ToEpoch(sc.Slot()), sc.Index)
}
func (n blobIdent) sszFname() string {
return fmt.Sprintf("%d.%s", n.index, sszExt)
}
func (n blobIdent) partFname(entropy string) string {
return fmt.Sprintf("%s-%d.%s", entropy, n.index, partExt)
}
func (n blobIdent) logFields() logrus.Fields {
return logrus.Fields{
"root": fmt.Sprintf("%#x", n.root),
"epoch": n.epoch,
"index": n.index,
}
}
type fsLayout interface {
name() string
dir(n blobIdent) string
sszPath(n blobIdent) string
partPath(n blobIdent, entropy string) string
iterateIdents(before primitives.Epoch) (*identIterator, error)
ident(root [32]byte, idx uint64) (blobIdent, error)
dirIdent(root [32]byte) (blobIdent, error)
summary(root [32]byte) BlobStorageSummary
notify(ident blobIdent) error
pruneBefore(before primitives.Epoch) (*pruneSummary, error)
remove(ident blobIdent) (int, error)
blockParentDirs(ident blobIdent) []string
}
func newLayout(name string, fs afero.Fs, cache *blobStorageSummaryCache, pruner *blobPruner) (fsLayout, error) {
switch name {
case LayoutNameFlat:
return newFlatLayout(fs, cache, pruner), nil
case LayoutNameByEpoch:
return newPeriodicEpochLayout(fs, cache, pruner), nil
default:
return nil, errors.Wrapf(errInvalidLayoutName, "name=%s", name)
}
}
func warmCache(l fsLayout, cache *blobStorageSummaryCache) error {
iter, err := l.iterateIdents(0)
if err != nil {
return errors.Wrap(errCacheWarmFailed, err.Error())
}
for ident, err := iter.next(); !errors.Is(err, io.EOF); ident, err = iter.next() {
if errors.Is(err, errIdentFailure) {
idf := &identificationError{}
if errors.As(err, &idf) {
log.WithFields(idf.LogFields()).WithError(err).Error("Failed to cache blob data for path")
}
continue
}
if err != nil {
return fmt.Errorf("%w: failed to populate blob data cache: %w", errCacheWarmFailed, err)
}
if err := cache.ensure(ident); err != nil {
return fmt.Errorf("%w: failed to write cache entry for %s: %w", errCacheWarmFailed, l.sszPath(ident), err)
}
}
return nil
}
func migrateLayout(fs afero.Fs, from, to fsLayout, cache *blobStorageSummaryCache) error {
start := time.Now()
iter, err := from.iterateIdents(0)
if err != nil {
return errors.Wrapf(errMigrationFailure, "failed to iterate legacy structure while migrating blobs, err=%s", err.Error())
}
if iter.atEOF() {
return errLayoutNotDetected
}
log.WithField("fromLayout", from.name()).WithField("toLayout", to.name()).Info("Migrating blob filesystem layout. This one-time operation can take extra time (up to a few minutes for systems with extended blob storage and a cold disk cache).")
lastMoved := ""
parentDirs := make(map[string]bool) // this map should have < 65k keys by design
moved := 0
dc := newDirCleaner()
for ident, err := iter.next(); !errors.Is(err, io.EOF); ident, err = iter.next() {
if err != nil {
if errors.Is(err, errIdentFailure) {
idf := &identificationError{}
if errors.As(err, &idf) {
log.WithFields(idf.LogFields()).WithError(err).Error("Failed to migrate blob path")
}
continue
}
return errors.Wrapf(errMigrationFailure, "failed to iterate previous layout structure while migrating blobs, err=%s", err.Error())
}
src := from.dir(ident)
target := to.dir(ident)
if src != lastMoved {
targetParent := filepath.Dir(target)
if targetParent != "" && targetParent != "." && !parentDirs[targetParent] {
if err := fs.MkdirAll(targetParent, directoryPermissions()); err != nil {
return errors.Wrapf(errMigrationFailure, "failed to make enclosing path before moving %s to %s", src, target)
}
parentDirs[targetParent] = true
}
if err := fs.Rename(src, target); err != nil {
return errors.Wrapf(errMigrationFailure, "could not rename %s to %s", src, target)
}
moved += 1
lastMoved = src
for _, dir := range from.blockParentDirs(ident) {
dc.add(dir)
}
}
if err := cache.ensure(ident); err != nil {
return errors.Wrapf(errMigrationFailure, "could not cache path %s, err=%s", to.sszPath(ident), err.Error())
}
}
dc.clean(fs)
if moved > 0 {
log.WithField("dirsMoved", moved).WithField("elapsed", time.Since(start)).
Info("Blob filesystem migration complete.")
}
return nil
}
type dirCleaner struct {
maxDepth int
layers map[int]map[string]struct{}
}
func newDirCleaner() *dirCleaner {
return &dirCleaner{layers: make(map[int]map[string]struct{})}
}
func (d *dirCleaner) add(dir string) {
nLayers := len(strings.Split(dir, string(filepath.Separator)))
_, ok := d.layers[nLayers]
if !ok {
d.layers[nLayers] = make(map[string]struct{})
}
d.layers[nLayers][dir] = struct{}{}
if nLayers > d.maxDepth {
d.maxDepth = nLayers
}
}
func (d *dirCleaner) clean(fs afero.Fs) {
for i := d.maxDepth; i >= 0; i-- {
d.cleanLayer(fs, i)
}
}
func (d *dirCleaner) cleanLayer(fs afero.Fs, layer int) {
dirs, ok := d.layers[layer]
if !ok {
return
}
for dir := range dirs {
// Use Remove rather than RemoveAll to make sure we're only removing empty directories
if err := fs.Remove(dir); err != nil {
log.WithField("dir", dir).WithError(err).Error("Failed to remove blob directory, please remove it manually if desired.")
contents, err := listDir(fs, dir)
if err != nil {
log.WithField("dir", dir).WithError(err).Error("Could not list blob directory contents to find reason for removal failure.")
continue
}
for _, c := range contents {
log.WithField("file", c).WithField("dir", dir).Debug("Unexpected file blocking migrated blob directory cleanup.")
}
}
}
}
type pruneSummary struct {
blobsPruned int
failedRemovals []string
}
func (s pruneSummary) LogFields() logrus.Fields {
return logrus.Fields{
"blobsPruned": s.blobsPruned,
"failedRemovals": len(s.failedRemovals),
}
}
func pruneBefore(before primitives.Epoch, l fsLayout) (map[primitives.Epoch]*pruneSummary, error) {
sums := make(map[primitives.Epoch]*pruneSummary)
iter, err := l.iterateIdents(before)
if err != nil {
return nil, errors.Wrap(err, "failed to iterate blob paths for pruning")
}
// We will get an ident for each index, but want to prune all indexes for the given root together.
var lastIdent blobIdent
for ident, err := iter.next(); !errors.Is(err, io.EOF); ident, err = iter.next() {
if err != nil {
if errors.Is(err, errIdentFailure) {
idf := &identificationError{}
if errors.As(err, &idf) {
log.WithFields(idf.LogFields()).WithError(err).Error("Failed to prune blob path due to identification errors")
}
continue
}
log.WithError(err).Error("encountered unhandled error during pruning")
return nil, errors.Wrap(errPruneFailed, err.Error())
}
if ident.epoch >= before {
continue
}
if lastIdent.root != ident.root {
pruneOne(lastIdent, l, sums)
lastIdent = ident
}
}
// handle the final ident
pruneOne(lastIdent, l, sums)
return sums, nil
}
func pruneOne(ident blobIdent, l fsLayout, sums map[primitives.Epoch]*pruneSummary) {
// Skip pruning the n-1 ident if we're on the first real ident (lastIdent will be zero value).
if ident.root == params.BeaconConfig().ZeroHash {
return
}
_, ok := sums[ident.epoch]
if !ok {
sums[ident.epoch] = &pruneSummary{}
}
s := sums[ident.epoch]
removed, err := l.remove(ident)
if err != nil {
s.failedRemovals = append(s.failedRemovals, l.dir(ident))
log.WithField("root", fmt.Sprintf("%#x", ident.root)).Error("Failed to delete blob directory for root")
}
s.blobsPruned += removed
}

View File

@@ -0,0 +1,212 @@
package filesystem
import (
"fmt"
"os"
"path"
"path/filepath"
"strconv"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/spf13/afero"
)
const epochsPerDirectory = 4096
type periodicEpochLayout struct {
fs afero.Fs
cache *blobStorageSummaryCache
pruner *blobPruner
}
var _ fsLayout = &periodicEpochLayout{}
func newPeriodicEpochLayout(fs afero.Fs, cache *blobStorageSummaryCache, pruner *blobPruner) fsLayout {
l := &periodicEpochLayout{fs: fs, cache: cache, pruner: pruner}
return l
}
func (l *periodicEpochLayout) name() string {
return LayoutNameByEpoch
}
func (l *periodicEpochLayout) blockParentDirs(ident blobIdent) []string {
return []string{
periodicEpochBaseDir,
l.periodDir(ident.epoch),
l.epochDir(ident.epoch),
}
}
func (l *periodicEpochLayout) notify(ident blobIdent) error {
if err := l.cache.ensure(ident); err != nil {
return err
}
l.pruner.notify(ident.epoch, l)
return nil
}
// If before == 0, it won't be used as a filter and all idents will be returned.
func (l *periodicEpochLayout) iterateIdents(before primitives.Epoch) (*identIterator, error) {
_, err := l.fs.Stat(periodicEpochBaseDir)
if err != nil {
if os.IsNotExist(err) {
return &identIterator{eof: true}, nil // The directory is non-existent, which is fine; stop iteration.
}
return nil, errors.Wrapf(err, "error reading path %s", periodicEpochBaseDir)
}
// iterate root, which should have directories named by "period"
entries, err := listDir(l.fs, periodicEpochBaseDir)
if err != nil {
return nil, errors.Wrapf(err, "failed to list %s", periodicEpochBaseDir)
}
return &identIterator{
fs: l.fs,
path: periodicEpochBaseDir,
// Please see comments on the `layers` field in `identIterator`` if the role of the layers is unclear.
layers: []layoutLayer{
{populateIdent: populateNoop, filter: isBeforePeriod(before)},
{populateIdent: populateEpoch, filter: isBeforeEpoch(before)},
{populateIdent: populateRoot, filter: isRootDir}, // extract root from path
{populateIdent: populateIndex, filter: isSszFile}, // extract index from filename
},
entries: entries,
}, nil
}
func (l *periodicEpochLayout) ident(root [32]byte, idx uint64) (blobIdent, error) {
return l.cache.identForIdx(root, idx)
}
func (l *periodicEpochLayout) dirIdent(root [32]byte) (blobIdent, error) {
return l.cache.identForRoot(root)
}
func (l *periodicEpochLayout) summary(root [32]byte) BlobStorageSummary {
return l.cache.Summary(root)
}
func (l *periodicEpochLayout) dir(n blobIdent) string {
return filepath.Join(l.epochDir(n.epoch), rootToString(n.root))
}
func (l *periodicEpochLayout) epochDir(epoch primitives.Epoch) string {
return filepath.Join(l.periodDir(epoch), fmt.Sprintf("%d", epoch))
}
func (l *periodicEpochLayout) periodDir(epoch primitives.Epoch) string {
return filepath.Join(periodicEpochBaseDir, fmt.Sprintf("%d", periodForEpoch(epoch)))
}
func (l *periodicEpochLayout) sszPath(n blobIdent) string {
return filepath.Join(l.dir(n), n.sszFname())
}
func (l *periodicEpochLayout) partPath(n blobIdent, entropy string) string {
return path.Join(l.dir(n), n.partFname(entropy))
}
func (l *periodicEpochLayout) pruneBefore(before primitives.Epoch) (*pruneSummary, error) {
sums, err := pruneBefore(before, l)
if err != nil {
return nil, err
}
// Roll up summaries and clean up per-epoch directories.
rollup := &pruneSummary{}
for epoch, sum := range sums {
rollup.blobsPruned += sum.blobsPruned
rollup.failedRemovals = append(rollup.failedRemovals, sum.failedRemovals...)
rmdir := l.epochDir(epoch)
if len(sum.failedRemovals) == 0 {
if err := l.fs.Remove(rmdir); err != nil {
log.WithField("dir", rmdir).WithError(err).Error("Failed to remove epoch directory while pruning")
}
} else {
log.WithField("dir", rmdir).WithField("numFailed", len(sum.failedRemovals)).WithError(err).Error("Unable to remove epoch directory due to pruning failures")
}
}
return rollup, nil
}
func (l *periodicEpochLayout) remove(ident blobIdent) (int, error) {
removed := l.cache.evict(ident.root)
// Skip the syscall if there are no blobs to remove.
if removed == 0 {
return 0, nil
}
if err := l.fs.RemoveAll(l.dir(ident)); err != nil {
return removed, err
}
return removed, nil
}
func periodForEpoch(epoch primitives.Epoch) primitives.Epoch {
return epoch / params.BeaconConfig().MinEpochsForBlobsSidecarsRequest
}
// Funcs below this line are iteration support methods that are specific to the epoch layout.
func isBeforePeriod(before primitives.Epoch) func(string) bool {
if before == 0 {
return filterNoop
}
beforePeriod := periodForEpoch(before)
if before%epochsPerDirectory != 0 {
// Add one because we need to include the period the epoch is in, unless it is the first epoch in the period,
// in which case we can just look at any previous period.
beforePeriod += 1
}
return func(p string) bool {
period, err := periodFromPath(p)
if err != nil {
return false
}
return primitives.Epoch(period) < beforePeriod
}
}
func isBeforeEpoch(before primitives.Epoch) func(string) bool {
if before == 0 {
return filterNoop
}
return func(p string) bool {
epoch, err := epochFromPath(p)
if err != nil {
return false
}
return epoch < before
}
}
func epochFromPath(p string) (primitives.Epoch, error) {
subdir := filepath.Base(p)
epoch, err := strconv.ParseUint(subdir, 10, 64)
if err != nil {
return 0, errors.Wrapf(errInvalidDirectoryLayout,
"failed to decode epoch as uint, err=%s, dir=%s", err.Error(), p)
}
return primitives.Epoch(epoch), nil
}
func periodFromPath(p string) (uint64, error) {
subdir := filepath.Base(p)
period, err := strconv.ParseUint(subdir, 10, 64)
if err != nil {
return 0, errors.Wrapf(errInvalidDirectoryLayout,
"failed to decode period from path as uint, err=%s, dir=%s", err.Error(), p)
}
return period, nil
}
func populateEpoch(namer blobIdent, dir string) (blobIdent, error) {
epoch, err := epochFromPath(dir)
if err != nil {
return namer, err
}
namer.epoch = epoch
return namer, nil
}

View File

@@ -0,0 +1,219 @@
package filesystem
import (
"encoding/binary"
"io"
"os"
"path"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/spf13/afero"
)
type flatLayout struct {
fs afero.Fs
cache *blobStorageSummaryCache
pruner *blobPruner
}
var _ fsLayout = &flatLayout{}
func newFlatLayout(fs afero.Fs, cache *blobStorageSummaryCache, pruner *blobPruner) fsLayout {
l := &flatLayout{fs: fs, cache: cache, pruner: pruner}
return l
}
func (l *flatLayout) iterateIdents(before primitives.Epoch) (*identIterator, error) {
if _, err := l.fs.Stat("."); err != nil {
if os.IsNotExist(err) {
return &identIterator{eof: true}, nil // The directory is non-existent, which is fine; stop iteration.
}
return nil, errors.Wrapf(err, "error reading path %s", periodicEpochBaseDir)
}
entries, err := listDir(l.fs, ".")
if err != nil {
return nil, errors.Wrapf(err, "could not list root directory")
}
slotAndIndex := &flatSlotReader{fs: l.fs, cache: l.cache, before: before}
return &identIterator{
fs: l.fs,
// Please see comments on the `layers` field in `identIterator`` if the role of the layers is unclear.
layers: []layoutLayer{
{populateIdent: populateRoot, filter: isFlatCachedAndBefore(l.cache, before)},
{populateIdent: slotAndIndex.populateEpoch, filter: slotAndIndex.isSSZAndBefore}},
entries: entries,
}, nil
}
func (*flatLayout) name() string {
return LayoutNameFlat
}
func (l *flatLayout) blockParentDirs(ident blobIdent) []string {
return []string{}
}
func (*flatLayout) dir(n blobIdent) string {
return rootToString(n.root)
}
func (l *flatLayout) sszPath(n blobIdent) string {
return path.Join(l.dir(n), n.sszFname())
}
func (l *flatLayout) partPath(n blobIdent, entropy string) string {
return path.Join(l.dir(n), n.partFname(entropy))
}
func (l *flatLayout) ident(root [32]byte, idx uint64) (blobIdent, error) {
return l.cache.identForIdx(root, idx)
}
func (l *flatLayout) dirIdent(root [32]byte) (blobIdent, error) {
return l.cache.identForRoot(root)
}
func (l *flatLayout) summary(root [32]byte) BlobStorageSummary {
return l.cache.Summary(root)
}
func (l *flatLayout) remove(ident blobIdent) (int, error) {
removed := l.cache.evict(ident.root)
if err := l.fs.RemoveAll(l.dir(ident)); err != nil {
return removed, err
}
return removed, nil
}
func (l *flatLayout) notify(ident blobIdent) error {
if err := l.cache.ensure(ident); err != nil {
return err
}
l.pruner.notify(ident.epoch, l)
return nil
}
func (l *flatLayout) pruneBefore(before primitives.Epoch) (*pruneSummary, error) {
sums, err := pruneBefore(before, l)
if err != nil {
return nil, err
}
// Roll up summaries and clean up per-epoch directories.
rollup := &pruneSummary{}
for _, sum := range sums {
rollup.blobsPruned += sum.blobsPruned
rollup.failedRemovals = append(rollup.failedRemovals, sum.failedRemovals...)
}
return rollup, nil
}
// Below this line are iteration support funcs and types that are specific to the flat layout.
// Read slot from marshaled BlobSidecar data in the given file. See slotFromBlob for details.
func slotFromFile(name string, fs afero.Fs) (primitives.Slot, error) {
f, err := fs.Open(name)
if err != nil {
return 0, err
}
defer func() {
if err := f.Close(); err != nil {
log.WithError(err).Errorf("Could not close blob file")
}
}()
return slotFromBlob(f)
}
// slotFromBlob reads the ssz data of a file at the specified offset (8 + 131072 + 48 + 48 = 131176 bytes),
// which is calculated based on the size of the BlobSidecar struct and is based on the size of the fields
// preceding the slot information within SignedBeaconBlockHeader.
func slotFromBlob(at io.ReaderAt) (primitives.Slot, error) {
b := make([]byte, 8)
_, err := at.ReadAt(b, 131176)
if err != nil {
return 0, err
}
rawSlot := binary.LittleEndian.Uint64(b)
return primitives.Slot(rawSlot), nil
}
type flatSlotReader struct {
before primitives.Epoch
fs afero.Fs
cache *blobStorageSummaryCache
}
func (l *flatSlotReader) populateEpoch(ident blobIdent, fname string) (blobIdent, error) {
ident, err := populateIndex(ident, fname)
if err != nil {
return ident, err
}
sum, ok := l.cache.get(ident.root)
if ok {
ident.epoch = sum.epoch
// Return early if the index is already known to the cache.
if sum.HasIndex(ident.index) {
return ident, nil
}
} else {
// If the root is not in the cache, we need to read the slot from the file.
slot, err := slotFromFile(fname, l.fs)
if err != nil {
return ident, err
}
ident.epoch = slots.ToEpoch(slot)
}
return ident, l.cache.ensure(ident)
}
func (l *flatSlotReader) isSSZAndBefore(fname string) bool {
if !isSszFile(fname) {
return false
}
// If 'before' != 0, assuming isSSZAndBefore is used as a filter on the same layer with populateEpoch, this will typically
// call populateEpoch before the iteration code calls it. So we can guarantee that the cache gets populated
// in either case, because if it is filtered out here, we either have a malformed path (root can't be determined) in which case
// the filter code won't call it anyway, or we have a valid path and the cache will be populated before the epoch can be compared.
if l.before == 0 {
return true
}
ident, err := populateRoot(blobIdent{}, path.Dir(fname))
// Filter out the path if we can't determine its root.
if err != nil {
return false
}
ident, err = l.populateEpoch(ident, fname)
// Filter out the path if we can't determine its epoch or properly cache it.
if err != nil {
return false
}
return ident.epoch < l.before
}
// isFlatCachedAndBefore returns a filter callback function to exclude roots that are known to be after the given epoch
// based on the cache. It's an opportunistic filter; if the cache is not populated, it will not attempt to populate it.
// isSSZAndBefore on the other hand, is a strict filter that will only return true if the file is an SSZ file and
// the epoch can be determined.
func isFlatCachedAndBefore(cache *blobStorageSummaryCache, before primitives.Epoch) func(string) bool {
if before == 0 {
return isRootDir
}
return func(p string) bool {
if !isRootDir(p) {
return false
}
root, err := rootFromPath(p)
if err != nil {
return false
}
sum, ok := cache.get(root)
// If we don't know the epoch by looking at the root, don't try to filter it.
if !ok {
return true
}
return sum.epoch < before
}
}

View File

@@ -0,0 +1,75 @@
package filesystem
import (
"testing"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/testing/require"
)
type mockLayout struct {
pruneBeforeFunc func(primitives.Epoch) (*pruneSummary, error)
}
var _ fsLayout = &mockLayout{}
func (m *mockLayout) name() string {
return "mock"
}
func (*mockLayout) dir(_ blobIdent) string {
return ""
}
func (*mockLayout) blockParentDirs(id blobIdent) []string {
return []string{}
}
func (*mockLayout) sszPath(_ blobIdent) string {
return ""
}
func (*mockLayout) partPath(_ blobIdent, _ string) string {
return ""
}
func (*mockLayout) iterateIdents(_ primitives.Epoch) (*identIterator, error) {
return nil, nil
}
func (*mockLayout) ident(_ [32]byte, _ uint64) (blobIdent, error) {
return blobIdent{}, nil
}
func (*mockLayout) dirIdent(_ [32]byte) (blobIdent, error) {
return blobIdent{}, nil
}
func (*mockLayout) summary(_ [32]byte) BlobStorageSummary {
return BlobStorageSummary{}
}
func (*mockLayout) notify(blobIdent) error {
return nil
}
func (m *mockLayout) pruneBefore(before primitives.Epoch) (*pruneSummary, error) {
return m.pruneBeforeFunc(before)
}
func (*mockLayout) remove(ident blobIdent) (int, error) {
return 0, nil
}
var _ fsLayout = &mockLayout{}
func TestCleaner(t *testing.T) {
l := &periodicEpochLayout{}
p := l.periodDir(11235813)
e := l.epochDir(11235813)
dc := newDirCleaner()
dc.add(p)
require.Equal(t, 2, dc.maxDepth)
dc.add(e)
require.Equal(t, 3, dc.maxDepth)
}

View File

@@ -0,0 +1,180 @@
package filesystem
import (
"os"
"testing"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/testing/require"
"github.com/prysmaticlabs/prysm/v5/testing/util"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/spf13/afero"
)
func ezIdent(t *testing.T, rootStr string, epoch primitives.Epoch, index uint64) blobIdent {
r, err := stringToRoot(rootStr)
require.NoError(t, err)
return blobIdent{root: r, epoch: epoch, index: index}
}
func setupTestBlobFile(t *testing.T, ident blobIdent, offset primitives.Slot, fs afero.Fs, l fsLayout) {
slot, err := slots.EpochStart(ident.epoch)
require.NoError(t, err)
slot += offset
_, sc := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, 1)
scb, err := sc[0].MarshalSSZ()
require.NoError(t, err)
dir := l.dir(ident)
require.NoError(t, fs.MkdirAll(dir, directoryPermissions()))
p := l.sszPath(ident)
require.NoError(t, afero.WriteFile(fs, p, scb, 0666))
_, err = fs.Stat(p)
require.NoError(t, err)
}
type migrationTestTarget struct {
ident blobIdent
slotOffset primitives.Slot
migrated bool
path string
}
func testAssertFsMigrated(t *testing.T, fs afero.Fs, ident blobIdent, before, after fsLayout) {
// Assert the pre-migration path is gone.
_, err := fs.Stat(before.sszPath(ident))
require.ErrorIs(t, err, os.ErrNotExist)
dir := before.dir(ident)
_, err = listDir(fs, dir)
require.ErrorIs(t, err, os.ErrNotExist)
// Assert the post-migration path present.
_, err = fs.Stat(after.sszPath(ident))
require.NoError(t, err)
}
func TestMigrations(t *testing.T) {
cases := []struct {
name string
forwardLayout string
backwardLayout string
targets []migrationTestTarget
}{
{
name: "all need migration",
backwardLayout: LayoutNameFlat,
forwardLayout: LayoutNameByEpoch,
targets: []migrationTestTarget{
{
ident: ezIdent(t, "0x0125e54c64c925018c9296965a5b622d9f5ab626c10917860dcfb6aa09a0a00b", 1234, 0),
},
{
ident: ezIdent(t, "0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86", 5330, 0),
slotOffset: 31,
},
{
ident: ezIdent(t, "0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86", 5330, 1),
slotOffset: 31,
},
{
ident: ezIdent(t, "0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c", 16777216, 0),
slotOffset: 16,
},
},
},
{
name: "mix old and new",
backwardLayout: LayoutNameFlat,
forwardLayout: LayoutNameByEpoch,
targets: []migrationTestTarget{
{
ident: ezIdent(t, "0x0125e54c64c925018c9296965a5b622d9f5ab626c10917860dcfb6aa09a0a00b", 1234, 0),
},
{
ident: ezIdent(t, "0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86", 5330, 0),
slotOffset: 31,
},
{
ident: ezIdent(t, "0x0127dba6fd30fdbb47e73e861d5c6e602b38ac3ddc945bb6a2fc4e10761e9a86", 5330, 1),
slotOffset: 31,
},
{
ident: ezIdent(t, "0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c", 16777216, 0),
slotOffset: 16,
migrated: true,
},
{
ident: ezIdent(t, "0x0232521756a0b965eab2c2245d7ad85feaeaf5f427cd14d1a7531f9d555b415c", 16777216, 1),
slotOffset: 16,
migrated: true,
},
{
ident: ezIdent(t, "0x42eabe3d2c125410cd226de6f2825fb7575ab896c3f52e43de1fa29e4c809aba", 16777217, 0),
slotOffset: 16,
migrated: true,
},
{
ident: ezIdent(t, "0x666cea5034e22bd3b849cb33914cad59afd88ee08e4d5bc0e997411c945fbc1d", 11235, 1),
migrated: true,
},
},
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
t.Run("forward", func(t *testing.T) {
testMigration(t, c.forwardLayout, c.backwardLayout, c.targets)
})
// run the same test in reverse - to cover both directions while making the test table smaller.
t.Run("backward", func(t *testing.T) {
testMigration(t, c.forwardLayout, c.backwardLayout, c.targets)
})
})
}
}
func testMigration(t *testing.T, forwardName, backwardName string, targets []migrationTestTarget) {
fs := afero.NewMemMapFs()
cache := newBlobStorageCache()
forward, err := newLayout(forwardName, fs, cache, nil)
require.NoError(t, err)
backward, err := newLayout(backwardName, fs, cache, nil)
require.NoError(t, err)
for _, tar := range targets {
if tar.migrated {
setupTestBlobFile(t, tar.ident, tar.slotOffset, fs, forward)
} else {
setupTestBlobFile(t, tar.ident, tar.slotOffset, fs, backward)
}
}
require.NoError(t, migrateLayout(fs, backward, forward, cache))
for _, tar := range targets {
// Make sure the file wound up in the right spot, according to the forward layout
// and that the old file is gone, according to the backward layout.
testAssertFsMigrated(t, fs, tar.ident, backward, forward)
entry, ok := cache.get(tar.ident.root)
// we only expect cache to be populated here by files that needed to be moved.
if !tar.migrated {
require.Equal(t, true, ok)
require.Equal(t, true, entry.HasIndex(tar.ident.index))
require.Equal(t, tar.ident.epoch, entry.epoch)
}
}
// Run migration in reverse - testing "undo"
cache = newBlobStorageCache()
forward, err = newLayout(forwardName, fs, cache, nil)
require.NoError(t, err)
backward, err = newLayout(backwardName, fs, cache, nil)
require.NoError(t, err)
// forward and backward are flipped compared to the above
require.NoError(t, migrateLayout(fs, forward, backward, cache))
for _, tar := range targets {
// just like the above, but forward and backward are flipped
testAssertFsMigrated(t, fs, tar.ident, forward, backward)
entry, ok := cache.get(tar.ident.root)
require.Equal(t, true, ok)
require.Equal(t, true, entry.HasIndex(tar.ident.index))
require.Equal(t, tar.ident.epoch, entry.epoch)
}
}

View File

@@ -4,30 +4,41 @@ import (
"testing"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/spf13/afero"
)
// NewEphemeralBlobStorage should only be used for tests.
// The instance of BlobStorage returned is backed by an in-memory virtual filesystem,
// improving test performance and simplifying cleanup.
func NewEphemeralBlobStorage(t testing.TB) *BlobStorage {
fs := afero.NewMemMapFs()
pruner, err := newBlobPruner(fs, params.BeaconConfig().MinEpochsForBlobsSidecarsRequest, withWarmedCache())
if err != nil {
t.Fatal("test setup issue", err)
}
return &BlobStorage{fs: fs, pruner: pruner}
func NewEphemeralBlobStorage(t testing.TB, opts ...BlobStorageOption) *BlobStorage {
return NewWarmedEphemeralBlobStorageUsingFs(t, afero.NewMemMapFs(), opts...)
}
// NewEphemeralBlobStorageWithFs can be used by tests that want access to the virtual filesystem
// NewEphemeralBlobStorageAndFs can be used by tests that want access to the virtual filesystem
// in order to interact with it outside the parameters of the BlobStorage api.
func NewEphemeralBlobStorageWithFs(t testing.TB) (afero.Fs, *BlobStorage) {
func NewEphemeralBlobStorageAndFs(t testing.TB, opts ...BlobStorageOption) (afero.Fs, *BlobStorage) {
fs := afero.NewMemMapFs()
pruner, err := newBlobPruner(fs, params.BeaconConfig().MinEpochsForBlobsSidecarsRequest, withWarmedCache())
bs := NewWarmedEphemeralBlobStorageUsingFs(t, fs, opts...)
return fs, bs
}
func NewEphemeralBlobStorageUsingFs(t testing.TB, fs afero.Fs, opts ...BlobStorageOption) *BlobStorage {
opts = append(opts,
WithBlobRetentionEpochs(params.BeaconConfig().MinEpochsForBlobsSidecarsRequest),
WithFs(fs))
bs, err := NewBlobStorage(opts...)
if err != nil {
t.Fatal("test setup issue", err)
t.Fatalf("error initializing test BlobStorage, err=%s", err.Error())
}
return fs, &BlobStorage{fs: fs, pruner: pruner}
return bs
}
func NewWarmedEphemeralBlobStorageUsingFs(t testing.TB, fs afero.Fs, opts ...BlobStorageOption) *BlobStorage {
bs := NewEphemeralBlobStorageUsingFs(t, fs, opts...)
bs.WarmCache()
return bs
}
type BlobMocker struct {
@@ -37,17 +48,9 @@ type BlobMocker struct {
// CreateFakeIndices creates empty blob sidecar files at the expected path for the given
// root and indices to influence the result of Indices().
func (bm *BlobMocker) CreateFakeIndices(root [32]byte, indices ...uint64) error {
func (bm *BlobMocker) CreateFakeIndices(root [32]byte, slot primitives.Slot, indices ...uint64) error {
for i := range indices {
n := blobNamer{root: root, index: indices[i]}
if err := bm.fs.MkdirAll(n.dir(), directoryPermissions); err != nil {
return err
}
f, err := bm.fs.Create(n.path())
if err != nil {
return err
}
if err := f.Close(); err != nil {
if err := bm.bs.layout.notify(newBlobIdent(root, slots.ToEpoch(slot), indices[i])); err != nil {
return err
}
}
@@ -56,9 +59,8 @@ func (bm *BlobMocker) CreateFakeIndices(root [32]byte, indices ...uint64) error
// NewEphemeralBlobStorageWithMocker returns a *BlobMocker value in addition to the BlobStorage value.
// BlockMocker encapsulates things blob path construction to avoid leaking implementation details.
func NewEphemeralBlobStorageWithMocker(_ testing.TB) (*BlobMocker, *BlobStorage) {
fs := afero.NewMemMapFs()
bs := &BlobStorage{fs: fs}
func NewEphemeralBlobStorageWithMocker(t testing.TB) (*BlobMocker, *BlobStorage) {
fs, bs := NewEphemeralBlobStorageAndFs(t)
return &BlobMocker{fs: fs, bs: bs}, bs
}
@@ -66,7 +68,7 @@ func NewMockBlobStorageSummarizer(t *testing.T, set map[[32]byte][]int) BlobStor
c := newBlobStorageCache()
for k, v := range set {
for i := range v {
if err := c.ensure(k, 0, uint64(v[i])); err != nil {
if err := c.ensure(blobIdent{root: k, epoch: 0, index: uint64(v[i])}); err != nil {
t.Fatal(err)
}
}

View File

@@ -1,319 +1,67 @@
package filesystem
import (
"context"
"encoding/binary"
"io"
"path"
"path/filepath"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/sirupsen/logrus"
"github.com/spf13/afero"
)
const retentionBuffer primitives.Epoch = 2
var (
errPruningFailures = errors.New("blobs could not be pruned for some roots")
errNotBlobSSZ = errors.New("not a blob ssz file")
)
var errNotBlobSSZ = errors.New("not a blob ssz file")
// blobPruner keeps track of the tail end of the retention period, based only the blobs it has seen via the notify method.
// If the retention period advances in response to notify being called,
// the pruner will invoke the pruneBefore method of the given layout in a new goroutine.
// The details of pruning are left entirely to the layout, with the pruner's only responsibility being to
// schedule just one pruning operation at a time, for each forward movement of the minimum retention epoch.
type blobPruner struct {
sync.Mutex
prunedBefore atomic.Uint64
windowSize primitives.Slot
cache *blobStorageCache
cacheReady chan struct{}
warmed bool
fs afero.Fs
mu sync.Mutex
prunedBefore atomic.Uint64
retentionPeriod primitives.Epoch
}
type prunerOpt func(*blobPruner) error
func withWarmedCache() prunerOpt {
return func(p *blobPruner) error {
return p.warmCache()
}
func newBlobPruner(retain primitives.Epoch) *blobPruner {
p := &blobPruner{retentionPeriod: retain + retentionBuffer}
return p
}
func newBlobPruner(fs afero.Fs, retain primitives.Epoch, opts ...prunerOpt) (*blobPruner, error) {
r, err := slots.EpochStart(retain + retentionBuffer)
if err != nil {
return nil, errors.Wrap(err, "could not set retentionSlots")
}
cw := make(chan struct{})
p := &blobPruner{fs: fs, windowSize: r, cache: newBlobStorageCache(), cacheReady: cw}
for _, o := range opts {
if err := o(p); err != nil {
return nil, err
}
}
return p, nil
}
// notify updates the pruner's view of root->blob mappings. This allows the pruner to build a cache
// of root->slot mappings and decide when to evict old blobs based on the age of present blobs.
func (p *blobPruner) notify(root [32]byte, latest primitives.Slot, idx uint64) error {
if err := p.cache.ensure(root, latest, idx); err != nil {
return err
}
pruned := uint64(windowMin(latest, p.windowSize))
if p.prunedBefore.Swap(pruned) == pruned {
return nil
// notify returns a channel that is closed when the pruning operation is complete.
// This is useful for tests, but at runtime fsLayouts or BlobStorage should not wait for completion.
func (p *blobPruner) notify(latest primitives.Epoch, layout fsLayout) chan struct{} {
done := make(chan struct{})
floor := periodFloor(latest, p.retentionPeriod)
if primitives.Epoch(p.prunedBefore.Swap(uint64(floor))) >= floor {
// Only trigger pruning if the atomic swap changed the previous value of prunedBefore.
close(done)
return done
}
go func() {
p.Lock()
defer p.Unlock()
if err := p.prune(primitives.Slot(pruned)); err != nil {
log.WithError(err).Errorf("Failed to prune blobs from slot %d", latest)
p.mu.Lock()
start := time.Now()
defer p.mu.Unlock()
sum, err := layout.pruneBefore(floor)
if err != nil {
log.WithError(err).WithFields(sum.LogFields()).Warn("Encountered errors during blob pruning.")
}
log.WithFields(logrus.Fields{
"upToEpoch": floor,
"duration": time.Since(start).String(),
"filesRemoved": sum.blobsPruned,
}).Debug("Pruned old blobs")
blobsPrunedCounter.Add(float64(sum.blobsPruned))
close(done)
}()
return nil
return done
}
func windowMin(latest, offset primitives.Slot) primitives.Slot {
// Safely compute the first slot in the epoch for the latest slot
latest = latest - latest%params.BeaconConfig().SlotsPerEpoch
if latest < offset {
func periodFloor(latest, period primitives.Epoch) primitives.Epoch {
if latest < period {
return 0
}
return latest - offset
}
func (p *blobPruner) warmCache() error {
p.Lock()
defer func() {
if !p.warmed {
p.warmed = true
close(p.cacheReady)
}
p.Unlock()
}()
if err := p.prune(0); err != nil {
return err
}
return nil
}
func (p *blobPruner) waitForCache(ctx context.Context) (*blobStorageCache, error) {
select {
case <-p.cacheReady:
return p.cache, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
// Prune prunes blobs in the base directory based on the retention epoch.
// It deletes blobs older than currentEpoch - (retentionEpochs+bufferEpochs).
// This is so that we keep a slight buffer and blobs are deleted after n+2 epochs.
func (p *blobPruner) prune(pruneBefore primitives.Slot) error {
start := time.Now()
totalPruned, totalErr := 0, 0
// Customize logging/metrics behavior for the initial cache warmup when slot=0.
// We'll never see a prune request for slot 0, unless this is the initial call to warm up the cache.
if pruneBefore == 0 {
defer func() {
log.WithField("duration", time.Since(start).String()).Debug("Warmed up pruner cache")
}()
} else {
defer func() {
log.WithFields(logrus.Fields{
"upToEpoch": slots.ToEpoch(pruneBefore),
"duration": time.Since(start).String(),
"filesRemoved": totalPruned,
}).Debug("Pruned old blobs")
blobsPrunedCounter.Add(float64(totalPruned))
}()
}
entries, err := listDir(p.fs, ".")
if err != nil {
return errors.Wrap(err, "unable to list root blobs directory")
}
dirs := filter(entries, filterRoot)
for _, dir := range dirs {
pruned, err := p.tryPruneDir(dir, pruneBefore)
if err != nil {
totalErr += 1
log.WithError(err).WithField("directory", dir).Error("Unable to prune directory")
}
totalPruned += pruned
}
if totalErr > 0 {
return errors.Wrapf(errPruningFailures, "pruning failed for %d root directories", totalErr)
}
return nil
}
func shouldRetain(slot, pruneBefore primitives.Slot) bool {
return slot >= pruneBefore
}
func (p *blobPruner) tryPruneDir(dir string, pruneBefore primitives.Slot) (int, error) {
root, err := rootFromDir(dir)
if err != nil {
return 0, errors.Wrapf(err, "invalid directory, could not parse subdir as root %s", dir)
}
slot, slotCached := p.cache.slot(root)
// Return early if the slot is cached and doesn't need pruning.
if slotCached && shouldRetain(slot, pruneBefore) {
return 0, nil
}
// entries will include things that aren't ssz files, like dangling .part files. We need these to
// completely clean up the directory.
entries, err := listDir(p.fs, dir)
if err != nil {
return 0, errors.Wrapf(err, "failed to list blobs in directory %s", dir)
}
// scFiles filters the dir listing down to the ssz encoded BlobSidecar files. This allows us to peek
// at the first one in the list to figure out the slot.
scFiles := filter(entries, filterSsz)
if len(scFiles) == 0 {
log.WithField("dir", dir).Warn("Pruner ignoring directory with no blob files")
return 0, nil
}
if !slotCached {
slot, err = slotFromFile(path.Join(dir, scFiles[0]), p.fs)
if err != nil {
return 0, errors.Wrapf(err, "slot could not be read from blob file %s", scFiles[0])
}
for i := range scFiles {
idx, err := idxFromPath(scFiles[i])
if err != nil {
return 0, errors.Wrapf(err, "index could not be determined for blob file %s", scFiles[i])
}
if err := p.cache.ensure(root, slot, idx); err != nil {
return 0, errors.Wrapf(err, "could not update prune cache for blob file %s", scFiles[i])
}
}
if shouldRetain(slot, pruneBefore) {
return 0, nil
}
}
removed := 0
for _, fname := range entries {
fullName := path.Join(dir, fname)
if err := p.fs.Remove(fullName); err != nil {
return removed, errors.Wrapf(err, "unable to remove %s", fullName)
}
// Don't count other files that happen to be in the dir, like dangling .part files.
if filterSsz(fname) {
removed += 1
}
// Log a warning whenever we clean up a .part file
if filterPart(fullName) {
log.WithField("file", fullName).Warn("Deleting abandoned blob .part file")
}
}
if err := p.fs.Remove(dir); err != nil {
return removed, errors.Wrapf(err, "unable to remove blob directory %s", dir)
}
p.cache.evict(root)
return len(scFiles), nil
}
func idxFromPath(fname string) (uint64, error) {
fname = path.Base(fname)
if filepath.Ext(fname) != dotSszExt {
return 0, errors.Wrap(errNotBlobSSZ, "does not have .ssz extension")
}
parts := strings.Split(fname, ".")
if len(parts) != 2 {
return 0, errors.Wrap(errNotBlobSSZ, "unexpected filename structure (want <index>.ssz)")
}
return strconv.ParseUint(parts[0], 10, 64)
}
func rootFromDir(dir string) ([32]byte, error) {
subdir := filepath.Base(dir) // end of the path should be the blob directory, named by hex encoding of root
root, err := stringToRoot(subdir)
if err != nil {
return root, errors.Wrapf(err, "invalid directory, could not parse subdir as root %s", dir)
}
return root, nil
}
// Read slot from marshaled BlobSidecar data in the given file. See slotFromBlob for details.
func slotFromFile(file string, fs afero.Fs) (primitives.Slot, error) {
f, err := fs.Open(file)
if err != nil {
return 0, err
}
defer func() {
if err := f.Close(); err != nil {
log.WithError(err).Errorf("Could not close blob file")
}
}()
return slotFromBlob(f)
}
// slotFromBlob reads the ssz data of a file at the specified offset (8 + 131072 + 48 + 48 = 131176 bytes),
// which is calculated based on the size of the BlobSidecar struct and is based on the size of the fields
// preceding the slot information within SignedBeaconBlockHeader.
func slotFromBlob(at io.ReaderAt) (primitives.Slot, error) {
b := make([]byte, 8)
_, err := at.ReadAt(b, 131176)
if err != nil {
return 0, err
}
rawSlot := binary.LittleEndian.Uint64(b)
return primitives.Slot(rawSlot), nil
}
func listDir(fs afero.Fs, dir string) ([]string, error) {
top, err := fs.Open(dir)
if err != nil {
return nil, errors.Wrap(err, "failed to open directory descriptor")
}
defer func() {
if err := top.Close(); err != nil {
log.WithError(err).Errorf("Could not close file %s", dir)
}
}()
// re the -1 param: "If n <= 0, Readdirnames returns all the names from the directory in a single slice"
dirs, err := top.Readdirnames(-1)
if err != nil {
return nil, errors.Wrap(err, "failed to read directory listing")
}
return dirs, nil
}
func filter(entries []string, filt func(string) bool) []string {
filtered := make([]string, 0, len(entries))
for i := range entries {
if filt(entries[i]) {
filtered = append(filtered, entries[i])
}
}
return filtered
}
func filterRoot(s string) bool {
return strings.HasPrefix(s, "0x")
}
var dotSszExt = "." + sszExt
var dotPartExt = "." + partExt
func filterSsz(s string) bool {
return filepath.Ext(s) == dotSszExt
}
func filterPart(s string) bool {
return filepath.Ext(s) == dotPartExt
return latest - period
}

View File

@@ -1,394 +1,197 @@
package filesystem
import (
"bytes"
"context"
"fmt"
"math"
"encoding/binary"
"os"
"path"
"sort"
"testing"
"time"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/verification"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
"github.com/prysmaticlabs/prysm/v5/testing/require"
"github.com/prysmaticlabs/prysm/v5/testing/util"
"github.com/prysmaticlabs/prysm/v5/time/slots"
"github.com/spf13/afero"
)
func TestTryPruneDir_CachedNotExpired(t *testing.T) {
fs := afero.NewMemMapFs()
pr, err := newBlobPruner(fs, 0)
require.NoError(t, err)
slot := pr.windowSize
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, params.BeaconConfig().MaxBlobsPerBlock(slot))
sc, err := verification.BlobSidecarNoop(sidecars[0])
require.NoError(t, err)
rootStr := rootString(sc.BlockRoot())
// This slot is right on the edge of what would need to be pruned, so by adding it to the cache and
// skipping any other test setup, we can be certain the hot cache path never touches the filesystem.
require.NoError(t, pr.cache.ensure(sc.BlockRoot(), sc.Slot(), 0))
pruned, err := pr.tryPruneDir(rootStr, pr.windowSize)
require.NoError(t, err)
require.Equal(t, 0, pruned)
type prunerScenario struct {
name string
prunedBefore primitives.Epoch
retentionPeriod primitives.Epoch
latest primitives.Epoch
expected pruneExpectation
}
func TestCacheWarmFail(t *testing.T) {
fs := afero.NewMemMapFs()
n := blobNamer{root: bytesutil.ToBytes32([]byte("derp")), index: 0}
bp := n.path()
mkdir := path.Dir(bp)
require.NoError(t, fs.MkdirAll(mkdir, directoryPermissions))
// Create an empty blob index in the fs by touching the file at a seemingly valid path.
fi, err := fs.Create(bp)
require.NoError(t, err)
require.NoError(t, fi.Close())
// Cache warm should fail due to the unexpected EOF.
pr, err := newBlobPruner(fs, 0)
require.NoError(t, err)
require.ErrorIs(t, pr.warmCache(), errPruningFailures)
// The cache warm has finished, so calling waitForCache with a super short deadline
// should not block or hit the context deadline.
ctx := context.Background()
ctx, cancel := context.WithDeadline(ctx, time.Now().Add(1*time.Millisecond))
defer cancel()
c, err := pr.waitForCache(ctx)
// We will get an error and a nil value for the cache if we hit the deadline.
require.NoError(t, err)
require.NotNil(t, c)
type pruneExpectation struct {
called bool
arg primitives.Epoch
summary *pruneSummary
err error
}
func TestTryPruneDir_CachedExpired(t *testing.T) {
t.Run("empty directory", func(t *testing.T) {
fs := afero.NewMemMapFs()
pr, err := newBlobPruner(fs, 0)
require.NoError(t, err)
var slot primitives.Slot = 0
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, 1)
sc, err := verification.BlobSidecarNoop(sidecars[0])
require.NoError(t, err)
rootStr := rootString(sc.BlockRoot())
require.NoError(t, fs.Mkdir(rootStr, directoryPermissions)) // make empty directory
require.NoError(t, pr.cache.ensure(sc.BlockRoot(), sc.Slot(), 0))
pruned, err := pr.tryPruneDir(rootStr, slot+1)
require.NoError(t, err)
require.Equal(t, 0, pruned)
})
t.Run("blobs to delete", func(t *testing.T) {
fs, bs := NewEphemeralBlobStorageWithFs(t)
var slot primitives.Slot = 0
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, 2)
scs, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
require.NoError(t, bs.Save(scs[0]))
require.NoError(t, bs.Save(scs[1]))
// check that the root->slot is cached
root := scs[0].BlockRoot()
rootStr := rootString(root)
cs, cok := bs.pruner.cache.slot(scs[0].BlockRoot())
require.Equal(t, true, cok)
require.Equal(t, slot, cs)
// ensure that we see the saved files in the filesystem
files, err := listDir(fs, rootStr)
require.NoError(t, err)
require.Equal(t, 2, len(files))
pruned, err := bs.pruner.tryPruneDir(rootStr, slot+1)
require.NoError(t, err)
require.Equal(t, 2, pruned)
files, err = listDir(fs, rootStr)
require.ErrorIs(t, err, os.ErrNotExist)
require.Equal(t, 0, len(files))
})
func (e *pruneExpectation) record(before primitives.Epoch) (*pruneSummary, error) {
e.called = true
e.arg = before
if e.summary == nil {
e.summary = &pruneSummary{}
}
return e.summary, e.err
}
func TestTryPruneDir_SlotFromFile(t *testing.T) {
t.Run("expired blobs deleted", func(t *testing.T) {
fs, bs := NewEphemeralBlobStorageWithFs(t)
var slot primitives.Slot = 0
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, 2)
scs, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
require.NoError(t, bs.Save(scs[0]))
require.NoError(t, bs.Save(scs[1]))
// check that the root->slot is cached
root := scs[0].BlockRoot()
rootStr := rootString(root)
cs, ok := bs.pruner.cache.slot(root)
require.Equal(t, true, ok)
require.Equal(t, slot, cs)
// evict it from the cache so that we trigger the file read path
bs.pruner.cache.evict(root)
_, ok = bs.pruner.cache.slot(root)
require.Equal(t, false, ok)
// ensure that we see the saved files in the filesystem
files, err := listDir(fs, rootStr)
require.NoError(t, err)
require.Equal(t, 2, len(files))
pruned, err := bs.pruner.tryPruneDir(rootStr, slot+1)
require.NoError(t, err)
require.Equal(t, 2, pruned)
files, err = listDir(fs, rootStr)
require.ErrorIs(t, err, os.ErrNotExist)
require.Equal(t, 0, len(files))
})
t.Run("not expired, intact", func(t *testing.T) {
fs, bs := NewEphemeralBlobStorageWithFs(t)
// Set slot equal to the window size, so it should be retained.
slot := bs.pruner.windowSize
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, 2)
scs, err := verification.BlobSidecarSliceNoop(sidecars)
require.NoError(t, err)
require.NoError(t, bs.Save(scs[0]))
require.NoError(t, bs.Save(scs[1]))
// Evict slot mapping from the cache so that we trigger the file read path.
root := scs[0].BlockRoot()
rootStr := rootString(root)
bs.pruner.cache.evict(root)
_, ok := bs.pruner.cache.slot(root)
require.Equal(t, false, ok)
// Ensure that we see the saved files in the filesystem.
files, err := listDir(fs, rootStr)
require.NoError(t, err)
require.Equal(t, 2, len(files))
// This should use the slotFromFile code (simulating restart).
// Setting pruneBefore == slot, so that the slot will be outside the window (at the boundary).
pruned, err := bs.pruner.tryPruneDir(rootStr, slot)
require.NoError(t, err)
require.Equal(t, 0, pruned)
// Ensure files are still present.
files, err = listDir(fs, rootStr)
require.NoError(t, err)
require.Equal(t, 2, len(files))
})
}
func TestSlotFromBlob(t *testing.T) {
cases := []struct {
slot primitives.Slot
}{
{slot: 0},
{slot: 2},
{slot: 1123581321},
{slot: math.MaxUint64},
func TestPrunerNotify(t *testing.T) {
defaultRetention := params.BeaconConfig().MinEpochsForBlobsSidecarsRequest
cases := []prunerScenario{
{
name: "last epoch of period",
retentionPeriod: defaultRetention,
prunedBefore: 11235,
latest: defaultRetention + 11235,
expected: pruneExpectation{called: false},
},
{
name: "within period",
retentionPeriod: defaultRetention,
prunedBefore: 11235,
latest: 11235 + defaultRetention - 1,
expected: pruneExpectation{called: false},
},
{
name: "triggers",
retentionPeriod: defaultRetention,
prunedBefore: 11235,
latest: 11235 + 1 + defaultRetention,
expected: pruneExpectation{called: true, arg: 11235 + 1},
},
{
name: "from zero - before first period",
retentionPeriod: defaultRetention,
prunedBefore: 0,
latest: defaultRetention - 1,
expected: pruneExpectation{called: false},
},
{
name: "from zero - at boundary",
retentionPeriod: defaultRetention,
prunedBefore: 0,
latest: defaultRetention,
expected: pruneExpectation{called: false},
},
{
name: "from zero - triggers",
retentionPeriod: defaultRetention,
prunedBefore: 0,
latest: defaultRetention + 1,
expected: pruneExpectation{called: true, arg: 1},
},
}
for _, c := range cases {
t.Run(fmt.Sprintf("slot %d", c.slot), func(t *testing.T) {
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, c.slot, 1)
sc := sidecars[0]
enc, err := sc.MarshalSSZ()
require.NoError(t, err)
slot, err := slotFromBlob(bytes.NewReader(enc))
require.NoError(t, err)
require.Equal(t, c.slot, slot)
t.Run(c.name, func(t *testing.T) {
actual := &pruneExpectation{}
l := &mockLayout{pruneBeforeFunc: actual.record}
pruner := &blobPruner{retentionPeriod: c.retentionPeriod}
pruner.prunedBefore.Store(uint64(c.prunedBefore))
done := pruner.notify(c.latest, l)
<-done
require.Equal(t, c.expected.called, actual.called)
require.Equal(t, c.expected.arg, actual.arg)
})
}
}
func TestSlotFromFile(t *testing.T) {
cases := []struct {
slot primitives.Slot
}{
{slot: 0},
{slot: 2},
{slot: 1123581321},
{slot: math.MaxUint64},
}
for _, c := range cases {
t.Run(fmt.Sprintf("slot %d", c.slot), func(t *testing.T) {
fs, bs := NewEphemeralBlobStorageWithFs(t)
_, sidecars := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, c.slot, 1)
sc, err := verification.BlobSidecarNoop(sidecars[0])
require.NoError(t, err)
require.NoError(t, bs.Save(sc))
fname := namerForSidecar(sc)
sszPath := fname.path()
slot, err := slotFromFile(sszPath, fs)
require.NoError(t, err)
require.Equal(t, c.slot, slot)
})
}
}
type dirFiles struct {
name string
isDir bool
children []dirFiles
}
func (df dirFiles) reify(t *testing.T, fs afero.Fs, base string) {
fullPath := path.Join(base, df.name)
if df.isDir {
if df.name != "" {
require.NoError(t, fs.Mkdir(fullPath, directoryPermissions))
}
for _, c := range df.children {
c.reify(t, fs, fullPath)
}
} else {
fp, err := fs.Create(fullPath)
func testSetupBlobIdentPaths(t *testing.T, fs afero.Fs, bs *BlobStorage, idents []testIdent) []blobIdent {
created := make([]blobIdent, len(idents))
for i, id := range idents {
slot, err := slots.EpochStart(id.epoch)
require.NoError(t, err)
_, err = fp.WriteString("derp")
slot += id.offset
_, scs := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, slot, 1)
sc := verification.FakeVerifyForTest(t, scs[0])
require.NoError(t, bs.Save(sc))
ident := identForSidecar(sc)
_, err = fs.Stat(bs.layout.sszPath(ident))
require.NoError(t, err)
created[i] = ident
}
return created
}
func testAssertBlobsPruned(t *testing.T, fs afero.Fs, bs *BlobStorage, pruned, remain []blobIdent) {
for _, id := range pruned {
_, err := fs.Stat(bs.layout.sszPath(id))
require.NotNil(t, err)
require.Equal(t, true, os.IsNotExist(err))
}
for _, id := range remain {
_, err := fs.Stat(bs.layout.sszPath(id))
require.NoError(t, err)
}
}
func (df dirFiles) childNames() []string {
cn := make([]string, len(df.children))
for i := range df.children {
cn[i] = df.children[i].name
}
return cn
type testIdent struct {
blobIdent
offset primitives.Slot
}
func TestListDir(t *testing.T) {
fs := afero.NewMemMapFs()
// parent directory
fsLayout := dirFiles{isDir: true}
// break out each subdir for easier assertions
notABlob := dirFiles{name: "notABlob", isDir: true}
childlessBlob := dirFiles{name: "0x0987654321", isDir: true}
blobWithSsz := dirFiles{name: "0x1123581321", isDir: true,
children: []dirFiles{{name: "1.ssz"}, {name: "2.ssz"}},
func testRoots(n int) [][32]byte {
roots := make([][32]byte, n)
for i := range roots {
binary.LittleEndian.PutUint32(roots[i][:], uint32(1+i))
}
blobWithSszAndTmp := dirFiles{name: "0x1234567890", isDir: true,
children: []dirFiles{{name: "5.ssz"}, {name: "0.part"}}}
fsLayout.children = append(fsLayout.children,
notABlob, childlessBlob, blobWithSsz, blobWithSszAndTmp)
return roots
}
topChildren := make([]string, len(fsLayout.children))
for i := range fsLayout.children {
topChildren[i] = fsLayout.children[i].name
}
fsLayout.reify(t, fs, "")
func TestLayoutPruneBefore(t *testing.T) {
roots := testRoots(10)
cases := []struct {
name string
dirPath string
expected []string
filter func(string) bool
err error
name string
pruned []testIdent
remain []testIdent
pruneBefore primitives.Epoch
err error
sum pruneSummary
}{
{
name: "non-existent",
dirPath: "derp",
expected: []string{},
err: os.ErrNotExist,
},
{
name: "empty",
dirPath: childlessBlob.name,
expected: []string{},
},
{
name: "top",
dirPath: ".",
expected: topChildren,
},
{
name: "custom filter: only notABlob",
dirPath: ".",
expected: []string{notABlob.name},
filter: func(s string) bool {
return s == notABlob.name
name: "none pruned",
pruneBefore: 1,
pruned: []testIdent{},
remain: []testIdent{
{offset: 1, blobIdent: blobIdent{root: roots[0], epoch: 1, index: 0}},
{offset: 1, blobIdent: blobIdent{root: roots[1], epoch: 1, index: 0}},
},
},
{
name: "root filter",
dirPath: ".",
expected: []string{childlessBlob.name, blobWithSsz.name, blobWithSszAndTmp.name},
filter: filterRoot,
},
{
name: "ssz filter",
dirPath: blobWithSsz.name,
expected: blobWithSsz.childNames(),
filter: filterSsz,
},
{
name: "ssz mixed filter",
dirPath: blobWithSszAndTmp.name,
expected: []string{"5.ssz"},
filter: filterSsz,
name: "expected pruned before epoch",
pruneBefore: 3,
pruned: []testIdent{
{offset: 0, blobIdent: blobIdent{root: roots[0], epoch: 1, index: 0}},
{offset: 31, blobIdent: blobIdent{root: roots[1], epoch: 1, index: 5}},
{offset: 0, blobIdent: blobIdent{root: roots[2], epoch: 2, index: 0}},
{offset: 31, blobIdent: blobIdent{root: roots[3], epoch: 2, index: 3}},
},
remain: []testIdent{
{offset: 0, blobIdent: blobIdent{root: roots[4], epoch: 3, index: 2}}, // boundary
{offset: 31, blobIdent: blobIdent{root: roots[5], epoch: 3, index: 0}}, // boundary
{offset: 0, blobIdent: blobIdent{root: roots[6], epoch: 4, index: 1}},
{offset: 31, blobIdent: blobIdent{root: roots[7], epoch: 4, index: 5}},
},
sum: pruneSummary{blobsPruned: 4},
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
result, err := listDir(fs, c.dirPath)
if c.filter != nil {
result = filter(result, c.filter)
}
if c.err != nil {
require.ErrorIs(t, err, c.err)
require.Equal(t, 0, len(result))
} else {
require.NoError(t, err)
sort.Strings(c.expected)
sort.Strings(result)
require.DeepEqual(t, c.expected, result)
}
})
}
}
func TestRootFromDir(t *testing.T) {
cases := []struct {
name string
dir string
err error
root [32]byte
}{
{
name: "happy path",
dir: "0xffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5cb",
root: [32]byte{255, 255, 135, 94, 29, 152, 92, 92, 203, 33, 72, 148, 152, 63, 36, 40,
237, 178, 113, 240, 248, 123, 104, 186, 112, 16, 228, 169, 157, 243, 181, 203},
},
{
name: "too short",
dir: "0xffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5c",
err: errInvalidRootString,
},
{
name: "too log",
dir: "0xffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5cbb",
err: errInvalidRootString,
},
{
name: "missing prefix",
dir: "ffff875e1d985c5ccb214894983f2428edb271f0f87b68ba7010e4a99df3b5cb",
err: errInvalidRootString,
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
root, err := stringToRoot(c.dir)
fs, bs := NewEphemeralBlobStorageAndFs(t, WithLayout(LayoutNameByEpoch))
pruned := testSetupBlobIdentPaths(t, fs, bs, c.pruned)
remain := testSetupBlobIdentPaths(t, fs, bs, c.remain)
sum, err := bs.layout.pruneBefore(c.pruneBefore)
if c.err != nil {
require.ErrorIs(t, err, c.err)
return
}
require.NoError(t, err)
require.Equal(t, c.root, root)
testAssertBlobsPruned(t, fs, bs, pruned, remain)
require.Equal(t, c.sum.blobsPruned, sum.blobsPruned)
require.Equal(t, len(c.pruned), sum.blobsPruned)
require.Equal(t, len(c.sum.failedRemovals), len(sum.failedRemovals))
})
}
}

View File

@@ -110,6 +110,7 @@ type HeadAccessDatabase interface {
// Block related methods.
HeadBlock(ctx context.Context) (interfaces.ReadOnlySignedBeaconBlock, error)
HeadBlockRoot() ([32]byte, error)
SaveHeadBlockRoot(ctx context.Context, blockRoot [32]byte) error
// Genesis operations.

View File

@@ -70,6 +70,21 @@ func (s *Store) OriginCheckpointBlockRoot(ctx context.Context) ([32]byte, error)
return root, err
}
// HeadBlockRoot returns the latest canonical block root in the Ethereum Beacon Chain.
func (s *Store) HeadBlockRoot() ([32]byte, error) {
var root [32]byte
err := s.db.View(func(tx *bolt.Tx) error {
bkt := tx.Bucket(blocksBucket)
headRoot := bkt.Get(headBlockRootKey)
if headRoot == nil {
return errors.New("no head block root found")
}
copy(root[:], headRoot)
return nil
})
return root, err
}
// HeadBlock returns the latest canonical block in the Ethereum Beacon Chain.
func (s *Store) HeadBlock(ctx context.Context) (interfaces.ReadOnlySignedBeaconBlock, error) {
ctx, span := trace.StartSpan(ctx, "BeaconDB.HeadBlock")

View File

@@ -3,15 +3,13 @@ package kv
import (
"context"
"fmt"
bolt "go.etcd.io/bbolt"
"testing"
"time"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/ethereum/go-ethereum/common"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/db/filters"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
@@ -22,6 +20,7 @@ import (
"github.com/prysmaticlabs/prysm/v5/testing/assert"
"github.com/prysmaticlabs/prysm/v5/testing/require"
"github.com/prysmaticlabs/prysm/v5/testing/util"
bolt "go.etcd.io/bbolt"
"google.golang.org/protobuf/proto"
)

View File

@@ -215,7 +215,7 @@ func (s *Store) LightClientUpdates(ctx context.Context, startPeriod, endPeriod u
if err != nil {
return nil, err
}
return updates, err
return updates, nil
}
func (s *Store) LightClientUpdate(ctx context.Context, period uint64) (interfaces.LightClientUpdate, error) {

View File

@@ -518,9 +518,9 @@ func (s *Store) unmarshalState(_ context.Context, enc []byte, validatorEntries [
switch {
case hasFuluKey(enc):
protoState := &ethpb.BeaconStateFulu{}
protoState := &ethpb.BeaconStateElectra{}
if err := protoState.UnmarshalSSZ(enc[len(fuluKey):]); err != nil {
return nil, errors.Wrap(err, "failed to unmarshal encoding for Electra")
return nil, errors.Wrap(err, "failed to unmarshal encoding for Fulu")
}
ok, err := s.isStateValidatorMigrationOver()
if err != nil {
@@ -690,7 +690,7 @@ func marshalState(ctx context.Context, st state.ReadOnlyBeaconState) ([]byte, er
}
return snappy.Encode(nil, append(ElectraKey, rawObj...)), nil
case version.Fulu:
rState, ok := st.ToProtoUnsafe().(*ethpb.BeaconStateFulu)
rState, ok := st.ToProtoUnsafe().(*ethpb.BeaconStateElectra)
if !ok {
return nil, errors.New("non valid inner state")
}

View File

@@ -103,6 +103,7 @@ go_test(
"//beacon-chain/core/helpers:go_default_library",
"//beacon-chain/core/signing:go_default_library",
"//beacon-chain/db:go_default_library",
"//beacon-chain/db/filesystem:go_default_library",
"//beacon-chain/db/testing:go_default_library",
"//beacon-chain/execution/testing:go_default_library",
"//beacon-chain/execution/types:go_default_library",

View File

@@ -36,17 +36,19 @@ var (
NewPayloadMethod,
NewPayloadMethodV2,
NewPayloadMethodV3,
NewPayloadMethodV4,
ForkchoiceUpdatedMethod,
ForkchoiceUpdatedMethodV2,
ForkchoiceUpdatedMethodV3,
GetPayloadMethod,
GetPayloadMethodV2,
GetPayloadMethodV3,
GetPayloadMethodV4,
GetPayloadBodiesByHashV1,
GetPayloadBodiesByRangeV1,
}
electraEngineEndpoints = []string{
NewPayloadMethodV4,
GetPayloadMethodV4,
}
)
const (
@@ -105,7 +107,7 @@ type Reconstructor interface {
ReconstructFullBellatrixBlockBatch(
ctx context.Context, blindedBlocks []interfaces.ReadOnlySignedBeaconBlock,
) ([]interfaces.SignedBeaconBlock, error)
ReconstructBlobSidecars(ctx context.Context, block interfaces.ReadOnlySignedBeaconBlock, blockRoot [32]byte, indices []bool) ([]blocks.VerifiedROBlob, error)
ReconstructBlobSidecars(ctx context.Context, block interfaces.ReadOnlySignedBeaconBlock, blockRoot [32]byte, hi func(uint64) bool) ([]blocks.VerifiedROBlob, error)
}
// EngineCaller defines a client that can interact with an Ethereum
@@ -296,6 +298,10 @@ func (s *Service) ExchangeCapabilities(ctx context.Context) ([]string, error) {
ctx, span := trace.StartSpan(ctx, "powchain.engine-api-client.ExchangeCapabilities")
defer span.End()
// Only check for electra related engine methods if it has been activated.
if params.ElectraEnabled() {
supportedEngineEndpoints = append(supportedEngineEndpoints, electraEngineEndpoints...)
}
var result []string
err := s.rpcClient.CallContext(ctx, &result, ExchangeCapabilities, supportedEngineEndpoints)
if err != nil {
@@ -531,32 +537,23 @@ func (s *Service) ReconstructFullBellatrixBlockBatch(
// It retrieves the KZG commitments from the block body, fetches the associated blobs and proofs,
// and constructs the corresponding verified read-only blob sidecars.
//
// The 'exists' argument is a boolean list (must be the same length as body.BlobKzgCommitments), where each element corresponds to whether a
// particular blob sidecar already exists. If exists[i] is true, the blob for the i-th KZG commitment
// has already been retrieved and does not need to be fetched again from the execution layer (EL).
//
// For example:
// - len(block.Body().BlobKzgCommitments()) == 6
// - If exists = [true, false, true, false, true, false], the function will fetch the blobs
// associated with indices 1, 3, and 5 (since those are marked as non-existent).
// - If exists = [false ... x 6], the function will attempt to fetch all blobs.
//
// Only the blobs that do not already exist (where exists[i] is false) are fetched using the KZG commitments from block body.
func (s *Service) ReconstructBlobSidecars(ctx context.Context, block interfaces.ReadOnlySignedBeaconBlock, blockRoot [32]byte, exists []bool) ([]blocks.VerifiedROBlob, error) {
// The 'hasIndex' argument is a function returns true if the given uint64 blob index already exists on disc.
// Only the blobs that do not already exist (where hasIndex(i) is false)
// will be fetched from the execution engine using the KZG commitments from block body.
func (s *Service) ReconstructBlobSidecars(ctx context.Context, block interfaces.ReadOnlySignedBeaconBlock, blockRoot [32]byte, hasIndex func(uint64) bool) ([]blocks.VerifiedROBlob, error) {
blockBody := block.Block().Body()
kzgCommitments, err := blockBody.BlobKzgCommitments()
if err != nil {
return nil, errors.Wrap(err, "could not get blob KZG commitments")
}
if len(kzgCommitments) > len(exists) {
return nil, fmt.Errorf("length of KZG commitments (%d) is greater than length of exists (%d)", len(kzgCommitments), len(exists))
}
// Collect KZG hashes for non-existing blobs
var kzgHashes []common.Hash
var kzgIndexes []int
for i, commitment := range kzgCommitments {
if !exists[i] {
if !hasIndex(uint64(i)) {
kzgHashes = append(kzgHashes, primitives.ConvertKzgCommitmentToVersionedHash(commitment))
kzgIndexes = append(kzgIndexes, i)
}
}
if len(kzgHashes) == 0 {
@@ -579,27 +576,21 @@ func (s *Service) ReconstructBlobSidecars(ctx context.Context, block interfaces.
// Reconstruct verified blob sidecars
var verifiedBlobs []blocks.VerifiedROBlob
for i, blobIndex := 0, 0; i < len(kzgCommitments); i++ {
if exists[i] {
for i := 0; i < len(kzgHashes); i++ {
if blobs[i] == nil {
continue
}
if blobIndex >= len(blobs) || blobs[blobIndex] == nil {
blobIndex++
continue
}
blob := blobs[blobIndex]
blobIndex++
proof, err := blocks.MerkleProofKZGCommitment(blockBody, i)
blob := blobs[i]
blobIndex := kzgIndexes[i]
proof, err := blocks.MerkleProofKZGCommitment(blockBody, blobIndex)
if err != nil {
log.WithError(err).WithField("index", i).Error("failed to get Merkle proof for KZG commitment")
log.WithError(err).WithField("index", blobIndex).Error("failed to get Merkle proof for KZG commitment")
continue
}
sidecar := &ethpb.BlobSidecar{
Index: uint64(i),
Index: uint64(blobIndex),
Blob: blob.Blob,
KzgCommitment: kzgCommitments[i],
KzgCommitment: kzgCommitments[blobIndex],
KzgProof: blob.KzgProof,
SignedBlockHeader: header,
CommitmentInclusionProof: proof,
@@ -607,14 +598,14 @@ func (s *Service) ReconstructBlobSidecars(ctx context.Context, block interfaces.
roBlob, err := blocks.NewROBlobWithRoot(sidecar, blockRoot)
if err != nil {
log.WithError(err).WithField("index", i).Error("failed to create RO blob with root")
log.WithError(err).WithField("index", blobIndex).Error("failed to create RO blob with root")
continue
}
v := s.blobVerifier(roBlob, verification.ELMemPoolRequirements)
verifiedBlob, err := v.VerifiedROBlob()
if err != nil {
log.WithError(err).WithField("index", i).Error("failed to verify RO blob")
log.WithError(err).WithField("index", blobIndex).Error("failed to verify RO blob")
continue
}

View File

@@ -20,6 +20,7 @@ import (
"github.com/ethereum/go-ethereum/rpc"
"github.com/holiman/uint256"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/db/filesystem"
mocks "github.com/prysmaticlabs/prysm/v5/beacon-chain/execution/testing"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/verification"
fieldparams "github.com/prysmaticlabs/prysm/v5/config/fieldparams"
@@ -2395,6 +2396,12 @@ func Test_ExchangeCapabilities(t *testing.T) {
})
}
func mockSummary(t *testing.T, exists []bool) func(uint64) bool {
hi, err := filesystem.NewBlobStorageSummary(params.BeaconConfig().DenebForkEpoch, exists)
require.NoError(t, err)
return hi.HasIndex
}
func TestReconstructBlobSidecars(t *testing.T) {
client := &Service{capabilityCache: &capabilityCache{}}
b := util.NewBeaconBlockDeneb()
@@ -2408,15 +2415,15 @@ func TestReconstructBlobSidecars(t *testing.T) {
ctx := context.Background()
t.Run("all seen", func(t *testing.T) {
exists := []bool{true, true, true, true, true, true}
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, exists)
hi := mockSummary(t, []bool{true, true, true, true, true, true})
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, hi)
require.NoError(t, err)
require.Equal(t, 0, len(verifiedBlobs))
})
t.Run("get-blobs end point is not supported", func(t *testing.T) {
exists := []bool{true, true, true, true, true, false}
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, exists)
hi := mockSummary(t, []bool{true, true, true, true, true, false})
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, hi)
require.NoError(t, err)
require.Equal(t, 0, len(verifiedBlobs))
})
@@ -2430,8 +2437,8 @@ func TestReconstructBlobSidecars(t *testing.T) {
rpcClient, client := setupRpcClient(t, srv.URL, client)
defer rpcClient.Close()
exists := [6]bool{}
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, exists[:])
hi := mockSummary(t, make([]bool, 6))
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, hi)
require.NoError(t, err)
require.Equal(t, 6, len(verifiedBlobs))
})
@@ -2443,22 +2450,29 @@ func TestReconstructBlobSidecars(t *testing.T) {
rpcClient, client := setupRpcClient(t, srv.URL, client)
defer rpcClient.Close()
exists := []bool{true, false, true, false, true, false}
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, exists)
hi := mockSummary(t, []bool{true, false, true, false, true, false})
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, hi)
require.NoError(t, err)
require.Equal(t, 3, len(verifiedBlobs))
})
t.Run("kzg is longer than exist", func(t *testing.T) {
srv := createBlobServer(t, 3)
t.Run("recovered 3 missing blobs with mutated blob mask", func(t *testing.T) {
exists := []bool{true, false, true, false, true, false}
hi := mockSummary(t, exists)
srv := createBlobServer(t, 3, func() {
// Mutate blob mask
exists[1] = true
exists[3] = true
})
defer srv.Close()
rpcClient, client := setupRpcClient(t, srv.URL, client)
defer rpcClient.Close()
exists := []bool{true, false, true, false, true}
_, err := client.ReconstructBlobSidecars(ctx, sb, r, exists)
require.ErrorContains(t, "length of KZG commitments (6) is greater than length of exists (5)", err)
verifiedBlobs, err := client.ReconstructBlobSidecars(ctx, sb, r, hi)
require.NoError(t, err)
require.Equal(t, 3, len(verifiedBlobs))
})
}
@@ -2472,12 +2486,16 @@ func createRandomKzgCommitments(t *testing.T, num int) [][]byte {
return kzgCommitments
}
func createBlobServer(t *testing.T, numBlobs int) *httptest.Server {
func createBlobServer(t *testing.T, numBlobs int, callbackFuncs ...func()) *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
defer func() {
require.NoError(t, r.Body.Close())
}()
// Execute callback functions for each request.
for _, f := range callbackFuncs {
f()
}
blobs := make([]pb.BlobAndProofJson, numBlobs)
for i := range blobs {

View File

@@ -109,7 +109,7 @@ func (e *EngineClient) ReconstructFullBellatrixBlockBatch(
}
// ReconstructBlobSidecars is a mock implementation of the ReconstructBlobSidecars method.
func (e *EngineClient) ReconstructBlobSidecars(context.Context, interfaces.ReadOnlySignedBeaconBlock, [32]byte, []bool) ([]blocks.VerifiedROBlob, error) {
func (e *EngineClient) ReconstructBlobSidecars(context.Context, interfaces.ReadOnlySignedBeaconBlock, [32]byte, func(uint64) bool) ([]blocks.VerifiedROBlob, error) {
return e.BlobSidecars, e.ErrorBlobSidecars
}

View File

@@ -1,8 +1,6 @@
package doublylinkedtree
import (
"fmt"
fieldparams "github.com/prysmaticlabs/prysm/v5/config/fieldparams"
"github.com/prysmaticlabs/prysm/v5/config/params"
)
@@ -15,7 +13,7 @@ func (f *ForkChoice) applyProposerBoostScore() error {
if s.previousProposerBoostRoot != params.BeaconConfig().ZeroHash {
previousNode, ok := s.nodeByRoot[s.previousProposerBoostRoot]
if !ok || previousNode == nil {
log.WithError(errInvalidProposerBoostRoot).Errorf(fmt.Sprintf("invalid prev root %#x", s.previousProposerBoostRoot))
log.WithError(errInvalidProposerBoostRoot).Errorf("invalid prev root %#x", s.previousProposerBoostRoot)
} else {
previousNode.balance -= s.previousProposerBoostScore
}
@@ -24,7 +22,7 @@ func (f *ForkChoice) applyProposerBoostScore() error {
if s.proposerBoostRoot != params.BeaconConfig().ZeroHash {
currentNode, ok := s.nodeByRoot[s.proposerBoostRoot]
if !ok || currentNode == nil {
log.WithError(errInvalidProposerBoostRoot).Errorf(fmt.Sprintf("invalid current root %#x", s.proposerBoostRoot))
log.WithError(errInvalidProposerBoostRoot).Errorf("invalid current root %#x", s.proposerBoostRoot)
} else {
proposerScore = (s.committeeWeight * params.BeaconConfig().ProposerScoreBoost) / 100
currentNode.balance += proposerScore

View File

@@ -78,7 +78,7 @@ func (f *ForkChoice) ShouldOverrideFCU() (override bool) {
// }
// Only orphan a block if the head LMD vote is weak
if head.weight*100 > f.store.committeeWeight*params.BeaconConfig().ReorgWeightThreshold {
if head.weight*100 > f.store.committeeWeight*params.BeaconConfig().ReorgHeadWeightThreshold {
return
}
@@ -143,7 +143,7 @@ func (f *ForkChoice) GetProposerHead() [32]byte {
}
// Only orphan a block if the head LMD vote is weak
if head.weight*100 > f.store.committeeWeight*params.BeaconConfig().ReorgWeightThreshold {
if head.weight*100 > f.store.committeeWeight*params.BeaconConfig().ReorgHeadWeightThreshold {
return head.root
}

View File

@@ -252,6 +252,13 @@ func (s *Store) tips() ([][32]byte, []primitives.Slot) {
return roots, slots
}
func (f *ForkChoice) HighestReceivedBlockRoot() [32]byte {
if f.store.highestReceivedNode == nil {
return [32]byte{}
}
return f.store.highestReceivedNode.root
}
// HighestReceivedBlockSlot returns the highest slot received by the forkchoice
func (f *ForkChoice) HighestReceivedBlockSlot() primitives.Slot {
if f.store.highestReceivedNode == nil {

View File

@@ -65,6 +65,7 @@ type FastGetter interface {
FinalizedPayloadBlockHash() [32]byte
HasNode([32]byte) bool
HighestReceivedBlockSlot() primitives.Slot
HighestReceivedBlockRoot() [32]byte
HighestReceivedBlockDelay() primitives.Slot
IsCanonical(root [32]byte) bool
IsOptimistic(root [32]byte) (bool, error)

View File

@@ -114,6 +114,13 @@ func (ro *ROForkChoice) HighestReceivedBlockSlot() primitives.Slot {
return ro.getter.HighestReceivedBlockSlot()
}
// HighestReceivedBlockRoot delegates to the underlying forkchoice call, under a lock.
func (ro *ROForkChoice) HighestReceivedBlockRoot() [32]byte {
ro.l.RLock()
defer ro.l.RUnlock()
return ro.getter.HighestReceivedBlockRoot()
}
// HighestReceivedBlockDelay delegates to the underlying forkchoice call, under a lock.
func (ro *ROForkChoice) HighestReceivedBlockDelay() primitives.Slot {
ro.l.RLock()

View File

@@ -29,6 +29,7 @@ const (
unrealizedJustifiedPayloadBlockHashCalled
nodeCountCalled
highestReceivedBlockSlotCalled
highestReceivedBlockRootCalled
highestReceivedBlockDelayCalled
receivedBlocksLastEpochCalled
weightCalled
@@ -252,6 +253,11 @@ func (ro *mockROForkchoice) HighestReceivedBlockSlot() primitives.Slot {
return 0
}
func (ro *mockROForkchoice) HighestReceivedBlockRoot() [32]byte {
ro.calls = append(ro.calls, highestReceivedBlockRootCalled)
return [32]byte{}
}
func (ro *mockROForkchoice) HighestReceivedBlockDelay() primitives.Slot {
ro.calls = append(ro.calls, highestReceivedBlockDelayCalled)
return 0

View File

@@ -166,6 +166,23 @@ func (s *Service) processIncludedAttestation(ctx context.Context, state state.Be
}
}
// processSingleAttestation logs when the beacon node observes a single attestation from tracked validator.
func (s *Service) processSingleAttestation(att ethpb.Att) {
s.RLock()
defer s.RUnlock()
single, ok := att.(*ethpb.SingleAttestation)
if !ok {
log.Errorf("Wrong attestation type (expected %T, got %T)", &ethpb.SingleAttestation{}, att)
return
}
if s.canUpdateAttestedValidator(single.AttesterIndex, single.GetData().Slot) {
logFields := logMessageTimelyFlagsForIndex(single.AttesterIndex, att.GetData())
log.WithFields(logFields).Info("Processed unaggregated attestation")
}
}
// processUnaggregatedAttestation logs when the beacon node observes an unaggregated attestation from tracked validator.
func (s *Service) processUnaggregatedAttestation(ctx context.Context, att ethpb.Att) {
s.RLock()

View File

@@ -241,7 +241,7 @@ func (s *Service) monitorRoutine(stateChannel chan *feed.Event, stateSub event.S
if !ok {
log.Error("Event feed data is not of type *operation.SingleAttReceivedData")
} else {
s.processUnaggregatedAttestation(s.ctx, data.Attestation)
s.processSingleAttestation(data.Attestation)
}
case operation.ExitReceived:
data, ok := e.Data.(*operation.ExitReceivedData)

View File

@@ -14,6 +14,7 @@ import (
func configureTracing(cliCtx *cli.Context) error {
return tracing.Setup(
cliCtx.Context,
"beacon-chain", // service name
cliCtx.String(cmd.TracingProcessNameFlag.Name),
cliCtx.String(cmd.TracingEndpointFlag.Name),
@@ -165,7 +166,7 @@ func configureExecutionSetting(cliCtx *cli.Context) error {
}
if !cliCtx.IsSet(flags.SuggestedFeeRecipient.Name) {
log.Warnf("In order to receive transaction fees from proposing blocks, " +
log.Warn("In order to receive transaction fees from proposing blocks, " +
"you must provide flag --" + flags.SuggestedFeeRecipient.Name + " with a valid ethereum address when starting your beacon node. " +
"Please see our documentation for more information on this requirement (https://docs.prylabs.network/docs/execution-node/fee-recipient).")
return nil

View File

@@ -6,6 +6,7 @@ import (
"github.com/prysmaticlabs/go-bitfield"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1/attestation"
log "github.com/sirupsen/logrus"
)
func (c *AttCaches) insertSeenBit(att ethpb.Att) error {
@@ -54,6 +55,10 @@ func (c *AttCaches) hasSeenBit(att ethpb.Att) (bool, error) {
}
for _, bit := range seenBits {
if c, err := bit.Contains(att.GetAggregationBits()); err != nil {
log.WithFields(log.Fields{
"bitLength": bit.Len(),
"incomingBitLength": att.GetAggregationBits().Len(),
}).Error("could not check if bit contains aggregation bits")
return false, err
} else if c {
return true, nil

View File

@@ -9,6 +9,7 @@ import (
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1/attestation"
"github.com/prysmaticlabs/prysm/v5/runtime/version"
log "github.com/sirupsen/logrus"
)
// SaveUnaggregatedAttestation saves an unaggregated attestation in cache.
@@ -60,7 +61,13 @@ func (c *AttCaches) UnaggregatedAttestations() ([]ethpb.Att, error) {
for _, att := range unAggregatedAtts {
seen, err := c.hasSeenBit(att)
if err != nil {
return nil, err
log.WithFields(log.Fields{
"slot": att.GetData().Slot,
"index": att.GetData().CommitteeIndex,
"version": att.Version(),
"aggBits": att.GetAggregationBits(),
}).WithError(err).Error("could not check if attestation has been seen")
continue
}
if !seen {
atts = append(atts, att.Clone())
@@ -163,7 +170,13 @@ func (c *AttCaches) DeleteSeenUnaggregatedAttestations() (int, error) {
if att == nil || att.IsNil() || att.IsAggregated() {
continue
}
if seen, err := c.hasSeenBit(att); err == nil && seen {
seen, err := c.hasSeenBit(att)
if err != nil {
log.WithError(err).Debug("Could not check if attestations bits have been seen")
delete(c.unAggregatedAtt, r)
count++
}
if seen {
delete(c.unAggregatedAtt, r)
count++
}

View File

@@ -36,6 +36,11 @@ func (s *Service) prepareForkChoiceAtts() {
select {
case slotInterval := <-ticker.C():
t := time.Now()
_, err := s.cfg.Pool.UnaggregatedAttestations()
if err != nil {
log.WithError(err).Error("Could not get unaggregated attestations")
continue
}
if err := s.batchForkChoiceAtts(s.ctx); err != nil {
log.WithError(err).Error("Could not prepare attestations for fork choice")
}

View File

@@ -6,8 +6,8 @@ go_library(
"doc.go",
"log.go",
"metrics.go",
"pool.go",
"service.go",
"service_new.go",
"types.go",
],
importpath = "github.com/prysmaticlabs/prysm/v5/beacon-chain/operations/slashings",

View File

@@ -0,0 +1,324 @@
package slashings
import (
"context"
"fmt"
"sort"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/blocks"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/helpers"
coretime "github.com/prysmaticlabs/prysm/v5/beacon-chain/core/time"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/container/slice"
"github.com/prysmaticlabs/prysm/v5/monitoring/tracing/trace"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/runtime/version"
"github.com/trailofbits/go-mutexasserts"
)
// NewPool returns an initialized attester slashing and proposer slashing pool.
func NewPool() *Pool {
return &Pool{
pendingProposerSlashing: make([]*ethpb.ProposerSlashing, 0),
pendingAttesterSlashing: make([]*PendingAttesterSlashing, 0),
included: make(map[primitives.ValidatorIndex]bool),
}
}
// PendingAttesterSlashings returns attester slashings that are able to be included into a block.
// This method will return the amount of pending attester slashings for a block transition unless parameter `noLimit` is true
// to indicate the request is for noLimit pending items.
func (p *Pool) PendingAttesterSlashings(ctx context.Context, state state.ReadOnlyBeaconState, noLimit bool) []ethpb.AttSlashing {
p.lock.Lock()
defer p.lock.Unlock()
_, span := trace.StartSpan(ctx, "operations.PendingAttesterSlashing")
defer span.End()
// Update prom metric.
numPendingAttesterSlashings.Set(float64(len(p.pendingAttesterSlashing)))
included := make(map[primitives.ValidatorIndex]bool)
// Allocate pending slice with a capacity of maxAttesterSlashings or len(p.pendingAttesterSlashing)) depending on the request.
maxSlashings := params.BeaconConfig().MaxAttesterSlashings
// EIP-7549: Beginning from Electra, the max attester slashings is reduced to 1.
if state.Version() >= version.Electra {
maxSlashings = params.BeaconConfig().MaxAttesterSlashingsElectra
}
if noLimit {
maxSlashings = uint64(len(p.pendingAttesterSlashing))
}
pending := make([]ethpb.AttSlashing, 0, maxSlashings)
for i := 0; i < len(p.pendingAttesterSlashing); i++ {
if uint64(len(pending)) >= maxSlashings {
break
}
slashing := p.pendingAttesterSlashing[i]
valid, err := p.validatorSlashingPreconditionCheck(state, slashing.validatorToSlash)
if err != nil {
log.WithError(err).Error("could not validate attester slashing")
continue
}
if included[slashing.validatorToSlash] || !valid {
p.pendingAttesterSlashing = append(p.pendingAttesterSlashing[:i], p.pendingAttesterSlashing[i+1:]...)
i--
continue
}
attSlashing := slashing.attesterSlashing
slashedVal := slice.IntersectionUint64(
attSlashing.FirstAttestation().GetAttestingIndices(),
attSlashing.SecondAttestation().GetAttestingIndices(),
)
for _, idx := range slashedVal {
included[primitives.ValidatorIndex(idx)] = true
}
pending = append(pending, attSlashing)
}
return pending
}
// PendingProposerSlashings returns proposer slashings that are able to be included into a block.
// This method will return the amount of pending proposer slashings for a block transition unless the `noLimit` parameter
// is set to true to indicate the request is for noLimit pending items.
func (p *Pool) PendingProposerSlashings(ctx context.Context, state state.ReadOnlyBeaconState, noLimit bool) []*ethpb.ProposerSlashing {
p.lock.Lock()
defer p.lock.Unlock()
_, span := trace.StartSpan(ctx, "operations.PendingProposerSlashing")
defer span.End()
// Update prom metric.
numPendingProposerSlashings.Set(float64(len(p.pendingProposerSlashing)))
// Allocate pending slice with a capacity of len(p.pendingProposerSlashing) or maxProposerSlashings depending on the request.
maxSlashings := params.BeaconConfig().MaxProposerSlashings
if noLimit {
maxSlashings = uint64(len(p.pendingProposerSlashing))
}
pending := make([]*ethpb.ProposerSlashing, 0, maxSlashings)
for i := 0; i < len(p.pendingProposerSlashing); i++ {
if uint64(len(pending)) >= maxSlashings {
break
}
slashing := p.pendingProposerSlashing[i]
valid, err := p.validatorSlashingPreconditionCheck(state, slashing.Header_1.Header.ProposerIndex)
if err != nil {
log.WithError(err).Error("could not validate proposer slashing")
continue
}
if !valid {
p.pendingProposerSlashing = append(p.pendingProposerSlashing[:i], p.pendingProposerSlashing[i+1:]...)
i--
continue
}
pending = append(pending, slashing)
}
return pending
}
// InsertAttesterSlashing into the pool. This method is a no-op if the attester slashing already exists in the pool,
// has been included into a block recently, or the validator is already exited.
func (p *Pool) InsertAttesterSlashing(
ctx context.Context,
state state.ReadOnlyBeaconState,
slashing ethpb.AttSlashing,
) error {
p.lock.Lock()
defer p.lock.Unlock()
ctx, span := trace.StartSpan(ctx, "operations.InsertAttesterSlashing")
defer span.End()
if err := blocks.VerifyAttesterSlashing(ctx, state, slashing); err != nil {
return errors.Wrap(err, "could not verify attester slashing")
}
slashedVal := slice.IntersectionUint64(slashing.FirstAttestation().GetAttestingIndices(), slashing.SecondAttestation().GetAttestingIndices())
cantSlash := make([]uint64, 0, len(slashedVal))
slashingReason := ""
for _, val := range slashedVal {
// Has this validator index been included recently?
ok, err := p.validatorSlashingPreconditionCheck(state, primitives.ValidatorIndex(val))
if err != nil {
return err
}
// If the validator has already exited, has already been slashed, or if its index
// has been recently included in the pool of slashings, skip including this indice.
if !ok {
slashingReason = "validator already exited/slashed or already recently included in slashings pool"
cantSlash = append(cantSlash, val)
continue
}
// Check if the validator already exists in the list of slashings.
// Use binary search to find the answer.
found := sort.Search(len(p.pendingAttesterSlashing), func(i int) bool {
return uint64(p.pendingAttesterSlashing[i].validatorToSlash) >= val
})
if found != len(p.pendingAttesterSlashing) && uint64(p.pendingAttesterSlashing[found].validatorToSlash) == val {
slashingReason = "validator already exist in list of pending slashings, no need to attempt to slash again"
cantSlash = append(cantSlash, val)
continue
}
pendingSlashing := &PendingAttesterSlashing{
attesterSlashing: slashing,
validatorToSlash: primitives.ValidatorIndex(val),
}
// Insert into pending list and sort again.
p.pendingAttesterSlashing = append(p.pendingAttesterSlashing, pendingSlashing)
sort.Slice(p.pendingAttesterSlashing, func(i, j int) bool {
return p.pendingAttesterSlashing[i].validatorToSlash < p.pendingAttesterSlashing[j].validatorToSlash
})
numPendingAttesterSlashings.Set(float64(len(p.pendingAttesterSlashing)))
}
if len(cantSlash) == len(slashedVal) {
return fmt.Errorf(
"could not slash any of %d validators in submitted slashing: %s",
len(slashedVal),
slashingReason,
)
}
return nil
}
// InsertProposerSlashing into the pool. This method is a no-op if the pending slashing already exists,
// has been included recently, the validator is already exited, or the validator was already slashed.
func (p *Pool) InsertProposerSlashing(
ctx context.Context,
state state.ReadOnlyBeaconState,
slashing *ethpb.ProposerSlashing,
) error {
p.lock.Lock()
defer p.lock.Unlock()
_, span := trace.StartSpan(ctx, "operations.InsertProposerSlashing")
defer span.End()
if err := blocks.VerifyProposerSlashing(state, slashing); err != nil {
return errors.Wrap(err, "could not verify proposer slashing")
}
idx := slashing.Header_1.Header.ProposerIndex
ok, err := p.validatorSlashingPreconditionCheck(state, idx)
if err != nil {
return err
}
// If the validator has already exited, has already been slashed, or if its index
// has been recently included in the pool of slashings, do not process this new
// slashing.
if !ok {
return fmt.Errorf("validator at index %d cannot be slashed", idx)
}
// Check if the validator already exists in the list of slashings.
// Use binary search to find the answer.
found := sort.Search(len(p.pendingProposerSlashing), func(i int) bool {
return p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex >= slashing.Header_1.Header.ProposerIndex
})
if found != len(p.pendingProposerSlashing) && p.pendingProposerSlashing[found].Header_1.Header.ProposerIndex ==
slashing.Header_1.Header.ProposerIndex {
return errors.New("slashing object already exists in pending proposer slashings")
}
// Insert into pending list and sort again.
p.pendingProposerSlashing = append(p.pendingProposerSlashing, slashing)
sort.Slice(p.pendingProposerSlashing, func(i, j int) bool {
return p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex < p.pendingProposerSlashing[j].Header_1.Header.ProposerIndex
})
numPendingProposerSlashings.Set(float64(len(p.pendingProposerSlashing)))
return nil
}
// MarkIncludedAttesterSlashing is used when an attester slashing has been included in a beacon block.
// Every block seen by this node that contains proposer slashings should call this method to include
// the proposer slashings.
func (p *Pool) MarkIncludedAttesterSlashing(as ethpb.AttSlashing) {
p.lock.Lock()
defer p.lock.Unlock()
slashedVal := slice.IntersectionUint64(as.FirstAttestation().GetAttestingIndices(), as.SecondAttestation().GetAttestingIndices())
for _, val := range slashedVal {
i := sort.Search(len(p.pendingAttesterSlashing), func(i int) bool {
return uint64(p.pendingAttesterSlashing[i].validatorToSlash) >= val
})
if i != len(p.pendingAttesterSlashing) && uint64(p.pendingAttesterSlashing[i].validatorToSlash) == val {
p.pendingAttesterSlashing = append(p.pendingAttesterSlashing[:i], p.pendingAttesterSlashing[i+1:]...)
}
p.included[primitives.ValidatorIndex(val)] = true
numAttesterSlashingsIncluded.Inc()
}
}
// MarkIncludedProposerSlashing is used when an proposer slashing has been included in a beacon block.
// Every block seen by this node that contains proposer slashings should call this method to include
// the proposer slashings.
func (p *Pool) MarkIncludedProposerSlashing(ps *ethpb.ProposerSlashing) {
p.lock.Lock()
defer p.lock.Unlock()
i := sort.Search(len(p.pendingProposerSlashing), func(i int) bool {
return p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex >= ps.Header_1.Header.ProposerIndex
})
if i != len(p.pendingProposerSlashing) && p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex == ps.Header_1.Header.ProposerIndex {
p.pendingProposerSlashing = append(p.pendingProposerSlashing[:i], p.pendingProposerSlashing[i+1:]...)
}
p.included[ps.Header_1.Header.ProposerIndex] = true
numProposerSlashingsIncluded.Inc()
}
// ConvertToElectra converts all Phase0 attester slashings to Electra attester slashings.
// This functionality is needed at the time of the Electra fork.
func (p *Pool) ConvertToElectra() {
p.lock.Lock()
defer p.lock.Unlock()
for _, pas := range p.pendingAttesterSlashing {
if pas.attesterSlashing.Version() == version.Phase0 {
first := pas.attesterSlashing.FirstAttestation()
second := pas.attesterSlashing.SecondAttestation()
pas.attesterSlashing = &ethpb.AttesterSlashingElectra{
Attestation_1: &ethpb.IndexedAttestationElectra{
AttestingIndices: first.GetAttestingIndices(),
Data: first.GetData(),
Signature: first.GetSignature(),
},
Attestation_2: &ethpb.IndexedAttestationElectra{
AttestingIndices: second.GetAttestingIndices(),
Data: second.GetData(),
Signature: second.GetSignature(),
},
}
}
}
}
// this function checks a few items about a validator before proceeding with inserting
// a proposer/attester slashing into the pool. First, it checks if the validator
// has been recently included in the pool, then it checks if the validator is slashable.
// Note: this method requires caller to hold the lock.
func (p *Pool) validatorSlashingPreconditionCheck(
state state.ReadOnlyBeaconState,
valIdx primitives.ValidatorIndex,
) (bool, error) {
if !mutexasserts.RWMutexLocked(&p.lock) && !mutexasserts.RWMutexRLocked(&p.lock) {
return false, errors.New("pool.validatorSlashingPreconditionCheck: caller must hold read/write lock")
}
// Check if the validator index has been included recently.
if p.included[valIdx] {
return false, nil
}
validator, err := state.ValidatorAtIndexReadOnly(valIdx)
if err != nil {
return false, err
}
// Checking if the validator is slashable.
if !helpers.IsSlashableValidatorUsingTrie(validator, coretime.CurrentEpoch(state)) {
return false, nil
}
return true, nil
}

View File

@@ -2,323 +2,102 @@ package slashings
import (
"context"
"fmt"
"sort"
"time"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/blocks"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/helpers"
coretime "github.com/prysmaticlabs/prysm/v5/beacon-chain/core/time"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/startup"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/container/slice"
"github.com/prysmaticlabs/prysm/v5/monitoring/tracing/trace"
ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
"github.com/prysmaticlabs/prysm/v5/runtime/version"
"github.com/trailofbits/go-mutexasserts"
"github.com/prysmaticlabs/prysm/v5/time/slots"
)
// NewPool returns an initialized attester slashing and proposer slashing pool.
func NewPool() *Pool {
return &Pool{
pendingProposerSlashing: make([]*ethpb.ProposerSlashing, 0),
pendingAttesterSlashing: make([]*PendingAttesterSlashing, 0),
included: make(map[primitives.ValidatorIndex]bool),
// WithElectraTimer includes functional options for the blockchain service related to CLI flags.
func WithElectraTimer(cw startup.ClockWaiter, currentSlotFn func() primitives.Slot) Option {
return func(p *PoolService) error {
p.runElectraTimer = true
p.cw = cw
p.currentSlotFn = currentSlotFn
return nil
}
}
// PendingAttesterSlashings returns attester slashings that are able to be included into a block.
// This method will return the amount of pending attester slashings for a block transition unless parameter `noLimit` is true
// to indicate the request is for noLimit pending items.
func (p *Pool) PendingAttesterSlashings(ctx context.Context, state state.ReadOnlyBeaconState, noLimit bool) []ethpb.AttSlashing {
p.lock.Lock()
defer p.lock.Unlock()
_, span := trace.StartSpan(ctx, "operations.PendingAttesterSlashing")
defer span.End()
// Update prom metric.
numPendingAttesterSlashings.Set(float64(len(p.pendingAttesterSlashing)))
included := make(map[primitives.ValidatorIndex]bool)
// Allocate pending slice with a capacity of maxAttesterSlashings or len(p.pendingAttesterSlashing)) depending on the request.
maxSlashings := params.BeaconConfig().MaxAttesterSlashings
// EIP-7549: Beginning from Electra, the max attester slashings is reduced to 1.
if state.Version() >= version.Electra {
maxSlashings = params.BeaconConfig().MaxAttesterSlashingsElectra
}
if noLimit {
maxSlashings = uint64(len(p.pendingAttesterSlashing))
}
pending := make([]ethpb.AttSlashing, 0, maxSlashings)
for i := 0; i < len(p.pendingAttesterSlashing); i++ {
if uint64(len(pending)) >= maxSlashings {
break
}
slashing := p.pendingAttesterSlashing[i]
valid, err := p.validatorSlashingPreconditionCheck(state, slashing.validatorToSlash)
if err != nil {
log.WithError(err).Error("could not validate attester slashing")
continue
}
if included[slashing.validatorToSlash] || !valid {
p.pendingAttesterSlashing = append(p.pendingAttesterSlashing[:i], p.pendingAttesterSlashing[i+1:]...)
i--
continue
}
attSlashing := slashing.attesterSlashing
slashedVal := slice.IntersectionUint64(
attSlashing.FirstAttestation().GetAttestingIndices(),
attSlashing.SecondAttestation().GetAttestingIndices(),
)
for _, idx := range slashedVal {
included[primitives.ValidatorIndex(idx)] = true
}
pending = append(pending, attSlashing)
// NewPoolService returns a service that manages the Pool.
func NewPoolService(ctx context.Context, pool PoolManager, opts ...Option) *PoolService {
ctx, cancel := context.WithCancel(ctx)
p := &PoolService{
ctx: ctx,
cancel: cancel,
poolManager: pool,
}
return pending
for _, opt := range opts {
if err := opt(p); err != nil {
return nil
}
}
return p
}
// PendingProposerSlashings returns proposer slashings that are able to be included into a block.
// This method will return the amount of pending proposer slashings for a block transition unless the `noLimit` parameter
// is set to true to indicate the request is for noLimit pending items.
func (p *Pool) PendingProposerSlashings(ctx context.Context, state state.ReadOnlyBeaconState, noLimit bool) []*ethpb.ProposerSlashing {
p.lock.Lock()
defer p.lock.Unlock()
_, span := trace.StartSpan(ctx, "operations.PendingProposerSlashing")
defer span.End()
// Update prom metric.
numPendingProposerSlashings.Set(float64(len(p.pendingProposerSlashing)))
// Allocate pending slice with a capacity of len(p.pendingProposerSlashing) or maxProposerSlashings depending on the request.
maxSlashings := params.BeaconConfig().MaxProposerSlashings
if noLimit {
maxSlashings = uint64(len(p.pendingProposerSlashing))
}
pending := make([]*ethpb.ProposerSlashing, 0, maxSlashings)
for i := 0; i < len(p.pendingProposerSlashing); i++ {
if uint64(len(pending)) >= maxSlashings {
break
}
slashing := p.pendingProposerSlashing[i]
valid, err := p.validatorSlashingPreconditionCheck(state, slashing.Header_1.Header.ProposerIndex)
if err != nil {
log.WithError(err).Error("could not validate proposer slashing")
continue
}
if !valid {
p.pendingProposerSlashing = append(p.pendingProposerSlashing[:i], p.pendingProposerSlashing[i+1:]...)
i--
continue
}
pending = append(pending, slashing)
}
return pending
// Start the slashing pool service.
func (p *PoolService) Start() {
go p.run()
}
// InsertAttesterSlashing into the pool. This method is a no-op if the attester slashing already exists in the pool,
// has been included into a block recently, or the validator is already exited.
func (p *Pool) InsertAttesterSlashing(
ctx context.Context,
state state.ReadOnlyBeaconState,
slashing ethpb.AttSlashing,
) error {
p.lock.Lock()
defer p.lock.Unlock()
ctx, span := trace.StartSpan(ctx, "operations.InsertAttesterSlashing")
defer span.End()
if err := blocks.VerifyAttesterSlashing(ctx, state, slashing); err != nil {
return errors.Wrap(err, "could not verify attester slashing")
func (p *PoolService) run() {
if !p.runElectraTimer {
return
}
slashedVal := slice.IntersectionUint64(slashing.FirstAttestation().GetAttestingIndices(), slashing.SecondAttestation().GetAttestingIndices())
cantSlash := make([]uint64, 0, len(slashedVal))
slashingReason := ""
for _, val := range slashedVal {
// Has this validator index been included recently?
ok, err := p.validatorSlashingPreconditionCheck(state, primitives.ValidatorIndex(val))
if err != nil {
return err
}
// If the validator has already exited, has already been slashed, or if its index
// has been recently included in the pool of slashings, skip including this indice.
if !ok {
slashingReason = "validator already exited/slashed or already recently included in slashings pool"
cantSlash = append(cantSlash, val)
continue
}
// Check if the validator already exists in the list of slashings.
// Use binary search to find the answer.
found := sort.Search(len(p.pendingAttesterSlashing), func(i int) bool {
return uint64(p.pendingAttesterSlashing[i].validatorToSlash) >= val
})
if found != len(p.pendingAttesterSlashing) && uint64(p.pendingAttesterSlashing[found].validatorToSlash) == val {
slashingReason = "validator already exist in list of pending slashings, no need to attempt to slash again"
cantSlash = append(cantSlash, val)
continue
}
pendingSlashing := &PendingAttesterSlashing{
attesterSlashing: slashing,
validatorToSlash: primitives.ValidatorIndex(val),
}
// Insert into pending list and sort again.
p.pendingAttesterSlashing = append(p.pendingAttesterSlashing, pendingSlashing)
sort.Slice(p.pendingAttesterSlashing, func(i, j int) bool {
return p.pendingAttesterSlashing[i].validatorToSlash < p.pendingAttesterSlashing[j].validatorToSlash
})
numPendingAttesterSlashings.Set(float64(len(p.pendingAttesterSlashing)))
electraSlot, err := slots.EpochStart(params.BeaconConfig().ElectraForkEpoch)
if err != nil {
return
}
if len(cantSlash) == len(slashedVal) {
return fmt.Errorf(
"could not slash any of %d validators in submitted slashing: %s",
len(slashedVal),
slashingReason,
)
// If run() is executed after the transition to Electra has already happened,
// there is nothing to convert because the slashing pool is empty at startup.
if p.currentSlotFn() >= electraSlot {
return
}
p.waitForChainInitialization()
electraTime, err := slots.ToTime(uint64(p.clock.GenesisTime().Unix()), electraSlot)
if err != nil {
return
}
t := time.NewTimer(electraTime.Sub(p.clock.Now()))
defer t.Stop()
select {
case <-t.C:
log.Info("Converting Phase0 slashings to Electra slashings")
p.poolManager.ConvertToElectra()
return
case <-p.ctx.Done():
log.Warn("Context cancelled, ConvertToElectra timer will not execute")
return
}
}
func (p *PoolService) waitForChainInitialization() {
clock, err := p.cw.WaitForClock(p.ctx)
if err != nil {
log.WithError(err).Error("Could not receive chain start notification")
}
p.clock = clock
log.WithField("genesisTime", clock.GenesisTime()).Info(
"Slashing pool service received chain initialization event",
)
}
// Stop the slashing pool service.
func (p *PoolService) Stop() error {
p.cancel()
return nil
}
// InsertProposerSlashing into the pool. This method is a no-op if the pending slashing already exists,
// has been included recently, the validator is already exited, or the validator was already slashed.
func (p *Pool) InsertProposerSlashing(
ctx context.Context,
state state.ReadOnlyBeaconState,
slashing *ethpb.ProposerSlashing,
) error {
p.lock.Lock()
defer p.lock.Unlock()
_, span := trace.StartSpan(ctx, "operations.InsertProposerSlashing")
defer span.End()
if err := blocks.VerifyProposerSlashing(state, slashing); err != nil {
return errors.Wrap(err, "could not verify proposer slashing")
}
idx := slashing.Header_1.Header.ProposerIndex
ok, err := p.validatorSlashingPreconditionCheck(state, idx)
if err != nil {
return err
}
// If the validator has already exited, has already been slashed, or if its index
// has been recently included in the pool of slashings, do not process this new
// slashing.
if !ok {
return fmt.Errorf("validator at index %d cannot be slashed", idx)
}
// Check if the validator already exists in the list of slashings.
// Use binary search to find the answer.
found := sort.Search(len(p.pendingProposerSlashing), func(i int) bool {
return p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex >= slashing.Header_1.Header.ProposerIndex
})
if found != len(p.pendingProposerSlashing) && p.pendingProposerSlashing[found].Header_1.Header.ProposerIndex ==
slashing.Header_1.Header.ProposerIndex {
return errors.New("slashing object already exists in pending proposer slashings")
}
// Insert into pending list and sort again.
p.pendingProposerSlashing = append(p.pendingProposerSlashing, slashing)
sort.Slice(p.pendingProposerSlashing, func(i, j int) bool {
return p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex < p.pendingProposerSlashing[j].Header_1.Header.ProposerIndex
})
numPendingProposerSlashings.Set(float64(len(p.pendingProposerSlashing)))
// Status of the slashing pool service.
func (p *PoolService) Status() error {
return nil
}
// MarkIncludedAttesterSlashing is used when an attester slashing has been included in a beacon block.
// Every block seen by this node that contains proposer slashings should call this method to include
// the proposer slashings.
func (p *Pool) MarkIncludedAttesterSlashing(as ethpb.AttSlashing) {
p.lock.Lock()
defer p.lock.Unlock()
slashedVal := slice.IntersectionUint64(as.FirstAttestation().GetAttestingIndices(), as.SecondAttestation().GetAttestingIndices())
for _, val := range slashedVal {
i := sort.Search(len(p.pendingAttesterSlashing), func(i int) bool {
return uint64(p.pendingAttesterSlashing[i].validatorToSlash) >= val
})
if i != len(p.pendingAttesterSlashing) && uint64(p.pendingAttesterSlashing[i].validatorToSlash) == val {
p.pendingAttesterSlashing = append(p.pendingAttesterSlashing[:i], p.pendingAttesterSlashing[i+1:]...)
}
p.included[primitives.ValidatorIndex(val)] = true
numAttesterSlashingsIncluded.Inc()
}
}
// MarkIncludedProposerSlashing is used when an proposer slashing has been included in a beacon block.
// Every block seen by this node that contains proposer slashings should call this method to include
// the proposer slashings.
func (p *Pool) MarkIncludedProposerSlashing(ps *ethpb.ProposerSlashing) {
p.lock.Lock()
defer p.lock.Unlock()
i := sort.Search(len(p.pendingProposerSlashing), func(i int) bool {
return p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex >= ps.Header_1.Header.ProposerIndex
})
if i != len(p.pendingProposerSlashing) && p.pendingProposerSlashing[i].Header_1.Header.ProposerIndex == ps.Header_1.Header.ProposerIndex {
p.pendingProposerSlashing = append(p.pendingProposerSlashing[:i], p.pendingProposerSlashing[i+1:]...)
}
p.included[ps.Header_1.Header.ProposerIndex] = true
numProposerSlashingsIncluded.Inc()
}
// ConvertToElectra converts all Phase0 attester slashings to Electra attester slashings.
// This functionality is needed at the time of the Electra fork.
func (p *Pool) ConvertToElectra() {
p.lock.Lock()
defer p.lock.Unlock()
for _, pas := range p.pendingAttesterSlashing {
if pas.attesterSlashing.Version() == version.Phase0 {
first := pas.attesterSlashing.FirstAttestation()
second := pas.attesterSlashing.SecondAttestation()
pas.attesterSlashing = &ethpb.AttesterSlashingElectra{
Attestation_1: &ethpb.IndexedAttestationElectra{
AttestingIndices: first.GetAttestingIndices(),
Data: first.GetData(),
Signature: first.GetSignature(),
},
Attestation_2: &ethpb.IndexedAttestationElectra{
AttestingIndices: second.GetAttestingIndices(),
Data: second.GetData(),
Signature: second.GetSignature(),
},
}
}
}
}
// this function checks a few items about a validator before proceeding with inserting
// a proposer/attester slashing into the pool. First, it checks if the validator
// has been recently included in the pool, then it checks if the validator is slashable.
// Note: this method requires caller to hold the lock.
func (p *Pool) validatorSlashingPreconditionCheck(
state state.ReadOnlyBeaconState,
valIdx primitives.ValidatorIndex,
) (bool, error) {
if !mutexasserts.RWMutexLocked(&p.lock) && !mutexasserts.RWMutexRLocked(&p.lock) {
return false, errors.New("pool.validatorSlashingPreconditionCheck: caller must hold read/write lock")
}
// Check if the validator index has been included recently.
if p.included[valIdx] {
return false, nil
}
validator, err := state.ValidatorAtIndexReadOnly(valIdx)
if err != nil {
return false, err
}
// Checking if the validator is slashable.
if !helpers.IsSlashableValidatorUsingTrie(validator, coretime.CurrentEpoch(state)) {
return false, nil
}
return true, nil
}

View File

@@ -1,105 +0,0 @@
package slashings
import (
"context"
"time"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/startup"
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/time/slots"
)
// WithElectraTimer includes functional options for the blockchain service related to CLI flags.
func WithElectraTimer(cw startup.ClockWaiter, currentSlotFn func() primitives.Slot) Option {
return func(p *PoolService) error {
p.runElectraTimer = true
p.cw = cw
p.currentSlotFn = currentSlotFn
return nil
}
}
// NewPoolService returns a service that manages the Pool.
func NewPoolService(ctx context.Context, pool PoolManager, opts ...Option) *PoolService {
ctx, cancel := context.WithCancel(ctx)
p := &PoolService{
ctx: ctx,
cancel: cancel,
poolManager: pool,
}
for _, opt := range opts {
if err := opt(p); err != nil {
return nil
}
}
return p
}
// Start the slashing pool service.
func (p *PoolService) Start() {
go p.run()
}
func (p *PoolService) run() {
if !p.runElectraTimer {
return
}
electraSlot, err := slots.EpochStart(params.BeaconConfig().ElectraForkEpoch)
if err != nil {
log.WithError(err).Error("Could not get Electra start slot")
return
}
// If run() is executed after the transition to Electra has already happened,
// there is nothing to convert because the slashing pool is empty at startup.
if p.currentSlotFn() >= electraSlot {
return
}
p.waitForChainInitialization()
electraTime, err := slots.ToTime(uint64(p.clock.GenesisTime().Unix()), electraSlot)
if err != nil {
log.WithError(err).Error("Could not get Electra start time")
return
}
t := time.NewTimer(electraTime.Sub(p.clock.Now()))
defer t.Stop()
select {
case <-t.C:
log.Info("Converting Phase0 slashings to Electra slashings")
p.poolManager.ConvertToElectra()
return
case <-p.ctx.Done():
log.Warn("Context cancelled, ConvertToElectra timer will not execute")
return
}
}
func (p *PoolService) waitForChainInitialization() {
clock, err := p.cw.WaitForClock(p.ctx)
if err != nil {
log.WithError(err).Error("Could not receive chain start notification")
}
p.clock = clock
log.WithField("genesisTime", clock.GenesisTime()).Info(
"Slashing pool service received chain initialization event",
)
}
// Stop the slashing pool service.
func (p *PoolService) Stop() error {
p.cancel()
return nil
}
// Status of the slashing pool service.
func (p *PoolService) Status() error {
return nil
}

View File

@@ -33,6 +33,9 @@ var (
// AggregateAttestationMap maps the fork-version to the underlying data type for that
// particular fork period.
AggregateAttestationMap map[[4]byte]func() (ethpb.SignedAggregateAttAndProof, error)
// AttesterSlashingMap maps the fork-version to the underlying data type for that particular
// fork period.
AttesterSlashingMap map[[4]byte]func() (ethpb.AttSlashing, error)
)
// InitializeDataMaps initializes all the relevant object maps. This function is called to
@@ -72,7 +75,7 @@ func InitializeDataMaps() {
},
bytesutil.ToBytes4(params.BeaconConfig().FuluForkVersion): func() (interfaces.ReadOnlySignedBeaconBlock, error) {
return blocks.NewSignedBeaconBlock(
&ethpb.SignedBeaconBlockFulu{Block: &ethpb.BeaconBlockFulu{Body: &ethpb.BeaconBlockBodyFulu{ExecutionPayload: &enginev1.ExecutionPayloadDeneb{}}}},
&ethpb.SignedBeaconBlockFulu{Block: &ethpb.BeaconBlockElectra{Body: &ethpb.BeaconBlockBodyElectra{ExecutionPayload: &enginev1.ExecutionPayloadDeneb{}}}},
)
},
}
@@ -151,4 +154,29 @@ func InitializeDataMaps() {
return &ethpb.SignedAggregateAttestationAndProofElectra{}, nil
},
}
// Reset our aggregate attestation map.
AttesterSlashingMap = map[[4]byte]func() (ethpb.AttSlashing, error){
bytesutil.ToBytes4(params.BeaconConfig().GenesisForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashing{}, nil
},
bytesutil.ToBytes4(params.BeaconConfig().AltairForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashing{}, nil
},
bytesutil.ToBytes4(params.BeaconConfig().BellatrixForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashing{}, nil
},
bytesutil.ToBytes4(params.BeaconConfig().CapellaForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashing{}, nil
},
bytesutil.ToBytes4(params.BeaconConfig().DenebForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashing{}, nil
},
bytesutil.ToBytes4(params.BeaconConfig().ElectraForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashingElectra{}, nil
},
bytesutil.ToBytes4(params.BeaconConfig().FuluForkVersion): func() (ethpb.AttSlashing, error) {
return &ethpb.AttesterSlashingElectra{}, nil
},
}
}

View File

@@ -76,6 +76,13 @@ func TestInitializeDataMaps(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, version.Phase0, agg.Version())
}
attSlashFunc, ok := AttesterSlashingMap[bytesutil.ToBytes4(params.BeaconConfig().GenesisForkVersion)]
assert.Equal(t, tt.exists, ok)
if tt.exists {
attSlash, err := attSlashFunc()
require.NoError(t, err)
assert.Equal(t, version.Phase0, attSlash.Version())
}
})
}
}

View File

@@ -1,7 +1,6 @@
package p2p
import (
"fmt"
"testing"
"github.com/ethereum/go-ethereum/crypto"
@@ -28,7 +27,7 @@ func TestVerifyConnectivity(t *testing.T) {
{"123.123.123.123", 19000, false, "Dialing an unreachable IP: 123.123.123.123:19000"},
}
for _, tc := range cases {
t.Run(fmt.Sprintf(tc.name),
t.Run(tc.name,
func(t *testing.T) {
verifyConnectivity(tc.address, tc.port, "tcp")
logMessage := "IP address is not accessible"

View File

@@ -161,6 +161,7 @@ func (s *Service) builderEndpoints(stater lookup.Stater) []endpoint {
const namespace = "builder"
return []endpoint{
{
// Deprecated: use SSE from /eth/v1/events for `Payload Attributes` instead
template: "/eth/v1/builder/states/{state_id}/expected_withdrawals",
name: namespace + ".ExpectedWithdrawals",
middleware: []middleware.Middleware{
@@ -225,6 +226,7 @@ func (s *Service) validatorEndpoints(
const namespace = "validator"
return []endpoint{
{
// Deprecated: use /eth/v2/validator/aggregate_attestation instead
template: "/eth/v1/validator/aggregate_attestation",
name: namespace + ".GetAggregateAttestation",
middleware: []middleware.Middleware{
@@ -253,6 +255,7 @@ func (s *Service) validatorEndpoints(
methods: []string{http.MethodPost},
},
{
// Deprecated: use /eth/v2/validator/aggregate_and_proofs instead
template: "/eth/v1/validator/aggregate_and_proofs",
name: namespace + ".SubmitAggregateAndProofs",
middleware: []middleware.Middleware{
@@ -583,6 +586,7 @@ func (s *Service) beaconEndpoints(
methods: []string{http.MethodGet},
},
{
// Deprecated: use /eth/v2/beacon/blocks instead
template: "/eth/v1/beacon/blocks",
name: namespace + ".PublishBlock",
middleware: []middleware.Middleware{
@@ -593,6 +597,7 @@ func (s *Service) beaconEndpoints(
methods: []string{http.MethodPost},
},
{
// Deprecated: use /eth/v2/beacon/blinded_blocks instead
template: "/eth/v1/beacon/blinded_blocks",
name: namespace + ".PublishBlindedBlock",
middleware: []middleware.Middleware{
@@ -632,6 +637,7 @@ func (s *Service) beaconEndpoints(
methods: []string{http.MethodGet},
},
{
// Deprecated: use /eth/v2/beacon/blocks/{block_id}/attestations instead
template: "/eth/v1/beacon/blocks/{block_id}/attestations",
name: namespace + ".GetBlockAttestations",
middleware: []middleware.Middleware{
@@ -668,6 +674,7 @@ func (s *Service) beaconEndpoints(
methods: []string{http.MethodGet},
},
{
// Deprecated: use /eth/v2/beacon/pool/attestations instead
template: "/eth/v1/beacon/pool/attestations",
name: namespace + ".ListAttestations",
middleware: []middleware.Middleware{
@@ -754,6 +761,7 @@ func (s *Service) beaconEndpoints(
methods: []string{http.MethodPost},
},
{
// Deprecated: use /eth/v2/beacon/pool/attester_slashings instead
template: "/eth/v1/beacon/pool/attester_slashings",
name: namespace + ".GetAttesterSlashings",
middleware: []middleware.Middleware{
@@ -876,6 +884,7 @@ func (s *Service) beaconEndpoints(
methods: []string{http.MethodGet, http.MethodPost},
},
{
// Deprecated: no longer needed post Electra
template: "/eth/v1/beacon/deposit_snapshot",
name: namespace + ".GetDepositSnapshot",
middleware: []middleware.Middleware{
@@ -884,6 +893,24 @@ func (s *Service) beaconEndpoints(
handler: server.GetDepositSnapshot,
methods: []string{http.MethodGet},
},
{
template: "/eth/v1/beacon/states/{state_id}/pending_deposits",
name: namespace + ".GetPendingDeposits",
middleware: []middleware.Middleware{
middleware.AcceptHeaderHandler([]string{api.JsonMediaType}),
},
handler: server.GetPendingDeposits,
methods: []string{http.MethodGet},
},
{
template: "/eth/v1/beacon/states/{state_id}/pending_partial_withdrawals",
name: namespace + ".GetPendingPartialWithdrawals",
middleware: []middleware.Middleware{
middleware.AcceptHeaderHandler([]string{api.JsonMediaType}),
},
handler: server.GetPendingPartialWithdrawals,
methods: []string{http.MethodGet},
},
}
}

View File

@@ -17,38 +17,40 @@ func Test_endpoints(t *testing.T) {
}
beaconRoutes := map[string][]string{
"/eth/v1/beacon/genesis": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/root": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/fork": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/finality_checkpoints": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/validators": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/states/{state_id}/validators/{validator_id}": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/validator_balances": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/states/{state_id}/committees": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/sync_committees": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/randao": {http.MethodGet},
"/eth/v1/beacon/headers": {http.MethodGet},
"/eth/v1/beacon/headers/{block_id}": {http.MethodGet},
"/eth/v1/beacon/blinded_blocks": {http.MethodPost},
"/eth/v2/beacon/blinded_blocks": {http.MethodPost},
"/eth/v1/beacon/blocks": {http.MethodPost},
"/eth/v2/beacon/blocks": {http.MethodPost},
"/eth/v2/beacon/blocks/{block_id}": {http.MethodGet},
"/eth/v1/beacon/blocks/{block_id}/root": {http.MethodGet},
"/eth/v1/beacon/blocks/{block_id}/attestations": {http.MethodGet},
"/eth/v2/beacon/blocks/{block_id}/attestations": {http.MethodGet},
"/eth/v1/beacon/blob_sidecars/{block_id}": {http.MethodGet},
"/eth/v1/beacon/deposit_snapshot": {http.MethodGet},
"/eth/v1/beacon/blinded_blocks/{block_id}": {http.MethodGet},
"/eth/v1/beacon/pool/attestations": {http.MethodGet, http.MethodPost},
"/eth/v2/beacon/pool/attestations": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/attester_slashings": {http.MethodGet, http.MethodPost},
"/eth/v2/beacon/pool/attester_slashings": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/proposer_slashings": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/sync_committees": {http.MethodPost},
"/eth/v1/beacon/pool/voluntary_exits": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/bls_to_execution_changes": {http.MethodGet, http.MethodPost},
"/prysm/v1/beacon/individual_votes": {http.MethodPost},
"/eth/v1/beacon/genesis": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/root": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/fork": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/finality_checkpoints": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/validators": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/states/{state_id}/validators/{validator_id}": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/validator_balances": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/states/{state_id}/committees": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/sync_committees": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/randao": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/pending_deposits": {http.MethodGet},
"/eth/v1/beacon/states/{state_id}/pending_partial_withdrawals": {http.MethodGet},
"/eth/v1/beacon/headers": {http.MethodGet},
"/eth/v1/beacon/headers/{block_id}": {http.MethodGet},
"/eth/v1/beacon/blinded_blocks": {http.MethodPost},
"/eth/v2/beacon/blinded_blocks": {http.MethodPost},
"/eth/v1/beacon/blocks": {http.MethodPost},
"/eth/v2/beacon/blocks": {http.MethodPost},
"/eth/v2/beacon/blocks/{block_id}": {http.MethodGet},
"/eth/v1/beacon/blocks/{block_id}/root": {http.MethodGet},
"/eth/v1/beacon/blocks/{block_id}/attestations": {http.MethodGet},
"/eth/v2/beacon/blocks/{block_id}/attestations": {http.MethodGet},
"/eth/v1/beacon/blob_sidecars/{block_id}": {http.MethodGet},
"/eth/v1/beacon/deposit_snapshot": {http.MethodGet},
"/eth/v1/beacon/blinded_blocks/{block_id}": {http.MethodGet},
"/eth/v1/beacon/pool/attestations": {http.MethodGet, http.MethodPost},
"/eth/v2/beacon/pool/attestations": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/attester_slashings": {http.MethodGet, http.MethodPost},
"/eth/v2/beacon/pool/attester_slashings": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/proposer_slashings": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/sync_committees": {http.MethodPost},
"/eth/v1/beacon/pool/voluntary_exits": {http.MethodGet, http.MethodPost},
"/eth/v1/beacon/pool/bls_to_execution_changes": {http.MethodGet, http.MethodPost},
"/prysm/v1/beacon/individual_votes": {http.MethodPost},
}
lightClientRoutes := map[string][]string{

File diff suppressed because it is too large Load Diff

View File

@@ -34,6 +34,7 @@ import (
const broadcastBLSChangesRateLimit = 128
// Deprecated: use ListAttestationsV2 instead
// ListAttestations retrieves attestations known by the node but
// not necessarily incorporated into any block. Allows filtering by committee index or slot.
func (s *Server) ListAttestations(w http.ResponseWriter, r *http.Request) {
@@ -707,6 +708,7 @@ func (s *Server) ListBLSToExecutionChanges(w http.ResponseWriter, r *http.Reques
})
}
// Deprecated: use GetAttesterSlashingsV2 instead
// GetAttesterSlashings retrieves attester slashings known by the node but
// not necessarily incorporated into any block.
func (s *Server) GetAttesterSlashings(w http.ResponseWriter, r *http.Request) {

File diff suppressed because it is too large Load Diff

View File

@@ -42,8 +42,7 @@ func TestBlobs(t *testing.T) {
denebBlock, blobs := util.GenerateTestDenebBlockWithSidecar(t, [32]byte{}, 123, 4)
require.NoError(t, db.SaveBlock(context.Background(), denebBlock))
bs := filesystem.NewEphemeralBlobStorage(t)
testSidecars, err := verification.BlobSidecarSliceNoop(blobs)
require.NoError(t, err)
testSidecars := verification.FakeVerifySliceForTest(t, blobs)
for i := range testSidecars {
require.NoError(t, bs.Save(testSidecars[i]))
}
@@ -418,8 +417,7 @@ func TestBlobs_Electra(t *testing.T) {
electraBlock, blobs := util.GenerateTestElectraBlockWithSidecar(t, [32]byte{}, 123, params.BeaconConfig().MaxBlobsPerBlockByVersion(version.Electra))
require.NoError(t, db.SaveBlock(context.Background(), electraBlock))
bs := filesystem.NewEphemeralBlobStorage(t)
testSidecars, err := verification.BlobSidecarSliceNoop(blobs)
require.NoError(t, err)
testSidecars := verification.FakeVerifySliceForTest(t, blobs)
for i := range testSidecars {
require.NoError(t, bs.Save(testSidecars[i]))
}

View File

@@ -17,6 +17,7 @@ import (
"github.com/prysmaticlabs/prysm/v5/time/slots"
)
// Deprecated: use SSE from events for `payload attributes` instead
// ExpectedWithdrawals get the withdrawals computed from the specified state, that will be included in the block that gets built on the specified state.
func (s *Server) ExpectedWithdrawals(w http.ResponseWriter, r *http.Request) {
// Retrieve beacon state
@@ -125,7 +126,7 @@ func buildExpectedWithdrawalsData(withdrawals []*enginev1.Withdrawal) []*structs
func handleWrapError(err error, message string, code int) *httputil.DefaultJsonError {
return &httputil.DefaultJsonError{
Message: errors.Wrapf(err, message).Error(),
Message: errors.Wrap(err, message).Error(),
Code: code,
}
}

View File

@@ -153,6 +153,13 @@ func TestGetSpec(t *testing.T) {
config.UnsetDepositRequestsStartIndex = 92
config.MaxDepositRequestsPerPayload = 93
config.MaxPendingDepositsPerEpoch = 94
config.MaxBlobCommitmentsPerBlock = 95
config.MaxBytesPerTransaction = 96
config.MaxExtraDataBytes = 97
config.BytesPerLogsBloom = 98
config.MaxTransactionsPerPayload = 99
config.FieldElementsPerBlob = 100
config.KzgCommitmentInclusionProofDepth = 101
var dbp [4]byte
copy(dbp[:], []byte{'0', '0', '0', '1'})
@@ -191,7 +198,7 @@ func TestGetSpec(t *testing.T) {
data, ok := resp.Data.(map[string]interface{})
require.Equal(t, true, ok)
assert.Equal(t, 161, len(data))
assert.Equal(t, 168, len(data))
for k, v := range data {
t.Run(k, func(t *testing.T) {
switch k {
@@ -437,7 +444,7 @@ func TestGetSpec(t *testing.T) {
assert.Equal(t, "76", v)
case "REORG_MAX_EPOCHS_SINCE_FINALIZATION":
assert.Equal(t, "2", v)
case "REORG_WEIGHT_THRESHOLD":
case "REORG_HEAD_WEIGHT_THRESHOLD":
assert.Equal(t, "20", v)
case "REORG_PARENT_WEIGHT_THRESHOLD":
assert.Equal(t, "160", v)
@@ -538,6 +545,20 @@ func TestGetSpec(t *testing.T) {
assert.Equal(t, "9", v)
case "MAX_REQUEST_BLOB_SIDECARS_ELECTRA":
assert.Equal(t, "1152", v)
case "MAX_BLOB_COMMITMENTS_PER_BLOCK":
assert.Equal(t, "95", v)
case "MAX_BYTES_PER_TRANSACTION":
assert.Equal(t, "96", v)
case "MAX_EXTRA_DATA_BYTES":
assert.Equal(t, "97", v)
case "BYTES_PER_LOGS_BLOOM":
assert.Equal(t, "98", v)
case "MAX_TRANSACTIONS_PER_PAYLOAD":
assert.Equal(t, "99", v)
case "FIELD_ELEMENTS_PER_BLOB":
assert.Equal(t, "100", v)
case "KZG_COMMITMENT_INCLUSION_PROOF_DEPTH":
assert.Equal(t, "101", v)
default:
t.Errorf("Incorrect key: %s", k)
}

View File

@@ -142,7 +142,7 @@ func newTopicRequest(topics []string) (*topicRequest, error) {
} else if topicsForOpsFeed[name] {
req.needOpsFeed = true
} else {
return nil, errors.Wrapf(errInvalidTopicName, name)
return nil, errors.Wrap(errInvalidTopicName, name)
}
req.topics[name] = true
}

View File

@@ -4,7 +4,6 @@ go_library(
name = "go_default_library",
srcs = [
"handlers.go",
"helpers.go",
"server.go",
],
importpath = "github.com/prysmaticlabs/prysm/v5/beacon-chain/rpc/eth/light-client",
@@ -17,11 +16,9 @@ go_library(
"//beacon-chain/db:go_default_library",
"//beacon-chain/rpc/eth/shared:go_default_library",
"//beacon-chain/rpc/lookup:go_default_library",
"//beacon-chain/state:go_default_library",
"//config/features:go_default_library",
"//config/params:go_default_library",
"//consensus-types/interfaces:go_default_library",
"//consensus-types/primitives:go_default_library",
"//monitoring/tracing/trace:go_default_library",
"//network/httputil:go_default_library",
"//runtime/version:go_default_library",

View File

@@ -10,6 +10,7 @@ import (
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/v5/api"
"github.com/prysmaticlabs/prysm/v5/api/server/structs"
lightclient "github.com/prysmaticlabs/prysm/v5/beacon-chain/core/light-client"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/rpc/eth/shared"
"github.com/prysmaticlabs/prysm/v5/config/features"
"github.com/prysmaticlabs/prysm/v5/config/params"
@@ -182,18 +183,31 @@ func (s *Server) GetLightClientFinalityUpdate(w http.ResponseWriter, req *http.R
return
}
update, err := newLightClientFinalityUpdateFromBeaconState(ctx, s.ChainInfoFetcher.CurrentSlot(), st, block, attestedState, attestedBlock, finalizedBlock)
update, err := lightclient.NewLightClientFinalityUpdateFromBeaconState(ctx, s.ChainInfoFetcher.CurrentSlot(), st, block, attestedState, attestedBlock, finalizedBlock)
if err != nil {
httputil.HandleError(w, "Could not get light client finality update: "+err.Error(), http.StatusInternalServerError)
return
}
response := &structs.LightClientFinalityUpdateResponse{
Version: version.String(attestedState.Version()),
Data: update,
if httputil.RespondWithSsz(req) {
ssz, err := update.MarshalSSZ()
if err != nil {
httputil.HandleError(w, "Could not marshal finality update to SSZ: "+err.Error(), http.StatusInternalServerError)
return
}
httputil.WriteSsz(w, ssz, "light_client_finality_update.ssz")
} else {
updateStruct, err := structs.LightClientFinalityUpdateFromConsensus(update)
if err != nil {
httputil.HandleError(w, "Could not convert light client finality update to API struct: "+err.Error(), http.StatusInternalServerError)
return
}
response := &structs.LightClientFinalityUpdateResponse{
Version: version.String(attestedState.Version()),
Data: updateStruct,
}
httputil.WriteJson(w, response)
}
httputil.WriteJson(w, response)
}
// GetLightClientOptimisticUpdate - implements https://github.com/ethereum/beacon-APIs/blob/263f4ed6c263c967f13279c7a9f5629b51c5fc55/apis/beacon/light_client/optimistic_update.yaml
@@ -232,18 +246,31 @@ func (s *Server) GetLightClientOptimisticUpdate(w http.ResponseWriter, req *http
return
}
update, err := newLightClientOptimisticUpdateFromBeaconState(ctx, s.ChainInfoFetcher.CurrentSlot(), st, block, attestedState, attestedBlock)
update, err := lightclient.NewLightClientOptimisticUpdateFromBeaconState(ctx, s.ChainInfoFetcher.CurrentSlot(), st, block, attestedState, attestedBlock)
if err != nil {
httputil.HandleError(w, "Could not get light client optimistic update: "+err.Error(), http.StatusInternalServerError)
return
}
response := &structs.LightClientOptimisticUpdateResponse{
Version: version.String(attestedState.Version()),
Data: update,
if httputil.RespondWithSsz(req) {
ssz, err := update.MarshalSSZ()
if err != nil {
httputil.HandleError(w, "Could not marshal optimistic update to SSZ: "+err.Error(), http.StatusInternalServerError)
return
}
httputil.WriteSsz(w, ssz, "light_client_optimistic_update.ssz")
} else {
updateStruct, err := structs.LightClientOptimisticUpdateFromConsensus(update)
if err != nil {
httputil.HandleError(w, "Could not convert light client optimistic update to API struct: "+err.Error(), http.StatusInternalServerError)
return
}
response := &structs.LightClientOptimisticUpdateResponse{
Version: version.String(attestedState.Version()),
Data: updateStruct,
}
httputil.WriteJson(w, response)
}
httputil.WriteJson(w, response)
}
// suitableBlock returns the latest block that satisfies all criteria required for creating a new update

File diff suppressed because it is too large Load Diff

View File

@@ -1,45 +0,0 @@
package lightclient
import (
"context"
"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/v5/api/server/structs"
lightclient "github.com/prysmaticlabs/prysm/v5/beacon-chain/core/light-client"
"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
)
func newLightClientFinalityUpdateFromBeaconState(
ctx context.Context,
currentSlot primitives.Slot,
state state.BeaconState,
block interfaces.ReadOnlySignedBeaconBlock,
attestedState state.BeaconState,
attestedBlock interfaces.ReadOnlySignedBeaconBlock,
finalizedBlock interfaces.ReadOnlySignedBeaconBlock,
) (*structs.LightClientFinalityUpdate, error) {
result, err := lightclient.NewLightClientFinalityUpdateFromBeaconState(ctx, currentSlot, state, block, attestedState, attestedBlock, finalizedBlock)
if err != nil {
return nil, err
}
return structs.LightClientFinalityUpdateFromConsensus(result)
}
func newLightClientOptimisticUpdateFromBeaconState(
ctx context.Context,
currentSlot primitives.Slot,
state state.BeaconState,
block interfaces.ReadOnlySignedBeaconBlock,
attestedState state.BeaconState,
attestedBlock interfaces.ReadOnlySignedBeaconBlock,
) (*structs.LightClientOptimisticUpdate, error) {
result, err := lightclient.NewLightClientOptimisticUpdateFromBeaconState(ctx, currentSlot, state, block, attestedState, attestedBlock)
if err != nil {
return nil, err
}
return structs.LightClientOptimisticUpdateFromConsensus(result)
}

Some files were not shown because too many files have changed in this diff Show More