Adds the processing of execution payload envelope
Corrects the protos for attestations and slashings in Electra versions
Adds generators of full blocks for Electra
* Add `execution_payload` and `payload_attestation_message` topics
* Set `SourcePubkey` to 48 bytes long
* Add randomly populated `PayloadAttestationMessage` object
* Add tests for `execution_payload` and `payload_attestation_message` topics
* Allow nodes with and without payload in forkchoice
This PR takes care of adding nodes to forkchoice that may or may not
have a corresponding payload. The rationale is as follows
- The node structure is kept almost the same as today.
- A zero payload hash is considered as if the node was empty (except for
the tree root)
- When inserting a node we check what the right parent node would be
depending on whether the parent had a payload or not.
- For pre-epbs forks all nodes are full, no logic changes except a new
steps to gather the parent hash that is needed for block insertion.
This PR had to change some core consensus types and interfaces.
- It removed the ROBlockEPBS interface and added the corresponding ePBS
fields to the ReadOnlyBeaconBlockBody
- It moved the setters and getters to epbs dedicated files.
It also added a checker for `IsParentFull` on forkchoice that simply
checks for the parent hash of the parent node.
* review
* Add `IndexedPayloadAttestation` container
* Add `GetPayloadAttestingIndices` and its unit test
* Add `GetIndexedPayloadAttestation` and its unit test
* Add `is_valid_indexed_payload_attestation` and its unit test
* Create a smaller set of validators for faster unit test
* Pass context to `GetPayloadTimelinessCommittee`
* Iterate `ValidatorsReadOnly` instead of copying all validators
This implements a helper to get the ptc committee from a state. It uses
the cached beacon committees if possible
It also implements a helper to compute the largest power of two of a
uint64 and a helper to test for nil payload attestation messages
* Remove Feature Flag From Prater (#12082)
* Use Epoch boundary cache to retrieve balances (#12083)
* Use Epoch boundary cache to retrieve balances
* save boundary states before inserting to forkchoice
* move up last block save
* remove boundary checks on balances
* fix ordering
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* chore: fix unfound image
* fix link
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* missed adding electra process epoch on state replay
* abstracting common functionality as to not forget in processing
* removing unnessesary error check
* breaking out core process slots logic out
* fixing small bug in epoch processing
* adding small log to fix linting and make use of logrus in the package
* adding some unit tests for process epoch
* potuz's review feedback
* reversing processing order based on feedback
* feat: implement ``is_better_update`` for light client
* replace ``!=`` with ``!reflect.DeepEqual``
* chore: import constant instead of redeclaring and some minor fixes
* feat: add test for ``newHasSupermajority != oldHasSupermajority``
* chore: capitalise first letter of function names
* refactor: rewrite test setups in a more readable manner
* two test cases
* feat: add tests for ``IsBetterUpdate``
* refactor: remove double import
* wording fix 1
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* wording fix 2
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* minor fixes
* fix: change function names to lowercase and fix dependencies
---------
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* moving beacon block fields from cloners to individual files and adding fuzzing tests
* adding missed tests
* fixing tests:
* removing deep not equals for now
* adding clarifying comment
* using fork instead of eip6110flag for validator index cache
* updating validator indicies on initialize from poroto unsafe
* fixing bad version check
* attempting to fix unit test
* reverting test changes
* changing incorrect initialization
* updating comment
* radek's comment'
* poc for payload electra cloning
* partial fixes
* fixing build
* addressing kasey's comment
* forgot to unexport interface
* making test more generic
* making fuzzing slightly more robust
* renaming based on kasey's comment
* using fuzz test in same package to avoid exporting interface
* fixing build
* fixing marshalling and adding more to unit test
* updating missed consolidation requests
* renaming variables
* adding test gen
* reverting config change
* `PruneProposalsAtEpoch`: Test return value in case of nothing to prune.
* `TestStore_PruneAttestations_OK`: Create unique validator indexes.
Before this commit, `attester2` for `j = n` was the same than
`attester1` for `j = n + 1`, resulting in erasure of a lot of attesters.
I guess it was not the initial intent.
* Slasher pruning: Check if the number of pruned items corresponds to the expectation.
Before this commit, if the pruning function did remove a superset
of the expected pruned items (including all the items), then the test would pass.
* Prune items that should be pruned and stop pruning items that should not be pruned.
The first 8 bytes of the key of `attestation-data-roots` and
`proposal-records` bytes correspond respectively to an encoded epoch and
and encoded slot.
The important word in this sentence is "encoded".
Before this commit, these slot/epoch are SSZ encoded, which means that
they are little-endian encoded.
However:
- `uint64PrefixGreaterThan` uses `bytes.Compare` which expects
big-endian encoded values.
- `for k, _ := c.First(); k != nil; k, _ = c.Next()` iters over the
keys in big-endian order.
The consequence is:
- Some items that should be pruned are not pruned, provoking a disk
usage leak.
- Some items that should not be pruned are pruned, provoking errors like
in https://github.com/prysmaticlabs/prysm/issues/13658.
This commit encodes the slot/epoch as big-endian before storing them in
the database keys.
Why this bug has not been detected in unit test before?
The values used in unit tests before this commit in
`TestStore_PruneProposalsAtEpoch` and `TestStore_PruneAttestations_OK`
are `10` and `20`.
Unfortunately, checking if `littleEndian(20) > littlenEndien(10)`
with the `>` operator considering operands as big-endian encoded returns
the expected result...
Just replacing `20` by `30` trigs the bug.
* Make deepsource happy.
* Slasher: Migrate database from little-endian to big-endian.
* Update beacon-chain/slasher/service.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/db/slasherkv/migrate.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `TestMigrate`: Fix documentation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Bump go version up
* Update to go 1.22 compatible version
* Fix NoSec declarations
* Skip Gosec in GolangCi
* Avoid Bug In Analyzer
* Add in Gohashtree and Update to 1.22.4
* Fix Go Sum
* validator/client: process Sync Committee roles separately
In a DV context, to be compatible with the proposed selection endpoint, the VC must push all partial selections to it instead of just one.
Process sync committee role separately within the RolesAt method, so that partial selections can be pushed to the DV client appropriately, if configured.
* testing/util: re-add erroneously deleted state_test.go
* validator/client: fix tests
* validator/client: always process sync committee validator
Process sync committee duty at slot boundary as well.
* don't fail if validator is marked as sync committee but it is not in the list
ignore the duty and continue
* address code review comments
* fix build
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* eip-7251: registry_updates
* Fix spectest violation
* More unit tests for process_registry_updates
* Update tests for ProcessRegistryUpdates
* Minor refactorings and benchmark
* Update beacon-chain/core/electra/registry_updates.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* Wrap errors from process_registry_updates for easier debugging
* Rewrite process_registry_updates not to use st.Validators()
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* wip changes
* wip changes refactoring deposit functions
* wip on deposit functions
* wip changes to fix deposit processing
* more wip function logic
* gaz
* wip refactoring deposit changes
* removing circlular dependency and other small fix
* fixing circular dependencies
* fixing validators file
* fixing more tests
* fixing unit tests
* wip deposit packing
* refactoring packing
* changing packing code some more
* fixing packing change
* updating more tests
* removing comment
* fixing transition code for eip6110
* including inserts for validator index cache
* adding tests and placeholder test
* moving deposit related unit tests over
* adding in test for electra proposer packing
* spec test wip
* moving cache saving to the correct spot
* eip-6110: deposit spectests
* adding deposit receipt spec tests
* reverting altair operations in this pr and will be handled separately
* fixing renames but need to refactor process deposits
* gaz
* fixing names
* fixing linting
* fixing unit test
* fixing a test and updating test util
* bal never used
* fixing more tests
* fixing one more test
* addressing feedback
* refactoring process deposits to be part of their own fork instead of in blocks package
* adding new tests to cover functions and removing redundancies
* removing comment based on feedback
* resolving easy deepsource issue
* refactoring for appropriate aliasing and readability
* reverting some changes to simplify diff
* small edit to comment
* fixing linting
* fixing merge changes and review comments
* Update beacon-chain/rpc/prysm/v1alpha1/validator/proposer_deposits.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* partial review comments
* addressing slack feedback
* moving function from deposits to validator.go
* adding in batch verification and improving logs
---------
Co-authored-by: Preston Van Loon <preston@pvl.dev>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `IndexedAtt` wrapper for the slasher feed
* test fixes
* fix simulator
* fix e2e
* Revert "Auxiliary commit to revert individual files from 191bbf77accfc2523fa9f909837a2e9dca132afa"
This reverts commit 2b0441a23a0e5f66e50cf36c3bbfbb39d587b17b.
* extract interface from channel
* adding logic for saving validator indicies in new cache
* updating index cache for eip6110
* Update setters_misc.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* linting
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* renaming functions and fields based on consensus changes
* execution api rename
* fixing test
* reverting spectests changes, it should be changed with new version
* reverting temporarily
* revert exclusions
* Increase health tracker channel buffer size
* keep only the latest value
* Make health test blocking as a regression test for PR #14807
* Fix new race conditions in the MockHealthClient
---------
Co-authored-by: Preston Van Loon <preston@pvl.dev>
* always close cache warm chan to prevent blocking
* test that waitForCache does not block
* combine defers to reduce cognitive overhead
* lint
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* eip-7251: process_effective_balance_updates
Spectests for process_effective_balance_updates
process_effective_balance_updates unit tests
* PR feedback from the amazing @rkapka
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Initial commit
* Fuzz all generated ssz en/decoders.
* Make proto message comparison aware of nested proto messages and differing unknown fields therein.
* Don't name receiver variable if not necessary.
* Run gazelle
* Cleanup part 1
* Cleanup part 2
* Cleanup part 3
* remove lock field init
* doc for SignerConfig
* remove vars
* use full Keymanager word in function
* revert interface rename
* linter
* fix build issues
* review
* eip-7251: process_pending_balance_deposits
* Update beacon-chain/core/electra/balance_deposits_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Remove defensive check. A unit test shows nothing bad happens
* Safe sub to protect from underflow
* Use @kasey's idea for safer subtraction
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* electra engine api support
* add marshaling support for ExecutionPayloadElectra
* add receipts to json tests
* deep source
* simplify slice handling
* deep source lint about type/method order
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* wip fork logic upgrade
* fixing replay and fork.go
* improving process function and adding tests for transition
* updating unit tests and temporarily removing some fields on state_trie.go
* updating state
* wip adding upgrade to electra code
* adding some comments
* adding spec tests
* fixing values used in state transition logic
* updating upgrade test
* gofmt
* avoid dup word linting
* fixing spec tests for fork
* gaz
* fixing tests
* improving unit test with new getters
* fixing bazel for minimal fork test
* adding bazel file
* Update beacon-chain/core/electra/upgrade.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* addressing some comments and adding more tests
* addressing more feedback
* one more feedback
* changing value to interface after talking to preston
* adding missed review feedback
* fixing linting
* noticed I was using the wrong function in the state upgrade
* fixing and ignoring some deepsource issues
* moving core electra validator functions to helper to remove circular dependencies in other PRs
* missed deepsource complaint
* Update upgrade.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update testing/util/electra_state.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update testing/util/electra_state.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* addressing feedback
* removing deepsoure ignore comments
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Electra: Beacon State
* Electra: Beacon state fixes from PR 13919
* Add missing tests - part 1
* Split eip_7251_root.go into different files and reuse/share code with historical state summaries root. It's identical!
* Add missing tests - part 2
* deposit receipts start index getters and setters (#13947)
* adding in getters and setters for deposit receipts start index
* adding tests
* gaz
* Add missing tests - part 3 of 3
Update the electra withdrawal example with a ssz state containing pending partial withdrawals
* add tests for beacon-chain/state/state-native/getters_balance_deposits.go
* Add electra field to testing/util/block.go execution payload
* godoc commentary on public methods
* Fix failing test
* Add balances index out of bounds check and relevant tests.
* Revert switch case electra
* Instead of copying spectest data into testdata, use the spectest dependency
* Deepsource fixes
* Address @rkapka PR feedback
* s/MaxPendingPartialsPerWithdrawalSweep/MaxPendingPartialsPerWithdrawalsSweep/
* Use multivalue slice compatible accessors for validator and balance in ActiveBalanceAtIndex
* More @rkapka feedback. What a great reviewer!
* More tests for branching logic in ExitEpochAndUpdateChurn
* fix build
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Electra: full beacon config
Fix TestGetSpec
* Fix beacon config spec compliance test so that it properly loads the config from spec tests. Tests failing for now.
* fix tests and comply with spec presets
* block protos
* proto and ssz
* stubs
* Enable Electra spec test
* Pull in EIP-7251 protobuf changes
From: https://github.com/prysmaticlabs/prysm/pull/13903
* All EIP7549 containers are passing
* All EIP7251 containers passing
* including changes from eip7002
* Everything passing except for beacon state hash tree root
* fixing eletra state to use electra payload
* Fix minimal test. Skip beacon state test
* Perston's feedback
---------
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>
* Remove EnableEIP4881 flag
* Gaz
* Fix missing error handler
* Remove old tree and fix tests
* Gaz
* Fix build import
* Replace depositcache
* Add pendingDeposit tests
* Nishant's fix
* Fix unsafe uint64 to int
* Fix other unsafe uint64 to int
* Remove: RemovePendingDeposit
* Deprecate and remove DisableEIP4881 flag
* Check: index not greater than deposit count
* Move index check
* GET
* POST
* Revert "Auxiliary commit to revert individual files from 615feb104004d6a945ededf5862ae38325fc7ec2"
This reverts commit 55cf071c684019f3d6124179154c10b2277fda49.
* comment fix
* deepsource
* Beacon-api: broadcast blobs in the event of seen block
* Fix parameters
* Fix test
* Check forkchoice
* Ran go format
* Revert "Ran go format"
This reverts commit 091e77e81d6e2b9861fecc27c0bad1898033f9a3.
* James feedback
* Radek's feedback
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Fix bad tests
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* allow other pkgs to check for blobs in pruning cache
* address deepsource complaints
* custom error to simplify test setup
* add AllAvailable method
* make storage summary slot field private
* unit test and off-by-one fix
* remove comment with copy of tested function
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Revert "Modify the algorithm of `updateFinalizedBlockRoots` (#13486)"
This reverts commit 32fb183392.
* migration to fix index corruption from pr 13486
* bail as soon as we see 10 epochs without the bug
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Remove check for duplicates in pending attestation queue
The current queue will only save 1 unaggregated attestation for a pending block because we wrap the object into a SignedAggregatedAttestationAndProof with a zeroed aggregator.
* fix tests
* add max/min span visualisation tool cli
* go mod tidy
* lint imports
* remove typo
* fix epoch table value
* fix deepsource
* add dep to bazel
* fix dep import order
* change command name from span to slasher-span-display
* change command args style using - instead of _
* sed s/CONFIGURATION/SLASHER PARAMS//
* change double neg to double pos condition
* remove unused anonymous func
* better function naming
* add range condition
* [deepsource] Fix Empty slice literal used to declare a variable
GO-W1027
* correct typo
* do not show incorrect epochs due to round robin
* fix import
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
ValidateClock in ValidateAttestationTime is useless
The check is that the attSlot is not > than the currentslot + 128 slots.
Later there's a check that the attSlot start time is not > than current slot
start time + clockDisparity.
if attSlot > than currentSlot + 128 slots, then the second check would fail
anyway.
The lattest check already guarantees that the attSlot cannot be larger than the
currentSlot, therefore it may never happen that attEpoch > currentEpoch. We just
need to check for Deneb that attEpoch >= currentEpoch - 1.
Removes also some duplicated variables like the attestation epoch being computed
twice.
* refactor batch verifier to share with pending queue
* unit test for batch verifier
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Remove Feature Flag From Prater (#12082)
* Use Epoch boundary cache to retrieve balances (#12083)
* Use Epoch boundary cache to retrieve balances
* save boundary states before inserting to forkchoice
* move up last block save
* remove boundary checks on balances
* fix ordering
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* create the log file along with its parent directory if not present
* only give ReadWritePermissions to the log file
* use io/file package to create the parent directories
* fix ci related issues
* add regression tests
* run gazelle
* fix tests
* remove print statements
* gazelle
* Remove failing test for MkdirAll, this failure is not expected
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* add log message if in da check at slot end
* don't bother logging late da check start
* break up defer with a var, too dense all together
* pass slot instead of block ref
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* addressing errors on events endpoint
* reverting timeout on get health
* fixing linting
* fixing more linting
* Update validator/client/beacon-api/beacon_api_validator_client.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/rpc/eth/events/events.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* reverting change and removing line on context done which creates a superfluous response.WriteHeader error
* gofmt
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* make it very big
* use new pruning implementation
* handle pre deneb
* revert cache change
* less verbose
* gaz
* Update beacon-chain/operations/attestations/prune_expired.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* gofmt
* be safer
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* handle special case of batch size=1
* unit test case for backfill batch len=1
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* flag to set an older backfill slot target
* wire up flag to main cli and usage
* fix deepsource complaints
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `subscribeStaticWithSubnets`: Fix docstring.
* `buildOptions`: Avoid `options` mutations.
* `dv5Cfg`: Avoid mutation.
* `RefreshENR`: Use default for all but Phase0.
* `udp4`, `udp6`: Create enum.
* `p2p.Config`: `BootstrapNodeAddr`==> `BootstrapNodeAddrs`.
* `p2p.Config`: `Discv5BootStrapAddr` ==> `Discv5BootStrapAddrs`.
* `TestScorers_BadResponses_Score`: Improve.
* `BeaconNode`: Avoid mutation.
* `TestStore_TrustedPeers`: Remove blankline.
* Remove blank identifiers.
* `privKey`: Keep the majority of code with low indentation.
* `P2PPreregistration`: Return error instead of fatal log.
* `parseBootStrapAddrs` => `ParseBootStrapAddrs` (export)
* `p2p.Config`: Remove `BootstrapNodeAddrs`.
* `NewService`: Avoid mutation when possible.
* `Service`: Remove blank identifier.
* `buildOptions`: Avoid `log.Fatalf` (make deepsource happy).
* `registerGRPCGateway`: Use `net.JoinHostPort` (make deepsource happy).
* `registerBuilderService`: Make deepsource happy.
* `scorers`: Add `NoLock` suffix (make deepsource happy).
* `scorerr`: Add some `NoLock`suffixes (making deepsource happy).
* `discovery_test.go`. Remove init.
Rationale:
`rand.Seed` is deprecated: As of Go 1.20 there is no reason to call Seed with a random value. Programs that call Seed with a known value to get a specific sequence of results should use New(NewSource(seed)) to obtain a local random generator.
This makes deepsource happy as well.
* `createListener`: Reduce cyclomatic complexity (make deepsource happy).
* `startDB`: Reduce cyclomatic complexity (make deepsource happy).
* `main`: Log a FATAL on error.
This way, the error message is very readable.
Before this commit, the error message is the less readable
message in the logs.
* `New`: Reduce cyclomatic complexity (make deepsource happy).
* `main`: Avoid `App` mutation, and make deepsource happy.
* Update beacon-chain/node/node.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* `bootnodes` ==> `BootNodes` (Fix PR comment).
* Remove duplicate `configureFastSSZHashingAlgorithm` since already done in `configureBeacon`. (Fix PR comment)
* Add `TestCreateLocalNode`. (PR comment fix.)
* `startModules` ==> `startBaseServices (Fix PR comment).
* `buildOptions` return errors consistently.
* `New`: Change ordering.
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* use real blob verifier in forkchoice spectest
* wip
* Use real blob sidecar for test
* Set file db correctly
* correctly handle blob cases where valid=false
* work-around spectest's weird Fork in genesis state
* gaz
* revert T-money's log level change
* rm whitespace
* unskip minimal test
* Preston's feedback
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* add them
* Ensure activation epoch does not overflow
* add them all in
* better check for overflow
* fix tests
* fix tests
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* `EpochFromString`: Use already defined `Uint64FromString` function.
* `Test_uint64FromString` => `Test_FromString`
This test function tests more functions than `Uint64FromString`.
* Slashing protection history: Remove unreachable code.
The function `NewKVStore` creates, via `kv.UpdatePublicKeysBuckets`,
a new item in the `proposal-history-bucket-interchange`.
IMO there is no real reason to prefer `proposal` than `attestation`
as a prefix for this bucket, but this is the way it is done right now
and renaming the bucket will probably be backward incompatible.
An `attestedPublicKey` cannot exist without
the corresponding `proposedPublicKey`.
Thus, the `else` portion of code removed in this commit is not reachable.
We raise an error if we get there.
This is also probably the reason why the removed `else` portion was not
tested.
* `NewKVStore`: Switch items in `createBuckets`.
So the order corresponds to `schema.go`
* `slashableAttestationCheck`: Fix comments and logs.
* `ValidatorClient.db`: Use `iface.ValidatorDB`.
* BoltDB database: Implement `GraffitiFileHash`.
* Filesystem database: Creates `db.go`.
This file defines the following structs:
- `Store`
- `Graffiti`
- `Configuration`
- `ValidatorSlashingProtection`
This files implements the following public functions:
- `NewStore`
- `Close`
- `Backup`
- `DatabasePath`
- `ClearDB`
- `UpdatePublicKeysBuckets`
This files implements the following private functions:
- `slashingProtectionDirPath`
- `configurationFilePath`
- `configuration`
- `saveConfiguration`
- `validatorSlashingProtection`
- `saveValidatorSlashingProtection`
- `publicKeys`
* Filesystem database: Creates `genesis.go`.
This file defines the following public functions:
- `GenesisValidatorsRoot`
- `SaveGenesisValidatorsRoot`
* Filesystem database: Creates `graffiti.go`.
This file defines the following public functions:
- `SaveGraffitiOrderedIndex`
- `GraffitiOrderedIndex`
* Filesystem database: Creates `migration.go`.
This file defines the following public functions:
- `RunUpMigrations`
- `RunDownMigrations`
* Filesystem database: Creates proposer_settings.go.
This file defines the following public functions:
- `ProposerSettings`
- `ProposerSettingsExists`
- `SaveProposerSettings`
* Filesystem database: Creates `attester_protection.go`.
This file defines the following public functions:
- `EIPImportBlacklistedPublicKeys`
- `SaveEIPImportBlacklistedPublicKeys`
- `SigningRootAtTargetEpoch`
- `LowestSignedTargetEpoch`
- `LowestSignedSourceEpoch`
- `AttestedPublicKeys`
- `CheckSlashableAttestation`
- `SaveAttestationForPubKey`
- `SaveAttestationsForPubKey`
- `AttestationHistoryForPubKey`
* Filesystem database: Creates `proposer_protection.go`.
This file defines the following public functions:
- `HighestSignedProposal`
- `LowestSignedProposal`
- `ProposalHistoryForPubKey`
- `ProposalHistoryForSlot`
- `ProposedPublicKeys`
* Ensure that the filesystem store implements the `ValidatorDB` interface.
* `slashableAttestationCheck`: Check the database type.
* `slashableProposalCheck`: Check the database type.
* `slashableAttestationCheck`: Allow usage of minimal slashing protection.
* `slashableProposalCheck`: Allow usage of minimal slashing protection.
* `ImportStandardProtectionJSON`: Check the database type.
* `ImportStandardProtectionJSON`: Allow usage of min slashing protection.
* Implement `RecursiveDirFind`.
* Implement minimal<->complete DB conversion.
3 public functions are implemented:
- `IsCompleteDatabaseExisting`
- `IsMinimalDatabaseExisting`
- `ConvertDatabase`
* `setupDB`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/node/node_test.go`.
The rest of this commit consists in minimal slashing protection testing.
* `setupWithKey`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/client/propose_test.go`.
The rest of this commit consists in tests wrapping.
* `setup`: Add `isSlashingProtectionMinimal` argument.
The added feature is located in the `validator/client/propose_test.go`
file.
The rest of this commit consists in tests wrapping.
* `initializeFromCLI` and `initializeForWeb`: Factorize db init.
* Add `convert-complete-to-minimal` command.
* Creates `--enable-minimal-slashing-protection` flag.
* `importSlashingProtectionJSON`: Check database type.
* `exportSlashingProtectionJSON`: Check database type.
* `TestClearDB`: Test with minimal slashing protection.
* KeyManager: Test with minimal slashing protection.
* RPC: KeyManager: Test with minimal slashing protection.
* `convert-complete-to-minimal`: Change option names.
Options were:
- `--source` (for source data directory), and
- `--target` (for target data directory)
However, since this command deals with slashing protection, which has
source (epochs) and target (epochs), the initial option names may confuse
the user.
In this commit:
`--source` ==> `--source-data-dir`
`--target` ==> `--target-data-dir`
* Set `SlashableAttestationCheck` as an iface method.
And delete `CheckSlashableAttestation` from iface.
* Move helpers functions in a more general directory.
No functional change.
* Extract common structs out of `kv`.
==> `filesystem` does not depend anymore on `kv`.
==> `iface` does not depend anymore on `kv`.
==> `slashing-protection` does not depend anymore on `kv`.
* Move `ValidateMetadata` in `validator/helpers`.
* `ValidateMetadata`: Test with mock.
This way, we can:
- Avoid any circular import for tests.
- Implement once for all `iface.ValidatorDB` implementations
the `ValidateMetadata`function.
- Have tests (and coverage) of `ValidateMetadata`in
its own package.
The ideal solution would have been to implement `ValidateMetadata` as
a method with the `iface.ValidatorDB`receiver.
Unfortunately, golang does not allow that.
* `iface.ValidatorDB`: Implement ImportStandardProtectionJSON.
The whole purpose of this commit is to avoid the `switch validatorDB.(type)`
in `ImportStandardProtectionJSON`.
* `iface.ValidatorDB`: Implement `SlashableProposalCheck`.
* Remove now useless `slashableProposalCheck`.
* Delete useless `ImportStandardProtectionJSON`.
* `file.Exists`: Detect directories and return an error.
Before, `Exists` was only able to detect if a file exists.
Now, this function takes an extra `File` or `Directory` argument.
It detects either if a file or a directory exists.
Before, if an error was returned by `os.Stat`, the the file was
considered as non existing.
Now, it is treated as a real error.
* Replace `os.Stat` by `file.Exists`.
* Remove `Is{Complete,Minimal}DatabaseExisting`.
* `publicKeys`: Add log if unexpected file found.
* Move `{Source,Target}DataDirFlag`in `db.go`.
* `failedAttLocalProtectionErr`: `var`==> `const`
* `signingRoot`: `32`==> `fieldparams.RootLength`.
* `validatorClientData`==> `validator-client-data`.
To be consistent with `slashing-protection`.
* Add progress bars for `import` and `convert`.
* `parseBlocksForUniquePublicKeys`: Move in `db/kv`.
* helpers: Remove unused `initializeProgressBar` function.
When a late block arrives and the beacon is proposing the next block, we
perform several checks to allow for the next block to reorg the incoming
late block.
Among those checks, we check that the parent block has been heavily
attested (currently 160% of the committee size).
We perform this check in these circumstances:
- When the late block arrives
- At 10 seconds into the slot
- At 0 seconds into the next slot (at proposing time)
The problem is that for blocks that arrive between 4 seconds and 10
seconds, the parent block will not have yet this expected weight since
attestations from the current committee were not imported yet, and thus
Prysm will send an FCU with payload attributes anyway at this time.
What happens is that Prysm keeps the EL building different blocks based
on different parents at the same time, when later in the next slot it
calls to propose, it will reorg the late block anyway and the EL would
have been computing a second payload uselessly.
This PR enables this check only when calling `ShouldOverrideFCU` after
10 seconds into the slot which we do only after having imported the
current attestations. We may want to actually remove this check entirely
from `ShouldOverrideFCU` and only keep it in `ProposerHead`.
Shout out to Anthithesis for reporting an issue that led to this
discoverly.
* employ a val reader to prevent constant copies
* clean it up and fix tests
* gaz
* radek's review
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Improve package documentation.
* `processAttestations`: Improve logging.
* Add `Benchmark_checkSurroundVotes` benchmark.
* Implement `saveChunksToDisk` as remplacement of `saveUpdatedChunks`.
The idea is to open only on DB transaction for all validator chunk indexes instead of
one DB transaction per validator chunk index.
It saves the overhead due to transaction start/stop of the DB.
Result of `Benchmark_checkSurroundVotes`:
- Before this commit: 133 seconds
- After this commit: 5.05 seconds
* `LoadSlasherChunks` and `SaveSlasherChunks`: Batch.
* `loadChunks` ==> `loadChunksFromDisk`
* `updatedChunkByChunkIndex`: Don't update if `latestEpochWritten == currentEpoch `.
* `updatedChunkByChunkIndex`: Load all needed chunks once.
* `latestEpochWritten` ==> `latestEpochUpdated`.
* `checkSurroundVotes`: Dump to disk at most every `25_600` chunks.
* `SaveAttestationRecordsForValidators`: Batch.
* `batchSize`: Use as package const and add comment.
* pending broadcast err if missing blobs and 0 peers
* compute request first for len check
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `getChunk` ==> `getChunkFromDatabase`.
* `loadChunks`: Rename variables.
* `Update`: Use explicit arguments.
* `detect_attestations.go`: Reduce abstraction layers.
* `loadAndUpdateChunks`: Change arguments order.
* `updatedChunkByChunkIndex`: Update all known validators in the chunk.
* `LastEpochWrittenForValidators`: Avoid avoidable `for`loop.
* `chunks.go`: Ensure implementations respect the interface.
* `LastEpochWrittenForValidators`: Stop considering lack of epoch as genesis epoch.
* `updatedChunkByChunkIndex`: Don't update latest updated epoch.
And add a bunch of tests.
* Improve slasher cold boot duration.
Before this commit, on a slasher cold boot (aka, without any db),
the `updatedChunkByChunkIndex` function looped for all validators
AND for all epochs between the genesis epoch and the current epoch.
This could take several dozen of minutes, and it is useless since the
min/max spans are actually a circular buffer with a limited lenght.
Cells of min/max spans can be overwritten (with the same value)
plenty of times.
After this commit, the `updatedChunkByChunkIndex` function loops
for all validators AND AT most 'historyLength' lenght.
Every cell of min/max spans are written AT MOST once.
Time needed for slasher boot goes from `O(nm)` to "only" `O(m)`, where:
- `n` is the number of epochs since the genesis.
- `m` is the number of validators.
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* retrieve and save blobs during backfill
* Update beacon-chain/sync/backfill/batch.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/sync/backfill/blobs.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/sync/backfill/metrics.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* make blobSync initialization path a little safer
* use bytes.Equal to avoid extra allocation
* stop using log.Fatal and other little cleanups
* wrap up post blob sync actions in batch mutator
* unit test coverage on verifier
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `LastEpochWrittenForValidators`: Use golang if.
* `SaveLastEpochsWrittenForValidators`: Refactor.
* `SaveLastEpochsWrittenForValidators`: Fix when `epochByValIndex` > `batchSize`.
Before this commit, `TestStore_LastEpochWrittenForValidators` works if `validatorsCount <= 10000`
and stops working if `validatorsCount > 10000`.
* Slasher: Detect surrounded votes in multiple batches.
Fixes https://github.com/prysmaticlabs/prysm/issues/13591.
* `Test_processAttestations`: Remove duplicated tests.
* Sort indexed attestations by data root.
* `processAttestations`: Don't return duplicate slashings anymore.
Fix https://github.com/prysmaticlabs/prysm/issues/13592.
* `AttesterDoubleVote`: Rename fields.
* Detect double votes in different batches.
In order to do that:
1. Each attestation of the batch is tested against the other attestations of the batch.
2. Each attestation of the batch is tested against the content of the database.
2. Attestations are saved into the database.
Fixes https://github.com/prysmaticlabs/prysm/issues/13590.
* `helpers.go`: Improve naming consistency.
* `detect_attestations.go`: Improve readability.
* `receive.go`: Add `attsQueueSize` in log message.
* `checkSlashableAttestations`: Improve logging.
`avgBatchProcessingTime` is not displayed any more if not batch is
processed.
* `loadChunks`: Use explicit `chunkKind` and `chunkIndices`.
* `getChunk`: Use specific `chunkIndex` and `chunkKind`.
* `validatorIndicesInChunk` -> `validatorIndexesInChunk`.
* `epochUpdateForValidator`: Use explicit arguments.
* `getChunk`: Change order of arguments.
* `latestEpochWrittenForValidator`: Use `ok` parameter.
So the default value is not any more considered as the absence of
value.
* `applyAttestationForValidator`: Use explicit arguments.
* `updateSpans`: Use explicit arguments.
* `saveUpdatedChunks`: Use explicit arguments.
* `checkSurrounds`: Use explicit arguments.
We see here that, previously, in `checkSlashableAttestations`,
`checkSurrounds` was called with the default value of `slashertypes`: `MinSpan`.
Now, we set it expliciterly at `MinSpan`, which may explicit a bug.
* `epochUpdateForValidator`: Set modified by the function argument first.
* `applyAttestationForValidator`: Set mutated argument `chunksByChunkIdx`first.
* `applyAttestationForValidator`: Rename variables.
* `Test_processQueuedAttestations`: Fix test.
Two tests were actually exactly the same.
* `updateSpans`: Keep happy path in the outer scope.
Even if in this case the "happy" path means slashing.
* `checkSurrounds`: Rename variable.
* `getChunk`: Avoid side effects.
It adds a few lines for callers, but it does not modify any more
arguments and it does what it says: getting a chunk.
* `CheckSlashable`: Flatten.
* `detect_attestations_test.go`: Simplify.
* `CheckSlashable`: Add error log in case of missing attestation.
* `processQueuedAttestations`: Extract a sub function.
So testing will be easier.
* `processAttesterSlashings` and `processProposerSlashings`: Improve.
* `processAttesterSlashings`: Return processed slashings.
* `createAttestationWrapper`: Rename variables.
* `signingRoot` ==> `headerRoot` or `dataRoot`.
Before this commit, there is two typse of `signing root`s floating around.
- The first one is a real signing root, aka a hash tree root computed from an object root and
a domain. This real signing root is the object ready to be signed.
- The second one is a "false" signing root, which is actually just the hash tree root of an object. This object is either the `Data` field of an attestation, or the `Header` field of a block.
Having 2 differents objects with the same name `signing root` is quite confusing.
This commit renames wrongly named `signing root` objects.
* `createAttestationWrapper` => `createAttestationWrapperEmptySig`.
So it's clear for the user that the created attestation wrapper has an empty signature.
* Implement `createAttestationWrapper`.
* `processAttestations`: Return processed attester slashings.
* Test `processAttestations` instead of `processQueuedAttestations`.
By testing `processAttestations` instead of `processQueuedAttestations`, we get rid of a lot of tests fixtures, including the 200 ms sleep.
The whole testing duration is shorter.
* `Test_processAttestations`: Allow multiple steps.
* `Test_processAttestations`: Add double steps tests.
Some new failing tests are commented with a corresponding github issue.
* `NextChunkStartEpoch`: Fix function comment.
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `chunks.go`: Avoid templating log messages.
* `checkSlashableAttestations`: Simplify duration computation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Add coreutils to docker images
* add coreutils dependencies
* Add a prysmaticlabs.com/uploads backup of the deb files
* Run gazelle and fix issues
* Remove broken tar, change http_archive deps to debian_archive, remove http mirrors in favor of snapshot
* Add comments about which deps are required by other deps
* Slasher: Ensure all gorouting are stopped before running `Stop` actions.
Fixes#13550.
In tests, `exitChan` are now useless since waitgroup are used to wait
for all goroutines to be stopped.
* `slasher.go`: Add comments and rename some variables. - NFC
* `detect_blocks.go`: Improve. - NFC
- Rename some variables.
- Add comments.
- Use second element of `range` when possible.
* `chunks.go`: Remove `_`receivers. - NFC
* `validateAttestationIntegrity`: Improve documentation. - NFC
* `filterAttestations`: Avoid `else`and rename variable. - NFC
* `slasher.go`: Fix and add comments.
* `SaveAttestationRecordsForValidators`: Remove unused code.
* `LastEpochWrittenForValidators`: Name variables consistently. - NFC
Avoid mixes between `indice(s)`and `index(es)`.
* `SaveLastEpochsWrittenForValidators`: Name variables consistently. - NFC
* `CheckAttesterDoubleVotes`: Rename variables and add comments. - NFC
* `schema.go`: Add comments. - NFC
* `processQueuedAttestations`: Add comments. - NFC
* `checkDoubleVotes`: Rename variable. - NFC
* `Test_processQueuedAttestations`: Ensure there is no error log.
* `shouldNotBeSlashable` => `shouldBeSlashable`
* `Test_processQueuedAttestations`: Add 2 test cases:
- Same target with different signing roots
- Same target with same signing roots
* `checkDoubleVotesOnDisk` ==> `checkDoubleVotes`.
Before this commit, `checkDoubleVotes` did two tasks:
- Checking if there are any slashable double votes in the input
list of attestations with respect to each other.
- Checking if there are any slashable double votes in the input
list of attestations with respect to our database.
However, `checkDoubleVotes` is called only in
`checkSlashableAttestations`.
And `checkSlashableAttestations` is called only in:
- `processQueuedAttestations`, and in
- `IsSlashableAttestation`
Study of case `processQueuedAttestations`:
---------------------------------------------
In `processQueuedAttestations`, `checkSlashableAttestations`
is ALWAYS called after
`Database.SaveAttestationRecordsForValidators`.
It means that, when calling `checkSlashableAttestations`,
`validAtts` are ALREADY stored in the DB.
Each attestation of `validAtts` will be checked twice:
- Against the other attestations of `validAtts` (the portion of
deleted code)
- Against the content of the database.
One of those two checks is redundent.
==> We can remove the check against other attestations in `validAtts`.
Study of case `Database.SaveAttestationRecordsForValidators`:
----------------------------------------------------------------
In `Database.SaveAttestationRecordsForValidators`,
`checkSlashableAttestations` is ALWAYS called with a list of
attestations containing only ONE attestation.
This only attestaion will be checked twice:
- Against itself, and an attestation cannot conflict with itself.
- Against the content of the database.
==> We can remove the check against other attestations in `validAtts`.
=========================
In both cases, we showed that we can remove the check of attestation
against the content of `validAtts`, and the corresponding test
`Test_checkDoubleVotes_SlashableInputAttestations`.
* `Test_processQueuedBlocks_DetectsDoubleProposals`: Wrap proposals.
So we can add new proposals later.
* Fix slasher multiple proposals false negative.
If a first batch of blocks is sent with:
- validator 1 - slot 4 - signing root 1
- validator 1 - slot 5 - signing root 1
Then, if a second batch of blocks is sent with:
- validator 1 - slot 4 - signing root 2
Because we have two blocks proposed by the same validator (1) and for
the same slot (4), but with two different signing roots (1 and 2), the
validator 1 should be slashed.
This is not the case before this commit.
A new test case has been added as well to check this.
Fixes#13551
* `params.go`: Change comments. - NFC
* `CheckSlashable`: Keep the happy path without indentation.
* `detectAllAttesterSlashings` => `checkSurrounds`.
* Update beacon-chain/db/slasherkv/slasher.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* Update beacon-chain/db/slasherkv/slasher.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* `CheckAttesterDoubleVotes`: Keep happy path without indentation.
Well, even if, in our case, "happy path" mean slashing.
* 'SaveAttestationRecordsForValidators': Save the first attestation.
In case of multiple votes, arbitrarily save the first attestation.
Saving the first one in particular has no functional impact,
since in any case all attestations will be tested against
the content of the database. So all but the first one will be
detected as slashable.
However, saving the first one and not an other one let us not
to modify the end to end tests, since they expect the first one
to be saved in the database.
* Rename `min` => `minimum`.
Not to conflict with the new `min` built-in function.
* `couldNotSaveSlashableAtt` ==> `couldNotCheckSlashableAtt`
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* backfill service
* fix bug where origin state is never unlocked
* support mvslice states
* use renamed interface
* refactor db code to skip block cache for backfill
* lint
* add test for verifier.verify
* enable service in service init test
* cancellation cleanup
* adding nil checks to configset juggling
* assume blocks are available by default
As long as we're sure the AvailableBlocker is initialized correctly
during node startup, defaulting to assuming we aren't in a checkpoint
sync simplifies things greatly for tests.
* block saving path refactor and bugfix
* fix fillback test
* fix BackfillStatus init tests
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
@@ -30,9 +30,6 @@ You can use `bazel run //tools/genesis-state-gen` to create a deterministic gene
- **--output-ssz** string: Output filename of the SSZ marshaling of the generated genesis state
- **--config-name=interop** string: name of the beacon chain config to use when generating the state. ex mainnet|minimal|interop
**deprecated flag: use --config-name instead**
- **--mainnet-config** bool: Select whether genesis state should be generated with mainnet or minimal (default) params
The example below creates 64 validator keys, instantiates a genesis state with those 64 validators and with genesis unix timestamp 1567542540,
and finally writes a ssz encoded output to ~/Desktop/genesis.ssz. This file can be used to kickstart the beacon chain in the next section. When using the `--interop-*` flags, the beacon node will assume the `interop` config should be used, unless a different config is specified on the command line.
returnð.GenericSignedBeaconBlock{Block:ð.GenericSignedBeaconBlock_BlindedBellatrix{BlindedBellatrix:block},IsBlinded:true,PayloadValue:0/* can't get payload value from blinded block */},nil
returnð.GenericBeaconBlock{Block:ð.GenericBeaconBlock_BlindedBellatrix{BlindedBellatrix:block},IsBlinded:true,PayloadValue:0/* can't get payload value from blinded block */},nil
returnð.GenericSignedBeaconBlock{Block:ð.GenericSignedBeaconBlock_BlindedCapella{BlindedCapella:block},IsBlinded:true,PayloadValue:0/* can't get payload value from blinded block */},nil
returnð.GenericBeaconBlock{Block:ð.GenericBeaconBlock_BlindedCapella{BlindedCapella:block},IsBlinded:true,PayloadValue:0/* can't get payload value from blinded block */},nil
// calculateChunkSize calculates a suitable chunk size for the purposes of parallelisation.
// calculateChunkSize calculates a suitable chunk size for the purposes of parallelization.
funccalculateChunkSize(itemsint)int{
// Start with a simple even split
chunkSize:=items/runtime.GOMAXPROCS(0)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
