* Update ckzg4844 to latest version
* Run go mod tidy
* Remove unnecessary tests & run goimports
* Remove fieldparams from blockchain/kzg
* Add back blank line
* Avoid large copies
* Run gazelle
* Use trusted setup from the specs & fix issue with struct
* Run goimports
* Fix mistake in makeCellsAndProofs
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* use recoverCellsAndKZGProofs
* make recoverAllCells and CellsToBlob private
* chore: all methods now return CellsAndProof struct
* chore: update code
* chore: move all ckzg related functionality into kzg package
* refactor code to match
* run: bazel run //:gazelle -- fix
* chore: add some docs and stop copying large objects when converting between types
* fixes
* manually add kzg.go dep to Build.Hazel
* move kzg methods to kzg.go
* chore: add RecoverCellsAndProofs method
* bazel run //:gazelle -- fix
* make Cells be flattened sequence of bytes
* chore: add test for flattening roundtrip
* chore: remove code that was doing the flattening outside of the kzg package
* fix merge
* fix
* remove now un-needed conversion
* use pointers for Cell parameters
* linter
* rename cell conversion methods (this only applies to old version of c-kzg)
* change recoverBlobs to recoverCellsAndProofs
* modify code to take in the cells and proofs for a particular blob instead of the blob itself
* add CellsAndProofs structure
* modify recoverCellsAndProofs to return `cellsAndProofs` structure
* modify `DataColumnSidecarsForReconstruct` to accept the `cellsAndKZGProofs` structure
* bazel run //:gazelle -- fix
* use kzg abstraction for kzg method
* move CellsAndProofs to kzg.go
* chore: move all ckzg related functionality into kzg package
* refactor code to match
* run: bazel run //:gazelle -- fix
* chore: add some docs and stop copying large objects when converting between types
* fixes
* manually add kzg.go dep to Build.Hazel
* move kzg methods to kzg.go
* chore: add RecoverCellsAndProofs method
* bazel run //:gazelle -- fix
* use BytesPerBlob constant
* chore: fix some deepsource issues
* one declaration for commans and blobs
* Wrap errors, add logs.
* `missingColumnRequest`: Fix blobs <-> data columns mix.
* `ColumnIndices`: Return `map[uint64]bool` instead of `[fieldparams.NumberOfColumns]bool`.
* `DataColumnSidecars`: `interfaces.SignedBeaconBlock` ==> `interfaces.ReadOnlySignedBeaconBlock`.
We don't need any of the non read-only methods.
* Fix comments.
* `handleUnblidedBlock` ==> `handleUnblindedBlock`.
* `SaveDataColumn`: Move log from debug to trace.
If we attempt to save an already existing data column sidecar,
a debug log was printed.
This case could be quite common now with the data column reconstruction enabled.
* `sampling_data_columns.go` --> `data_columns_sampling.go`.
* Reconstruct data columns.
* Support Data Columns For By Root Requests
* Revert Config Changes
* Fix Panic
* Fix Process Block
* Fix Flags
* Lint
* Support Checkpoint Sync
* Manu's Review
* Add Support For Columns in Remaining Methods
* Unmarshal Uncorrectly
* Add Data Structure for New Request Type
* Add Data Column By Range Handler
* Add Data Column Request Methods
* Add new validation for columns by range requests
* Fix Build
* Allow Prysm Node To Fetch Data Columns
* Allow Prysm Node To Fetch Data Columns And Sync
* Bug Fixes For Interop
* GoFmt
* Use different var
* Manu's Review
* Enable E2E And Add Fixes
* Register Same Topic For Data Columns
* Initialize Capacity Of Slice
* Fix Initialization of Data Column Receiver
* Remove Mix In From Merkle Proof
* E2E: Subscribe to all subnets.
* Remove Index Check
* Remaining Bug Fixes to Get It Working
* Change Evaluator to Allow Test to Finish
* Fix Build
* Add Data Column Verification
* Fix LoopVar Bug
* Do Not Allocate Memory
* Update beacon-chain/blockchain/process_block.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/core/peerdas/helpers.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/core/peerdas/helpers.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Gofmt
* Fix It Again
* Fix Test Setup
* Fix Build
* Fix Trusted Setup panic
* Fix Trusted Setup panic
* Use New Test
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Add new DA check
* Exit early in the event no commitments exist.
* Gazelle
* Fix Mock Broadcaster
* Fix Test Setup
* Update beacon-chain/blockchain/process_block.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Manu's Review
* Fix Build
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Remove capital letter from error messages.
* `[4]byte` => `[fieldparams.VersionLength]byte`.
* Prometheus: Remove extra `committee`.
They are probably due to a bad copy/paste.
Note: The name of the probe itself is remaining,
to ensure backward compatibility.
* Implement Proposer RPC for data columns.
* Fix TestProposer_ProposeBlock_OK test.
* Remove default peerDAS activation.
* `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
* Bump `c-kzg-4844` lib to the `das` branch.
* Implement `MerkleProofKZGCommitments`.
* Implement `das-core.md`.
* Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`.
* `CustodyColumnSubnets`: Include `i` in the for loop.
* Remove `computeSubscribedColumnSubnet`.
* Remove `peerdas.CustodyColumns` out of the for loop.
* `PruneProposalsAtEpoch`: Test return value in case of nothing to prune.
* `TestStore_PruneAttestations_OK`: Create unique validator indexes.
Before this commit, `attester2` for `j = n` was the same than
`attester1` for `j = n + 1`, resulting in erasure of a lot of attesters.
I guess it was not the initial intent.
* Slasher pruning: Check if the number of pruned items corresponds to the expectation.
Before this commit, if the pruning function did remove a superset
of the expected pruned items (including all the items), then the test would pass.
* Prune items that should be pruned and stop pruning items that should not be pruned.
The first 8 bytes of the key of `attestation-data-roots` and
`proposal-records` bytes correspond respectively to an encoded epoch and
and encoded slot.
The important word in this sentence is "encoded".
Before this commit, these slot/epoch are SSZ encoded, which means that
they are little-endian encoded.
However:
- `uint64PrefixGreaterThan` uses `bytes.Compare` which expects
big-endian encoded values.
- `for k, _ := c.First(); k != nil; k, _ = c.Next()` iters over the
keys in big-endian order.
The consequence is:
- Some items that should be pruned are not pruned, provoking a disk
usage leak.
- Some items that should not be pruned are pruned, provoking errors like
in https://github.com/prysmaticlabs/prysm/issues/13658.
This commit encodes the slot/epoch as big-endian before storing them in
the database keys.
Why this bug has not been detected in unit test before?
The values used in unit tests before this commit in
`TestStore_PruneProposalsAtEpoch` and `TestStore_PruneAttestations_OK`
are `10` and `20`.
Unfortunately, checking if `littleEndian(20) > littlenEndien(10)`
with the `>` operator considering operands as big-endian encoded returns
the expected result...
Just replacing `20` by `30` trigs the bug.
* Make deepsource happy.
* Slasher: Migrate database from little-endian to big-endian.
* Update beacon-chain/slasher/service.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/db/slasherkv/migrate.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `TestMigrate`: Fix documentation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Bump go version up
* Update to go 1.22 compatible version
* Fix NoSec declarations
* Skip Gosec in GolangCi
* Avoid Bug In Analyzer
* Add in Gohashtree and Update to 1.22.4
* Fix Go Sum
* validator/client: process Sync Committee roles separately
In a DV context, to be compatible with the proposed selection endpoint, the VC must push all partial selections to it instead of just one.
Process sync committee role separately within the RolesAt method, so that partial selections can be pushed to the DV client appropriately, if configured.
* testing/util: re-add erroneously deleted state_test.go
* validator/client: fix tests
* validator/client: always process sync committee validator
Process sync committee duty at slot boundary as well.
* don't fail if validator is marked as sync committee but it is not in the list
ignore the duty and continue
* address code review comments
* fix build
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* eip-7251: registry_updates
* Fix spectest violation
* More unit tests for process_registry_updates
* Update tests for ProcessRegistryUpdates
* Minor refactorings and benchmark
* Update beacon-chain/core/electra/registry_updates.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* Wrap errors from process_registry_updates for easier debugging
* Rewrite process_registry_updates not to use st.Validators()
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* wip changes
* wip changes refactoring deposit functions
* wip on deposit functions
* wip changes to fix deposit processing
* more wip function logic
* gaz
* wip refactoring deposit changes
* removing circlular dependency and other small fix
* fixing circular dependencies
* fixing validators file
* fixing more tests
* fixing unit tests
* wip deposit packing
* refactoring packing
* changing packing code some more
* fixing packing change
* updating more tests
* removing comment
* fixing transition code for eip6110
* including inserts for validator index cache
* adding tests and placeholder test
* moving deposit related unit tests over
* adding in test for electra proposer packing
* spec test wip
* moving cache saving to the correct spot
* eip-6110: deposit spectests
* adding deposit receipt spec tests
* reverting altair operations in this pr and will be handled separately
* fixing renames but need to refactor process deposits
* gaz
* fixing names
* fixing linting
* fixing unit test
* fixing a test and updating test util
* bal never used
* fixing more tests
* fixing one more test
* addressing feedback
* refactoring process deposits to be part of their own fork instead of in blocks package
* adding new tests to cover functions and removing redundancies
* removing comment based on feedback
* resolving easy deepsource issue
* refactoring for appropriate aliasing and readability
* reverting some changes to simplify diff
* small edit to comment
* fixing linting
* fixing merge changes and review comments
* Update beacon-chain/rpc/prysm/v1alpha1/validator/proposer_deposits.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* partial review comments
* addressing slack feedback
* moving function from deposits to validator.go
* adding in batch verification and improving logs
---------
Co-authored-by: Preston Van Loon <preston@pvl.dev>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `IndexedAtt` wrapper for the slasher feed
* test fixes
* fix simulator
* fix e2e
* Revert "Auxiliary commit to revert individual files from 191bbf77accfc2523fa9f909837a2e9dca132afa"
This reverts commit 2b0441a23a0e5f66e50cf36c3bbfbb39d587b17b.
* extract interface from channel
* adding logic for saving validator indicies in new cache
* updating index cache for eip6110
* Update setters_misc.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* linting
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* renaming functions and fields based on consensus changes
* execution api rename
* fixing test
* reverting spectests changes, it should be changed with new version
* reverting temporarily
* revert exclusions
* Increase health tracker channel buffer size
* keep only the latest value
* Make health test blocking as a regression test for PR #14807
* Fix new race conditions in the MockHealthClient
---------
Co-authored-by: Preston Van Loon <preston@pvl.dev>
* always close cache warm chan to prevent blocking
* test that waitForCache does not block
* combine defers to reduce cognitive overhead
* lint
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* eip-7251: process_effective_balance_updates
Spectests for process_effective_balance_updates
process_effective_balance_updates unit tests
* PR feedback from the amazing @rkapka
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Initial commit
* Fuzz all generated ssz en/decoders.
* Make proto message comparison aware of nested proto messages and differing unknown fields therein.
* Don't name receiver variable if not necessary.
* Run gazelle
* Cleanup part 1
* Cleanup part 2
* Cleanup part 3
* remove lock field init
* doc for SignerConfig
* remove vars
* use full Keymanager word in function
* revert interface rename
* linter
* fix build issues
* review
* eip-7251: process_pending_balance_deposits
* Update beacon-chain/core/electra/balance_deposits_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Remove defensive check. A unit test shows nothing bad happens
* Safe sub to protect from underflow
* Use @kasey's idea for safer subtraction
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* electra engine api support
* add marshaling support for ExecutionPayloadElectra
* add receipts to json tests
* deep source
* simplify slice handling
* deep source lint about type/method order
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* wip fork logic upgrade
* fixing replay and fork.go
* improving process function and adding tests for transition
* updating unit tests and temporarily removing some fields on state_trie.go
* updating state
* wip adding upgrade to electra code
* adding some comments
* adding spec tests
* fixing values used in state transition logic
* updating upgrade test
* gofmt
* avoid dup word linting
* fixing spec tests for fork
* gaz
* fixing tests
* improving unit test with new getters
* fixing bazel for minimal fork test
* adding bazel file
* Update beacon-chain/core/electra/upgrade.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* addressing some comments and adding more tests
* addressing more feedback
* one more feedback
* changing value to interface after talking to preston
* adding missed review feedback
* fixing linting
* noticed I was using the wrong function in the state upgrade
* fixing and ignoring some deepsource issues
* moving core electra validator functions to helper to remove circular dependencies in other PRs
* missed deepsource complaint
* Update upgrade.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update testing/util/electra_state.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update testing/util/electra_state.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* addressing feedback
* removing deepsoure ignore comments
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Electra: Beacon State
* Electra: Beacon state fixes from PR 13919
* Add missing tests - part 1
* Split eip_7251_root.go into different files and reuse/share code with historical state summaries root. It's identical!
* Add missing tests - part 2
* deposit receipts start index getters and setters (#13947)
* adding in getters and setters for deposit receipts start index
* adding tests
* gaz
* Add missing tests - part 3 of 3
Update the electra withdrawal example with a ssz state containing pending partial withdrawals
* add tests for beacon-chain/state/state-native/getters_balance_deposits.go
* Add electra field to testing/util/block.go execution payload
* godoc commentary on public methods
* Fix failing test
* Add balances index out of bounds check and relevant tests.
* Revert switch case electra
* Instead of copying spectest data into testdata, use the spectest dependency
* Deepsource fixes
* Address @rkapka PR feedback
* s/MaxPendingPartialsPerWithdrawalSweep/MaxPendingPartialsPerWithdrawalsSweep/
* Use multivalue slice compatible accessors for validator and balance in ActiveBalanceAtIndex
* More @rkapka feedback. What a great reviewer!
* More tests for branching logic in ExitEpochAndUpdateChurn
* fix build
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Electra: full beacon config
Fix TestGetSpec
* Fix beacon config spec compliance test so that it properly loads the config from spec tests. Tests failing for now.
* fix tests and comply with spec presets
* block protos
* proto and ssz
* stubs
* Enable Electra spec test
* Pull in EIP-7251 protobuf changes
From: https://github.com/prysmaticlabs/prysm/pull/13903
* All EIP7549 containers are passing
* All EIP7251 containers passing
* including changes from eip7002
* Everything passing except for beacon state hash tree root
* fixing eletra state to use electra payload
* Fix minimal test. Skip beacon state test
* Perston's feedback
---------
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>
* Remove EnableEIP4881 flag
* Gaz
* Fix missing error handler
* Remove old tree and fix tests
* Gaz
* Fix build import
* Replace depositcache
* Add pendingDeposit tests
* Nishant's fix
* Fix unsafe uint64 to int
* Fix other unsafe uint64 to int
* Remove: RemovePendingDeposit
* Deprecate and remove DisableEIP4881 flag
* Check: index not greater than deposit count
* Move index check
* GET
* POST
* Revert "Auxiliary commit to revert individual files from 615feb104004d6a945ededf5862ae38325fc7ec2"
This reverts commit 55cf071c684019f3d6124179154c10b2277fda49.
* comment fix
* deepsource
* Beacon-api: broadcast blobs in the event of seen block
* Fix parameters
* Fix test
* Check forkchoice
* Ran go format
* Revert "Ran go format"
This reverts commit 091e77e81d6e2b9861fecc27c0bad1898033f9a3.
* James feedback
* Radek's feedback
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Fix bad tests
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* allow other pkgs to check for blobs in pruning cache
* address deepsource complaints
* custom error to simplify test setup
* add AllAvailable method
* make storage summary slot field private
* unit test and off-by-one fix
* remove comment with copy of tested function
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Revert "Modify the algorithm of `updateFinalizedBlockRoots` (#13486)"
This reverts commit 32fb183392.
* migration to fix index corruption from pr 13486
* bail as soon as we see 10 epochs without the bug
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Remove check for duplicates in pending attestation queue
The current queue will only save 1 unaggregated attestation for a pending block because we wrap the object into a SignedAggregatedAttestationAndProof with a zeroed aggregator.
* fix tests
* add max/min span visualisation tool cli
* go mod tidy
* lint imports
* remove typo
* fix epoch table value
* fix deepsource
* add dep to bazel
* fix dep import order
* change command name from span to slasher-span-display
* change command args style using - instead of _
* sed s/CONFIGURATION/SLASHER PARAMS//
* change double neg to double pos condition
* remove unused anonymous func
* better function naming
* add range condition
* [deepsource] Fix Empty slice literal used to declare a variable
GO-W1027
* correct typo
* do not show incorrect epochs due to round robin
* fix import
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
ValidateClock in ValidateAttestationTime is useless
The check is that the attSlot is not > than the currentslot + 128 slots.
Later there's a check that the attSlot start time is not > than current slot
start time + clockDisparity.
if attSlot > than currentSlot + 128 slots, then the second check would fail
anyway.
The lattest check already guarantees that the attSlot cannot be larger than the
currentSlot, therefore it may never happen that attEpoch > currentEpoch. We just
need to check for Deneb that attEpoch >= currentEpoch - 1.
Removes also some duplicated variables like the attestation epoch being computed
twice.
* refactor batch verifier to share with pending queue
* unit test for batch verifier
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Remove Feature Flag From Prater (#12082)
* Use Epoch boundary cache to retrieve balances (#12083)
* Use Epoch boundary cache to retrieve balances
* save boundary states before inserting to forkchoice
* move up last block save
* remove boundary checks on balances
* fix ordering
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* create the log file along with its parent directory if not present
* only give ReadWritePermissions to the log file
* use io/file package to create the parent directories
* fix ci related issues
* add regression tests
* run gazelle
* fix tests
* remove print statements
* gazelle
* Remove failing test for MkdirAll, this failure is not expected
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* add log message if in da check at slot end
* don't bother logging late da check start
* break up defer with a var, too dense all together
* pass slot instead of block ref
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* addressing errors on events endpoint
* reverting timeout on get health
* fixing linting
* fixing more linting
* Update validator/client/beacon-api/beacon_api_validator_client.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/rpc/eth/events/events.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* reverting change and removing line on context done which creates a superfluous response.WriteHeader error
* gofmt
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* make it very big
* use new pruning implementation
* handle pre deneb
* revert cache change
* less verbose
* gaz
* Update beacon-chain/operations/attestations/prune_expired.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* gofmt
* be safer
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* handle special case of batch size=1
* unit test case for backfill batch len=1
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* flag to set an older backfill slot target
* wire up flag to main cli and usage
* fix deepsource complaints
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `subscribeStaticWithSubnets`: Fix docstring.
* `buildOptions`: Avoid `options` mutations.
* `dv5Cfg`: Avoid mutation.
* `RefreshENR`: Use default for all but Phase0.
* `udp4`, `udp6`: Create enum.
* `p2p.Config`: `BootstrapNodeAddr`==> `BootstrapNodeAddrs`.
* `p2p.Config`: `Discv5BootStrapAddr` ==> `Discv5BootStrapAddrs`.
* `TestScorers_BadResponses_Score`: Improve.
* `BeaconNode`: Avoid mutation.
* `TestStore_TrustedPeers`: Remove blankline.
* Remove blank identifiers.
* `privKey`: Keep the majority of code with low indentation.
* `P2PPreregistration`: Return error instead of fatal log.
* `parseBootStrapAddrs` => `ParseBootStrapAddrs` (export)
* `p2p.Config`: Remove `BootstrapNodeAddrs`.
* `NewService`: Avoid mutation when possible.
* `Service`: Remove blank identifier.
* `buildOptions`: Avoid `log.Fatalf` (make deepsource happy).
* `registerGRPCGateway`: Use `net.JoinHostPort` (make deepsource happy).
* `registerBuilderService`: Make deepsource happy.
* `scorers`: Add `NoLock` suffix (make deepsource happy).
* `scorerr`: Add some `NoLock`suffixes (making deepsource happy).
* `discovery_test.go`. Remove init.
Rationale:
`rand.Seed` is deprecated: As of Go 1.20 there is no reason to call Seed with a random value. Programs that call Seed with a known value to get a specific sequence of results should use New(NewSource(seed)) to obtain a local random generator.
This makes deepsource happy as well.
* `createListener`: Reduce cyclomatic complexity (make deepsource happy).
* `startDB`: Reduce cyclomatic complexity (make deepsource happy).
* `main`: Log a FATAL on error.
This way, the error message is very readable.
Before this commit, the error message is the less readable
message in the logs.
* `New`: Reduce cyclomatic complexity (make deepsource happy).
* `main`: Avoid `App` mutation, and make deepsource happy.
* Update beacon-chain/node/node.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* `bootnodes` ==> `BootNodes` (Fix PR comment).
* Remove duplicate `configureFastSSZHashingAlgorithm` since already done in `configureBeacon`. (Fix PR comment)
* Add `TestCreateLocalNode`. (PR comment fix.)
* `startModules` ==> `startBaseServices (Fix PR comment).
* `buildOptions` return errors consistently.
* `New`: Change ordering.
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* use real blob verifier in forkchoice spectest
* wip
* Use real blob sidecar for test
* Set file db correctly
* correctly handle blob cases where valid=false
* work-around spectest's weird Fork in genesis state
* gaz
* revert T-money's log level change
* rm whitespace
* unskip minimal test
* Preston's feedback
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* add them
* Ensure activation epoch does not overflow
* add them all in
* better check for overflow
* fix tests
* fix tests
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* `EpochFromString`: Use already defined `Uint64FromString` function.
* `Test_uint64FromString` => `Test_FromString`
This test function tests more functions than `Uint64FromString`.
* Slashing protection history: Remove unreachable code.
The function `NewKVStore` creates, via `kv.UpdatePublicKeysBuckets`,
a new item in the `proposal-history-bucket-interchange`.
IMO there is no real reason to prefer `proposal` than `attestation`
as a prefix for this bucket, but this is the way it is done right now
and renaming the bucket will probably be backward incompatible.
An `attestedPublicKey` cannot exist without
the corresponding `proposedPublicKey`.
Thus, the `else` portion of code removed in this commit is not reachable.
We raise an error if we get there.
This is also probably the reason why the removed `else` portion was not
tested.
* `NewKVStore`: Switch items in `createBuckets`.
So the order corresponds to `schema.go`
* `slashableAttestationCheck`: Fix comments and logs.
* `ValidatorClient.db`: Use `iface.ValidatorDB`.
* BoltDB database: Implement `GraffitiFileHash`.
* Filesystem database: Creates `db.go`.
This file defines the following structs:
- `Store`
- `Graffiti`
- `Configuration`
- `ValidatorSlashingProtection`
This files implements the following public functions:
- `NewStore`
- `Close`
- `Backup`
- `DatabasePath`
- `ClearDB`
- `UpdatePublicKeysBuckets`
This files implements the following private functions:
- `slashingProtectionDirPath`
- `configurationFilePath`
- `configuration`
- `saveConfiguration`
- `validatorSlashingProtection`
- `saveValidatorSlashingProtection`
- `publicKeys`
* Filesystem database: Creates `genesis.go`.
This file defines the following public functions:
- `GenesisValidatorsRoot`
- `SaveGenesisValidatorsRoot`
* Filesystem database: Creates `graffiti.go`.
This file defines the following public functions:
- `SaveGraffitiOrderedIndex`
- `GraffitiOrderedIndex`
* Filesystem database: Creates `migration.go`.
This file defines the following public functions:
- `RunUpMigrations`
- `RunDownMigrations`
* Filesystem database: Creates proposer_settings.go.
This file defines the following public functions:
- `ProposerSettings`
- `ProposerSettingsExists`
- `SaveProposerSettings`
* Filesystem database: Creates `attester_protection.go`.
This file defines the following public functions:
- `EIPImportBlacklistedPublicKeys`
- `SaveEIPImportBlacklistedPublicKeys`
- `SigningRootAtTargetEpoch`
- `LowestSignedTargetEpoch`
- `LowestSignedSourceEpoch`
- `AttestedPublicKeys`
- `CheckSlashableAttestation`
- `SaveAttestationForPubKey`
- `SaveAttestationsForPubKey`
- `AttestationHistoryForPubKey`
* Filesystem database: Creates `proposer_protection.go`.
This file defines the following public functions:
- `HighestSignedProposal`
- `LowestSignedProposal`
- `ProposalHistoryForPubKey`
- `ProposalHistoryForSlot`
- `ProposedPublicKeys`
* Ensure that the filesystem store implements the `ValidatorDB` interface.
* `slashableAttestationCheck`: Check the database type.
* `slashableProposalCheck`: Check the database type.
* `slashableAttestationCheck`: Allow usage of minimal slashing protection.
* `slashableProposalCheck`: Allow usage of minimal slashing protection.
* `ImportStandardProtectionJSON`: Check the database type.
* `ImportStandardProtectionJSON`: Allow usage of min slashing protection.
* Implement `RecursiveDirFind`.
* Implement minimal<->complete DB conversion.
3 public functions are implemented:
- `IsCompleteDatabaseExisting`
- `IsMinimalDatabaseExisting`
- `ConvertDatabase`
* `setupDB`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/node/node_test.go`.
The rest of this commit consists in minimal slashing protection testing.
* `setupWithKey`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/client/propose_test.go`.
The rest of this commit consists in tests wrapping.
* `setup`: Add `isSlashingProtectionMinimal` argument.
The added feature is located in the `validator/client/propose_test.go`
file.
The rest of this commit consists in tests wrapping.
* `initializeFromCLI` and `initializeForWeb`: Factorize db init.
* Add `convert-complete-to-minimal` command.
* Creates `--enable-minimal-slashing-protection` flag.
* `importSlashingProtectionJSON`: Check database type.
* `exportSlashingProtectionJSON`: Check database type.
* `TestClearDB`: Test with minimal slashing protection.
* KeyManager: Test with minimal slashing protection.
* RPC: KeyManager: Test with minimal slashing protection.
* `convert-complete-to-minimal`: Change option names.
Options were:
- `--source` (for source data directory), and
- `--target` (for target data directory)
However, since this command deals with slashing protection, which has
source (epochs) and target (epochs), the initial option names may confuse
the user.
In this commit:
`--source` ==> `--source-data-dir`
`--target` ==> `--target-data-dir`
* Set `SlashableAttestationCheck` as an iface method.
And delete `CheckSlashableAttestation` from iface.
* Move helpers functions in a more general directory.
No functional change.
* Extract common structs out of `kv`.
==> `filesystem` does not depend anymore on `kv`.
==> `iface` does not depend anymore on `kv`.
==> `slashing-protection` does not depend anymore on `kv`.
* Move `ValidateMetadata` in `validator/helpers`.
* `ValidateMetadata`: Test with mock.
This way, we can:
- Avoid any circular import for tests.
- Implement once for all `iface.ValidatorDB` implementations
the `ValidateMetadata`function.
- Have tests (and coverage) of `ValidateMetadata`in
its own package.
The ideal solution would have been to implement `ValidateMetadata` as
a method with the `iface.ValidatorDB`receiver.
Unfortunately, golang does not allow that.
* `iface.ValidatorDB`: Implement ImportStandardProtectionJSON.
The whole purpose of this commit is to avoid the `switch validatorDB.(type)`
in `ImportStandardProtectionJSON`.
* `iface.ValidatorDB`: Implement `SlashableProposalCheck`.
* Remove now useless `slashableProposalCheck`.
* Delete useless `ImportStandardProtectionJSON`.
* `file.Exists`: Detect directories and return an error.
Before, `Exists` was only able to detect if a file exists.
Now, this function takes an extra `File` or `Directory` argument.
It detects either if a file or a directory exists.
Before, if an error was returned by `os.Stat`, the the file was
considered as non existing.
Now, it is treated as a real error.
* Replace `os.Stat` by `file.Exists`.
* Remove `Is{Complete,Minimal}DatabaseExisting`.
* `publicKeys`: Add log if unexpected file found.
* Move `{Source,Target}DataDirFlag`in `db.go`.
* `failedAttLocalProtectionErr`: `var`==> `const`
* `signingRoot`: `32`==> `fieldparams.RootLength`.
* `validatorClientData`==> `validator-client-data`.
To be consistent with `slashing-protection`.
* Add progress bars for `import` and `convert`.
* `parseBlocksForUniquePublicKeys`: Move in `db/kv`.
* helpers: Remove unused `initializeProgressBar` function.
When a late block arrives and the beacon is proposing the next block, we
perform several checks to allow for the next block to reorg the incoming
late block.
Among those checks, we check that the parent block has been heavily
attested (currently 160% of the committee size).
We perform this check in these circumstances:
- When the late block arrives
- At 10 seconds into the slot
- At 0 seconds into the next slot (at proposing time)
The problem is that for blocks that arrive between 4 seconds and 10
seconds, the parent block will not have yet this expected weight since
attestations from the current committee were not imported yet, and thus
Prysm will send an FCU with payload attributes anyway at this time.
What happens is that Prysm keeps the EL building different blocks based
on different parents at the same time, when later in the next slot it
calls to propose, it will reorg the late block anyway and the EL would
have been computing a second payload uselessly.
This PR enables this check only when calling `ShouldOverrideFCU` after
10 seconds into the slot which we do only after having imported the
current attestations. We may want to actually remove this check entirely
from `ShouldOverrideFCU` and only keep it in `ProposerHead`.
Shout out to Anthithesis for reporting an issue that led to this
discoverly.
* employ a val reader to prevent constant copies
* clean it up and fix tests
* gaz
* radek's review
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Improve package documentation.
* `processAttestations`: Improve logging.
* Add `Benchmark_checkSurroundVotes` benchmark.
* Implement `saveChunksToDisk` as remplacement of `saveUpdatedChunks`.
The idea is to open only on DB transaction for all validator chunk indexes instead of
one DB transaction per validator chunk index.
It saves the overhead due to transaction start/stop of the DB.
Result of `Benchmark_checkSurroundVotes`:
- Before this commit: 133 seconds
- After this commit: 5.05 seconds
* `LoadSlasherChunks` and `SaveSlasherChunks`: Batch.
* `loadChunks` ==> `loadChunksFromDisk`
* `updatedChunkByChunkIndex`: Don't update if `latestEpochWritten == currentEpoch `.
* `updatedChunkByChunkIndex`: Load all needed chunks once.
* `latestEpochWritten` ==> `latestEpochUpdated`.
* `checkSurroundVotes`: Dump to disk at most every `25_600` chunks.
* `SaveAttestationRecordsForValidators`: Batch.
* `batchSize`: Use as package const and add comment.
* pending broadcast err if missing blobs and 0 peers
* compute request first for len check
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `getChunk` ==> `getChunkFromDatabase`.
* `loadChunks`: Rename variables.
* `Update`: Use explicit arguments.
* `detect_attestations.go`: Reduce abstraction layers.
* `loadAndUpdateChunks`: Change arguments order.
* `updatedChunkByChunkIndex`: Update all known validators in the chunk.
* `LastEpochWrittenForValidators`: Avoid avoidable `for`loop.
* `chunks.go`: Ensure implementations respect the interface.
* `LastEpochWrittenForValidators`: Stop considering lack of epoch as genesis epoch.
* `updatedChunkByChunkIndex`: Don't update latest updated epoch.
And add a bunch of tests.
* Improve slasher cold boot duration.
Before this commit, on a slasher cold boot (aka, without any db),
the `updatedChunkByChunkIndex` function looped for all validators
AND for all epochs between the genesis epoch and the current epoch.
This could take several dozen of minutes, and it is useless since the
min/max spans are actually a circular buffer with a limited lenght.
Cells of min/max spans can be overwritten (with the same value)
plenty of times.
After this commit, the `updatedChunkByChunkIndex` function loops
for all validators AND AT most 'historyLength' lenght.
Every cell of min/max spans are written AT MOST once.
Time needed for slasher boot goes from `O(nm)` to "only" `O(m)`, where:
- `n` is the number of epochs since the genesis.
- `m` is the number of validators.
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* retrieve and save blobs during backfill
* Update beacon-chain/sync/backfill/batch.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/sync/backfill/blobs.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/sync/backfill/metrics.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* make blobSync initialization path a little safer
* use bytes.Equal to avoid extra allocation
* stop using log.Fatal and other little cleanups
* wrap up post blob sync actions in batch mutator
* unit test coverage on verifier
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `LastEpochWrittenForValidators`: Use golang if.
* `SaveLastEpochsWrittenForValidators`: Refactor.
* `SaveLastEpochsWrittenForValidators`: Fix when `epochByValIndex` > `batchSize`.
Before this commit, `TestStore_LastEpochWrittenForValidators` works if `validatorsCount <= 10000`
and stops working if `validatorsCount > 10000`.
* Slasher: Detect surrounded votes in multiple batches.
Fixes https://github.com/prysmaticlabs/prysm/issues/13591.
* `Test_processAttestations`: Remove duplicated tests.
* Sort indexed attestations by data root.
* `processAttestations`: Don't return duplicate slashings anymore.
Fix https://github.com/prysmaticlabs/prysm/issues/13592.
* `AttesterDoubleVote`: Rename fields.
* Detect double votes in different batches.
In order to do that:
1. Each attestation of the batch is tested against the other attestations of the batch.
2. Each attestation of the batch is tested against the content of the database.
2. Attestations are saved into the database.
Fixes https://github.com/prysmaticlabs/prysm/issues/13590.
* `helpers.go`: Improve naming consistency.
* `detect_attestations.go`: Improve readability.
* `receive.go`: Add `attsQueueSize` in log message.
* `checkSlashableAttestations`: Improve logging.
`avgBatchProcessingTime` is not displayed any more if not batch is
processed.
* `loadChunks`: Use explicit `chunkKind` and `chunkIndices`.
* `getChunk`: Use specific `chunkIndex` and `chunkKind`.
* `validatorIndicesInChunk` -> `validatorIndexesInChunk`.
* `epochUpdateForValidator`: Use explicit arguments.
* `getChunk`: Change order of arguments.
* `latestEpochWrittenForValidator`: Use `ok` parameter.
So the default value is not any more considered as the absence of
value.
* `applyAttestationForValidator`: Use explicit arguments.
* `updateSpans`: Use explicit arguments.
* `saveUpdatedChunks`: Use explicit arguments.
* `checkSurrounds`: Use explicit arguments.
We see here that, previously, in `checkSlashableAttestations`,
`checkSurrounds` was called with the default value of `slashertypes`: `MinSpan`.
Now, we set it expliciterly at `MinSpan`, which may explicit a bug.
* `epochUpdateForValidator`: Set modified by the function argument first.
* `applyAttestationForValidator`: Set mutated argument `chunksByChunkIdx`first.
* `applyAttestationForValidator`: Rename variables.
* `Test_processQueuedAttestations`: Fix test.
Two tests were actually exactly the same.
* `updateSpans`: Keep happy path in the outer scope.
Even if in this case the "happy" path means slashing.
* `checkSurrounds`: Rename variable.
* `getChunk`: Avoid side effects.
It adds a few lines for callers, but it does not modify any more
arguments and it does what it says: getting a chunk.
* `CheckSlashable`: Flatten.
* `detect_attestations_test.go`: Simplify.
* `CheckSlashable`: Add error log in case of missing attestation.
* `processQueuedAttestations`: Extract a sub function.
So testing will be easier.
* `processAttesterSlashings` and `processProposerSlashings`: Improve.
* `processAttesterSlashings`: Return processed slashings.
* `createAttestationWrapper`: Rename variables.
* `signingRoot` ==> `headerRoot` or `dataRoot`.
Before this commit, there is two typse of `signing root`s floating around.
- The first one is a real signing root, aka a hash tree root computed from an object root and
a domain. This real signing root is the object ready to be signed.
- The second one is a "false" signing root, which is actually just the hash tree root of an object. This object is either the `Data` field of an attestation, or the `Header` field of a block.
Having 2 differents objects with the same name `signing root` is quite confusing.
This commit renames wrongly named `signing root` objects.
* `createAttestationWrapper` => `createAttestationWrapperEmptySig`.
So it's clear for the user that the created attestation wrapper has an empty signature.
* Implement `createAttestationWrapper`.
* `processAttestations`: Return processed attester slashings.
* Test `processAttestations` instead of `processQueuedAttestations`.
By testing `processAttestations` instead of `processQueuedAttestations`, we get rid of a lot of tests fixtures, including the 200 ms sleep.
The whole testing duration is shorter.
* `Test_processAttestations`: Allow multiple steps.
* `Test_processAttestations`: Add double steps tests.
Some new failing tests are commented with a corresponding github issue.
* `NextChunkStartEpoch`: Fix function comment.
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `chunks.go`: Avoid templating log messages.
* `checkSlashableAttestations`: Simplify duration computation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
@@ -30,9 +30,6 @@ You can use `bazel run //tools/genesis-state-gen` to create a deterministic gene
- **--output-ssz** string: Output filename of the SSZ marshaling of the generated genesis state
- **--config-name=interop** string: name of the beacon chain config to use when generating the state. ex mainnet|minimal|interop
**deprecated flag: use --config-name instead**
- **--mainnet-config** bool: Select whether genesis state should be generated with mainnet or minimal (default) params
The example below creates 64 validator keys, instantiates a genesis state with those 64 validators and with genesis unix timestamp 1567542540,
and finally writes a ssz encoded output to ~/Desktop/genesis.ssz. This file can be used to kickstart the beacon chain in the next section. When using the `--interop-*` flags, the beacon node will assume the `interop` config should be used, unless a different config is specified on the command line.
line:=scanner.Text()// TODO(13730): scanner does not handle /r and does not fully adhere to https://html.spec.whatwg.org/multipage/server-sent-events.html#the-eventsource-interface
// Handle the event based on your specific format
ifline==""{
// Empty line indicates the end of an event
ifeventType!=""&&data!=""{
// Process the event when both eventType and data are set
log.WithError(err).Error("could not update forkchoice with payload attributes for proposal")
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.