* `ColumnAlignsWithBlock`: Split lines.
* Data columns verifications: Batch
* Remove completely `DataColumnBatchVerifier`.
Only `DataColumnsVerifier` (with `s`) on columns remains.
It is the responsability of the function which receive the data column
(either by gossip, by range request or by root request) to verify the
data column wrt. corresponding checks.
* Fix Nishant's comment.
* `validateDataColumnsByRange`: `current` ==> `currentSlot`.
* `validateRequest`: Extract `remotePeer` variable.
* `dataColumnSidecarsByRangeRPCHandler`: Small non functional refactor.
* `streamDataColumnBatch`: Fix major bug.
Before this commit, the node was unable to respond with a data column index higher than the count of stored data columns.
For example, if there is 8 data columns stored for a given block, the node was
able to respond for data columns indices 1, 3, and 5, but not for 10, 16 or 127.
The issue was visible only for full nodes, since super nodes always store 128 data columns.
* Initial sync: Fetch data columns from all peers.
(Not only from supernodes.)
* Nishant's comment: Fix `lastSlot` and `endSlot` duplication.
* Address Nishant's comment.
* Fix Commitments Check
* `highestFinalizedEpoch`: Refactor (no functional change).
* `retrieveMissingDataColumnsFromPeers`: Fix logs.
* `VerifyDataColumnSidecarKZGProofs`: Optimise with capacity.
* Save data columns when initial syncing.
* `dataColumnSidecarsByRangeRPCHandler`: Add logs when a request enters.
* Improve logging.
* Improve logging.
* `peersWithDataColumns: Do not filter any more on peer head slot.
* Fix Nishant's comment.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Improve logging.
* `retrieveMissingDataColumnsFromPeers`: Limit to `512` items per request.
* `retrieveMissingDataColumnsFromPeers`: Allow `nil` peers.
Before this commit:
If, when this funcion is called, we are not yet connected to enough peers, then `peers` will be possibly not be satisfaying,
and, if new peers are connected, we will never see them.
After this commit:
If `peers` is `nil`, then we regularly check for all connected peers.
If `peers` is not `nil`, then we use them.
* `BestFinalized`: Refactor (no functional change).
* `BestNonFinalized`: Refactor (no functional change).
* `beaconBlocksByRangeRPCHandler`: Remove useless log.
The same is already printed at the start of the function.
* `calculateHeadAndTargetEpochs`: Avoid `else`.
* `ConvertPeerIDToNodeID`: Improve error.
* Stop printing noisy "peer should be banned" logs.
* Initial sync: Request data columns from peers which:
- custody a superset of columns we need, and
- have a head slot >= our target slot.
* `requestDataColumnsFromPeers`: Shuffle peers before requesting.
Before this commit, we always requests peers in the same order,
until one responds something.
Without shuffling, we always requests data columns from the same
peer.
* `requestDataColumnsFromPeers`: If error from a peer, just log the error and skip the peer.
* Improve logging.
* Fix tests.
* `retrieveMissingDataColumnsFromPeers`: Improve logging.
* `dataColumnSidecarByRootRPCHandler`: Stop decreasing peer's score if asking for a column we do not custody.
* `dataColumnSidecarByRootRPCHandler`: If a data column is unavailable, stop waiting for it.
This behaviour was useful for peer sampling.
Now, just return the data column if we store it.
If we don't, skip.
* Dirty code comment.
* `retrieveMissingDataColumnsFromPeers`: Improve logs.
* `SendDataColumnsByRangeRequest`: Improve logs.
* `dataColumnSidecarsByRangeRPCHandler`: Improve logs.
* `columnErrBuilder`: Uses `Wrap` instead of `Join`.
Reason: `Join` makes a carriage return. The log is quite unreadable.
* `validateDataColumn`: Improve log.
* `areDataColumnsAvailable`: Improve log.
* `SendDataColumnSidecarByRoot` ==> `SendDataColumnSidecarsByRootRequest`.
* `handleDA`: Refactor error message.
* `sendRecentBeaconBlocksRequest` ==> `sendBeaconBlocksRequest`.
Reason: There is no notion at all of "recent" in the function.
If the caller decides to call this function only with "recent" blocks, that's fine.
However, the function itself will know nothing about the "recentness" of these blocks.
* `sendBatchRootRequest`: Improve comments.
* `sendBeaconBlocksRequest`: Avoid `else` usage and use map of bool instead of `struct{}`.
* `wrapAndReportValidation`: Remove `agent` from log.
Reason: This prevent the log to hold on one line, and it is not really useful to debug.
* `validateAggregateAndProof`: Add comments.
* `GetValidCustodyPeers`: Fix typo.
* `GetValidCustodyPeers` ==> `DataColumnsAdmissibleCustodyPeers`.
* `CustodyHandler` ==> `DataColumnsHandler`.
* `CustodyCountFromRemotePeer` ==> `DataColumnsCustodyCountFromRemotePeer`.
* Implement `DataColumnsAdmissibleSubnetSamplingPeers`.
* Use `SubnetSamplingSize` instead of `CustodySubnetCount` where needed.
* Revert "`wrapAndReportValidation`: Remove `agent` from log."
This reverts commit 55db351102.
* `scheduleReconstructedDataColumnsBroadcast`: Really minor refactor.
* `receivedDataColumnsFromRootLock` -> `dataColumnsFromRootLock`
* `reconstructDataColumns`: Stop looking into the DB to know if we have some columns.
Before this commit:
Each time we receive a column, we look into the filesystem for all columns we store.
==> For 128 columns, it looks for 1 + 2 + 3 + ... + 128 = 128(128+1)/2 = 8256 files look.
Also, as soon as a column is saved into the file system, then if, right after, we
look at the filesystem again, we assume the column will be available (strict consistency).
It happens not to be always true.
==> Sometimes, we can reconstruct and reseed columns more than once, because of this lack of filesystem strict consistency.
After this commit:
We use a (strictly consistent) cache to determine if we received a column or not.
==> No more consistency issue, and less stress for the filesystem.
* `dataColumnSidecarByRootRPCHandler`: Improve logging.
Before this commit, logged values assumed that all requested columns correspond to
the same block root, which is not always the case.
After this commit, we know which columns are requested for which root.
* Add a log when broadcasting a data column.
This is useful to debug "lost data columns" in devnet.
* Address Nishant's comment
* `sendBatchRootRequest`: Refactor and add comments.
* `sendBatchRootRequest`: Do send requests to peers that custodies a superset of our columns.
Before this commit, we sent "data columns by root requests" for data columns peers do not custody.
* Data columns: Use subnet sampling only.
(Instead of peer sampling.)
aaa
* `areDataColumnsAvailable`: Improve logs.
* `GetBeaconBlock`: Improve logs.
Rationale: A `begin` log should always be followed by a `success` log or a `failure` log.
* `CustodyCountFromRemotePeer`: Set happy path in the outer scope.
* `FindPeersWithSubnet`: Improve logging.
* `listenForNewNodes`: Avoid infinite loop in a small subnet.
* Address Nishant's comment.
* FIx Nishant's comment.
* `pingPeers`: Add log with new ENR when modified.
* `p2p Start`: Use idiomatic go error syntax.
* P2P `start`: Fix error message.
* Use not bootnodes at all if the `--chain-config-file` flag is used and no `--bootstrap-node` flag is used.
Before this commit, if the `--chain-config-file` flag is used and no `--bootstrap-node` flag is used, then bootnodes are (incorrectly) defaulted on `mainnet` ones.
* `validPeersExist`: Centralize logs.
* `AddConnectionHandler`: Improve logging.
"Peer connected" does not really reflect the fact that a new peer is actually connected. --> "New peer connection" is more clear.
Also, instead of writing `0`, `1`or `2` for direction, now it's writted "Unknown", "Inbound", "Outbound".
* Logging: Add 2 decimals for timestamt in text and JSON logs.
* Improve "no valid peers" logging.
* Improve "Some columns have no peers responsible for custody" logging.
* `pubsubSubscriptionRequestLimit`: Increase to be consistent with data columns.
* `sendPingRequest`: Improve logging.
* `FindPeersWithSubnet`: Regularly recheck in our current set of peers if we have enough peers for this topic.
Before this commit, new peers HAD to be found, even if current peers are eventually acceptable.
For very small network, it used to lead to infinite search.
* `subscribeDynamicWithSyncSubnets`: Use exactly the same subscription function initially and every slot.
* Make deepsource happier.
* Nishant's commend: Change peer disconnected log.
* NIshant's comment: Change `Too many incoming subscription` log from error to debug.
* `FindPeersWithSubnet`: Address Nishant's comment.
* `batchSize`: Address Nishant's comment.
* `pingPeers` ==> `pingPeersAndLogEnr`.
* Update beacon-chain/sync/subscriber.go
Co-authored-by: Nishant Das <nishdas93@gmail.com>
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
* `broadcastAndReceiveDataColumns`: Use real `sidecar.ColumnIndex` instead of position in the slice.
And improve logging as well.
* `isDataColumnsAvailable`: Improve logging.
* `validateDataColumn`: Print `Accepted data column sidecar gossip` really at the end.
* Subscriber: Improve logging.
* `sendAndSaveDataColumnSidecars`: Use common used function for logging.
* `dataColumnSidecarByRootRPCHandler`: Logging - Pring `all` instead of all the columns for a super node.
* Verification: Improve logging.
* `DataColumnsWithholdCount`: Set as `uint64` instead `int`.
* `DataColumnFields`: Improve logging.
* Logging: Remove now useless private `columnFields`function.
* Avoid useless goroutines blocking for reconstruction.
* Update beacon-chain/sync/subscriber.go
Co-authored-by: Nishant Das <nishdas93@gmail.com>
* Address Nishant's comment.
* Improve logging.
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
* Persist All Changes
* Fix All Tests
* Fix Build
* Fix Build
* Fix Build
* Fix Test Again
* Add missing verification
* Add Test Cases for Data Column Validation
* Fix comments for methods
* Fix comments for methods
* Fix Test
* Manu's Review
* Fix the obvious...
* Data columns sampling: Modify logging.
* `waitForChainStart`: Set it threadsafe - Do only wait once.
* Sampling: Wait for chain start before running the sampling.
Reason: `newDataColumnSampler1D` needs `s.ctxMap`.
`s.ctxMap` is only set when chain is started.
Previously `waitForChainStart` was only called in `s.registerHandlers`, it self called in a go-routine.
==> We had a race condition here: Sometimes `newDataColumnSampler1D` were called once `s.ctxMap` were set, sometimes not.
* Adresse Nishant's comments.
* Sampling: Improve logging.
* `waitForChainStart`: Remove `chainIsStarted` check.
* `SendDataColumnsByRangeRequest`: Add some new fields in logs.
* `BlobStorageSummary`: Implement `HasDataColumnIndex` and `AllDataColumnsAvailable`.
* Implement `fetchDataColumnsFromPeers`.
* `fetchBlobsFromPeer`: Return only one error.
* DeepSource: Pass heavy objects by pointers.
* `removeBlockFromQueue`: Remove redundant error checking.
* `fetchBlobsFromPeer`: Use same variable for `append`.
* Remove unused arguments.
* Combine types.
* `Persist`: Add documentation.
* Remove unused receiver
* Remove duplicated import.
* Stop using both pointer and value receiver at the same time.
* `verifyAndPopulateColumns`: Remove unused parameter
* Stop using mpty slice literal used to declare a variable.
* PeerDAS: Run reconstruction in parallel.
* `isDataAvailableDataColumns` --> `isDataColumnsAvailable`
* `isDataColumnsAvailable`: Return `nil` as soon as half of the columns are received.
* Make deepsource happy.
* Update ckzg4844 to latest version
* Run go mod tidy
* Remove unnecessary tests & run goimports
* Remove fieldparams from blockchain/kzg
* Add back blank line
* Avoid large copies
* Run gazelle
* Use trusted setup from the specs & fix issue with struct
* Run goimports
* Fix mistake in makeCellsAndProofs
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* use recoverCellsAndKZGProofs
* make recoverAllCells and CellsToBlob private
* chore: all methods now return CellsAndProof struct
* chore: update code
* chore: move all ckzg related functionality into kzg package
* refactor code to match
* run: bazel run //:gazelle -- fix
* chore: add some docs and stop copying large objects when converting between types
* fixes
* manually add kzg.go dep to Build.Hazel
* move kzg methods to kzg.go
* chore: add RecoverCellsAndProofs method
* bazel run //:gazelle -- fix
* make Cells be flattened sequence of bytes
* chore: add test for flattening roundtrip
* chore: remove code that was doing the flattening outside of the kzg package
* fix merge
* fix
* remove now un-needed conversion
* use pointers for Cell parameters
* linter
* rename cell conversion methods (this only applies to old version of c-kzg)
* change recoverBlobs to recoverCellsAndProofs
* modify code to take in the cells and proofs for a particular blob instead of the blob itself
* add CellsAndProofs structure
* modify recoverCellsAndProofs to return `cellsAndProofs` structure
* modify `DataColumnSidecarsForReconstruct` to accept the `cellsAndKZGProofs` structure
* bazel run //:gazelle -- fix
* use kzg abstraction for kzg method
* move CellsAndProofs to kzg.go
* chore: move all ckzg related functionality into kzg package
* refactor code to match
* run: bazel run //:gazelle -- fix
* chore: add some docs and stop copying large objects when converting between types
* fixes
* manually add kzg.go dep to Build.Hazel
* move kzg methods to kzg.go
* chore: add RecoverCellsAndProofs method
* bazel run //:gazelle -- fix
* use BytesPerBlob constant
* chore: fix some deepsource issues
* one declaration for commans and blobs
* Wrap errors, add logs.
* `missingColumnRequest`: Fix blobs <-> data columns mix.
* `ColumnIndices`: Return `map[uint64]bool` instead of `[fieldparams.NumberOfColumns]bool`.
* `DataColumnSidecars`: `interfaces.SignedBeaconBlock` ==> `interfaces.ReadOnlySignedBeaconBlock`.
We don't need any of the non read-only methods.
* Fix comments.
* `handleUnblidedBlock` ==> `handleUnblindedBlock`.
* `SaveDataColumn`: Move log from debug to trace.
If we attempt to save an already existing data column sidecar,
a debug log was printed.
This case could be quite common now with the data column reconstruction enabled.
* `sampling_data_columns.go` --> `data_columns_sampling.go`.
* Reconstruct data columns.
* Support Data Columns For By Root Requests
* Revert Config Changes
* Fix Panic
* Fix Process Block
* Fix Flags
* Lint
* Support Checkpoint Sync
* Manu's Review
* Add Support For Columns in Remaining Methods
* Unmarshal Uncorrectly
* Add Data Structure for New Request Type
* Add Data Column By Range Handler
* Add Data Column Request Methods
* Add new validation for columns by range requests
* Fix Build
* Allow Prysm Node To Fetch Data Columns
* Allow Prysm Node To Fetch Data Columns And Sync
* Bug Fixes For Interop
* GoFmt
* Use different var
* Manu's Review
* Enable E2E And Add Fixes
* Register Same Topic For Data Columns
* Initialize Capacity Of Slice
* Fix Initialization of Data Column Receiver
* Remove Mix In From Merkle Proof
* E2E: Subscribe to all subnets.
* Remove Index Check
* Remaining Bug Fixes to Get It Working
* Change Evaluator to Allow Test to Finish
* Fix Build
* Add Data Column Verification
* Fix LoopVar Bug
* Do Not Allocate Memory
* Update beacon-chain/blockchain/process_block.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/core/peerdas/helpers.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/core/peerdas/helpers.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Gofmt
* Fix It Again
* Fix Test Setup
* Fix Build
* Fix Trusted Setup panic
* Fix Trusted Setup panic
* Use New Test
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Add new DA check
* Exit early in the event no commitments exist.
* Gazelle
* Fix Mock Broadcaster
* Fix Test Setup
* Update beacon-chain/blockchain/process_block.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Manu's Review
* Fix Build
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Remove capital letter from error messages.
* `[4]byte` => `[fieldparams.VersionLength]byte`.
* Prometheus: Remove extra `committee`.
They are probably due to a bad copy/paste.
Note: The name of the probe itself is remaining,
to ensure backward compatibility.
* Implement Proposer RPC for data columns.
* Fix TestProposer_ProposeBlock_OK test.
* Remove default peerDAS activation.
* `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
* Bump `c-kzg-4844` lib to the `das` branch.
* Implement `MerkleProofKZGCommitments`.
* Implement `das-core.md`.
* Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`.
* `CustodyColumnSubnets`: Include `i` in the for loop.
* Remove `computeSubscribedColumnSubnet`.
* Remove `peerdas.CustodyColumns` out of the for loop.
* sse implementation that sheds stuck clients
* Radek and James feedback
* Refactor event streamer code for readability
* less-flaky test signaling
* test case where queue fills; fixes
* add changelog entry
* james and preston feedback
* swap our Subscription interface with an alias
* event.Data can be nil for the payload attr event
* deepsource
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* allow users to publish blobs
Allowing users to publish blobs before publishing blocks, gives the blobs a head start. They can begin to propagate around the network while the block is being validated.
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
* Update beacon-chain/rpc/prysm/beacon/handlers.go
---------
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* removing extra container for ssz blob sidecar response
* gaz and changelog
* gaz
* fixing dependencies
* fixing test
* updating values based on feedback"
* version bump on k8s io client-go and apimachinery
* bazel file served
* fixing build issues
* some changes in noops functions
* Update CHANGELOG.md
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Use read only validators on ApplyToEveryValidator
* Use ReadFromEveryValidator on slashing
* change changelog
* Revert "Use ReadFromEveryValidator on slashing"
This reverts commit 74c055bddb56e0573075c71df8a40f1c6a9bfdfd.
* looking at ways to reduce validator registration calls
* small mistake, should be epoch start
* adding more optimizations for reducing registration calls while covering more edgecases
* linting
* adding change log and force full push override
* fixing bug and adding tests
* changing if statement just to be safe
* potuz feedback for easier readability
* more review feedback for simplicity
* more review suggestions from potuz
* fix unit test
* reduce redundancy
* Update CHANGELOG.md
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* small nitpick
* fixing typo
* updating logs
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* e2e release test breaks due to changes in flag naming
* changing approach to fix
* adding some small alias test
* fixing test, changelog, and flag name
* Update config_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* wip passing e2e
* reverting temp comment
* remove unneeded comments
* fixing merge errors
* fixing more bugs from merge
* fixing test
* WIP moving code around and fixing tests
* unused linting
* gaz
* temp removing these tests as we need placeholder/wrapper APIs for them with the removal of the gateway
* attempting to remove dependencies to gRPC gateway , 1 mroe left in deps.bzl
* renaming flags and other gateway services to http
* goimport
* fixing deepsource
* git mv
* Update validator/package/validator.yaml
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update validator/package/validator.yaml
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update cmd/beacon-chain/flags/base.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update cmd/beacon-chain/flags/base.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update cmd/beacon-chain/flags/base.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* addressing feedback
* missed lint
* renaming import
* reversal based on feedback
* fixing web ui registration
* don't require mux handler
* gaz
* removing gRPC service from validator completely, merged with http service, renames are a work in progress
* updating go.sum
* linting
* trailing white space
* realized there was more cleanup i could do with code reuse
* adding wrapper for routes
* reverting version
* fixing dependencies from merging develop
* gaz
* fixing unit test
* fixing dependencies
* reverting unit test
* fixing conflict
* updating change log
* Update log.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* gaz
* Update api/server/httprest/server.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* addressing some feedback
* forgot to remove deprecated flag in usage
* gofmt
* fixing test
* fixing deepsource issue
* moving deprecated flag and adding timeout handler
* missed removal of a flag
* fixing test:
* Update CHANGELOG.md
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* addressing feedback
* updating comments based on feedback
* removing unused field for now, we can add it back in if we need to use the option
* removing unused struct
* changing api-timeout flag based on feedback
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* change LCUpdate to use LCHeader
* fix api struct usages
* fix api struct finalized_header
* add lightclientheader to proto structs
* fix proto usages
* fix proto usages in events
* fix uppercase field in protobuf defenition
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Adds a wrapper package
* Gazelle
* Add in Empty Span
* Revert It Back
* Add back reference
* Set It As Empty
* fix missing import
* remove redundant alias
* remove unused
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
Co-authored-by: Saolyn <sammy@rosso.red>
* hardcoded GenesisValidatorsRoot
* added in mainnet.config
* updated desc
* added it in all testnets
* minor change
* added roots instead of empty and fn to compute byte32 from hex
* added in e2e testnet_config
* fixed test
* minor fix
* removed fn and added bytes output directly
* Add test for genesis validator root mainnet value
* removed root from minimal and testnet
* removed root
* Update CHANGELOG.md
* Fix bazel package visiblity
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Preston Van Loon <preston@pvl.dev>
* PR to update workflow check dependencies
* Updated build checkout version to v4
* Updated to go 1.23.0
* Updated lint version to v1.60.3
* Revert to 1.22.3
* Updated go to 1.23
* revert
* Updated setup-go to v5
* Update lint to 1.60.2
* Revert changes
* Update Lint version to v1.60.3
* Update lint to go 1.23.0
* Update golanci.yml to 1.23.0
* Revert and keep to golang 1.22.4
* Disable mnd
* Downgrade to current version
* Add update to go 1.26
* Update to go 1.22.6
* Update .golangci.yml to 1.22.6
* Removed the default value of the bootnode flag to prevent it from being overridden during testnet usage
* bugfix for checking stringslice flag to use isSet
* fix: replace `BeaconBlockHeader` in `createLightClientBootstrap` with `LightClientHeader`
* minor fix in `handlers_test.go`
* check if `beacon` is `nil` instead of `header`
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* create light-client-updates bucket
* Electra committe validation for aggregate and proof (#14317)
* Electra committe validation for aggregate and proof
* review
* update comments
* Refactor get local payload (#14327)
* Refactor get local payload
* Fix go lint: new line
* add lightclient db kv functions
* lightclient db tests
* move blockchain/lightclient.go to core/light-client package
* add comparison check for start and end period
* create testing/utils/lightcilent.go
* lightclient db tests
* fix imports and usages
* fix imports and usages in process_block_helpers
* fix bazel dependencies
* remove unnecessary nil check
* add more tests for lightclient kv functions
* refactor tests
* refactor kv.LightClientUpdates
* fix db to return every update that is available in the requested range
* run gazzele fix
* return empty map in case of empty db
* fix goimports errors
* goimports
* Revert "Auxiliary commit to revert individual files from aa7ce6f37cb6767cf11642b022b2ce59d42ae621"
This reverts commit 33c707f5bd164386449dc14ff27d95ad5f195161.
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: terence <terence@prysmaticlabs.com>
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
* feat: implement function `PayloadProof` to calculate proof of execution payload
* remove comments
* feat: implement function to compute field roots of
* feat: implement function to compute `BeaconBlock` field roots and add tests
* fix dependencies
* check if interface implements the assserted type
* fix: lint
* replace `ok != true` with `!ok`
* remove unused parameter from `PayloadProof`
* remove test and move `PayloadProof` to `blocks/proofs.go`
* remove `PayloadProof` from `fieldtrie`
* replace `fieldtrie.ProofFromMerkleLayers` with `trie.ProofFromMerkleLayers`
* Update container/trie/sparse_merkle.go
* update dependencies
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* Allow blobs in Electra
* Add Test
* Add Error for unknown request type
* Fix Test
* fix tests
* clean up logic
---------
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
* feat: (WIP)implement ``ComputeFieldRootsForBlockBody`` function to compute roots of fields in BlockBody
* calculate field roots upto ``deposits``
* add some required constants fix some errors
* implement ``ComputeFieldRootsForBlockBody`` function for all fields in ``BeaconBlockBody``
* populate `fieldRoots` based on block body version
* fix constants
* `bazel run //:gazelle -- fix`
* remove nested `if`s
* formatting 1
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* formatting 2
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* remove unrequired check
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* update naming 1
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* update function name
* add test for Phase0 block body
* update Phase0 test
* update phase0 test without setting any fields
* fix some errors in Phase0 test
* don't set fields
* add altair test
* add tests upto Electra
* fix the function for deneb and newer forks
* update dependencies
* add checks to ensure interface implements the asserted type
---------
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Remove Feature Flag From Prater (#12082)
* Use Epoch boundary cache to retrieve balances (#12083)
* Use Epoch boundary cache to retrieve balances
* save boundary states before inserting to forkchoice
* move up last block save
* remove boundary checks on balances
* fix ordering
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* chore: fix unfound image
* fix link
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* missed adding electra process epoch on state replay
* abstracting common functionality as to not forget in processing
* removing unnessesary error check
* breaking out core process slots logic out
* fixing small bug in epoch processing
* adding small log to fix linting and make use of logrus in the package
* adding some unit tests for process epoch
* potuz's review feedback
* reversing processing order based on feedback
* feat: implement ``is_better_update`` for light client
* replace ``!=`` with ``!reflect.DeepEqual``
* chore: import constant instead of redeclaring and some minor fixes
* feat: add test for ``newHasSupermajority != oldHasSupermajority``
* chore: capitalise first letter of function names
* refactor: rewrite test setups in a more readable manner
* two test cases
* feat: add tests for ``IsBetterUpdate``
* refactor: remove double import
* wording fix 1
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* wording fix 2
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* minor fixes
* fix: change function names to lowercase and fix dependencies
---------
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* moving beacon block fields from cloners to individual files and adding fuzzing tests
* adding missed tests
* fixing tests:
* removing deep not equals for now
* adding clarifying comment
* using fork instead of eip6110flag for validator index cache
* updating validator indicies on initialize from poroto unsafe
* fixing bad version check
* attempting to fix unit test
* reverting test changes
* changing incorrect initialization
* updating comment
* radek's comment'
* poc for payload electra cloning
* partial fixes
* fixing build
* addressing kasey's comment
* forgot to unexport interface
* making test more generic
* making fuzzing slightly more robust
* renaming based on kasey's comment
* using fuzz test in same package to avoid exporting interface
* fixing build
* fixing marshalling and adding more to unit test
* updating missed consolidation requests
* renaming variables
* adding test gen
* reverting config change
* `PruneProposalsAtEpoch`: Test return value in case of nothing to prune.
* `TestStore_PruneAttestations_OK`: Create unique validator indexes.
Before this commit, `attester2` for `j = n` was the same than
`attester1` for `j = n + 1`, resulting in erasure of a lot of attesters.
I guess it was not the initial intent.
* Slasher pruning: Check if the number of pruned items corresponds to the expectation.
Before this commit, if the pruning function did remove a superset
of the expected pruned items (including all the items), then the test would pass.
* Prune items that should be pruned and stop pruning items that should not be pruned.
The first 8 bytes of the key of `attestation-data-roots` and
`proposal-records` bytes correspond respectively to an encoded epoch and
and encoded slot.
The important word in this sentence is "encoded".
Before this commit, these slot/epoch are SSZ encoded, which means that
they are little-endian encoded.
However:
- `uint64PrefixGreaterThan` uses `bytes.Compare` which expects
big-endian encoded values.
- `for k, _ := c.First(); k != nil; k, _ = c.Next()` iters over the
keys in big-endian order.
The consequence is:
- Some items that should be pruned are not pruned, provoking a disk
usage leak.
- Some items that should not be pruned are pruned, provoking errors like
in https://github.com/prysmaticlabs/prysm/issues/13658.
This commit encodes the slot/epoch as big-endian before storing them in
the database keys.
Why this bug has not been detected in unit test before?
The values used in unit tests before this commit in
`TestStore_PruneProposalsAtEpoch` and `TestStore_PruneAttestations_OK`
are `10` and `20`.
Unfortunately, checking if `littleEndian(20) > littlenEndien(10)`
with the `>` operator considering operands as big-endian encoded returns
the expected result...
Just replacing `20` by `30` trigs the bug.
* Make deepsource happy.
* Slasher: Migrate database from little-endian to big-endian.
* Update beacon-chain/slasher/service.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/db/slasherkv/migrate.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `TestMigrate`: Fix documentation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Bump go version up
* Update to go 1.22 compatible version
* Fix NoSec declarations
* Skip Gosec in GolangCi
* Avoid Bug In Analyzer
* Add in Gohashtree and Update to 1.22.4
* Fix Go Sum
* validator/client: process Sync Committee roles separately
In a DV context, to be compatible with the proposed selection endpoint, the VC must push all partial selections to it instead of just one.
Process sync committee role separately within the RolesAt method, so that partial selections can be pushed to the DV client appropriately, if configured.
* testing/util: re-add erroneously deleted state_test.go
* validator/client: fix tests
* validator/client: always process sync committee validator
Process sync committee duty at slot boundary as well.
* don't fail if validator is marked as sync committee but it is not in the list
ignore the duty and continue
* address code review comments
* fix build
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* eip-7251: registry_updates
* Fix spectest violation
* More unit tests for process_registry_updates
* Update tests for ProcessRegistryUpdates
* Minor refactorings and benchmark
* Update beacon-chain/core/electra/registry_updates.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* Wrap errors from process_registry_updates for easier debugging
* Rewrite process_registry_updates not to use st.Validators()
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* wip changes
* wip changes refactoring deposit functions
* wip on deposit functions
* wip changes to fix deposit processing
* more wip function logic
* gaz
* wip refactoring deposit changes
* removing circlular dependency and other small fix
* fixing circular dependencies
* fixing validators file
* fixing more tests
* fixing unit tests
* wip deposit packing
* refactoring packing
* changing packing code some more
* fixing packing change
* updating more tests
* removing comment
* fixing transition code for eip6110
* including inserts for validator index cache
* adding tests and placeholder test
* moving deposit related unit tests over
* adding in test for electra proposer packing
* spec test wip
* moving cache saving to the correct spot
* eip-6110: deposit spectests
* adding deposit receipt spec tests
* reverting altair operations in this pr and will be handled separately
* fixing renames but need to refactor process deposits
* gaz
* fixing names
* fixing linting
* fixing unit test
* fixing a test and updating test util
* bal never used
* fixing more tests
* fixing one more test
* addressing feedback
* refactoring process deposits to be part of their own fork instead of in blocks package
* adding new tests to cover functions and removing redundancies
* removing comment based on feedback
* resolving easy deepsource issue
* refactoring for appropriate aliasing and readability
* reverting some changes to simplify diff
* small edit to comment
* fixing linting
* fixing merge changes and review comments
* Update beacon-chain/rpc/prysm/v1alpha1/validator/proposer_deposits.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* partial review comments
* addressing slack feedback
* moving function from deposits to validator.go
* adding in batch verification and improving logs
---------
Co-authored-by: Preston Van Loon <preston@pvl.dev>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `IndexedAtt` wrapper for the slasher feed
* test fixes
* fix simulator
* fix e2e
* Revert "Auxiliary commit to revert individual files from 191bbf77accfc2523fa9f909837a2e9dca132afa"
This reverts commit 2b0441a23a0e5f66e50cf36c3bbfbb39d587b17b.
* extract interface from channel
* adding logic for saving validator indicies in new cache
* updating index cache for eip6110
* Update setters_misc.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* linting
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* renaming functions and fields based on consensus changes
* execution api rename
* fixing test
* reverting spectests changes, it should be changed with new version
* reverting temporarily
* revert exclusions
* Increase health tracker channel buffer size
* keep only the latest value
* Make health test blocking as a regression test for PR #14807
* Fix new race conditions in the MockHealthClient
---------
Co-authored-by: Preston Van Loon <preston@pvl.dev>
* always close cache warm chan to prevent blocking
* test that waitForCache does not block
* combine defers to reduce cognitive overhead
* lint
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* eip-7251: process_effective_balance_updates
Spectests for process_effective_balance_updates
process_effective_balance_updates unit tests
* PR feedback from the amazing @rkapka
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Initial commit
* Fuzz all generated ssz en/decoders.
* Make proto message comparison aware of nested proto messages and differing unknown fields therein.
* Don't name receiver variable if not necessary.
* Run gazelle
* Cleanup part 1
* Cleanup part 2
* Cleanup part 3
* remove lock field init
* doc for SignerConfig
* remove vars
* use full Keymanager word in function
* revert interface rename
* linter
* fix build issues
* review
* eip-7251: process_pending_balance_deposits
* Update beacon-chain/core/electra/balance_deposits_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Remove defensive check. A unit test shows nothing bad happens
* Safe sub to protect from underflow
* Use @kasey's idea for safer subtraction
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* electra engine api support
* add marshaling support for ExecutionPayloadElectra
* add receipts to json tests
* deep source
* simplify slice handling
* deep source lint about type/method order
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* wip fork logic upgrade
* fixing replay and fork.go
* improving process function and adding tests for transition
* updating unit tests and temporarily removing some fields on state_trie.go
* updating state
* wip adding upgrade to electra code
* adding some comments
* adding spec tests
* fixing values used in state transition logic
* updating upgrade test
* gofmt
* avoid dup word linting
* fixing spec tests for fork
* gaz
* fixing tests
* improving unit test with new getters
* fixing bazel for minimal fork test
* adding bazel file
* Update beacon-chain/core/electra/upgrade.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* addressing some comments and adding more tests
* addressing more feedback
* one more feedback
* changing value to interface after talking to preston
* adding missed review feedback
* fixing linting
* noticed I was using the wrong function in the state upgrade
* fixing and ignoring some deepsource issues
* moving core electra validator functions to helper to remove circular dependencies in other PRs
* missed deepsource complaint
* Update upgrade.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update testing/util/electra_state.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update testing/util/electra_state.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* addressing feedback
* removing deepsoure ignore comments
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Electra: Beacon State
* Electra: Beacon state fixes from PR 13919
* Add missing tests - part 1
* Split eip_7251_root.go into different files and reuse/share code with historical state summaries root. It's identical!
* Add missing tests - part 2
* deposit receipts start index getters and setters (#13947)
* adding in getters and setters for deposit receipts start index
* adding tests
* gaz
* Add missing tests - part 3 of 3
Update the electra withdrawal example with a ssz state containing pending partial withdrawals
* add tests for beacon-chain/state/state-native/getters_balance_deposits.go
* Add electra field to testing/util/block.go execution payload
* godoc commentary on public methods
* Fix failing test
* Add balances index out of bounds check and relevant tests.
* Revert switch case electra
* Instead of copying spectest data into testdata, use the spectest dependency
* Deepsource fixes
* Address @rkapka PR feedback
* s/MaxPendingPartialsPerWithdrawalSweep/MaxPendingPartialsPerWithdrawalsSweep/
* Use multivalue slice compatible accessors for validator and balance in ActiveBalanceAtIndex
* More @rkapka feedback. What a great reviewer!
* More tests for branching logic in ExitEpochAndUpdateChurn
* fix build
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Electra: full beacon config
Fix TestGetSpec
* Fix beacon config spec compliance test so that it properly loads the config from spec tests. Tests failing for now.
* fix tests and comply with spec presets
* block protos
* proto and ssz
* stubs
* Enable Electra spec test
* Pull in EIP-7251 protobuf changes
From: https://github.com/prysmaticlabs/prysm/pull/13903
* All EIP7549 containers are passing
* All EIP7251 containers passing
* including changes from eip7002
* Everything passing except for beacon state hash tree root
* fixing eletra state to use electra payload
* Fix minimal test. Skip beacon state test
* Perston's feedback
---------
Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>
* Remove EnableEIP4881 flag
* Gaz
* Fix missing error handler
* Remove old tree and fix tests
* Gaz
* Fix build import
* Replace depositcache
* Add pendingDeposit tests
* Nishant's fix
* Fix unsafe uint64 to int
* Fix other unsafe uint64 to int
* Remove: RemovePendingDeposit
* Deprecate and remove DisableEIP4881 flag
* Check: index not greater than deposit count
* Move index check
* GET
* POST
* Revert "Auxiliary commit to revert individual files from 615feb104004d6a945ededf5862ae38325fc7ec2"
This reverts commit 55cf071c684019f3d6124179154c10b2277fda49.
* comment fix
* deepsource
* Beacon-api: broadcast blobs in the event of seen block
* Fix parameters
* Fix test
* Check forkchoice
* Ran go format
* Revert "Ran go format"
This reverts commit 091e77e81d6e2b9861fecc27c0bad1898033f9a3.
* James feedback
* Radek's feedback
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Fix bad tests
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* allow other pkgs to check for blobs in pruning cache
* address deepsource complaints
* custom error to simplify test setup
* add AllAvailable method
* make storage summary slot field private
* unit test and off-by-one fix
* remove comment with copy of tested function
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Revert "Modify the algorithm of `updateFinalizedBlockRoots` (#13486)"
This reverts commit 32fb183392.
* migration to fix index corruption from pr 13486
* bail as soon as we see 10 epochs without the bug
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Remove check for duplicates in pending attestation queue
The current queue will only save 1 unaggregated attestation for a pending block because we wrap the object into a SignedAggregatedAttestationAndProof with a zeroed aggregator.
* fix tests
* add max/min span visualisation tool cli
* go mod tidy
* lint imports
* remove typo
* fix epoch table value
* fix deepsource
* add dep to bazel
* fix dep import order
* change command name from span to slasher-span-display
* change command args style using - instead of _
* sed s/CONFIGURATION/SLASHER PARAMS//
* change double neg to double pos condition
* remove unused anonymous func
* better function naming
* add range condition
* [deepsource] Fix Empty slice literal used to declare a variable
GO-W1027
* correct typo
* do not show incorrect epochs due to round robin
* fix import
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
ValidateClock in ValidateAttestationTime is useless
The check is that the attSlot is not > than the currentslot + 128 slots.
Later there's a check that the attSlot start time is not > than current slot
start time + clockDisparity.
if attSlot > than currentSlot + 128 slots, then the second check would fail
anyway.
The lattest check already guarantees that the attSlot cannot be larger than the
currentSlot, therefore it may never happen that attEpoch > currentEpoch. We just
need to check for Deneb that attEpoch >= currentEpoch - 1.
Removes also some duplicated variables like the attestation epoch being computed
twice.
* refactor batch verifier to share with pending queue
* unit test for batch verifier
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Remove Feature Flag From Prater (#12082)
* Use Epoch boundary cache to retrieve balances (#12083)
* Use Epoch boundary cache to retrieve balances
* save boundary states before inserting to forkchoice
* move up last block save
* remove boundary checks on balances
* fix ordering
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* create the log file along with its parent directory if not present
* only give ReadWritePermissions to the log file
* use io/file package to create the parent directories
* fix ci related issues
* add regression tests
* run gazelle
* fix tests
* remove print statements
* gazelle
* Remove failing test for MkdirAll, this failure is not expected
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* add log message if in da check at slot end
* don't bother logging late da check start
* break up defer with a var, too dense all together
* pass slot instead of block ref
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* addressing errors on events endpoint
* reverting timeout on get health
* fixing linting
* fixing more linting
* Update validator/client/beacon-api/beacon_api_validator_client.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/rpc/eth/events/events.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* reverting change and removing line on context done which creates a superfluous response.WriteHeader error
* gofmt
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* make it very big
* use new pruning implementation
* handle pre deneb
* revert cache change
* less verbose
* gaz
* Update beacon-chain/operations/attestations/prune_expired.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* gofmt
* be safer
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* handle special case of batch size=1
* unit test case for backfill batch len=1
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* flag to set an older backfill slot target
* wire up flag to main cli and usage
* fix deepsource complaints
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `subscribeStaticWithSubnets`: Fix docstring.
* `buildOptions`: Avoid `options` mutations.
* `dv5Cfg`: Avoid mutation.
* `RefreshENR`: Use default for all but Phase0.
* `udp4`, `udp6`: Create enum.
* `p2p.Config`: `BootstrapNodeAddr`==> `BootstrapNodeAddrs`.
* `p2p.Config`: `Discv5BootStrapAddr` ==> `Discv5BootStrapAddrs`.
* `TestScorers_BadResponses_Score`: Improve.
* `BeaconNode`: Avoid mutation.
* `TestStore_TrustedPeers`: Remove blankline.
* Remove blank identifiers.
* `privKey`: Keep the majority of code with low indentation.
* `P2PPreregistration`: Return error instead of fatal log.
* `parseBootStrapAddrs` => `ParseBootStrapAddrs` (export)
* `p2p.Config`: Remove `BootstrapNodeAddrs`.
* `NewService`: Avoid mutation when possible.
* `Service`: Remove blank identifier.
* `buildOptions`: Avoid `log.Fatalf` (make deepsource happy).
* `registerGRPCGateway`: Use `net.JoinHostPort` (make deepsource happy).
* `registerBuilderService`: Make deepsource happy.
* `scorers`: Add `NoLock` suffix (make deepsource happy).
* `scorerr`: Add some `NoLock`suffixes (making deepsource happy).
* `discovery_test.go`. Remove init.
Rationale:
`rand.Seed` is deprecated: As of Go 1.20 there is no reason to call Seed with a random value. Programs that call Seed with a known value to get a specific sequence of results should use New(NewSource(seed)) to obtain a local random generator.
This makes deepsource happy as well.
* `createListener`: Reduce cyclomatic complexity (make deepsource happy).
* `startDB`: Reduce cyclomatic complexity (make deepsource happy).
* `main`: Log a FATAL on error.
This way, the error message is very readable.
Before this commit, the error message is the less readable
message in the logs.
* `New`: Reduce cyclomatic complexity (make deepsource happy).
* `main`: Avoid `App` mutation, and make deepsource happy.
* Update beacon-chain/node/node.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* `bootnodes` ==> `BootNodes` (Fix PR comment).
* Remove duplicate `configureFastSSZHashingAlgorithm` since already done in `configureBeacon`. (Fix PR comment)
* Add `TestCreateLocalNode`. (PR comment fix.)
* `startModules` ==> `startBaseServices (Fix PR comment).
* `buildOptions` return errors consistently.
* `New`: Change ordering.
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* use real blob verifier in forkchoice spectest
* wip
* Use real blob sidecar for test
* Set file db correctly
* correctly handle blob cases where valid=false
* work-around spectest's weird Fork in genesis state
* gaz
* revert T-money's log level change
* rm whitespace
* unskip minimal test
* Preston's feedback
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* add them
* Ensure activation epoch does not overflow
* add them all in
* better check for overflow
* fix tests
* fix tests
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* `EpochFromString`: Use already defined `Uint64FromString` function.
* `Test_uint64FromString` => `Test_FromString`
This test function tests more functions than `Uint64FromString`.
* Slashing protection history: Remove unreachable code.
The function `NewKVStore` creates, via `kv.UpdatePublicKeysBuckets`,
a new item in the `proposal-history-bucket-interchange`.
IMO there is no real reason to prefer `proposal` than `attestation`
as a prefix for this bucket, but this is the way it is done right now
and renaming the bucket will probably be backward incompatible.
An `attestedPublicKey` cannot exist without
the corresponding `proposedPublicKey`.
Thus, the `else` portion of code removed in this commit is not reachable.
We raise an error if we get there.
This is also probably the reason why the removed `else` portion was not
tested.
* `NewKVStore`: Switch items in `createBuckets`.
So the order corresponds to `schema.go`
* `slashableAttestationCheck`: Fix comments and logs.
* `ValidatorClient.db`: Use `iface.ValidatorDB`.
* BoltDB database: Implement `GraffitiFileHash`.
* Filesystem database: Creates `db.go`.
This file defines the following structs:
- `Store`
- `Graffiti`
- `Configuration`
- `ValidatorSlashingProtection`
This files implements the following public functions:
- `NewStore`
- `Close`
- `Backup`
- `DatabasePath`
- `ClearDB`
- `UpdatePublicKeysBuckets`
This files implements the following private functions:
- `slashingProtectionDirPath`
- `configurationFilePath`
- `configuration`
- `saveConfiguration`
- `validatorSlashingProtection`
- `saveValidatorSlashingProtection`
- `publicKeys`
* Filesystem database: Creates `genesis.go`.
This file defines the following public functions:
- `GenesisValidatorsRoot`
- `SaveGenesisValidatorsRoot`
* Filesystem database: Creates `graffiti.go`.
This file defines the following public functions:
- `SaveGraffitiOrderedIndex`
- `GraffitiOrderedIndex`
* Filesystem database: Creates `migration.go`.
This file defines the following public functions:
- `RunUpMigrations`
- `RunDownMigrations`
* Filesystem database: Creates proposer_settings.go.
This file defines the following public functions:
- `ProposerSettings`
- `ProposerSettingsExists`
- `SaveProposerSettings`
* Filesystem database: Creates `attester_protection.go`.
This file defines the following public functions:
- `EIPImportBlacklistedPublicKeys`
- `SaveEIPImportBlacklistedPublicKeys`
- `SigningRootAtTargetEpoch`
- `LowestSignedTargetEpoch`
- `LowestSignedSourceEpoch`
- `AttestedPublicKeys`
- `CheckSlashableAttestation`
- `SaveAttestationForPubKey`
- `SaveAttestationsForPubKey`
- `AttestationHistoryForPubKey`
* Filesystem database: Creates `proposer_protection.go`.
This file defines the following public functions:
- `HighestSignedProposal`
- `LowestSignedProposal`
- `ProposalHistoryForPubKey`
- `ProposalHistoryForSlot`
- `ProposedPublicKeys`
* Ensure that the filesystem store implements the `ValidatorDB` interface.
* `slashableAttestationCheck`: Check the database type.
* `slashableProposalCheck`: Check the database type.
* `slashableAttestationCheck`: Allow usage of minimal slashing protection.
* `slashableProposalCheck`: Allow usage of minimal slashing protection.
* `ImportStandardProtectionJSON`: Check the database type.
* `ImportStandardProtectionJSON`: Allow usage of min slashing protection.
* Implement `RecursiveDirFind`.
* Implement minimal<->complete DB conversion.
3 public functions are implemented:
- `IsCompleteDatabaseExisting`
- `IsMinimalDatabaseExisting`
- `ConvertDatabase`
* `setupDB`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/node/node_test.go`.
The rest of this commit consists in minimal slashing protection testing.
* `setupWithKey`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/client/propose_test.go`.
The rest of this commit consists in tests wrapping.
* `setup`: Add `isSlashingProtectionMinimal` argument.
The added feature is located in the `validator/client/propose_test.go`
file.
The rest of this commit consists in tests wrapping.
* `initializeFromCLI` and `initializeForWeb`: Factorize db init.
* Add `convert-complete-to-minimal` command.
* Creates `--enable-minimal-slashing-protection` flag.
* `importSlashingProtectionJSON`: Check database type.
* `exportSlashingProtectionJSON`: Check database type.
* `TestClearDB`: Test with minimal slashing protection.
* KeyManager: Test with minimal slashing protection.
* RPC: KeyManager: Test with minimal slashing protection.
* `convert-complete-to-minimal`: Change option names.
Options were:
- `--source` (for source data directory), and
- `--target` (for target data directory)
However, since this command deals with slashing protection, which has
source (epochs) and target (epochs), the initial option names may confuse
the user.
In this commit:
`--source` ==> `--source-data-dir`
`--target` ==> `--target-data-dir`
* Set `SlashableAttestationCheck` as an iface method.
And delete `CheckSlashableAttestation` from iface.
* Move helpers functions in a more general directory.
No functional change.
* Extract common structs out of `kv`.
==> `filesystem` does not depend anymore on `kv`.
==> `iface` does not depend anymore on `kv`.
==> `slashing-protection` does not depend anymore on `kv`.
* Move `ValidateMetadata` in `validator/helpers`.
* `ValidateMetadata`: Test with mock.
This way, we can:
- Avoid any circular import for tests.
- Implement once for all `iface.ValidatorDB` implementations
the `ValidateMetadata`function.
- Have tests (and coverage) of `ValidateMetadata`in
its own package.
The ideal solution would have been to implement `ValidateMetadata` as
a method with the `iface.ValidatorDB`receiver.
Unfortunately, golang does not allow that.
* `iface.ValidatorDB`: Implement ImportStandardProtectionJSON.
The whole purpose of this commit is to avoid the `switch validatorDB.(type)`
in `ImportStandardProtectionJSON`.
* `iface.ValidatorDB`: Implement `SlashableProposalCheck`.
* Remove now useless `slashableProposalCheck`.
* Delete useless `ImportStandardProtectionJSON`.
* `file.Exists`: Detect directories and return an error.
Before, `Exists` was only able to detect if a file exists.
Now, this function takes an extra `File` or `Directory` argument.
It detects either if a file or a directory exists.
Before, if an error was returned by `os.Stat`, the the file was
considered as non existing.
Now, it is treated as a real error.
* Replace `os.Stat` by `file.Exists`.
* Remove `Is{Complete,Minimal}DatabaseExisting`.
* `publicKeys`: Add log if unexpected file found.
* Move `{Source,Target}DataDirFlag`in `db.go`.
* `failedAttLocalProtectionErr`: `var`==> `const`
* `signingRoot`: `32`==> `fieldparams.RootLength`.
* `validatorClientData`==> `validator-client-data`.
To be consistent with `slashing-protection`.
* Add progress bars for `import` and `convert`.
* `parseBlocksForUniquePublicKeys`: Move in `db/kv`.
* helpers: Remove unused `initializeProgressBar` function.
When a late block arrives and the beacon is proposing the next block, we
perform several checks to allow for the next block to reorg the incoming
late block.
Among those checks, we check that the parent block has been heavily
attested (currently 160% of the committee size).
We perform this check in these circumstances:
- When the late block arrives
- At 10 seconds into the slot
- At 0 seconds into the next slot (at proposing time)
The problem is that for blocks that arrive between 4 seconds and 10
seconds, the parent block will not have yet this expected weight since
attestations from the current committee were not imported yet, and thus
Prysm will send an FCU with payload attributes anyway at this time.
What happens is that Prysm keeps the EL building different blocks based
on different parents at the same time, when later in the next slot it
calls to propose, it will reorg the late block anyway and the EL would
have been computing a second payload uselessly.
This PR enables this check only when calling `ShouldOverrideFCU` after
10 seconds into the slot which we do only after having imported the
current attestations. We may want to actually remove this check entirely
from `ShouldOverrideFCU` and only keep it in `ProposerHead`.
Shout out to Anthithesis for reporting an issue that led to this
discoverly.
* employ a val reader to prevent constant copies
* clean it up and fix tests
* gaz
* radek's review
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Improve package documentation.
* `processAttestations`: Improve logging.
* Add `Benchmark_checkSurroundVotes` benchmark.
* Implement `saveChunksToDisk` as remplacement of `saveUpdatedChunks`.
The idea is to open only on DB transaction for all validator chunk indexes instead of
one DB transaction per validator chunk index.
It saves the overhead due to transaction start/stop of the DB.
Result of `Benchmark_checkSurroundVotes`:
- Before this commit: 133 seconds
- After this commit: 5.05 seconds
* `LoadSlasherChunks` and `SaveSlasherChunks`: Batch.
* `loadChunks` ==> `loadChunksFromDisk`
* `updatedChunkByChunkIndex`: Don't update if `latestEpochWritten == currentEpoch `.
* `updatedChunkByChunkIndex`: Load all needed chunks once.
* `latestEpochWritten` ==> `latestEpochUpdated`.
* `checkSurroundVotes`: Dump to disk at most every `25_600` chunks.
* `SaveAttestationRecordsForValidators`: Batch.
* `batchSize`: Use as package const and add comment.
2024-02-21 15:12:37 +00:00
1755 changed files with 129297 additions and 73435 deletions
@@ -6,6 +6,9 @@ Excited by our work and want to get involved in building out our sharding releas
You can explore our [Open Issues](https://github.com/prysmaticlabs/prysm/issues) in-the works for our different releases. Feel free to fork our repo and start creating PR’s after assigning yourself to an issue of interest. We are always chatting on [Discord](https://discord.gg/CTYGPUJ) drop us a line there if you want to get more involved or have any questions on our implementation!
> [!IMPORTANT]
> Please, **do not send pull requests for trivial changes**, such as typos, these will be rejected. These types of pull requests incur a cost to reviewers and do not provide much value to the project. If you are unsure, please open an issue first to discuss the change.
## Contribution Steps
**1. Set up Prysm following the instructions in README.md.**
Navigate to your fork of the repo on GitHub. On the upper left where the current branch is listed, change the branch to your feature-in-progress-branch. Open the files that you have worked on and check to make sure they include your changes.
**16. Create a pull request.**
**16. Add an entry to CHANGELOG.md.**
Navigate your browser to https://github.com/prysmaticlabs/prysm and click on the new pull request button. In the “base” box on the left, leave the default selection “base master”, the branch that you want your changes to be applied to. In the “compare” box on the right, select feature-in-progress-branch, the branch containing the changes you want to apply. You will then be asked to answer a few questions about your pull request. After you complete the questionnaire, the pull request will appear in the list of pull requests at https://github.com/prysmaticlabs/prysm/pulls.
If your change is user facing, you must include a CHANGELOG.md entry. See the [Maintaining CHANGELOG.md](#maintaining-changelogmd) section for more information.
**17. Respond to comments by Core Contributors.**
**17. Create a pull request.**
Navigate your browser to https://github.com/prysmaticlabs/prysm and click on the new pull request button. In the “base” box on the left, leave the default selection “base master”, the branch that you want your changes to be applied to. In the “compare” box on the right, select feature-in-progress-branch, the branch containing the changes you want to apply. You will then be asked to answer a few questions about your pull request. After you complete the questionnaire, the pull request will appear in the list of pull requests at https://github.com/prysmaticlabs/prysm/pulls. Ensure that you have added an entry to CHANGELOG.md if your PR is a user-facing change. See the [Maintaining CHANGELOG.md](#maintaining-changelogmd) section for more information.
**18. Respond to comments by Core Contributors.**
Core Contributors may ask questions and request that you make edits. If you set notifications at the top of the page to “not watching,” you will still be notified by email whenever someone comments on the page of a pull request you have created. If you are asked to modify your pull request, repeat steps 8 through 15, then leave a comment to notify the Core Contributors that the pull request is ready for further review.
**18. If the number of commits becomes excessive, you may be asked to squash your commits.**
**19. If the number of commits becomes excessive, you may be asked to squash your commits.**
You can do this with an interactive rebase. Start by running the following command to determine the commit that is the base of your branch...
@@ -136,7 +143,7 @@ Core Contributors may ask questions and request that you make edits. If you set
**19. The previous command will return a commit-hash that you should use in the following command.**
**20. The previous command will return a commit-hash that you should use in the following command.**
```
$ git rebase -i commit-hash
@@ -160,13 +167,30 @@ squash hash add a feature
Save and close the file, then a commit command will appear in the terminal that squashes the smaller commits into one. Check to be sure the commit message accurately reflects your changes and then hit enter to execute it.
**20. Update your pull request with the following command.**
**21. Update your pull request with the following command.**
```
$ git push myrepo feature-in-progress-branch -f
```
**21. Finally, again leave a comment to the Core Contributors on the pull request to let them know that the pull request has been updated.**
**22. Finally, again leave a comment to the Core Contributors on the pull request to let them know that the pull request has been updated.**
## Maintaining CHANGELOG.md
This project follows the changelog guidelines from [keepachangelog.com](https://keepachangelog.com/en/1.1.0/).
All PRs with user facing changes should have an entry in the CHANGELOG.md file and the change should be categorized in the appropriate category within the "Unreleased" section. The categories are:
-`Added` for new features.
-`Changed` for changes in existing functionality.
-`Deprecated` for soon-to-be removed features.
-`Removed` for now removed features.
-`Fixed` for any bug fixes.
-`Security` in case of vulnerabilities. Please see the [Security Policy](SECURITY.md) for responsible disclosure before adding a change with this category.
### Releasing
When a new release is made, the "Unreleased" section should be moved to a new section with the release version and the current date. Then a new "Unreleased" section is made at the top of the file with the categories listed above.
// NextSyncCommittee should be be empty after `ProcessSyncCommitteeUpdates`. Validator should get indices.
// NextSyncCommittee should be empty after `ProcessSyncCommitteeUpdates`. Validator should get indices.
require.NotEqual(t,0,len(indices))
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
