* `Broadcasted data column sidecar` log: Add `blobCount`.
* `broadcastAndReceiveDataColumns`: Broadcast and receive data columns in parallel.
* `ProposeBeaconBlock`: First broadcast/receive block, and then sidecars.
* `broadcastReceiveBlock`: Add log.
* Add changelog
* Fix deadlock-option 1.
* Fix deadlock-option 2.
* Take notifier out of the critical section
* only compute common info once, for all sidecars
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Add bitvector field for FixedTestContainer
* Handle Bitvector type using isBitfield flag
* Add Bitvector case for Stringify
* Add bitlist field for VariableTestContainer
* Add bitlistInfo
* Changelog
* Add bitvectorInfo
* Remove analyzeBit* functions and just inline them
* Fix misleading comments
* Add comments for bitlistInfo's Size
* Apply reviews from Radek
* Fix misleading log msg on shutdown
gRPCServer.GracefulStop blocks until it has been shutdown. Logging
"Initiated graceful stop" after it has been completed is misleading.
Names are added to the message to discern services. Also, a minimum test
is added mainly to verify the change made with this commit.
* Add changelog fragment file
* Capitalize log messages
* Update endtoend test for fixed log messages
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* `reconstructSaveBroadcastDataColumnSidecars`: Use the `s.columnIndicesToSample` instead of recoding its content.
ddd
* Rename `custodyColumns` ==> `columnIndicesToSample`.
* `DataColumnStorage.Save`: Remove wrong godoc.
* Implement `receiveDataColumnSidecars` and transform `receiveDataColumnSidecar` as a subcase of the plural version.
* `dataColumnSubscriber`: Add godoc and remove only once used variable.
* `processDataColumnSidecarsFromExecution`: Use single flight directly in the function.
So the caller does not have any more the responsability to deal with multiple simultaneous calls.
* `processDataColumnSidecarsFromReconstruction`: Guard against a single flight.
In `dataColumnSubscriber`, trig in parallel `processDataColumnSidecarsFromReconstruction` and `processDataColumnSidecarsFromExecution`.
Stop when the first of them is successful.
* `processDataColumnSidecarsFromExecution`: Use `receiveDataColumnSidecars` instead of `receiveDataColumnSidecar`.
* Implement and use `broadcastAndReceiveUnseenDataColumnSidecars`.
* Add changelog.
* Fix James' comment.
* Fix James' comment.
* `processDataColumnSidecarsFromReconstruction`: Log reconstruction duration.
* fix: use assigned committee index in attestation data after Electra
* update change log
* update add unit test
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Change InsertChain
InsertChain uses `ROBlock` since #14571, this allows it to insert the
last block of the chain as well. We change the semantics of InsertChain
to include all blocks and take them in increasing order.
* Fix tests
* Use slices.Reverse
* PeerDAS: Implement sync
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Implement `TestFetchDataColumnSidecarsFromPeers`.
* Implement `TestSelectPeers`.
* Fix James' comment.
* Fix flakiness in `TestSelectPeers`.
* Revert "Fix Potuz's comment."
This reverts commit c45230b455.
* Revert "Fix James' comment."
This reverts commit a3f919205a.
* `selectPeers`: Avoid map with key but empty value.
* Fix Potuz's comment.
* Add DataColumnStorage and SubscribeAllDataSubnets flag.
* getBlobsV2: retry if reconstruction isnt successful
* test: engine client and sync package, metrics
* lint: fmt and log capitalisation
* lint: return error when it is not nil
* config: make retry interval configurable
* sidecar: recover function and different context for retrying
* lint: remove unused field
* beacon: default retry interval
* reconstruct: load once, correctly deliver the result to all waiting goroutines
* reconstruct: simplify multi goroutine case and avoid race condition
* engine: remove isDataAlreadyAvailable function
* sync: no goroutine, getblobsv2 in absence of block as well, wrap error
* exec: hardcode retry interval
* da: non blocking checks
* sync: remove unwanted checks
* execution: fix test
* execution: retry atomicity test
* da: updated IsDataAvailable
* sync: remove unwanted tests
* bazel: bazel run //:gazelle -- fix
* blockchain: fix CustodyGroupCount return
* lint: formatting
* lint: lint and use unused metrics
* execution: retry logic inside ReconstructDataColumnSidecars itself
* lint: format
* execution: ensure the retry actually happens when it needs to
* execution: ensure single responsibility, execution should not do DA check
* sync: don't call ReconstructDataColumnSidecars if not required
* blockchain: move IsDataAvailable interface to blockchain package
* execution: make reconstructSingleflight part of the service struct
* blockchain: cleaner DA check
* lint: formatting and remove confusing comment
* sync: fix lint, test and add extra test for when data is actually not available
* sync: new appropriate mock service
* execution: edge case - delete activeRetries on success
* execution: use service context instead of function's for retry
* blockchain: get variable samplesPerSlot only when required
* remove redundant function and fix name
* fix test
* fix more tests
* put samplesPerSlot at appropriate place
* tidy up IsDataAvailable
* correct bad merge
* fix bad merge
* remove redundant flag option
* refactor to deduplicate sidecar construction code
* - Add godocs
- Rename some functions to be closer to the spec
- Add err in return of commitments
* Replace mutating public method (but only internally used) `Populate` but private not mutating method `extract`.
* Implement a unique `processDataColumnSidecarsFromExecution` instead 2 separate functions from block and from sidecar.
* `ReceiveBlock`: Wrap errors.
* Remove useless tests.
* `ConstructionPopulator`: Add tests.
* Fix tests
* Move functions to be consistent with blobs.
* `fetchCellsAndProofsFromExecution`: Avoid useless flattening.
* `processDataColumnSidecarsFromExecution`: Stop using DB cache.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* create lc cache to track branches
* save lc stuff
* remove finalized data from LC cache on finalization
* read lc stuff
* edit tests
* changelog
* linter
* address commments
* address commments 2
* address commments 3
* address commments 4
* lint
* address commments 5 x_x
* set beacon lcStore to mimick registrable services
* clean up the error propagation
* pass the state to saveLCBootstrap since it's not saved in db yet
* remove "experimental" from backfill flag name
* backwards-compatible alias
* changelog
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `computeIndicesByRootByPeer`: Add 1 slack epoch regarding peer head slot.
* `FetchDataColumnSidecars`: Switch mode.
Before this commit, this function returned on error as long as at least ONE requested sidecar was not retrieved.
Now, this function retrieves what it can (best effort mode) and returns an additional value which is the map of missing sidecars after running this function.
It is now the role of the caller to check this extra returned value and decide what to do in case some requested sidecars are still missing.
* `fetchOriginDataColumnSidecars`: Optimize
Before this commit, when running `fetchOriginDataColumnSidecars`, all the missing sidecars had to been retrieved in a single shot for the sidecars to be considered as available. The issue was, if for example `sync.FetchDataColumnSidecars` returned all but one sidecar, the returned sidecars were NOT saved, and on the next iteration, all the previously fetched sidecars had to be requested again (from peers.)
After this commit, we greedily save all fetched sidecars, solving this issue.
* Initial sync: Do not fetch data column sidecars before the retention period.
* Implement perfect peerdas syncing.
* Add changelog.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Update beacon-chain/sync/data_column_sidecars.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Update beacon-chain/sync/data_column_sidecars.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Update beacon-chain/sync/data_column_sidecars.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Update after Potuz's comment.
* Fix Potuz's commit.
* Fix James' comment.
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Add flag for colocation whitelisting. --p2p-ip-colocation-whitelist
This change updates the peer IP colocation checking to respect the
configured CIDR whitelist (--p2p-ip-colocation-whitelist flag).
Changes:
- Added IPColocationWhitelist field to peers.StatusConfig
- Added ipColocationWhitelist field to Status struct to store parsed IPNets
- Parse CIDR strings into net.IPNet in NewStatus constructor
- Updated isfromBadIP method to skip colocation limits for whitelisted IPs
- Pass IPColocationWhitelist from Service config when creating Status
The IP colocation whitelist allows operators to exempt specific IP ranges
from the colocation limit, useful for deployments with known trusted
address ranges or legitimate node clustering.
Only check if an IP is in the whitelist when the colocation limit
is actually exceeded, rather than checking for every IP. This is
more efficient and matches the intended behavior.
* Changelog fragment
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* @kasey feedback: Move IP colocation parsing to the node construction
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Implement KZG proof batch verification for data column gossip validation
* Manu's feedback
* Add tests
* Update beacon-chain/sync/batch_verifier.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/batch_verifier.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Fix tests
* Kasey's feedback
* `validateWithKzgBatchVerifier`: Give up after a full slot.
Before this commit:
After 100 ms, an un-batched verification is launched concurrently to the batched one.
As a result, a stressed node could start to be even more stressed by the multiple verifications.
Also, it is always hard to choose a correct timeout value.
100ms may be OK for a given node with a given BPO version, and not ok for the same node with a BPO version with 10x more blobs.
However, we know this gossip validation won't be useful after a full slot duration.
After this commit:
After a full slot duration, we just ignore the incoming gossip message.
It's important to ignore it and not to reject it, since rejecting it would downscore the peer sending this message.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Calculate max epoch and churn for slashing once
* calculate once for proposer and attester slashings
* changelog <3
* introduce struct
* check if err is nil in ProcessVoluntaryExits
* rename exitData to exitInfo and return from functions
* cleanup + tests
* cleanup after rebase
* Potuz's review
* pre-calculate total active balance
* remove `slashValidatorFunc` closure
* Avoid a second validator loop
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* remove balance parameter from slashing functions
---------
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: potuz <potuz@prysmaticlabs.com>
* adding web 3 signer changes for fulu
* missed adding fulu values
* add accounting for fulu version
* updated web3signer version and fixing linting
* Update validator/keymanager/remote-web3signer/types/requests_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update validator/keymanager/remote-web3signer/types/mock/mocks.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* radek suggestions
* removing redundant check and removing old function, changed changelog to reflect
* gaz
* radek's suggestion
* fixing test as v1 proof is no longer used
* fixing more tests
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
When peers return invalid data during initial sync, log the specific
validation failure reason. This helps identify:
- Whether peer exceeded requested block count
- Whether peer exceeded MAX_REQUEST_BLOCKS protocol limit
- Whether blocks are outside the requested slot range
- Whether blocks are out of order (not increasing or wrong step)
Each log includes the specific condition that failed, making it easier
to debug whether the issue is with peer implementations or request
validation logic.
* Add vectorInfo
* Add 2D bytes field for test
* Add tag_parser for parsing SSZ tags
* Integrate tag parser with analyzer
* Add ByteList test case
* Changelog
* Better printing feature with Stringer implementation
* Return error for non-determined case without printing other values
* Update tag_parser.go: handle Vector and List mutually exclusive (inspired by OffchainLabs/fastssz)
* Make linter happy
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Subscription: Get fanout peers in all data column subnets when at least one validator is connected
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Updated test structure to @nalepae suggestion
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Add VariableTestContainer in ssz_query.proto
* Add listInfo
* Use errors.New for making an error with a static string literal
* Add listInfo field when analyzing the List type
* Persist the field order in the container
* Add actualOffset and goFieldName at fieldInfo
* Add PopulateFromValue function & update test runner
* Handle slice of ssz object for marshalling
* Add CalculateOffsetAndLength test
* Add comments for better doc
* Changelog :)
* Apply reviews from Radek
* Remove actualOffset and update offset field instead
* Add Nested container of variable-sized for testing nested path
* Fix offset adding logics: for variable-sized field, always add 4 instead of its fixed size
* Fix multiple import issue
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
