* Avoid unnecessary calls to ExitInformation()
ExitInformation runs a loop over the whole validator set. This is needed
in case that there are slashings or exits to be processed in a block (we
could be caching or avoid this entirely post-Electra though). This PR
removes these calls on normal state transition to this function. h/t to
@terencechain for finding out this bug.
In addition, on processing withdrawal requests and registry updates, we
kept recomputing the exit information at the same time that the state is
updated and the function that updates the state already takes care of
tracking and updating the right exit information. So this PR removes the
calls to compute this exit information on a loop. Notice that this bug
has been present even before we had a function `ExitInformation()` so I
will document here to help the reviewer
Our previous behavior is to do this in a loop
```
st, err = validators.InitiateValidatorExit(ctx, st, vIdx, validators.ExitInformation(st))
```
This is a bit problematic since `ExitInformation` loops over the whole validator set to compute the exit information (and the total active balance) and then the function `InitiateValidatorExit` actually recomputes the total active balance looping again over the whole validator set and overwriting the pointer returned by `ExitInformation`.
On the other hand, the funciton `InitiateValidatorExit` does mutate the state `st` itself. So each call to `ExitInformation(st)` may actually return a different pointer.
The function ExitInformation computes as follows
```
err := s.ReadFromEveryValidator(func(idx int, val state.ReadOnlyValidator) error {
e := val.ExitEpoch()
if e != farFutureEpoch {
if e > exitInfo.HighestExitEpoch {
exitInfo.HighestExitEpoch = e
exitInfo.Churn = 1
} else if e == exitInfo.HighestExitEpoch {
exitInfo.Churn++
}
```
So it simply increases the churn for each validator that has epoch equal to the highest exit epoch.
The function `InitiateValidatorExit` mutates this pointer in the following way
if the state is post-electra, it disregards completely this pointer and computes the highest exit epoch and updates churn inconditionally, so the pointer `exitInfo.HighestExitEpoch` will always have the right value and is not even neded to be computed before. We could even avoid the fist loop even. If the state is pre-Electra then the function itself updates correctly the exit info for the next iteration.
* Only care about exits pre-Electra
* Update beacon-chain/core/transition/transition_no_verify_sig.go
Co-authored-by: terence <terence@prysmaticlabs.com>
* Radek's review
---------
Co-authored-by: terence <terence@prysmaticlabs.com>
* attempting to improve duties v2
* removing go routine
* changelog
* unnessesary variable
* fixing test
* small optimization existing early on CommitteeAssignments function
* fixing small bug
* fixes performance issues with duties v2
* fixed changelog
* gofmt
* Sort sidecars by index before calling `RecoverCellsAndKZGProofs`.
Reason: Starting at `c-kzg-4844 v2.1.2`, the library needs input to be sorted.
* Update `c-kzg-4844` to `v2.1.3`
* Update `c-kzg-4844` to `v2.1.5`
* additional log information around invalid payloads
* fix test with reversed require.ErrorIs args
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* slasherkv: Set a 1 minute timeout on PruneAttestationOnEpoch operations to prevent very large bolt transactions.
* Fix CI
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Change wrap message to avoid the could not...: could not...: could not... effect.
Reference: https://github.com/uber-go/guide/blob/master/style.md#error-wrapping.
* Log: remove period at the end of the latest sentence.
* Dirty quick fix to ensure that the custody group count is set at P2P service start.
A real fix would involve a chan implement a proper synchronization scheme.
* Add changelog.
* Support Fulu genesis block
* `NewGenesisBlockForState`: Factorize Electra and Fulu, which share the same `BeaconBlock`.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* `Broadcasted data column sidecar` log: Add `blobCount`.
* `broadcastAndReceiveDataColumns`: Broadcast and receive data columns in parallel.
* `ProposeBeaconBlock`: First broadcast/receive block, and then sidecars.
* `broadcastReceiveBlock`: Add log.
* Add changelog
* Fix deadlock-option 1.
* Fix deadlock-option 2.
* Take notifier out of the critical section
* only compute common info once, for all sidecars
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Fix misleading log msg on shutdown
gRPCServer.GracefulStop blocks until it has been shutdown. Logging
"Initiated graceful stop" after it has been completed is misleading.
Names are added to the message to discern services. Also, a minimum test
is added mainly to verify the change made with this commit.
* Add changelog fragment file
* Capitalize log messages
* Update endtoend test for fixed log messages
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* `reconstructSaveBroadcastDataColumnSidecars`: Use the `s.columnIndicesToSample` instead of recoding its content.
ddd
* Rename `custodyColumns` ==> `columnIndicesToSample`.
* `DataColumnStorage.Save`: Remove wrong godoc.
* Implement `receiveDataColumnSidecars` and transform `receiveDataColumnSidecar` as a subcase of the plural version.
* `dataColumnSubscriber`: Add godoc and remove only once used variable.
* `processDataColumnSidecarsFromExecution`: Use single flight directly in the function.
So the caller does not have any more the responsability to deal with multiple simultaneous calls.
* `processDataColumnSidecarsFromReconstruction`: Guard against a single flight.
In `dataColumnSubscriber`, trig in parallel `processDataColumnSidecarsFromReconstruction` and `processDataColumnSidecarsFromExecution`.
Stop when the first of them is successful.
* `processDataColumnSidecarsFromExecution`: Use `receiveDataColumnSidecars` instead of `receiveDataColumnSidecar`.
* Implement and use `broadcastAndReceiveUnseenDataColumnSidecars`.
* Add changelog.
* Fix James' comment.
* Fix James' comment.
* `processDataColumnSidecarsFromReconstruction`: Log reconstruction duration.
* Change InsertChain
InsertChain uses `ROBlock` since #14571, this allows it to insert the
last block of the chain as well. We change the semantics of InsertChain
to include all blocks and take them in increasing order.
* Fix tests
* Use slices.Reverse
* PeerDAS: Implement sync
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Implement `TestFetchDataColumnSidecarsFromPeers`.
* Implement `TestSelectPeers`.
* Fix James' comment.
* Fix flakiness in `TestSelectPeers`.
* Revert "Fix Potuz's comment."
This reverts commit c45230b455.
* Revert "Fix James' comment."
This reverts commit a3f919205a.
* `selectPeers`: Avoid map with key but empty value.
* Fix Potuz's comment.
* Add DataColumnStorage and SubscribeAllDataSubnets flag.
* getBlobsV2: retry if reconstruction isnt successful
* test: engine client and sync package, metrics
* lint: fmt and log capitalisation
* lint: return error when it is not nil
* config: make retry interval configurable
* sidecar: recover function and different context for retrying
* lint: remove unused field
* beacon: default retry interval
* reconstruct: load once, correctly deliver the result to all waiting goroutines
* reconstruct: simplify multi goroutine case and avoid race condition
* engine: remove isDataAlreadyAvailable function
* sync: no goroutine, getblobsv2 in absence of block as well, wrap error
* exec: hardcode retry interval
* da: non blocking checks
* sync: remove unwanted checks
* execution: fix test
* execution: retry atomicity test
* da: updated IsDataAvailable
* sync: remove unwanted tests
* bazel: bazel run //:gazelle -- fix
* blockchain: fix CustodyGroupCount return
* lint: formatting
* lint: lint and use unused metrics
* execution: retry logic inside ReconstructDataColumnSidecars itself
* lint: format
* execution: ensure the retry actually happens when it needs to
* execution: ensure single responsibility, execution should not do DA check
* sync: don't call ReconstructDataColumnSidecars if not required
* blockchain: move IsDataAvailable interface to blockchain package
* execution: make reconstructSingleflight part of the service struct
* blockchain: cleaner DA check
* lint: formatting and remove confusing comment
* sync: fix lint, test and add extra test for when data is actually not available
* sync: new appropriate mock service
* execution: edge case - delete activeRetries on success
* execution: use service context instead of function's for retry
* blockchain: get variable samplesPerSlot only when required
* remove redundant function and fix name
* fix test
* fix more tests
* put samplesPerSlot at appropriate place
* tidy up IsDataAvailable
* correct bad merge
* fix bad merge
* remove redundant flag option
* refactor to deduplicate sidecar construction code
* - Add godocs
- Rename some functions to be closer to the spec
- Add err in return of commitments
* Replace mutating public method (but only internally used) `Populate` but private not mutating method `extract`.
* Implement a unique `processDataColumnSidecarsFromExecution` instead 2 separate functions from block and from sidecar.
* `ReceiveBlock`: Wrap errors.
* Remove useless tests.
* `ConstructionPopulator`: Add tests.
* Fix tests
* Move functions to be consistent with blobs.
* `fetchCellsAndProofsFromExecution`: Avoid useless flattening.
* `processDataColumnSidecarsFromExecution`: Stop using DB cache.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* create lc cache to track branches
* save lc stuff
* remove finalized data from LC cache on finalization
* read lc stuff
* edit tests
* changelog
* linter
* address commments
* address commments 2
* address commments 3
* address commments 4
* lint
* address commments 5 x_x
* set beacon lcStore to mimick registrable services
* clean up the error propagation
* pass the state to saveLCBootstrap since it's not saved in db yet
* remove "experimental" from backfill flag name
* backwards-compatible alias
* changelog
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `computeIndicesByRootByPeer`: Add 1 slack epoch regarding peer head slot.
* `FetchDataColumnSidecars`: Switch mode.
Before this commit, this function returned on error as long as at least ONE requested sidecar was not retrieved.
Now, this function retrieves what it can (best effort mode) and returns an additional value which is the map of missing sidecars after running this function.
It is now the role of the caller to check this extra returned value and decide what to do in case some requested sidecars are still missing.
* `fetchOriginDataColumnSidecars`: Optimize
Before this commit, when running `fetchOriginDataColumnSidecars`, all the missing sidecars had to been retrieved in a single shot for the sidecars to be considered as available. The issue was, if for example `sync.FetchDataColumnSidecars` returned all but one sidecar, the returned sidecars were NOT saved, and on the next iteration, all the previously fetched sidecars had to be requested again (from peers.)
After this commit, we greedily save all fetched sidecars, solving this issue.
* Initial sync: Do not fetch data column sidecars before the retention period.
* Implement perfect peerdas syncing.
* Add changelog.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Fix James' comment.
* Update beacon-chain/sync/data_column_sidecars.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Update beacon-chain/sync/data_column_sidecars.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Update beacon-chain/sync/data_column_sidecars.go
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Update after Potuz's comment.
* Fix Potuz's commit.
* Fix James' comment.
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
* Add flag for colocation whitelisting. --p2p-ip-colocation-whitelist
This change updates the peer IP colocation checking to respect the
configured CIDR whitelist (--p2p-ip-colocation-whitelist flag).
Changes:
- Added IPColocationWhitelist field to peers.StatusConfig
- Added ipColocationWhitelist field to Status struct to store parsed IPNets
- Parse CIDR strings into net.IPNet in NewStatus constructor
- Updated isfromBadIP method to skip colocation limits for whitelisted IPs
- Pass IPColocationWhitelist from Service config when creating Status
The IP colocation whitelist allows operators to exempt specific IP ranges
from the colocation limit, useful for deployments with known trusted
address ranges or legitimate node clustering.
Only check if an IP is in the whitelist when the colocation limit
is actually exceeded, rather than checking for every IP. This is
more efficient and matches the intended behavior.
* Changelog fragment
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* @kasey feedback: Move IP colocation parsing to the node construction
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Implement KZG proof batch verification for data column gossip validation
* Manu's feedback
* Add tests
* Update beacon-chain/sync/batch_verifier.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/batch_verifier.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Fix tests
* Kasey's feedback
* `validateWithKzgBatchVerifier`: Give up after a full slot.
Before this commit:
After 100 ms, an un-batched verification is launched concurrently to the batched one.
As a result, a stressed node could start to be even more stressed by the multiple verifications.
Also, it is always hard to choose a correct timeout value.
100ms may be OK for a given node with a given BPO version, and not ok for the same node with a BPO version with 10x more blobs.
However, we know this gossip validation won't be useful after a full slot duration.
After this commit:
After a full slot duration, we just ignore the incoming gossip message.
It's important to ignore it and not to reject it, since rejecting it would downscore the peer sending this message.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Calculate max epoch and churn for slashing once
* calculate once for proposer and attester slashings
* changelog <3
* introduce struct
* check if err is nil in ProcessVoluntaryExits
* rename exitData to exitInfo and return from functions
* cleanup + tests
* cleanup after rebase
* Potuz's review
* pre-calculate total active balance
* remove `slashValidatorFunc` closure
* Avoid a second validator loop
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* remove balance parameter from slashing functions
---------
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: potuz <potuz@prysmaticlabs.com>
When peers return invalid data during initial sync, log the specific
validation failure reason. This helps identify:
- Whether peer exceeded requested block count
- Whether peer exceeded MAX_REQUEST_BLOCKS protocol limit
- Whether blocks are outside the requested slot range
- Whether blocks are out of order (not increasing or wrong step)
Each log includes the specific condition that failed, making it easier
to debug whether the issue is with peer implementations or request
validation logic.
* Subscription: Get fanout peers in all data column subnets when at least one validator is connected
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Updated test structure to @nalepae suggestion
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
