* Use head for block validation when possible
When validating blocks for pubsub, we always copy a state and advance
when we simply need to get a read only beacon state without a copy in
most cases since the head state normally works.
* fix test
* fix tests
* fix more tests
* fix more tests
* Add nil check to be safe
* fix more tests
* add test case
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Set default value of `--blob-batch-limit` to 384.
So, using default values, `--blob-batch-limit * --blob-batch-limit-burst-factor = 384*3 = MAX_REQUEST_BLOB_SIDECARS = 1152.`
* `blobSidecarByRootRPCHandler`: Add rate limiting.
Bacause now the rate limiter validation is done before the request validation,
adapt `TestBlobsByRootValidation` consequently and add new specific tests for
`validateBlobByRootRequest` to cover the now untested case.
* Set default value of `--data-column-batch-limit-burst-factor` to 4.
So, using default values, `--data-column-batch-limit * --data-column-batch-limit-burst-factor = 4096*2 = MAX_REQUEST_DATA_COLUMN_SIDECARS_ELECTRA = 16384`.
* `validateDataColumnsByRootRequest`: Take a count instead of idents.
* `dataColumnSidecarByRootRPCHandler`: Add rate limiting.
* `signatureData`: Add `string` function.
* `RODataColumnsVerifier.ValidProposerSignature`: Ensure the expensive signature verification is only performed once for concurrent requests for the same signature data.
Share flight group
* `parentState` ==> `state`.
* `RODataColumnsVerifier.SidecarProposerExpected: Ensure the expensive index computation is only performed once for concurrent requests.`
* Add `wrapAttestationError`
* Fix Kasey's comment.
* Fix Terence's comment.
* Implement `AvailableBlocks`.
* `blobSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.
* `dataColumnSidecarByRootRPCHandler`: Do not do extra work if only needed for TRACE logging.
* `TestDataColumnSidecarsByRootRPCHandler`: Re-arrange (no functional change).
* `TestDataColumnSidecarsByRootRPCHandler`: Save blocks corresponding to sidecars into DB.
* `dataColumnSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.
* Add changelog
* `TestDataColumnSidecarsByRootRPCHandler`: Use `assert` instead of `require` in goroutines.
https://github.com/stretchr/testify?tab=readme-ov-file#require-package
* Define TCP and QUIC as `InternetProtocol` (no functional change).
* Group types. (No functional changes)
* Rename variables and use range syntax.
* Add `p2pMaxPeers` and `p2pPeerCountDirectionType` metrics
* `p2p_subscribed_topic_peer_total`: Reset to avoid dangling values.
* `validateConfig`:
- Use `Warning` with fields instead of `Warnf`.
- Avoid to both modify in place the input value and return it.
* Add `p2p_minimum_peers_per_subnet` metric.
* `beaconConfig` => `cfg`.
https://github.com/OffchainLabs/prysm/pull/15880#discussion_r2436826215
* Add changelog
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Update Earliest available slot when pruning
* bazel run //:gazelle -- fix
* custodyUpdater interface to avoid import cycle
* bazel run //:gazelle -- fix
* simplify test
* separation of concerns
* debug log for updating eas
* UpdateEarliestAvailableSlot function in CustodyManager
* fix test
* UpdateEarliestAvailableSlot function for FakeP2P
* lint
* UpdateEarliestAvailableSlot instead of UpdateCustodyInfo + check for Fulu
* fix test and lint
* bugfix: enforce minimum retention period in pruner
* remove MinEpochsForBlockRequests function and use from config
* remove modifying earliest_available_slot after data column pruning
* correct earliestAvailableSlot validation: allow backfill decrease but prevent increase within MIN_EPOCHS_FOR_BLOCK_REQUESTS
* lint
* bazel run //:gazelle -- fix
* lint and remove unwanted debug logs
* Return a wrapped error, and let the caller decide what to do
* fix tests because updateEarliestSlot returns error now
* avoid re-doing computation in the test function
* lint and correct changelog
* custody updater should be a mandatory part of the pruner service
* ensure never increase eas if we are in the block requests window
* slot level granularity edge case
* update the value stored in the DB
* log tidy up
* use errNoCustodyInfo
* allow earliestAvailableSlot edit when custodyGroupCount doesnt change
* undo the minimal config change
* add context to CustodyGroupCount after merging from develop
* cosmetic change
* shift responsibility from caller to callee, protection for updateEarliestSlot. UpdateEarliestAvailableSlot returns cgc
* allow increase in earliestAvailableSlot only when custodyGroupCount also increases
* remove CustodyGroupCount as it is no longer needed as UpdateEarliestAvailableSlot returns cgc now
* proper place for log and name refactor
* test for Nil custody info
* allow decreasing earliest slot in DB (just like in memory)
* invert if statement to make more readable
* UpdateEarliestAvailableSlot for DB (equivalent of p2p's UpdateEarliestAvailableSlot) & undo changes made to UpdateCustodyInfo
* in UpdateEarliestAvailableSlot, no need to return unused values
* no need to log stored group count
* log.WithField instead of log.WithFields
* reject committee index >= committees_per_slot in unaggregated attestation validation
* Create phrwlk_fix-attestation-committee-index-bound.md
* add a unit test
* fix test
* fixing test
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>
* `TestDataColumnSidecarsByRangeRPCHandler`: Remove commented code.
* Remove double import
* `dataColumnSidecarsByRangeRPCHandler`: Gracefully close the stream if no data to return.
* Tests: Change `require` to `assert` in goroutines in tests.
https://pkg.go.dev/github.com/stretchr/testify/require#hdr-Assertions
* Add changelog.
* make registerSubscribers idempotent
* clean up debugging changes
* test fix
* rm unused var
* sobbing noises
* naming feedback and separate test for digestActionDone
* gazelle
* manu's feedback
* refactor to enable immediate sub after init sync
* preston comment re panic causing db corruption risk
* ensure we check that we're 1 epoch past the fork
* manu feedback
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `findPeersWithSubnets`: If the `filter` function returns an error for a given peer, log an error and skip the peer instead of aborting the whole function.
* `computeIndicesByRootByPeer`: If the loop returns an error for a given peer, log an error and skip the peer instead of aborting the whole function.
* Add changelog.
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* `buildStatusFromStream`: Use parent context.
* Status tests: Use `t.Context` everywhere.
* `buildStatusFromStream`: Respond statusV2 only if Fulu is enabled.
Without doing so, earliest available slot is never defined, and then `s.cfg.p2p.EarliestAvailableSlot` will block until the context is canceled.
* Send our real earliest available slot when sending a Status request post Fulu instead of `0`.
* Add changelog.
* Prettify logs for byRange/byRoot data column sidecar requests.
* Moving byRoot/byRange data column sidecars requests from peers to TRACE level.
* Move "Peer requested blob sidecar by root not found in db" in TRACE.
* Add changelog.
* Fix Kasey's comment.
* Apply Kasey's suggestion.
* Revert "`createLocalNode`: Wait before retrying to retrieve the custody group count if not present. (#15735)"
This reverts commit 4585cdc932.
* Revert "Fix no custody info available at start (#15732)"
This reverts commit 80eba4e6dd.
* Add context to `EarliestAvailableSlot` and `CustodyGroupCount` (no functional change).
* Remove double imports.
* `EarliestAvailableSlot` and `CustodyGroupCount`: Wait for custody info to be initialized.
* exclude unscheduled forks from the schedule
* update tests that relied on fulu being far future
* don't mess with the version->epoch mapping
* LastFork cheats the tests by filtering by far_future_epoch
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `reconstructSaveBroadcastDataColumnSidecars`: Use the `s.columnIndicesToSample` instead of recoding its content.
ddd
* Rename `custodyColumns` ==> `columnIndicesToSample`.
* `DataColumnStorage.Save`: Remove wrong godoc.
* Implement `receiveDataColumnSidecars` and transform `receiveDataColumnSidecar` as a subcase of the plural version.
* `dataColumnSubscriber`: Add godoc and remove only once used variable.
* `processDataColumnSidecarsFromExecution`: Use single flight directly in the function.
So the caller does not have any more the responsability to deal with multiple simultaneous calls.
* `processDataColumnSidecarsFromReconstruction`: Guard against a single flight.
In `dataColumnSubscriber`, trig in parallel `processDataColumnSidecarsFromReconstruction` and `processDataColumnSidecarsFromExecution`.
Stop when the first of them is successful.
* `processDataColumnSidecarsFromExecution`: Use `receiveDataColumnSidecars` instead of `receiveDataColumnSidecar`.
* Implement and use `broadcastAndReceiveUnseenDataColumnSidecars`.
* Add changelog.
* Fix James' comment.
* Fix James' comment.
* `processDataColumnSidecarsFromReconstruction`: Log reconstruction duration.
* PeerDAS: Implement sync
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Implement `TestFetchDataColumnSidecarsFromPeers`.
* Implement `TestSelectPeers`.
* Fix James' comment.
* Fix flakiness in `TestSelectPeers`.
* Revert "Fix Potuz's comment."
This reverts commit c45230b455.
* Revert "Fix James' comment."
This reverts commit a3f919205a.
* `selectPeers`: Avoid map with key but empty value.
* Fix Potuz's comment.
* Add DataColumnStorage and SubscribeAllDataSubnets flag.
* getBlobsV2: retry if reconstruction isnt successful
* test: engine client and sync package, metrics
* lint: fmt and log capitalisation
* lint: return error when it is not nil
* config: make retry interval configurable
* sidecar: recover function and different context for retrying
* lint: remove unused field
* beacon: default retry interval
* reconstruct: load once, correctly deliver the result to all waiting goroutines
* reconstruct: simplify multi goroutine case and avoid race condition
* engine: remove isDataAlreadyAvailable function
* sync: no goroutine, getblobsv2 in absence of block as well, wrap error
* exec: hardcode retry interval
* da: non blocking checks
* sync: remove unwanted checks
* execution: fix test
* execution: retry atomicity test
* da: updated IsDataAvailable
* sync: remove unwanted tests
* bazel: bazel run //:gazelle -- fix
* blockchain: fix CustodyGroupCount return
* lint: formatting
* lint: lint and use unused metrics
* execution: retry logic inside ReconstructDataColumnSidecars itself
* lint: format
* execution: ensure the retry actually happens when it needs to
* execution: ensure single responsibility, execution should not do DA check
* sync: don't call ReconstructDataColumnSidecars if not required
* blockchain: move IsDataAvailable interface to blockchain package
* execution: make reconstructSingleflight part of the service struct
* blockchain: cleaner DA check
* lint: formatting and remove confusing comment
* sync: fix lint, test and add extra test for when data is actually not available
* sync: new appropriate mock service
* execution: edge case - delete activeRetries on success
* execution: use service context instead of function's for retry
* blockchain: get variable samplesPerSlot only when required
* remove redundant function and fix name
* fix test
* fix more tests
* put samplesPerSlot at appropriate place
* tidy up IsDataAvailable
* correct bad merge
* fix bad merge
* remove redundant flag option
* refactor to deduplicate sidecar construction code
* - Add godocs
- Rename some functions to be closer to the spec
- Add err in return of commitments
* Replace mutating public method (but only internally used) `Populate` but private not mutating method `extract`.
* Implement a unique `processDataColumnSidecarsFromExecution` instead 2 separate functions from block and from sidecar.
* `ReceiveBlock`: Wrap errors.
* Remove useless tests.
* `ConstructionPopulator`: Add tests.
* Fix tests
* Move functions to be consistent with blobs.
* `fetchCellsAndProofsFromExecution`: Avoid useless flattening.
* `processDataColumnSidecarsFromExecution`: Stop using DB cache.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* create lc cache to track branches
* save lc stuff
* remove finalized data from LC cache on finalization
* read lc stuff
* edit tests
* changelog
* linter
* address commments
* address commments 2
* address commments 3
* address commments 4
* lint
* address commments 5 x_x
* set beacon lcStore to mimick registrable services
* clean up the error propagation
* pass the state to saveLCBootstrap since it's not saved in db yet
