* make registerSubscribers idempotent
* clean up debugging changes
* test fix
* rm unused var
* sobbing noises
* naming feedback and separate test for digestActionDone
* gazelle
* manu's feedback
* refactor to enable immediate sub after init sync
* preston comment re panic causing db corruption risk
* ensure we check that we're 1 epoch past the fork
* manu feedback
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `reconstructSaveBroadcastDataColumnSidecars`: Use the `s.columnIndicesToSample` instead of recoding its content.
ddd
* Rename `custodyColumns` ==> `columnIndicesToSample`.
* `DataColumnStorage.Save`: Remove wrong godoc.
* Implement `receiveDataColumnSidecars` and transform `receiveDataColumnSidecar` as a subcase of the plural version.
* `dataColumnSubscriber`: Add godoc and remove only once used variable.
* `processDataColumnSidecarsFromExecution`: Use single flight directly in the function.
So the caller does not have any more the responsability to deal with multiple simultaneous calls.
* `processDataColumnSidecarsFromReconstruction`: Guard against a single flight.
In `dataColumnSubscriber`, trig in parallel `processDataColumnSidecarsFromReconstruction` and `processDataColumnSidecarsFromExecution`.
Stop when the first of them is successful.
* `processDataColumnSidecarsFromExecution`: Use `receiveDataColumnSidecars` instead of `receiveDataColumnSidecar`.
* Implement and use `broadcastAndReceiveUnseenDataColumnSidecars`.
* Add changelog.
* Fix James' comment.
* Fix James' comment.
* `processDataColumnSidecarsFromReconstruction`: Log reconstruction duration.
* PeerDAS: Implement sync
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Fix Potuz's comment.
* Implement `TestFetchDataColumnSidecarsFromPeers`.
* Implement `TestSelectPeers`.
* Fix James' comment.
* Fix flakiness in `TestSelectPeers`.
* Revert "Fix Potuz's comment."
This reverts commit c45230b455.
* Revert "Fix James' comment."
This reverts commit a3f919205a.
* `selectPeers`: Avoid map with key but empty value.
* Fix Potuz's comment.
* Add DataColumnStorage and SubscribeAllDataSubnets flag.
* getBlobsV2: retry if reconstruction isnt successful
* test: engine client and sync package, metrics
* lint: fmt and log capitalisation
* lint: return error when it is not nil
* config: make retry interval configurable
* sidecar: recover function and different context for retrying
* lint: remove unused field
* beacon: default retry interval
* reconstruct: load once, correctly deliver the result to all waiting goroutines
* reconstruct: simplify multi goroutine case and avoid race condition
* engine: remove isDataAlreadyAvailable function
* sync: no goroutine, getblobsv2 in absence of block as well, wrap error
* exec: hardcode retry interval
* da: non blocking checks
* sync: remove unwanted checks
* execution: fix test
* execution: retry atomicity test
* da: updated IsDataAvailable
* sync: remove unwanted tests
* bazel: bazel run //:gazelle -- fix
* blockchain: fix CustodyGroupCount return
* lint: formatting
* lint: lint and use unused metrics
* execution: retry logic inside ReconstructDataColumnSidecars itself
* lint: format
* execution: ensure the retry actually happens when it needs to
* execution: ensure single responsibility, execution should not do DA check
* sync: don't call ReconstructDataColumnSidecars if not required
* blockchain: move IsDataAvailable interface to blockchain package
* execution: make reconstructSingleflight part of the service struct
* blockchain: cleaner DA check
* lint: formatting and remove confusing comment
* sync: fix lint, test and add extra test for when data is actually not available
* sync: new appropriate mock service
* execution: edge case - delete activeRetries on success
* execution: use service context instead of function's for retry
* blockchain: get variable samplesPerSlot only when required
* remove redundant function and fix name
* fix test
* fix more tests
* put samplesPerSlot at appropriate place
* tidy up IsDataAvailable
* correct bad merge
* fix bad merge
* remove redundant flag option
* refactor to deduplicate sidecar construction code
* - Add godocs
- Rename some functions to be closer to the spec
- Add err in return of commitments
* Replace mutating public method (but only internally used) `Populate` but private not mutating method `extract`.
* Implement a unique `processDataColumnSidecarsFromExecution` instead 2 separate functions from block and from sidecar.
* `ReceiveBlock`: Wrap errors.
* Remove useless tests.
* `ConstructionPopulator`: Add tests.
* Fix tests
* Move functions to be consistent with blobs.
* `fetchCellsAndProofsFromExecution`: Avoid useless flattening.
* `processDataColumnSidecarsFromExecution`: Stop using DB cache.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Implement KZG proof batch verification for data column gossip validation
* Manu's feedback
* Add tests
* Update beacon-chain/sync/batch_verifier.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/batch_verifier.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update beacon-chain/sync/kzg_batch_verifier_test.go
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Fix tests
* Kasey's feedback
* `validateWithKzgBatchVerifier`: Give up after a full slot.
Before this commit:
After 100 ms, an un-batched verification is launched concurrently to the batched one.
As a result, a stressed node could start to be even more stressed by the multiple verifications.
Also, it is always hard to choose a correct timeout value.
100ms may be OK for a given node with a given BPO version, and not ok for the same node with a BPO version with 10x more blobs.
However, we know this gossip validation won't be useful after a full slot duration.
After this commit:
After a full slot duration, we just ignore the incoming gossip message.
It's important to ignore it and not to reject it, since rejecting it would downscore the peer sending this message.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Subscription: Get fanout peers in all data column subnets when at least one validator is connected
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Apply suggestion from @nalepae
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Updated test structure to @nalepae suggestion
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* `dataColumnsRPCMinValidSlot`: Add new test case.
* peerDAS: Implement `dataColumnSidecarByRangeRPCHandler`.
* `rateLimitingAmount`: Define out of the function.
* Fix James' comment.
* Fix James' comment.
* PeerDAS: Implement the validation pipeline for data column sidecars received via gossip
* Fix Terence's comment
* Fix Terence's comment.
* Fix Terence's comment.
* Add equivocation detection logic; broadcast slashing immediately on equivocation
* nit: comments
* move equivocation detection to validateBeaconBlockPubSub
* include broadcasting logic within the helper function
* fix lint
* Add unit tests for equivocation detection
* remove comment that are not required
* Add changelog file
* Add descriptive comment for detectAndBroadcastEquivocation
* use head block instead of block cache for equivocation detection
* add more equivocation unit tests; update a mock to include HeadState error
* update the order of the checks
* move slashing before state fetch; update Tests
* update changelog
* use verifyProposerSlashing to verify and reject block; remove verifySlashableBlock; update tests
* Update changelog
* nit: cleaner error check
* nit: clean up
* revert code logic; update string check; add a unit test
* improve errors; merge tests
* Update a unit test
* fix lint
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Migrate Prysm repo to Offchain Labs organization ahead of Pectra upgrade v6
* Replace prysmaticlabs with OffchainLabs on general markdowns
* Update mock
* Gazelle and add mock.go to excluded generated mock file
* Adds a wrapper package
* Gazelle
* Add in Empty Span
* Revert It Back
* Add back reference
* Set It As Empty
* fix missing import
* remove redundant alias
* remove unused
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
Co-authored-by: Saolyn <sammy@rosso.red>
* `IndexedAtt` wrapper for the slasher feed
* test fixes
* fix simulator
* fix e2e
* Revert "Auxiliary commit to revert individual files from 191bbf77accfc2523fa9f909837a2e9dca132afa"
This reverts commit 2b0441a23a0e5f66e50cf36c3bbfbb39d587b17b.
* extract interface from channel
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Add pending blobs queue for missing parent block
* Prune sidecars older than previous slot
* Prune sidecar based on time
* Tests
* Fix state notifier
* Wait for chain to start
* Remove logs
* Remove bad logs
* James feedback
* Fix conflict
* Rm outdated check
* Potuz's feedback
* Kasey's feedback
* Use 11s mark
* Use secs
* Add pending blobs queue for missing parent block
* Prune sidecars older than previous slot
* Prune sidecar based on time
* Tests
* Fix state notifier
* Wait for chain to start
* Remove logs
* Remove bad logs
* James feedback
* Fix conflict
* Rm outdated check
* Potuz's feedback
* Kasey's feedback
* Use 11s mark
* Use secs
* Add test case for duplicates
* Radek's feedback
* Fix test
* refactor initialization to blocking startup method
* require genesisSetter in blockchain, fix tests
* work-around gazelle weirdness
* fix dep gazelle ignores
* only call SetGenesis once
* fix typo
* validator test setup and fix to return right error
* move waitForChainStart to Start
* wire up sync Service.genesisWaiter
* fix p2p genesisWaiter plumbing
* remove extra clock type, integrate into genesis
and rename
* use time.Now when no Nower is specified
* remove unused ClockSetter
* simplify rpc context checking
* fix typo
* use clock everywhere in sync; [32]byte val root
* don't use DeepEqual to compare [32]byte and []byte
* don't use clock in init sync, not wired up yet
* use clock waiter in blockchain as well
* use cancelable contexts in tests with goroutines
* missed a reference to WithClockSetter
* Update beacon-chain/startup/genesis.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update beacon-chain/blockchain/service_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* more clear docs
* doc for NewClock
* move clock typedef to more logical file name
* adding documentation
* gaz
* fixes for capella
* reducing test raciness
* fix races in committee cache tests
* lint
* add tests on Duration slot math helper
* startup package test coverage
* fix bad merge
* set non-zero genesis time in tests that call Start
* happy deepsource, happy me-epsource
* replace Synced event with channel
* remove unused error
* remove accidental wip commit
* gaz!
* remove unused event constants
* remove sync statefeed subscription to fix deadlock
* remove state notifier
* fix build
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: nisdas <nishdas93@gmail.com>
* fix slashing checks
* fix to make it more performant
* gaz
* fix up
* potuz's comment
* potuz's comment
* fix cache
* change index in test for better case
* gaz
---------
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
