* Use head for block validation when possible
When validating blocks for pubsub, we always copy a state and advance
when we simply need to get a read only beacon state without a copy in
most cases since the head state normally works.
* fix test
* fix tests
* fix more tests
* fix more tests
* Add nil check to be safe
* fix more tests
* add test case
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Only use head if it's compatible with target
* Allow blocks from the previous epoch to be viable for checkpoints
* Add feature flag to make it configurable
* fix tests
* @satushh's review
* Manu's nit
* Use fields in logs
* Set default value of `--blob-batch-limit` to 384.
So, using default values, `--blob-batch-limit * --blob-batch-limit-burst-factor = 384*3 = MAX_REQUEST_BLOB_SIDECARS = 1152.`
* `blobSidecarByRootRPCHandler`: Add rate limiting.
Bacause now the rate limiter validation is done before the request validation,
adapt `TestBlobsByRootValidation` consequently and add new specific tests for
`validateBlobByRootRequest` to cover the now untested case.
* Set default value of `--data-column-batch-limit-burst-factor` to 4.
So, using default values, `--data-column-batch-limit * --data-column-batch-limit-burst-factor = 4096*2 = MAX_REQUEST_DATA_COLUMN_SIDECARS_ELECTRA = 16384`.
* `validateDataColumnsByRootRequest`: Take a count instead of idents.
* `dataColumnSidecarByRootRPCHandler`: Add rate limiting.
* `SidecarProposerExpected`: Add the slot in the single flight key.
* Fix Kasey's comment.
* Revert "Fix Kasey's comment."
This reverts commit 9e3b4b7acf.
* `signatureData`: Add `string` function.
* `RODataColumnsVerifier.ValidProposerSignature`: Ensure the expensive signature verification is only performed once for concurrent requests for the same signature data.
Share flight group
* `parentState` ==> `state`.
* `RODataColumnsVerifier.SidecarProposerExpected: Ensure the expensive index computation is only performed once for concurrent requests.`
* Add `wrapAttestationError`
* Fix Kasey's comment.
* Fix Terence's comment.
* updated path processing data types, refactored ParsePath and fixed tests
* updated generalized index accordingly, changed input parameter path type from []PathElemen to Path
* updated query.go accordingly, changed input parameter path type from []PathElemen to Path
* added descriptive changelog
* Update encoding/ssz/query/path.go
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
* Added documentation for Path struct and renamed to for clarity
* Update encoding/ssz/query/path.go
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* updated changelog to its correct type: Changed
* updated outdated comment in generalized_index.go and removed test in generalized_index_test.go as this one belongs in path_test.go
* Added validateRawPath with strict raw-path validation only - no raw-path fixing is added. Added test suite covering
* added extra tests for wrongly formated paths
---------
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Add a lock for p2p computation of active validator count and limit only to topics that need it.
* Changelog fragment
* Update gossip_scoring_params.go
Wrap errors
* Implement `AvailableBlocks`.
* `blobSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.
* `dataColumnSidecarByRootRPCHandler`: Do not do extra work if only needed for TRACE logging.
* `TestDataColumnSidecarsByRootRPCHandler`: Re-arrange (no functional change).
* `TestDataColumnSidecarsByRootRPCHandler`: Save blocks corresponding to sidecars into DB.
* `dataColumnSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.
* Add changelog
* `TestDataColumnSidecarsByRootRPCHandler`: Use `assert` instead of `require` in goroutines.
https://github.com/stretchr/testify?tab=readme-ov-file#require-package
* added tests for calculating generalized indices
* added first version of GI calculation walking the specified path with no recursion. Extended test coverage for bitlist and bitvectors.
vectors need more testing
* refactored code. Detached PathElement processing, currently done at the beginning. Swap to regex to gain flexibility.
* added an updateRoot function with the GI formula. more refactoring
* added changelog
* replaced TODO tag
* udpated some comments
* simplified code - removed duplicated code in processingLengthField function
* run gazelle
* merging all input path processing into path.go
* reviewed Jun's feedback
* removed unnecessary idx pointer var + fixed error with length data type (uint64 instead of uint8)
* refactored path.go after merging path elements from generalized_indices.go
* re-computed GIs for tests as VariableTestContainer added a new field.
* added minor comment - rawPath MUST be snake case
removed extractFieldName func.
* fixed vector GI calculation - updated tests GIs
* removed updateRoot function in favor of inline code
* path input data enforced to be snake case
* added sanity checks for accessing outbound element indices - checked against vector.length/list.limit
* fixed issues triggered after merging develop
* Removed redundant comment
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
* removed unreachable condition as `strings.Split` always return a slice with length >= 1
If s does not contain sep and sep is not empty, Split returns a slice of
length 1 whose only element is s.
* added tests to cover edge cases + cleaned code (toLower is no longer needed in extractFieldName function
* added Jun's feedback + more testing
* postponed snake case conversion to do it on a per-element-basis. Added more testing focused mainly in snake case conversion
* addressed several Jun's comments.
* added sanity check to prevent length of a multi-dimensional array. added more tests with extended paths
* Update encoding/ssz/query/generalized_index.go
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* Update encoding/ssz/query/generalized_index.go
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* Update encoding/ssz/query/generalized_index.go
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* placed constant bitsPerChunk in the right place. Exported BitsPerChunk and BytesPerChunk and updated code that use them
* added helpers for computing GI of each data type
* changed %q in favor of %s
* Update encoding/ssz/query/path.go
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
* removed the least restrictive condition isBasicType
* replaced length of containerInfo.order for containerInfo.fields for clarity
* removed outdated comment
* removed toSnakeCase conversion.
* moved isBasicType func to its natural place, SSZType
* cosmetic refactor
- renamed itemLengthFromInfo to itemLength (same name is in spec).
- arranged all SSZ helpers.
* cleaned tests
* renamed "root" to "index"
* removed unnecessary check for negative integers. Replaced %q for %s.
* refactored regex variables and prevented re-assignation
* added length regex explanation
* added more testing for stressing regex for path processing
* renamed currentIndex to parentIndex for clarity and documented the returns from calculate<Type>GeneralizedIndex functions
* Update encoding/ssz/query/generalized_index.go
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
* run gazelle
* fixed never asserted error. Updated error message
---------
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Define TCP and QUIC as `InternetProtocol` (no functional change).
* Group types. (No functional changes)
* Rename variables and use range syntax.
* Add `p2pMaxPeers` and `p2pPeerCountDirectionType` metrics
* `p2p_subscribed_topic_peer_total`: Reset to avoid dangling values.
* `validateConfig`:
- Use `Warning` with fields instead of `Warnf`.
- Avoid to both modify in place the input value and return it.
* Add `p2p_minimum_peers_per_subnet` metric.
* `beaconConfig` => `cfg`.
https://github.com/OffchainLabs/prysm/pull/15880#discussion_r2436826215
* Add changelog
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* default new blob storage layouts to by-epoch
also, do not log migration message until we see a directory that needs to be migrated
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* manu feedback
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* Update Earliest available slot when pruning
* bazel run //:gazelle -- fix
* custodyUpdater interface to avoid import cycle
* bazel run //:gazelle -- fix
* simplify test
* separation of concerns
* debug log for updating eas
* UpdateEarliestAvailableSlot function in CustodyManager
* fix test
* UpdateEarliestAvailableSlot function for FakeP2P
* lint
* UpdateEarliestAvailableSlot instead of UpdateCustodyInfo + check for Fulu
* fix test and lint
* bugfix: enforce minimum retention period in pruner
* remove MinEpochsForBlockRequests function and use from config
* remove modifying earliest_available_slot after data column pruning
* correct earliestAvailableSlot validation: allow backfill decrease but prevent increase within MIN_EPOCHS_FOR_BLOCK_REQUESTS
* lint
* bazel run //:gazelle -- fix
* lint and remove unwanted debug logs
* Return a wrapped error, and let the caller decide what to do
* fix tests because updateEarliestSlot returns error now
* avoid re-doing computation in the test function
* lint and correct changelog
* custody updater should be a mandatory part of the pruner service
* ensure never increase eas if we are in the block requests window
* slot level granularity edge case
* update the value stored in the DB
* log tidy up
* use errNoCustodyInfo
* allow earliestAvailableSlot edit when custodyGroupCount doesnt change
* undo the minimal config change
* add context to CustodyGroupCount after merging from develop
* cosmetic change
* shift responsibility from caller to callee, protection for updateEarliestSlot. UpdateEarliestAvailableSlot returns cgc
* allow increase in earliestAvailableSlot only when custodyGroupCount also increases
* remove CustodyGroupCount as it is no longer needed as UpdateEarliestAvailableSlot returns cgc now
* proper place for log and name refactor
* test for Nil custody info
* allow decreasing earliest slot in DB (just like in memory)
* invert if statement to make more readable
* UpdateEarliestAvailableSlot for DB (equivalent of p2p's UpdateEarliestAvailableSlot) & undo changes made to UpdateCustodyInfo
* in UpdateEarliestAvailableSlot, no need to return unused values
* no need to log stored group count
* log.WithField instead of log.WithFields
* Add serialization code for state diffs
Adds serialization code for state diffs.
Adds code to create and apply state diffs
Adds fuzz tests and benchmarks for serialization/deserialization
Co-authored-by: Claude <noreply@anthropic.com>
* Add Fulu support
* Review #1
* gazelle
* Fix some fuzzers
* Failing cases from the fuzzers in consensus-types/hdiff
* Fix more fuzz tests
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* add comparison tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Use ConvertToElectra in UpgradeToElectra
* Add comments on constants
* Fix readEth1Data
* remove colons from error messages
* Add design doc
* Apply suggestions from code review
Bast
Co-authored-by: Bastin <43618253+Inspector-Butters@users.noreply.github.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Preston Van Loon <preston@pvl.dev>
Co-authored-by: Bastin <43618253+Inspector-Butters@users.noreply.github.com>
* Move ssz_query objects into testing folder (ensuring test objects only used in test environment)
* Add containers for response
* Export sszInfo
* Add QueryBeaconState/Block
* Add comments and few refactor
* Fix merge conflict issues
* Return 500 when calculate offset fails
* Add test for QueryBeaconState
* Add test for QueryBeaconBlock
* Changelog :)
* Rename `QuerySSZRequest` to `SSZQueryRequest`
* Fix middleware hooks for RPC to accept JSON from client and return SSZ
* Convert to `SSZObject` directly from proto
* Move marshalling/calculating hash tree root part after `CalculateOffsetAndLength`
* Make nogo happy
* Add informing comment for using proto unsafe conversion
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Fixed metadata extraction on Windows by correctly splitting file paths
* `TestExtractFileMetadata`: Refactor a bit.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
* `VerifiedRODataColumnError`: Don't reuse Blob error.
* `VerifiedRODataColumnFromDisk`: Use a specific error when the count of read bytes is lower than expected.
* Add changelog.
* Add basic parsing feature for accessing by index
* Add more tests for 2d byte vector
* Add List case for access indexing
* Handle 2D bytes List example
* Fix misleading cases for CalculateOffsetAndLength
* Use elementSizes[index] if it is the last path element
* Add variable_container_list field for mocking attester_slashings in BeaconBlockBody
* Remove redundant protobuf message
* Better documentation
* Changelog
* Fix `expectedSize` of `VariableTestContainer`: as we added `variable_container_list` here
* Apply reviews from Radek
* reject committee index >= committees_per_slot in unaggregated attestation validation
* Create phrwlk_fix-attestation-committee-index-bound.md
* add a unit test
* fix test
* fixing test
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>
