Compare commits

..

19 Commits

Author SHA1 Message Date
james-prysm
462cc74b47 Merge branch 'develop' into change-p2p-forkchoice-error 2025-11-03 09:55:30 -08:00
Manu NALEPA
d394f00e9f beacon_data_column_sidecar_gossip_verification_milliseconds: Adjust buckets. (#15964)
* `beacon_data_column_sidecar_gossip_verification_milliseconds`: Divide by 10.

* Fix Kasey's comment.
2025-11-03 17:00:37 +00:00
Muzry
11c6325b54 use filepath to perform path operations correctly on Windows (#15953) 2025-11-03 15:14:08 +00:00
Manu NALEPA
ec524ce99c RODataColumnsVerifier.ValidProposerSignature: Ensure the expensive signature verification is only performed once for concurrent requests for the same signature data. (#15954)
* `signatureData`: Add `string` function.

* `RODataColumnsVerifier.ValidProposerSignature`: Ensure the expensive signature verification is only performed once for concurrent requests for the same signature data.

Share flight group

* `parentState` ==> `state`.

* `RODataColumnsVerifier.SidecarProposerExpected: Ensure the expensive index computation is only performed once for concurrent requests.`

* Add `wrapAttestationError`

* Fix Kasey's comment.

* Fix Terence's comment.
2025-11-03 14:48:41 +00:00
Manu NALEPA
b2a9db0826 BeaconBlockContainerToSignedBeaconBlock: Add Fulu. (#15940) 2025-11-01 23:18:42 +00:00
Manu NALEPA
040661bd68 Update go-netroute to v0.4.0. (#15949) 2025-10-31 20:46:06 +00:00
fernantho
d3bd0eaa30 SSZ-QL: update "path parsing" data types (#15935)
* updated path processing data types, refactored ParsePath and fixed tests

* updated generalized index accordingly, changed input parameter path type from []PathElemen to Path

* updated query.go accordingly, changed input parameter path type from []PathElemen to Path

* added descriptive changelog

* Update encoding/ssz/query/path.go

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

* Added documentation for Path struct and renamed  to  for clarity

* Update encoding/ssz/query/path.go

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>

* updated changelog to its correct type: Changed

* updated outdated comment in generalized_index.go and removed test in generalized_index_test.go as this one belongs in path_test.go

* Added validateRawPath with strict raw-path validation only - no raw-path fixing is added. Added test suite covering

* added extra tests for wrongly formated paths

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-10-31 17:37:59 +00:00
Preston Van Loon
577899bfec P2p active val count lock (#15955)
* Add a lock for p2p computation of active validator count and limit only to topics that need it.

* Changelog fragment

* Update gossip_scoring_params.go

Wrap errors
2025-10-31 15:25:18 +00:00
Muzry
374bae9c81 Fix incorrect version used when sending attestation version in Fulu (#15950)
* Fix incorrect version used when sending attestation version in Fulu

* update typo

* fix Eth-Consensus-Version in submit_signed_aggregate_proof.go
2025-10-31 13:17:44 +00:00
kasey
3e0492a636 also ignore errors from readdirnames (#15947)
* also ignore errors from readdirnames

* test case for empty blobs dir

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2025-10-30 19:02:25 +00:00
rocksload
5a1a5b5ae5 refactor: use slices.Contains to simplify code (#15646)
Signed-off-by: rocksload <rocksload@outlook.com>
2025-10-29 14:40:33 +00:00
james-prysm
dbb2f0b047 changelog (#15929) 2025-10-28 15:42:05 +00:00
Manu NALEPA
7b3c11c818 Do not serve sidecars if corresponding block is not available in the database (#15933)
* Implement `AvailableBlocks`.

* `blobSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.

* `dataColumnSidecarByRootRPCHandler`: Do not do extra work if only needed for TRACE logging.

* `TestDataColumnSidecarsByRootRPCHandler`: Re-arrange (no functional change).

* `TestDataColumnSidecarsByRootRPCHandler`: Save blocks corresponding to sidecars into DB.

* `dataColumnSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.

* Add changelog

* `TestDataColumnSidecarsByRootRPCHandler`: Use `assert` instead of `require` in goroutines.

https://github.com/stretchr/testify?tab=readme-ov-file#require-package
2025-10-28 15:39:35 +00:00
Manu NALEPA
c9b34d556d Update go-netroute to v0.3.0 (#15934) 2025-10-28 12:57:14 +00:00
fernantho
10a2f0687b SSZ-QL: calculate generalized indices for elements (#15873)
* added tests for calculating generalized indices

* added first version of GI calculation walking the specified path with no recursion. Extended test coverage for bitlist and bitvectors.
vectors need more testing

* refactored code. Detached PathElement processing, currently done at the beginning. Swap to regex to gain flexibility.

* added an updateRoot function with the GI formula. more refactoring

* added changelog

* replaced TODO tag

* udpated some comments

* simplified code - removed duplicated code in processingLengthField function

* run gazelle

* merging all input path processing into path.go

* reviewed Jun's feedback

* removed unnecessary idx pointer var + fixed error with length data type (uint64 instead of uint8)

* refactored path.go after merging path elements from generalized_indices.go

* re-computed GIs for tests as VariableTestContainer added a new field.

* added minor comment - rawPath MUST be snake case

removed extractFieldName func.

* fixed vector GI calculation - updated tests GIs

* removed updateRoot function in favor of inline code

* path input data enforced to be snake case

* added sanity checks for accessing outbound element indices - checked against vector.length/list.limit

* fixed issues triggered after merging develop

* Removed redundant comment

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

* removed unreachable condition as `strings.Split` always return a slice with length >= 1

If s does not contain sep and sep is not empty, Split returns a slice of
length 1 whose only element is s.

* added tests to cover edge cases + cleaned code (toLower is no longer needed in extractFieldName function

* added Jun's feedback + more testing

* postponed snake case conversion to do it on a per-element-basis. Added more testing focused mainly in snake case conversion

* addressed several Jun's comments.

* added sanity check to prevent length of a multi-dimensional array. added more tests with extended paths

* Update encoding/ssz/query/generalized_index.go

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>

* Update encoding/ssz/query/generalized_index.go

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>

* Update encoding/ssz/query/generalized_index.go

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>

* placed constant bitsPerChunk in the right place. Exported BitsPerChunk and BytesPerChunk and updated code that use them

* added helpers for computing GI of each data type

* changed %q in favor of %s

* Update encoding/ssz/query/path.go

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

* removed the least restrictive condition isBasicType

* replaced length of containerInfo.order for containerInfo.fields for clarity

* removed outdated comment

* removed toSnakeCase conversion.

* moved isBasicType func to its natural place, SSZType

* cosmetic refactor

- renamed itemLengthFromInfo to itemLength (same name is in spec).
- arranged all SSZ helpers.

* cleaned tests

* renamed "root" to "index"

* removed unnecessary check for negative integers. Replaced %q for %s.

* refactored regex variables and prevented re-assignation

* added length regex explanation

* added more testing for stressing regex for path processing

* renamed currentIndex to parentIndex for clarity and documented the returns from calculate<Type>GeneralizedIndex functions

* Update encoding/ssz/query/generalized_index.go

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>

* run gazelle

* fixed never asserted error. Updated error message

---------

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2025-10-27 23:27:34 +00:00
james-prysm
959bfef56e gofmt 2025-10-10 12:01:24 -05:00
james-prysm
f7ceb346d0 fixing test and changelog 2025-10-10 11:55:03 -05:00
james-prysm
571e4593e0 Merge branch 'develop' into change-p2p-forkchoice-error 2025-10-09 16:52:40 -05:00
james-prysm
9aa01e2cfc changeing ErrNotDescendantOfFinalized to ErrRootNotInForkChoice 2025-10-09 16:51:20 -05:00
81 changed files with 1738 additions and 322 deletions

View File

@@ -4,6 +4,29 @@ All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
## [v6.1.4](https://github.com/prysmaticlabs/prysm/compare/v6.1.3...v6.1.4) - 2025-10-24
This release includes a bug fix affecting block proposals in rare cases, along with an important update for Windows users running post-Fusaka fork.
### Added
- SSZ-QL: Add endpoints for `BeaconState`/`BeaconBlock`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15888)
- Add native state diff type and marshalling functions. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15250)
- Update the earliest available slot after pruning operations in beacon chain database pruner. This ensures the P2P layer accurately knows which historical data is available after pruning, preventing nodes from advertising or attempting to serve data that has been pruned. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15694)
### Fixed
- Correctly advertise (in ENR and beacon API) attestation subnets when using `--subscribe-all-subnets`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15880)
- `randomPeer`: Return if the context is cancelled when waiting for peers. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15876)
- Improve error message when the byte count read from disk when reading a data column sidecars is lower than expected. (Mostly, because the file is truncated.). [[PR]](https://github.com/prysmaticlabs/prysm/pull/15881)
- Delete the genesis state file when --clear-db / --force-clear-db is specified. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15883)
- Fix sync committee subscription to use subnet indices instead of committee indices. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15885)
- Fixed metadata extraction on Windows by correctly splitting file paths. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15899)
- `VerifyDataColumnsSidecarKZGProofs`: Check if sizes match. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15892)
- Fix recoverStateSummary to persist state summaries in stateSummaryBucket instead of stateBucket (#15896). [[PR]](https://github.com/prysmaticlabs/prysm/pull/15896)
- `updateCustodyInfoInDB`: Use `NumberOfCustodyGroups` instead of `NumberOfColumns`. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15908)
- Sync committee uses correct state to calculate position. [[PR]](https://github.com/prysmaticlabs/prysm/pull/15905)
## [v6.1.3](https://github.com/prysmaticlabs/prysm/compare/v6.1.2...v6.1.3) - 2025-10-20
This release has several important beacon API and p2p fixes.

View File

@@ -2,8 +2,10 @@ package blockchain
import (
stderrors "errors"
"fmt"
"github.com/OffchainLabs/prysm/v6/beacon-chain/verification"
fieldparams "github.com/OffchainLabs/prysm/v6/config/fieldparams"
"github.com/pkg/errors"
)
@@ -48,6 +50,11 @@ var (
errBlockBeingSynced = errors.New("block is being synced")
)
// ErrRootNotInForkchoice is returned when a root cannot be found in forkchoice.
func ErrRootNotInForkchoice(root [fieldparams.RootLength]byte) invalidBlock {
return invalidBlock{error: fmt.Errorf("root %#x not in forkchoice", root), root: root}
}
// An invalid block is the block that fails state transition based on the core protocol rules.
// The beacon node shall not be accepting nor building blocks that branch off from an invalid block.
// Some examples of invalid blocks are:

View File

@@ -401,7 +401,7 @@ func (s *Service) fillInForkChoiceMissingBlocks(ctx context.Context, signed inte
return nil
}
if root != s.ensureRootNotZeros(finalized.Root) && !s.cfg.ForkChoiceStore.HasNode(root) {
return ErrNotDescendantOfFinalized
return ErrRootNotInForkchoice(root)
}
slices.Reverse(pendingNodes)
return s.cfg.ForkChoiceStore.InsertChain(ctx, pendingNodes)

View File

@@ -374,7 +374,7 @@ func TestFillForkChoiceMissingBlocks_FinalizedSibling(t *testing.T) {
err = service.fillInForkChoiceMissingBlocks(
t.Context(), wsb, beaconState.FinalizedCheckpoint(), beaconState.CurrentJustifiedCheckpoint())
require.Equal(t, ErrNotDescendantOfFinalized.Error(), err.Error())
require.Equal(t, ErrRootNotInForkchoice(bytesutil.ToBytes32(roots[8])), err.Error())
}
func TestFillForkChoiceMissingBlocks_ErrorCases(t *testing.T) {

View File

@@ -507,7 +507,7 @@ func (s *Service) validateStateTransition(ctx context.Context, preState state.Be
// Verify that the parent block is in forkchoice
parentRoot := b.ParentRoot()
if !s.InForkchoice(parentRoot) {
return nil, ErrNotDescendantOfFinalized
return nil, ErrRootNotInForkchoice(parentRoot)
}
stateTransitionStartTime := time.Now()
postState, err := transition.ExecuteStateTransition(ctx, preState, signed)

View File

@@ -472,6 +472,36 @@ func (s *ChainService) HasBlock(ctx context.Context, rt [32]byte) bool {
return s.InitSyncBlockRoots[rt]
}
func (s *ChainService) AvailableBlocks(ctx context.Context, blockRoots [][32]byte) map[[32]byte]bool {
if s.DB == nil {
return nil
}
count := len(blockRoots)
availableRoots := make(map[[32]byte]bool, count)
notInDBRoots := make([][32]byte, 0, count)
for _, root := range blockRoots {
if s.DB.HasBlock(ctx, root) {
availableRoots[root] = true
continue
}
notInDBRoots = append(notInDBRoots, root)
}
if s.InitSyncBlockRoots == nil {
return availableRoots
}
for _, root := range notInDBRoots {
if s.InitSyncBlockRoots[root] {
availableRoots[root] = true
}
}
return availableRoots
}
// RecentBlockSlot mocks the same method in the chain service.
func (s *ChainService) RecentBlockSlot([32]byte) (primitives.Slot, error) {
return s.BlockSlot, nil

View File

@@ -3,6 +3,7 @@ package blockchain
import (
"context"
"fmt"
"slices"
"github.com/OffchainLabs/prysm/v6/beacon-chain/db/filters"
"github.com/OffchainLabs/prysm/v6/config/params"
@@ -81,12 +82,10 @@ func (v *WeakSubjectivityVerifier) VerifyWeakSubjectivity(ctx context.Context, f
if err != nil {
return errors.Wrap(err, "error while retrieving block roots to verify weak subjectivity")
}
for _, root := range roots {
if v.root == root {
log.Info("Weak subjectivity check has passed!!")
v.verified = true
return nil
}
if slices.Contains(roots, v.root) {
log.Info("Weak subjectivity check has passed!!")
v.verified = true
return nil
}
return errors.Wrap(errWSBlockNotFoundInEpoch, fmt.Sprintf("root=%#x, epoch=%d", v.root, v.epoch))
}

View File

@@ -28,6 +28,7 @@ type ReadOnlyDatabase interface {
BlocksBySlot(ctx context.Context, slot primitives.Slot) ([]interfaces.ReadOnlySignedBeaconBlock, error)
BlockRootsBySlot(ctx context.Context, slot primitives.Slot) (bool, [][32]byte, error)
HasBlock(ctx context.Context, blockRoot [32]byte) bool
AvailableBlocks(ctx context.Context, blockRoots [][32]byte) map[[32]byte]bool
GenesisBlock(ctx context.Context) (interfaces.ReadOnlySignedBeaconBlock, error)
GenesisBlockRoot(ctx context.Context) ([32]byte, error)
IsFinalizedBlock(ctx context.Context, blockRoot [32]byte) bool

View File

@@ -336,6 +336,42 @@ func (s *Store) HasBlock(ctx context.Context, blockRoot [32]byte) bool {
return exists
}
// AvailableBlocks returns a set of roots indicating which blocks corresponding to `blockRoots` are available in the storage.
func (s *Store) AvailableBlocks(ctx context.Context, blockRoots [][32]byte) map[[32]byte]bool {
_, span := trace.StartSpan(ctx, "BeaconDB.AvailableBlocks")
defer span.End()
count := len(blockRoots)
availableRoots := make(map[[32]byte]bool, count)
// First, check the cache for each block root.
notInCacheRoots := make([][32]byte, 0, count)
for _, root := range blockRoots {
if v, ok := s.blockCache.Get(string(root[:])); v != nil && ok {
availableRoots[root] = true
continue
}
notInCacheRoots = append(notInCacheRoots, root)
}
// Next, check the database for the remaining block roots.
if err := s.db.View(func(tx *bolt.Tx) error {
bkt := tx.Bucket(blocksBucket)
for _, root := range notInCacheRoots {
if bkt.Get(root[:]) != nil {
availableRoots[root] = true
}
}
return nil
}); err != nil {
panic(err) // lint:nopanic -- View never returns an error.
}
return availableRoots
}
// BlocksBySlot retrieves a list of beacon blocks and its respective roots by slot.
func (s *Store) BlocksBySlot(ctx context.Context, slot primitives.Slot) ([]interfaces.ReadOnlySignedBeaconBlock, error) {
ctx, span := trace.StartSpan(ctx, "BeaconDB.BlocksBySlot")

View File

@@ -656,6 +656,44 @@ func TestStore_BlocksCRUD_NoCache(t *testing.T) {
}
}
func TestAvailableBlocks(t *testing.T) {
ctx := t.Context()
db := setupDB(t)
b0, b1, b2 := util.NewBeaconBlock(), util.NewBeaconBlock(), util.NewBeaconBlock()
b0.Block.Slot, b1.Block.Slot, b2.Block.Slot = 10, 20, 30
sb0, err := blocks.NewSignedBeaconBlock(b0)
require.NoError(t, err)
r0, err := b0.Block.HashTreeRoot()
require.NoError(t, err)
// Save b0 but remove it from cache.
err = db.SaveBlock(ctx, sb0)
require.NoError(t, err)
db.blockCache.Del(string(r0[:]))
// b1 is not saved at all.
r1, err := b1.Block.HashTreeRoot()
require.NoError(t, err)
// Save b2 in cache and DB.
sb2, err := blocks.NewSignedBeaconBlock(b2)
require.NoError(t, err)
r2, err := b2.Block.HashTreeRoot()
require.NoError(t, err)
require.NoError(t, db.SaveBlock(ctx, sb2))
require.NoError(t, err)
expected := map[[32]byte]bool{r0: true, r2: true}
actual := db.AvailableBlocks(ctx, [][32]byte{r0, r1, r2})
require.Equal(t, len(expected), len(actual))
for i := range expected {
require.Equal(t, true, actual[i])
}
}
func TestStore_Blocks_FiltersCorrectly(t *testing.T) {
for _, tt := range blockTests {
t.Run(tt.name, func(t *testing.T) {

View File

@@ -12,6 +12,7 @@ import (
"os"
"os/signal"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
@@ -1135,10 +1136,8 @@ func (b *BeaconNode) registerLightClientStore() {
func hasNetworkFlag(cliCtx *cli.Context) bool {
for _, flag := range features.NetworkFlags {
for _, name := range flag.Names() {
if cliCtx.IsSet(name) {
return true
}
if slices.ContainsFunc(flag.Names(), cliCtx.IsSet) {
return true
}
}
return false

View File

@@ -2,6 +2,7 @@ package p2p
import (
"context"
"fmt"
"math"
"net"
"reflect"
@@ -106,18 +107,26 @@ func peerScoringParams(colocationWhitelist []*net.IPNet) (*pubsub.PeerScoreParam
}
func (s *Service) topicScoreParams(topic string) (*pubsub.TopicScoreParams, error) {
activeValidators, err := s.retrieveActiveValidators()
if err != nil {
return nil, err
}
switch {
case strings.Contains(topic, GossipBlockMessage):
return defaultBlockTopicParams(), nil
case strings.Contains(topic, GossipAggregateAndProofMessage):
activeValidators, err := s.retrieveActiveValidators()
if err != nil {
return nil, fmt.Errorf("failed to compute active validator count for topic %s: %w", GossipAggregateAndProofMessage, err)
}
return defaultAggregateTopicParams(activeValidators), nil
case strings.Contains(topic, GossipAttestationMessage):
activeValidators, err := s.retrieveActiveValidators()
if err != nil {
return nil, fmt.Errorf("failed to compute active validator count for topic %s: %w", GossipAttestationMessage, err)
}
return defaultAggregateSubnetTopicParams(activeValidators), nil
case strings.Contains(topic, GossipSyncCommitteeMessage):
activeValidators, err := s.retrieveActiveValidators()
if err != nil {
return nil, fmt.Errorf("failed to compute active validator count for topic %s: %w", GossipSyncCommitteeMessage, err)
}
return defaultSyncSubnetTopicParams(activeValidators), nil
case strings.Contains(topic, GossipContributionAndProofMessage):
return defaultSyncContributionTopicParams(), nil
@@ -142,6 +151,8 @@ func (s *Service) topicScoreParams(topic string) (*pubsub.TopicScoreParams, erro
}
func (s *Service) retrieveActiveValidators() (uint64, error) {
s.activeValidatorCountLock.Lock()
defer s.activeValidatorCountLock.Unlock()
if s.activeValidatorCount != 0 {
return s.activeValidatorCount, nil
}

View File

@@ -25,6 +25,7 @@ package peers
import (
"context"
"net"
"slices"
"sort"
"strings"
"time"
@@ -306,11 +307,8 @@ func (p *Status) SubscribedToSubnet(index uint64) []peer.ID {
connectedStatus := peerData.ConnState == Connecting || peerData.ConnState == Connected
if connectedStatus && peerData.MetaData != nil && !peerData.MetaData.IsNil() && peerData.MetaData.AttnetsBitfield() != nil {
indices := indicesFromBitfield(peerData.MetaData.AttnetsBitfield())
for _, idx := range indices {
if idx == index {
peers = append(peers, pid)
break
}
if slices.Contains(indices, index) {
peers = append(peers, pid)
}
}
}

View File

@@ -65,35 +65,36 @@ var (
// Service for managing peer to peer (p2p) networking.
type Service struct {
started bool
isPreGenesis bool
pingMethod func(ctx context.Context, id peer.ID) error
pingMethodLock sync.RWMutex
cancel context.CancelFunc
cfg *Config
peers *peers.Status
addrFilter *multiaddr.Filters
ipLimiter *leakybucket.Collector
privKey *ecdsa.PrivateKey
metaData metadata.Metadata
pubsub *pubsub.PubSub
joinedTopics map[string]*pubsub.Topic
joinedTopicsLock sync.RWMutex
subnetsLock map[uint64]*sync.RWMutex
subnetsLockLock sync.Mutex // Lock access to subnetsLock
initializationLock sync.Mutex
dv5Listener ListenerRebooter
startupErr error
ctx context.Context
host host.Host
genesisTime time.Time
genesisValidatorsRoot []byte
activeValidatorCount uint64
peerDisconnectionTime *cache.Cache
custodyInfo *custodyInfo
custodyInfoLock sync.RWMutex // Lock access to custodyInfo
custodyInfoSet chan struct{}
allForkDigests map[[4]byte]struct{}
started bool
isPreGenesis bool
pingMethod func(ctx context.Context, id peer.ID) error
pingMethodLock sync.RWMutex
cancel context.CancelFunc
cfg *Config
peers *peers.Status
addrFilter *multiaddr.Filters
ipLimiter *leakybucket.Collector
privKey *ecdsa.PrivateKey
metaData metadata.Metadata
pubsub *pubsub.PubSub
joinedTopics map[string]*pubsub.Topic
joinedTopicsLock sync.RWMutex
subnetsLock map[uint64]*sync.RWMutex
subnetsLockLock sync.Mutex // Lock access to subnetsLock
initializationLock sync.Mutex
dv5Listener ListenerRebooter
startupErr error
ctx context.Context
host host.Host
genesisTime time.Time
genesisValidatorsRoot []byte
activeValidatorCount uint64
activeValidatorCountLock sync.Mutex
peerDisconnectionTime *cache.Cache
custodyInfo *custodyInfo
custodyInfoLock sync.RWMutex // Lock access to custodyInfo
custodyInfoSet chan struct{}
allForkDigests map[[4]byte]struct{}
}
type custodyInfo struct {

View File

@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"net/http"
"slices"
"strconv"
"strings"
@@ -388,12 +389,9 @@ func syncRewardsVals(
scIndices := make([]primitives.ValidatorIndex, 0, len(allScIndices))
scVals := make([]*precompute.Validator, 0, len(allScIndices))
for _, valIdx := range valIndices {
for _, scIdx := range allScIndices {
if valIdx == scIdx {
scVals = append(scVals, allVals[valIdx])
scIndices = append(scIndices, valIdx)
break
}
if slices.Contains(allScIndices, valIdx) {
scVals = append(scVals, allVals[valIdx])
scIndices = append(scIndices, valIdx)
}
}

View File

@@ -1133,14 +1133,9 @@ func randomPeer(
"delay": waitPeriod,
}).Debug("Waiting for a peer with enough bandwidth for data column sidecars")
timer := time.NewTimer(waitPeriod)
select {
case <-timer.C:
// Timer expired, retry the loop
case <-time.After(waitPeriod):
case <-ctx.Done():
// Context cancelled - stop timer to prevent leak
timer.Stop()
return "", ctx.Err()
}
}

View File

@@ -214,7 +214,7 @@ var (
prometheus.HistogramOpts{
Name: "beacon_data_column_sidecar_gossip_verification_milliseconds",
Help: "Captures the time taken to verify data column sidecars.",
Buckets: []float64{100, 250, 500, 750, 1000, 1500, 2000, 4000, 8000, 12000, 16000},
Buckets: []float64{2, 5, 10, 25, 50, 75, 100, 250, 500, 1000, 2000},
},
)

View File

@@ -5,6 +5,7 @@ import (
"context"
"encoding/hex"
"fmt"
"slices"
"github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain"
"github.com/OffchainLabs/prysm/v6/beacon-chain/core/blocks"
@@ -117,7 +118,7 @@ func (s *Service) processAttestationBucket(ctx context.Context, bucket *attestat
// Shared validations for the entire bucket.
if !s.cfg.chain.InForkchoice(bytesutil.ToBytes32(data.BeaconBlockRoot)) {
log.WithError(blockchain.ErrNotDescendantOfFinalized).WithField("root", fmt.Sprintf("%#x", data.BeaconBlockRoot)).Debug("Failed forkchoice check for bucket")
log.WithError(blockchain.ErrRootNotInForkchoice(bytesutil.ToBytes32(data.BeaconBlockRoot))).Debug("Failed forkchoice check for bucket")
return
}
@@ -382,10 +383,8 @@ func (s *Service) savePending(root [32]byte, pending any, isEqual func(other any
// Skip if the attestation/aggregate from the same validator already exists in
// the pending queue.
for _, a := range s.blkRootToPendingAtts[root] {
if isEqual(a) {
return
}
if slices.ContainsFunc(s.blkRootToPendingAtts[root], isEqual) {
return
}
pendingAttCount.Inc()

View File

@@ -10,6 +10,7 @@ import (
"github.com/OffchainLabs/prysm/v6/beacon-chain/p2p"
"github.com/OffchainLabs/prysm/v6/beacon-chain/p2p/types"
"github.com/OffchainLabs/prysm/v6/cmd/beacon-chain/flags"
fieldparams "github.com/OffchainLabs/prysm/v6/config/fieldparams"
"github.com/OffchainLabs/prysm/v6/config/params"
"github.com/OffchainLabs/prysm/v6/consensus-types/primitives"
"github.com/OffchainLabs/prysm/v6/encoding/bytesutil"
@@ -58,6 +59,17 @@ func (s *Service) blobSidecarByRootRPCHandler(ctx context.Context, msg interface
return errors.Wrapf(err, "unexpected error computing min valid blob request slot, current_slot=%d", cs)
}
// Extract all needed roots.
roots := make([][fieldparams.RootLength]byte, 0, len(blobIdents))
for _, ident := range blobIdents {
root := bytesutil.ToBytes32(ident.BlockRoot)
roots = append(roots, root)
}
// Filter all available roots in block storage.
availableRoots := s.cfg.beaconDB.AvailableBlocks(ctx, roots)
// Serve each requested blob sidecar.
for i := range blobIdents {
if err := ctx.Err(); err != nil {
closeStream(stream, log)
@@ -69,7 +81,15 @@ func (s *Service) blobSidecarByRootRPCHandler(ctx context.Context, msg interface
<-ticker.C
}
s.rateLimiter.add(stream, 1)
root, idx := bytesutil.ToBytes32(blobIdents[i].BlockRoot), blobIdents[i].Index
// Do not serve a blob sidecar if the corresponding block is not available.
if !availableRoots[root] {
log.Trace("Peer requested blob sidecar by root but corresponding block not found in db")
continue
}
sc, err := s.cfg.blobStorage.Get(root, idx)
if err != nil {
log := log.WithFields(logrus.Fields{

View File

@@ -56,18 +56,6 @@ func (s *Service) dataColumnSidecarByRootRPCHandler(ctx context.Context, msg int
return errors.Wrap(err, "validate data columns by root request")
}
requestedColumnsByRoot := make(map[[fieldparams.RootLength]byte][]uint64)
for _, columnIdent := range requestedColumnIdents {
var root [fieldparams.RootLength]byte
copy(root[:], columnIdent.BlockRoot)
requestedColumnsByRoot[root] = append(requestedColumnsByRoot[root], columnIdent.Columns...)
}
// Sort by column index for each root.
for _, columns := range requestedColumnsByRoot {
slices.Sort(columns)
}
// Compute the oldest slot we'll allow a peer to request, based on the current slot.
minReqSlot, err := dataColumnsRPCMinValidSlot(s.cfg.clock.CurrentSlot())
if err != nil {
@@ -84,6 +72,12 @@ func (s *Service) dataColumnSidecarByRootRPCHandler(ctx context.Context, msg int
}
if log.Logger.Level >= logrus.TraceLevel {
requestedColumnsByRoot := make(map[[fieldparams.RootLength]byte][]uint64)
for _, ident := range requestedColumnIdents {
root := bytesutil.ToBytes32(ident.BlockRoot)
requestedColumnsByRoot[root] = append(requestedColumnsByRoot[root], ident.Columns...)
}
// We optimistially assume the peer requests the same set of columns for all roots,
// pre-sizing the map accordingly.
requestedRootsByColumnSet := make(map[string][]string, 1)
@@ -96,6 +90,17 @@ func (s *Service) dataColumnSidecarByRootRPCHandler(ctx context.Context, msg int
log.WithField("requested", requestedRootsByColumnSet).Trace("Serving data column sidecars by root")
}
// Extract all requested roots.
roots := make([][fieldparams.RootLength]byte, 0, len(requestedColumnIdents))
for _, ident := range requestedColumnIdents {
root := bytesutil.ToBytes32(ident.BlockRoot)
roots = append(roots, root)
}
// Filter all available roots in block storage.
availableRoots := s.cfg.beaconDB.AvailableBlocks(ctx, roots)
// Serve each requested data column sidecar.
count := 0
for _, ident := range requestedColumnIdents {
if err := ctx.Err(); err != nil {
@@ -117,6 +122,12 @@ func (s *Service) dataColumnSidecarByRootRPCHandler(ctx context.Context, msg int
s.rateLimiter.add(stream, int64(len(columns)))
// Do not serve a blob sidecar if the corresponding block is not available.
if !availableRoots[root] {
log.Trace("Peer requested blob sidecar by root but corresponding block not found in db")
continue
}
// Retrieve the requested sidecars from the store.
verifiedRODataColumns, err := s.cfg.dataColumnStorage.Get(root, columns)
if err != nil {

View File

@@ -10,6 +10,7 @@ import (
chainMock "github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain/testing"
"github.com/OffchainLabs/prysm/v6/beacon-chain/db/filesystem"
testDB "github.com/OffchainLabs/prysm/v6/beacon-chain/db/testing"
"github.com/OffchainLabs/prysm/v6/beacon-chain/p2p"
p2ptest "github.com/OffchainLabs/prysm/v6/beacon-chain/p2p/testing"
"github.com/OffchainLabs/prysm/v6/beacon-chain/p2p/types"
@@ -19,6 +20,7 @@ import (
"github.com/OffchainLabs/prysm/v6/config/params"
"github.com/OffchainLabs/prysm/v6/consensus-types/blocks"
"github.com/OffchainLabs/prysm/v6/consensus-types/primitives"
"github.com/OffchainLabs/prysm/v6/testing/assert"
"github.com/OffchainLabs/prysm/v6/testing/require"
"github.com/OffchainLabs/prysm/v6/testing/util"
"github.com/libp2p/go-libp2p/core/network"
@@ -103,23 +105,47 @@ func TestDataColumnSidecarsByRootRPCHandler(t *testing.T) {
localP2P := p2ptest.NewTestP2P(t)
clock := startup.NewClock(time.Now(), [fieldparams.RootLength]byte{})
params := []util.DataColumnParam{
{Slot: 10, Index: 1}, {Slot: 10, Index: 2}, {Slot: 10, Index: 3},
{Slot: 40, Index: 4}, {Slot: 40, Index: 6},
{Slot: 45, Index: 7}, {Slot: 45, Index: 8}, {Slot: 45, Index: 9},
_, verifiedRODataColumns := util.CreateTestVerifiedRoDataColumnSidecars(
t,
[]util.DataColumnParam{
{Slot: 10, Index: 1}, {Slot: 10, Index: 2}, {Slot: 10, Index: 3},
{Slot: 40, Index: 4}, {Slot: 40, Index: 6},
{Slot: 45, Index: 7}, {Slot: 45, Index: 8}, {Slot: 45, Index: 9},
{Slot: 46, Index: 10}, // Corresponding block won't be saved in DB
},
)
dataColumnStorage := filesystem.NewEphemeralDataColumnStorage(t)
err := dataColumnStorage.Save(verifiedRODataColumns)
require.NoError(t, err)
beaconDB := testDB.SetupDB(t)
indices := [...]int{0, 3, 5}
roBlocks := make([]blocks.ROBlock, 0, len(indices))
for _, i := range indices {
blockPb := util.NewBeaconBlock()
signedBeaconBlock, err := blocks.NewSignedBeaconBlock(blockPb)
require.NoError(t, err)
// Here the block root has to match the sidecar's block root.
// (However, the block root does not match the actual root of the block, but we don't care for this test.)
roBlock, err := blocks.NewROBlockWithRoot(signedBeaconBlock, verifiedRODataColumns[i].BlockRoot())
require.NoError(t, err)
roBlocks = append(roBlocks, roBlock)
}
_, verifiedRODataColumns := util.CreateTestVerifiedRoDataColumnSidecars(t, params)
storage := filesystem.NewEphemeralDataColumnStorage(t)
err := storage.Save(verifiedRODataColumns)
err = beaconDB.SaveROBlocks(ctx, roBlocks, false /*cache*/)
require.NoError(t, err)
service := &Service{
cfg: &config{
p2p: localP2P,
beaconDB: beaconDB,
clock: clock,
dataColumnStorage: storage,
dataColumnStorage: dataColumnStorage,
chain: &chainMock.ChainService{},
},
rateLimiter: newRateLimiter(localP2P),
@@ -134,6 +160,7 @@ func TestDataColumnSidecarsByRootRPCHandler(t *testing.T) {
root0 := verifiedRODataColumns[0].BlockRoot()
root3 := verifiedRODataColumns[3].BlockRoot()
root5 := verifiedRODataColumns[5].BlockRoot()
root8 := verifiedRODataColumns[8].BlockRoot()
remoteP2P.BHost.SetStreamHandler(protocolID, func(stream network.Stream) {
defer wg.Done()
@@ -147,22 +174,22 @@ func TestDataColumnSidecarsByRootRPCHandler(t *testing.T) {
break
}
require.NoError(t, err)
assert.NoError(t, err)
sidecars = append(sidecars, sidecar)
}
require.Equal(t, 5, len(sidecars))
require.Equal(t, root3, sidecars[0].BlockRoot())
require.Equal(t, root3, sidecars[1].BlockRoot())
require.Equal(t, root5, sidecars[2].BlockRoot())
require.Equal(t, root5, sidecars[3].BlockRoot())
require.Equal(t, root5, sidecars[4].BlockRoot())
assert.Equal(t, 5, len(sidecars))
assert.Equal(t, root3, sidecars[0].BlockRoot())
assert.Equal(t, root3, sidecars[1].BlockRoot())
assert.Equal(t, root5, sidecars[2].BlockRoot())
assert.Equal(t, root5, sidecars[3].BlockRoot())
assert.Equal(t, root5, sidecars[4].BlockRoot())
require.Equal(t, uint64(4), sidecars[0].Index)
require.Equal(t, uint64(6), sidecars[1].Index)
require.Equal(t, uint64(7), sidecars[2].Index)
require.Equal(t, uint64(8), sidecars[3].Index)
require.Equal(t, uint64(9), sidecars[4].Index)
assert.Equal(t, uint64(4), sidecars[0].Index)
assert.Equal(t, uint64(6), sidecars[1].Index)
assert.Equal(t, uint64(7), sidecars[2].Index)
assert.Equal(t, uint64(8), sidecars[3].Index)
assert.Equal(t, uint64(9), sidecars[4].Index)
})
localP2P.Connect(remoteP2P)
@@ -182,6 +209,10 @@ func TestDataColumnSidecarsByRootRPCHandler(t *testing.T) {
BlockRoot: root5[:],
Columns: []uint64{7, 8, 9},
},
{
BlockRoot: root8[:],
Columns: []uint64{10},
},
}
err = service.dataColumnSidecarByRootRPCHandler(ctx, msg, stream)

View File

@@ -3,6 +3,7 @@ package sync
import (
"context"
"fmt"
"slices"
"github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain"
"github.com/OffchainLabs/prysm/v6/beacon-chain/core/blocks"
@@ -166,8 +167,8 @@ func (s *Service) validateAggregatedAtt(ctx context.Context, signed ethpb.Signed
// Verify current finalized checkpoint is an ancestor of the block defined by the attestation's beacon block root.
if !s.cfg.chain.InForkchoice(bytesutil.ToBytes32(data.BeaconBlockRoot)) {
tracing.AnnotateError(span, blockchain.ErrNotDescendantOfFinalized)
return pubsub.ValidationIgnore, blockchain.ErrNotDescendantOfFinalized
tracing.AnnotateError(span, blockchain.ErrRootNotInForkchoice(bytesutil.ToBytes32(data.BeaconBlockRoot)))
return pubsub.ValidationIgnore, blockchain.ErrRootNotInForkchoice(bytesutil.ToBytes32(data.BeaconBlockRoot))
}
bs, err := s.cfg.chain.AttestationTargetState(ctx, data.Target)
@@ -290,11 +291,8 @@ func (s *Service) validateIndexInCommittee(ctx context.Context, a ethpb.Att, val
}
var withinCommittee bool
for _, i := range committee {
if validatorIndex == i {
withinCommittee = true
break
}
if slices.Contains(committee, validatorIndex) {
withinCommittee = true
}
if !withinCommittee {
return pubsub.ValidationReject, fmt.Errorf("validator index %d is not within the committee: %v",

View File

@@ -5,6 +5,7 @@ import (
"encoding/binary"
"fmt"
"reflect"
"slices"
"strings"
"github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain"
@@ -15,6 +16,7 @@ import (
"github.com/OffchainLabs/prysm/v6/beacon-chain/p2p"
"github.com/OffchainLabs/prysm/v6/beacon-chain/slasher/types"
"github.com/OffchainLabs/prysm/v6/beacon-chain/state"
"github.com/OffchainLabs/prysm/v6/config/params"
"github.com/OffchainLabs/prysm/v6/consensus-types/primitives"
"github.com/OffchainLabs/prysm/v6/encoding/bytesutil"
"github.com/OffchainLabs/prysm/v6/monitoring/tracing"
@@ -66,7 +68,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
return pubsub.ValidationReject, errWrongMessage
}
if err := helpers.ValidateNilAttestation(att); err != nil {
return pubsub.ValidationReject, err
return pubsub.ValidationReject, wrapAttestationError(err, att)
}
data := att.GetData()
@@ -83,7 +85,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
return pubsub.ValidationIgnore, err
}
if err := helpers.ValidateSlotTargetEpoch(data); err != nil {
return pubsub.ValidationReject, err
return pubsub.ValidationReject, wrapAttestationError(err, att)
}
committeeIndex := att.GetCommitteeIndex()
@@ -105,7 +107,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
s.hasBadBlock(bytesutil.ToBytes32(data.Target.Root)) ||
s.hasBadBlock(bytesutil.ToBytes32(data.Source.Root)) {
attBadBlockCount.Inc()
return pubsub.ValidationReject, errors.New("attestation data references bad block root")
return pubsub.ValidationReject, wrapAttestationError(errors.New("attestation data references bad block root"), att)
}
}
@@ -119,13 +121,13 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
}
// Block exists - verify it's in forkchoice (i.e., it's a descendant of the finalized checkpoint)
if !s.cfg.chain.InForkchoice(blockRoot) {
tracing.AnnotateError(span, blockchain.ErrNotDescendantOfFinalized)
return pubsub.ValidationIgnore, blockchain.ErrNotDescendantOfFinalized
tracing.AnnotateError(span, blockchain.ErrRootNotInForkchoice(blockRoot))
return pubsub.ValidationIgnore, blockchain.ErrRootNotInForkchoice(blockRoot)
}
if err = s.cfg.chain.VerifyLmdFfgConsistency(ctx, att); err != nil {
tracing.AnnotateError(span, err)
attBadLmdConsistencyCount.Inc()
return pubsub.ValidationReject, err
return pubsub.ValidationReject, wrapAttestationError(err, att)
}
preState, err := s.cfg.chain.AttestationTargetState(ctx, data.Target)
@@ -136,7 +138,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
validationRes, err := s.validateUnaggregatedAttTopic(ctx, att, preState, *msg.Topic)
if validationRes != pubsub.ValidationAccept {
return validationRes, err
return validationRes, wrapAttestationError(err, att)
}
committee, err := helpers.BeaconCommitteeFromState(ctx, preState, data.Slot, committeeIndex)
@@ -147,7 +149,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
validationRes, err = validateAttesterData(ctx, att, committee)
if validationRes != pubsub.ValidationAccept {
return validationRes, err
return validationRes, wrapAttestationError(err, att)
}
// Consolidated handling of Electra SingleAttestation vs Phase0 unaggregated attestation
@@ -182,7 +184,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(
validationRes, err = s.validateUnaggregatedAttWithState(ctx, attForValidation, preState)
if validationRes != pubsub.ValidationAccept {
return validationRes, err
return validationRes, wrapAttestationError(err, att)
}
if s.slasherEnabled {
@@ -336,15 +338,9 @@ func validateAttestingIndex(
// _[REJECT]_ The attester is a member of the committee -- i.e.
// `attestation.attester_index in get_beacon_committee(state, attestation.data.slot, index)`.
inCommittee := false
for _, ix := range committee {
if attestingIndex == ix {
inCommittee = true
break
}
}
inCommittee := slices.Contains(committee, attestingIndex)
if !inCommittee {
return pubsub.ValidationReject, errors.New("attester is not a member of the committee")
return pubsub.ValidationReject, errors.Errorf("attester %d is not a member of the committee", attestingIndex)
}
return pubsub.ValidationAccept, nil
@@ -397,3 +393,24 @@ func (s *Service) hasBlockAndState(ctx context.Context, blockRoot [32]byte) bool
hasState := hasStateSummary || s.cfg.beaconDB.HasState(ctx, blockRoot)
return hasState && s.cfg.chain.HasBlock(ctx, blockRoot)
}
func wrapAttestationError(err error, att eth.Att) error {
slotsPerEpoch := params.BeaconConfig().SlotsPerEpoch
committeeIndex := att.GetCommitteeIndex()
attData := att.GetData()
slot := attData.Slot
slotInEpoch := slot % slotsPerEpoch
oldCommitteeIndex := attData.CommitteeIndex
blockRoot := fmt.Sprintf("%#x", attData.BeaconBlockRoot)
sourceRoot := fmt.Sprintf("%#x", attData.Source.Root)
sourceEpoch := attData.Source.Epoch
targetEpoch := attData.Target.Epoch
targetRoot := fmt.Sprintf("%#x", attData.Target.Root)
return errors.Wrapf(
err,
"attSlot: %d, attSlotInEpoch: %d, attOldCommitteeIndex: %d, attCommitteeIndex: %d, attBlockRoot: %s, attSource: {root: %s, epoch: %d}, attTarget: {root: %s, epoch: %d}",
slot, slotInEpoch, oldCommitteeIndex, committeeIndex, blockRoot, sourceRoot, sourceEpoch, targetRoot, targetEpoch,
)
}

View File

@@ -285,7 +285,7 @@ func (s *Service) validateBeaconBlock(ctx context.Context, blk interfaces.ReadOn
func (s *Service) validatePhase0Block(ctx context.Context, blk interfaces.ReadOnlySignedBeaconBlock, blockRoot [32]byte) (state.BeaconState, error) {
if !s.cfg.chain.InForkchoice(blk.Block().ParentRoot()) {
s.setBadBlock(ctx, blockRoot)
return nil, blockchain.ErrNotDescendantOfFinalized
return nil, blockchain.ErrRootNotInForkchoice(blk.Block().ParentRoot())
}
parentState, err := s.cfg.stateGen.StateByRoot(ctx, blk.Block().ParentRoot())

View File

@@ -44,6 +44,7 @@ go_library(
"@com_github_prometheus_client_golang//prometheus/promauto:go_default_library",
"@com_github_sirupsen_logrus//:go_default_library",
"@com_github_spf13_afero//:go_default_library",
"@org_golang_x_sync//singleflight:go_default_library",
],
)

View File

@@ -50,6 +50,10 @@ type signatureData struct {
Slot primitives.Slot
}
func (d signatureData) concat() string {
return string(d.Root[:]) + string(d.Signature[:])
}
func (d signatureData) logFields() logrus.Fields {
return logrus.Fields{
"root": fmt.Sprintf("%#x", d.Root),

View File

@@ -21,6 +21,18 @@ func testSignedBlockBlobKeys(t *testing.T, valRoot []byte, slot primitives.Slot,
return block, blobs, sks[0], pks[0]
}
func TestSignatureDataString(t *testing.T) {
const expected = "\x01\x02\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x05\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
sigData := signatureData{
Root: [32]byte{1, 2, 3},
Signature: [96]byte{4, 5, 6},
}
actual := sigData.concat()
require.Equal(t, expected, actual)
}
func TestVerifySignature(t *testing.T) {
valRoot := [32]byte{}
_, blobs, _, pk := testSignedBlockBlobKeys(t, valRoot[:], 0, 1)

View File

@@ -257,17 +257,25 @@ func (dv *RODataColumnsVerifier) ValidProposerSignature(ctx context.Context) (er
continue
}
columnVerificationProposerSignatureCache.WithLabelValues("miss").Inc()
// Ensure the expensive signature verification is only performed once for
// concurrent requests for the same signature data.
if _, err, _ = dv.sg.Do(signatureData.concat(), func() (any, error) {
columnVerificationProposerSignatureCache.WithLabelValues("miss").Inc()
// Retrieve the parent state.
parentState, err := dv.parentState(ctx, dataColumn)
if err != nil {
return columnErrBuilder(errors.Wrap(err, "parent state"))
}
// Retrieve the parent state.
parentState, err := dv.state(ctx, dataColumn.ParentRoot())
if err != nil {
return nil, columnErrBuilder(errors.Wrap(err, "parent state"))
}
// Full verification, which will subsequently be cached for anything sharing the signature cache.
if err = dv.sc.VerifySignature(signatureData, parentState); err != nil {
return columnErrBuilder(errors.Wrap(err, "verify signature"))
// Full verification, which will subsequently be cached for anything sharing the signature cache.
if err = dv.sc.VerifySignature(signatureData, parentState); err != nil {
return nil, columnErrBuilder(errors.Wrap(err, "verify signature"))
}
return nil, nil
}); err != nil {
return err
}
}
@@ -470,15 +478,25 @@ func (dv *RODataColumnsVerifier) SidecarProposerExpected(ctx context.Context) (e
idx, cached := dv.pc.Proposer(checkpoint, dataColumnSlot)
if !cached {
// Retrieve the parent state.
parentState, err := dv.parentState(ctx, dataColumn)
if err != nil {
return columnErrBuilder(errors.Wrap(err, "parent state"))
}
parentRoot := dataColumn.ParentRoot()
// Ensure the expensive index computation is only performed once for
// concurrent requests for the same signature data.
if _, err, _ := dv.sg.Do(fmt.Sprintf("%#x", parentRoot), func() (any, error) {
// Retrieve the parent state.
parentState, err := dv.state(ctx, parentRoot)
if err != nil {
return nil, columnErrBuilder(errors.Wrap(err, "parent state"))
}
idx, err = dv.pc.ComputeProposer(ctx, parentRoot, dataColumnSlot, parentState)
if err != nil {
return columnErrBuilder(errors.Wrap(err, "compute proposer"))
// Compute the proposer index.
idx, err = dv.pc.ComputeProposer(ctx, parentRoot, dataColumnSlot, parentState)
if err != nil {
return nil, columnErrBuilder(errors.Wrap(err, "compute proposer"))
}
return nil, nil
}); err != nil {
return err
}
}
@@ -490,23 +508,21 @@ func (dv *RODataColumnsVerifier) SidecarProposerExpected(ctx context.Context) (e
return nil
}
// parentState retrieves the parent state of the data column from the cache if possible, else retrieves it from the state by rooter.
func (dv *RODataColumnsVerifier) parentState(ctx context.Context, dataColumn blocks.RODataColumn) (state.BeaconState, error) {
parentRoot := dataColumn.ParentRoot()
// state retrieves the state of the corresponding root from the cache if possible, else retrieves it from the state by rooter.
func (dv *RODataColumnsVerifier) state(ctx context.Context, root [fieldparams.RootLength]byte) (state.BeaconState, error) {
// If the parent root is already in the cache, return it.
if st, ok := dv.stateByRoot[parentRoot]; ok {
if st, ok := dv.stateByRoot[root]; ok {
return st, nil
}
// Retrieve the parent state from the state by rooter.
st, err := dv.sr.StateByRoot(ctx, parentRoot)
st, err := dv.sr.StateByRoot(ctx, root)
if err != nil {
return nil, errors.Wrap(err, "state by root")
}
// Store the parent state in the cache.
dv.stateByRoot[parentRoot] = st
dv.stateByRoot[root] = st
return st, nil
}

View File

@@ -14,6 +14,7 @@ import (
"github.com/OffchainLabs/prysm/v6/consensus-types/blocks"
"github.com/OffchainLabs/prysm/v6/consensus-types/primitives"
ethpb "github.com/OffchainLabs/prysm/v6/proto/prysm/v1alpha1"
"golang.org/x/sync/singleflight"
)
// Forkchoicer represents the forkchoice methods that the verifiers need.
@@ -41,6 +42,7 @@ type sharedResources struct {
pc proposerCache
sr StateByRooter
ic *inclusionProofCache
sg singleflight.Group
}
// Initializer is used to create different Verifiers.

View File

@@ -1,3 +0,0 @@
### Fixed
- Fix recoverStateSummary to persist state summaries in stateSummaryBucket instead of stateBucket (#15896).

View File

@@ -0,0 +1,3 @@
### Added
- Added GeneralizedIndicesFromPath function to calculate the GIs for a given sszInfo object and a PathElement

View File

@@ -0,0 +1,3 @@
## Changed
- Introduced Path type for SSZ-QL queries and updated PathElement (removed Length field, kept Index) enforcing that len queries are terminal (at most one per path).
- Changed length query syntax from `block.payload.len(transactions)` to `len(block.payload.transactions)`

View File

@@ -0,0 +1,3 @@
### Changed
- changed ErrNotDescendantOfFinalized to ErrRootNotInForkChoice.

View File

@@ -1,3 +0,0 @@
### Ignored
- Changelog entries for v6.1.3 through v6.1.2

View File

@@ -0,0 +1,3 @@
### Ignored
- Changelog entries for v6.1.4 through v6.1.3

View File

@@ -1,2 +0,0 @@
### Fixed
- Delete the genesis state file when --clear-db / --force-clear-db is specified.

View File

@@ -0,0 +1,2 @@
### Ignored
- Fix bug with layout detection when readdirnames returns io.EOF.

View File

@@ -1,2 +0,0 @@
### Fixed
- Correctly advertise (in ENR and beacon API) attestation subnets when using `--subscribe-all-subnets`.

View File

@@ -0,0 +1,3 @@
### Fixed
- `blobSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.
- `dataColumnSidecarByRootRPCHandler`: Do not serve a sidecar if the corresponding block is not available.

View File

@@ -0,0 +1,2 @@
### Ignored
- `BeaconBlockContainerToSignedBeaconBlock`: Add Fulu.

View File

@@ -0,0 +1,2 @@
### Changed
- Update `go-netroute` to `v0.4.0`

View File

@@ -0,0 +1,2 @@
### Changed
- Update go-netroute to `v0.3.0`

2
changelog/manu-metric.md Normal file
View File

@@ -0,0 +1,2 @@
### Ignored
- `beacon_data_column_sidecar_gossip_verification_milliseconds`: Divide by 10.

View File

@@ -1,2 +0,0 @@
### Fixed
- `updateCustodyInfoInDB`: Use `NumberOfCustodyGroups` instead of `NumberOfColumns`.

View File

@@ -1,2 +0,0 @@
### Fixed
- `randomPeer`: Return if the context is cancelled when waiting for peers.

View File

@@ -1,2 +0,0 @@
### Fixed
- Improve error message when the byte count read from disk when reading a data column sidecars is lower than expected. (Mostly, because the file is truncated.)

View File

@@ -0,0 +1,2 @@
### Fixed
- `RODataColumnsVerifier.ValidProposerSignature`: Ensure the expensive signature verification is only performed once for concurrent requests for the same signature data.

View File

@@ -1,2 +0,0 @@
### Fixed
- `VerifyDataColumnsSidecarKZGProofs`: Check if sizes match.

View File

@@ -0,0 +1,2 @@
### Fixed
- Fix incorrect version used when sending attestation version in Fulu

View File

@@ -1,2 +0,0 @@
### Fixed
- Fixed metadata extraction on Windows by correctly splitting file paths

View File

@@ -0,0 +1,3 @@
### Fixed
- use filepath for path operations (clean, join, etc.) to ensure correct behavior on Windows

View File

@@ -1,3 +0,0 @@
### Added
- Add native state diff type and marshalling functions

View File

@@ -0,0 +1,4 @@
### Fixed
- Changed the behavior of topic subscriptions such that only topics that require the active validator count will compute that value.
- Added a Mutex to the computation of active validator count during topic subscription to avoid a race condition where multiple goroutines are computing the same work.

View File

@@ -0,0 +1,3 @@
### Ignored
- Use slices.Contains to simplify code

View File

@@ -1,3 +0,0 @@
### Added
- Update the earliest available slot after pruning operations in beacon chain database pruner. This ensures the P2P layer accurately knows which historical data is available after pruning, preventing nodes from advertising or attempting to serve data that has been pruned.

View File

@@ -1,3 +0,0 @@
### Added
- SSZ-QL: Add endpoints for `BeaconState`/`BeaconBlock`.

View File

@@ -1,3 +0,0 @@
### Fixed
- Sync committee uses correct state to calculate position

View File

@@ -1,3 +0,0 @@
### Fixed
- Fix sync committee subscription to use subnet indices instead of committee indices

View File

@@ -1,3 +0,0 @@
### Changed
- Replace `time.After()` with `time.NewTimer()` and explicitly stop the timer when the context is cancelled

View File

@@ -2,8 +2,10 @@ package storage
import (
"fmt"
"io"
"os"
"path"
"path/filepath"
"slices"
"strings"
"github.com/OffchainLabs/prysm/v6/beacon-chain/db/filesystem"
@@ -55,10 +57,8 @@ func layoutFlagUsage() string {
}
func validateLayoutFlag(_ *cli.Context, v string) error {
for _, l := range filesystem.LayoutNames {
if v == l {
return nil
}
if slices.Contains(filesystem.LayoutNames, v) {
return nil
}
return errors.Errorf("invalid value '%s' for flag --%s, %s", v, BlobStorageLayout.Name, layoutOptions())
}
@@ -120,7 +120,7 @@ func detectLayout(dir string, c stringFlagGetter) (string, error) {
return explicit, nil
}
dir = path.Clean(dir)
dir = filepath.Clean(dir)
// nosec: this path is provided by the node operator via flag
base, err := os.Open(dir) // #nosec G304
if err != nil {
@@ -140,6 +140,10 @@ func detectLayout(dir string, c stringFlagGetter) (string, error) {
// amount of wiggle room to be confident that we'll likely see a by-root director if one exists.
entries, err := base.Readdirnames(16)
if err != nil {
// We can get this error if the directory exists and is empty
if errors.Is(err, io.EOF) {
return filesystem.LayoutNameByEpoch, nil
}
return "", errors.Wrap(err, "reading blob storage directory")
}
for _, entry := range entries {
@@ -154,7 +158,7 @@ func blobStoragePath(c *cli.Context) string {
blobsPath := c.Path(BlobStoragePathFlag.Name)
if blobsPath == "" {
// append a "blobs" subdir to the end of the data dir path
blobsPath = path.Join(c.String(cmd.DataDirFlag.Name), "blobs")
blobsPath = filepath.Join(c.String(cmd.DataDirFlag.Name), "blobs")
}
return blobsPath
}
@@ -163,7 +167,7 @@ func dataColumnStoragePath(c *cli.Context) string {
dataColumnsPath := c.Path(DataColumnStoragePathFlag.Name)
if dataColumnsPath == "" {
// append a "data-columns" subdir to the end of the data dir path
dataColumnsPath = path.Join(c.String(cmd.DataDirFlag.Name), "data-columns")
dataColumnsPath = filepath.Join(c.String(cmd.DataDirFlag.Name), "data-columns")
}
return dataColumnsPath

View File

@@ -192,6 +192,13 @@ func TestDetectLayout(t *testing.T) {
},
expectedErr: syscall.ENOTDIR,
},
{
name: "empty blobs dir",
setup: func(t *testing.T, dir string) {
require.NoError(t, os.MkdirAll(dir, 0o755))
},
expected: filesystem.LayoutNameByEpoch,
},
}
for _, tc := range cases {

View File

@@ -4,6 +4,7 @@ package flags
import (
"fmt"
"slices"
"strings"
"github.com/urfave/cli/v2"
@@ -19,11 +20,9 @@ type EnumValue struct {
}
func (e *EnumValue) Set(value string) error {
for _, enum := range e.Enum {
if enum == value {
*e.Destination = value
return nil
}
if slices.Contains(e.Enum, value) {
*e.Destination = value
return nil
}
return fmt.Errorf("allowed values are %s", strings.Join(e.Enum, ", "))

View File

@@ -640,6 +640,10 @@ func BuildSignedBeaconBlockFromExecutionPayload(blk interfaces.ReadOnlySignedBea
// This is particularly useful for using the values from API calls.
func BeaconBlockContainerToSignedBeaconBlock(obj *eth.BeaconBlockContainer) (interfaces.ReadOnlySignedBeaconBlock, error) {
switch obj.Block.(type) {
case *eth.BeaconBlockContainer_BlindedFuluBlock:
return NewSignedBeaconBlock(obj.GetBlindedFuluBlock())
case *eth.BeaconBlockContainer_FuluBlock:
return NewSignedBeaconBlock(obj.GetFuluBlock())
case *eth.BeaconBlockContainer_BlindedElectraBlock:
return NewSignedBeaconBlock(obj.GetBlindedElectraBlock())
case *eth.BeaconBlockContainer_ElectraBlock:

View File

@@ -268,20 +268,16 @@ func (s *Slice[V]) At(obj Identifiable, index uint64) (V, error) {
return s.sharedItems[index], nil
}
for _, v := range ind.Values {
for _, id := range v.ids {
if id == obj.Id() {
return v.val, nil
}
if slices.Contains(v.ids, obj.Id()) {
return v.val, nil
}
}
return s.sharedItems[index], nil
} else {
item := s.appendedItems[index-uint64(len(s.sharedItems))]
for _, v := range item.Values {
for _, id := range v.ids {
if id == obj.Id() {
return v.val, nil
}
if slices.Contains(v.ids, obj.Id()) {
return v.val, nil
}
}
var def V

View File

@@ -2023,8 +2023,8 @@ def prysm_deps():
go_repository(
name = "com_github_libp2p_go_netroute",
importpath = "github.com/libp2p/go-netroute",
sum = "h1:Dejd8cQ47Qx2kRABg6lPwknU7+nBnFRpko45/fFPuZ8=",
version = "v0.2.2",
sum = "h1:sZZx9hyANYUx9PZyqcgE/E1GUG3iEtTZHUEvdtXT7/Q=",
version = "v0.4.0",
)
go_repository(
name = "com_github_libp2p_go_reuseport",

View File

@@ -11,7 +11,10 @@ import (
"github.com/prysmaticlabs/go-bitfield"
)
const bytesPerChunk = 32
const (
BitsPerChunk = 256
BytesPerChunk = 32
)
// BitlistRoot returns the mix in length of a bitwise Merkleized bitfield.
func BitlistRoot(bfield bitfield.Bitfield, maxCapacity uint64) ([32]byte, error) {
@@ -54,14 +57,14 @@ func BitwiseMerkleize(chunks [][32]byte, count, limit uint64) ([32]byte, error)
}
// PackByChunk a given byte array's final chunk with zeroes if needed.
func PackByChunk(serializedItems [][]byte) ([][bytesPerChunk]byte, error) {
var emptyChunk [bytesPerChunk]byte
func PackByChunk(serializedItems [][]byte) ([][BytesPerChunk]byte, error) {
var emptyChunk [BytesPerChunk]byte
// If there are no items, we return an empty chunk.
if len(serializedItems) == 0 {
return [][bytesPerChunk]byte{emptyChunk}, nil
} else if len(serializedItems[0]) == bytesPerChunk {
return [][BytesPerChunk]byte{emptyChunk}, nil
} else if len(serializedItems[0]) == BytesPerChunk {
// If each item has exactly BYTES_PER_CHUNK length, we return the list of serialized items.
chunks := make([][bytesPerChunk]byte, 0, len(serializedItems))
chunks := make([][BytesPerChunk]byte, 0, len(serializedItems))
for _, c := range serializedItems {
chunks = append(chunks, bytesutil.ToBytes32(c))
}
@@ -75,12 +78,12 @@ func PackByChunk(serializedItems [][]byte) ([][bytesPerChunk]byte, error) {
// If all our serialized item slices are length zero, we
// exit early.
if len(orderedItems) == 0 {
return [][bytesPerChunk]byte{emptyChunk}, nil
return [][BytesPerChunk]byte{emptyChunk}, nil
}
numItems := len(orderedItems)
var chunks [][bytesPerChunk]byte
for i := 0; i < numItems; i += bytesPerChunk {
j := i + bytesPerChunk
var chunks [][BytesPerChunk]byte
for i := 0; i < numItems; i += BytesPerChunk {
j := i + BytesPerChunk
// We create our upper bound index of the chunk, if it is greater than numItems,
// we set it as numItems itself.
if j > numItems {
@@ -89,7 +92,7 @@ func PackByChunk(serializedItems [][]byte) ([][bytesPerChunk]byte, error) {
// We create chunks from the list of items based on the
// indices determined above.
// Right-pad the last chunk with zero bytes if it does not
// have length bytesPerChunk from the helper.
// have length BytesPerChunk from the helper.
// The ToBytes32 helper allocates a 32-byte array, before
// copying the ordered items in. This ensures that even if
// the last chunk is != 32 in length, we will right-pad it with

View File

@@ -7,6 +7,7 @@ go_library(
"bitlist.go",
"bitvector.go",
"container.go",
"generalized_index.go",
"list.go",
"path.go",
"query.go",
@@ -18,12 +19,16 @@ go_library(
],
importpath = "github.com/OffchainLabs/prysm/v6/encoding/ssz/query",
visibility = ["//visibility:public"],
deps = ["@com_github_prysmaticlabs_go_bitfield//:go_default_library"],
deps = [
"//encoding/ssz:go_default_library",
"@com_github_prysmaticlabs_go_bitfield//:go_default_library",
],
)
go_test(
name = "go_default_test",
srcs = [
"generalized_index_test.go",
"path_test.go",
"query_test.go",
"tag_parser_test.go",

View File

@@ -0,0 +1,321 @@
package query
import (
"errors"
"fmt"
"github.com/OffchainLabs/prysm/v6/encoding/ssz"
)
const listBaseIndex = 2
// GetGeneralizedIndexFromPath calculates the generalized index for a given path.
// To calculate the generalized index, two inputs are needed:
// 1. The sszInfo of the root object, to be able to navigate the SSZ structure
// 2. The path to the field (e.g., "field_a.field_b[3].field_c")
// It walks the path step by step, updating the generalized index at each step.
func GetGeneralizedIndexFromPath(info *SszInfo, path Path) (uint64, error) {
if info == nil {
return 0, errors.New("SszInfo is nil")
}
// If path is empty, no generalized index can be computed.
if len(path.Elements) == 0 {
return 0, errors.New("cannot compute generalized index for an empty path")
}
// Starting from the root generalized index
currentIndex := uint64(1)
currentInfo := info
for index, pathElement := range path.Elements {
element := pathElement
// Check that we are in a container to access fields
if currentInfo.sszType != Container {
return 0, fmt.Errorf("indexing requires a container field step first, got %s", currentInfo.sszType)
}
// Retrieve the field position and SSZInfo for the field in the current container
fieldPos, fieldSsz, err := getContainerFieldByName(currentInfo, element.Name)
if err != nil {
return 0, fmt.Errorf("container field %s not found: %w", element.Name, err)
}
// Get the chunk count for the current container
chunkCount, err := getChunkCount(currentInfo)
if err != nil {
return 0, fmt.Errorf("chunk count error: %w", err)
}
// Update the generalized index to point to the specified field
currentIndex = currentIndex*nextPowerOfTwo(chunkCount) + fieldPos
currentInfo = fieldSsz
// Check for length access: element is the last in the path and requests length
if path.Length && index == len(path.Elements)-1 {
currentInfo, currentIndex, err = calculateLengthGeneralizedIndex(fieldSsz, element, currentIndex)
if err != nil {
return 0, fmt.Errorf("length calculation error: %w", err)
}
continue
}
if element.Index == nil {
continue
}
switch fieldSsz.sszType {
case List:
currentInfo, currentIndex, err = calculateListGeneralizedIndex(fieldSsz, element, currentIndex)
if err != nil {
return 0, fmt.Errorf("list calculation error: %w", err)
}
case Vector:
currentInfo, currentIndex, err = calculateVectorGeneralizedIndex(fieldSsz, element, currentIndex)
if err != nil {
return 0, fmt.Errorf("vector calculation error: %w", err)
}
case Bitlist:
currentInfo, currentIndex, err = calculateBitlistGeneralizedIndex(fieldSsz, element, currentIndex)
if err != nil {
return 0, fmt.Errorf("bitlist calculation error: %w", err)
}
case Bitvector:
currentInfo, currentIndex, err = calculateBitvectorGeneralizedIndex(fieldSsz, element, currentIndex)
if err != nil {
return 0, fmt.Errorf("bitvector calculation error: %w", err)
}
default:
return 0, fmt.Errorf("indexing not supported for type %s", fieldSsz.sszType)
}
}
return currentIndex, nil
}
// getContainerFieldByName finds a container field by its name
// and returns its index and SSZInfo.
func getContainerFieldByName(info *SszInfo, fieldName string) (uint64, *SszInfo, error) {
containerInfo, err := info.ContainerInfo()
if err != nil {
return 0, nil, err
}
for index, name := range containerInfo.order {
if name == fieldName {
fieldInfo := containerInfo.fields[name]
if fieldInfo == nil || fieldInfo.sszInfo == nil {
return 0, nil, fmt.Errorf("field %s has no ssz info", name)
}
return uint64(index), fieldInfo.sszInfo, nil
}
}
return 0, nil, fmt.Errorf("field %s not found", fieldName)
}
// Helpers for Generalized Index calculation per type
// calculateLengthGeneralizedIndex calculates the generalized index for a length field.
// note: length fields are only valid for List and Bitlist types. Multi-dimensional arrays are not supported.
// Returns:
// - its descendant SSZInfo (length field i.e. uint64)
// - its generalized index.
func calculateLengthGeneralizedIndex(fieldSsz *SszInfo, element PathElement, parentIndex uint64) (*SszInfo, uint64, error) {
if element.Index != nil {
return nil, 0, fmt.Errorf("len() is not supported for multi-dimensional arrays")
}
// Length field is only valid for List and Bitlist types
if fieldSsz.sszType != List && fieldSsz.sszType != Bitlist {
return nil, 0, fmt.Errorf("len() is only supported for List and Bitlist types, got %s", fieldSsz.sszType)
}
// Length is a uint64 per SSZ spec
currentInfo := &SszInfo{sszType: Uint64}
lengthIndex := parentIndex*2 + 1
return currentInfo, lengthIndex, nil
}
// calculateListGeneralizedIndex calculates the generalized index for a list element.
// Returns:
// - its descendant SSZInfo (list element)
// - its generalized index.
func calculateListGeneralizedIndex(fieldSsz *SszInfo, element PathElement, parentIndex uint64) (*SszInfo, uint64, error) {
li, err := fieldSsz.ListInfo()
if err != nil {
return nil, 0, fmt.Errorf("list info error: %w", err)
}
elem, err := li.Element()
if err != nil {
return nil, 0, fmt.Errorf("list element error: %w", err)
}
if *element.Index >= li.Limit() {
return nil, 0, fmt.Errorf("index %d out of bounds for list with limit %d", *element.Index, li.Limit())
}
// Compute chunk position for the element
var chunkPos uint64
if elem.sszType.isBasic() {
start := *element.Index * itemLength(elem)
chunkPos = start / ssz.BytesPerChunk
} else {
chunkPos = *element.Index
}
innerChunkCount, err := getChunkCount(fieldSsz)
if err != nil {
return nil, 0, fmt.Errorf("chunk count error: %w", err)
}
// root = root * base_index * pow2ceil(chunk_count(container)) + fieldPos
listIndex := parentIndex*listBaseIndex*nextPowerOfTwo(innerChunkCount) + chunkPos
currentInfo := elem
return currentInfo, listIndex, nil
}
// calculateVectorGeneralizedIndex calculates the generalized index for a vector element.
// Returns:
// - its descendant SSZInfo (vector element)
// - its generalized index.
func calculateVectorGeneralizedIndex(fieldSsz *SszInfo, element PathElement, parentIndex uint64) (*SszInfo, uint64, error) {
vi, err := fieldSsz.VectorInfo()
if err != nil {
return nil, 0, fmt.Errorf("vector info error: %w", err)
}
elem, err := vi.Element()
if err != nil {
return nil, 0, fmt.Errorf("vector element error: %w", err)
}
if *element.Index >= vi.Length() {
return nil, 0, fmt.Errorf("index %d out of bounds for vector with length %d", *element.Index, vi.Length())
}
var chunkPos uint64
if elem.sszType.isBasic() {
start := *element.Index * itemLength(elem)
chunkPos = start / ssz.BytesPerChunk
} else {
chunkPos = *element.Index
}
innerChunkCount, err := getChunkCount(fieldSsz)
if err != nil {
return nil, 0, fmt.Errorf("chunk count error: %w", err)
}
vectorIndex := parentIndex*nextPowerOfTwo(innerChunkCount) + chunkPos
currentInfo := elem
return currentInfo, vectorIndex, nil
}
// calculateBitlistGeneralizedIndex calculates the generalized index for a bitlist element.
// Returns:
// - its descendant SSZInfo (bitlist element i.e. a boolean)
// - its generalized index.
func calculateBitlistGeneralizedIndex(fieldSsz *SszInfo, element PathElement, parentIndex uint64) (*SszInfo, uint64, error) {
// Bits packed into 256-bit chunks; select the chunk containing the bit
chunkPos := *element.Index / ssz.BitsPerChunk
innerChunkCount, err := getChunkCount(fieldSsz)
if err != nil {
return nil, 0, fmt.Errorf("chunk count error: %w", err)
}
bitlistIndex := parentIndex*listBaseIndex*nextPowerOfTwo(innerChunkCount) + chunkPos
// Bits element is not further descendable; set to basic to guard further steps
currentInfo := &SszInfo{sszType: Boolean}
return currentInfo, bitlistIndex, nil
}
// calculateBitvectorGeneralizedIndex calculates the generalized index for a bitvector element.
// Returns:
// - its descendant SSZInfo (bitvector element i.e. a boolean)
// - its generalized index.
func calculateBitvectorGeneralizedIndex(fieldSsz *SszInfo, element PathElement, parentIndex uint64) (*SszInfo, uint64, error) {
chunkPos := *element.Index / ssz.BitsPerChunk
innerChunkCount, err := getChunkCount(fieldSsz)
if err != nil {
return nil, 0, fmt.Errorf("chunk count error: %w", err)
}
bitvectorIndex := parentIndex*nextPowerOfTwo(innerChunkCount) + chunkPos
// Bits element is not further descendable; set to basic to guard further steps
currentInfo := &SszInfo{sszType: Boolean}
return currentInfo, bitvectorIndex, nil
}
// Helper functions from SSZ spec
// itemLength calculates the byte length of an SSZ item based on its type information.
// For basic SSZ types (uint8, uint16, uint32, uint64, bool, etc.), it returns the actual
// size of the type in bytes. For compound types (containers, lists, vectors), it returns
// BytesPerChunk which represents the standard SSZ chunk size (32 bytes) used for
// Merkle tree operations in the SSZ serialization format.
func itemLength(info *SszInfo) uint64 {
if info.sszType.isBasic() {
return info.Size()
}
return ssz.BytesPerChunk
}
// nextPowerOfTwo computes the next power of two greater than or equal to v.
func nextPowerOfTwo(v uint64) uint64 {
v--
v |= v >> 1
v |= v >> 2
v |= v >> 4
v |= v >> 8
v |= v >> 16
v++
return uint64(v)
}
// getChunkCount returns the number of chunks for the given SSZInfo (equivalent to chunk_count in the spec)
func getChunkCount(info *SszInfo) (uint64, error) {
switch info.sszType {
case Uint8, Uint16, Uint32, Uint64, Boolean:
return 1, nil
case Container:
containerInfo, err := info.ContainerInfo()
if err != nil {
return 0, err
}
return uint64(len(containerInfo.fields)), nil
case List:
listInfo, err := info.ListInfo()
if err != nil {
return 0, err
}
elementInfo, err := listInfo.Element()
if err != nil {
return 0, err
}
elemLength := itemLength(elementInfo)
return (listInfo.Limit()*elemLength + 31) / ssz.BytesPerChunk, nil
case Vector:
vectorInfo, err := info.VectorInfo()
if err != nil {
return 0, err
}
elementInfo, err := vectorInfo.Element()
if err != nil {
return 0, err
}
elemLength := itemLength(elementInfo)
return (vectorInfo.Length()*elemLength + 31) / ssz.BytesPerChunk, nil
case Bitlist:
bitlistInfo, err := info.BitlistInfo()
if err != nil {
return 0, err
}
return (bitlistInfo.Limit() + 255) / ssz.BitsPerChunk, nil // Bits are packed into 256-bit chunks
case Bitvector:
bitvectorInfo, err := info.BitvectorInfo()
if err != nil {
return 0, err
}
return (bitvectorInfo.Length() + 255) / ssz.BitsPerChunk, nil // Bits are packed into 256-bit chunks
default:
return 0, errors.New("unsupported SSZ type for chunk count calculation")
}
}

View File

@@ -0,0 +1,364 @@
package query_test
import (
"strings"
"testing"
"github.com/OffchainLabs/prysm/v6/encoding/ssz/query"
sszquerypb "github.com/OffchainLabs/prysm/v6/proto/ssz_query/testing"
"github.com/OffchainLabs/prysm/v6/testing/require"
)
func TestGetIndicesFromPath_FixedNestedContainer(t *testing.T) {
fixedNestedContainer := &sszquerypb.FixedNestedContainer{}
info, err := query.AnalyzeObject(fixedNestedContainer)
require.NoError(t, err)
require.NotNil(t, info, "Expected non-nil SSZ info")
testCases := []struct {
name string
path string
expectedIndex uint64
expectError bool
errorMessage string
}{
{
name: "Value1 field",
path: ".value1",
expectedIndex: 2,
expectError: false,
},
{
name: "Value3 field",
path: ".value3",
expectError: true,
errorMessage: "field value3 not found",
},
{
name: "Basic field cannot descend",
path: "value1.value1",
expectError: true,
errorMessage: "indexing requires a container field step first, got Uint64",
},
{
name: "Indexing without container step",
path: "value2.value2[0]",
expectError: true,
errorMessage: "indexing requires a container field step first",
},
{
name: "Value2 field",
path: "value2",
expectedIndex: 3,
expectError: false,
},
{
name: "Value2 -> element[0]",
path: "value2[0]",
expectedIndex: 3,
expectError: false,
},
{
name: "Value2 -> element[31]",
path: "value2[31]",
expectedIndex: 3,
expectError: false,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
provingFields, err := query.ParsePath(tc.path)
require.NoError(t, err)
actualIndex, err := query.GetGeneralizedIndexFromPath(info, provingFields)
if tc.expectError {
require.NotNil(t, err)
if tc.errorMessage != "" {
if !strings.Contains(err.Error(), tc.errorMessage) {
t.Errorf("Expected error message to contain '%s', but got: %s", tc.errorMessage, err.Error())
}
}
} else {
require.NoError(t, err)
require.Equal(t, tc.expectedIndex, actualIndex, "Generalized index mismatch for path: %s", tc.path)
t.Logf("Path: %s -> Generalized Index: %v", tc.path, actualIndex)
}
})
}
}
func TestGetIndicesFromPath_VariableTestContainer(t *testing.T) {
testSpec := &sszquerypb.VariableTestContainer{}
info, err := query.AnalyzeObject(testSpec)
require.NoError(t, err)
require.NotNil(t, info, "Expected non-nil SSZ info")
testCases := []struct {
name string
path string
expectedIndex uint64
expectError bool
errorMessage string
}{
{
name: "leading_field",
path: "leading_field",
expectedIndex: 16,
expectError: false,
},
{
name: "field_list_uint64",
path: "field_list_uint64",
expectedIndex: 17,
expectError: false,
},
{
name: "len(field_list_uint64)",
path: "len(field_list_uint64)",
expectedIndex: 35,
expectError: false,
},
{
name: "field_list_uint64[0]",
path: "field_list_uint64[0]",
expectedIndex: 17408,
expectError: false,
},
{
name: "field_list_uint64[2047]",
path: "field_list_uint64[2047]",
expectedIndex: 17919,
expectError: false,
},
{
name: "bitlist_field",
path: "bitlist_field",
expectedIndex: 22,
expectError: false,
},
{
name: "bitlist_field[0]",
path: "bitlist_field[0]",
expectedIndex: 352,
expectError: false,
},
{
name: "bitlist_field[1]",
path: "bitlist_field[1]",
expectedIndex: 352,
expectError: false,
},
{
name: "len(bitlist_field)",
path: "len(bitlist_field)",
expectedIndex: 45,
expectError: false,
},
{
name: "len(trailing_field)",
path: "len(trailing_field)",
expectError: true,
errorMessage: "len() is only supported for List and Bitlist types, got Vector",
},
{
name: "field_list_container[0]",
path: "field_list_container[0]",
expectedIndex: 4608,
expectError: false,
},
{
name: "nested",
path: "nested",
expectedIndex: 20,
expectError: false,
},
{
name: "nested.field_list_uint64[10]",
path: "nested.field_list_uint64[10]",
expectedIndex: 5186,
expectError: false,
},
{
name: "variable_container_list",
path: "variable_container_list",
expectedIndex: 21,
expectError: false,
},
{
name: "len(variable_container_list)",
path: "len(variable_container_list)",
expectedIndex: 43,
expectError: false,
},
{
name: "variable_container_list[0]",
path: "variable_container_list[0]",
expectedIndex: 672,
expectError: false,
},
{
name: "variable_container_list[0].inner_1",
path: "variable_container_list[0].inner_1",
expectedIndex: 1344,
expectError: false,
},
{
name: "variable_container_list[0].inner_1.field_list_uint64[1]",
path: "variable_container_list[0].inner_1.field_list_uint64[1]",
expectedIndex: 344128,
expectError: false,
},
{
name: "len(variable_container_list[0].inner_1.nested_list_field[3])",
path: "len(variable_container_list[0].inner_1.nested_list_field[3])",
expectError: true,
errorMessage: "length calculation error: len() is not supported for multi-dimensional arrays",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
provingFields, err := query.ParsePath(tc.path)
require.NoError(t, err)
actualIndex, err := query.GetGeneralizedIndexFromPath(info, provingFields)
if tc.expectError {
require.NotNil(t, err)
if tc.errorMessage != "" {
if !strings.Contains(err.Error(), tc.errorMessage) {
t.Errorf("Expected error message to contain '%s', but got: %s", tc.errorMessage, err.Error())
}
}
} else {
require.NoError(t, err)
require.Equal(t, tc.expectedIndex, actualIndex, "Generalized index mismatch for path: %s", tc.path)
t.Logf("Path: %s -> Generalized Index: %v", tc.path, actualIndex)
}
})
}
}
func TestGetIndicesFromPath_FixedTestContainer(t *testing.T) {
testSpec := &sszquerypb.FixedTestContainer{}
info, err := query.AnalyzeObject(testSpec)
require.NoError(t, err)
require.NotNil(t, info, "Expected non-nil SSZ info")
testCases := []struct {
name string
path string
expectedIndex uint64
expectError bool
errorMessage string
}{
{
name: "field_uint32",
path: "field_uint32",
expectedIndex: 16,
expectError: false,
},
{
name: ".field_uint64",
path: ".field_uint64",
expectedIndex: 17,
expectError: false,
},
{
name: "field_bool",
path: "field_bool",
expectedIndex: 18,
expectError: false,
},
{
name: "field_bytes32",
path: "field_bytes32",
expectedIndex: 19,
expectError: false,
},
{
name: "nested",
path: "nested",
expectedIndex: 20,
expectError: false,
},
{
name: "vector_field",
path: "vector_field",
expectedIndex: 21,
expectError: false,
},
{
name: "two_dimension_bytes_field",
path: "two_dimension_bytes_field",
expectedIndex: 22,
expectError: false,
},
{
name: "bitvector64_field",
path: "bitvector64_field",
expectedIndex: 23,
expectError: false,
},
{
name: "bitvector512_field",
path: "bitvector512_field",
expectedIndex: 24,
expectError: false,
},
{
name: "bitvector64_field[0]",
path: "bitvector64_field[0]",
expectedIndex: 23,
expectError: false,
},
{
name: "bitvector64_field[63]",
path: "bitvector64_field[63]",
expectedIndex: 23,
expectError: false,
},
{
name: "bitvector512_field[0]",
path: "bitvector512_field[0]",
expectedIndex: 48,
expectError: false,
},
{
name: "bitvector512_field[511]",
path: "bitvector512_field[511]",
expectedIndex: 49,
expectError: false,
},
{
name: "trailing_field",
path: "trailing_field",
expectedIndex: 25,
expectError: false,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
provingFields, err := query.ParsePath(tc.path)
require.NoError(t, err)
actualIndex, err := query.GetGeneralizedIndexFromPath(info, provingFields)
if tc.expectError {
require.NotNil(t, err)
if tc.errorMessage != "" {
if !strings.Contains(err.Error(), tc.errorMessage) {
t.Errorf("Expected error message to contain '%s', but got: %s", tc.errorMessage, err.Error())
}
}
} else {
require.NoError(t, err)
require.Equal(t, tc.expectedIndex, actualIndex, "Generalized index mismatch for path: %s", tc.path)
t.Logf("Path: %s -> Generalized Index: %v", tc.path, actualIndex)
}
})
}
}

View File

@@ -3,6 +3,7 @@ package query
import (
"errors"
"fmt"
"regexp"
"strconv"
"strings"
)
@@ -14,11 +15,55 @@ type PathElement struct {
Index *uint64
}
func ParsePath(rawPath string) ([]PathElement, error) {
// We use dot notation, so we split the path by '.'.
rawElements := strings.Split(rawPath, ".")
if len(rawElements) == 0 {
return nil, errors.New("empty path provided")
// Path represents the entire path structure for SSZ-QL queries. It consists of multiple PathElements
// and a flag indicating if the path is querying for length.
type Path struct {
// If true, the path is querying for the length of the final element in Elements field
Length bool
// Sequence of path elements representing the navigation through the SSZ structure
Elements []PathElement
}
// Matches an array index expression like [123] or [ foo ] and captures the inner content without the brackets.
var arrayIndexRegex = regexp.MustCompile(`\[(\d+)\]`)
// Matches an entire string thats a len(<expr>) call (whitespace flexible), capturing the inner expression and disallowing any trailing characters.
var lengthRegex = regexp.MustCompile(`^\s*len\s*\(\s*([^)]+?)\s*\)\s*$`)
// Valid path characters: letters, digits, dot, slash, square brackets and parentheses only.
// Any other character will render the path invalid.
var validPathChars = regexp.MustCompile(`^[A-Za-z0-9._\[\]\(\)]*$`)
// Invalid patterns: a closing bracket followed directly by a letter or underscore
var invalidBracketPattern = regexp.MustCompile(`\][^.\[\)]|\).`)
// ParsePath parses a raw path string into a slice of PathElements.
// note: field names are stored in snake case format. rawPath has to be provided in snake case.
// 1. Supports dot notation for field access (e.g., "field1.field2").
// 2. Supports array indexing using square brackets (e.g., "array_field[0]").
// 3. Supports length access using len() notation (e.g., "len(array_field)").
// 4. Handles leading dots and validates path format.
func ParsePath(rawPath string) (Path, error) {
if err := validateRawPath(rawPath); err != nil {
return Path{}, err
}
var rawElements []string
var processedPath Path
matches := lengthRegex.FindStringSubmatch(rawPath)
// FindStringSubmatch matches a whole string like "len(field_name)" and its inner expression.
// For a path element to be a length query, len(matches) should be 2:
// 1. Full match: "len(field_name)"
// 2. Inner expression: "field_name"
if len(matches) == 2 {
processedPath.Length = true
// If we have found a len() expression, we only want to parse its inner expression.
rawElements = strings.Split(matches[1], ".")
} else {
// Normal path parsing
rawElements = strings.Split(rawPath, ".")
}
if rawElements[0] == "" {
@@ -26,37 +71,118 @@ func ParsePath(rawPath string) ([]PathElement, error) {
rawElements = rawElements[1:]
}
var path []PathElement
var pathElements []PathElement
for _, elem := range rawElements {
if elem == "" {
return nil, errors.New("invalid path: consecutive dots or trailing dot")
return Path{}, errors.New("invalid path: consecutive dots or trailing dot")
}
fieldName := elem
var index *uint64
// Processing element string
processingField := elem
var pathElement PathElement
// Check for index notation, e.g., "field[0]"
if strings.Contains(elem, "[") {
parts := strings.SplitN(elem, "[", 2)
if len(parts) != 2 {
return nil, fmt.Errorf("invalid index notation in path element %s", elem)
}
// Default name is the full working string (may be updated below if it contains indices)
pathElement.Name = processingField
fieldName = parts[0]
indexPart := strings.TrimSuffix(parts[1], "]")
if indexPart == "" {
return nil, errors.New("index cannot be empty")
}
indexValue, err := strconv.ParseUint(indexPart, 10, 64)
if strings.Contains(processingField, "[") {
// Split into field and indices, e.g., "array[0][1]" -> name:"array", indices:{0,1}
pathElement.Name = extractFieldName(processingField)
indices, err := extractArrayIndices(processingField)
if err != nil {
return nil, fmt.Errorf("invalid index in path element %s: %w", elem, err)
return Path{}, err
}
index = &indexValue
// Although extractArrayIndices supports multiple indices,
// only a single index is supported per PathElement, e.g., "transactions[0]" is valid
// while "transactions[0][0]" is rejected explicitly.
if len(indices) != 1 {
return Path{}, fmt.Errorf("multiple indices not supported in token %s", processingField)
}
pathElement.Index = &indices[0]
}
path = append(path, PathElement{Name: fieldName, Index: index})
pathElements = append(pathElements, pathElement)
}
return path, nil
processedPath.Elements = pathElements
return processedPath, nil
}
// validateRawPath performs initial validation of the raw path string:
// 1. Rejects invalid characters (only letters, digits, '.', '[]', and '()' are allowed).
// 2. Validates balanced parentheses
// 3. Validates balanced brackets.
// 4. Ensures len() calls are only at the start of the path.
// 5. Rejects empty len() calls.
// 6. Rejects invalid patterns like "][a" or "][_" which indicate malformed paths.
func validateRawPath(rawPath string) error {
// 1. Reject any path containing invalid characters (this includes spaces).
if !validPathChars.MatchString(rawPath) {
return fmt.Errorf("invalid character in path: only letters, digits, '.', '[]' and '()' are allowed")
}
// 2. Basic validation for balanced parentheses: wrongly formatted paths like "test))((" are not rejected in this condition but later.
if strings.Count(rawPath, "(") != strings.Count(rawPath, ")") {
return fmt.Errorf("unmatched parentheses in path: %s", rawPath)
}
// 3. Basic validation for balanced brackets:
// wrongly formatted paths like "array][0][" are rejected by checking bracket counts and format.
matches := arrayIndexRegex.FindAllStringSubmatch(rawPath, -1)
openBracketsCount := strings.Count(rawPath, "[")
closeBracketsCount := strings.Count(rawPath, "]")
if openBracketsCount != closeBracketsCount {
return fmt.Errorf("unmatched brackets in path: %s", rawPath)
}
if len(matches) != openBracketsCount || len(matches) != closeBracketsCount {
return fmt.Errorf("invalid bracket format in path: %s", rawPath)
}
// 4. Reject len() calls not at the start of the path
if strings.Index(rawPath, "len(") > 0 {
return fmt.Errorf("len() call must be at the start of the path: %s", rawPath)
}
// 5. Reject empty len() calls
if strings.Contains(rawPath, "len()") {
return fmt.Errorf("len() call must not be empty: %s", rawPath)
}
// 6. Reject invalid patterns like "][a" or "][_" which indicate malformed paths
if invalidBracketPattern.MatchString(rawPath) {
return fmt.Errorf("invalid path format near brackets in path: %s", rawPath)
}
return nil
}
// extractFieldName extracts the field name from a path element name (removes array indices)
// For example: "field_name[5]" returns "field_name"
func extractFieldName(name string) string {
if idx := strings.Index(name, "["); idx != -1 {
return name[:idx]
}
return name
}
// extractArrayIndices returns every bracketed, non-negative index in the name,
// e.g. "array[0][1]" -> []uint64{0, 1}. Errors if none are found or if any index is invalid.
func extractArrayIndices(name string) ([]uint64, error) {
// Match all bracketed content, then we'll parse as unsigned to catch negatives explicitly
matches := arrayIndexRegex.FindAllStringSubmatch(name, -1)
if len(matches) == 0 {
return nil, errors.New("no array indices found")
}
indices := make([]uint64, 0, len(matches))
for _, m := range matches {
raw := strings.TrimSpace(m[1])
idx, err := strconv.ParseUint(raw, 10, 64)
if err != nil {
return nil, fmt.Errorf("invalid array index: %w", err)
}
indices = append(indices, idx)
}
return indices, nil
}

View File

@@ -7,33 +7,293 @@ import (
"github.com/OffchainLabs/prysm/v6/testing/require"
)
// Helper to get pointer to uint64
func u64(v uint64) *uint64 { return &v }
func TestParsePath(t *testing.T) {
tests := []struct {
name string
path string
expected []query.PathElement
expected query.Path
wantErr bool
}{
{
name: "simple nested path",
path: "data.target.root",
expected: []query.PathElement{
{Name: "data"},
{Name: "target"},
{Name: "root"},
name: "simple path",
path: "data",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "data"},
},
},
wantErr: false,
},
{
name: "simple nested path with leading dot",
path: ".data.target.root",
expected: []query.PathElement{
{Name: "data"},
{Name: "target"},
{Name: "root"},
name: "simple path beginning with dot",
path: ".data",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "data"},
},
},
wantErr: false,
},
{
name: "simple path trailing dot",
path: "data.",
wantErr: true,
},
{
name: "simple path surrounded by dot",
path: ".data.",
wantErr: true,
},
{
name: "simple path beginning with two dots",
path: "..data",
wantErr: true,
},
{
name: "simple nested path",
path: "data.target.root",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "data"},
{Name: "target"},
{Name: "root"},
},
},
wantErr: false,
},
{
name: "len with top-level identifier",
path: "len(data)",
expected: query.Path{
Length: true,
Elements: []query.PathElement{
{Name: "data"},
},
},
wantErr: false,
},
{
name: "len with top-level identifier and leading dot",
path: "len(.data)",
expected: query.Path{
Length: true,
Elements: []query.PathElement{
{Name: "data"},
},
},
wantErr: false,
},
{
name: "len with top-level identifier and trailing dot",
path: "len(data.)",
wantErr: true,
},
{
name: "len with top-level identifier beginning dot",
path: ".len(data)",
wantErr: true,
},
{
name: "len with dotted path inside",
path: "len(data.target.root)",
expected: query.Path{
Length: true,
Elements: []query.PathElement{
{Name: "data"},
{Name: "target"},
{Name: "root"},
},
},
wantErr: false,
},
{
name: "simple length path with non-outer length field",
path: "data.target.len(root)",
wantErr: true,
},
{
name: "simple path with `len` used as a field name",
path: "data.len",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "data"},
{Name: "len"},
},
},
wantErr: false,
},
{
name: "simple path with `len` used as a field name + trailing field",
path: "data.len.value",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "data"},
{Name: "len"},
{Name: "value"},
},
},
wantErr: false,
},
{
name: "simple path with `len`",
path: "len.len",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "len"},
{Name: "len"},
},
},
wantErr: false,
},
{
name: "simple length path with length field",
path: "len.len(root)",
wantErr: true,
},
{
name: "empty length field",
path: "len()",
wantErr: true,
},
{
name: "length field not terminal",
path: "len(data).foo",
wantErr: true,
},
{
name: "length field with missing closing paren",
path: "len(data",
wantErr: true,
},
{
name: "length field with two closing paren",
path: "len(data))",
wantErr: true,
},
{
name: "len with comma-separated args",
path: "len(a,b)",
wantErr: true,
},
{
name: "array index path",
path: "arr[42]",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "arr", Index: u64(42)},
},
},
wantErr: false,
},
{
name: "array index path with max uint64",
path: "arr[18446744073709551615]",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "arr", Index: u64(18446744073709551615)},
},
},
wantErr: false,
},
{
name: "array element in wrong nested path",
path: "arr[42]foo",
wantErr: true,
},
{
name: "array index in nested path",
path: "arr[42].foo",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "arr", Index: u64(42)},
{Name: "foo"},
},
},
wantErr: false,
},
{
name: "array index in deeper nested path",
path: "arr[42].foo.bar[10]",
expected: query.Path{
Length: false,
Elements: []query.PathElement{
{Name: "arr", Index: u64(42)},
{Name: "foo"},
{Name: "bar", Index: u64(10)},
},
},
wantErr: false,
},
{
name: "length of array element",
path: "len(arr[42])",
expected: query.Path{
Length: true,
Elements: []query.PathElement{
{Name: "arr", Index: u64(42)},
},
},
wantErr: false,
},
{
name: "length of array + trailing item",
path: "len(arr)[0]",
wantErr: true,
},
{
name: "length of nested path within array element",
path: "len(arr[42].foo)",
expected: query.Path{
Length: true,
Elements: []query.PathElement{
{Name: "arr", Index: u64(42)},
{Name: "foo"},
},
},
wantErr: false,
},
{
name: "empty spaces in path",
path: "data . target",
wantErr: true,
},
{
name: "leading dot + empty spaces",
path: ". data",
wantErr: true,
},
{
name: "length with leading dot + empty spaces",
path: "len(. data)",
wantErr: true,
},
{
name: "Empty path error",
path: "",
expected: query.Path{},
},
{
name: "length with leading dot + empty spaces",
path: "test))((",
wantErr: true,
},
{
name: "length with leading dot + empty spaces",
path: "array][0][",
wantErr: true,
},
}
for _, tt := range tests {
@@ -41,12 +301,12 @@ func TestParsePath(t *testing.T) {
parsedPath, err := query.ParsePath(tt.path)
if tt.wantErr {
require.NotNil(t, err, "Expected error but got none")
require.NotNil(t, err, "Expected error did not occur")
return
}
require.NoError(t, err)
require.Equal(t, len(tt.expected), len(parsedPath), "Expected %d path elements, got %d", len(tt.expected), len(parsedPath))
require.Equal(t, len(tt.expected.Elements), len(parsedPath.Elements), "Expected %d path elements, got %d", len(tt.expected.Elements), len(parsedPath.Elements))
require.DeepEqual(t, tt.expected, parsedPath, "Parsed path does not match expected path")
})
}

View File

@@ -7,19 +7,19 @@ import (
// CalculateOffsetAndLength calculates the offset and length of a given path within the SSZ object.
// By walking the given path, it accumulates the offsets based on SszInfo.
func CalculateOffsetAndLength(sszInfo *SszInfo, path []PathElement) (*SszInfo, uint64, uint64, error) {
func CalculateOffsetAndLength(sszInfo *SszInfo, path Path) (*SszInfo, uint64, uint64, error) {
if sszInfo == nil {
return nil, 0, 0, errors.New("sszInfo is nil")
}
if len(path) == 0 {
if len(path.Elements) == 0 {
return nil, 0, 0, errors.New("path is empty")
}
walk := sszInfo
offset := uint64(0)
for pathIndex, elem := range path {
for pathIndex, elem := range path.Elements {
containerInfo, err := walk.ContainerInfo()
if err != nil {
return nil, 0, 0, fmt.Errorf("could not get field infos: %w", err)
@@ -56,7 +56,7 @@ func CalculateOffsetAndLength(sszInfo *SszInfo, path []PathElement) (*SszInfo, u
// to the next field's sszInfo, which would have the correct size information.
// However, if this is the last element in the path, we need to ensure we return the correct size
// for the indexed element. Hence, we return the size from elementSizes.
if pathIndex == len(path)-1 {
if pathIndex == len(path.Elements)-1 {
return walk, offset, listInfo.elementSizes[index], nil
}
} else {

View File

@@ -57,3 +57,8 @@ func (t SSZType) String() string {
return fmt.Sprintf("Unknown(%d)", t)
}
}
// isBasic returns true if the SSZType is a basic type.
func (t SSZType) isBasic() bool {
return t == Uint8 || t == Uint16 || t == Uint32 || t == Uint64 || t == Boolean
}

2
go.mod
View File

@@ -177,7 +177,7 @@ require (
github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect
github.com/libp2p/go-msgio v0.3.0 // indirect
github.com/libp2p/go-nat v0.2.0 // indirect
github.com/libp2p/go-netroute v0.2.2 // indirect
github.com/libp2p/go-netroute v0.4.0 // indirect
github.com/libp2p/go-reuseport v0.4.0 // indirect
github.com/libp2p/go-yamux/v4 v4.0.2 // indirect
github.com/lunixbochs/vtclean v1.0.0 // indirect

4
go.sum
View File

@@ -599,8 +599,8 @@ github.com/libp2p/go-msgio v0.3.0 h1:mf3Z8B1xcFN314sWX+2vOTShIE0Mmn2TXn3YCUQGNj0
github.com/libp2p/go-msgio v0.3.0/go.mod h1:nyRM819GmVaF9LX3l03RMh10QdOroF++NBbxAb0mmDM=
github.com/libp2p/go-nat v0.2.0 h1:Tyz+bUFAYqGyJ/ppPPymMGbIgNRH+WqC5QrT5fKrrGk=
github.com/libp2p/go-nat v0.2.0/go.mod h1:3MJr+GRpRkyT65EpVPBstXLvOlAPzUVlG6Pwg9ohLJk=
github.com/libp2p/go-netroute v0.2.2 h1:Dejd8cQ47Qx2kRABg6lPwknU7+nBnFRpko45/fFPuZ8=
github.com/libp2p/go-netroute v0.2.2/go.mod h1:Rntq6jUAH0l9Gg17w5bFGhcC9a+vk4KNXs6s7IljKYE=
github.com/libp2p/go-netroute v0.4.0 h1:sZZx9hyANYUx9PZyqcgE/E1GUG3iEtTZHUEvdtXT7/Q=
github.com/libp2p/go-netroute v0.4.0/go.mod h1:Nkd5ShYgSMS5MUKy/MU2T57xFoOKvvLR92Lic48LEyA=
github.com/libp2p/go-reuseport v0.4.0 h1:nR5KU7hD0WxXCJbmw7r2rhRYruNRl2koHw8fQscQm2s=
github.com/libp2p/go-reuseport v0.4.0/go.mod h1:ZtI03j/wO5hZVDFo2jKywN6bYKWLOy8Se6DrI2E1cLU=
github.com/libp2p/go-yamux/v4 v4.0.2 h1:nrLh89LN/LEiqcFiqdKDRHjGstN300C1269K/EX0CPU=

View File

@@ -1,6 +1,7 @@
package util
import (
"slices"
"testing"
"time"
@@ -66,13 +67,7 @@ func assertNoHooks(t *testing.T, logger *logrus.Logger) {
func assertRegistered(t *testing.T, logger *logrus.Logger, hook ComparableHook) {
for _, lvl := range hook.Levels() {
registered := logger.Hooks[lvl]
found := false
for _, h := range registered {
if hook.Equal(h) {
found = true
break
}
}
found := slices.ContainsFunc(registered, hook.Equal)
require.Equal(t, true, found, "Expected hook %v to be registered at level %s, but it was not", hook, lvl.String())
}
}

View File

@@ -6,6 +6,7 @@ import (
"errors"
"go/ast"
"go/token"
"slices"
"strconv"
"strings"
"unicode"
@@ -178,12 +179,7 @@ func isLoggingCall(call *ast.CallExpr, logFunctions []string, aliases map[string
// isCommonLogPackage checks for common logging package names
func isCommonLogPackage(pkg string) bool {
common := []string{"log", "logrus", "zerolog", "zap", "glog", "klog"}
for _, c := range common {
if pkg == c {
return true
}
}
return false
return slices.Contains(common, pkg)
}
// isFormatFunction checks if this is a format function (ending with 'f')
@@ -274,10 +270,8 @@ func isAcceptableStart(firstRune rune, s string) bool {
// Special characters that are OK to start with
acceptableChars := []rune{'%', '$', '/', '\\', '[', '(', '{', '"', '\'', '`', '-'}
for _, char := range acceptableChars {
if firstRune == char {
return true
}
if slices.Contains(acceptableChars, firstRune) {
return true
}
// URLs/paths are OK

View File

@@ -10,6 +10,7 @@ import (
"github.com/OffchainLabs/prysm/v6/network/httputil"
ethpb "github.com/OffchainLabs/prysm/v6/proto/prysm/v1alpha1"
"github.com/OffchainLabs/prysm/v6/runtime/version"
"github.com/OffchainLabs/prysm/v6/time/slots"
"github.com/pkg/errors"
)
@@ -67,7 +68,8 @@ func (c *beaconApiValidatorClient) proposeAttestationElectra(ctx context.Context
if err != nil {
return nil, err
}
headers := map[string]string{"Eth-Consensus-Version": version.String(attestation.Version())}
consensusVersion := version.String(slots.ToForkVersion(attestation.Data.Slot))
headers := map[string]string{"Eth-Consensus-Version": consensusVersion}
if err = c.jsonRestHandler.Post(
ctx,
"/eth/v2/beacon/pool/attestations",

View File

@@ -8,11 +8,14 @@ import (
"testing"
"github.com/OffchainLabs/prysm/v6/beacon-chain/core/helpers"
"github.com/OffchainLabs/prysm/v6/config/params"
"github.com/OffchainLabs/prysm/v6/consensus-types/primitives"
"github.com/OffchainLabs/prysm/v6/network/httputil"
ethpb "github.com/OffchainLabs/prysm/v6/proto/prysm/v1alpha1"
"github.com/OffchainLabs/prysm/v6/runtime/version"
"github.com/OffchainLabs/prysm/v6/testing/assert"
"github.com/OffchainLabs/prysm/v6/testing/require"
"github.com/OffchainLabs/prysm/v6/time/slots"
"github.com/OffchainLabs/prysm/v6/validator/client/beacon-api/mock"
testhelpers "github.com/OffchainLabs/prysm/v6/validator/client/beacon-api/test-helpers"
"go.uber.org/mock/gomock"
@@ -214,36 +217,58 @@ func TestProposeAttestationFallBack(t *testing.T) {
}
func TestProposeAttestationElectra(t *testing.T) {
attestation := &ethpb.SingleAttestation{
AttesterIndex: 74,
Data: &ethpb.AttestationData{
Slot: 75,
CommitteeIndex: 76,
BeaconBlockRoot: testhelpers.FillByteSlice(32, 38),
Source: &ethpb.Checkpoint{
Epoch: 78,
Root: testhelpers.FillByteSlice(32, 79),
params.SetupTestConfigCleanup(t)
params.BeaconConfig().ElectraForkEpoch = 0
params.BeaconConfig().FuluForkEpoch = 1
buildSingleAttestation := func(slot primitives.Slot) *ethpb.SingleAttestation {
targetEpoch := slots.ToEpoch(slot)
sourceEpoch := targetEpoch
if targetEpoch > 0 {
sourceEpoch = targetEpoch - 1
}
return &ethpb.SingleAttestation{
AttesterIndex: 74,
Data: &ethpb.AttestationData{
Slot: slot,
CommitteeIndex: 76,
BeaconBlockRoot: testhelpers.FillByteSlice(32, 38),
Source: &ethpb.Checkpoint{
Epoch: sourceEpoch,
Root: testhelpers.FillByteSlice(32, 79),
},
Target: &ethpb.Checkpoint{
Epoch: targetEpoch,
Root: testhelpers.FillByteSlice(32, 81),
},
},
Target: &ethpb.Checkpoint{
Epoch: 80,
Root: testhelpers.FillByteSlice(32, 81),
},
},
Signature: testhelpers.FillByteSlice(96, 82),
CommitteeId: 83,
Signature: testhelpers.FillByteSlice(96, 82),
CommitteeId: 83,
}
}
attestationElectra := buildSingleAttestation(0)
attestationFulu := buildSingleAttestation(params.BeaconConfig().SlotsPerEpoch)
tests := []struct {
name string
attestation *ethpb.SingleAttestation
expectedErrorMessage string
endpointError error
endpointCall int
name string
attestation *ethpb.SingleAttestation
expectedConsensusVersion string
expectedErrorMessage string
endpointError error
endpointCall int
}{
{
name: "valid",
attestation: attestation,
endpointCall: 1,
name: "valid electra",
attestation: attestationElectra,
expectedConsensusVersion: version.String(slots.ToForkVersion(attestationElectra.GetData().GetSlot())),
endpointCall: 1,
},
{
name: "valid fulu consensus version",
attestation: attestationFulu,
expectedConsensusVersion: version.String(slots.ToForkVersion(attestationFulu.GetData().GetSlot())),
endpointCall: 1,
},
{
name: "nil attestation",
@@ -283,8 +308,11 @@ func TestProposeAttestationElectra(t *testing.T) {
expectedErrorMessage: "attestation's target can't be nil",
},
{
name: "bad request",
attestation: attestation,
name: "bad request",
attestation: attestationElectra,
expectedConsensusVersion: version.String(
slots.ToForkVersion(attestationElectra.GetData().GetSlot()),
),
expectedErrorMessage: "bad request",
endpointError: errors.New("bad request"),
endpointCall: 1,
@@ -304,11 +332,14 @@ func TestProposeAttestationElectra(t *testing.T) {
}
ctx := t.Context()
headers := map[string]string{"Eth-Consensus-Version": version.String(test.attestation.Version())}
headerMatcher := gomock.Any()
if test.expectedConsensusVersion != "" {
headerMatcher = gomock.Eq(map[string]string{"Eth-Consensus-Version": test.expectedConsensusVersion})
}
jsonRestHandler.EXPECT().Post(
gomock.Any(),
"/eth/v2/beacon/pool/attestations",
headers,
headerMatcher,
bytes.NewBuffer(marshalledAttestations),
nil,
).Return(
@@ -325,7 +356,7 @@ func TestProposeAttestationElectra(t *testing.T) {
require.NoError(t, err)
require.NotNil(t, proposeResponse)
expectedAttestationDataRoot, err := attestation.Data.HashTreeRoot()
expectedAttestationDataRoot, err := test.attestation.Data.HashTreeRoot()
require.NoError(t, err)
// Make sure that the attestation data root is set

View File

@@ -10,6 +10,7 @@ import (
"github.com/OffchainLabs/prysm/v6/network/httputil"
ethpb "github.com/OffchainLabs/prysm/v6/proto/prysm/v1alpha1"
"github.com/OffchainLabs/prysm/v6/runtime/version"
"github.com/OffchainLabs/prysm/v6/time/slots"
"github.com/pkg/errors"
)
@@ -54,7 +55,9 @@ func (c *beaconApiValidatorClient) submitSignedAggregateSelectionProofElectra(ct
if err != nil {
return nil, errors.Wrap(err, "failed to marshal SignedAggregateAttestationAndProofElectra")
}
headers := map[string]string{"Eth-Consensus-Version": version.String(in.SignedAggregateAndProof.Version())}
dataSlot := in.SignedAggregateAndProof.Message.Aggregate.Data.Slot
consensusVersion := version.String(slots.ToForkVersion(dataSlot))
headers := map[string]string{"Eth-Consensus-Version": consensusVersion}
if err = c.jsonRestHandler.Post(ctx, "/eth/v2/validator/aggregate_and_proofs", headers, bytes.NewBuffer(body), nil); err != nil {
return nil, err
}

View File

@@ -7,11 +7,13 @@ import (
"testing"
"github.com/OffchainLabs/prysm/v6/api/server/structs"
"github.com/OffchainLabs/prysm/v6/config/params"
"github.com/OffchainLabs/prysm/v6/network/httputil"
ethpb "github.com/OffchainLabs/prysm/v6/proto/prysm/v1alpha1"
"github.com/OffchainLabs/prysm/v6/runtime/version"
"github.com/OffchainLabs/prysm/v6/testing/assert"
"github.com/OffchainLabs/prysm/v6/testing/require"
"github.com/OffchainLabs/prysm/v6/time/slots"
"github.com/OffchainLabs/prysm/v6/validator/client/beacon-api/mock"
testhelpers "github.com/OffchainLabs/prysm/v6/validator/client/beacon-api/test-helpers"
"github.com/pkg/errors"
@@ -123,6 +125,10 @@ func TestSubmitSignedAggregateSelectionProof_Fallback(t *testing.T) {
}
func TestSubmitSignedAggregateSelectionProofElectra_Valid(t *testing.T) {
params.SetupTestConfigCleanup(t)
params.BeaconConfig().ElectraForkEpoch = 0
params.BeaconConfig().FuluForkEpoch = 100
ctrl := gomock.NewController(t)
defer ctrl.Finish()
@@ -131,7 +137,8 @@ func TestSubmitSignedAggregateSelectionProofElectra_Valid(t *testing.T) {
require.NoError(t, err)
ctx := t.Context()
headers := map[string]string{"Eth-Consensus-Version": version.String(signedAggregateAndProofElectra.Message.Version())}
expectedVersion := version.String(slots.ToForkVersion(signedAggregateAndProofElectra.Message.Aggregate.Data.Slot))
headers := map[string]string{"Eth-Consensus-Version": expectedVersion}
jsonRestHandler := mock.NewMockJsonRestHandler(ctrl)
jsonRestHandler.EXPECT().Post(
gomock.Any(),
@@ -155,6 +162,10 @@ func TestSubmitSignedAggregateSelectionProofElectra_Valid(t *testing.T) {
}
func TestSubmitSignedAggregateSelectionProofElectra_BadRequest(t *testing.T) {
params.SetupTestConfigCleanup(t)
params.BeaconConfig().ElectraForkEpoch = 0
params.BeaconConfig().FuluForkEpoch = 100
ctrl := gomock.NewController(t)
defer ctrl.Finish()
@@ -163,7 +174,8 @@ func TestSubmitSignedAggregateSelectionProofElectra_BadRequest(t *testing.T) {
require.NoError(t, err)
ctx := t.Context()
headers := map[string]string{"Eth-Consensus-Version": version.String(signedAggregateAndProofElectra.Message.Version())}
expectedVersion := version.String(slots.ToForkVersion(signedAggregateAndProofElectra.Message.Aggregate.Data.Slot))
headers := map[string]string{"Eth-Consensus-Version": expectedVersion}
jsonRestHandler := mock.NewMockJsonRestHandler(ctrl)
jsonRestHandler.EXPECT().Post(
gomock.Any(),
@@ -182,6 +194,43 @@ func TestSubmitSignedAggregateSelectionProofElectra_BadRequest(t *testing.T) {
assert.ErrorContains(t, "bad request", err)
}
func TestSubmitSignedAggregateSelectionProofElectra_FuluVersion(t *testing.T) {
params.SetupTestConfigCleanup(t)
params.BeaconConfig().ElectraForkEpoch = 0
params.BeaconConfig().FuluForkEpoch = 1
ctrl := gomock.NewController(t)
defer ctrl.Finish()
signedAggregateAndProofElectra := generateSignedAggregateAndProofElectraJson()
marshalledSignedAggregateSignedAndProofElectra, err := json.Marshal([]*structs.SignedAggregateAttestationAndProofElectra{jsonifySignedAggregateAndProofElectra(signedAggregateAndProofElectra)})
require.NoError(t, err)
ctx := t.Context()
expectedVersion := version.String(slots.ToForkVersion(signedAggregateAndProofElectra.Message.Aggregate.Data.Slot))
headers := map[string]string{"Eth-Consensus-Version": expectedVersion}
jsonRestHandler := mock.NewMockJsonRestHandler(ctrl)
jsonRestHandler.EXPECT().Post(
gomock.Any(),
"/eth/v2/validator/aggregate_and_proofs",
headers,
bytes.NewBuffer(marshalledSignedAggregateSignedAndProofElectra),
nil,
).Return(
nil,
).Times(1)
attestationDataRoot, err := signedAggregateAndProofElectra.Message.Aggregate.Data.HashTreeRoot()
require.NoError(t, err)
validatorClient := &beaconApiValidatorClient{jsonRestHandler: jsonRestHandler}
resp, err := validatorClient.submitSignedAggregateSelectionProofElectra(ctx, &ethpb.SignedAggregateSubmitElectraRequest{
SignedAggregateAndProof: signedAggregateAndProofElectra,
})
require.NoError(t, err)
assert.DeepEqual(t, attestationDataRoot[:], resp.AttestationDataRoot)
}
func generateSignedAggregateAndProofJson() *ethpb.SignedAggregateAttestationAndProof {
return &ethpb.SignedAggregateAttestationAndProof{
Message: &ethpb.AggregateAttestationAndProof{