Files
prysm/encoding/bytesutil/integers.go
fernantho 0476eeda57 SSZ-QL: custom Generic Merkle Proofs building the tree and collecting the hashes in one sweep (#16177)
<!-- Thanks for sending a PR! Before submitting:

1. If this is your first PR, check out our contribution guide here
https://docs.prylabs.network/docs/contribute/contribution-guidelines
You will then need to sign our Contributor License Agreement (CLA),
which will show up as a comment from a bot in this pull request after
you open it. We cannot review code without a signed CLA.
2. Please file an associated tracking issue if this pull request is
non-trivial and requires context for our team to understand. All
features and most bug fixes should have
an associated issue with a design discussed and decided upon. Small bug
   fixes and documentation improvements don't need issues.
3. New features and bug fixes must have tests. Documentation may need to
be updated. If you're unsure what to update, send the PR, and we'll
discuss
   in review.
4. Note that PRs updating dependencies and new Go versions are not
accepted.
   Please file an issue instead.
5. A changelog entry is required for user facing issues.
-->

**What type of PR is this?**
Feature

**What does this PR do? Why is it needed?**
This PR replaces the previous PR
https://github.com/OffchainLabs/prysm/pull/16121, which built the entire
Merkle tree and generated proofs only after the tree was complete. In
this PR, the Merkle proof is produced by collecting hashes while the
Merkle tree is being built. This approach has proven to be more
efficient than the one in
https://github.com/OffchainLabs/prysm/pull/16121.

- **ProofCollector**: 
- New `ProofCollector` type in `encoding/ssz/query/proof_collector.go`:
Collects sibling hashes and leaves needed for Merkle proofs during
merkleization.
- Multiproof-ready design with `requiredSiblings`/`requiredLeaves` maps
for registering target gindices before merkleization.
- Thread-safe: read-only required maps during merkleization,
mutex-protected writes to `siblings`/`leaves`.
- `AddTarget(gindex)` registers a target leaf and computes all required
sibling gindices along the path to root.
- `toProof()` converts collected data into `fastssz.Proof` structure.
- Parallel execution in `merkleizeVectorBody` for composite elements
with worker pool pattern.
- Optimized container hashing: Generalized
`stateutil.OptimizedValidatorRoots` pattern for any SSZ container type:
- `optimizedContainerRoots`: Parallelized field root computation +
level-by-level vectorized hashing via `VectorizedSha256`.
- `hashContainerHelper`: Worker goroutine for processing container
subsets.
- `containerFieldRoots`: Computes field roots for a single container
using reflection and SszInfo metadata.

- **`Prove(gindex)` method** in `encoding/ssz/query/merkle_proof.go`:
Entry point for generating SSZ Merkle proofs for a given generalized
index.

- **Testing**
- Added `merkle_proof_test.go` and `proof_collector_test.go` to test and
benchmark this feature.

The main outcomes of the optimizations are here:
```
❯ go test ./encoding/ssz/query -run=^$ -bench='Benchmark(OptimizedContainerRoots|OptimizedValidatorRoots|ProofCollectorMerkleize)$' -benchmem
goos: darwin
goarch: arm64
pkg: github.com/OffchainLabs/prysm/v7/encoding/ssz/query
cpu: Apple M2 Pro
BenchmarkOptimizedValidatorRoots-10         3237            361029 ns/op          956858 B/op       6024 allocs/op
BenchmarkOptimizedContainerRoots-10         1138            969002 ns/op         3245223 B/op      11024 allocs/op
BenchmarkProofCollectorMerkleize-10          522           2262066 ns/op         3216000 B/op      19000 allocs/op
PASS
ok      github.com/OffchainLabs/prysm/v7/encoding/ssz/query     4.619s
```
Knowing that `OptimizedValidatorRoots` implements very effective
optimizations, `OptimizedContainerRoots` mimics them.
In the benchmark we can see that `OptimizedValidatorRoots` remain as the
most performant and tit the baseline here:
- `ProofCollectorMerkleize` is **~6.3× slower**, uses **~3.4× more
memory** (B/op), and performs **~3.2× more allocations**.
- `OptimizedContainerRoots` sits in between: it’s **~2.7× slower** than
`OptimizedValidatorRoots` (and **~3.4× higher B/op**, **~1.8× more
allocations**), but it is a clear win over `ProofCollectorMerkleize` for
lists/vectors: **~2.3× faster** with **~1.7× fewer allocations** (and
essentially the same memory footprint).

The main drawback is that `OptimizedContainerRoots` can only be applied
to vector/list subtrees where we don’t need to collect any sibling/leaf
data (i.e., no proof targets within that subtree); integrating it into
the recursive merkleize(...) flow when targets are outside the subtree
is expected to land in a follow-up PR.

**Which issues(s) does this PR fix?**
Partially https://github.com/OffchainLabs/prysm/issues/15598

**Other notes for review**
In this [write-up](https://hackmd.io/@fernantho/BJbZ1xmmbg), I depict
the process to come up with this solution.

Future improvements:
- Defensive check that the gindex is not too big, depicted [here](
https://github.com/OffchainLabs/prysm/pull/16177#discussion_r2671684100).
- Integrate optimizedContainerRoots into the recursive merkleize(...)
flow when proof targets are not within the subtree (skip full traversal
for container lists).
- Add multiproofs.
- Connect `proofCollector` to SSZ-QL endpoints (direct integration of
`proofCollector` for BeaconBlock endpoint and "hybrid" approach for
BeaconState endpoint).

**Acknowledgements**

- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).

---------

Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com>
Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
2026-01-28 13:01:22 +00:00

181 lines
5.2 KiB
Go

package bytesutil
import (
"encoding/binary"
"errors"
"fmt"
"math/big"
"github.com/OffchainLabs/prysm/v7/math"
)
// ToBytes returns integer x to bytes in little-endian format at the specified length.
// Spec defines similar method uint_to_bytes(n: uint) -> bytes, which is equivalent to ToBytes(n, 8).
func ToBytes(x uint64, length int) []byte {
if length < 0 {
length = 0
}
makeLength := max(length, 8)
bytes := make([]byte, makeLength)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:length]
}
// Bytes1 returns integer x to bytes in little-endian format, x.to_bytes(1, 'little').
func Bytes1(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:1]
}
// Bytes2 returns integer x to bytes in little-endian format, x.to_bytes(2, 'little').
func Bytes2(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:2]
}
// Bytes3 returns integer x to bytes in little-endian format, x.to_bytes(3, 'little').
func Bytes3(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:3]
}
// Bytes4 returns integer x to bytes in little-endian format, x.to_bytes(4, 'little').
func Bytes4(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:4]
}
// Bytes8 returns integer x to bytes in little-endian format, x.to_bytes(8, 'little').
func Bytes8(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes
}
// Bytes32 returns integer x to bytes in little-endian format, x.to_bytes(32, 'little').
func Bytes32(x uint64) []byte {
bytes := make([]byte, 32)
binary.LittleEndian.PutUint64(bytes, x)
return bytes
}
// FromBytes2 returns an integer which is stored in the little-endian format(2, 'little')
// from a byte array.
func FromBytes2(x []byte) uint16 {
if len(x) < 2 {
return 0
}
return binary.LittleEndian.Uint16(x[:2])
}
// FromBytes4 returns an integer which is stored in the little-endian format(4, 'little')
// from a byte array.
func FromBytes4(x []byte) uint64 {
if len(x) < 4 {
return 0
}
empty4bytes := make([]byte, 4)
return binary.LittleEndian.Uint64(append(x[:4], empty4bytes...))
}
// FromBytes8 returns an integer which is stored in the little-endian format(8, 'little')
// from a byte array.
func FromBytes8(x []byte) uint64 {
if len(x) < 8 {
return 0
}
return binary.LittleEndian.Uint64(x)
}
// ToLowInt64 returns the lowest 8 bytes interpreted as little endian.
func ToLowInt64(x []byte) int64 {
if len(x) < 8 {
return 0
}
// Use the first 8 bytes.
x = x[:8]
return int64(binary.LittleEndian.Uint64(x)) // lint:ignore uintcast -- A negative number might be the expected result.
}
// Uint32ToBytes4 is a convenience method for converting uint32 to a fix
// sized 4 byte array in big endian order. Returns 4 byte array.
func Uint32ToBytes4(i uint32) [4]byte {
buf := make([]byte, 4)
binary.BigEndian.PutUint32(buf, i)
return ToBytes4(buf)
}
// Uint64ToBytesLittleEndian conversion.
func Uint64ToBytesLittleEndian(i uint64) []byte {
buf := make([]byte, 8)
binary.LittleEndian.PutUint64(buf, i)
return buf
}
// Uint64ToBytesLittleEndian32 conversion of a uint64 to a fix
// sized 32 byte array in little endian order. Returns 32 byte array.
func Uint64ToBytesLittleEndian32(i uint64) []byte {
buf := make([]byte, 32)
binary.LittleEndian.PutUint64(buf, i)
return buf
}
// Uint64ToBytesBigEndian conversion.
func Uint64ToBytesBigEndian(i uint64) []byte {
buf := make([]byte, 8)
binary.BigEndian.PutUint64(buf, i)
return buf
}
// BytesToUint64BigEndian conversion. Returns 0 if empty bytes or byte slice with length less
// than 8.
func BytesToUint64BigEndian(b []byte) uint64 {
if len(b) < 8 { // This will panic otherwise.
return 0
}
return binary.BigEndian.Uint64(b)
}
// LittleEndianBytesToBigInt takes bytes of a number stored as little-endian and returns a big integer
func LittleEndianBytesToBigInt(bytes []byte) *big.Int {
// Integers are stored as little-endian, but big.Int expects big-endian. So we need to reverse the byte order before decoding.
return new(big.Int).SetBytes(ReverseByteOrder(bytes))
}
// BigIntToLittleEndianBytes takes a big integer and returns its bytes stored as little-endian
func BigIntToLittleEndianBytes(bigInt *big.Int) []byte {
// big.Int.Bytes() returns bytes in big-endian order, so we need to reverse the byte order
return ReverseByteOrder(bigInt.Bytes())
}
// Uint256ToSSZBytes takes a string representation of uint256 and returns its bytes stored as little-endian
func Uint256ToSSZBytes(num string) ([]byte, error) {
uint256, ok := new(big.Int).SetString(num, 10)
if !ok {
return nil, errors.New("could not parse Uint256")
}
if !math.IsValidUint256(uint256) {
return nil, fmt.Errorf("%s is not a valid Uint256", num)
}
return PadTo(ReverseByteOrder(uint256.Bytes()), 32), nil
}
// PutLittleEndian writes an unsigned integer value in little-endian format.
// Supports sizes 1, 2, 4, or 8 bytes for uint8/16/32/64 respectively.
func PutLittleEndian(dst []byte, val uint64, size int) {
switch size {
case 1:
dst[0] = byte(val)
case 2:
binary.LittleEndian.PutUint16(dst, uint16(val))
case 4:
binary.LittleEndian.PutUint32(dst, uint32(val))
case 8:
binary.LittleEndian.PutUint64(dst, val)
}
}