mirror of
https://github.com/OffchainLabs/prysm.git
synced 2026-02-07 19:45:16 -05:00
<!-- Thanks for sending a PR! Before submitting: 1. If this is your first PR, check out our contribution guide here https://docs.prylabs.network/docs/contribute/contribution-guidelines You will then need to sign our Contributor License Agreement (CLA), which will show up as a comment from a bot in this pull request after you open it. We cannot review code without a signed CLA. 2. Please file an associated tracking issue if this pull request is non-trivial and requires context for our team to understand. All features and most bug fixes should have an associated issue with a design discussed and decided upon. Small bug fixes and documentation improvements don't need issues. 3. New features and bug fixes must have tests. Documentation may need to be updated. If you're unsure what to update, send the PR, and we'll discuss in review. 4. Note that PRs updating dependencies and new Go versions are not accepted. Please file an issue instead. 5. A changelog entry is required for user facing issues. --> **What type of PR is this?** Feature **What does this PR do? Why is it needed?** This PR replaces the previous PR https://github.com/OffchainLabs/prysm/pull/16121, which built the entire Merkle tree and generated proofs only after the tree was complete. In this PR, the Merkle proof is produced by collecting hashes while the Merkle tree is being built. This approach has proven to be more efficient than the one in https://github.com/OffchainLabs/prysm/pull/16121. - **ProofCollector**: - New `ProofCollector` type in `encoding/ssz/query/proof_collector.go`: Collects sibling hashes and leaves needed for Merkle proofs during merkleization. - Multiproof-ready design with `requiredSiblings`/`requiredLeaves` maps for registering target gindices before merkleization. - Thread-safe: read-only required maps during merkleization, mutex-protected writes to `siblings`/`leaves`. - `AddTarget(gindex)` registers a target leaf and computes all required sibling gindices along the path to root. - `toProof()` converts collected data into `fastssz.Proof` structure. - Parallel execution in `merkleizeVectorBody` for composite elements with worker pool pattern. - Optimized container hashing: Generalized `stateutil.OptimizedValidatorRoots` pattern for any SSZ container type: - `optimizedContainerRoots`: Parallelized field root computation + level-by-level vectorized hashing via `VectorizedSha256`. - `hashContainerHelper`: Worker goroutine for processing container subsets. - `containerFieldRoots`: Computes field roots for a single container using reflection and SszInfo metadata. - **`Prove(gindex)` method** in `encoding/ssz/query/merkle_proof.go`: Entry point for generating SSZ Merkle proofs for a given generalized index. - **Testing** - Added `merkle_proof_test.go` and `proof_collector_test.go` to test and benchmark this feature. The main outcomes of the optimizations are here: ``` ❯ go test ./encoding/ssz/query -run=^$ -bench='Benchmark(OptimizedContainerRoots|OptimizedValidatorRoots|ProofCollectorMerkleize)$' -benchmem goos: darwin goarch: arm64 pkg: github.com/OffchainLabs/prysm/v7/encoding/ssz/query cpu: Apple M2 Pro BenchmarkOptimizedValidatorRoots-10 3237 361029 ns/op 956858 B/op 6024 allocs/op BenchmarkOptimizedContainerRoots-10 1138 969002 ns/op 3245223 B/op 11024 allocs/op BenchmarkProofCollectorMerkleize-10 522 2262066 ns/op 3216000 B/op 19000 allocs/op PASS ok github.com/OffchainLabs/prysm/v7/encoding/ssz/query 4.619s ``` Knowing that `OptimizedValidatorRoots` implements very effective optimizations, `OptimizedContainerRoots` mimics them. In the benchmark we can see that `OptimizedValidatorRoots` remain as the most performant and tit the baseline here: - `ProofCollectorMerkleize` is **~6.3× slower**, uses **~3.4× more memory** (B/op), and performs **~3.2× more allocations**. - `OptimizedContainerRoots` sits in between: it’s **~2.7× slower** than `OptimizedValidatorRoots` (and **~3.4× higher B/op**, **~1.8× more allocations**), but it is a clear win over `ProofCollectorMerkleize` for lists/vectors: **~2.3× faster** with **~1.7× fewer allocations** (and essentially the same memory footprint). The main drawback is that `OptimizedContainerRoots` can only be applied to vector/list subtrees where we don’t need to collect any sibling/leaf data (i.e., no proof targets within that subtree); integrating it into the recursive merkleize(...) flow when targets are outside the subtree is expected to land in a follow-up PR. **Which issues(s) does this PR fix?** Partially https://github.com/OffchainLabs/prysm/issues/15598 **Other notes for review** In this [write-up](https://hackmd.io/@fernantho/BJbZ1xmmbg), I depict the process to come up with this solution. Future improvements: - Defensive check that the gindex is not too big, depicted [here]( https://github.com/OffchainLabs/prysm/pull/16177#discussion_r2671684100). - Integrate optimizedContainerRoots into the recursive merkleize(...) flow when proof targets are not within the subtree (skip full traversal for container lists). - Add multiproofs. - Connect `proofCollector` to SSZ-QL endpoints (direct integration of `proofCollector` for BeaconBlock endpoint and "hybrid" approach for BeaconState endpoint). **Acknowledgements** - [x] I have read [CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md). - [x] I have included a uniquely named [changelog fragment file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd). - [x] I have added a description with sufficient context for reviewers to understand this PR. - [x] I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable). --------- Co-authored-by: Radosław Kapka <radoslaw.kapka@gmail.com> Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>
181 lines
5.2 KiB
Go
181 lines
5.2 KiB
Go
package bytesutil
|
|
|
|
import (
|
|
"encoding/binary"
|
|
"errors"
|
|
"fmt"
|
|
"math/big"
|
|
|
|
"github.com/OffchainLabs/prysm/v7/math"
|
|
)
|
|
|
|
// ToBytes returns integer x to bytes in little-endian format at the specified length.
|
|
// Spec defines similar method uint_to_bytes(n: uint) -> bytes, which is equivalent to ToBytes(n, 8).
|
|
func ToBytes(x uint64, length int) []byte {
|
|
if length < 0 {
|
|
length = 0
|
|
}
|
|
makeLength := max(length, 8)
|
|
bytes := make([]byte, makeLength)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes[:length]
|
|
}
|
|
|
|
// Bytes1 returns integer x to bytes in little-endian format, x.to_bytes(1, 'little').
|
|
func Bytes1(x uint64) []byte {
|
|
bytes := make([]byte, 8)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes[:1]
|
|
}
|
|
|
|
// Bytes2 returns integer x to bytes in little-endian format, x.to_bytes(2, 'little').
|
|
func Bytes2(x uint64) []byte {
|
|
bytes := make([]byte, 8)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes[:2]
|
|
}
|
|
|
|
// Bytes3 returns integer x to bytes in little-endian format, x.to_bytes(3, 'little').
|
|
func Bytes3(x uint64) []byte {
|
|
bytes := make([]byte, 8)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes[:3]
|
|
}
|
|
|
|
// Bytes4 returns integer x to bytes in little-endian format, x.to_bytes(4, 'little').
|
|
func Bytes4(x uint64) []byte {
|
|
bytes := make([]byte, 8)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes[:4]
|
|
}
|
|
|
|
// Bytes8 returns integer x to bytes in little-endian format, x.to_bytes(8, 'little').
|
|
func Bytes8(x uint64) []byte {
|
|
bytes := make([]byte, 8)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes
|
|
}
|
|
|
|
// Bytes32 returns integer x to bytes in little-endian format, x.to_bytes(32, 'little').
|
|
func Bytes32(x uint64) []byte {
|
|
bytes := make([]byte, 32)
|
|
binary.LittleEndian.PutUint64(bytes, x)
|
|
return bytes
|
|
}
|
|
|
|
// FromBytes2 returns an integer which is stored in the little-endian format(2, 'little')
|
|
// from a byte array.
|
|
func FromBytes2(x []byte) uint16 {
|
|
if len(x) < 2 {
|
|
return 0
|
|
}
|
|
return binary.LittleEndian.Uint16(x[:2])
|
|
}
|
|
|
|
// FromBytes4 returns an integer which is stored in the little-endian format(4, 'little')
|
|
// from a byte array.
|
|
func FromBytes4(x []byte) uint64 {
|
|
if len(x) < 4 {
|
|
return 0
|
|
}
|
|
empty4bytes := make([]byte, 4)
|
|
return binary.LittleEndian.Uint64(append(x[:4], empty4bytes...))
|
|
}
|
|
|
|
// FromBytes8 returns an integer which is stored in the little-endian format(8, 'little')
|
|
// from a byte array.
|
|
func FromBytes8(x []byte) uint64 {
|
|
if len(x) < 8 {
|
|
return 0
|
|
}
|
|
return binary.LittleEndian.Uint64(x)
|
|
}
|
|
|
|
// ToLowInt64 returns the lowest 8 bytes interpreted as little endian.
|
|
func ToLowInt64(x []byte) int64 {
|
|
if len(x) < 8 {
|
|
return 0
|
|
}
|
|
// Use the first 8 bytes.
|
|
x = x[:8]
|
|
return int64(binary.LittleEndian.Uint64(x)) // lint:ignore uintcast -- A negative number might be the expected result.
|
|
}
|
|
|
|
// Uint32ToBytes4 is a convenience method for converting uint32 to a fix
|
|
// sized 4 byte array in big endian order. Returns 4 byte array.
|
|
func Uint32ToBytes4(i uint32) [4]byte {
|
|
buf := make([]byte, 4)
|
|
binary.BigEndian.PutUint32(buf, i)
|
|
return ToBytes4(buf)
|
|
}
|
|
|
|
// Uint64ToBytesLittleEndian conversion.
|
|
func Uint64ToBytesLittleEndian(i uint64) []byte {
|
|
buf := make([]byte, 8)
|
|
binary.LittleEndian.PutUint64(buf, i)
|
|
return buf
|
|
}
|
|
|
|
// Uint64ToBytesLittleEndian32 conversion of a uint64 to a fix
|
|
// sized 32 byte array in little endian order. Returns 32 byte array.
|
|
func Uint64ToBytesLittleEndian32(i uint64) []byte {
|
|
buf := make([]byte, 32)
|
|
binary.LittleEndian.PutUint64(buf, i)
|
|
return buf
|
|
}
|
|
|
|
// Uint64ToBytesBigEndian conversion.
|
|
func Uint64ToBytesBigEndian(i uint64) []byte {
|
|
buf := make([]byte, 8)
|
|
binary.BigEndian.PutUint64(buf, i)
|
|
return buf
|
|
}
|
|
|
|
// BytesToUint64BigEndian conversion. Returns 0 if empty bytes or byte slice with length less
|
|
// than 8.
|
|
func BytesToUint64BigEndian(b []byte) uint64 {
|
|
if len(b) < 8 { // This will panic otherwise.
|
|
return 0
|
|
}
|
|
return binary.BigEndian.Uint64(b)
|
|
}
|
|
|
|
// LittleEndianBytesToBigInt takes bytes of a number stored as little-endian and returns a big integer
|
|
func LittleEndianBytesToBigInt(bytes []byte) *big.Int {
|
|
// Integers are stored as little-endian, but big.Int expects big-endian. So we need to reverse the byte order before decoding.
|
|
return new(big.Int).SetBytes(ReverseByteOrder(bytes))
|
|
}
|
|
|
|
// BigIntToLittleEndianBytes takes a big integer and returns its bytes stored as little-endian
|
|
func BigIntToLittleEndianBytes(bigInt *big.Int) []byte {
|
|
// big.Int.Bytes() returns bytes in big-endian order, so we need to reverse the byte order
|
|
return ReverseByteOrder(bigInt.Bytes())
|
|
}
|
|
|
|
// Uint256ToSSZBytes takes a string representation of uint256 and returns its bytes stored as little-endian
|
|
func Uint256ToSSZBytes(num string) ([]byte, error) {
|
|
uint256, ok := new(big.Int).SetString(num, 10)
|
|
if !ok {
|
|
return nil, errors.New("could not parse Uint256")
|
|
}
|
|
if !math.IsValidUint256(uint256) {
|
|
return nil, fmt.Errorf("%s is not a valid Uint256", num)
|
|
}
|
|
return PadTo(ReverseByteOrder(uint256.Bytes()), 32), nil
|
|
}
|
|
|
|
// PutLittleEndian writes an unsigned integer value in little-endian format.
|
|
// Supports sizes 1, 2, 4, or 8 bytes for uint8/16/32/64 respectively.
|
|
func PutLittleEndian(dst []byte, val uint64, size int) {
|
|
switch size {
|
|
case 1:
|
|
dst[0] = byte(val)
|
|
case 2:
|
|
binary.LittleEndian.PutUint16(dst, uint16(val))
|
|
case 4:
|
|
binary.LittleEndian.PutUint32(dst, uint32(val))
|
|
case 8:
|
|
binary.LittleEndian.PutUint64(dst, val)
|
|
}
|
|
}
|