**What type of PR is this?**
Optimisation (@nisdas)
**What does this PR do? Why is it needed?**
The beacon state contains the list of all validators that have been at
least once deposited on the network.
| Network | Active | Total | Ratio |
|----------|-----------|---------------|---------|
| Hoodi | 1,224,855 | **1,294,521** | 94.1% |
| Mainnet | 948,883 | **2,229,313** | 43.56% |
Currently, the validators are stored in the beacon state like this:
```go
type BeaconState struct {
...
validatorsMultiValue *MultiValueValidators
...
}
type MultiValueValidators = multi_value_slice.Slice[*ethpb.Validator]
type Validator struct {
state protoimpl.MessageState `protogen:"open.v1"`
PublicKey []byte `protobuf:"bytes,1,opt,name=public_key,json=publicKey,proto3" json:"public_key,omitempty" spec-name:"pubkey" ssz-size:"48"`
WithdrawalCredentials []byte `protobuf:"bytes,2,opt,name=withdrawal_credentials,json=withdrawalCredentials,proto3" json:"withdrawal_credentials,omitempty" ssz-size:"32"`
EffectiveBalance uint64 `protobuf:"varint,3,opt,name=effective_balance,json=effectiveBalance,proto3" json:"effective_balance,omitempty"`
Slashed bool `protobuf:"varint,4,opt,name=slashed,proto3" json:"slashed,omitempty"`
ActivationEligibilityEpoch github_com_OffchainLabs_prysm_v7_consensus_types_primitives.Epoch `protobuf:"varint,5,opt,name=activation_eligibility_epoch,json=activationEligibilityEpoch,proto3" json:"activation_eligibility_epoch,omitempty" cast-type:"github.com/OffchainLabs/prysm/v7/consensus-types/primitives.Epoch"`
ActivationEpoch github_com_OffchainLabs_prysm_v7_consensus_types_primitives.Epoch `protobuf:"varint,6,opt,name=activation_epoch,json=activationEpoch,proto3" json:"activation_epoch,omitempty" cast-type:"github.com/OffchainLabs/prysm/v7/consensus-types/primitives.Epoch"`
ExitEpoch github_com_OffchainLabs_prysm_v7_consensus_types_primitives.Epoch `protobuf:"varint,7,opt,name=exit_epoch,json=exitEpoch,proto3" json:"exit_epoch,omitempty" cast-type:"github.com/OffchainLabs/prysm/v7/consensus-types/primitives.Epoch"`
WithdrawableEpoch github_com_OffchainLabs_prysm_v7_consensus_types_primitives.Epoch `protobuf:"varint,8,opt,name=withdrawable_epoch,json=withdrawableEpoch,proto3" json:"withdrawable_epoch,omitempty" cast-type:"github.com/OffchainLabs/prysm/v7/consensus-types/primitives.Epoch"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
```
Some fields, used only by protobuf (`state`, `unknownFields`,
`sizeCache`) are not used by the beacon node.
Some other fields, stored as slices (`PublicKey`,
`WithdrawalCredentials`) need extra memory (`ptr+len+cap`) while they
could be stored as arrays because their sizes are fixed and known in
advance.
We define a new custom type called `CompactValidator`, which is a
fixed-size, pointer-free representation of a validator:
```go
type CompactValidator struct {
PublicKey [fieldparams.BLSPubkeyLength]byte // 48 bytes
WithdrawalCredentials [32]byte // 32 bytes
EffectiveBalance uint64 // 8 bytes
Slashed bool // 1 byte
ActivationEligibilityEpoch primitives.Epoch // 8 bytes
ActivationEpoch primitives.Epoch // 8 bytes
ExitEpoch primitives.Epoch // 8 bytes
WithdrawableEpoch primitives.Epoch // 8 bytes
}
```
Here is the comparison, per validator, between storing a
`*ethpb.Validator` and a `CompactValidator`.
| Component | `*ethpb.Validator` | `CompactValidator` | Wasted |
|------------------------------------------|--------------------|--------------------|---------|
| Pointer to struct | 8 B | 0 B | 8 B |
| `state` (protoimpl.MessageState) | 8 B | 0 B | 8 B |
| `PublicKey` slice header (ptr+len+cap) | 24 B | 0 B | 24 B |
| `PublicKey` backing array (heap) | 48 B | 48 B (inline) | 0 B |
| `PublicKey` malloc overhead | ~16 B | 0 B | ~16 B |
| `WithdrawalCredentials` slice header | 24 B | 0 B | 24 B |
| `WithdrawalCredentials` backing array | 32 B | 32 B (inline) | 0 B |
| `WithdrawalCredentials` malloc overhead | ~16 B | 0 B | ~16 B |
| `EffectiveBalance` (uint64) | 8 B | 8 B | 0 B |
| `Slashed` (bool + padding) | 8 B | 8 B | 0 B |
| `ActivationEligibilityEpoch` | 8 B | 8 B | 0 B |
| `ActivationEpoch` | 8 B | 8 B | 0 B |
| `ExitEpoch` | 8 B | 8 B | 0 B |
| `WithdrawableEpoch` | 8 B | 8 B | 0 B |
| `unknownFields` ([]byte header) | 24 B | 0 B | 24 B |
| `sizeCache` (int32 + padding) | 8 B | 0 B | 8 B |
| Struct malloc overhead | ~16 B | 0 B | ~16 B |
| **Total** | **~264 B** | **128 B** | **~136 B (52%)** |
We waste `~136 B` per validator.
With a total of `2,229,313` validators (mainnet), it represents `~290
MB`.
Storing `CompactValidator` instead of `*ethpb.Validator` also reduces
the garbage collection pressure.
The following graph represents, for a Hoodi supernode (200 validators
managed):
1. The heap memory usage (`go_memstats_alloc_bytes`) without this PR
(`ref`) and with this PR (`current`)
2. `1.`, but averaged over the latest four hours.
3. The diff of the graphs in `2`. (The bigger, the better)
> [!NOTE]
> The improvement is, after stabilisation, **~300MB-450MB**,
representing **~8%-11%**.
>
> <img width="1028" height="917" alt="image"
src="https://github.com/user-attachments/assets/9179d9b1-bc50-4248-9f47-82a7646d58ab"
/>
>
> For some reasons, the gain is higher than initialy expected.
**For the reviewers:**
This PR should be reviewed commit by commit:
Commits 1 and 2 are independent and unrelated to this PR
Commit 3 is an oversight of:
- https://github.com/OffchainLabs/prysm/pull/16420
Commit 4 implements the `CompactValidator`
Commit 5 uses the `CompactValidator`. The key change is:
```go
// MultiValueValidators is a multi-value slice of compact validators.
type MultiValueValidators = multi_value_slice.Slice[stateutil.CompactValidator]
```
**Acknowledgements**
- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
---------
Co-authored-by: Bastin <43618253+Inspector-Butters@users.noreply.github.com>
**What type of PR is this?**
Optimisation
**What does this PR do? Why is it needed?**
Use `InitializeFromProtoUnsafeXXX` instead of `InitializeFromProtoXXX`
when possible. (@nisdas' change)
This change avoids useless copies.
The difference between these two methods is the latest copies the input
state. However, if the input state is guaranteed not to be modified
after the use of this function (and, in particular, if the input state
is not used at all after the use of this function), then copying it is
unnecessarry. ==> Using the "unsafe" method is fine in this case.
**Other notes for review**
As a reviewer, you should ensure that when the unsafe method is used,
then the input state is never modified after the use of the function.
This PR is much easier to review directly in VSCode, because it displays
more context by default.
**Acknowledgements**
- [x] I have read
[CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md).
- [x] I have included a uniquely named [changelog fragment
file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd).
- [x] I have added a description with sufficient context for reviewers
to understand this PR.
- [x] I have tested that my changes work as expected and I added a
testing plan to the PR description (if applicable).
- moves deposit-related helpers (deposit signature verification, batch
verification, merkle proof verification, and activation helper) from
`beacon-chain/core/blocks` into `beacon-chain/core/helpers`
- updates call sites (Altair/Electra) to use helpers
Why?
- In gloas, the blocks package needs to call into gloas logic (e.g.
clearing builder pending payments/withdrawals on proposer slashing)
- gloas also introduces deposit-request processing which needs deposit
signature verification previously located in blocks.
That creates a Bazel/Go dependency cycle (blocks -> gloas -> blocks)
- the natural layering is for blocks and fork logic to depend on a lower
level util package for deposit verification, so moving deposit helpers
to core/helpers breaks the cycle
* Migrate Prysm repo to Offchain Labs organization ahead of Pectra upgrade v6
* Replace prysmaticlabs with OffchainLabs on general markdowns
* Update mock
* Gazelle and add mock.go to excluded generated mock file
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Remove native state flag and use native state in spectests
* remove feature from tests
* use e2e config in slasher simulator
* use params.BeaconConfig in testutil
* use correct function
* use minimal config in go_test
* fix TestListValidators
* parameterize sync committee bits and aggregation bits
* Fix TestServer_ListIndexedAttestations_GenesisEpoch
(cherry picked from commit 254ab623dde08ae8886b152facdbbd8889ed79db)
* fix more tests
* fix even more
* moreeee
* aaaand more
* one more fix
* one more
* simplify TestGetAltairDuties_UnknownPubkey
* comment out problematic test
* one more fix
* one more
* aaaand one more
* another
* use fieldparams in HydrateBlindedBeaconBlockBodyBellatrix
* create new package for mainnet tests
* TestServer_GetBellatrixBeaconBlock
* change slashed validator index
* clear cache in reward_test.go
* deprecate flag
* create bazel mainnet target
* move attester mainnet test to mainnet target
* "fix" proposer tests
* use minimal config in TestServer_circuitBreakBuilder
* fix TestProposer_ProposeBlock_OK
* more fixes in validator package
* more fixes
* more fixes
* test code
* move TestProposer_GetBeaconBlock_BellatrixEpoch to minimal
* finally
* remove proposer_bellatrix_mainnet_test.go
* fix TestServer_GetBellatrixBeaconBlock_HappyCase
* fix TestServer_GetBellatrixBeaconBlock_BuilderCase
* Preston needs to fix this!
* Revert "Preston needs to fix this!"
This reverts commit b03d97a16e.
* remove proto state tests
* fix migration tests
* static analysis fix
* review
* remove proto state
* swap state in tests
* fix BUILD file in /proto/testing
* remove metrics test with nil state
* panic in SizeSSZ
* moving slowly
* adapt old code to new interfaces
* return interfaces from factory functions
* replace the rest of WrappedSignedBeaconBlock
* WrappedBeaconBlock
* WrappedBeaconBlockBody
* miscellaneous
* Test_BeaconBlockIsNil
* replace usages of BeaconBlockIsNil
* replace usages of mutator
* fix all build errors
* fix some more issues
* mutator changes
* relax assertions when initializing
* revert changes in object_mapping.go
* allow calling Proto on nil
* Revert "allow calling Proto on nil"
This reverts commit ecc84e4553.
* modify Copy and Proto methods
* remove unused var
* fix block batch tests
* correct BUILD file
* Error when initializing nil objects
* one more error fix
* add missing comma
* rename alias to blocktest
* add logging
* error when SignedBeaconBlock is nil
* fix last test
* import fix
* broken
* working
* test fixes
* reduce complexity of processPendingBlocks
* simplified
* replace eth2 types
* replace protos
* regen proto
* replace
* gaz
* deps
* amend
* regen proto
* mod
* gaz
* gaz
* ensure build
* ssz
* add dep
* no more eth2 types
* no more eth2
* remg
* all builds
* buidl
* tidy
* clean
* fmt
* val serv
* gaz
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Move domain function and all signing root functions from beacon-chain/core/helpers to beacon-chain/core
* @terencechain suggestion to put these methods under core/signing
