Files
prysm/shared/stateutil/validators.go
Raul Jordan cc741ed8af Ensure New State Type Tests Pass in Prysm (#4646)
* begin state service

* begin on the state trie idea

* created beacon state structure

* add in the full clone getter

* return by value instead

* add all setters

* new state setters are being completed

* arrays roots exposed

*  close to finishing all these headerssss

* functionality complete

* added in proto benchmark test

* test for compatibility

* add test for compat

* comments fixed

* add clone

* add clone

* remove underlying copies

* make it immutable

* integrate it into chainservice

* revert

* wrap up comments for package

* address all comments and godocs

* address all comments

* clone the pending attestation properly

* properly clone remaining items

* tests pass fixed bug

* begin using it instead of head state

* prevent nil pointer exceptions

* begin using new struct in db

* integrated new type into db package

* add proper nil checks

* using new state in archiver

* refactored much of core

* editing all the precompute functions

* done with most core refactor

* fixed up some bugs in the clone comparisons

* append current epoch atts

* add missing setters

* add new setters

* fix other core methods

* fix up transition

* main service and forkchoice

* fix rpc

* integrated to powchain

* some more changes

* fix build

* improve processing of deposits

* fix error

* prevent panic

* comment

* fix process att

* gaz

* fix up att process

* resolve existing review comments

* resolve another batch of gh comments

* resolve broken cpt state

* revise testutil to use the new state

* begin updating the state transition func to pass in more compartmentalized args

* finish editing transition function to return errors

* block operations pretty much done with refactor

* state transition fully refactored

* got epoch processing completed

* fix build in fork choice

* fixing more of the build

* fix up broken sync package

* it builds nowww it buildssss

* revert registry changes

* Recompute on Read (#4627)

* compute on read

* fix up eth1 data votes

* looking into slashings bug introduced in core/

* able to advance more slots

* add logging

* can now sync with testnet yay

* remove the leaves algorithm and other merkle imports

* expose initialize unsafe funcs

* Update beacon-chain/db/kv/state.go

* lint

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* More Optimizations for New State (#4641)

* map optimization

* more optimizations

* use a custom hasher

* comment

* block operations optimizations

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* fixed up various operations to use the validator index map access

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* archiver tests pass

* fixing cache tests

* cache tests passing

* edited validator tests

* powchain tests passing

* halfway thru sync tests

* more sync test fixes

* add in tests for state/

* working through rpc tests

* assignments tests passed

* almost done with rpc/beacon tests

* resolved painful validator test

* fixed up even more tests

* resolve tests

* fix build

* reduce a randao mixes copy

* fixes under //beacon-chain/blockchain/...

* build //beacon-chain/core/...

* fixes

* Runtime Optimizations (#4648)

* parallelize shuffling

* clean up

* lint

* fix build

* use callback to read from registry

* fix array roots and size map

* new improvements

* reduce hash allocs

* improved shuffling

* terence's review

* use different method

* raul's comment

* new array roots

* remove clone in pre-compute

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* raul's review

* lint

* fix build issues

* fix visibility

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* fix visibility

* build works for all

* fix blockchain test

* fix a few tests

* fix more tests

* update validator in slashing

* archiver passing

* fixed rpc/validator

* progress on core tests

* resolve broken rpc tests

* blockchain tests passed

* fix up some tests in core

* fix message diff

* remove unnecessary save

* Save validator after slashing

* Update validators one by one

* another update

* fix everything

* fix more precompute tests

* fix blocks tests

* more elegant fix

* more helper fixes

* change back ?

* fix test

* fix skip slot

* fix test

* reset caches

* fix testutil

* raceoff fixed

* passing

* Retrieve cached state in the beginning

* lint

* Fixed tests part 1

* Fixed rest of the tests

* Minor changes to avoid copying, small refactor to reduce deplicated code

* Handle att req for slot 0

* New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689)

* Only populate merkle layers as needed, copy merkle layers on copy/clone.

* use custom copy

* Make maps of correct size

* slightly fast, doesn't wait for lock

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Target root can't be 0x00

* Don't use cache for current slot (may not be the right fix)

* fixed up tests

* Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing...

* Align with prev logic for process slots cachedState.Slot() < slot

* Fix Initial Sync Flag (#4692)

* fixes

* fix up some test failures due to lack of nil checks

* fix up some test failures due to lack of nil checks

* fix up imports

* revert some changes

* imports

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* resolving further conflicts

* Better skip slot cache (#4694)

* Return copy of skip slot cache state, disable skip slot cache on sync

* fix

* Fix pruning

* fix up issues with broken tests

Co-authored-by: Nishant Das <nish1993@hotmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: shayzluf <thezluf@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-01-31 12:57:01 -08:00

188 lines
6.1 KiB
Go

package stateutil
import (
"bytes"
"encoding/binary"
"github.com/pkg/errors"
ethpb "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
"github.com/prysmaticlabs/prysm/shared/featureconfig"
"github.com/prysmaticlabs/prysm/shared/hashutil"
"github.com/prysmaticlabs/prysm/shared/params"
)
// ValidatorRegistryRoot computes the HashTreeRoot Merkleization of
// a list of validator structs according to the eth2
// Simple Serialize specification.
func ValidatorRegistryRoot(vals []*ethpb.Validator) ([32]byte, error) {
if featureconfig.Get().EnableSSZCache {
return cachedHasher.validatorRegistryRoot(vals)
}
return nocachedHasher.validatorRegistryRoot(vals)
}
// ValidatorBalancesRoot computes the HashTreeRoot Merkleization of
// a list of validator uint64 balances according to the eth2
// Simple Serialize specification.
func ValidatorBalancesRoot(balances []uint64) ([32]byte, error) {
balancesMarshaling := make([][]byte, 0)
for i := 0; i < len(balances); i++ {
balanceBuf := make([]byte, 8)
binary.LittleEndian.PutUint64(balanceBuf, balances[i])
balancesMarshaling = append(balancesMarshaling, balanceBuf)
}
balancesChunks, err := pack(balancesMarshaling)
if err != nil {
return [32]byte{}, errors.Wrap(err, "could not pack balances into chunks")
}
maxBalCap := params.BeaconConfig().ValidatorRegistryLimit
elemSize := uint64(8)
balLimit := (maxBalCap*elemSize + 31) / 32
if balLimit == 0 {
if len(balances) == 0 {
balLimit = 1
} else {
balLimit = uint64(len(balances))
}
}
balancesRootsRoot, err := bitwiseMerkleize(balancesChunks, uint64(len(balancesChunks)), balLimit)
if err != nil {
return [32]byte{}, errors.Wrap(err, "could not compute balances merkleization")
}
balancesRootsBuf := new(bytes.Buffer)
if err := binary.Write(balancesRootsBuf, binary.LittleEndian, uint64(len(balances))); err != nil {
return [32]byte{}, errors.Wrap(err, "could not marshal balances length")
}
balancesRootsBufRoot := make([]byte, 32)
copy(balancesRootsBufRoot, balancesRootsBuf.Bytes())
return mixInLength(balancesRootsRoot, balancesRootsBufRoot), nil
}
func (h *stateRootHasher) validatorRegistryRoot(validators []*ethpb.Validator) ([32]byte, error) {
hashKeyElements := make([]byte, len(validators)*32)
roots := make([][32]byte, len(validators))
emptyKey := hashutil.FastSum256(hashKeyElements)
bytesProcessed := 0
for i := 0; i < len(validators); i++ {
val, err := h.validatorRoot(validators[i])
if err != nil {
return [32]byte{}, errors.Wrap(err, "could not compute validators merkleization")
}
copy(hashKeyElements[bytesProcessed:bytesProcessed+32], val[:])
roots[i] = val
bytesProcessed += 32
}
hashKey := hashutil.FastSum256(hashKeyElements)
if hashKey != emptyKey && h.rootsCache != nil {
if found, ok := h.rootsCache.Get(string(hashKey[:])); found != nil && ok {
return found.([32]byte), nil
}
}
validatorsRootsRoot, err := bitwiseMerkleizeArrays(roots, uint64(len(roots)), params.BeaconConfig().ValidatorRegistryLimit)
if err != nil {
return [32]byte{}, errors.Wrap(err, "could not compute validator registry merkleization")
}
validatorsRootsBuf := new(bytes.Buffer)
if err := binary.Write(validatorsRootsBuf, binary.LittleEndian, uint64(len(validators))); err != nil {
return [32]byte{}, errors.Wrap(err, "could not marshal validator registry length")
}
// We need to mix in the length of the slice.
var validatorsRootsBufRoot [32]byte
copy(validatorsRootsBufRoot[:], validatorsRootsBuf.Bytes())
res := mixInLength(validatorsRootsRoot, validatorsRootsBufRoot[:])
if hashKey != emptyKey && h.rootsCache != nil {
h.rootsCache.Set(string(hashKey[:]), res, 32)
}
return res, nil
}
func (h *stateRootHasher) validatorRoot(validator *ethpb.Validator) ([32]byte, error) {
// Validator marshaling for caching.
enc := make([]byte, 122)
fieldRoots := make([][32]byte, 2, 8)
if validator != nil {
copy(enc[0:48], validator.PublicKey)
copy(enc[48:80], validator.WithdrawalCredentials)
effectiveBalanceBuf := [32]byte{}
binary.LittleEndian.PutUint64(effectiveBalanceBuf[:8], validator.EffectiveBalance)
copy(enc[80:88], effectiveBalanceBuf[:8])
if validator.Slashed {
enc[88] = uint8(1)
} else {
enc[88] = uint8(0)
}
activationEligibilityBuf := [32]byte{}
binary.LittleEndian.PutUint64(activationEligibilityBuf[:8], validator.ActivationEligibilityEpoch)
copy(enc[89:97], activationEligibilityBuf[:8])
activationBuf := [32]byte{}
binary.LittleEndian.PutUint64(activationBuf[:8], validator.ActivationEpoch)
copy(enc[97:105], activationBuf[:8])
exitBuf := [32]byte{}
binary.LittleEndian.PutUint64(exitBuf[:8], validator.ExitEpoch)
copy(enc[105:113], exitBuf[:8])
withdrawalBuf := [32]byte{}
binary.LittleEndian.PutUint64(withdrawalBuf[:8], validator.WithdrawableEpoch)
copy(enc[113:121], withdrawalBuf[:8])
// Check if it exists in cache:
if h.rootsCache != nil {
if found, ok := h.rootsCache.Get(string(enc)); found != nil && ok {
return found.([32]byte), nil
}
}
// Public key.
pubKeyChunks, err := pack([][]byte{validator.PublicKey})
if err != nil {
return [32]byte{}, err
}
pubKeyRoot, err := bitwiseMerkleize(pubKeyChunks, uint64(len(pubKeyChunks)), uint64(len(pubKeyChunks)))
if err != nil {
return [32]byte{}, err
}
fieldRoots[0] = pubKeyRoot
// Withdrawal credentials.
copy(fieldRoots[1][:], validator.WithdrawalCredentials)
// Effective balance.
fieldRoots = append(fieldRoots, effectiveBalanceBuf)
// Slashed.
slashBuf := [32]byte{}
if validator.Slashed {
slashBuf[0] = uint8(1)
} else {
slashBuf[0] = uint8(0)
}
fieldRoots = append(fieldRoots, slashBuf)
// Activation eligibility epoch.
fieldRoots = append(fieldRoots, activationEligibilityBuf)
// Activation epoch.
fieldRoots = append(fieldRoots, activationBuf)
// Exit epoch.
fieldRoots = append(fieldRoots, exitBuf)
// Withdrawable epoch.
fieldRoots = append(fieldRoots, withdrawalBuf)
}
valRoot, err := bitwiseMerkleizeArrays(fieldRoots, uint64(len(fieldRoots)), uint64(len(fieldRoots)))
if err != nil {
return [32]byte{}, err
}
if h.rootsCache != nil {
h.rootsCache.Set(string(enc), valRoot, 32)
}
return valRoot, nil
}