* HTTP proxy server for Eth2 APIs (#8904)
* Implement API HTTP proxy server
* cleanup + more comments
* gateway will no longer be dependent on beaconv1
* handle error during ErrorJson type assertion
* simplify handling of endpoint data
* fix mux v1 route
* use URL encoding for all requests
* comment fieldProcessor
* fix failing test
* change proxy port to not interfere with e2e
* gzl
* simplify conditional expression
* Move appending custom error header to grpcutils package
* add api-middleware-port flag
* fix documentation for processField
* modify e2e port
* change field processing error message
* better error message for field processing
* simplify base64ToHexProcessor
* fix json structs
* Run several new endpoints through API middleware (#8922)
* Implement API HTTP proxy server
* cleanup + more comments
* gateway will no longer be dependent on beaconv1
* handle error during ErrorJson type assertion
* simplify handling of endpoint data
* fix mux v1 route
* use URL encoding for all requests
* comment fieldProcessor
* fix failing test
* change proxy port to not interfere with e2e
* gzl
* simplify conditional expression
* Move appending custom error header to grpcutils package
* add api-middleware-port flag
* fix documentation for processField
* modify e2e port
* change field processing error message
* better error message for field processing
* simplify base64ToHexProcessor
* fix json structs
* /eth/v1/beacon/states/{state_id}/validators
* /eth/v1/beacon/states/{state_id}/validators/{validator_id}
* /eth/v1/beacon/states/{state_id}/validator_balances
* /eth/v1/beacon/states/{state_id}/committees
* allow skipping base64-encoding for query params
* /eth/v1/beacon/pool/attestations
* replace break with continue
* Remove unused functions (#8924)
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Process SSZ-serialized beacon state through API middleware (#8925)
* update field names
* Process SSZ-serialized beacon state through API middleware
* revert changes to go.mod and go.sum
* Revert "Merge branch '__develop' into feature/api-middleware"
This reverts commit 7c739a8fd7, reversing
changes made to 2d0f8e012e.
* update ethereumapis
* update validator field name
* update deps.bzl
* update json tags (#8942)
* Run `/node/syncing` through API Middleware (#8944)
* add IsSyncing field to grpc response
* run /node/syncing through the middleware
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Return HTTP status codes other than 200 and 500 from node and debug endpoints (#8937)
* error codes for node endpoints
* error codes for debug endpoints
* better comment about headers
* gzl
* review comments
* comment on return value
* update fakeChecker used for fuzz tests
* fix failing tests
* Allow to pass URL params literally, without encoding to base64 (#8938)
* Allow to pass URL params literally, without encoding to base64
* fix compile error
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Process SSZ-serialized beacon state through API middleware (#8925)
* update field names
* Process SSZ-serialized beacon state through API middleware
* revert changes to go.mod and go.sum
* Revert "Merge branch '__develop' into feature/api-middleware"
This reverts commit 7c739a8fd7, reversing
changes made to 2d0f8e012e.
* update ethereumapis
* update validator field name
* update deps.bzl
* update json tags (#8942)
* Run `/node/syncing` through API Middleware (#8944)
* add IsSyncing field to grpc response
* run /node/syncing through the middleware
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Return HTTP status codes other than 200 and 500 from node and debug endpoints (#8937)
* error codes for node endpoints
* error codes for debug endpoints
* better comment about headers
* gzl
* review comments
* comment on return value
* update fakeChecker used for fuzz tests
* fix failing tests
* Allow to pass URL params literally, without encoding to base64 (#8938)
* Allow to pass URL params literally, without encoding to base64
* fix compile error
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* unused import
* Return correct status codes from beacon endpoints (#8960)
* Various API Middleware fixes (#8963)
* Return correct status codes from `/states` endpoints
* better error messages in debug and node
* better error messages in state
* returning correct error codes from validator endpoints
* correct error codes for getting a block header
* gzl
* fix err variable name
* fix nil block comparison
* test fixes
* make status enum test better
* fix ineffectual assignment
* make PR unstuck
* return proper status codes
* return uppercase keys from /config/spec
* return lowercase validator status
* convert requested enum values to uppercase
* validator fixes
* Implement `/beacon/headers` endpoint (#8966)
* Refactor API Middleware into more manageable code (#8984)
* move endpoint registration out of shared package
* divide main function into smaller components
* return early on error
* implement hooks
* implement custom handlers and add documentation
* fix test compile error
* restrict package visibility
* remove redundant error checking
* rename file
* API Middleware unit tests (#8998)
* move endpoint registration out of shared package
* divide main function into smaller components
* return early on error
* implement hooks
* implement custom handlers and add documentation
* fix test compile error
* restrict package visibility
* remove redundant error checking
* rename file
* api_middleware_processing
* endpoints
* gzl
* remove gazelle:ignore
* merge
* Implement SSZ version of `/blocks/{block_id}` (#8970)
* Implement SSZ version of `/blocks/{block_id}`
* add dependencies back
* fix indentation in deps.bzl
* parameterize ssz functions
* get block ssz
* update ethereumapis dependency
* gzl
* Do not reuse `Endpoint` structs between API calls (#9007)
* code refactor
* implement endpoint factory
* fix test
* fmt
* include pbs
* gaz
* test naming fixes
* remove unused code
* radek comments
* revert endpoint test
* bring back bytes test case
* move `signedBeaconBlock` to `migration` package
* change `fmt.Errorf` to `errors.Wrap`
* capitalize SSZ
* capitalize URL
* more review feedback
* rename `handleGetBlockSSZ` to `handleGetBeaconBlockSSZ`
* rename `IndexOutOfRangeError` to `ValidatorIndexOutOfRangeError`
* simplify parameter names
* test header
* more corrections
* properly allocate array capacity
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
* Revert "Rename `NewService` to `New` (#8337)"
This reverts commit d121b19145.
# Conflicts:
# beacon-chain/sync/initial-sync/round_robin_test.go
* fix name in test
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* remove unused cache states map
* correct typo
* Remove unused array
* Add lock around deposits cache chainstart pubkeys
* Copy attestation before grabbing lock. This may reduce lock contention time as other callers wanting the lock do not need to wait as long for the lock to become available.
* Copy attestation before grabbing lock. This may reduce lock contention time as other callers wanting the lock do not need to wait as long for the lock to become available.
* Set capacity to 1 since it is known that the slice will be 1 after insertion
* require validatorSlashingPreconditionCheck caller to hold lock
* Add lock for voluntary exits pool HasBeenIncluded
* Require rate limiter retrieveCollector to hold lock
* Add lock requirement assertions in sync
* Remove unused struct
* remove ClearCachedStates API
* field initSyncState is unused
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Add, use and test `VerifyWeakSubjectivityRoot`
* Comments
* Make bazel test happy
* Merge branch 'master' into verify-ws
* Add flag to help
* Merge branch 'verify-ws' of github.com:prysmaticlabs/prysm into verify-ws
* Merge refs/heads/master into verify-ws
* Merge refs/heads/master into verify-ws
* Merge refs/heads/master into verify-ws
* Merge branch 'master' of github.com:prysmaticlabs/prysm into verify-ws
* Update beacon-chain/blockchain/service.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update beacon-chain/blockchain/weak_subjectivity_checks.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Merge branch 'verify-ws' of github.com:prysmaticlabs/prysm into verify-ws
* s/&&/|| for VerifyWeakSubjectivityRoot circuit breaker
* Merge refs/heads/master into verify-ws
* Merge refs/heads/master into verify-ws
* removes redundant code
* removes redundant code
* removes redundant db
* removes redundant p2p
* servive - test future genesis
* more tests for service.Start()
* service - already synced test
* Merge branch 'master' into init-sync-more-service-tests
* Merge branch 'master' into init-sync-more-service-tests
* NewInitialSync to New (avoid stutter)
* TestService_markSynced
* more assertions
* TestService_Resync
* renames test
* Merge branch 'master' into init-sync-more-service-tests
* Merge refs/heads/master into init-sync-more-service-tests
* Merge branch 'master' into init-sync-more-service-tests
* Merge refs/heads/master into init-sync-more-service-tests
* Merge branch 'master' into init-sync-more-service-tests
* fixes e2e test
* Merge branch 'init-sync-more-service-tests' of github.com:prysmaticlabs/prysm into init-sync-more-service-tests
* Merge refs/heads/master into init-sync-more-service-tests
* Merge refs/heads/master into init-sync-more-service-tests
* Merge refs/heads/master into init-sync-more-service-tests
* Merge refs/heads/master into init-sync-more-service-tests
* New -> NewInitialSync
* refactor state initialization into waitForStateInitialization
* more tests
* redundant blank line
* Merge branch 'master' into init-sync-refactor-state-init
* moves comment up
* Nishant's suggestions + bit more refactoring
* Explicitly process error (Nishant's suggestion)
* removes duplicate code
* Merge branch 'master' into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* Merge branch 'master' into init-sync-queue-dedup-blocks
* removes notification
* Merge branch 'init-sync-queue-dedup-blocks' of github.com:prysmaticlabs/prysm into init-sync-queue-dedup-blocks
* adds test
* updates service
* get rid of deepequal for err check
* Merge refs/heads/master into init-sync-queue-dedup-blocks
* implements weighted round robin in init-sync
* protection against evil peer
* reshuffle
* shorten flag
* deterministic order of wrr
* Merge branch 'master' into init-sync-wrr
* Merge refs/heads/master into init-sync-wrr
* Merge refs/heads/master into init-sync-wrr
* allow peers with the same capacity to pe selected at random
* Merge branch 'init-sync-wrr' of github.com:prysmaticlabs/prysm into init-sync-wrr
* adds wrr to e2e tests
* Merge refs/heads/master into init-sync-wrr
* Merge refs/heads/master into init-sync-wrr
* Merge refs/heads/master into init-sync-wrr
* refactors block fetching function to fetcher
* more comments fixed
* Merge refs/heads/master into init-sync-upd
* Merge refs/heads/master into init-sync-upd
* Merge refs/heads/master into init-sync-upd
* Enforce error handling and checking type assertions
* Reference issue #5404 in the TODO message
* doc description
* Merge branch 'master' into errcheck
* fix tests and address @nisdas feedbacK
* gaz
* fix docker image
* Clear initial sync state caches after round robin sync
* fix test mock
* lint
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* begin state service
* begin on the state trie idea
* created beacon state structure
* add in the full clone getter
* return by value instead
* add all setters
* new state setters are being completed
* arrays roots exposed
* close to finishing all these headerssss
* functionality complete
* added in proto benchmark test
* test for compatibility
* add test for compat
* comments fixed
* add clone
* add clone
* remove underlying copies
* make it immutable
* integrate it into chainservice
* revert
* wrap up comments for package
* address all comments and godocs
* address all comments
* clone the pending attestation properly
* properly clone remaining items
* tests pass fixed bug
* begin using it instead of head state
* prevent nil pointer exceptions
* begin using new struct in db
* integrated new type into db package
* add proper nil checks
* using new state in archiver
* refactored much of core
* editing all the precompute functions
* done with most core refactor
* fixed up some bugs in the clone comparisons
* append current epoch atts
* add missing setters
* add new setters
* fix other core methods
* fix up transition
* main service and forkchoice
* fix rpc
* integrated to powchain
* some more changes
* fix build
* improve processing of deposits
* fix error
* prevent panic
* comment
* fix process att
* gaz
* fix up att process
* resolve existing review comments
* resolve another batch of gh comments
* resolve broken cpt state
* revise testutil to use the new state
* begin updating the state transition func to pass in more compartmentalized args
* finish editing transition function to return errors
* block operations pretty much done with refactor
* state transition fully refactored
* got epoch processing completed
* fix build in fork choice
* fixing more of the build
* fix up broken sync package
* it builds nowww it buildssss
* revert registry changes
* Recompute on Read (#4627)
* compute on read
* fix up eth1 data votes
* looking into slashings bug introduced in core/
* able to advance more slots
* add logging
* can now sync with testnet yay
* remove the leaves algorithm and other merkle imports
* expose initialize unsafe funcs
* Update beacon-chain/db/kv/state.go
* lint
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* More Optimizations for New State (#4641)
* map optimization
* more optimizations
* use a custom hasher
* comment
* block operations optimizations
* Update beacon-chain/state/types.go
Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>
* fixed up various operations to use the validator index map access
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* archiver tests pass
* fixing cache tests
* cache tests passing
* edited validator tests
* powchain tests passing
* halfway thru sync tests
* more sync test fixes
* add in tests for state/
* working through rpc tests
* assignments tests passed
* almost done with rpc/beacon tests
* resolved painful validator test
* fixed up even more tests
* resolve tests
* fix build
* reduce a randao mixes copy
* fixes under //beacon-chain/blockchain/...
* build //beacon-chain/core/...
* fixes
* Runtime Optimizations (#4648)
* parallelize shuffling
* clean up
* lint
* fix build
* use callback to read from registry
* fix array roots and size map
* new improvements
* reduce hash allocs
* improved shuffling
* terence's review
* use different method
* raul's comment
* new array roots
* remove clone in pre-compute
* Update beacon-chain/state/types.go
Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>
* raul's review
* lint
* fix build issues
* fix visibility
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* fix visibility
* build works for all
* fix blockchain test
* fix a few tests
* fix more tests
* update validator in slashing
* archiver passing
* fixed rpc/validator
* progress on core tests
* resolve broken rpc tests
* blockchain tests passed
* fix up some tests in core
* fix message diff
* remove unnecessary save
* Save validator after slashing
* Update validators one by one
* another update
* fix everything
* fix more precompute tests
* fix blocks tests
* more elegant fix
* more helper fixes
* change back ?
* fix test
* fix skip slot
* fix test
* reset caches
* fix testutil
* raceoff fixed
* passing
* Retrieve cached state in the beginning
* lint
* Fixed tests part 1
* Fixed rest of the tests
* Minor changes to avoid copying, small refactor to reduce deplicated code
* Handle att req for slot 0
* New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689)
* Only populate merkle layers as needed, copy merkle layers on copy/clone.
* use custom copy
* Make maps of correct size
* slightly fast, doesn't wait for lock
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Target root can't be 0x00
* Don't use cache for current slot (may not be the right fix)
* fixed up tests
* Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing...
* Align with prev logic for process slots cachedState.Slot() < slot
* Fix Initial Sync Flag (#4692)
* fixes
* fix up some test failures due to lack of nil checks
* fix up some test failures due to lack of nil checks
* fix up imports
* revert some changes
* imports
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* resolving further conflicts
* Better skip slot cache (#4694)
* Return copy of skip slot cache state, disable skip slot cache on sync
* fix
* Fix pruning
* fix up issues with broken tests
Co-authored-by: Nishant Das <nish1993@hotmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: shayzluf <thezluf@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Use a client side rate limit to reduce chance of getting banned
* fix test
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
