p2p/sync: Wrong genesis digest is used to register pubsub topic subscriptions (race condition) (#5500)

* Return error when genesis time or genesis root is not set
* Merge refs/heads/master into fail-on-unset-fork-digest
* move to own helpers
* make it non public
* Merge branch 'fail-on-unset-fork-digest' of https://github.com/prysmaticlabs/geth-sharding into fail-on-unset-fork-digest
* lint
* fix
* return error
* fix tests and error
* Merge refs/heads/master into fail-on-unset-fork-digest
* first round of test fixes
* second round of fixes
* Merge branch 'fail-on-unset-fork-digest' of https://github.com/prysmaticlabs/geth-sharding into fail-on-unset-fork-digest
* lint
* Merge refs/heads/master into fail-on-unset-fork-digest
* gaz
* Merge branch 'fail-on-unset-fork-digest' of https://github.com/prysmaticlabs/geth-sharding into fail-on-unset-fork-digest

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,59 @@
+---
+name: "\U0001F41EBug report"
+about: Report a bug or problem with running Prysm
+---
+<!--💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎
+
+Hellooo! 😄 
+
+To help us tend to your issue faster, please search our currently open issues before submitting a new one.
+Existing issues often contain information about workarounds, resolution, or progress updates.
+
+💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎-->
+
+# 🐞 Bug Report
+
+### Description
+
+<!-- ✍️--> A clear and concise description of the problem...
+
+### Has this worked before in a previous version?
+
+<!-- Did this behavior use to work in the previous version? -->
+<!-- ✍️--> Yes, the previous version in which this bug was not present was: ....
+
+## 🔬 Minimal Reproduction
+
+<!--
+Please let us know how we can reproduce this issue. Include the exact method you used to run Prysm along with any flags used in your beacon chain and/or validator. Make sure you don't upload any confidential files or private keys.
+-->
+
+## 🔥 Error
+
+<pre><code>
+<!-- If the issue is accompanied by an error, please share the error logs with us below. If you have a lot of logs, place make a paste bin with your logs and share the link with us here: -->
+<!-- ✍️-->
+
+</code></pre>
+
+
+## 🌍  Your Environment
+
+**Operating System:**
+
+<pre>
+  <code>
+
+  </code>
+</pre>
+
+**What version of Prysm are you running? (Which release)**
+
+<pre>
+  <code>
+
+  </code>
+</pre>
+
+**Anything else relevant (validator index / public key)?**
+

.github/ISSUE_TEMPLATE/discussion.md

@@ -0,0 +1,21 @@
+---
+name: "\U0001F48EGeneral issue / discussion"
+about: Any other type of general issue or discussion
+
+---
+<!--💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎
+
+Hellooo! 😄 
+
+To help us tend to your issue faster, please search our currently open issues before submitting a new one.
+Existing issues often contain information about workarounds, resolution, or progress updates.
+
+💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎-->
+
+# 💎 Issue
+
+### Background
+
+<!-- ✍️--> Context and background information on the discussion...
+
+### Description

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,27 @@
+---
+name: "\U0001F680Feature request"
+about: Suggest a feature for Prysm
+
+---
+<!--💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎
+
+Hellooo! 😄 
+
+To help us tend to your issue faster, please search our currently open issues before submitting a new one.
+Existing issues often contain information about workarounds, resolution, or progress updates.
+
+💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎-->
+
+# 🚀 Feature Request
+
+### Description
+
+<!-- ✍️--> A clear and concise description of the problem or missing capability...
+
+### Describe the solution you'd like
+
+<!-- ✍️--> If you have a solution in mind, please describe it.
+
+### Describe alternatives you've considered
+
+<!-- ✍️--> Have you considered any alternative solutions or workarounds?

.github/workflows/dappnode-release-trigger.yml

@@ -0,0 +1,41 @@
+name: Update DAppNodePackages
+
+on:
+  push:
+    tags:
+    - '*'
+
+jobs:
+  dappnode-update-beacon-chain:
+    name: Trigger a beacon-chain release
+    runs-on: ubuntu-latest
+    steps:
+    - name: Get latest tag
+      id: get_tag
+      run: echo ::set-output name=TAG::${GITHUB_REF/refs\/tags\//}
+    - name: Send dispatch event to DAppNodePackage-prysm-beacon-chain
+      env: 
+        DISPATCH_REPO: dappnode/DAppNodePackage-prysm-beacon-chain
+      run: |
+        curl -v -X POST -u "${{ secrets.PAT_GITHUB }}" \
+        -H "Accept: application/vnd.github.everest-preview+json" \
+        -H "Content-Type: application/json" \
+        --data '{"event_type":"new_release", "client_payload": { "tag":"${{ steps.get_tag.outputs.TAG }}"}}' \
+        https://api.github.com/repos/$DISPATCH_REPO/dispatches
+        
+  dappnode-update-validator:
+    name: Trigger a validator release
+    runs-on: ubuntu-latest
+    steps:
+    - name: Get latest tag
+      id: get_tag
+      run: echo ::set-output name=TAG::${GITHUB_REF/refs\/tags\//}
+    - name: Send dispatch event to DAppNodePackage validator repository
+      env: 
+        DISPATCH_REPO: dappnode/DAppNodePackage-prysm-validator
+      run: |
+        curl -v -X POST -u "${{ secrets.PAT_GITHUB }}" \
+        -H "Accept: application/vnd.github.everest-preview+json" \
+        -H "Content-Type: application/json" \
+        --data '{"event_type":"new_release", "client_payload": { "tag":"${{ steps.get_tag.outputs.TAG }}"}}' \
+        https://api.github.com/repos/$DISPATCH_REPO/dispatches

WORKSPACE

@@ -1508,74 +1508,89 @@ go_repository(
 
 go_repository(
     name = "com_github_wealdtech_go_eth2_wallet",
-    commit = "6970d62e60d86fdae3c3e510e800e8a60d755a7d",
     importpath = "github.com/wealdtech/go-eth2-wallet",
+    sum = "h1:Hna/w4EKBJIs86VprIq7ez063A6kwk31d/O3Gs+MpYc=",
+    version = "v1.9.3",
 )
 
 go_repository(
-    name = "com_github_wealdtech_go_eth2_wallet_hd",
-    commit = "ce0a252a01c621687e9786a64899cfbfe802ba73",
-    importpath = "github.com/wealdtech/go-eth2-wallet-hd",
+    name = "com_github_wealdtech_go_eth2_wallet_hd_v2",
+    importpath = "github.com/wealdtech/go-eth2-wallet-hd/v2",
+    sum = "h1:kiCvdexK3zRC2GwZHSHq+hS+irVNtMs5pNADyumeeRM=",
+    version = "v2.0.1",
 )
 
 go_repository(
-    name = "com_github_wealdtech_go_eth2_wallet_nd",
-    commit = "12c8c41cdbd16797ff292e27f58e126bb89e9706",
-    importpath = "github.com/wealdtech/go-eth2-wallet-nd",
+    name = "com_github_wealdtech_go_eth2_wallet_nd_v2",
+    importpath = "github.com/wealdtech/go-eth2-wallet-nd/v2",
+    sum = "h1:4lB6GY5oHQn5xwn/Sxm1e9SeVCaxa7q/0hqXUQYUNwU=",
+    version = "v2.0.1",
 )
 
 go_repository(
     name = "com_github_wealdtech_go_eth2_wallet_store_filesystem",
-    commit = "1eea6a48d75380047d2ebe7c8c4bd8985bcfdeca",
     importpath = "github.com/wealdtech/go-eth2-wallet-store-filesystem",
+    sum = "h1:Lc6wVTjIYeD+2hLAIzq1SugTWR527vEX4tEr5v3zxJc=",
+    version = "v1.7.2",
 )
 
 go_repository(
     name = "com_github_wealdtech_go_eth2_wallet_store_s3",
-    commit = "1c821b5161f7bb0b3efa2030eff687eea5e70e53",
     importpath = "github.com/wealdtech/go-eth2-wallet-store-s3",
+    sum = "h1:xzyQDxbe5nr7xG0ByevTV2S8qkeOZvvjp+leBJcpxXQ=",
+    version = "v1.6.2",
 )
 
 go_repository(
     name = "com_github_wealdtech_go_eth2_wallet_encryptor_keystorev4",
-    commit = "0c11c07b9544eb662210fadded94f40f309d8c8f",
     importpath = "github.com/wealdtech/go-eth2-wallet-encryptor-keystorev4",
+    sum = "h1:IcpS4VpXhYz+TVupB5n6C6IQzaKwG+Rc8nvgCa/da4c=",
+    version = "v1.0.0",
 )
 
 go_repository(
-    name = "com_github_wealdtech_go_eth2_wallet_types",
-    commit = "af67d8101be61e7c4dd8126d2b3eba20cff5dab2",
-    importpath = "github.com/wealdtech/go-eth2-wallet-types",
+    name = "com_github_wealdtech_go_eth2_wallet_types_v2",
+    importpath = "github.com/wealdtech/go-eth2-wallet-types/v2",
+    sum = "h1:Ct3RrNJTapBiG2GxVl53Kfgy96f0GEUV7bediTu91u8=",
+    version = "v2.0.1",
 )
 
 go_repository(
-    name = "com_github_wealdtech_go_eth2_types",
-    commit = "f9c31ddf180537dd5712d5998a3d56c45864d71f",
-    importpath = "github.com/wealdtech/go-eth2-types",
+    name = "com_github_wealdtech_go_eth2_types_v2",
+    importpath = "github.com/wealdtech/go-eth2-types/v2",
+    sum = "h1:qfmgaCBkH2N11LHCXsRWYz7OOxc+1QXrKHlS9yDnFsw=",
+    version = "v2.3.0",
+    build_directives = [
+        "gazelle:resolve go github.com/herumi/bls-eth-go-binary/bls @herumi_bls_eth_go_binary//:go_default_library",
+    ],
 )
 
 go_repository(
     name = "com_github_wealdtech_go_eth2_util",
-    commit = "326ebb1755651131bb8f4506ea9a23be6d9ad1dd",
     importpath = "github.com/wealdtech/go-eth2-util",
+    sum = "h1:MyM16V7Qhd9q2ZaRa0WteBg2bWb8UplIKjZr8aeBZP0=",
+    version = "v1.1.4",
 )
 
 go_repository(
     name = "com_github_wealdtech_go_ecodec",
-    commit = "7473d835445a3490e61a5fcf48fe4e9755a37957",
     importpath = "github.com/wealdtech/go-ecodec",
+    sum = "h1:yggrTSckcPJRaxxOxQF7FPm21kgE8WA6+f5jdq5Kr8o=",
+    version = "v1.1.0",
 )
 
 go_repository(
     name = "com_github_wealdtech_go_bytesutil",
-    commit = "e564d0ade555b9f97494f0f669196ddcc6bc531d",
     importpath = "github.com/wealdtech/go-bytesutil",
+    sum = "h1:ocEg3Ke2GkZ4vQw5lp46rmO+pfqCCTgq35gqOy8JKVc=",
+    version = "v1.1.1",
 )
 
 go_repository(
     name = "com_github_wealdtech_go_indexer",
-    commit = "334862c32b1e3a5c6738a2618f5c0a8ebeb8cd51",
     importpath = "github.com/wealdtech/go-indexer",
+    sum = "h1:/S4rfWQbSOnnYmwnvuTVatDibZ8o1s9bmTCHO16XINg=",
+    version = "v1.0.0",
 )
 
 go_repository(

beacon-chain/BUILD.bazel

@@ -20,6 +20,7 @@ go_library(
         "//shared/featureconfig:go_default_library",
         "//shared/logutil:go_default_library",
         "//shared/version:go_default_library",
+        "@com_github_ethereum_go_ethereum//log:go_default_library",
         "@com_github_ipfs_go_log//:go_default_library",
         "@com_github_joonix_log//:go_default_library",
         "@com_github_sirupsen_logrus//:go_default_library",
@@ -57,6 +58,7 @@ go_image(
         "//shared/featureconfig:go_default_library",
         "//shared/logutil:go_default_library",
         "//shared/version:go_default_library",
+        "@com_github_ethereum_go_ethereum//log:go_default_library",
         "@com_github_ipfs_go_log//:go_default_library",
         "@com_github_joonix_log//:go_default_library",
         "@com_github_sirupsen_logrus//:go_default_library",

beacon-chain/blockchain/chain_info.go

@@ -11,6 +11,7 @@ import (
 	"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
 	"github.com/prysmaticlabs/prysm/beacon-chain/state"
 	pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
+	"github.com/prysmaticlabs/prysm/shared/bytesutil"
 	"github.com/prysmaticlabs/prysm/shared/params"
 )
 
@@ -19,6 +20,7 @@ import (
 type ChainInfoFetcher interface {
 	HeadFetcher
 	FinalizationFetcher
+	GenesisFetcher
 }
 
 // TimeFetcher retrieves the Eth2 data that's related to time.
@@ -27,6 +29,11 @@ type TimeFetcher interface {
 	CurrentSlot() uint64
 }
 
+// GenesisFetcher retrieves the eth2 data related to its genesis.
+type GenesisFetcher interface {
+	GenesisValidatorRoot() [32]byte
+}
+
 // HeadFetcher defines a common interface for methods in blockchain service which
 // directly retrieves head related data.
 type HeadFetcher interface {
@@ -178,6 +185,15 @@ func (s *Service) GenesisTime() time.Time {
 	return s.genesisTime
 }
 
+// GenesisValidatorRoot returns the genesis validator
+// root of the chain.
+func (s *Service) GenesisValidatorRoot() [32]byte {
+	if !s.hasHeadState() {
+		return [32]byte{}
+	}
+	return bytesutil.ToBytes32(s.head.state.GenesisValidatorRoot())
+}
+
 // CurrentFork retrieves the latest fork information of the beacon chain.
 func (s *Service) CurrentFork() *pb.Fork {
 	if !s.hasHeadState() {

beacon-chain/blockchain/init_sync_process_block.go

@@ -170,6 +170,12 @@ func (s *Service) generateState(ctx context.Context, startRoot [32]byte, endRoot
 		return nil, err
 	}
 	if preState == nil {
+		if !s.stateGen.HasState(ctx, startRoot) {
+			if err := s.beaconDB.SaveBlocks(ctx, s.getInitSyncBlocks()); err != nil {
+				return nil, errors.Wrap(err, "could not save initial sync blocks")
+			}
+			s.clearInitSyncBlocks()
+		}
 		preState, err = s.stateGen.StateByRoot(ctx, startRoot)
 		if err != nil {
 			return nil, err

beacon-chain/blockchain/process_attestation.go

@@ -71,6 +71,16 @@ func (s *Service) onAttestation(ctx context.Context, a *ethpb.Attestation) ([]ui
 	ctx, span := trace.StartSpan(ctx, "blockchain.onAttestation")
 	defer span.End()
 
+	if a == nil {
+		return nil, errors.New("nil attestation")
+	}
+	if a.Data == nil {
+		return nil, errors.New("nil attestation.Data field")
+	}
+	if a.Data.Target == nil {
+		return nil, errors.New("nil attestation.Data.Target field")
+	}
+
 	tgt := stateTrie.CopyCheckpoint(a.Data.Target)
 	tgtSlot := helpers.StartSlot(tgt.Epoch)
 
@@ -128,12 +138,6 @@ func (s *Service) onAttestation(ctx context.Context, a *ethpb.Attestation) ([]ui
 	if indexedAtt.AttestingIndices == nil {
 		return nil, errors.New("nil attesting indices")
 	}
-	if a.Data == nil {
-		return nil, errors.New("nil att data")
-	}
-	if a.Data.Target == nil {
-		return nil, errors.New("nil att target")
-	}
 
 	// Update forkchoice store with the new attestation for updating weight.
 	s.forkChoiceStore.ProcessAttestation(ctx, indexedAtt.AttestingIndices, bytesutil.ToBytes32(a.Data.BeaconBlockRoot), a.Data.Target.Epoch)

beacon-chain/blockchain/process_attestation_helpers.go

@@ -33,6 +33,12 @@ func (s *Service) getAttPreState(ctx context.Context, c *ethpb.Checkpoint) (*sta
 
 	var baseState *stateTrie.BeaconState
 	if !featureconfig.Get().DisableNewStateMgmt {
+		if !s.stateGen.HasState(ctx, bytesutil.ToBytes32(c.Root)) {
+			if err := s.beaconDB.SaveBlocks(ctx, s.getInitSyncBlocks()); err != nil {
+				return nil, errors.Wrap(err, "could not save initial sync blocks")
+			}
+			s.clearInitSyncBlocks()
+		}
 		baseState, err = s.stateGen.StateByRoot(ctx, bytesutil.ToBytes32(c.Root))
 		if err != nil {
 			return nil, errors.Wrapf(err, "could not get pre state for slot %d", helpers.StartSlot(c.Epoch))
@@ -129,6 +135,12 @@ func (s *Service) verifyAttestation(ctx context.Context, baseState *stateTrie.Be
 			var aState *stateTrie.BeaconState
 			var err error
 			if !featureconfig.Get().DisableNewStateMgmt {
+				if !s.stateGen.HasState(ctx, bytesutil.ToBytes32(a.Data.BeaconBlockRoot)) {
+					if err := s.beaconDB.SaveBlocks(ctx, s.getInitSyncBlocks()); err != nil {
+						return nil, errors.Wrap(err, "could not save initial sync blocks")
+					}
+					s.clearInitSyncBlocks()
+				}
 				aState, err = s.stateGen.StateByRoot(ctx, bytesutil.ToBytes32(a.Data.BeaconBlockRoot))
 				if err != nil {
 					return nil, err

beacon-chain/blockchain/process_attestation_test.go

@@ -114,7 +114,7 @@ func TestStore_OnAttestation(t *testing.T) {
 			a:             &ethpb.Attestation{Data: &ethpb.AttestationData{Target: &ethpb.Checkpoint{Root: BlkWithOutStateRoot[:]}}},
 			s:             &pb.BeaconState{},
 			wantErr:       true,
-			wantErrString: "could not get pre state for slot 0: unknown boundary state",
+			wantErrString: "could not get pre state for slot 0: could not get ancestor state",
 		},
 		{
 			name: "process attestation doesn't match current epoch",
@@ -124,13 +124,34 @@ func TestStore_OnAttestation(t *testing.T) {
 			wantErr:       true,
 			wantErrString: "does not match current epoch",
 		},
+		{
+			name:          "process nil field (a.Target) in attestation",
+			a:             nil,
+			s:             &pb.BeaconState{},
+			wantErr:       true,
+			wantErrString: "nil attestation",
+		},
+		{
+			name:          "process nil field (a.Data) in attestation",
+			a:             &ethpb.Attestation{},
+			s:             &pb.BeaconState{},
+			wantErr:       true,
+			wantErrString: "nil attestation.Data field",
+		},
+		{
+			name:          "process nil field (a.Target) in attestation",
+			a:             &ethpb.Attestation{Data: &ethpb.AttestationData{}},
+			s:             &pb.BeaconState{},
+			wantErr:       true,
+			wantErrString: "nil attestation.Data.Target field",
+		},
 	}
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			_, err := service.onAttestation(ctx, tt.a)
 			if tt.wantErr {
-				if !strings.Contains(err.Error(), tt.wantErrString) {
+				if err == nil || !strings.Contains(err.Error(), tt.wantErrString) {
 					t.Errorf("Store.onAttestation() error = %v, wantErr = %v", err, tt.wantErrString)
 				}
 			} else {

beacon-chain/blockchain/process_block_helpers.go

@@ -21,7 +21,12 @@ import (
 
 // CurrentSlot returns the current slot based on time.
 func (s *Service) CurrentSlot() uint64 {
-	return uint64(roughtime.Now().Unix()-s.genesisTime.Unix()) / params.BeaconConfig().SecondsPerSlot
+	now := roughtime.Now().Unix()
+	genesis := s.genesisTime.Unix()
+	if now < genesis {
+		return 0
+	}
+	return uint64(now-genesis) / params.BeaconConfig().SecondsPerSlot
 }
 
 // getBlockPreState returns the pre state of an incoming block. It uses the parent root of the block
@@ -62,8 +67,17 @@ func (s *Service) verifyBlkPreState(ctx context.Context, b *ethpb.BeaconBlock) (
 
 	if !featureconfig.Get().DisableNewStateMgmt {
 		parentRoot := bytesutil.ToBytes32(b.ParentRoot)
-		if !s.stateGen.StateSummaryExists(ctx, parentRoot) {
-			return nil, errors.New("provided block root does not have block saved in the db")
+		// Loosen the check to HasBlock because state summary gets saved in batches
+		// during initial syncing. There's no risk given a state summary object is just a
+		// a subset of the block object.
+		if !s.stateGen.StateSummaryExists(ctx, parentRoot) && !s.beaconDB.HasBlock(ctx, parentRoot) {
+			return nil, errors.New("could not reconstruct parent state")
+		}
+		if !s.stateGen.HasState(ctx, parentRoot) {
+			if err := s.beaconDB.SaveBlocks(ctx, s.getInitSyncBlocks()); err != nil {
+				return nil, errors.Wrap(err, "could not save initial sync blocks")
+			}
+			s.clearInitSyncBlocks()
 		}
 		preState, err := s.stateGen.StateByRoot(ctx, parentRoot)
 		if err != nil {

beacon-chain/blockchain/process_block_test.go

@@ -20,6 +20,7 @@ import (
 	pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
 	"github.com/prysmaticlabs/prysm/shared/bytesutil"
 	"github.com/prysmaticlabs/prysm/shared/params"
+	"github.com/prysmaticlabs/prysm/shared/roughtime"
 	"github.com/prysmaticlabs/prysm/shared/testutil"
 )
 
@@ -87,7 +88,7 @@ func TestStore_OnBlock(t *testing.T) {
 			name:          "parent block root does not have a state",
 			blk:           &ethpb.BeaconBlock{},
 			s:             st.Copy(),
-			wantErrString: "provided block root does not have block saved in the db",
+			wantErrString: "could not reconstruct parent state",
 		},
 		{
 			name:          "block is from the feature",
@@ -353,7 +354,7 @@ func TestCachedPreState_CanGetFromDB(t *testing.T) {
 
 	service.finalizedCheckpt = &ethpb.Checkpoint{Root: r[:]}
 	_, err = service.verifyBlkPreState(ctx, b)
-	wanted := "provided block root does not have block saved in the db"
+	wanted := "could not reconstruct parent state"
 	if err.Error() != wanted {
 		t.Error("Did not get wanted error")
 	}
@@ -761,3 +762,13 @@ func blockTree1(db db.Database, genesisRoot []byte) ([][]byte, error) {
 	}
 	return [][]byte{r0[:], r1[:], nil, r3[:], r4[:], r5[:], r6[:], r7[:], r8[:]}, nil
 }
+
+func TestCurrentSlot_HandlesOverflow(t *testing.T) {
+	svc := Service{genesisTime: roughtime.Now().Add(1 * time.Hour)}
+
+	slot := svc.CurrentSlot()
+
+	if slot != 0 {
+		t.Fatalf("Expected slot to be 0, got %d", slot)
+	}
+}

beacon-chain/blockchain/testing/mock.go

@@ -32,6 +32,7 @@ type ChainService struct {
 	BlocksReceived              []*ethpb.SignedBeaconBlock
 	Balance                     *precompute.Balance
 	Genesis                     time.Time
+	ValidatorsRoot              [32]byte
 	Fork                        *pb.Fork
 	DB                          db.Database
 	stateNotifier               statefeed.Notifier
@@ -217,6 +218,11 @@ func (ms *ChainService) GenesisTime() time.Time {
 	return ms.Genesis
 }
 
+// GenesisValidatorRoot mocks the same method in the chain service.
+func (ms *ChainService) GenesisValidatorRoot() [32]byte {
+	return ms.ValidatorsRoot
+}
+
 // CurrentSlot mocks the same method in the chain service.
 func (ms *ChainService) CurrentSlot() uint64 {
 	return uint64(time.Now().Unix()-ms.Genesis.Unix()) / params.BeaconConfig().SecondsPerSlot

beacon-chain/core/blocks/block_operations.go

@@ -379,7 +379,7 @@ func ProcessRandaoNoVerify(
 	for i, x := range blockRandaoReveal {
 		latestMixSlice[i] ^= x
 	}
-	if err := beaconState.UpdateRandaoMixesAtIndex(latestMixSlice, currentEpoch%latestMixesLength); err != nil {
+	if err := beaconState.UpdateRandaoMixesAtIndex(currentEpoch%latestMixesLength, latestMixSlice); err != nil {
 		return nil, err
 	}
 	return beaconState, nil
@@ -987,7 +987,7 @@ func ProcessDeposit(
 		depositSig := deposit.Data.Signature
 		if err := verifyDepositDataSigningRoot(deposit.Data, pubKey, depositSig, domain); err != nil {
 			// Ignore this error as in the spec pseudo code.
-			log.Errorf("Skipping deposit: could not verify deposit data signature: %v", err)
+			log.Debugf("Skipping deposit: could not verify deposit data signature: %v", err)
 			return beaconState, nil
 		}
 

beacon-chain/core/epoch/epoch_processing.go

@@ -293,7 +293,7 @@ func ProcessFinalUpdates(state *stateTrie.BeaconState) (*stateTrie.BeaconState,
 	if err != nil {
 		return nil, err
 	}
-	if err := state.UpdateRandaoMixesAtIndex(mix, nextEpoch%randaoMixLength); err != nil {
+	if err := state.UpdateRandaoMixesAtIndex(nextEpoch%randaoMixLength, mix); err != nil {
 		return nil, err
 	}
 

beacon-chain/core/feed/state/events.go

@@ -9,6 +9,8 @@ const (
 	ChainStarted
 	// Initialized is sent when the internal beacon node's state is ready to be accessed.
 	Initialized
+	// Synced is sent when the beacon node has completed syncing and is ready to participate in the network.
+	Synced
 )
 
 // BlockProcessedData is the data sent with BlockProcessed events.
@@ -27,6 +29,12 @@ type ChainStartedData struct {
 	StartTime time.Time
 }
 
+// SyncedData is the data sent with Synced events.
+type SyncedData struct {
+	// StartTime is the time at which the chain started.
+	StartTime time.Time
+}
+
 // InitializedData is the data sent with Initialized events.
 type InitializedData struct {
 	// StartTime is the time at which the chain started.

beacon-chain/core/helpers/BUILD.bazel

@@ -16,14 +16,15 @@ go_library(
     importpath = "github.com/prysmaticlabs/prysm/beacon-chain/core/helpers",
     visibility = [
         "//beacon-chain:__subpackages__",
+        "//endtoend/evaluators:__pkg__",
         "//shared/benchutil/benchmark_files:__subpackages__",
-        "//shared/testutil:__pkg__",
-        "//shared/keystore:__pkg__",
         "//shared/interop:__pkg__",
+        "//shared/keystore:__pkg__",
+        "//shared/p2putils:__pkg__",
+        "//shared/testutil:__pkg__",
         "//slasher:__subpackages__",
         "//tools:__subpackages__",
         "//validator:__subpackages__",
-        "//endtoend/evaluators:__pkg__",
     ],
     deps = [
         "//beacon-chain/cache:go_default_library",

beacon-chain/db/iface/interface.go

@@ -130,4 +130,7 @@ type Database interface {
 
 	// Backup and restore methods
 	Backup(ctx context.Context) error
+
+	// HistoricalStatesDeleted verifies historical states exist in DB.
+	HistoricalStatesDeleted(ctx context.Context) error
 }

beacon-chain/db/kafka/passthrough.go

@@ -357,3 +357,8 @@ func (e Exporter) SaveLastArchivedIndex(ctx context.Context, index uint64) error
 func (e Exporter) LastArchivedIndex(ctx context.Context) (uint64, error) {
 	return e.db.LastArchivedIndex(ctx)
 }
+
+// HistoricalStatesDeleted -- passthrough
+func (e Exporter) HistoricalStatesDeleted(ctx context.Context) error {
+	return e.db.HistoricalStatesDeleted(ctx)
+}

beacon-chain/db/kv/check_historical_state.go

@@ -11,7 +11,8 @@ import (
 
 var historicalStateDeletedKey = []byte("historical-states-deleted")
 
-func (kv *Store) ensureNewStateServiceCompatible(ctx context.Context) error {
+// HistoricalStatesDeleted verifies historical states exist in DB.
+func (kv *Store) HistoricalStatesDeleted(ctx context.Context) error {
 	if featureconfig.Get().DisableNewStateMgmt {
 		return kv.db.Update(func(tx *bolt.Tx) error {
 			bkt := tx.Bucket(newStateServiceCompatibleBucket)
@@ -32,9 +33,9 @@ func (kv *Store) ensureNewStateServiceCompatible(ctx context.Context) error {
 	regenHistoricalStatesConfirmed := false
 	var err error
 	if historicalStateDeleted {
-		actionText := "--disable-new-state-mgmt was used. To proceed without the flag, the db will need " +
-			"to generate and save historical states. This process may take a while, - do you want to proceed? (Y/N)"
-		deniedText := "Historical states will not be generated. Please continue use --disable-new-state-mgmt"
+		actionText := "--disable-new-state-mgmt was previously used and historical states cannot be found. To proceed without using the flag, the db will need " +
+			"to generate and re-save historical states. This process may take a while, - do you want to proceed? (Y/N)"
+		deniedText := "Historical states will not be generated. Please continue using --disable-new-state-mgmt"
 
 		regenHistoricalStatesConfirmed, err = cmd.ConfirmAction(actionText, deniedText)
 		if err != nil {

beacon-chain/db/kv/kv.go

@@ -1,7 +1,6 @@
 package kv
 
 import (
-	"context"
 	"os"
 	"path"
 	"sync"
@@ -120,10 +119,6 @@ func NewKVStore(dirPath string, stateSummaryCache *cache.StateSummaryCache) (*St
 		return nil, err
 	}
 
-	if err := kv.ensureNewStateServiceCompatible(context.Background()); err != nil {
-		return nil, err
-	}
-
 	err = prometheus.Register(createBoltCollector(kv.db))
 
 	return kv, err

beacon-chain/main.go

@@ -7,6 +7,7 @@ import (
 	"runtime"
 	runtimeDebug "runtime/debug"
 
+	gethlog "github.com/ethereum/go-ethereum/log"
 	golog "github.com/ipfs/go-log"
 	joonix "github.com/joonix/log"
 	"github.com/prysmaticlabs/prysm/beacon-chain/flags"
@@ -62,6 +63,7 @@ var appFlags = []cli.Flag{
 	cmd.P2PMetadata,
 	cmd.P2PWhitelist,
 	cmd.P2PEncoding,
+	cmd.P2PPubsub,
 	cmd.DataDirFlag,
 	cmd.VerbosityFlag,
 	cmd.EnableTracingFlag,
@@ -167,7 +169,12 @@ func startNode(ctx *cli.Context) error {
 	}
 	logrus.SetLevel(level)
 	if level == logrus.TraceLevel {
+		// libp2p specific logging.
 		golog.SetAllLoggers(gologging.DEBUG)
+		// Geth specific logging.
+		glogger := gethlog.NewGlogHandler(gethlog.StreamHandler(os.Stderr, gethlog.TerminalFormat(true)))
+		glogger.Verbosity(gethlog.LvlTrace)
+		gethlog.Root().SetHandler(glogger)
 	}
 
 	beacon, err := node.NewBeaconNode(ctx)

beacon-chain/node/node.go

@@ -259,7 +259,12 @@ func (b *BeaconNode) startDB(ctx *cli.Context) error {
 		if err != nil {
 			return err
 		}
+	} else {
+		if err := d.HistoricalStatesDeleted(ctx); err != nil {
+			return err
+		}
 	}
+
 	log.WithField("database-path", dbPath).Info("Checking DB")
 	b.db = d
 	b.depositCache = depositcache.NewDepositCache()
@@ -307,6 +312,7 @@ func (b *BeaconNode) registerP2P(ctx *cli.Context) error {
 		DisableDiscv5:     ctx.Bool(flags.DisableDiscv5.Name),
 		Encoding:          ctx.String(cmd.P2PEncoding.Name),
 		StateNotifier:     b,
+		PubSub:            ctx.String(cmd.P2PPubsub.Name),
 	})
 	if err != nil {
 		return err

beacon-chain/p2p/BUILD.bazel

@@ -43,6 +43,7 @@ go_library(
         "//shared/featureconfig:go_default_library",
         "//shared/hashutil:go_default_library",
         "//shared/iputils:go_default_library",
+        "//shared/p2putils:go_default_library",
         "//shared/params:go_default_library",
         "//shared/runutil:go_default_library",
         "//shared/sliceutil:go_default_library",
@@ -114,6 +115,7 @@ go_test(
         "//proto/beacon/p2p/v1:go_default_library",
         "//proto/testing:go_default_library",
         "//shared/iputils:go_default_library",
+        "//shared/p2putils:go_default_library",
         "//shared/params:go_default_library",
         "//shared/testutil:go_default_library",
         "@com_github_ethereum_go_ethereum//p2p/discover:go_default_library",

beacon-chain/p2p/broadcaster.go

@@ -22,7 +22,7 @@ var ErrMessageNotMapped = errors.New("message type is not mapped to a PubSub top
 func (s *Service) Broadcast(ctx context.Context, msg proto.Message) error {
 	ctx, span := trace.StartSpan(ctx, "p2p.Broadcast")
 	defer span.End()
-	forkDigest, err := s.ForkDigest()
+	forkDigest, err := s.forkDigest()
 	if err != nil {
 		return err
 	}
@@ -44,7 +44,7 @@ func (s *Service) Broadcast(ctx context.Context, msg proto.Message) error {
 	span.AddAttributes(trace.StringAttribute("topic", topic))
 
 	buf := new(bytes.Buffer)
-	if _, err := s.Encoding().Encode(buf, msg); err != nil {
+	if _, err := s.Encoding().EncodeGossip(buf, msg); err != nil {
 		err := errors.Wrap(err, "could not encode message")
 		traceutil.AnnotateError(span, err)
 		return err

beacon-chain/p2p/broadcaster_test.go

@@ -10,8 +10,8 @@ import (
 
 	"github.com/gogo/protobuf/proto"
 	eth "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
-	testpb "github.com/prysmaticlabs/prysm/proto/testing"
 	p2ptest "github.com/prysmaticlabs/prysm/beacon-chain/p2p/testing"
+	testpb "github.com/prysmaticlabs/prysm/proto/testing"
 	"github.com/prysmaticlabs/prysm/shared/testutil"
 )
 
@@ -29,6 +29,8 @@ func TestService_Broadcast(t *testing.T) {
 		cfg: &Config{
 			Encoding: "ssz",
 		},
+		genesisTime:           time.Now(),
+		genesisValidatorsRoot: []byte{'A'},
 	}
 
 	msg := &testpb.TestSimpleMessage{
@@ -38,7 +40,7 @@ func TestService_Broadcast(t *testing.T) {
 	topic := "/eth2/%x/testing"
 	// Set a test gossip mapping for testpb.TestSimpleMessage.
 	GossipTypeMapping[reflect.TypeOf(msg)] = topic
-	digest, err := p.ForkDigest()
+	digest, err := p.forkDigest()
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -67,7 +69,7 @@ func TestService_Broadcast(t *testing.T) {
 		}
 
 		result := &testpb.TestSimpleMessage{}
-		if err := p.Encoding().Decode(incomingMessage.Data, result); err != nil {
+		if err := p.Encoding().DecodeGossip(incomingMessage.Data, result); err != nil {
 			tt.Fatal(err)
 		}
 		if !proto.Equal(result, msg) {
@@ -85,7 +87,10 @@ func TestService_Broadcast(t *testing.T) {
 }
 
 func TestService_Broadcast_ReturnsErr_TopicNotMapped(t *testing.T) {
-	p := Service{}
+	p := Service{
+		genesisTime:           time.Now(),
+		genesisValidatorsRoot: []byte{'A'},
+	}
 	if err := p.Broadcast(context.Background(), &testpb.AddressBook{}); err != ErrMessageNotMapped {
 		t.Fatalf("Expected error %v, got %v", ErrMessageNotMapped, err)
 	}

beacon-chain/p2p/config.go

@@ -27,4 +27,5 @@ type Config struct {
 	WhitelistCIDR         string
 	Encoding              string
 	StateNotifier         statefeed.Notifier
+	PubSub                string
 }

beacon-chain/p2p/discovery_test.go

@@ -53,7 +53,9 @@ func TestCreateListener(t *testing.T) {
 	port := 1024
 	ipAddr, pkey := createAddrAndPrivKey(t)
 	s := &Service{
-		cfg: &Config{UDPPort: uint(port)},
+		genesisTime:           time.Now(),
+		genesisValidatorsRoot: []byte{'A'},
+		cfg:                   &Config{UDPPort: uint(port)},
 	}
 	listener := s.createListener(ipAddr, pkey)
 	defer listener.Close()
@@ -130,7 +132,10 @@ func TestStartDiscV5_DiscoverAllPeers(t *testing.T) {
 func TestMultiAddrsConversion_InvalidIPAddr(t *testing.T) {
 	addr := net.ParseIP("invalidIP")
 	_, pkey := createAddrAndPrivKey(t)
-	s := &Service{}
+	s := &Service{
+		genesisTime:           time.Now(),
+		genesisValidatorsRoot: []byte{'A'},
+	}
 	node, err := s.createLocalNode(pkey, addr, 0, 0)
 	if err != nil {
 		t.Fatal(err)
@@ -149,6 +154,8 @@ func TestMultiAddrConversion_OK(t *testing.T) {
 			TCPPort: 0,
 			UDPPort: 0,
 		},
+		genesisTime:           time.Now(),
+		genesisValidatorsRoot: []byte{'A'},
 	}
 	listener := s.createListener(ipAddr, pkey)
 	defer listener.Close()

beacon-chain/p2p/encoder/BUILD.bazel

@@ -13,8 +13,10 @@ go_library(
         "//beacon-chain:__subpackages__",
     ],
     deps = [
+        "//shared/params:go_default_library",
         "@com_github_gogo_protobuf//proto:go_default_library",
         "@com_github_golang_snappy//:go_default_library",
+        "@com_github_pkg_errors//:go_default_library",
         "@com_github_prysmaticlabs_go_ssz//:go_default_library",
         "@com_github_sirupsen_logrus//:go_default_library",
     ],

beacon-chain/p2p/encoder/network_encoding.go

@@ -12,8 +12,10 @@ const (
 
 // NetworkEncoding represents an encoder compatible with Ethereum 2.0 p2p.
 type NetworkEncoding interface {
-	// Decodes to the provided message. The interface must be a pointer to the decoding destination.
+	// Decode to the provided message. The interface must be a pointer to the decoding destination.
 	Decode([]byte, interface{}) error
+	// DecodeGossip to the provided gossip message. The interface must be a pointer to the decoding destination.
+	DecodeGossip([]byte, interface{}) error
 	// DecodeWithLength a bytes from a reader with a varint length prefix. The interface must be a pointer to the
 	// decoding destination.
 	DecodeWithLength(io.Reader, interface{}) error
@@ -22,6 +24,8 @@ type NetworkEncoding interface {
 	DecodeWithMaxLength(io.Reader, interface{}, uint64) error
 	// Encode an arbitrary message to the provided writer. The interface must be a pointer object to encode.
 	Encode(io.Writer, interface{}) (int, error)
+	// EncodeGossip an arbitrary gossip message to the provided writer. The interface must be a pointer object to encode.
+	EncodeGossip(io.Writer, interface{}) (int, error)
 	// EncodeWithLength an arbitrary message to the provided writer with a varint length prefix. The interface must be
 	// a pointer object to encode.
 	EncodeWithLength(io.Writer, interface{}) (int, error)

beacon-chain/p2p/encoder/ssz.go

@@ -7,14 +7,19 @@ import (
 
 	"github.com/gogo/protobuf/proto"
 	"github.com/golang/snappy"
+	errors "github.com/pkg/errors"
 	"github.com/prysmaticlabs/go-ssz"
+	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/sirupsen/logrus"
 )
 
 var _ = NetworkEncoding(&SszNetworkEncoder{})
 
 // MaxChunkSize allowed for decoding messages.
-const MaxChunkSize = uint64(1 << 20) // 1Mb
+var MaxChunkSize = params.BeaconNetworkConfig().MaxChunkSize // 1Mib
+
+// MaxGossipSize allowed for gossip messages.
+var MaxGossipSize = params.BeaconNetworkConfig().GossipMaxSize // 1 Mib
 
 // SszNetworkEncoder supports p2p networking encoding using SimpleSerialize
 // with snappy compression (if enabled).
@@ -41,6 +46,24 @@ func (e SszNetworkEncoder) Encode(w io.Writer, msg interface{}) (int, error) {
 	return w.Write(b)
 }
 
+// EncodeGossip the proto gossip message to the io.Writer.
+func (e SszNetworkEncoder) EncodeGossip(w io.Writer, msg interface{}) (int, error) {
+	if msg == nil {
+		return 0, nil
+	}
+	b, err := e.doEncode(msg)
+	if err != nil {
+		return 0, err
+	}
+	if len(b) > int(MaxGossipSize) {
+		return 0, errors.Errorf("gossip message exceeds max gossip size: %d bytes > %d bytes", len(b), MaxGossipSize)
+	}
+	if e.UseSnappyCompression {
+		b = snappy.Encode(nil /*dst*/, b)
+	}
+	return w.Write(b)
+}
+
 // EncodeWithLength the proto message to the io.Writer. This encoding prefixes the byte slice with a protobuf varint
 // to indicate the size of the message.
 func (e SszNetworkEncoder) EncodeWithLength(w io.Writer, msg interface{}) (int, error) {
@@ -105,6 +128,21 @@ func (e SszNetworkEncoder) Decode(b []byte, to interface{}) error {
 	return e.doDecode(b, to)
 }
 
+// DecodeGossip decodes the bytes to the protobuf gossip message provided.
+func (e SszNetworkEncoder) DecodeGossip(b []byte, to interface{}) error {
+	if e.UseSnappyCompression {
+		var err error
+		b, err = snappy.Decode(nil /*dst*/, b)
+		if err != nil {
+			return err
+		}
+	}
+	if len(b) > int(MaxGossipSize) {
+		return errors.Errorf("gossip message exceeds max gossip size: %d bytes > %d bytes", len(b), MaxGossipSize)
+	}
+	return e.doDecode(b, to)
+}
+
 // DecodeWithLength the bytes from io.Reader to the protobuf message provided.
 func (e SszNetworkEncoder) DecodeWithLength(r io.Reader, to interface{}) error {
 	return e.DecodeWithMaxLength(r, to, MaxChunkSize)

beacon-chain/p2p/encoder/ssz_test.go

@@ -15,12 +15,14 @@ func TestSszNetworkEncoder_RoundTrip(t *testing.T) {
 	e := &encoder.SszNetworkEncoder{UseSnappyCompression: false}
 	testRoundTrip(t, e)
 	testRoundTripWithLength(t, e)
+	testRoundTripWithGossip(t, e)
 }
 
 func TestSszNetworkEncoder_RoundTrip_Snappy(t *testing.T) {
 	e := &encoder.SszNetworkEncoder{UseSnappyCompression: true}
 	testRoundTrip(t, e)
 	testRoundTripWithLength(t, e)
+	testRoundTripWithGossip(t, e)
 }
 
 func testRoundTrip(t *testing.T, e *encoder.SszNetworkEncoder) {
@@ -63,6 +65,26 @@ func testRoundTripWithLength(t *testing.T, e *encoder.SszNetworkEncoder) {
 	}
 }
 
+func testRoundTripWithGossip(t *testing.T, e *encoder.SszNetworkEncoder) {
+	buf := new(bytes.Buffer)
+	msg := &testpb.TestSimpleMessage{
+		Foo: []byte("fooooo"),
+		Bar: 9001,
+	}
+	_, err := e.EncodeGossip(buf, msg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	decoded := &testpb.TestSimpleMessage{}
+	if err := e.DecodeGossip(buf.Bytes(), decoded); err != nil {
+		t.Fatal(err)
+	}
+	if !proto.Equal(decoded, msg) {
+		t.Logf("decoded=%+v\n", decoded)
+		t.Error("Decoded message is not the same as original")
+	}
+}
+
 func TestSszNetworkEncoder_EncodeWithMaxLength(t *testing.T) {
 	buf := new(bytes.Buffer)
 	msg := &testpb.TestSimpleMessage{

beacon-chain/p2p/fork.go

@@ -10,19 +10,19 @@ import (
 	"github.com/ethereum/go-ethereum/p2p/enr"
 	"github.com/pkg/errors"
 	"github.com/prysmaticlabs/go-ssz"
-	"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
 	pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
+	"github.com/prysmaticlabs/prysm/shared/p2putils"
 	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/sirupsen/logrus"
 )
 
 // ENR key used for eth2-related fork data.
-const eth2ENRKey = "eth2"
+var eth2ENRKey = params.BeaconNetworkConfig().ETH2Key
 
 // ForkDigest returns the current fork digest of
 // the node.
-func (s *Service) ForkDigest() ([4]byte, error) {
-	return createForkDigest(s.genesisTime, s.genesisValidatorsRoot)
+func (s *Service) forkDigest() ([4]byte, error) {
+	return p2putils.CreateForkDigest(s.genesisTime, s.genesisValidatorsRoot)
 }
 
 // Compares fork ENRs between an incoming peer's record and our node's
@@ -72,35 +72,6 @@ func (s *Service) compareForkENR(record *enr.Record) error {
 	return nil
 }
 
-// Creates a fork digest from a genesis time and genesis
-// validators root, utilizing the current slot to determine
-// the active fork version in the node.
-func createForkDigest(
-	genesisTime time.Time,
-	genesisValidatorsRoot []byte,
-) ([4]byte, error) {
-	currentSlot := helpers.SlotsSince(genesisTime)
-	currentEpoch := helpers.SlotToEpoch(currentSlot)
-
-	// We retrieve a list of scheduled forks by epoch.
-	// We loop through the keys in this map to determine the current
-	// fork version based on the current, time-based epoch number
-	// since the genesis time.
-	currentForkVersion := params.BeaconConfig().GenesisForkVersion
-	scheduledForks := params.BeaconConfig().ForkVersionSchedule
-	for epoch, forkVersion := range scheduledForks {
-		if epoch <= currentEpoch {
-			currentForkVersion = forkVersion
-		}
-	}
-
-	digest, err := helpers.ComputeForkDigest(currentForkVersion, genesisValidatorsRoot)
-	if err != nil {
-		return [4]byte{}, err
-	}
-	return digest, nil
-}
-
 // Adds a fork entry as an ENR record under the eth2EnrKey for
 // the local node. The fork entry is an ssz-encoded enrForkID type
 // which takes into account the current fork version from the current
@@ -111,7 +82,7 @@ func addForkEntry(
 	genesisTime time.Time,
 	genesisValidatorsRoot []byte,
 ) (*enode.LocalNode, error) {
-	digest, err := createForkDigest(genesisTime, genesisValidatorsRoot)
+	digest, err := p2putils.CreateForkDigest(genesisTime, genesisValidatorsRoot)
 	if err != nil {
 		return nil, err
 	}

beacon-chain/p2p/fork_test.go

@@ -16,6 +16,7 @@ import (
 	"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
 	testDB "github.com/prysmaticlabs/prysm/beacon-chain/db/testing"
 	pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
+	"github.com/prysmaticlabs/prysm/shared/p2putils"
 	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/prysmaticlabs/prysm/shared/testutil"
 	"github.com/sirupsen/logrus"
@@ -213,7 +214,7 @@ func TestDiscv5_AddRetrieveForkEntryENR(t *testing.T) {
 
 	genesisTime := time.Now()
 	genesisValidatorsRoot := make([]byte, 32)
-	digest, err := createForkDigest(genesisTime, make([]byte, 32))
+	digest, err := p2putils.CreateForkDigest(genesisTime, make([]byte, 32))
 	if err != nil {
 		t.Fatal(err)
 	}

beacon-chain/p2p/interfaces.go

@@ -44,7 +44,6 @@ type ConnectionHandler interface {
 // EncodingProvider provides p2p network encoding.
 type EncodingProvider interface {
 	Encoding() encoder.NetworkEncoding
-	ForkDigest() ([4]byte, error)
 }
 
 // PubSubProvider provides the p2p pubsub protocol.

beacon-chain/p2p/peers/status.go

@@ -226,8 +226,8 @@ func (p *Status) SubscribedToSubnet(index uint64) []peer.ID {
 	peers := make([]peer.ID, 0)
 	for pid, status := range p.status {
 		// look at active peers
-		if status.peerState == PeerConnecting || status.peerState == PeerConnected &&
-			status.metaData != nil {
+		connectedStatus := status.peerState == PeerConnecting || status.peerState == PeerConnected
+		if connectedStatus && status.metaData != nil && status.metaData.Attnets != nil {
 			indices := retrieveIndicesFromBitfield(status.metaData.Attnets)
 			for _, idx := range indices {
 				if idx == index {

beacon-chain/p2p/sender.go

@@ -7,6 +7,7 @@ import (
 	"github.com/libp2p/go-libp2p-core/network"
 	"github.com/libp2p/go-libp2p-core/peer"
 	"github.com/libp2p/go-libp2p-core/protocol"
+	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/prysmaticlabs/prysm/shared/traceutil"
 	"go.opencensus.io/trace"
 )
@@ -20,7 +21,7 @@ func (s *Service) Send(ctx context.Context, message interface{}, baseTopic strin
 	span.AddAttributes(trace.StringAttribute("topic", topic))
 
 	// TTFB_TIME (5s) + RESP_TIMEOUT (10s).
-	const deadline = 15 * time.Second
+	var deadline = params.BeaconNetworkConfig().TtfbTimeout + params.BeaconNetworkConfig().RespTimeout
 	ctx, cancel := context.WithTimeout(ctx, deadline)
 	defer cancel()
 

beacon-chain/p2p/service.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"context"
 	"crypto/ecdsa"
+	"fmt"
 	"strconv"
 	"strings"
 	"time"
@@ -56,6 +57,12 @@ const prysmProtocolPrefix = "/prysm/0.0.0"
 // maxBadResponses is the maximum number of bad responses from a peer before we stop talking to it.
 const maxBadResponses = 3
 
+const (
+	pubsubFlood  = "flood"
+	pubsubGossip = "gossip"
+	pubsubRandom = "random"
+)
+
 // Service for managing peer to peer (p2p) networking.
 type Service struct {
 	started               bool
@@ -153,7 +160,20 @@ func NewService(cfg *Config) (*Service, error) {
 		pubsub.WithStrictSignatureVerification(false),
 		pubsub.WithMessageIdFn(msgIDFunction),
 	}
-	gs, err := pubsub.NewGossipSub(s.ctx, s.host, psOpts...)
+
+	var gs *pubsub.PubSub
+	if cfg.PubSub == "" {
+		cfg.PubSub = pubsubGossip
+	}
+	if cfg.PubSub == pubsubFlood {
+		gs, err = pubsub.NewFloodSub(s.ctx, s.host, psOpts...)
+	} else if cfg.PubSub == pubsubGossip {
+		gs, err = pubsub.NewGossipSub(s.ctx, s.host, psOpts...)
+	} else if cfg.PubSub == pubsubRandom {
+		gs, err = pubsub.NewRandomSub(s.ctx, s.host, psOpts...)
+	} else {
+		return nil, fmt.Errorf("unknown pubsub type %s", cfg.PubSub)
+	}
 	if err != nil {
 		log.WithError(err).Error("Failed to start pubsub")
 		return nil, err
@@ -391,6 +411,10 @@ func (s *Service) RefreshENR(epoch uint64) {
 // with those peers.
 func (s *Service) FindPeersWithSubnet(index uint64) (bool, error) {
 	nodes := make([]*enode.Node, searchLimit)
+	if s.dv5Listener == nil {
+		// return if discovery isn't set
+		return false, nil
+	}
 	num := s.dv5Listener.ReadRandomNodes(nodes)
 	exists := false
 	for _, node := range nodes[:num] {
@@ -429,7 +453,7 @@ func (s *Service) FindPeersWithSubnet(index uint64) (bool, error) {
 				}
 				s.peers.Add(node.Record(), info.ID, multiAddr, network.DirUnknown)
 				if err := s.connectWithPeer(*info); err != nil {
-					log.Errorf("Could not connect with peer %s: %v", info.String(), err)
+					log.WithError(err).Debugf("Could not connect with peer %s", info.String())
 				}
 				exists = true
 			}
@@ -499,7 +523,7 @@ func (s *Service) connectWithAllPeers(multiAddrs []ma.Multiaddr) {
 		// make each dial non-blocking
 		go func(info peer.AddrInfo) {
 			if err := s.connectWithPeer(info); err != nil {
-				log.Errorf("Could not connect with peer %s: %v", info.String(), err)
+				log.WithError(err).Debugf("Could not connect with peer %s", info.String())
 			}
 		}(info)
 	}

beacon-chain/p2p/subnets.go

@@ -4,10 +4,12 @@ import (
 	"github.com/ethereum/go-ethereum/p2p/enode"
 	"github.com/ethereum/go-ethereum/p2p/enr"
 	"github.com/prysmaticlabs/go-bitfield"
+	"github.com/prysmaticlabs/prysm/shared/params"
 )
 
-const attestationSubnetCount = 64
-const attSubnetEnrKey = "attnets"
+var attestationSubnetCount = params.BeaconNetworkConfig().AttestationSubnetCount
+
+var attSubnetEnrKey = params.BeaconNetworkConfig().AttSubnetKey
 
 func intializeAttSubnets(node *enode.LocalNode) *enode.LocalNode {
 	bitV := bitfield.NewBitvector64()

beacon-chain/p2p/testing/p2p.go

@@ -40,6 +40,7 @@ type TestP2P struct {
 	pubsub          *pubsub.PubSub
 	BroadcastCalled bool
 	DelaySend       bool
+	Digest          [4]byte
 	peers           *peers.Status
 	LocalMetadata   *pb.MetaData
 }
@@ -119,7 +120,7 @@ func (p *TestP2P) ReceivePubSub(topic string, msg proto.Message) {
 	time.Sleep(time.Millisecond * 100)
 
 	buf := new(bytes.Buffer)
-	if _, err := p.Encoding().Encode(buf, msg); err != nil {
+	if _, err := p.Encoding().EncodeGossip(buf, msg); err != nil {
 		p.t.Fatalf("Failed to encode message: %v", err)
 	}
 	digest, err := p.ForkDigest()
@@ -257,7 +258,7 @@ func (p *TestP2P) RefreshENR(epoch uint64) {
 
 // ForkDigest mocks the p2p func.
 func (p *TestP2P) ForkDigest() ([4]byte, error) {
-	return [4]byte{}, nil
+	return p.Digest, nil
 }
 
 // Metadata mocks the peer's metadata.

beacon-chain/powchain/log_processing.go

@@ -156,13 +156,12 @@ func (s *Service) ProcessDepositLog(ctx context.Context, depositLog gethTypes.Lo
 
 	// Make sure duplicates are rejected pre-chainstart.
 	if !s.chainStartData.Chainstarted {
-		var pubkey = fmt.Sprintf("#%x", depositData.PublicKey)
+		var pubkey = fmt.Sprintf("%#x", depositData.PublicKey)
 		if s.depositCache.PubkeyInChainstart(ctx, pubkey) {
-			log.Warnf("Pubkey %#x has already been submitted for chainstart", pubkey)
+			log.WithField("publicKey", pubkey).Debug("Pubkey has already been submitted for chainstart")
 		} else {
 			s.depositCache.MarkPubkeyForChainstart(ctx, pubkey)
 		}
-
 	}
 
 	// We always store all historical deposits in the DB.

beacon-chain/rpc/testing/beacon_node_validator_service_mock.go

@@ -1,42 +1,160 @@
 // Code generated by MockGen. DO NOT EDIT.
-// Source: github.com/prysmaticlabs/ethereumapis/eth/v1alpha1 (interfaces: BeaconNodeValidator_WaitForActivationServer,BeaconNodeValidator_WaitForChainStartServer)
+// Source: github.com/prysmaticlabs/ethereumapis/eth/v1alpha1 (interfaces: BeaconNodeValidator_WaitForSyncedServer,BeaconNodeValidator_WaitForActivationServer,BeaconNodeValidator_WaitForChainStartServer)
 
-// Package testing is a generated GoMock package.
+// Package mock is a generated GoMock package.
 package testing
 
 import (
 	context "context"
-	reflect "reflect"
-
 	gomock "github.com/golang/mock/gomock"
-	v1alpha1 "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
+	eth "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
 	metadata "google.golang.org/grpc/metadata"
+	reflect "reflect"
 )
 
-// MockBeaconNodeValidator_WaitForActivationServer is a mock of BeaconNodeValidator_WaitForActivationServer interface
+// MockBeaconNodeValidator_WaitForSyncedServer is a mock of BeaconNodeValidator_WaitForSyncedServer interface.
+type MockBeaconNodeValidator_WaitForSyncedServer struct {
+	ctrl     *gomock.Controller
+	recorder *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder
+}
+
+// MockBeaconNodeValidator_WaitForSyncedServerMockRecorder is the mock recorder for MockBeaconNodeValidator_WaitForSyncedServer.
+type MockBeaconNodeValidator_WaitForSyncedServerMockRecorder struct {
+	mock *MockBeaconNodeValidator_WaitForSyncedServer
+}
+
+// NewMockBeaconNodeValidator_WaitForSyncedServer creates a new mock instance.
+func NewMockBeaconNodeValidator_WaitForSyncedServer(ctrl *gomock.Controller) *MockBeaconNodeValidator_WaitForSyncedServer {
+	mock := &MockBeaconNodeValidator_WaitForSyncedServer{ctrl: ctrl}
+	mock.recorder = &MockBeaconNodeValidator_WaitForSyncedServerMockRecorder{mock}
+	return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) EXPECT() *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder {
+	return m.recorder
+}
+
+// Context mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) Context() context.Context {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Context")
+	ret0, _ := ret[0].(context.Context)
+	return ret0
+}
+
+// Context indicates an expected call of Context.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) Context() *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Context", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).Context))
+}
+
+// RecvMsg mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) RecvMsg(arg0 interface{}) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "RecvMsg", arg0)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// RecvMsg indicates an expected call of RecvMsg.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) RecvMsg(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RecvMsg", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).RecvMsg), arg0)
+}
+
+// Send mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) Send(arg0 *eth.SyncedResponse) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Send", arg0)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// Send indicates an expected call of Send.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) Send(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Send", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).Send), arg0)
+}
+
+// SendHeader mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) SendHeader(arg0 metadata.MD) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "SendHeader", arg0)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// SendHeader indicates an expected call of SendHeader.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) SendHeader(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SendHeader", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).SendHeader), arg0)
+}
+
+// SendMsg mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) SendMsg(arg0 interface{}) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "SendMsg", arg0)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// SendMsg indicates an expected call of SendMsg.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) SendMsg(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SendMsg", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).SendMsg), arg0)
+}
+
+// SetHeader mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) SetHeader(arg0 metadata.MD) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "SetHeader", arg0)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// SetHeader indicates an expected call of SetHeader.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) SetHeader(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetHeader", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).SetHeader), arg0)
+}
+
+// SetTrailer mocks base method.
+func (m *MockBeaconNodeValidator_WaitForSyncedServer) SetTrailer(arg0 metadata.MD) {
+	m.ctrl.T.Helper()
+	m.ctrl.Call(m, "SetTrailer", arg0)
+}
+
+// SetTrailer indicates an expected call of SetTrailer.
+func (mr *MockBeaconNodeValidator_WaitForSyncedServerMockRecorder) SetTrailer(arg0 interface{}) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetTrailer", reflect.TypeOf((*MockBeaconNodeValidator_WaitForSyncedServer)(nil).SetTrailer), arg0)
+}
+
+// MockBeaconNodeValidator_WaitForActivationServer is a mock of BeaconNodeValidator_WaitForActivationServer interface.
 type MockBeaconNodeValidator_WaitForActivationServer struct {
 	ctrl     *gomock.Controller
 	recorder *MockBeaconNodeValidator_WaitForActivationServerMockRecorder
 }
 
-// MockBeaconNodeValidator_WaitForActivationServerMockRecorder is the mock recorder for MockBeaconNodeValidator_WaitForActivationServer
+// MockBeaconNodeValidator_WaitForActivationServerMockRecorder is the mock recorder for MockBeaconNodeValidator_WaitForActivationServer.
 type MockBeaconNodeValidator_WaitForActivationServerMockRecorder struct {
 	mock *MockBeaconNodeValidator_WaitForActivationServer
 }
 
-// NewMockBeaconNodeValidator_WaitForActivationServer creates a new mock instance
+// NewMockBeaconNodeValidator_WaitForActivationServer creates a new mock instance.
 func NewMockBeaconNodeValidator_WaitForActivationServer(ctrl *gomock.Controller) *MockBeaconNodeValidator_WaitForActivationServer {
 	mock := &MockBeaconNodeValidator_WaitForActivationServer{ctrl: ctrl}
 	mock.recorder = &MockBeaconNodeValidator_WaitForActivationServerMockRecorder{mock}
 	return mock
 }
 
-// EXPECT returns an object that allows the caller to indicate expected use
+// EXPECT returns an object that allows the caller to indicate expected use.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) EXPECT() *MockBeaconNodeValidator_WaitForActivationServerMockRecorder {
 	return m.recorder
 }
 
-// Context mocks base method
+// Context mocks base method.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) Context() context.Context {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "Context")
@@ -44,13 +162,13 @@ func (m *MockBeaconNodeValidator_WaitForActivationServer) Context() context.Cont
 	return ret0
 }
 
-// Context indicates an expected call of Context
+// Context indicates an expected call of Context.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) Context() *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Context", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).Context))
 }
 
-// RecvMsg mocks base method
+// RecvMsg mocks base method.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) RecvMsg(arg0 interface{}) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "RecvMsg", arg0)
@@ -58,27 +176,27 @@ func (m *MockBeaconNodeValidator_WaitForActivationServer) RecvMsg(arg0 interface
 	return ret0
 }
 
-// RecvMsg indicates an expected call of RecvMsg
+// RecvMsg indicates an expected call of RecvMsg.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) RecvMsg(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RecvMsg", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).RecvMsg), arg0)
 }
 
-// Send mocks base method
-func (m *MockBeaconNodeValidator_WaitForActivationServer) Send(arg0 *v1alpha1.ValidatorActivationResponse) error {
+// Send mocks base method.
+func (m *MockBeaconNodeValidator_WaitForActivationServer) Send(arg0 *eth.ValidatorActivationResponse) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "Send", arg0)
 	ret0, _ := ret[0].(error)
 	return ret0
 }
 
-// Send indicates an expected call of Send
+// Send indicates an expected call of Send.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) Send(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Send", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).Send), arg0)
 }
 
-// SendHeader mocks base method
+// SendHeader mocks base method.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) SendHeader(arg0 metadata.MD) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "SendHeader", arg0)
@@ -86,13 +204,13 @@ func (m *MockBeaconNodeValidator_WaitForActivationServer) SendHeader(arg0 metada
 	return ret0
 }
 
-// SendHeader indicates an expected call of SendHeader
+// SendHeader indicates an expected call of SendHeader.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) SendHeader(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SendHeader", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).SendHeader), arg0)
 }
 
-// SendMsg mocks base method
+// SendMsg mocks base method.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) SendMsg(arg0 interface{}) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "SendMsg", arg0)
@@ -100,13 +218,13 @@ func (m *MockBeaconNodeValidator_WaitForActivationServer) SendMsg(arg0 interface
 	return ret0
 }
 
-// SendMsg indicates an expected call of SendMsg
+// SendMsg indicates an expected call of SendMsg.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) SendMsg(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SendMsg", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).SendMsg), arg0)
 }
 
-// SetHeader mocks base method
+// SetHeader mocks base method.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) SetHeader(arg0 metadata.MD) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "SetHeader", arg0)
@@ -114,48 +232,48 @@ func (m *MockBeaconNodeValidator_WaitForActivationServer) SetHeader(arg0 metadat
 	return ret0
 }
 
-// SetHeader indicates an expected call of SetHeader
+// SetHeader indicates an expected call of SetHeader.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) SetHeader(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetHeader", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).SetHeader), arg0)
 }
 
-// SetTrailer mocks base method
+// SetTrailer mocks base method.
 func (m *MockBeaconNodeValidator_WaitForActivationServer) SetTrailer(arg0 metadata.MD) {
 	m.ctrl.T.Helper()
 	m.ctrl.Call(m, "SetTrailer", arg0)
 }
 
-// SetTrailer indicates an expected call of SetTrailer
+// SetTrailer indicates an expected call of SetTrailer.
 func (mr *MockBeaconNodeValidator_WaitForActivationServerMockRecorder) SetTrailer(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetTrailer", reflect.TypeOf((*MockBeaconNodeValidator_WaitForActivationServer)(nil).SetTrailer), arg0)
 }
 
-// MockBeaconNodeValidator_WaitForChainStartServer is a mock of BeaconNodeValidator_WaitForChainStartServer interface
+// MockBeaconNodeValidator_WaitForChainStartServer is a mock of BeaconNodeValidator_WaitForChainStartServer interface.
 type MockBeaconNodeValidator_WaitForChainStartServer struct {
 	ctrl     *gomock.Controller
 	recorder *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder
 }
 
-// MockBeaconNodeValidator_WaitForChainStartServerMockRecorder is the mock recorder for MockBeaconNodeValidator_WaitForChainStartServer
+// MockBeaconNodeValidator_WaitForChainStartServerMockRecorder is the mock recorder for MockBeaconNodeValidator_WaitForChainStartServer.
 type MockBeaconNodeValidator_WaitForChainStartServerMockRecorder struct {
 	mock *MockBeaconNodeValidator_WaitForChainStartServer
 }
 
-// NewMockBeaconNodeValidator_WaitForChainStartServer creates a new mock instance
+// NewMockBeaconNodeValidator_WaitForChainStartServer creates a new mock instance.
 func NewMockBeaconNodeValidator_WaitForChainStartServer(ctrl *gomock.Controller) *MockBeaconNodeValidator_WaitForChainStartServer {
 	mock := &MockBeaconNodeValidator_WaitForChainStartServer{ctrl: ctrl}
 	mock.recorder = &MockBeaconNodeValidator_WaitForChainStartServerMockRecorder{mock}
 	return mock
 }
 
-// EXPECT returns an object that allows the caller to indicate expected use
+// EXPECT returns an object that allows the caller to indicate expected use.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) EXPECT() *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder {
 	return m.recorder
 }
 
-// Context mocks base method
+// Context mocks base method.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) Context() context.Context {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "Context")
@@ -163,13 +281,13 @@ func (m *MockBeaconNodeValidator_WaitForChainStartServer) Context() context.Cont
 	return ret0
 }
 
-// Context indicates an expected call of Context
+// Context indicates an expected call of Context.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) Context() *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Context", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).Context))
 }
 
-// RecvMsg mocks base method
+// RecvMsg mocks base method.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) RecvMsg(arg0 interface{}) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "RecvMsg", arg0)
@@ -177,27 +295,27 @@ func (m *MockBeaconNodeValidator_WaitForChainStartServer) RecvMsg(arg0 interface
 	return ret0
 }
 
-// RecvMsg indicates an expected call of RecvMsg
+// RecvMsg indicates an expected call of RecvMsg.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) RecvMsg(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RecvMsg", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).RecvMsg), arg0)
 }
 
-// Send mocks base method
-func (m *MockBeaconNodeValidator_WaitForChainStartServer) Send(arg0 *v1alpha1.ChainStartResponse) error {
+// Send mocks base method.
+func (m *MockBeaconNodeValidator_WaitForChainStartServer) Send(arg0 *eth.ChainStartResponse) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "Send", arg0)
 	ret0, _ := ret[0].(error)
 	return ret0
 }
 
-// Send indicates an expected call of Send
+// Send indicates an expected call of Send.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) Send(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Send", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).Send), arg0)
 }
 
-// SendHeader mocks base method
+// SendHeader mocks base method.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) SendHeader(arg0 metadata.MD) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "SendHeader", arg0)
@@ -205,13 +323,13 @@ func (m *MockBeaconNodeValidator_WaitForChainStartServer) SendHeader(arg0 metada
 	return ret0
 }
 
-// SendHeader indicates an expected call of SendHeader
+// SendHeader indicates an expected call of SendHeader.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) SendHeader(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SendHeader", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).SendHeader), arg0)
 }
 
-// SendMsg mocks base method
+// SendMsg mocks base method.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) SendMsg(arg0 interface{}) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "SendMsg", arg0)
@@ -219,13 +337,13 @@ func (m *MockBeaconNodeValidator_WaitForChainStartServer) SendMsg(arg0 interface
 	return ret0
 }
 
-// SendMsg indicates an expected call of SendMsg
+// SendMsg indicates an expected call of SendMsg.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) SendMsg(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SendMsg", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).SendMsg), arg0)
 }
 
-// SetHeader mocks base method
+// SetHeader mocks base method.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) SetHeader(arg0 metadata.MD) error {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "SetHeader", arg0)
@@ -233,19 +351,19 @@ func (m *MockBeaconNodeValidator_WaitForChainStartServer) SetHeader(arg0 metadat
 	return ret0
 }
 
-// SetHeader indicates an expected call of SetHeader
+// SetHeader indicates an expected call of SetHeader.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) SetHeader(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetHeader", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).SetHeader), arg0)
 }
 
-// SetTrailer mocks base method
+// SetTrailer mocks base method.
 func (m *MockBeaconNodeValidator_WaitForChainStartServer) SetTrailer(arg0 metadata.MD) {
 	m.ctrl.T.Helper()
 	m.ctrl.Call(m, "SetTrailer", arg0)
 }
 
-// SetTrailer indicates an expected call of SetTrailer
+// SetTrailer indicates an expected call of SetTrailer.
 func (mr *MockBeaconNodeValidator_WaitForChainStartServerMockRecorder) SetTrailer(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetTrailer", reflect.TypeOf((*MockBeaconNodeValidator_WaitForChainStartServer)(nil).SetTrailer), arg0)

beacon-chain/rpc/validator/aggregator.go

@@ -84,7 +84,8 @@ func (as *Server) SubmitAggregateSelectionProof(ctx context.Context, req *ethpb.
 // SubmitSignedAggregateSelectionProof is called by a validator to broadcast a signed
 // aggregated and proof object.
 func (as *Server) SubmitSignedAggregateSelectionProof(ctx context.Context, req *ethpb.SignedAggregateSubmitRequest) (*ethpb.SignedAggregateSubmitResponse, error) {
-	if req.SignedAggregateAndProof == nil {
+	if req.SignedAggregateAndProof == nil || req.SignedAggregateAndProof.Message == nil ||
+		req.SignedAggregateAndProof.Message.Aggregate == nil || req.SignedAggregateAndProof.Message.Aggregate.Data == nil {
 		return nil, status.Error(codes.InvalidArgument, "Signed aggregate request can't be nil")
 	}
 

beacon-chain/rpc/validator/server.go

@@ -7,6 +7,7 @@ import (
 
 	"github.com/prysmaticlabs/prysm/beacon-chain/state/stategen"
 	"github.com/prysmaticlabs/prysm/shared/bytesutil"
+	"github.com/prysmaticlabs/prysm/shared/params"
 
 	ptypes "github.com/gogo/protobuf/types"
 	ethpb "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
@@ -90,7 +91,8 @@ func (vs *Server) WaitForActivation(req *ethpb.ValidatorActivationRequest, strea
 
 	for {
 		select {
-		case <-time.After(6 * time.Second):
+		// Pinging every slot for activation.
+		case <-time.After(time.Duration(params.BeaconConfig().SecondsPerSlot) * time.Second):
 			activeValidatorExists, validatorStatuses, err := vs.multipleValidatorStatus(stream.Context(), req.PublicKeys)
 			if err != nil {
 				return status.Errorf(codes.Internal, "Could not fetch validator status: %v", err)
@@ -196,7 +198,44 @@ func (vs *Server) WaitForChainStart(req *ptypes.Empty, stream ethpb.BeaconNodeVa
 	}
 }
 
-// WaitForSynced is to be implemented.
-func (vs *Server) WaitForSynced(_ *ptypes.Empty, stream ethpb.BeaconNodeValidator_WaitForSyncedServer) error {
-	return status.Error(codes.Unimplemented, "not implemented")
+// WaitForSynced subscribes to the state channel and ends the stream when the state channel
+// indicates the beacon node has been initialized and is ready
+func (vs *Server) WaitForSynced(req *ptypes.Empty, stream ethpb.BeaconNodeValidator_WaitForSyncedServer) error {
+	head, err := vs.HeadFetcher.HeadState(context.Background())
+	if err != nil {
+		return status.Errorf(codes.Internal, "Could not retrieve head state: %v", err)
+	}
+	if head != nil && !vs.SyncChecker.Syncing() {
+		res := &ethpb.SyncedResponse{
+			Synced:      true,
+			GenesisTime: head.GenesisTime(),
+		}
+		return stream.Send(res)
+	}
+
+	stateChannel := make(chan *feed.Event, 1)
+	stateSub := vs.StateNotifier.StateFeed().Subscribe(stateChannel)
+	defer stateSub.Unsubscribe()
+	for {
+		select {
+		case event := <-stateChannel:
+			if event.Type == statefeed.Synced {
+				data, ok := event.Data.(*statefeed.SyncedData)
+				if !ok {
+					return errors.New("event data is not type *statefeed.SyncedData")
+				}
+				log.WithField("starttime", data.StartTime).Debug("Received sync completed event")
+				log.Info("Sending genesis time notification to connected validator clients")
+				res := &ethpb.SyncedResponse{
+					Synced:      true,
+					GenesisTime: uint64(data.StartTime.Unix()),
+				}
+				return stream.Send(res)
+			}
+		case <-stateSub.Err():
+			return status.Error(codes.Aborted, "Subscriber closed, exiting goroutine")
+		case <-vs.Ctx.Done():
+			return status.Error(codes.Canceled, "Context canceled")
+		}
+	}
 }

beacon-chain/rpc/validator/server_test.go

@@ -22,6 +22,7 @@ import (
 	internal "github.com/prysmaticlabs/prysm/beacon-chain/rpc/testing"
 	mockRPC "github.com/prysmaticlabs/prysm/beacon-chain/rpc/testing"
 	stateTrie "github.com/prysmaticlabs/prysm/beacon-chain/state"
+	mockSync "github.com/prysmaticlabs/prysm/beacon-chain/sync/initial-sync/testing"
 	pbp2p "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
 	"github.com/prysmaticlabs/prysm/shared/bls"
 	"github.com/prysmaticlabs/prysm/shared/event"
@@ -333,3 +334,121 @@ func TestWaitForChainStart_NotStartedThenLogFired(t *testing.T) {
 	exitRoutine <- true
 	testutil.AssertLogsContain(t, hook, "Sending genesis time")
 }
+
+func TestWaitForSynced_ContextClosed(t *testing.T) {
+	db := dbutil.SetupDB(t)
+	defer dbutil.TeardownDB(t, db)
+	ctx := context.Background()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	chainService := &mockChain.ChainService{}
+	Server := &Server{
+		Ctx: ctx,
+		ChainStartFetcher: &mockPOW.FaultyMockPOWChain{
+			ChainFeed: new(event.Feed),
+		},
+		StateNotifier: chainService.StateNotifier(),
+		BeaconDB:      db,
+		HeadFetcher:   chainService,
+	}
+
+	exitRoutine := make(chan bool)
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+	mockStream := mockRPC.NewMockBeaconNodeValidator_WaitForSyncedServer(ctrl)
+	go func(tt *testing.T) {
+		if err := Server.WaitForSynced(&ptypes.Empty{}, mockStream); err == nil || !strings.Contains(err.Error(), "Context canceled") {
+			tt.Errorf("Could not call RPC method: %v", err)
+		}
+		<-exitRoutine
+	}(t)
+	cancel()
+	exitRoutine <- true
+}
+
+func TestWaitForSynced_AlreadySynced(t *testing.T) {
+	db := dbutil.SetupDB(t)
+	defer dbutil.TeardownDB(t, db)
+	ctx := context.Background()
+	headBlockRoot := [32]byte{0x01, 0x02}
+	trie := testutil.NewBeaconState()
+	if err := trie.SetSlot(3); err != nil {
+		t.Fatal(err)
+	}
+	if err := db.SaveState(ctx, trie, headBlockRoot); err != nil {
+		t.Fatal(err)
+	}
+	if err := db.SaveHeadBlockRoot(ctx, headBlockRoot); err != nil {
+		t.Fatal(err)
+	}
+
+	chainService := &mockChain.ChainService{State: trie}
+	Server := &Server{
+		Ctx: context.Background(),
+		ChainStartFetcher: &mockPOW.POWChain{
+			ChainFeed: new(event.Feed),
+		},
+		BeaconDB:      db,
+		StateNotifier: chainService.StateNotifier(),
+		HeadFetcher:   chainService,
+		SyncChecker:   &mockSync.Sync{IsSyncing: false},
+	}
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+	mockStream := mockRPC.NewMockBeaconNodeValidator_WaitForSyncedServer(ctrl)
+	mockStream.EXPECT().Send(
+		&ethpb.SyncedResponse{
+			Synced:      true,
+			GenesisTime: uint64(time.Unix(0, 0).Unix()),
+		},
+	).Return(nil)
+	if err := Server.WaitForSynced(&ptypes.Empty{}, mockStream); err != nil {
+		t.Errorf("Could not call RPC method: %v", err)
+	}
+}
+
+func TestWaitForSynced_NotStartedThenLogFired(t *testing.T) {
+	db := dbutil.SetupDB(t)
+	defer dbutil.TeardownDB(t, db)
+
+	hook := logTest.NewGlobal()
+	chainService := &mockChain.ChainService{}
+	Server := &Server{
+		Ctx: context.Background(),
+		ChainStartFetcher: &mockPOW.FaultyMockPOWChain{
+			ChainFeed: new(event.Feed),
+		},
+		BeaconDB:      db,
+		StateNotifier: chainService.StateNotifier(),
+		HeadFetcher:   chainService,
+	}
+	exitRoutine := make(chan bool)
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+	mockStream := mockRPC.NewMockBeaconNodeValidator_WaitForSyncedServer(ctrl)
+	mockStream.EXPECT().Send(
+		&ethpb.SyncedResponse{
+			Synced:      true,
+			GenesisTime: uint64(time.Unix(0, 0).Unix()),
+		},
+	).Return(nil)
+	go func(tt *testing.T) {
+		if err := Server.WaitForSynced(&ptypes.Empty{}, mockStream); err != nil {
+			tt.Errorf("Could not call RPC method: %v", err)
+		}
+		<-exitRoutine
+	}(t)
+
+	// Send in a loop to ensure it is delivered (busy wait for the service to subscribe to the state feed).
+	for sent := 0; sent == 0; {
+		sent = Server.StateNotifier.StateFeed().Send(&feed.Event{
+			Type: statefeed.Synced,
+			Data: &statefeed.SyncedData{
+				StartTime: time.Unix(0, 0),
+			},
+		})
+	}
+
+	exitRoutine <- true
+	testutil.AssertLogsContain(t, hook, "Sending genesis time")
+}

beacon-chain/state/field_trie_test.go

@@ -85,10 +85,10 @@ func TestFieldTrie_CopyTrieImmutable(t *testing.T) {
 	changedIdx := []uint64{2, 29}
 
 	changedVals := [][32]byte{{'A', 'B'}, {'C', 'D'}}
-	if err := newState.UpdateRandaoMixesAtIndex(changedVals[0][:], changedIdx[0]); err != nil {
+	if err := newState.UpdateRandaoMixesAtIndex(changedIdx[0], changedVals[0][:]); err != nil {
 		t.Fatal(err)
 	}
-	if err := newState.UpdateRandaoMixesAtIndex(changedVals[1][:], changedIdx[1]); err != nil {
+	if err := newState.UpdateRandaoMixesAtIndex(changedIdx[1], changedVals[1][:]); err != nil {
 		t.Fatal(err)
 	}
 

beacon-chain/state/references_test.go

@@ -36,7 +36,7 @@ func TestStateReferenceSharing_Finalizer(t *testing.T) {
 	if b.sharedFieldReferences[randaoMixes].refs != 2 {
 		t.Error("Expected 2 shared references to randao mixes")
 	}
-	if err := b.UpdateRandaoMixesAtIndex([]byte("bar"), 0); err != nil {
+	if err := b.UpdateRandaoMixesAtIndex(0, []byte("bar")); err != nil {
 		t.Fatal(err)
 	}
 	if b.sharedFieldReferences[randaoMixes].refs != 1 || a.sharedFieldReferences[randaoMixes].refs != 1 {

beacon-chain/state/setters.go

@@ -427,7 +427,7 @@ func (b *BeaconState) SetRandaoMixes(val [][]byte) error {
 
 // UpdateRandaoMixesAtIndex for the beacon state. This PR updates the randao mixes
 // at a specific index to a new value.
-func (b *BeaconState) UpdateRandaoMixesAtIndex(val []byte, idx uint64) error {
+func (b *BeaconState) UpdateRandaoMixesAtIndex(idx uint64, val []byte) error {
 	if !b.HasInnerState() {
 		return ErrNilInnerState
 	}

beacon-chain/state/stategen/getter_test.go

@@ -69,6 +69,9 @@ func TestStateByRoot_HotStateDB(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
+	if err := service.beaconDB.SaveBlock(ctx, blk); err != nil {
+		t.Fatal(err)
+	}
 	if err := service.beaconDB.SaveGenesisBlockRoot(ctx, blkRoot); err != nil {
 		t.Fatal(err)
 	}
@@ -76,7 +79,21 @@ func TestStateByRoot_HotStateDB(t *testing.T) {
 		t.Fatal(err)
 	}
 	targetSlot := uint64(10)
-	targetRoot := [32]byte{'a'}
+	targetBlock := &ethpb.SignedBeaconBlock{Block: &ethpb.BeaconBlock{Slot: targetSlot, ParentRoot: blkRoot[:], ProposerIndex: 7}}
+	if err := service.beaconDB.SaveBlock(ctx, targetBlock); err != nil {
+		t.Fatal(err)
+	}
+	targetRoot, err := ssz.HashTreeRoot(targetBlock.Block)
+	if err != nil {
+		t.Fatal(err)
+	}
+	beaconState, _ = testutil.DeterministicGenesisState(t, 32)
+	if err := beaconState.SetSlot(10); err != nil {
+		t.Fatal(err)
+	}
+	if err := service.beaconDB.SaveState(ctx, beaconState, targetRoot); err != nil {
+		t.Fatal(err)
+	}
 	if err := service.beaconDB.SaveStateSummary(ctx, &pb.StateSummary{
 		Slot: targetSlot,
 		Root: targetRoot[:],

beacon-chain/state/stategen/hot.go

@@ -13,6 +13,15 @@ import (
 	"go.opencensus.io/trace"
 )
 
+// HasState returns true if the state exists in cache or in DB.
+func (s *State) HasState(ctx context.Context, blockRoot [32]byte) bool {
+	if s.hotStateCache.Has(blockRoot) {
+		return true
+	}
+
+	return s.beaconDB.HasState(ctx, blockRoot)
+}
+
 // This saves a post finalized beacon state in the hot section of the DB. On the epoch boundary,
 // it saves a full state. On an intermediate slot, it saves a back pointer to the
 // nearest epoch boundary state.
@@ -70,9 +79,11 @@ func (s *State) loadHotStateByRoot(ctx context.Context, blockRoot [32]byte) (*st
 		return nil, errors.Wrap(err, "could not get state summary")
 	}
 
-	startState, err := s.lastSavedState(ctx, summary.Slot)
+	// Since the hot state is not in cache nor DB, start replaying using the parent state which is
+	// retrieved using input block's parent root.
+	startState, err := s.lastAncestorState(ctx, blockRoot)
 	if err != nil {
-		return nil, err
+		return nil, errors.Wrap(err, "could not get ancestor state")
 	}
 	if startState == nil {
 		return nil, errUnknownBoundaryState
@@ -130,3 +141,37 @@ func (s *State) loadHotStateBySlot(ctx context.Context, slot uint64) (*state.Bea
 
 	return s.ReplayBlocks(ctx, startState, replayBlks, slot)
 }
+
+// This returns the last saved in DB ancestor state of the input block root.
+// It recursively look up block's parent until a corresponding state of the block root
+// is found in the DB.
+func (s *State) lastAncestorState(ctx context.Context, root [32]byte) (*state.BeaconState, error) {
+	ctx, span := trace.StartSpan(ctx, "stateGen.lastAncestorState")
+	defer span.End()
+
+	b, err := s.beaconDB.Block(ctx, root)
+	if err != nil {
+		return nil, err
+	}
+	if b == nil {
+		return nil, errUnknownBlock
+	}
+
+	for {
+		if ctx.Err() != nil {
+			return nil, ctx.Err()
+		}
+		parentRoot := bytesutil.ToBytes32(b.Block.ParentRoot)
+		if s.beaconDB.HasState(ctx, parentRoot) {
+			return s.beaconDB.State(ctx, parentRoot)
+		}
+
+		b, err = s.beaconDB.Block(ctx, parentRoot)
+		if err != nil {
+			return nil, err
+		}
+		if b == nil {
+			return nil, errUnknownBlock
+		}
+	}
+}

beacon-chain/state/stategen/hot_test.go

@@ -158,6 +158,13 @@ func TestLoadHoteStateByRoot_FromDBCanProcess(t *testing.T) {
 	}); err != nil {
 		t.Fatal(err)
 	}
+	beaconState, _ = testutil.DeterministicGenesisState(t, 32)
+	if err := beaconState.SetSlot(10); err != nil {
+		t.Fatal(err)
+	}
+	if err := service.beaconDB.SaveState(ctx, beaconState, targetRoot); err != nil {
+		t.Fatal(err)
+	}
 
 	// This tests where hot state was not cached and needs processing.
 	loadedState, err := service.loadHotStateByRoot(ctx, targetRoot)
@@ -238,3 +245,60 @@ func TestLoadHoteStateBySlot_CanAdvanceSlotUsingDB(t *testing.T) {
 		t.Error("Did not correctly load state")
 	}
 }
+
+func TestLastAncestorState_CanGet(t *testing.T) {
+	ctx := context.Background()
+	db := testDB.SetupDB(t)
+	defer testDB.TeardownDB(t, db)
+	service := New(db, cache.NewStateSummaryCache())
+
+	b0 := &ethpb.BeaconBlock{Slot: 0, ParentRoot: []byte{'a'}}
+	r0, err := ssz.HashTreeRoot(b0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	b1 := &ethpb.BeaconBlock{Slot: 1, ParentRoot: r0[:]}
+	r1, err := ssz.HashTreeRoot(b1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	b2 := &ethpb.BeaconBlock{Slot: 2, ParentRoot: r1[:]}
+	r2, err := ssz.HashTreeRoot(b2)
+	if err != nil {
+		t.Fatal(err)
+	}
+	b3 := &ethpb.BeaconBlock{Slot: 3, ParentRoot: r2[:]}
+	r3, err := ssz.HashTreeRoot(b3)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	b1State := testutil.NewBeaconState()
+	if err := b1State.SetSlot(1); err != nil {
+		t.Fatal(err)
+	}
+
+	if err := service.beaconDB.SaveBlock(ctx, &ethpb.SignedBeaconBlock{Block: b0}); err != nil {
+		t.Fatal(err)
+	}
+	if err := service.beaconDB.SaveBlock(ctx, &ethpb.SignedBeaconBlock{Block: b1}); err != nil {
+		t.Fatal(err)
+	}
+	if err := service.beaconDB.SaveBlock(ctx, &ethpb.SignedBeaconBlock{Block: b2}); err != nil {
+		t.Fatal(err)
+	}
+	if err := service.beaconDB.SaveBlock(ctx, &ethpb.SignedBeaconBlock{Block: b3}); err != nil {
+		t.Fatal(err)
+	}
+	if err := service.beaconDB.SaveState(ctx, b1State, r1); err != nil {
+		t.Fatal(err)
+	}
+
+	lastState, err := service.lastAncestorState(ctx, r3)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if lastState.Slot() != b1State.Slot() {
+		t.Error("Did not get wanted state")
+	}
+}

beacon-chain/state/types_test.go

@@ -208,7 +208,7 @@ func TestBeaconState_ImmutabilityWithSharedResources(t *testing.T) {
 	if !reflect.DeepEqual(a.RandaoMixes(), b.RandaoMixes()) {
 		t.Fatal("Test precondition failed, fields are not equal")
 	}
-	if err := a.UpdateRandaoMixesAtIndex([]byte("foo"), 1); err != nil {
+	if err := a.UpdateRandaoMixesAtIndex(1, []byte("foo")); err != nil {
 		t.Fatal(err)
 	}
 	if reflect.DeepEqual(a.RandaoMixes(), b.RandaoMixes()) {

beacon-chain/sync/BUILD.bazel

@@ -62,6 +62,7 @@ go_library(
         "//shared/featureconfig:go_default_library",
         "//shared/hashutil:go_default_library",
         "//shared/messagehandler:go_default_library",
+        "//shared/p2putils:go_default_library",
         "//shared/params:go_default_library",
         "//shared/roughtime:go_default_library",
         "//shared/runutil:go_default_library",

beacon-chain/sync/deadlines.go

@@ -4,10 +4,11 @@ import (
 	"time"
 
 	"github.com/libp2p/go-libp2p-core/network"
+	"github.com/prysmaticlabs/prysm/shared/params"
 )
 
-const defaultReadDuration = ttfbTimeout
-const defaultWriteDuration = 10 * time.Second // RESP_TIMEOUT
+var defaultReadDuration = ttfbTimeout
+var defaultWriteDuration = params.BeaconNetworkConfig().RespTimeout // RESP_TIMEOUT
 
 func setRPCStreamDeadlines(stream network.Stream) {
 	setStreamReadDeadline(stream, defaultReadDuration)

beacon-chain/sync/decode_pubsub.go

@@ -22,7 +22,7 @@ func (r *Service) decodePubsubMessage(msg *pubsub.Message) (proto.Message, error
 		return nil, fmt.Errorf("no message mapped for topic %s", topic)
 	}
 	m := proto.Clone(base)
-	if err := r.p2p.Encoding().Decode(msg.Data, m); err != nil {
+	if err := r.p2p.Encoding().DecodeGossip(msg.Data, m); err != nil {
 		return nil, err
 	}
 	return m, nil

beacon-chain/sync/initial-sync/service.go

@@ -120,6 +120,12 @@ func (s *Service) Start() {
 	if helpers.SlotToEpoch(currentSlot) == 0 {
 		log.Info("Chain started within the last epoch - not syncing")
 		s.synced = true
+		s.stateNotifier.StateFeed().Send(&feed.Event{
+			Type: statefeed.Synced,
+			Data: &statefeed.SyncedData{
+				StartTime: genesis,
+			},
+		})
 		return
 	}
 	log.Info("Starting initial chain sync...")
@@ -127,6 +133,12 @@ func (s *Service) Start() {
 	if helpers.SlotToEpoch(s.chain.HeadSlot()) == helpers.SlotToEpoch(currentSlot) {
 		log.Info("Already synced to the current chain head")
 		s.synced = true
+		s.stateNotifier.StateFeed().Send(&feed.Event{
+			Type: statefeed.Synced,
+			Data: &statefeed.SyncedData{
+				StartTime: genesis,
+			},
+		})
 		return
 	}
 	s.waitForMinimumPeers()
@@ -135,6 +147,12 @@ func (s *Service) Start() {
 	}
 	log.Infof("Synced up to slot %d", s.chain.HeadSlot())
 	s.synced = true
+	s.stateNotifier.StateFeed().Send(&feed.Event{
+		Type: statefeed.Synced,
+		Data: &statefeed.SyncedData{
+			StartTime: genesis,
+		},
+	})
 }
 
 // Stop initial sync.

beacon-chain/sync/metrics.go

@@ -72,8 +72,13 @@ var (
 )
 
 func (r *Service) updateMetrics() {
+	// do not update metrics if genesis time
+	// has not been initialized
+	if r.chain.GenesisTime().IsZero() {
+		return
+	}
 	// We update the dynamic subnet topics.
-	digest, err := r.p2p.ForkDigest()
+	digest, err := r.forkDigest()
 	if err != nil {
 		log.WithError(err).Errorf("Could not compute fork digest")
 	}

beacon-chain/sync/rpc.go

@@ -4,12 +4,12 @@ import (
 	"context"
 	"reflect"
 	"strings"
-	"time"
 
 	libp2pcore "github.com/libp2p/go-libp2p-core"
 	"github.com/libp2p/go-libp2p-core/network"
 	"github.com/prysmaticlabs/prysm/beacon-chain/p2p"
 	pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
+	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/prysmaticlabs/prysm/shared/roughtime"
 	"github.com/prysmaticlabs/prysm/shared/traceutil"
 	"go.opencensus.io/trace"
@@ -18,12 +18,12 @@ import (
 // Time to first byte timeout. The maximum time to wait for first byte of
 // request response (time-to-first-byte). The client is expected to give up if
 // they don't receive the first byte within 5 seconds.
-const ttfbTimeout = 5 * time.Second
+var ttfbTimeout = params.BeaconNetworkConfig().TtfbTimeout
 
 // maxChunkSize would be the maximum allowed size that a request/response chunk can be.
 // any size beyond that would be rejected and the corresponding stream reset. This would
 // be 1048576 bytes or 1 MiB.
-const maxChunkSize = 1 << 20
+var maxChunkSize = params.BeaconNetworkConfig().MaxChunkSize
 
 // rpcHandler is responsible for handling and responding to any incoming message.
 // This method may return an error to internal monitoring, but the error will

beacon-chain/sync/rpc_status.go

@@ -78,7 +78,7 @@ func (r *Service) sendRPCStatusRequest(ctx context.Context, id peer.ID) error {
 		return err
 	}
 
-	forkDigest, err := r.p2p.ForkDigest()
+	forkDigest, err := r.forkDigest()
 	if err != nil {
 		return err
 	}
@@ -179,7 +179,7 @@ func (r *Service) statusRPCHandler(ctx context.Context, msg interface{}, stream
 		return err
 	}
 
-	forkDigest, err := r.p2p.ForkDigest()
+	forkDigest, err := r.forkDigest()
 	if err != nil {
 		return err
 	}
@@ -200,7 +200,7 @@ func (r *Service) statusRPCHandler(ctx context.Context, msg interface{}, stream
 }
 
 func (r *Service) validateStatusMessage(msg *pb.Status, stream network.Stream) error {
-	forkDigest, err := r.p2p.ForkDigest()
+	forkDigest, err := r.forkDigest()
 	if err != nil {
 		return err
 	}

beacon-chain/sync/rpc_status_test.go

@@ -35,7 +35,11 @@ func TestHelloRPCHandler_Disconnects_OnForkVersionMismatch(t *testing.T) {
 		t.Error("Expected peers to be connected")
 	}
 
-	r := &Service{p2p: p1}
+	r := &Service{p2p: p1,
+		chain: &mock.ChainService{
+			Genesis:        time.Now(),
+			ValidatorsRoot: [32]byte{'A'},
+		}}
 	pcl := protocol.ID("/testing")
 
 	var wg sync.WaitGroup
@@ -116,8 +120,14 @@ func TestHelloRPCHandler_ReturnsHelloMessage(t *testing.T) {
 				PreviousVersion: params.BeaconConfig().GenesisForkVersion,
 				CurrentVersion:  params.BeaconConfig().GenesisForkVersion,
 			},
+			ValidatorsRoot: [32]byte{'A'},
+			Genesis:        time.Now(),
 		},
 	}
+	digest, err := r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	// Setup streams
 	pcl := protocol.ID("/testing")
@@ -131,7 +141,7 @@ func TestHelloRPCHandler_ReturnsHelloMessage(t *testing.T) {
 			t.Fatal(err)
 		}
 		expected := &pb.Status{
-			ForkDigest:     params.BeaconConfig().GenesisForkVersion,
+			ForkDigest:     digest[:],
 			HeadSlot:       genesisState.Slot(),
 			HeadRoot:       headRoot[:],
 			FinalizedEpoch: 5,
@@ -146,7 +156,7 @@ func TestHelloRPCHandler_ReturnsHelloMessage(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = r.statusRPCHandler(context.Background(), &pb.Status{ForkDigest: params.BeaconConfig().GenesisForkVersion}, stream1)
+	err = r.statusRPCHandler(context.Background(), &pb.Status{ForkDigest: digest[:]}, stream1)
 	if err != nil {
 		t.Errorf("Unxpected error: %v", err)
 	}
@@ -187,12 +197,23 @@ func TestHandshakeHandlers_Roundtrip(t *testing.T) {
 				PreviousVersion: params.BeaconConfig().GenesisForkVersion,
 				CurrentVersion:  params.BeaconConfig().GenesisForkVersion,
 			},
+			Genesis:        time.Now(),
+			ValidatorsRoot: [32]byte{'A'},
 		},
 		ctx: context.Background(),
 	}
+	p1.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
+
 	r2 := &Service{
 		p2p: p2,
 	}
+	p2.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	r.Start()
 
@@ -208,7 +229,7 @@ func TestHandshakeHandlers_Roundtrip(t *testing.T) {
 		}
 		log.WithField("status", out).Warn("received status")
 
-		resp := &pb.Status{HeadSlot: 100, ForkDigest: params.BeaconConfig().GenesisForkVersion}
+		resp := &pb.Status{HeadSlot: 100, ForkDigest: p2.Digest[:]}
 
 		if _, err := stream.Write([]byte{responseCodeSuccess}); err != nil {
 			t.Fatal(err)
@@ -331,6 +352,8 @@ func TestStatusRPCRequest_RequestSent(t *testing.T) {
 				PreviousVersion: params.BeaconConfig().GenesisForkVersion,
 				CurrentVersion:  params.BeaconConfig().GenesisForkVersion,
 			},
+			Genesis:        time.Now(),
+			ValidatorsRoot: [32]byte{'A'},
 		},
 		ctx: context.Background(),
 	}
@@ -345,8 +368,12 @@ func TestStatusRPCRequest_RequestSent(t *testing.T) {
 		if err := r.p2p.Encoding().DecodeWithLength(stream, out); err != nil {
 			t.Fatal(err)
 		}
+		digest, err := r.forkDigest()
+		if err != nil {
+			t.Fatal(err)
+		}
 		expected := &pb.Status{
-			ForkDigest:     params.BeaconConfig().GenesisForkVersion,
+			ForkDigest:     digest[:],
 			HeadSlot:       genesisState.Slot(),
 			HeadRoot:       headRoot[:],
 			FinalizedEpoch: 5,
@@ -407,6 +434,8 @@ func TestStatusRPCRequest_BadPeerHandshake(t *testing.T) {
 				PreviousVersion: params.BeaconConfig().GenesisForkVersion,
 				CurrentVersion:  params.BeaconConfig().GenesisForkVersion,
 			},
+			Genesis:        time.Now(),
+			ValidatorsRoot: [32]byte{'A'},
 		},
 		ctx: context.Background(),
 	}

beacon-chain/sync/service.go

@@ -59,6 +59,7 @@ type blockchainService interface {
 	blockchain.ForkFetcher
 	blockchain.AttestationReceiver
 	blockchain.TimeFetcher
+	blockchain.GenesisFetcher
 }
 
 // NewRegularSync service.

beacon-chain/sync/subscriber.go

@@ -12,20 +12,22 @@ import (
 	"github.com/libp2p/go-libp2p-core/peer"
 	pubsub "github.com/libp2p/go-libp2p-pubsub"
 	pb "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
-	"go.opencensus.io/trace"
 	"github.com/prysmaticlabs/prysm/beacon-chain/core/feed"
 	statefeed "github.com/prysmaticlabs/prysm/beacon-chain/core/feed/state"
 	"github.com/prysmaticlabs/prysm/beacon-chain/p2p"
 	"github.com/prysmaticlabs/prysm/shared/featureconfig"
 	"github.com/prysmaticlabs/prysm/shared/messagehandler"
+	"github.com/prysmaticlabs/prysm/shared/p2putils"
 	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/prysmaticlabs/prysm/shared/roughtime"
 	"github.com/prysmaticlabs/prysm/shared/slotutil"
 	"github.com/prysmaticlabs/prysm/shared/traceutil"
+	"go.opencensus.io/trace"
 )
 
 const pubsubMessageTimeout = 30 * time.Second
-const maximumGossipClockDisparity = 500 * time.Millisecond
+
+var maximumGossipClockDisparity = params.BeaconNetworkConfig().MaximumGossipClockDisparity
 
 // subHandler represents handler for a given subscription.
 type subHandler func(context.Context, proto.Message) error
@@ -218,7 +220,7 @@ func (r *Service) subscribeDynamicWithSubnets(
 	if base == nil {
 		log.Fatalf("%s is not mapped to any message in GossipTopicMappings", topicFormat)
 	}
-	digest, err := r.p2p.ForkDigest()
+	digest, err := r.forkDigest()
 	if err != nil {
 		log.WithError(err).Fatal("Could not compute fork digest")
 	}
@@ -266,7 +268,7 @@ func (r *Service) subscribeDynamic(topicFormat string, determineSubsLen func() i
 	if base == nil {
 		log.Fatalf("%s is not mapped to any message in GossipTopicMappings", topicFormat)
 	}
-	digest, err := r.p2p.ForkDigest()
+	digest, err := r.forkDigest()
 	if err != nil {
 		log.WithError(err).Fatal("Could not compute fork digest")
 	}
@@ -384,9 +386,14 @@ func (r *Service) addDigestToTopic(topic string) string {
 	if !strings.Contains(topic, "%x") {
 		log.Fatal("Topic does not have appropriate formatter for digest")
 	}
-	digest, err := r.p2p.ForkDigest()
+	digest, err := r.forkDigest()
 	if err != nil {
 		log.WithError(err).Fatal("Could not compute fork digest")
 	}
 	return fmt.Sprintf(topic, digest)
 }
+
+func (r *Service) forkDigest() ([4]byte, error) {
+	genRoot := r.chain.GenesisValidatorRoot()
+	return p2putils.CreateForkDigest(r.chain.GenesisTime(), genRoot[:])
+}

beacon-chain/sync/subscriber_committee_index_beacon_attestation_test.go

@@ -61,6 +61,7 @@ func TestService_committeeIndexBeaconAttestationSubscriber_ValidMessage(t *testi
 			State:            s,
 			Genesis:          time.Now(),
 			ValidAttestation: true,
+			ValidatorsRoot:   [32]byte{'A'},
 		},
 		chainStarted:         true,
 		p2p:                  p,
@@ -72,6 +73,10 @@ func TestService_committeeIndexBeaconAttestationSubscriber_ValidMessage(t *testi
 		seenAttestationCache: c,
 		stateSummaryCache:    cache.NewStateSummaryCache(),
 	}
+	p.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 	r.registerSubscribers()
 	r.stateNotifier.StateFeed().Send(&feed.Event{
 		Type: statefeed.Initialized,

beacon-chain/sync/subscriber_test.go

@@ -31,6 +31,15 @@ func TestSubscribe_ReceivesValidMessage(t *testing.T) {
 		ctx:         context.Background(),
 		p2p:         p2p,
 		initialSync: &mockSync.Sync{IsSyncing: false},
+		chain: &mockChain.ChainService{
+			ValidatorsRoot: [32]byte{'A'},
+			Genesis:        time.Now(),
+		},
+	}
+	var err error
+	p2p.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
 	}
 	topic := "/eth2/%x/voluntary_exit"
 	var wg sync.WaitGroup
@@ -61,7 +70,10 @@ func TestSubscribe_ReceivesAttesterSlashing(t *testing.T) {
 	ctx := context.Background()
 	d := db.SetupDB(t)
 	defer db.TeardownDB(t, d)
-	chainService := &mockChain.ChainService{}
+	chainService := &mockChain.ChainService{
+		Genesis:        time.Now(),
+		ValidatorsRoot: [32]byte{'A'},
+	}
 	c, err := lru.New(10)
 	if err != nil {
 		t.Fatal(err)
@@ -101,6 +113,10 @@ func TestSubscribe_ReceivesAttesterSlashing(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
+	p2p.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 	p2p.ReceivePubSub(topic, attesterSlashing)
 
 	if testutil.WaitTimeout(&wg, time.Second) {
@@ -115,7 +131,10 @@ func TestSubscribe_ReceivesAttesterSlashing(t *testing.T) {
 func TestSubscribe_ReceivesProposerSlashing(t *testing.T) {
 	p2p := p2ptest.NewTestP2P(t)
 	ctx := context.Background()
-	chainService := &mockChain.ChainService{}
+	chainService := &mockChain.ChainService{
+		ValidatorsRoot: [32]byte{'A'},
+		Genesis:        time.Now(),
+	}
 	d := db.SetupDB(t)
 	defer db.TeardownDB(t, d)
 	c, err := lru.New(10)
@@ -157,6 +176,10 @@ func TestSubscribe_ReceivesProposerSlashing(t *testing.T) {
 	if err := r.db.SaveState(ctx, beaconState, root); err != nil {
 		t.Fatal(err)
 	}
+	p2p.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 	p2p.ReceivePubSub(topic, proposerSlashing)
 
 	if testutil.WaitTimeout(&wg, time.Second) {
@@ -170,7 +193,10 @@ func TestSubscribe_ReceivesProposerSlashing(t *testing.T) {
 
 func TestSubscribe_WaitToSync(t *testing.T) {
 	p2p := p2ptest.NewTestP2P(t)
-	chainService := &mockChain.ChainService{}
+	chainService := &mockChain.ChainService{
+		Genesis:        time.Now(),
+		ValidatorsRoot: [32]byte{'A'},
+	}
 	r := Service{
 		ctx:           context.Background(),
 		p2p:           p2p,
@@ -217,8 +243,17 @@ func TestSubscribe_HandlesPanic(t *testing.T) {
 	p := p2ptest.NewTestP2P(t)
 	r := Service{
 		ctx: context.Background(),
+		chain: &mockChain.ChainService{
+			Genesis:        time.Now(),
+			ValidatorsRoot: [32]byte{'A'},
+		},
 		p2p: p,
 	}
+	var err error
+	p.Digest, err = r.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	topic := p2p.GossipTypeMapping[reflect.TypeOf(&pb.SignedVoluntaryExit{})]
 	var wg sync.WaitGroup

beacon-chain/sync/validate_aggregate_proof.go

@@ -197,7 +197,7 @@ func validateIndexInCommittee(ctx context.Context, s *stateTrie.BeaconState, a *
 func validateAggregateAttTime(attSlot uint64, genesisTime uint64) error {
 	// in milliseconds
 	attTime := 1000 * (genesisTime + (attSlot * params.BeaconConfig().SecondsPerSlot))
-	attSlotRange := attSlot + params.BeaconConfig().AttestationPropagationSlotRange
+	attSlotRange := attSlot + params.BeaconNetworkConfig().AttestationPropagationSlotRange
 	attTimeRange := 1000 * (genesisTime + (attSlotRange * params.BeaconConfig().SecondsPerSlot))
 	currentTimeInSec := roughtime.Now().Unix()
 	currentTime := 1000 * currentTimeInSec
@@ -206,7 +206,7 @@ func validateAggregateAttTime(attSlot uint64, genesisTime uint64) error {
 	currentSlot := (uint64(currentTimeInSec) - genesisTime) / params.BeaconConfig().SecondsPerSlot
 	if attTime-uint64(maximumGossipClockDisparity.Milliseconds()) > uint64(currentTime) ||
 		uint64(currentTime-maximumGossipClockDisparity.Milliseconds()) > attTimeRange {
-		return fmt.Errorf("attestation slot out of range %d <= %d <= %d", attSlot, currentSlot, attSlot+params.BeaconConfig().AttestationPropagationSlotRange)
+		return fmt.Errorf("attestation slot out of range %d <= %d <= %d", attSlot, currentSlot, attSlot+params.BeaconNetworkConfig().AttestationPropagationSlotRange)
 	}
 	return nil
 }

beacon-chain/sync/validate_aggregate_proof_test.go

@@ -549,11 +549,13 @@ func TestVerifyIndexInCommittee_SeenAggregatorSlot(t *testing.T) {
 		db:          db,
 		initialSync: &mockSync.Sync{IsSyncing: false},
 		chain: &mock.ChainService{Genesis: time.Now(),
+			ValidatorsRoot:   [32]byte{'A'},
 			State:            beaconState,
 			ValidAttestation: true,
 			FinalizedCheckPoint: &ethpb.Checkpoint{
 				Epoch: 0,
 			}},
+
 		attPool:              attestations.NewPool(),
 		seenAttestationCache: c,
 		stateSummaryCache:    cache.NewStateSummaryCache(),

beacon-chain/sync/validate_beacon_blocks.go

@@ -10,6 +10,7 @@ import (
 	"github.com/prysmaticlabs/prysm/beacon-chain/core/blocks"
 	"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
 	"github.com/prysmaticlabs/prysm/shared/bytesutil"
+	"github.com/prysmaticlabs/prysm/shared/featureconfig"
 	"github.com/prysmaticlabs/prysm/shared/traceutil"
 	"go.opencensus.io/trace"
 )
@@ -89,36 +90,38 @@ func (r *Service) validateBeaconBlockPubSub(ctx context.Context, pid peer.ID, ms
 		return false
 	}
 
-	hasStateSummaryDB := r.db.HasStateSummary(ctx, bytesutil.ToBytes32(blk.Block.ParentRoot))
-	hasStateSummaryCache := r.stateSummaryCache.Has(bytesutil.ToBytes32(blk.Block.ParentRoot))
-	if !hasStateSummaryDB && !hasStateSummaryCache {
-		log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("No access to parent state")
-		return false
-	}
-	parentState, err := r.stateGen.StateByRoot(ctx, bytesutil.ToBytes32(blk.Block.ParentRoot))
-	if err != nil {
-		log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not get parent state")
-		return false
-	}
+	if !featureconfig.Get().DisableNewStateMgmt {
+		hasStateSummaryDB := r.db.HasStateSummary(ctx, bytesutil.ToBytes32(blk.Block.ParentRoot))
+		hasStateSummaryCache := r.stateSummaryCache.Has(bytesutil.ToBytes32(blk.Block.ParentRoot))
+		if !hasStateSummaryDB && !hasStateSummaryCache {
+			log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("No access to parent state")
+			return false
+		}
+		parentState, err := r.stateGen.StateByRoot(ctx, bytesutil.ToBytes32(blk.Block.ParentRoot))
+		if err != nil {
+			log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not get parent state")
+			return false
+		}
 
-	if err := blocks.VerifyBlockHeaderSignature(parentState, blk); err != nil {
-		log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not verify block signature")
-		return false
-	}
+		if err := blocks.VerifyBlockHeaderSignature(parentState, blk); err != nil {
+			log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not verify block signature")
+			return false
+		}
 
-	err = parentState.SetSlot(blk.Block.Slot)
-	if err != nil {
-		log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not set parent state slot")
-		return false
-	}
-	idx, err := helpers.BeaconProposerIndex(parentState)
-	if err != nil {
-		log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not get proposer index using parent state")
-		return false
-	}
-	if blk.Block.ProposerIndex != idx {
-		log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Incorrect proposer index")
-		return false
+		err = parentState.SetSlot(blk.Block.Slot)
+		if err != nil {
+			log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not set parent state slot")
+			return false
+		}
+		idx, err := helpers.BeaconProposerIndex(parentState)
+		if err != nil {
+			log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Could not get proposer index using parent state")
+			return false
+		}
+		if blk.Block.ProposerIndex != idx {
+			log.WithError(err).WithField("blockSlot", blk.Block.Slot).Warn("Incorrect proposer index")
+			return false
+		}
 	}
 
 	msg.ValidatorData = blk // Used in downstream subscriber

beacon-chain/sync/validate_committee_index_beacon_attestation.go

@@ -62,7 +62,7 @@ func (s *Service) validateCommitteeIndexBeaconAttestation(ctx context.Context, p
 	}
 
 	// The attestation's committee index (attestation.data.index) is for the correct subnet.
-	digest, err := s.p2p.ForkDigest()
+	digest, err := s.forkDigest()
 	if err != nil {
 		log.WithError(err).Error("Failed to compute fork digest")
 		traceutil.AnnotateError(span, err)

beacon-chain/sync/validate_committee_index_beacon_attestation_test.go

@@ -3,6 +3,7 @@ package sync
 import (
 	"bytes"
 	"context"
+	"fmt"
 	"testing"
 	"time"
 
@@ -29,6 +30,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 	defer dbtest.TeardownDB(t, db)
 	chain := &mockChain.ChainService{
 		Genesis:          time.Now().Add(time.Duration(-64*int64(params.BeaconConfig().SecondsPerSlot)) * time.Second), // 64 slots ago
+		ValidatorsRoot:   [32]byte{'A'},
 		ValidAttestation: true,
 	}
 
@@ -45,6 +47,10 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 		seenAttestationCache: c,
 		stateSummaryCache:    cache.NewStateSummaryCache(),
 	}
+	digest, err := s.forkDigest()
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	blk := &ethpb.SignedBeaconBlock{
 		Block: &ethpb.BeaconBlock{
@@ -82,7 +88,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 					Slot:            63,
 				},
 			},
-			topic:                     "/eth2/00000000/committee_index1_beacon_attestation",
+			topic:                     fmt.Sprintf("/eth2/%x/committee_index1_beacon_attestation", digest),
 			validAttestationSignature: true,
 			want:                      true,
 		},
@@ -96,7 +102,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 					Slot:            63,
 				},
 			},
-			topic:                     "/eth2/00000000/committee_index1_beacon_attestation",
+			topic:                     fmt.Sprintf("/eth2/%x/committee_index1_beacon_attestation", digest),
 			validAttestationSignature: true,
 			want:                      false,
 		},
@@ -110,7 +116,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 					Slot:            63,
 				},
 			},
-			topic:                     "/eth2/00000000/committee_index3_beacon_attestation",
+			topic:                     fmt.Sprintf("/eth2/%x/committee_index3_beacon_attestation", digest),
 			validAttestationSignature: true,
 			want:                      false,
 		},
@@ -124,7 +130,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 					Slot:            63,
 				},
 			},
-			topic:                     "/eth2/00000000/committee_index1_beacon_attestation",
+			topic:                     fmt.Sprintf("/eth2/%x/committee_index1_beacon_attestation", digest),
 			validAttestationSignature: true,
 			want:                      false,
 		},
@@ -138,7 +144,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 					Slot:            63,
 				},
 			},
-			topic:                     "/eth2/00000000/committee_index1_beacon_attestation",
+			topic:                     fmt.Sprintf("/eth2/%x/committee_index1_beacon_attestation", digest),
 			validAttestationSignature: true,
 			want:                      false,
 		},
@@ -152,7 +158,7 @@ func TestService_validateCommitteeIndexBeaconAttestation(t *testing.T) {
 					Slot:            63,
 				},
 			},
-			topic:                     "/eth2/00000000/committee_index1_beacon_attestation",
+			topic:                     fmt.Sprintf("/eth2/%x/committee_index1_beacon_attestation", digest),
 			validAttestationSignature: false,
 			want:                      false,
 		},

beacon-chain/usage.go

@@ -108,6 +108,7 @@ var appHelpFlagGroups = []flagGroup{
 			cmd.StaticPeers,
 			cmd.EnableUPnPFlag,
 			cmd.P2PEncoding,
+			cmd.P2PPubsub,
 			flags.MinSyncPeers,
 		},
 	},

docs/DEMO_INFORMATION.md

@@ -1,58 +0,0 @@
-# Ethereum Serenity Prysm Demo
-
-**IMPORTANT: This document describes our v0.0.0 demo release from October 2018, which is no longer relevant to our current work and is only here for historical purposes**
-
-## Overview & Research Background
-
-At Prysmatic Labs, we started working on Ethereum Serenity all the way back since Vitalik first had a Sharding FAQ as the only reference for the system at the start of 2018. A lot has happened, with the specification evolving from a series of ETHResearch blog posts into a minimal viable blueprint for how to design a scalable, secure Ethereum blockchain using Casper Proof of Stake and Sharding at its core.
-
-Now, the plan is to deploy ETH2.0 as a sidechain of Ethereum known as a beacon chain, where validators can stake their Ether and participate in consensus to vote on occurrences on shards known as cross-links.
-
-## Version 0.0.0: Beacon Chain+Validator Demo
-
-We call our Ethereum Serenity project Prysm, which will serve as the scaffold for a production-oriented release fully written in Go. We have been working hard to ensure we have a robust implementation of the Ethereum Serenity specification as created by the Ethereum Research Team along with industry standard approaches to building large scale applications.
-
-We are proud to announce our very first release of Prysm, v0.0.0, which will serve as the building block for all future releases as we get to production. We want to show the community we have a project we have put a lot of work into through careful thought and design decisions that we hope will set a standard for future ETH2.0 developments.
-
-## What This Release Encompasses
-
-Version 0.0.0 includes a basic beacon-chain+validator demo which can do the following:
-
-- Spin up a beacon chain node from a genesis configuration
-- Connect a validator client via RPC
-- The validator client gets shuffled into a specific shard at a given slot
-- Validators propose/attest to canonical beacon blocks during their assigned slot
-- Validators get reshuffled into shards every new cycle transition
-- Casper FFG rewards/penalties are included in this release even though they are a constant area of research
-- Basic, locally networked p2p via libp2p and the mDNS discovery protocol
-- Beacon chain block sync through p2p (listening for incoming blocks + syncing to a latest head from scratch)
-- A useful simulator of beacon blocks (this allows us to simulate other beacon nodes relaying info to our node locally)
-- Storing blocks/attestations/states to a write-optimized key-value backend known as BoltDB
-- gRPC public API client/server for querying a beacon node for canonical blocks, states, and latest validator assignments
-- A robust, scalable build system known as Bazel used in production at Google, Pinterest, Dropbox, and other industry giants
-- A Web3 Subscription service to listen to latest mainchain blocks and validator registrations
-
-## Not Included in the Release
-
-- Although a Validator Registration Contract is included, validator rotation, withdrawals, and dynasty transitions are not yet included
-- Shards, their associated design, and cross-shard transactions are not included in this release
-- Fork-choice rule for ETH2.0 is not yet included, we use a naive fork choice rule for this release
-- Signature aggregation and verification are not included in this release
-- Randomness via RANDAO and VDF are not included in this release as they are an active area of research - we use basic block hashes as a stub for now
-- Serialization format for ETH2.0 is still an active area of research and is not included here
-- Shardp2p and beacon node p2p peer discovery have not yet been designed beyond mDNS
-- State execution is not included as it depends on shard functionality
-
-## How to Run the Demo
-
-Curious to see the system working and running a validator client locally? We have put together comprehensive instructions on running our demo in our Github repository! Try it out and see for yourself :). 
-
-[Running Instructions](https://github.com/prysmaticlabs/prysm/blob/master/README.md#instructions)
-
-You’ll be able to spin up a beacon node, connect a validator client, and start getting assigned to shards where you will then create beacon blocks or vote on beacon blocks through structures called “attestations”. If you are not quite as familiar with the Ethereum Serenity Roadmap and Spec, check out the following links:
-
-- [Ethereum Serenity Devs Handbook](https://notes.ethereum.org/s/BkSZAJNwX#)
-- [Ethereum Serenity Casper+Sharding Specification](https://github.com/ethereum/eth2.0-specs/tree/master/specs)
-
-Even though canonical blocks are created in the demo due to your activity as a validator, you’ll quickly see not much can be done with these blocks until real, state execution comes into play. However, this beacon chain is a critical piece of consensus and coordination for all actors participating in Ethereum Serenity and will as the foundation for a full-fledged, sharding system. 
-

docs/PRIOR_RESEARCH.md

@@ -1,113 +0,0 @@
-# Beacon Chain Research Synopsis
-
-This doc will summarize some historical discussions and roadmap updates around integrating Casper/Sharding through a beacon chain and what it means for Ethereum's end-game. 
-
-### Research Notes Leading Up To This
-
--   [Offchain Collation Headers](https://ethresear.ch/t/offchain-collation-headers/1679)
--   [RANDAO Beacon Exploitability Part. 1](https://ethresear.ch/t/rng-exploitability-analysis-assuming-pure-randao-based-main-chain/1825/9)
--   [Leaderless k-of-n Random Beacon](https://ethresear.ch/t/leaderless-k-of-n-random-beacon/2046)
--   [Two Ways to do Cross Links](https://ethresear.ch/t/two-ways-to-do-cross-links/2074)
--   [Registrations, Shard Count, and Shuffling](https://ethresear.ch/t/registrations-shard-count-and-shuffling/2129)
--   [Committee Based Sharded Casper](https://ethresear.ch/t/committee-based-sharded-casper/2197)
--   [Attestation Committee Based Full PoS Chains](https://ethresear.ch/t/attestation-committee-based-full-pos-chains/2259)
-
-The beacon chain idea emerged from the research around notarization of shard information by a committee of notaries, similar to a committee of validators in Casper. The key difference in sharding, however, is the pseudorandomness generation required for fast reshuffling of actors across shards within the system.
-
-Piggybacking off the VRF (verifiable random function) research alongside the ideas of BLS signatures put forth by DFINITY, there are many elements that can be taken to create a sidechain that would potentially merge sharding/casper :heart:.
-
-## The Beacon Chain
-
-### A Sidechain Instead of a Sharding Manager Contract
-
-When handling the sharded system via smart contract on the Ethereum mainchain, we were able to derive implicit finality via the transactions that submit a collation header into the contract that would then be mined onto a block in the mainchain. However, we were _bounded_ by gas and the current functioning of the EVM 1.0. That is, the number of shards realistically could only grow as much as the sharding manager contract could handle as a load of incoming transactions. 
-
-This system is also more affected by hard-forks and changes occurring in the main Ethereum network. Moreover, the incoming integration of a hybrid Casper PoS system would create a complicated co-existence of two types of validators: namely Casper and sharding validators. Signature verification, in particular, is an extremely expensive operation if done entirely through a smart contract on the mainchain, creating inherent bottlenecks in a hybrid system.
-
-Instead, we propose the creation of a _sidechain_ known as a **beacon chain** that has links to the mainchain by containing hashes of canonical mainchain blocks within its own block construction.
-
-### Desiderata
-
-There are a few important traits we will include in this sidechain construct that are particularly important for sharding.
-
--   Block references to the main chain
--   Full Proof of Stake via Casper FFG
--   RANDAO for random selection of committees
--   One-Way Deposits
--   Ability to store verifiable metadata of occurrences across shards (we refer to this information as a cross-link)
-
-In Vitalik Buterin's [words](https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ?both):
-
-> A cross-link is a special type of transaction that says “here is the hash of some recent block on shard X. Here are signatures from at least 2/3 of some randomly selected sample of M validators (eg. M = 1024) that attest to the validity of the cross-link”. Every shard (eg. there might be 4000 shards total) is itself a PoS chain, and the shard chains are where the transactions and accounts will be stored. The cross-links serve to “confirm” segments of the shard chains into the main chain, and are also the primary way through which the different shards will be able to talk to each other.
-
-The main purpose of beacon chain is to handle these shard cross-links as well as the set of validators that are locked into the system. This initial set is seeded by users burning 32ETH into a contract on the mainchain and specifying their public key, which can then be verified by the beacon chain which interacts with this contract.
-
-The exact specification of the beacon chain is a work-in-progress being written [here](https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ?both). Instead of paraphrasing its spec, we will elaborate on the research that led to this point and what it means moving forward.
-
-## Research History
-
-Recall that the **Sharding Manager Contract** was originally going to be used for management of the sharded system.
-
-> Whole point of this is to give a background and better understanding of the system...I think I won't follow the structure below in the final writing but will help as an outline.
-
-### Limitations of the Sharding Manager Contract
-
-Using a Sharding Manager Contract, albeit attractive from a development standpoint, poses the following challenges:
-
--   miners on the main chain can censor transactions
--   the number of shards is limited by gas costs of writes to the storage of this contract
--   any upgrades to this contract/sharding system would require a hard fork on the main chain
-
-#### Offchain Collation Headers
-
-[ETHResearch Link](https://ethresear.ch/t/offchain-collation-headers/1679)
-
-One idea that naturally arose was to store collation headers offchain, allowing for more shards to exist as there would be no processing bottleneck and faster finality guarantees. Reminiscent of DFINITY's chain, this process could be done through a construct known as a **random beacon side chain** that is pegged to the main chain via checkpoints on main chain blocks.
-
-Reminiscent of plasma chains without the exit mechanism according to Justin Drake, this construct would provide a solid ground for experimentation of shard designs without modifications to the main chain. The beacon chain would provide the pseudorandomness required for committee selection of the sharding system, through BLS distributed key generation, as well as better finality given that shards only care about finalized deposits from the main chain.
-
-Management and submission of collation headers can now be done within each respective shard, with the beacon chain only serving as a coordination device for summarizing what was voted on within each shard. That is, the beacon chain would be responsible for handling what we call a **cross-link**, which is a piece of metadata summarizing which collations were voted on as canonical within shards, who voted on these collations, and what are their blob merkle roots.
-
-#### Randomness in Committee Selections
-
-[ETHResearch Link](https://ethresear.ch/t/rng-exploitability-analysis-assuming-pure-randao-based-main-chain/1825/9)
-
-While DFINITY's beacon chain uses the BLS Signature scheme integrated into something called Threshold Relay for distributed randomness, this is not needed for the random beacon chain construct we are building for sharding. That is, there are other satisfactory ways to achieve randomness that make more sense for sharding.
-
-Ethereum's sharding beacon chain will use a system called **RANDAO** which is a Decentralized Autonomous Organization (a DAO) where randomness is generated by participants contributing a value to a "hash-onion" in the system 
-
-$$H(H(H(.....S.....)))$$
-
-where participants creating a block have to reveal the pre-image (the value before the hash) of their commitment value and update the current commitment value to this pre-image. In a recursive manner, the next participants will have to do the same and reveal their pre-images when creating a block. This system updates a random value $R$ during each iteration by taking the $XOR$ of it with the revealed pre-image. This value, $R$, is then used for randomization of committee selection. For sharding in particular, Justin mentioned the source of randomness can be this global $R$ value also $XOR$ with the proposer's pre-image commitment for that particular shard, restricting the visibility of $R\_{shard}$ to only proposers participating in that shard.
-
-#### Leaderless Random Beacon - Alternative to RANDAO
-
-[ETHResearch Link](https://ethresear.ch/t/leaderless-k-of-n-random-beacon/2046)
-
-The downside of RANDAO is the need for a leader in each step of the hash pre-image reveal step for determining the next value of $R$. In an alternative approach, we have a "committee of size $n$ generate random numbers if $k$ participants generate correctly" (Justin Drake). In this approach, we have every participant commit a temporary secret key, public key pair and form a polynomial via point interpolation. At a reveal phase of the protocol, k-of-n polynomial encrypted shares. At a reveal phase, $k$ participants reveal their private keys and it is then easy to check which ones did not reveal correctly. Even then, we can reconstruct an appropriate candidate polynomial using the revealed keys. Then, the randomness becomes random output to be the "sum of the secret keys for which the corresponding participants committed correctly" as explained by Justin.
-
-### Shard Metadata & Finality
-
-[ETHResearch Link](https://ethresear.ch/t/two-ways-to-do-cross-links/2074)
-
-The main idea behind sharding is to leverage this beacon chain construct for consensus on **cross-links** which are the heart of the sharding spec. In a way, cross-links are metadata that summarize the latest occurrences on shards. That is, they compress the results of PoS consensus from each shard into a simple message that uses proposer signatures to confirm this information. They are called **cross-links** because they will eventually be the way shards can communicate between each other, as they are linked to the finality of the main chain.
-
-#### Registrations and Committee Reshuffling
-
-[ETHResearch Link](https://ethresear.ch/t/registrations-shard-count-and-shuffling/2129)
-
-Leading up to the beacon chain spec, a few proposals for structuring committees were being explored, including this one by Justin Drake. We refer to participants in the Ethereum protocol 2.0 (Casper + Sharding) as validators which can be in one of three states: either pending_registration, registered, or pending_deregistration (Justin Drake).
-
-The main idea behind structuring committees of validators is that shards will be empty and uninitialized until the beacon chain reaches a certain minimum number of registered validators.
-
-In the post, it is mentioned that
-
-> Proposers and notaries are shuffled (via pseudo-random permutations) across shards in a staggered fashion and at a constant rate. Proposers are assigned to shards for 2^19 periods (~30 days) and the oldest proposer from each shard are shuffled every 2^(19 - 10) periods. Notaries are assigned to a shard for 2^7 periods (~10 minutes) and the oldest 2^(10 - 7) notaries from each shard are shuffled every period.
->
-> However, the current spec for the beacon chain mentions a fixed number for the SHARD_COUNT set to 1024.
-
-The entire reshuffling mechanism was revamped given this fixed shard_count number. In the current beacon chain spec, it is mentioned that
-
-> For shard crosslinks, the process is somewhat more complicated. First, we choose the set of shards active during each epoch. We want to target some fixed number of notaries per crosslink, but this means that since there is a fixed number of shards and a variable number of validators, we won’t be able to go through every shard in every epoch. Hence, we first need to select which shards we will be crosslinking in some given epoch
-
-Additionally, the current spec forces casper validators to also be sharding validators, which enforces greater security and takes advantage of the enshrined randomness + full PoS properties of the beacon chain.

docs/READINGS.md

@@ -1,149 +0,0 @@
-# Required Readings
-
-This document serves as an outline of the required readings to catch up with and contribute to the implementation for of the Ethereum 2.0 (Serenity) protocol. Whether you are an expert at Ethereum and want to contribute, or are a beginner and want to take the time to learn the concepts, this will serve as a comprehensive set of readings and links to help you up to speed.
-
-## Required Blockchain & Ethereum Knowledge
-
-**Blockchain Fundamentals**
-
--   [What is Blockchain Technology? A Step by Step Guide for Beginners](https://blockgeeks.com/guides/what-is-blockchain-technology/)
--   [What is Bitcoin? A Step by Step Guide for Beginners](https://blockgeeks.com/guides/what-is-bitcoin/)
--   [The Science Behind Cryptocurrencies' Cryptography](https://blockgeeks.com/guides/cryptocurrencies-cryptography/)
--   [The Ins and Outs of Cryptographic Hash Functions](https://blockgeeks.com/guides/cryptographic-hash-functions/)
--   [Blockchain Glossary from A-Z](https://blockgeeks.com/guides/blockchain-glossary-from-a-z/)
--   [Blockchain Addresses 101: What Are They?](https://blockgeeks.com/guides/blockchain-address-101/)
-
-**Ethereum**
-
--   [What is Ethereum?](https://www.ethereum.org/beginners/)
--   [How Does Ethereum Work Anyway?](https://medium.com/@preethikasireddy/how-does-ethereum-work-anyway-22d1df506369)
--   [Ethereum Introduction](https://github.com/ethereum/wiki/wiki/Ethereum-introduction)
--   [Ethereum Frequently Asked Questions](https://github.com/ethereum/wiki/wiki/FAQs)
--   [What is Hashing?](https://blockgeeks.com/guides/what-is-hashing/)
--   [Hashing Algorithms and Security](https://www.youtube.com/watch?v=b4b8ktEV4Bg)
--   [Understanding Merkle Trees](https://www.codeproject.com/Articles/1176140/Understanding-Merkle-Trees-Why-use-them-who-uses-t)
--   [Ethereum White Paper](https://github.com/ethereum/wiki/wiki/White-Paper)
--   [Ethereum Block Architecture](https://ethereum.stackexchange.com/questions/268/ethereum-block-architecture/6413#6413)
--   [Ethereum Beige Paper](https://github.com/chronaeon/beigepaper/blob/master/beigepaper.pdf)
--   [What is an Ethereum Token?](https://blockgeeks.com/guides/ethereum-token/)
--   [What is Ethereum Gas?](https://blockgeeks.com/guides/ethereum-gas-step-by-step-guide/)
-
-**Ethereum Client Implementations**
--   [Geth](https://github.com/ethereum/go-ethereum) (known also as go-ethereum) is the Golang implementation of the Ethereum protocol
--   [Parity](https://github.com/paritytech/parity) the fastest and most performant implementation - written in Rust
--   [Trinity](https://github.com/ethereum/py-evm/tree/master/trinity) new project implements Ethereum in Python
--   [Cpp-Ethereum](https://github.com/ethereum/cpp-ethereum) a C++ implementation of Ethereum
-
-**Consensus**
-
--   [Bitcoin Original White Paper](https://bitcoin.org/bitcoin.pdf)
--   [Basic Primer: Blockchain Consensus](https://blockgeeks.com/guides/blockchain-consensus/)
--   [Understanding Blockchain Fundamentals: Byzantine Fault Tolerance](https://medium.com/loom-network/understanding-blockchain-fundamentals-part-1-byzantine-fault-tolerance-245f46fe8419)
--   [Understanding Blockchain Fundamentals: Proof of Work vs. Proof of Stake](https://medium.com/loom-network/understanding-blockchain-fundamentals-part-2-proof-of-work-proof-of-stake-b6ae907c7edb)
--   [Proof of Work vs. Proof of Stake](https://blockgeeks.com/guides/proof-of-work-vs-proof-of-stake/)
--   [Proof of Stake FAQ](https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ)
--   [How Does Ethereum Mining Work?](https://www.coindesk.com/information/ethereum-mining-works/)
--   [ETHash Algorithm](https://github.com/ethereum/wiki/wiki/Ethash)
-
-**Ethereum Virtual Machine**
-
--   [What is the Ethereum Virtual Machine?](https://themerkle.com/what-is-the-ethereum-virtual-machine/)
--   [Ethereum VM](https://medium.com/@jeff.ethereum/go-ethereums-jit-evm-27ef88277520)
--   [Ethereum Protocol Subtleties](https://github.com/ethereum/wiki/wiki/Subtleties)
--   [Awesome Ethereum Virtual Machine](https://github.com/ethereum/wiki/wiki/Ethereum-Virtual-Machine-(EVM)-Awesome-List)
-
-**Ethereum-flavored WebAssembly**
-
--   [Ewasm background, motivation, goals, and design](https://github.com/ewasm/design)
--   [The current Ewasm spec](https://github.com/ewasm/design/blob/master/eth_interface.md)
--   [Latest Ewasm community call including live demo of the testnet](https://www.youtube.com/watch?v=apIHpBSdBio)
--   [Why eWASM? by Alex Beregszaszi](https://www.youtube.com/watch?v=VF7f_s2P3U0)
--   [Panel: entire eWASM team discussion and Q&A](https://youtu.be/ThvForkdPyc?t=119)
--   [Ewasm community meetup at ETHBuenosAires](https://www.youtube.com/watch?v=qDzrbj7dtyU)
-
-**Smart Contracts, dApps, and Cryptoeconomics**
-
--   [What are dApps? The New Decentralized Future](https://blockgeeks.com/guides/dapps/)
--   [How to Learn Solidity](https://blockgeeks.com/guides/solidity/)
--   [Ethereum Development Tutorial](https://github.com/ethereum/wiki/wiki/Ethereum-Development-Tutorial)
--   [What is Cryptocurrency Game Theory?](https://blockgeeks.com/guides/cryptocurrency-game-theory/)
--   [What is Cryptoeconomics?](https://blockgeeks.com/guides/what-is-cryptoeconomics/)
--   [Mechanism Design for Cryptoeconomic Applications](https://medium.com/blockchannel/a-crash-course-in-mechanism-design-for-cryptoeconomic-applications-a9f06ab6a976)
--   [Cryptoeconomics: An Introduction](https://cryptoeconomics.study/)
-
-**Peer-to-Peer Networking**
-
--   [Ethereum Peer to Peer Networking](https://github.com/ethereum/go-ethereum/wiki/Peer-to-Peer)
--   [How Does the P2P on Ethereum Work?](https://www.reddit.com/r/ethereum/comments/3918u0/how_does_the_p2p_network_on_ethereum_work/)
--   [How Does Kademlia Work?](http://gleamly.com/article/introduction-kademlia-dht-how-it-works)
--   [Kademlia Protocol](http://www.divms.uiowa.edu/~ghosh/kademlia.pdf)
-
-## Required Sharding Knowledge
-
-With respect to knowing enough about sharding, we will cover the requirements for both part-time and core contributors to Prysmatic Labs.
-
-### For Part Time Contributors
-
--   [Blockchain Scalability: Why?](https://blockgeeks.com/guides/blockchain-scalability/)
--   [What Are Ethereum Nodes and Sharding](https://blockgeeks.com/guides/what-are-ethereum-nodes-and-sharding/)
--   [How to Scale Ethereum: Sharding Explained](https://medium.com/prysmatic-labs/how-to-scale-ethereum-sharding-explained-ba2e283b7fce)
--   [Sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ)
--   [Sharding Introduction: R&D Compendium](https://github.com/ethereum/wiki/wiki/Sharding-introduction-R&D-compendium)
-
-### For Core Contributors
-
-**Fundamentals**
-
--   [Blockchain Scalability: Why?](https://blockgeeks.com/guides/blockchain-scalability/)
--   [What Are Ethereum Nodes and Sharding](https://blockgeeks.com/guides/what-are-ethereum-nodes-and-sharding/)
--   [How to Scale Ethereum: Sharding Explained](https://medium.com/prysmatic-labs/how-to-scale-ethereum-sharding-explained-ba2e283b7fce)
--   [Sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ)
--   [Sharding Introduction: R&D Compendium](https://github.com/ethereum/wiki/wiki/Sharding-introduction-R&D-compendium)
-
-After reading the Sharding FAQ, it is important to understand the minimal implementation notes and the research that went into writing it up.
-
-**Serenity Concepts and Notes**
-
--   [Sharding Concepts Mental Map](https://www.mindomo.com/zh/mindmap/sharding-d7cf8b6dee714d01a77388cb5d9d2a01)
--   [Taiwan Sharding Workshop Notes](https://hackmd.io/s/HJ_BbgCFz#%E2%9F%A0-General-Introduction)
--   [Sharding Research Compendium](http://notes.ethereum.org/s/BJc_eGVFM)
--   [Torus Shaped Sharding Network](https://ethresear.ch/t/torus-shaped-sharding-network/1720/8)
--   [General Theory of Sharding](https://ethresear.ch/t/a-general-theory-of-what-quadratically-sharded-validation-is/1730/10)
--   [Sharding Design Compendium](https://ethresear.ch/t/sharding-designs-compendium/1888/25)
-
-**Serenity Research Posts in Order**
-
--   [Sharding v2.1 Spec](https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ)
--   [Casper/Sharding/Beacon Chain FAQs](https://notes.ethereum.org/9MMuzWeFTTSg-3Tz_YeiBA?view)
--   [RETIRED! Sharding Phase 1 Spec](https://ethresear.ch/t/sharding-phase-1-spec-retired/1407/92)
--   [Exploring the Proposer/Collator Spec and Why it Was Retired](https://ethresear.ch/t/exploring-the-proposer-collator-split/1632/24)
--   [The Stateless Client Concept](https://ethresear.ch/t/the-stateless-client-concept/172/4)
--   [Shard Chain Blocks vs. Collators](https://ethresear.ch/t/shard-chain-blocks-vs-collators/429)
--   [Ethereum Concurrency Actors and Per Contract Sharding](https://ethresear.ch/t/ethereum-concurrency-actors-and-per-contract-sharding/375)
--   [Future Compatibility for Sharding](https://ethresear.ch/t/future-compatibility-for-sharding/386)
--   [Fork Choice Rule for Collation Proposal Mechanisms](https://ethresear.ch/t/fork-choice-rule-for-collation-proposal-mechanisms/922/8)
--   [State Execution](https://ethresear.ch/t/state-execution-scalability-and-cost-under-dos-attacks/1048)
--   [Fast Shard Chains With Notarization](https://ethresear.ch/t/as-fast-as-possible-shard-chains-with-notarization/1806/2)
--   [RANDAO Notary Committees](https://ethresear.ch/t/fork-free-randao/1835/3)
--   [Safe Notary Pool Size](https://ethresear.ch/t/safe-notary-pool-size/1728/3)
--   [Cross Links Between Main and Shard Chains](https://ethresear.ch/t/cross-links-between-main-chain-and-shards/1860/2)
-
-**Serenity Talk from Conferences & Meet Ups in Order**
--   [Sharding Presentation by Vitalik from IC3-ETH Bootcamp](https://vod.video.cornell.edu/media/Sharding+-+Vitalik+Buterin/1_1xezsfb4/97851101)
--   [Latest Research and Sharding by Justin Drake from Tech Crunch](https://www.youtube.com/watch?v=J6xO7DH20Js)
--   [Beacon Casper Chain by Vitalik and Justin Drake](https://www.youtube.com/watch?v=GAywmwGToUI)
--   [Proofs of Custody by Vitalik and Justin Drake](https://www.youtube.com/watch?v=jRcS9D_gw_o)
--   [So You Want To Be a Casper Validator by Vitalik](https://www.youtube.com/watch?v=rl63S6kCKbA)
--   [Ethereum Sharding from EDCon by Justin Drake](https://www.youtube.com/watch?v=J4rylD6w2S4)
--   [Casper CBC and Sharding by Vlad Zamfir](https://www.youtube.com/watch?v=qDa4xjQq1RE&t=1951s)
--   [Casper FFG in Depth by Carl](https://www.youtube.com/watch?v=uQ3IqLDf-oo)
--   [Ethereum & Scalability Technology from Asia Pacific ETH meet up by Hsiao Wei](https://www.youtube.com/watch?v=GhuWWShfqBI)
-
-## Necessary Go Knowledge & Readings
-
--   [The Go Programming Language (Only Recommended Book)](https://www.amazon.com/Programming-Language-Addison-Wesley-Professional-Computing/dp/0134190440)
--   [Ethereum Development with Go](https://goethereumbook.org)
--   [How to Write Go Code](http://golang.org/doc/code.html)
--   [The Go Programming Language Tour](http://tour.golang.org/)
--   [Getting Started With Go](http://www.youtube.com/watch?v=2KmHtgtEZ1s)
--   [Go Official Website](https://golang.org/)

docs/ROADMAP.md

@@ -1,148 +0,0 @@
-# Prysmatic Labs Roadmap Reference
-
-This document serves as a main reference for Prysmatic Labs' sharding and beacon chain implementation in Go, along with our roadmap and compilation of active research.
-
-# Table of Contents
-
--   [Sharding Introduction](#sharding-introduction)
-    -   [Basic Sharding Idea and Design](#basic-sharding-idea-and-design)
--   [Roadmap Phases](#roadmap-phases)
-    -   [The Ruby Release: Local Network](#the-ruby-release-local-network)
-    -   [The Sapphire Release: Goerli Testnet](#the-sapphire-release-goerli-testnet)
-    -   [The Diamond Release: Ethereum Mainnet](#the-diamond-release-ethereum-mainnet)
-    -   [System Architecture](#system-architecture)
--   [Acknowledgements](#acknowledgements)
--   [References](#references)
-
-# Sharding Introduction
-
-Currently, every single node running the Ethereum network has to process every single transaction that goes through the network. This gives the blockchain a high amount of security because of how much validation goes into each block, but at the same time it means that an entire blockchain is only as fast as its individual nodes and not the sum of its parts. Currently, transactions on the EVM are non-parallelizable, and every transaction is executed in sequence globally. The scalability problem then has to do with the idea that a blockchain can have at most 2 of these 3 properties: decentralization, security, and scalability.
-
-If we have scalability and security, it would mean that our blockchain is centralized and that would allow it to have a faster throughput. Right now, Ethereum is decentralized and secure, but not scalable.
-
-An approach to solving the scalability trilemma is the idea of blockchain sharding, where we split the entire state of the network into partitions called shards that contain their own independent piece of state and transaction history. In this system, certain nodes would process transactions only for certain shards, allowing the throughput of transactions processed in total across all shards to be much higher than having a single shard do all the work as the main chain does now.
-
-## Basic Sharding Idea and Design
-
-A sharded blockchain system is made possible by having nodes store “signed metadata” in the main chain of latest changes within each shard chain. Through this, we manage to create a layer of abstraction that tells us enough information about the global, synced state of parallel shard chains. These messages are called **cross-links**, which are specific structures that encompass important information about blocks in shards across the network. We coordinate these crosslinks on a chain known as a **beacon-chain**. In this chain, blocks are created by actors known as **proposers** that are tasked with information about cross-links together. These blocks are then voted on by a party of actors known as **attesters**. These attesters are randomly selected for particular periods of time in certain shards and are then tasked into reaching consensus on these chains and on beacon chain blocks via a **proof of stake** system. Attesters and proposers are the names of two different responsibilities a participant in Ethereum 2.0 consensus can have. These participants are known in general as **validators**.
-
-Cross-links are stored in blocks on a full proof of stake chain known as a **beacon chain**, which will be implemented as a sidechain to the Ethereum main chain initially.
-
-We still keep the Ethereum main chain and deploy a smart contract into it known as the **Validator Deposit Contract**, where users can deposit and burn 32 ETH. Beacon chain nodes would listen to deposits in this contract and consequently queue up a user with the associated address as a validator in the beacon chain PoS system. Validators then become part of a registered validator set in the beacon chain, and are committees of validators are selected to become attesters or proposers for a certain period of time on shards until they are ventually reshuffled into different shards.
-
-Given that we are splitting up the global state of the Ethereum blockchain into shards, new types of attacks arise because fewer resources are required to completely dominate a shard. This is why a **source of randomness** and periods are critical components to ensuring the integrity of the system.
-
-The Ethereum Wiki’s [Sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ) suggests pseudorandom sampling of validators on each shard. The goal is so that these attesters will not know which shard they will get in advance. Otherwise, malicious actors could concentrate resources into a single shard and try to overtake it (See: [1% Attack](https://medium.com/@icebearhww/ethereum-sharding-and-finality-65248951f649)).
-
-Sharding revolves around being able to store shard metadata in a full proof of stake chain known as a beacon chain. For pseudorandomness generation, a mechanism known as RANDAO can be used in the beacon chain to shuffle validators securely.
-
-# Roadmap Phases
-
-Prysmatic Labs will implement the official beacon chain specification and the full roadmap for Ethereum 2.0 as written in its official repository [here](https://github.com/ethereum/eth2.0-specs) by the community of core researchers and developers in Ethereum.
-
-To roll out these phases, we will be releasing our implementation in a series of steps:
-
-## The Ruby Release: Local Network
-
-Our current work is focused on creating a localized version of a beacon chain that would include the following:
-
--  A minimal, **beacon chain node** that will interact with an Ethereum 1.0 node via JSON-RPC
--  A **Validator Deposit Contract** deployed on an Ethereum 1.0 chain where a beacon node can read logs to check for registered validators
--  A minimal, floodsub p2p network
--  Ability for proposers/attesters to be selected by the beacon chain's randomness into committees that work on specific shards
--  Ability to execute state transitions, apply a fork choice rule, and advance a full proof of stake blockchain through distributed consensus
-
-We released our Ruby release on October 2018
-
-## The Sapphire Release: Goerli Testnet
-
-Part 1 of the **Sapphire Release** will focus around getting the **Ruby Release** polished enough to be live on an Ethereum testnet and manage a a beacon chain system. 
-
-Part 2 of the **Sapphire Release** will focus on implementing state execution and defining the State Transition Function for sharding on a local testnet (as outlined in [Beyond Phase 1](#beyond-phase-1)) as an extenstion to the Ruby Release.
-
-ETA: End of Q1 2019
-
-## The Diamond Release: Ethereum Mainnet
-
-The **Diamond Release** will reconcile the best parts of the previous releases and deploy a full-featured, cross-shard transaction system through a beacon chain, casper FFG-enabled, sharding release. As expected, this is the most difficult and time consuming release on the horizon for Prysmatic Labs. We plan on growing our community effort significantly over the first few releases to get all hands-on deck preparing for this.
-
-The Diamond Release should be considered the production release candidate for sharding Ethereum on the mainnet.
-
-ETA: To Be determined
-
-# Beacon Chain Phase 0 Implementation
-
-Prysmatic Labs will begin by focusing its implementation entirely on Phase 0 from the Ethereum 2.0 roadmap. We plan on being as pragmatic as possible to create something that can be locally run by any developer as soon as possible. Our initial deliverable will center around a command line tool that will serve as an entry point into a beacon chain node that allows for users to become validators and earn rewrads for participating in Proof of Stake consensus.
-
-Here is a reference spec explaining how our initial system will function:
-
-## System Architecture
-
-Our implementation revolves around the following core components:
-
--   A **beacon chain** that connects to this main chain node via JSON-RPC
--   A **validator client** that connects to a beacon node and allows for users to earn rewards for staking 32ETH to secure the protocol
-
-A basic, end-to-end example of the system is as follows:
-
-1.  _**User deposits 32 ETH into a Validator Deposit Contract on the Eth 1.0 chain:**_ the beacon chain listens for the logs in the chain to queue that validator into the beacon chain chain's main event loop
-
-2.  _**Registered validator begins PoS process to propose blocks:**_ the PoS validator has the resposibility to participate in the addition of new blocks to the beacon chain
-
-3.  _**RANDAO mechanism selects committees of proposers/attesters for shards:**_ the beacon chain node will use its RANDAO mechanism to select committees of proposers, attesters that each have responsibilities within the system. 
-
-4. _**Beacon Chain State Advances, Committees are Reshuffled:**_ upon completing responsibilities, the different actors of the system are them reshuffled into new committees on different shards.
-
-## The EVM: What You Need to Know
-
-As an important aside, we’ll take a brief detour into the EVM and what we need to understand before we modify it for a sharded blockchain. At its core, the functionality of the EVM optimizes for _security_ and not for computational power with the following restrictions:
-
--   Every single step must be paid for upfront with gas costs to prevent DDoS
--   Programs can't interact with each other without a single byte array
-    -   This also means programs can't access other programs' state
--   Sandboxed Execution - the EVM can only modify its internal state and nothing else
--   Deterministic execution guarantees
-
-So what exactly is the EVM? The EVM was purposely designed to be a stack based machine with memory-byte arrays and key-value stores that are kept on a trie
-
--   Every single keys and storage values are 32 bytes
--   There are 100 total opcodes in the EVM
--   The EVM comes with a temporary memory byte-array and storage trie to hold persistent memory.
-
-Cryptographic operations are done using pre-compiled contracts. Aside from that, the EVM provides a bunch of blockchain access-level context that allows certain opcodes to fetch useful information from the external system. For example, LOG opcodes store useful information in the log bloom filter that can be synced with light clients. This can be used as a low-gas form of storage, since LOG does not modify the state.
-
-Additionally, the EVM contains a call-depth limit such that recursive invocations or chains of calls will eventually halt, preventing a drastic use of resources.
-
-It is important to note that the merkle root of an Ethereum account is updated any time an `SSTORE` opcode is executed successfully by a program on the EVM that results in a key or value changing in the state merklix (merkle radix) tree.
-
-How is this relevant to sharding? It is important to note the importance of certain opcodes in our implementation and how we will need to introduce and modify several of them for both security and scalability considerations in a sharded chain.
-
-# Acknowledgements
-
-A special thanks for entire [Prysmatic Labs](https://discord.gg/che9auJ) team for helping put this together and to Ethereum Research (Hsiao-Wei Wang, Vitalik, Justin Drake) for the help and guidance in our approach.
-
-# References
-
-[Sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ)
-
-[Sharding Reference Spec](https://github.com/ethereum/sharding/blob/develop/docs/doc.md)
-
-[Data Availability and Erasure Coding](https://github.com/ethereum/research/wiki/A-note-on-data-availability-and-erasure-coding)
-
-[Proof of Visibility for Data Availability](https://ethresear.ch/t/proof-of-visibility-for-data-availability/1073)
-
-[History, State, and Asynchronous Accumulators in the Stateless Model](https://ethresear.ch/t/history-state-and-asynchronous-accumulators-in-the-stateless-model/287)
-
-[Torus Shaped Sharding Network](https://ethresear.ch/t/torus-shaped-sharding-network/1720)
-
-[Data Availability Proof-friendly State Tree Transitions](https://ethresear.ch/t/data-availability-proof-friendly-state-tree-transitions/1453)
-
-[Safety Attester Pool Size](https://ethresear.ch/t/safe-attester-pool-size/1728)
-
-[Fixed Size Deposits and Rewards Penalties Quadleak](https://ethresear.ch/t/fixed-size-deposits-and-rewards-penalties-quad-leak/2073/7)
-
-[Two Ways To Do Cross Links](https://ethresear.ch/t/two-ways-to-do-cross-links/2074/2)
-
-[Extending Minimal Sharding with Cross Links](https://ethresear.ch/t/two-ways-to-do-cross-links/2074/2)
-
-[Leaderless K of N Random Beacon](https://ethresear.ch/t/leaderless-k-of-n-random-beacon/2046/3)

endtoend/components/beacon_node.go

@@ -48,7 +48,7 @@ func StartNewBeaconNode(t *testing.T, config *types.E2EConfig, index int, enr st
 		fmt.Sprintf("--min-sync-peers=%d", e2e.TestParams.BeaconNodeCount-1),
 		fmt.Sprintf("--p2p-udp-port=%d", e2e.TestParams.BeaconNodeRPCPort+index+10),
 		fmt.Sprintf("--p2p-tcp-port=%d", e2e.TestParams.BeaconNodeRPCPort+index+20),
-		fmt.Sprintf("--monitoring-port=%d", e2e.TestParams.BeaconNodeRPCPort+index+30),
+		fmt.Sprintf("--monitoring-port=%d", e2e.TestParams.BeaconNodeMetricsPort+index),
 		fmt.Sprintf("--grpc-gateway-port=%d", e2e.TestParams.BeaconNodeRPCPort+index+40),
 		fmt.Sprintf("--contract-deployment-block=%d", 0),
 		fmt.Sprintf("--rpc-max-page-size=%d", params.BeaconConfig().MinGenesisActiveValidatorCount),

endtoend/evaluators/BUILD.bazel

@@ -13,6 +13,7 @@ go_library(
     visibility = ["//endtoend:__subpackages__"],
     deps = [
         "//beacon-chain/core/helpers:go_default_library",
+        "//endtoend/params:go_default_library",
         "//endtoend/types:go_default_library",
         "//shared/bytesutil:go_default_library",
         "//shared/params:go_default_library",

endtoend/evaluators/node.go

@@ -4,10 +4,13 @@ import (
 	"bytes"
 	"context"
 	"fmt"
+	"io/ioutil"
+	"net/http"
 
 	ptypes "github.com/gogo/protobuf/types"
 	"github.com/pkg/errors"
 	eth "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
+	e2e "github.com/prysmaticlabs/prysm/endtoend/params"
 	"github.com/prysmaticlabs/prysm/endtoend/types"
 	"google.golang.org/grpc"
 )
@@ -19,6 +22,13 @@ var PeersConnect = types.Evaluator{
 	Evaluation: peersConnect,
 }
 
+// HealthzCheck pings healthz and errors if it doesn't have the expected OK status.
+var HealthzCheck = types.Evaluator{
+	Name:       "healthz_check_epoch_%d",
+	Policy:     afterNthEpoch(0),
+	Evaluation: healthzCheck,
+}
+
 // FinishedSyncing returns whether the beacon node with the given rpc port has finished syncing.
 var FinishedSyncing = types.Evaluator{
 	Name:       "finished_syncing",
@@ -40,6 +50,42 @@ func onEpoch(epoch uint64) func(uint64) bool {
 	}
 }
 
+func healthzCheck(conns ...*grpc.ClientConn) error {
+	count := len(conns)
+	for i := 0; i < count; i++ {
+		resp, err := http.Get(fmt.Sprintf("http://localhost:%d/healthz", e2e.TestParams.BeaconNodeMetricsPort+i))
+		if err != nil {
+			return errors.Wrapf(err, "could not connect to beacon node %d", i)
+		}
+		if resp.StatusCode != http.StatusOK {
+			body, err := ioutil.ReadAll(resp.Body)
+			if err != nil {
+				return err
+			}
+			return fmt.Errorf("expected status code OK for beacon node %d, received %v with body %s", i, resp.StatusCode, body)
+		}
+		if err := resp.Body.Close(); err != nil {
+			return err
+		}
+
+		resp, err = http.Get(fmt.Sprintf("http://localhost:%d/healthz", e2e.TestParams.ValidatorMetricsPort+i))
+		if err != nil {
+			return errors.Wrapf(err, "could not connect to validator client %d", i)
+		}
+		if resp.StatusCode != http.StatusOK {
+			body, err := ioutil.ReadAll(resp.Body)
+			if err != nil {
+				return err
+			}
+			return fmt.Errorf("expected status code OK for validator client %d, received %v with body %s", i, resp.StatusCode, body)
+		}
+		if err := resp.Body.Close(); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
 func peersConnect(conns ...*grpc.ClientConn) error {
 	if len(conns) == 1 {
 		return nil

endtoend/minimal_antiflake_e2e_1_test.go

@@ -22,6 +22,7 @@ func TestEndToEnd_AntiFlake_MinimalConfig_1(t *testing.T) {
 		TestSlasher:    false,
 		Evaluators: []types.Evaluator{
 			ev.PeersConnect,
+			ev.HealthzCheck,
 			ev.ValidatorsAreActive,
 			ev.ValidatorsParticipating,
 		},

endtoend/minimal_antiflake_e2e_2_test.go

@@ -22,6 +22,7 @@ func TestEndToEnd_AntiFlake_MinimalConfig_2(t *testing.T) {
 		TestSlasher:    false,
 		Evaluators: []types.Evaluator{
 			ev.PeersConnect,
+			ev.HealthzCheck,
 			ev.ValidatorsAreActive,
 			ev.ValidatorsParticipating,
 		},

endtoend/minimal_e2e_test.go

@@ -33,6 +33,7 @@ func TestEndToEnd_MinimalConfig(t *testing.T) {
 		TestSlasher:    true,
 		Evaluators: []types.Evaluator{
 			ev.PeersConnect,
+			ev.HealthzCheck,
 			ev.ValidatorsAreActive,
 			ev.ValidatorsParticipating,
 			ev.FinalizationOccurs,

endtoend/minimal_slashing_e2e_test.go

@@ -22,6 +22,7 @@ func TestEndToEnd_Slashing_MinimalConfig(t *testing.T) {
 		TestSlasher:    true,
 		Evaluators: []types.Evaluator{
 			ev.PeersConnect,
+			ev.HealthzCheck,
 			ev.ValidatorsSlashed,
 			ev.SlashedValidatorsLoseBalance,
 			ev.InjectDoubleVote,

prysm.sh

@@ -71,7 +71,20 @@ readonly wrapper_dir="$(dirname "$(get_realpath "${BASH_SOURCE[0]}")")/dist"
 arch=$(uname -m)
 arch=${arch/x86_64/amd64}
 arch=${arch/aarch64/arm64}
-readonly os_arch_suffix="$(uname -s | tr '[:upper:]' '[:lower:]')-$arch"
+
+system=""
+case "$OSTYPE" in
+  darwin*)  system="darwin" ;; 
+  linux*)   system="linux" ;;
+  msys*)    system="windows" ;;
+  cygwin*)  system="windows" ;;
+  *)        exit 1 ;;
+esac
+
+if [ "$system" == "windows" ]
+then 
+	arch="amd64.exe"
+fi
 
 mkdir -p $wrapper_dir
 
@@ -91,13 +104,12 @@ get_prysm_version
 
 color "37" "Latest Prysm version is $prysm_version."
 
-BEACON_CHAIN_REAL="${wrapper_dir}/beacon-chain-${prysm_version}-${os_arch_suffix}"
-VALIDATOR_REAL="${wrapper_dir}/validator-${prysm_version}-${os_arch_suffix}"
+BEACON_CHAIN_REAL="${wrapper_dir}/beacon-chain-${prysm_version}-${system}-${arch}"
+VALIDATOR_REAL="${wrapper_dir}/validator-${prysm_version}-${system}-${arch}"
 
 if [[ ! -x $BEACON_CHAIN_REAL ]]; then 
     color "34" "Downloading beacon chain@${prysm_version} to ${BEACON_CHAIN_REAL} (${reason})"
-
-    curl -L "https://github.com/prysmaticlabs/prysm/releases/download/${prysm_version}/beacon-chain-${prysm_version}-${os_arch_suffix}" -o $BEACON_CHAIN_REAL
+    curl -L "https://github.com/prysmaticlabs/prysm/releases/download/${prysm_version}/beacon-chain-${prysm_version}-${system}-${arch}" -o $BEACON_CHAIN_REAL
     chmod +x $BEACON_CHAIN_REAL
 else
     color "37" "Beacon chain is up to date."
@@ -106,7 +118,7 @@ fi
 if [[ ! -x $VALIDATOR_REAL ]]; then 
     color "34" "Downloading validator@${prysm_version} to ${VALIDATOR_REAL} (${reason})"
 
-    curl -L "https://github.com/prysmaticlabs/prysm/releases/download/${prysm_version}/validator-${prysm_version}-${os_arch_suffix}" -o $VALIDATOR_REAL
+    curl -L "https://github.com/prysmaticlabs/prysm/releases/download/${prysm_version}/validator-${prysm_version}-${system}-${arch}" -o $VALIDATOR_REAL
     chmod +x $VALIDATOR_REAL
 else
     color "37" "Validator is up to date."
@@ -128,4 +140,4 @@ case $1 in
 esac
 
 color "36" "Starting Prysm $1 ${@:2}"
-exec -a "$0" "${process}" "${@:2}"
+exec -a "$0" "${process}" "${@:2}"

shared/cmd/flags.go

@@ -67,7 +67,7 @@ var (
 	BootstrapNode = &cli.StringFlag{
 		Name:  "bootstrap-node",
 		Usage: "The address of bootstrap node. Beacon node will connect for peer discovery via DHT.  Multiple nodes can be separated with a comma",
-		Value: "/dns4/prylabs.net/tcp/30001/p2p/16Uiu2HAm7Qwe19vz9WzD2Mxn7fXd1vgHHp4iccuyq7TxwRXoAGfc",
+		Value: "/dns4/prylabs.net/tcp/30001/p2p/16Uiu2HAm7Qwe19vz9WzD2Mxn7fXd1vgHHp4iccuyq7TxwRXoAGfc,enr:-Ku4QAGwOT9StqmwI5LHaIymIO4ooFKfNkEjWa0f1P8OsElgBh2Ijb-GrD_-b9W4kcPFcwmHQEy5RncqXNqdpVo1heoBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpAAAAAAAAAAAP__________gmlkgnY0gmlwhBLf22SJc2VjcDI1NmsxoQJxCnE6v_x2ekgY_uoE1rtwzvGy40mq9eD66XfHPBWgIIN1ZHCCD6A",
 	}
 	// RelayNode tells the beacon node which relay node to connect to.
 	RelayNode = &cli.StringFlag{
@@ -135,7 +135,13 @@ var (
 	P2PEncoding = &cli.StringFlag{
 		Name:  "p2p-encoding",
 		Usage: "The encoding format of messages sent over the wire. The default is 0, which represents ssz",
-		Value: "ssz",
+		Value: "ssz-snappy",
+	}
+	// P2PPubsub defines the pubsub router to use for p2p messages.
+	P2PPubsub = &cli.StringFlag{
+		Name:  "p2p-pubsub",
+		Usage: "The name of the pubsub router to use. Supported values are: gossip, flood, random",
+		Value: "gossip",
 	}
 	// ForceClearDB removes any previously stored data at the data directory.
 	ForceClearDB = &cli.BoolFlag{

shared/featureconfig/flags.go

@@ -75,11 +75,13 @@ var (
 		Name: "disable-protect-proposer",
 		Usage: "Disables functionality to prevent the validator client from signing and " +
 			"broadcasting 2 different block proposals in the same epoch. Protects from slashing.",
+		Value: true,
 	}
 	disableProtectAttesterFlag = &cli.BoolFlag{
 		Name: "disable-protect-attester",
 		Usage: "Disables functionality to prevent the validator client from signing and " +
 			"broadcasting 2 any slashable attestations.",
+		Value: true,
 	}
 	disableStrictAttestationPubsubVerificationFlag = &cli.BoolFlag{
 		Name:  "disable-strict-attestation-pubsub-verification",

shared/p2putils/BUILD.bazel

@@ -0,0 +1,13 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["fork.go"],
+    importpath = "github.com/prysmaticlabs/prysm/shared/p2putils",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//beacon-chain/core/helpers:go_default_library",
+        "//shared/params:go_default_library",
+        "@com_github_pkg_errors//:go_default_library",
+    ],
+)

shared/p2putils/fork.go

@@ -0,0 +1,44 @@
+package p2putils
+
+import (
+	"time"
+
+	"github.com/pkg/errors"
+	"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
+	"github.com/prysmaticlabs/prysm/shared/params"
+)
+
+// CreateForkDigest creates a fork digest from a genesis time and genesis
+// validators root, utilizing the current slot to determine
+// the active fork version in the node.
+func CreateForkDigest(
+	genesisTime time.Time,
+	genesisValidatorsRoot []byte,
+) ([4]byte, error) {
+	if genesisTime.IsZero() {
+		return [4]byte{}, errors.New("genesis time is not set")
+	}
+	if len(genesisValidatorsRoot) == 0 {
+		return [4]byte{}, errors.New("genesis validators root is not set")
+	}
+	currentSlot := helpers.SlotsSince(genesisTime)
+	currentEpoch := helpers.SlotToEpoch(currentSlot)
+
+	// We retrieve a list of scheduled forks by epoch.
+	// We loop through the keys in this map to determine the current
+	// fork version based on the current, time-based epoch number
+	// since the genesis time.
+	currentForkVersion := params.BeaconConfig().GenesisForkVersion
+	scheduledForks := params.BeaconConfig().ForkVersionSchedule
+	for epoch, forkVersion := range scheduledForks {
+		if epoch <= currentEpoch {
+			currentForkVersion = forkVersion
+		}
+	}
+
+	digest, err := helpers.ComputeForkDigest(currentForkVersion, genesisValidatorsRoot)
+	if err != nil {
+		return [4]byte{}, err
+	}
+	return digest, nil
+}

shared/params/BUILD.bazel

@@ -2,7 +2,10 @@ load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
 
 go_library(
     name = "go_default_library",
-    srcs = ["config.go"],
+    srcs = [
+        "config.go",
+        "network_config.go",
+    ],
     importpath = "github.com/prysmaticlabs/prysm/shared/params",
     visibility = ["//visibility:public"],
     deps = ["//shared/bytesutil:go_default_library"],

shared/params/config.go

@@ -53,7 +53,6 @@ type BeaconChainConfig struct {
 	MinEpochsToInactivityPenalty     uint64 `yaml:"MIN_EPOCHS_TO_INACTIVITY_PENALTY"`    // MinEpochsToInactivityPenalty defines the minimum amount of epochs since finality to begin penalizing inactivity.
 	Eth1FollowDistance               uint64 // Eth1FollowDistance is the number of eth1.0 blocks to wait before considering a new deposit for voting. This only applies after the chain as been started.
 	SafeSlotsToUpdateJustified       uint64 // SafeSlotsToUpdateJustified is the minimal slots needed to update justified check point.
-	AttestationPropagationSlotRange  uint64 // AttestationPropagationSlotRange is the maximum number of slots during which an attestation can be propagated.
 	SecondsPerETH1Block              uint64 `yaml:"SECONDS_PER_ETH1_BLOCK"` // SecondsPerETH1Block is the approximate time for a single eth1 block to be produced.
 	// State list lengths
 	EpochsPerHistoricalVector uint64 `yaml:"EPOCHS_PER_HISTORICAL_VECTOR"` // EpochsPerHistoricalVector defines max length in epoch to store old historical stats in beacon state.
@@ -155,7 +154,6 @@ var defaultBeaconConfig = &BeaconChainConfig{
 	MinEpochsToInactivityPenalty:     4,
 	Eth1FollowDistance:               1024,
 	SafeSlotsToUpdateJustified:       8,
-	AttestationPropagationSlotRange:  32,
 	SecondsPerETH1Block:              14,
 
 	// State list length constants.

shared/params/network_config.go

@@ -0,0 +1,36 @@
+package params
+
+import "time"
+
+// NetworkConfig defines the spec based network parameters.
+type NetworkConfig struct {
+	GossipMaxSize                   uint64        `yaml:"GOSSIP_MAX_SIZE"`                    // GossipMaxSize is the maximum allowed size of uncompressed gossip messages.
+	MaxChunkSize                    uint64        `yaml:"MAX_CHUNK_SIZE"`                     // MaxChunkSize is the the maximum allowed size of uncompressed req/resp chunked responses.
+	AttestationSubnetCount          uint64        `yaml:"ATTESTATION_SUBNET_COUNT"`           // AttestationSubnetCount is the number of attestation subnets used in the gossipsub protocol.
+	AttestationPropagationSlotRange uint64        `yaml:"ATTESTATION_PROPAGATION_SLOT_RANGE"` // AttestationPropagationSlotRange is the maximum number of slots during which an attestation can be propagated.
+	TtfbTimeout                     time.Duration `yaml:"TTFB_TIMEOUT"`                       // TtfbTimeout is the maximum time to wait for first byte of request response (time-to-first-byte).
+	RespTimeout                     time.Duration `yaml:"RESP_TIMEOUT"`                       // RespTimeout is the maximum time for complete response transfer.
+	MaximumGossipClockDisparity     time.Duration `yaml:"MAXIMUM_GOSSIP_CLOCK_DISPARITY"`     // MaximumGossipClockDisparity is the maximum milliseconds of clock disparity assumed between honest nodes.
+
+	// DiscoveryV5 Config
+	ETH2Key      string // ETH2Key is the ENR key of the eth2 object in an enr.
+	AttSubnetKey string // AttSubnetKey is the ENR key of the subnet bitfield in the enr.
+}
+
+var defaultNetworkConfig = &NetworkConfig{
+	GossipMaxSize:                   1 << 20, // 1 MiB
+	MaxChunkSize:                    1 << 20, // 1 MiB
+	AttestationSubnetCount:          64,
+	AttestationPropagationSlotRange: 32,
+	TtfbTimeout:                     5 * time.Second,
+	RespTimeout:                     10 * time.Second,
+	MaximumGossipClockDisparity:     500 * time.Millisecond,
+	ETH2Key:                         "eth2",
+	AttSubnetKey:                    "attnets",
+}
+
+// BeaconNetworkConfig returns the current network config for
+// the beacon chain.
+func BeaconNetworkConfig() *NetworkConfig {
+	return defaultNetworkConfig
+}

third_party/herumi/bls_eth_go_binary.BUILD

@@ -146,5 +146,6 @@ go_library(
     visibility = [
         # Additional access will require security approval.
         "@prysm//shared/bls:__pkg__",
+        "@com_github_wealdtech_go_eth2_types_v2//:__pkg__",
     ],
 )

tools/bootnode/BUILD.bazel

@@ -10,6 +10,7 @@ go_library(
     visibility = ["//visibility:private"],
     deps = [
         "//proto/beacon/p2p/v1:go_default_library",
+        "//shared/iputils:go_default_library",
         "//shared/logutil:go_default_library",
         "//shared/params:go_default_library",
         "//shared/version:go_default_library",
@@ -47,8 +48,9 @@ go_image(
     visibility = ["//visibility:private"],
     deps = [
         "//proto/beacon/p2p/v1:go_default_library",
-        "//shared/params:go_default_library",
+        "//shared/iputils:go_default_library",
         "//shared/logutil:go_default_library",
+        "//shared/params:go_default_library",
         "//shared/version:go_default_library",
         "@com_github_btcsuite_btcd//btcec:go_default_library",
         "@com_github_ethereum_go_ethereum//log:go_default_library",
@@ -60,7 +62,6 @@ go_image(
         "@com_github_ipfs_go_log//:go_default_library",
         "@com_github_libp2p_go_libp2p//:go_default_library",
         "@com_github_libp2p_go_libp2p_core//crypto:go_default_library",
-        "@com_github_libp2p_go_libp2p_core//protocol:go_default_library",
         "@com_github_libp2p_go_libp2p_kad_dht//:go_default_library",
         "@com_github_libp2p_go_libp2p_kad_dht//opts:go_default_library",
         "@com_github_multiformats_go_multiaddr//:go_default_library",

tools/bootnode/bootnode.go

@@ -38,6 +38,7 @@ import (
 	"github.com/prysmaticlabs/go-bitfield"
 	"github.com/prysmaticlabs/go-ssz"
 	pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
+	"github.com/prysmaticlabs/prysm/shared/iputils"
 	"github.com/prysmaticlabs/prysm/shared/logutil"
 	"github.com/prysmaticlabs/prysm/shared/params"
 	"github.com/prysmaticlabs/prysm/shared/version"
@@ -52,12 +53,13 @@ var (
 	discv5port   = flag.Int("discv5-port", 4000, "Port to listen for discv5 connections")
 	kademliaPort = flag.Int("kad-port", 4500, "Port to listen for connections to kad DHT")
 	metricsPort  = flag.Int("metrics-port", 5000, "Port to listen for connections")
-	externalIP   = flag.String("external-ip", "127.0.0.1", "External IP for the bootnode")
-
-	log = logrus.WithField("prefix", "bootnode")
+	externalIP   = flag.String("external-ip", "", "External IP for the bootnode")
+	disableKad   = flag.Bool("disable-kad", false, "Disables the bootnode from running kademlia dht")
+	log          = logrus.WithField("prefix", "bootnode")
 )
 
 const dhtProtocol = "/prysm/0.0.0/dht"
+const defaultIP = "127.0.0.1"
 
 type handler struct {
 	listener *discover.UDPv5
@@ -88,12 +90,18 @@ func main() {
 	cfg := discover.Config{
 		PrivateKey: privKey,
 	}
-	listener := createListener(*externalIP, *discv5port, cfg)
+	ipAddr, err := iputils.ExternalIPv4()
+	if err != nil {
+		log.Fatal(err)
+	}
+	listener := createListener(ipAddr, *discv5port, cfg)
 
 	node := listener.Self()
 	log.Infof("Running bootnode: %s", node.String())
 
-	startKademliaDHT(interfacePrivKey)
+	if !*disableKad {
+		startKademliaDHT(interfacePrivKey)
+	}
 
 	handler := &handler{
 		listener: listener,
@@ -113,8 +121,12 @@ func startKademliaDHT(privKey crypto.PrivKey) {
 	if *debug {
 		logging.SetDebugLogging()
 	}
+	ipAddr := defaultIP
+	if *externalIP != "" {
+		ipAddr = *externalIP
+	}
 
-	listen, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", *externalIP, *kademliaPort))
+	listen, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/%s/tcp/%d", ipAddr, *kademliaPort))
 	if err != nil {
 		log.Fatalf("Failed to construct new multiaddress. %v", err)
 	}
@@ -144,7 +156,7 @@ func startKademliaDHT(privKey crypto.PrivKey) {
 		log.Fatalf("Failed to bootstrap DHT. %v", err)
 	}
 
-	fmt.Printf("Running Kademlia DHT bootnode: /ip4/%s/tcp/%d/p2p/%s\n", *externalIP, *kademliaPort, host.ID().Pretty())
+	fmt.Printf("Running Kademlia DHT bootnode: /ip4/%s/tcp/%d/p2p/%s\n", ipAddr, *kademliaPort, host.ID().Pretty())
 }
 
 func createListener(ipAddr string, port int, cfg discover.Config) *discover.UDPv5 {
@@ -196,6 +208,10 @@ func createLocalNode(privKey *ecdsa.PrivateKey, ipAddr net.IP, port int) (*enode
 	if err != nil {
 		return nil, errors.Wrap(err, "Could not open node's peer database")
 	}
+	external := net.ParseIP(*externalIP)
+	if *externalIP == "" {
+		external = ipAddr
+	}
 
 	forkID := &pb.ENRForkID{
 		CurrentForkDigest: []byte{0, 0, 0, 0},
@@ -208,14 +224,10 @@ func createLocalNode(privKey *ecdsa.PrivateKey, ipAddr net.IP, port int) (*enode
 	}
 
 	localNode := enode.NewLocalNode(db, privKey)
-	ipEntry := enr.IP(ipAddr)
-	udpEntry := enr.UDP(port)
-	localNode.SetFallbackIP(ipAddr)
-	localNode.SetFallbackUDP(port)
-	localNode.Set(ipEntry)
-	localNode.Set(udpEntry)
 	localNode.Set(enr.WithEntry("eth2", forkEntry))
 	localNode.Set(enr.WithEntry("attnets", bitfield.NewBitvector64()))
+	localNode.SetFallbackIP(external)
+	localNode.SetFallbackUDP(port)
 
 	return localNode, nil
 }

validator/keymanager/BUILD.bazel

@@ -26,7 +26,7 @@ go_library(
         "@com_github_wealdtech_eth2_signer_api//pb/v1:go_default_library",
         "@com_github_wealdtech_go_eth2_wallet//:go_default_library",
         "@com_github_wealdtech_go_eth2_wallet_store_filesystem//:go_default_library",
-        "@com_github_wealdtech_go_eth2_wallet_types//:go_default_library",
+        "@com_github_wealdtech_go_eth2_wallet_types_v2//:go_default_library",
         "@org_golang_google_grpc//:go_default_library",
         "@org_golang_google_grpc//credentials:go_default_library",
         "@org_golang_x_crypto//ssh/terminal:go_default_library",
@@ -48,7 +48,7 @@ go_test(
         "//shared/bytesutil:go_default_library",
         "//shared/testutil:go_default_library",
         "@com_github_wealdtech_go_eth2_wallet_encryptor_keystorev4//:go_default_library",
-        "@com_github_wealdtech_go_eth2_wallet_nd//:go_default_library",
+        "@com_github_wealdtech_go_eth2_wallet_nd_v2//:go_default_library",
         "@com_github_wealdtech_go_eth2_wallet_store_filesystem//:go_default_library",
     ],
 )

validator/keymanager/wallet.go

@@ -11,7 +11,7 @@ import (
 	"github.com/prysmaticlabs/prysm/shared/bytesutil"
 	e2wallet "github.com/wealdtech/go-eth2-wallet"
 	filesystem "github.com/wealdtech/go-eth2-wallet-store-filesystem"
-	e2wtypes "github.com/wealdtech/go-eth2-wallet-types"
+	e2wtypes "github.com/wealdtech/go-eth2-wallet-types/v2"
 )
 
 type walletOpts struct {
@@ -128,7 +128,7 @@ func (km *Wallet) Sign(pubKey [48]byte, root [32]byte) (*bls.Signature, error) {
 		return nil, ErrNoSuchKey
 	}
 	// TODO(#4817) Update with new library to remove domain here.
-	sig, err := account.Sign(root[:], 0)
+	sig, err := account.Sign(root[:])
 	if err != nil {
 		return nil, err
 	}