From 17221c806501e11943fd8359754f79b65713468d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 12 Nov 2020 10:26:25 -0700 Subject: [PATCH 01/90] more clearly define min epoch range for blocksbyrange requests --- README.md | 1 + specs/phase0/p2p-interface.md | 41 ++++++++++++++++++++++++++++--- specs/phase0/weak-subjectivity.md | 5 +++- 3 files changed, 42 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index a3ca7c586..8104ec7cc 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ Core specifications for Eth2 clients be found in [specs](specs/). These are divi * [Deposit Contract](specs/phase0/deposit-contract.md) * [Honest Validator](specs/phase0/validator.md) * [P2P Networking](specs/phase0/p2p-interface.md) +* [Weak Subjectivity](specs/phase0/weak-subjectivity.md.md) ### Phase 1 * [From Phase 0 to Phase 1](specs/phase1/phase1-fork.md) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 5dc892991..8104b54c7 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -93,6 +93,7 @@ It consists of four main sections: - [Why is it called Req/Resp and not RPC?](#why-is-it-called-reqresp-and-not-rpc) - [Why do we allow empty responses in block requests?](#why-do-we-allow-empty-responses-in-block-requests) - [Why does `BeaconBlocksByRange` let the server choose which branch to send blocks from?](#why-does-beaconblocksbyrange-let-the-server-choose-which-branch-to-send-blocks-from) + - [Why are `BlocksByRange` requests only required to be served for the latest `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs?](#why-are-blocksbyrange-requests-only-required-to-be-served-for-the-latest-min_epochs_for_block_requests-epochs) - [What's the effect of empty slots on the sync algorithm?](#whats-the-effect-of-empty-slots-on-the-sync-algorithm) - [Discovery](#discovery) - [Why are we using discv5 and not libp2p Kademlia DHT?](#why-are-we-using-discv5-and-not-libp2p-kademlia-dht) @@ -171,6 +172,7 @@ This section outlines constants that are used in this spec. |---|---|---| | `GOSSIP_MAX_SIZE` | `2**20` (= 1048576, 1 MiB) | The maximum allowed size of uncompressed gossip messages. | | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request | +| `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks | | `MAX_CHUNK_SIZE` | `2**20` (1048576, 1 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. | | `TTFB_TIMEOUT` | `5s` | The maximum time to wait for first byte of request response (time-to-first-byte). | | `RESP_TIMEOUT` | `10s` | The maximum time for complete response transfer. | @@ -179,7 +181,6 @@ This section outlines constants that are used in this spec. | `MESSAGE_DOMAIN_INVALID_SNAPPY` | `0x00000000` | 4-byte domain for gossip message-id isolation of *invalid* snappy messages | | `MESSAGE_DOMAIN_VALID_SNAPPY` | `0x01000000` | 4-byte domain for gossip message-id isolation of *valid* snappy messages | - ## MetaData Clients MUST locally store the following `MetaData`: @@ -746,10 +747,17 @@ The request MUST be encoded as an SSZ-container. The response MUST consist of zero or more `response_chunk`. Each _successful_ `response_chunk` MUST contain a single `SignedBeaconBlock` payload. -Clients MUST keep a record of signed blocks seen since the start of the weak subjectivity period -and MUST support serving requests of blocks up to their own `head_block_root`. +Clients MUST keep a record of signed blocks seen on the epoch range +`[max(GENESIS_EPOCH, current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS), current_epoch]` +where `current_epoch` is defined by the current wall-clock time, +and clients MUST support serving requests of blocks on this range. -Clients MUST respond with at least the first block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOCKS` blocks. +*Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint +MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` +to be compliant with `BlocksByRange` requests. + +Clients MUST respond with at least the first block that exists in the range, if they have it, +and no more than `MAX_REQUEST_BLOCKS` blocks. The following blocks, where they exist, MUST be sent in consecutive order. @@ -1394,6 +1402,31 @@ To avoid this race condition, we allow the responding side to choose which branc The requesting client then goes on to validate the blocks and incorporate them in their own database -- because they follow the same rules, they should at this point arrive at the same canonical chain. +### Why are `BlocksByRange` requests only required to be served for the latest `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs? + +Due to economic finality and weak subjectivity requirements of a proof-of-stake blockchain, for a new node to safely join the network +the node must provide a recent checkpoint found out-of-band. This checkpoint can be in the form of a `root` & `epoch` or it can be the entire +beacon state and then a simple block sync from there to the head. We expect the latter to be the dominant UX strategy. + +These checkpoints *in the worst case* (i.e. very large validator set and maximal allowed safety decay) must be from the +most recent `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, and thus a user must be able to block sync to the head from this starting point. +Thus, this defines the epoch range outside which nodes may prune blocks, and +the epoch range that a new node syncing from a checkpoint must backfill. + +`MIN_EPOCHS_FOR_BLOCK_REQUESTS` is calculated using the arithmetic from `compute_weak_subjectivity_period` found in the +[weak subjectivity guide](./weak-subjectivity.md). Specifically to find this max epoch range, we use the worst case event of a very large validator size +(`>= MIN_PER_EPOCH_CHURN_LIMIT * CHURN_LIMIT_QUOTIENT`). + +```python +MIN_EPOCHS_FOR_BLOCK_REQUESTS = ( + MIN_VALIDATOR_WITHDRAWABILITY_DELAY + + MAX_SAFETY_DECAY * CHURN_LIMIT_QUOTIENT // (2 * 100) +) +``` + +Where `MAX_SAFETY_DECAY = 100` and thus `MIN_EPOCHS_FOR_BLOCK_REQUESTS = 33024` (~5 months). + + ### What's the effect of empty slots on the sync algorithm? When syncing one can only tell that a slot has been skipped on a particular branch diff --git a/specs/phase0/weak-subjectivity.md b/specs/phase0/weak-subjectivity.md index b4d78cb12..797f972a1 100644 --- a/specs/phase0/weak-subjectivity.md +++ b/specs/phase0/weak-subjectivity.md @@ -94,7 +94,9 @@ A brief reference for what these values look like in practice: ## Weak Subjectivity Sync -Clients should allow users to input a Weak Subjectivity Checkpoint at startup, and guarantee that any successful sync leads to the given Weak Subjectivity Checkpoint along the canonical chain. If such a sync is not possible, the client should treat this as a critical and irrecoverable failure. +Clients should allow users to input a Weak Subjectivity Checkpoint at startup, +and guarantee that any successful sync leads to the given Weak Subjectivity Checkpoint along the canonical chain. +If such a sync is not possible, the client should treat this as a critical and irrecoverable failure. ### Weak Subjectivity Sync Procedure @@ -112,6 +114,7 @@ Clients should allow users to input a Weak Subjectivity Checkpoint at startup, a Emit descriptive critical error if this assert fails, then exit client process. ### Checking for Stale Weak Subjectivity Checkpoint + Clients may choose to validate that the input Weak Subjectivity Checkpoint is not stale at the time of startup. To support this mechanism, the client needs to take the state at the Weak Subjectivity Checkpoint as a CLI parameter input (or fetch the state associated with the input Weak Subjectivity Checkpoint from some source). From 56aafbe533c6f5ed7881a65b22c41c1d7367c50e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 9 Dec 2020 12:37:56 -0700 Subject: [PATCH 02/90] add note about signature check when backfilling beaconblocks --- specs/phase0/p2p-interface.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 8104b54c7..df17258ab 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -94,6 +94,7 @@ It consists of four main sections: - [Why do we allow empty responses in block requests?](#why-do-we-allow-empty-responses-in-block-requests) - [Why does `BeaconBlocksByRange` let the server choose which branch to send blocks from?](#why-does-beaconblocksbyrange-let-the-server-choose-which-branch-to-send-blocks-from) - [Why are `BlocksByRange` requests only required to be served for the latest `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs?](#why-are-blocksbyrange-requests-only-required-to-be-served-for-the-latest-min_epochs_for_block_requests-epochs) + - [Why must the proposer signature be checked when backfilling blocks in the database?](#why-must-the-proposer-signature-be-checked-when-backfilling-blocks-in-the-database) - [What's the effect of empty slots on the sync algorithm?](#whats-the-effect-of-empty-slots-on-the-sync-algorithm) - [Discovery](#discovery) - [Why are we using discv5 and not libp2p Kademlia DHT?](#why-are-we-using-discv5-and-not-libp2p-kademlia-dht) @@ -754,7 +755,10 @@ and clients MUST support serving requests of blocks on this range. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` -to be compliant with `BlocksByRange` requests. +to be compliant with `BlocksByRange` requests. To safely perform such a +backfill of blocks to the recent state, the node MUST validate both (1) the +proposer signatures and (2) that the blocks form a valid chain up to the most +recent block referenced in the weak subjectivity state. Clients MUST respond with at least the first block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOCKS` blocks. @@ -1426,6 +1430,20 @@ MIN_EPOCHS_FOR_BLOCK_REQUESTS = ( Where `MAX_SAFETY_DECAY = 100` and thus `MIN_EPOCHS_FOR_BLOCK_REQUESTS = 33024` (~5 months). +### Why must the proposer signature be checked when backfilling blocks in the database? + +When backfilling blocks in a database from a know safe block/state (e.g. when starting from a weak subjectivity state), +the node not only must ensure the `BeaconBlock`s form a chain to the known safe block, +but also must check that the proposer signature is valid in the `SignedBeaconBlock` wrapper. + +This is because the signature is not part of the `BeaconBlock` hash chain, and +thus could be corrupted by an attacker serving valid `BeaconBlock`s but invalid +signatures contained in `SignedBeaconBlock`. + +Although in this particular use case this does not represent a decay in safety +(due to the assumptions of starting at a weak subjectivity checkpoint), it +would represent invalid historic data and could be unwittingly transmitted to +additional nodes. ### What's the effect of empty slots on the sync algorithm? From 2ad8fdb818f58778391d93540892d35570f33237 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 13 Jan 2021 18:03:40 -0700 Subject: [PATCH 03/90] add ability for node to randomly request and descore if not serving blocks on WS period --- specs/phase0/p2p-interface.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index df17258ab..4a412ac19 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -753,6 +753,11 @@ Clients MUST keep a record of signed blocks seen on the epoch range where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blocks on this range. +Synced clients unable to reply to Block requests within the +`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epoch range MAY get descored or disconnected at any time. +Note, due to this it is risky behaviour to begin participating as a full node at the head if having +not yet backfilled on this range. + *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` to be compliant with `BlocksByRange` requests. To safely perform such a From 68d6e4319a5a7292653606b9ff2dcaac2c3c0c5c Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Sat, 1 May 2021 16:30:23 -0700 Subject: [PATCH 04/90] Adjust sync committee size and duration --- configs/mainnet/altair.yaml | 8 ++++---- specs/altair/beacon-chain.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..32b75fc9c 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -12,8 +12,8 @@ PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 # Misc # --------------------------------------------------------------- -# 2**10 (= 1,024) -SYNC_COMMITTEE_SIZE: 1024 +# 2**9 (= 512) +SYNC_COMMITTEE_SIZE: 512 # 2**6 (= 64) SYNC_PUBKEYS_PER_AGGREGATE: 64 # 2**2 (= 4) @@ -22,8 +22,8 @@ INACTIVITY_SCORE_BIAS: 4 # Time parameters # --------------------------------------------------------------- -# 2**8 (= 256) -EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 256 +# 2**9 (= 512) +EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 512 # Signature domains diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f498c40c5..aa961d956 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -118,7 +118,7 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | | - | - | -| `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | +| `SYNC_COMMITTEE_SIZE` | `uint64(2**9)` (= 512) | | `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | @@ -126,7 +126,7 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**9)` (= 512) | epochs | ~54 hours | ### Domain types From 0438f2f27c5c3eb1b43c940962fec702361ee1b2 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Sat, 1 May 2021 16:07:43 -0700 Subject: [PATCH 05/90] whitespace --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 726747c59..25b8bae54 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -409,7 +409,7 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S #### Modified `slash_validator` -*Note*: The function `slash_validator` is modified to use `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` +*Note*: The function `slash_validator` is modified to use `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` and use `PROPOSER_WEIGHT` when calculating the proposer reward. ```python From 7b33c1119a383ea693ada3793699e0bb1d43c8de Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Sat, 1 May 2021 16:08:36 -0700 Subject: [PATCH 06/90] simplify sync committee pubkey aggregation in altair --- configs/mainnet/altair.yaml | 2 -- configs/minimal/altair.yaml | 2 -- specs/altair/beacon-chain.md | 8 +++----- 3 files changed, 3 insertions(+), 9 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..46b74b3aa 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -14,8 +14,6 @@ PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 # --------------------------------------------------------------- # 2**10 (= 1,024) SYNC_COMMITTEE_SIZE: 1024 -# 2**6 (= 64) -SYNC_PUBKEYS_PER_AGGREGATE: 64 # 2**2 (= 4) INACTIVITY_SCORE_BIAS: 4 diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index f9b30eea2..42e464ab4 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -14,8 +14,6 @@ PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 # --------------------------------------------------------------- # [customized] SYNC_COMMITTEE_SIZE: 32 -# [customized] -SYNC_PUBKEYS_PER_AGGREGATE: 16 # 2**2 (= 4) INACTIVITY_SCORE_BIAS: 4 diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 25b8bae54..96a14fe21 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -119,7 +119,6 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | | - | - | | `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | -| `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | ### Time parameters @@ -212,7 +211,7 @@ class SyncAggregate(Container): ```python class SyncCommittee(Container): pubkeys: Vector[BLSPubkey, SYNC_COMMITTEE_SIZE] - pubkey_aggregates: Vector[BLSPubkey, SYNC_COMMITTEE_SIZE // SYNC_PUBKEYS_PER_AGGREGATE] + aggregate_pubkey: BLSPubkey ``` ## Helper functions @@ -306,9 +305,8 @@ def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] - partition = [pubkeys[i:i + SYNC_PUBKEYS_PER_AGGREGATE] for i in range(0, len(pubkeys), SYNC_PUBKEYS_PER_AGGREGATE)] - pubkey_aggregates = [bls.AggregatePKs(preaggregate) for preaggregate in partition] - return SyncCommittee(pubkeys=pubkeys, pubkey_aggregates=pubkey_aggregates) + aggregate_pubkey = bls.AggregatePKs(pubkeys) + return SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=aggregate_pubkey) ``` #### `get_base_reward_per_increment` From 9c3d5982cfbe9a52b02e2bd028a873c9226a34c9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 12:16:39 -0700 Subject: [PATCH 07/90] add documentation about duplicate pubkeys --- specs/altair/beacon-chain.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 96a14fe21..b75835d55 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -302,6 +302,14 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ Return the sync committee for a given ``state`` and ``epoch``. + + ``SyncCommittee`` contains an aggregate pubkey that enables + resource-constrained clients to save some computation when verifying + the sync committee's signature. + + ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_sync_committee_indices`` + returns duplicate indices. Implementations must take care when handling + optimizations relating to aggregation and verification in the presence of duplicates. """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] From 7d236561bd39bb92f3d0c7662c5d4d2762a5fe45 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 12:41:19 -0700 Subject: [PATCH 08/90] adjust subnet count to reflect smaller sync committees --- specs/altair/validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index c69e490b1..c58086b8e 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -74,7 +74,7 @@ This document is currently illustrative for early Altair testnets and some parts | Name | Value | Unit | | - | - | :-: | | `TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE` | `2**2` (= 4) | validators | -| `SYNC_COMMITTEE_SUBNET_COUNT` | `8` | The number of sync committee subnets used in the gossipsub aggregation protocol. | +| `SYNC_COMMITTEE_SUBNET_COUNT` | `4` | The number of sync committee subnets used in the gossipsub aggregation protocol. | ## Containers From 7a168be862294d580003126d881cc7f4ba4e2584 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 13:25:34 -0700 Subject: [PATCH 09/90] allow fault tolerance equal to threshold, not just above --- specs/altair/sync-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index 925c882ef..fba263bb1 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -185,7 +185,7 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda store.valid_updates.append(update) if ( - sum(update.sync_committee_bits) * 3 > len(update.sync_committee_bits) * 2 + sum(update.sync_committee_bits) * 3 >= len(update.sync_committee_bits) * 2 and update.finality_header != BeaconBlockHeader() ): # Apply update if (1) 2/3 quorum is reached and (2) we have a finality proof. From 3b803241193083696c2922f5983712a269d8d62b Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 13:42:02 -0700 Subject: [PATCH 10/90] Compute `LIGHT_CLIENT_UPDATE_TIMEOUT` in lieu of maintaining a constant --- configs/mainnet/altair.yaml | 2 -- configs/minimal/altair.yaml | 2 -- specs/altair/sync-protocol.md | 10 ++-------- 3 files changed, 2 insertions(+), 12 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..6db260cca 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -47,8 +47,6 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 # 2**13 MAX_VALID_LIGHT_CLIENT_UPDATES: 8192 -# 2**13 (=8192) -LIGHT_CLIENT_UPDATE_TIMEOUT: 8192 # Validator diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index f9b30eea2..5c5781010 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -47,8 +47,6 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 # [customized] MAX_VALID_LIGHT_CLIENT_UPDATES: 32 -# [customized] -LIGHT_CLIENT_UPDATE_TIMEOUT: 32 # Validator # --------------------------------------------------------------- diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index fba263bb1..ca55c92f7 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -12,7 +12,6 @@ - [Constants](#constants) - [Configuration](#configuration) - [Misc](#misc) - - [Time parameters](#time-parameters) - [Containers](#containers) - [`LightClientSnapshot`](#lightclientsnapshot) - [`LightClientUpdate`](#lightclientupdate) @@ -53,12 +52,6 @@ uses sync committees introduced in [this beacon chain extension](./beacon-chain. | `MIN_SYNC_COMMITTEE_PARTICIPANTS` | `1` | | `MAX_VALID_LIGHT_CLIENT_UPDATES` | `uint64(2**64 - 1)` | -### Time parameters - -| Name | Value | Unit | Duration | -| - | - | :-: | :-: | -| `LIGHT_CLIENT_UPDATE_TIMEOUT` | `Slot(2**13)` | slots | ~27 hours | - ## Containers ### `LightClientSnapshot` @@ -184,6 +177,7 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda validate_light_client_update(store.snapshot, update, genesis_validators_root) store.valid_updates.append(update) + update_timeout = SLOTS_PER_EPOCH * EPOCHS_PER_SYNC_COMMITTEE_PERIOD if ( sum(update.sync_committee_bits) * 3 >= len(update.sync_committee_bits) * 2 and update.finality_header != BeaconBlockHeader() @@ -193,7 +187,7 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda # It may be changed to re-organizable light client design. See the on-going issue eth2.0-specs#2182. apply_light_client_update(store.snapshot, update) store.valid_updates = [] - elif current_slot > store.snapshot.header.slot + LIGHT_CLIENT_UPDATE_TIMEOUT: + elif current_slot > store.snapshot.header.slot + update_timeout: # Forced best update when the update timeout has elapsed apply_light_client_update(store.snapshot, max(store.valid_updates, key=lambda update: sum(update.sync_committee_bits))) From 165c960cf15b63c30bed2dd2ce9b1a36c7734093 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 13:46:30 -0700 Subject: [PATCH 11/90] Update config value to reflect spec --- configs/mainnet/altair.yaml | 4 ++-- configs/minimal/altair.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 6db260cca..b7b7603a2 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -45,8 +45,8 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# 2**13 -MAX_VALID_LIGHT_CLIENT_UPDATES: 8192 +# 2**64 - 1 +MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 # Validator diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index 5c5781010..5618af8ff 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -45,8 +45,8 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# [customized] -MAX_VALID_LIGHT_CLIENT_UPDATES: 32 +# 2**64 - 1 +MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 # Validator # --------------------------------------------------------------- From f37f9a367d4f55957ab70c59e6502993cf43ad3a Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:15:15 +0200 Subject: [PATCH 12/90] include merge in generators --- tests/generators/epoch_processing/main.py | 10 ++++++++-- tests/generators/finality/main.py | 9 ++++++--- tests/generators/fork_choice/main.py | 8 ++++++-- tests/generators/operations/main.py | 13 +++++++++++-- tests/generators/rewards/main.py | 11 +++++++++-- tests/generators/sanity/main.py | 11 +++++++++-- tests/generators/ssz_static/main.py | 2 +- 7 files changed, 50 insertions(+), 14 deletions(-) diff --git a/tests/generators/epoch_processing/main.py b/tests/generators/epoch_processing/main.py index 9efd96534..2aa6381ff 100644 --- a/tests/generators/epoch_processing/main.py +++ b/tests/generators/epoch_processing/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -27,6 +28,10 @@ if __name__ == "__main__": **phase_0_mods, } # also run the previous phase 0 tests + # No epoch-processing changes in Merge and previous testing repeats with new types, so no additional tests required. + # TODO: rebase onto Altair testing later. + merge_mods = phase_0_mods + # TODO Custody Game testgen is disabled for now # custody_game_mods = {**{key: 'eth2spec.test.custody_game.epoch_processing.test_process_' + key for key in [ # 'reveal_deadlines', @@ -37,6 +42,7 @@ if __name__ == "__main__": all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="epoch_processing", specs=specs, all_mods=all_mods) diff --git a/tests/generators/finality/main.py b/tests/generators/finality/main.py index e50425da4..148ddef96 100644 --- a/tests/generators/finality/main.py +++ b/tests/generators/finality/main.py @@ -1,19 +1,22 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": phase_0_mods = {'finality': 'eth2spec.test.phase0.finality.test_finality'} - altair_mods = phase_0_mods # No additional altair specific finality tests + altair_mods = phase_0_mods # No additional Altair specific finality tests + merge_mods = phase_0_mods # No additional Merge specific finality tests all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: spec_merge, } run_state_test_generators(runner_name="finality", specs=specs, all_mods=all_mods) diff --git a/tests/generators/fork_choice/main.py b/tests/generators/fork_choice/main.py index 445ee629c..ae15caa1d 100644 --- a/tests/generators/fork_choice/main.py +++ b/tests/generators/fork_choice/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -13,10 +14,13 @@ if __name__ == "__main__": ]} # No additional Altair specific finality tests, yet. altair_mods = phase_0_mods + # No specific Merge tests yet. TODO: rebase onto Altair testing later. + merge_mods = phase_0_mods all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="fork_choice", specs=specs, all_mods=all_mods) diff --git a/tests/generators/operations/main.py b/tests/generators/operations/main.py index 0c1b84c24..316ccdf10 100644 --- a/tests/generators/operations/main.py +++ b/tests/generators/operations/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -23,6 +24,13 @@ if __name__ == "__main__": **phase_0_mods, } # also run the previous phase 0 tests + merge_mods = { + **{key: 'eth2spec.test.merge.block_processing.test_process_' + key for key in [ + 'execution_payload', + ]}, + **phase_0_mods, # TODO: runs phase0 tests. Rebase to include `altair_mods` testing later. + } + # TODO Custody Game testgen is disabled for now # custody_game_mods = {**{key: 'eth2spec.test.custody_game.block_processing.test_process_' + key for key in [ # 'attestation', @@ -35,6 +43,7 @@ if __name__ == "__main__": all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="operations", specs=specs, all_mods=all_mods) diff --git a/tests/generators/rewards/main.py b/tests/generators/rewards/main.py index 2a0f14863..8e50732e1 100644 --- a/tests/generators/rewards/main.py +++ b/tests/generators/rewards/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -16,9 +17,15 @@ if __name__ == "__main__": # No additional altair specific rewards tests, yet. altair_mods = phase_0_mods + # No additional merge specific rewards tests, yet. + # Note: Block rewards are non-epoch rewards and are tested as part of block processing tests. + # Transaction fees are part of the execution-layer. + merge_mods = phase_0_mods + all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="rewards", specs=specs, all_mods=all_mods) diff --git a/tests/generators/sanity/main.py b/tests/generators/sanity/main.py index 7f7fecc67..3bed6672d 100644 --- a/tests/generators/sanity/main.py +++ b/tests/generators/sanity/main.py @@ -1,11 +1,12 @@ from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -17,9 +18,15 @@ if __name__ == "__main__": 'blocks', ]}, **phase_0_mods} # also run the previous phase 0 tests + # Altair-specific test cases are ignored, but should be included after the Merge is rebased onto Altair work. + merge_mods = {**{key: 'eth2spec.test.merge.sanity.test_' + key for key in [ + 'blocks', + ]}, **phase_0_mods} # TODO: Merge inherits phase0 tests for now. + all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="sanity", specs=specs, all_mods=all_mods) diff --git a/tests/generators/ssz_static/main.py b/tests/generators/ssz_static/main.py index 94ce02335..1e810f71e 100644 --- a/tests/generators/ssz_static/main.py +++ b/tests/generators/ssz_static/main.py @@ -94,7 +94,7 @@ if __name__ == "__main__": settings.append((seed, MAINNET, random_value.RandomizationMode.mode_random, False, 5)) seed += 1 # TODO: enable testing for the whole merge spec. - for fork in TESTGEN_FORKS + (MERGE,): + for fork in TESTGEN_FORKS: gen_runner.run_generator("ssz_static", [ create_provider(fork, config_name, seed, mode, chaos, cases_if_random) for (seed, config_name, mode, chaos, cases_if_random) in settings From ab693c972445a7ba362d2f527447461f377ce82a Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:16:22 +0200 Subject: [PATCH 13/90] update spec test constants for merge --- tests/core/pyspec/eth2spec/test/helpers/constants.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index ccd7b20a2..d8f2a37ba 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -7,21 +7,24 @@ from .typing import SpecForkName, ConfigName # Some of the Spec module functionality is exposed here to deal with phase-specific changes. PHASE0 = SpecForkName('phase0') ALTAIR = SpecForkName('altair') +MERGE = SpecForkName('merge') # Experimental phases (not included in default "ALL_PHASES"): -MERGE = SpecForkName('merge') SHARDING = SpecForkName('sharding') CUSTODY_GAME = SpecForkName('custody_game') DAS = SpecForkName('das') # The forks that pytest runs with. -ALL_PHASES = (PHASE0, ALTAIR) +ALL_PHASES = (PHASE0, ALTAIR, MERGE) # The forks that output to the test vectors. -TESTGEN_FORKS = (PHASE0, ALTAIR) +TESTGEN_FORKS = (PHASE0, ALTAIR, MERGE) # TODO: everything runs in parallel to Altair. # After features are rebased on the Altair fork, this can be reduced to just PHASE0. FORKS_BEFORE_ALTAIR = (PHASE0, MERGE, SHARDING, CUSTODY_GAME, DAS) +# TODO: when rebasing Merge onto Altair, add ALTAIR to this tuple. +FORKS_BEFORE_MERGE = (PHASE0,) + # # Config # From 521cffc3e9c4de950489470eb1eefbf2871a5c33 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:17:10 +0200 Subject: [PATCH 14/90] update execution-payload processing to isolate payload from block body --- specs/merge/beacon-chain.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index b9c443962..d665f1151 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -146,8 +146,8 @@ def is_transition_completed(state: BeaconState) -> bool: #### `is_transition_block` ```python -def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> bool: - return not is_transition_completed(state) and block_body.execution_payload != ExecutionPayload() +def is_transition_block(state: BeaconState, block: BeaconBlock) -> bool: + return not is_transition_completed(state) and block.body.execution_payload != ExecutionPayload() ``` #### `compute_time_at_slot` @@ -168,7 +168,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) - process_execution_payload(state, block.body) # [New in Merge] + # Pre-merge, skip execution payload processing + if is_transition_completed(state) or is_transition_block(state, block): + process_execution_payload(state, block.body.execution_payload) # [New in Merge] ``` #### Execution payload processing @@ -181,16 +183,10 @@ The body of the function is implementation dependent. ##### `process_execution_payload` ```python -def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None: +def process_execution_payload(state: BeaconState, execution_payload: ExecutionPayload) -> None: """ Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ - # Pre-merge, skip processing - if not is_transition_completed(state) and not is_transition_block(state, body): - return - - execution_payload = body.execution_payload - if is_transition_completed(state): assert execution_payload.parent_hash == state.latest_execution_payload_header.block_hash assert execution_payload.number == state.latest_execution_payload_header.number + 1 From 25d0d673a9b0950c0aa2d0b2406f849c9dca0470 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:18:01 +0200 Subject: [PATCH 15/90] start testing of merge functionality --- tests/core/pyspec/eth2spec/test/context.py | 8 +- .../pyspec/eth2spec/test/helpers/block.py | 6 +- .../test/helpers/execution_payload.py | 26 ++++ .../pyspec/eth2spec/test/merge/__init__.py | 0 .../test/merge/block_processing/__init__.py | 0 .../test_process_execution_payload.py | 147 ++++++++++++++++++ .../eth2spec/test/merge/sanity/__init__.py | 0 .../eth2spec/test/merge/sanity/test_blocks.py | 25 +++ 8 files changed, 210 insertions(+), 2 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/helpers/execution_payload.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/block_processing/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/sanity/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 438e611cf..8699c65ad 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -8,7 +8,7 @@ from eth2spec.utils import bls from .exceptions import SkippedTest from .helpers.constants import ( PHASE0, ALTAIR, - ALL_PHASES, FORKS_BEFORE_ALTAIR, + ALL_PHASES, FORKS_BEFORE_ALTAIR, FORKS_BEFORE_MERGE, ) from .helpers.genesis import create_genesis_state from .utils import vector_test, with_meta_tags @@ -365,3 +365,9 @@ def is_post_altair(spec): if spec.fork in FORKS_BEFORE_ALTAIR: return False return True + + +def is_post_merge(spec): + if spec.fork in FORKS_BEFORE_MERGE: + return False + return True diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 6949b54bc..64f406633 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -1,4 +1,5 @@ -from eth2spec.test.context import is_post_altair +from eth2spec.test.context import is_post_altair, is_post_merge +from eth2spec.test.helpers.execution_payload import build_empty_execution_payload from eth2spec.test.helpers.keys import privkeys from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls @@ -94,6 +95,9 @@ def build_empty_block(spec, state, slot=None): if is_post_altair(spec): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY + if is_post_merge(spec): + empty_block.body.execution_payload = build_empty_execution_payload(spec, state) + apply_randao_reveal(spec, state, empty_block) return empty_block diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py new file mode 100644 index 000000000..e829e07c7 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -0,0 +1,26 @@ + +def build_empty_execution_payload(spec, state): + """ + Assuming a pre-state of the same slot, build a valid ExecutionPayload without any transactions. + """ + latest = state.latest_execution_payload_header + timestamp = spec.compute_time_at_slot(state, state.slot) + empty_txs = spec.List[spec.OpaqueTransaction, spec.MAX_EXECUTION_TRANSACTIONS]() + + payload = spec.ExecutionPayload( + block_hash=spec.Hash32(), + parent_hash=latest.block_hash, + coinbase=spec.Bytes20(), + state_root=latest.state_root, # no changes to the state + number=latest.number + 1, + gas_limit=latest.gas_limit, # retain same limit + gas_used=0, # empty block, 0 gas + timestamp=timestamp, + receipt_root=b"no receipts here" + b"\x00"*16, # TODO: root of empty MPT may be better. + logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? + transactions_root=empty_txs, + ) + # TODO: real RLP + block hash logic would be nice, requires RLP and keccak256 dependency however. + payload.block_hash = spec.Hash32(spec.hash(payload.hash_tree_root() + b"FAKE RLP HASH")) + + return payload diff --git a/tests/core/pyspec/eth2spec/test/merge/__init__.py b/tests/core/pyspec/eth2spec/test/merge/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/__init__.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py new file mode 100644 index 000000000..0cf5cef5f --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -0,0 +1,147 @@ +from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases_except + +with_merge_and_later = with_all_phases_except([PHASE0, ALTAIR]) + + +def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): + """ + Run ``process_execution_payload``, yielding: + - pre-state ('pre') + - execution payload ('execution_payload') + - execution details, to mock EVM execution ('execution.yml', a dict with 'execution_valid' key and boolean value) + - post-state ('post'). + If ``valid == False``, run expecting ``AssertionError`` + """ + + yield 'pre', state + yield 'execution', {'execution_valid': execution_valid} + yield 'execution_payload', execution_payload + + if not valid: + expect_assertion_error(lambda: spec.process_execution_payload(state, execution_payload)) + yield 'post', None + return + + spec.process_execution_payload(state, execution_payload) + + yield 'post', state + + # TODO: any assertions to make? + + +@with_merge_and_later +@spec_state_test +def test_success_first_payload(spec, state): + assert not spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_regular_payload(spec, state): + # TODO: setup state + assert spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_first_payload_with_gap_slot(spec, state): + # TODO: transition gap slot + + assert not spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_regular_payload_with_gap_slot(spec, state): + # TODO: setup state + assert spec.is_transition_completed(state) + # TODO: transition gap slot + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_bad_execution_first_payload(spec, state): + # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) + + # TODO: execution payload. + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_execution_regular_payload(spec, state): + # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_parent_hash_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_number_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_everything_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_timestamp_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_timestamp_regular_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + diff --git a/tests/core/pyspec/eth2spec/test/merge/sanity/__init__.py b/tests/core/pyspec/eth2spec/test/merge/sanity/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py new file mode 100644 index 000000000..1f9e69167 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py @@ -0,0 +1,25 @@ +from eth2spec.test.helpers.state import ( + state_transition_and_sign_block +) +from eth2spec.test.helpers.block import ( + build_empty_block_for_next_slot +) +from eth2spec.test.context import ( + with_all_phases, spec_state_test +) + + +@with_all_phases +@spec_state_test +def test_empty_block_transition(spec, state): + yield 'pre', state + + block = build_empty_block_for_next_slot(spec, state) + assert len(block.body.execution_payload.transactions) == 0 + + signed_block = state_transition_and_sign_block(spec, state, block) + + yield 'blocks', [signed_block] + yield 'post', state + +# TODO: tests with EVM, mock or replacement? From 36032fd115c1be0eef52ac4a68c3cc8672b5ed70 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:37:00 +0200 Subject: [PATCH 16/90] update doc about format --- tests/formats/operations/README.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/tests/formats/operations/README.md b/tests/formats/operations/README.md index 033c5fdef..f562a6f2a 100644 --- a/tests/formats/operations/README.md +++ b/tests/formats/operations/README.md @@ -33,17 +33,23 @@ This excludes the other parts of the block-transition. Operations: -| *`operation-name`* | *`operation-object`* | *`input name`* | *`processing call`* | -|-------------------------|-----------------------|----------------------|-----------------------------------------------------------------| -| `attestation` | `Attestation` | `attestation` | `process_attestation(state, attestation)` | -| `attester_slashing` | `AttesterSlashing` | `attester_slashing` | `process_attester_slashing(state, attester_slashing)` | -| `block_header` | `BeaconBlock` | **`block`** | `process_block_header(state, block)` | -| `deposit` | `Deposit` | `deposit` | `process_deposit(state, deposit)` | -| `proposer_slashing` | `ProposerSlashing` | `proposer_slashing` | `process_proposer_slashing(state, proposer_slashing)` | -| `voluntary_exit` | `SignedVoluntaryExit` | `voluntary_exit` | `process_voluntary_exit(state, voluntary_exit)` | -| `sync_aggregate` | `SyncAggregate` | `sync_aggregate` | `process_sync_committee(state, sync_aggregate)` (new in Altair) | +| *`operation-name`* | *`operation-object`* | *`input name`* | *`processing call`* | +|-------------------------|-----------------------|----------------------|----------------------------------------------------------------------| +| `attestation` | `Attestation` | `attestation` | `process_attestation(state, attestation)` | +| `attester_slashing` | `AttesterSlashing` | `attester_slashing` | `process_attester_slashing(state, attester_slashing)` | +| `block_header` | `BeaconBlock` | **`block`** | `process_block_header(state, block)` | +| `deposit` | `Deposit` | `deposit` | `process_deposit(state, deposit)` | +| `proposer_slashing` | `ProposerSlashing` | `proposer_slashing` | `process_proposer_slashing(state, proposer_slashing)` | +| `voluntary_exit` | `SignedVoluntaryExit` | `voluntary_exit` | `process_voluntary_exit(state, voluntary_exit)` | +| `sync_aggregate` | `SyncAggregate` | `sync_aggregate` | `process_sync_committee(state, sync_aggregate)` (new in Altair) | +| `execution_payload` | `ExecutionPayload` | `execution_payload` | `process_execution_payload(state, execution_payload)` (new in Merge) | Note that `block_header` is not strictly an operation (and is a full `Block`), but processed in the same manner, and hence included here. +The `execution_payload` processing normally requires a `verify_execution_state_transition(execution_payload)`, +a responsibility of an (external) execution engine. +During testing this execution is mocked, an `execution.yml` is provided instead: +a dict containing an `execution_valid` boolean field with the verification result. + The resulting state should match the expected `post` state, or if the `post` state is left blank, the handler should reject the input operation as invalid. From 9dcdbafba318fe42333c66f51e703cec2adf1434 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 4 May 2021 17:19:15 -0700 Subject: [PATCH 17/90] Reorganization of config params to put sync committee in one section This may be marginally "cleaner" than the previous approach, keeping constants with the same topic together. --- specs/altair/beacon-chain.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 726747c59..49169eae8 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -114,20 +114,19 @@ This patch updates a few configuration values to move penalty parameters toward | `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` | `uint64(2**6)` (= 64) | | `PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR` | `uint64(2)` | +### Sync committee + +| Name | Value | Unit | Duration | +| - | - | - | - | +| `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | | | +| `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | + ### Misc | Name | Value | -| - | - | -| `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | -| `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | -### Time parameters - -| Name | Value | Unit | Duration | -| - | - | :-: | :-: | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | - ### Domain types | Name | Value | From b310482bce960f77271c1929c47235b63b4f3842 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 4 May 2021 17:19:51 -0700 Subject: [PATCH 18/90] Update specs/altair/beacon-chain.md --- specs/altair/beacon-chain.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 49169eae8..60bbd69d8 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -125,6 +125,7 @@ This patch updates a few configuration values to move penalty parameters toward ### Misc | Name | Value | +| - | - | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | ### Domain types From 79fc41146d898d80358ff6c746217d827587ee2c Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 5 May 2021 13:37:07 +0600 Subject: [PATCH 19/90] Adjust is_transition_block call in fork-choice --- specs/merge/fork-choice.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index 34647f45d..f478dd7e6 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -75,7 +75,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: assert get_ancestor(store, block.parent_root, finalized_slot) == store.finalized_checkpoint.root # [New in Merge] - if is_transition_block(pre_state, block.body): + if is_transition_block(pre_state, block): # Delay consideration of block until PoW block is processed by the PoW node pow_block = get_pow_block(block.body.execution_payload.parent_hash) assert pow_block.is_processed From 786e611c71d5c62bf30c988c692ab88d1a54d8f3 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 12:05:32 +0100 Subject: [PATCH 20/90] Flags to BitVector --- specs/altair/beacon-chain.md | 31 +++++-------------------------- 1 file changed, 5 insertions(+), 26 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 7bf0c2d99..66b069cc5 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -69,7 +69,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `uint8` | a succinct representation of 8 boolean participation flags | +| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | ## Constants @@ -99,6 +99,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | +| `PARTICIPATION_FLAGS_LENGTH` | `8` | ## Configuration @@ -247,28 +248,6 @@ def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: ) ``` -#### `add_flag` - -```python -def add_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: - """ - Return a new ``ParticipationFlags`` adding ``flag_index`` to ``flags``. - """ - flag = ParticipationFlags(2**flag_index) - return flags | flag -``` - -#### `has_flag` - -```python -def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: - """ - Return whether ``flags`` has ``flag_index`` set. - """ - flag = ParticipationFlags(2**flag_index) - return flags & flag == flag -``` - ### Beacon state accessors #### `get_sync_committee_indices` @@ -348,7 +327,7 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo else: epoch_participation = state.previous_epoch_participation active_validator_indices = get_active_validator_indices(state, epoch) - participating_indices = [i for i in active_validator_indices if has_flag(epoch_participation[i], flag_index)] + participating_indices = [i for i in active_validator_indices if epoch_participation[i][flag_index]] return set(filter(lambda index: not state.validators[index].slashed, participating_indices)) ``` @@ -492,8 +471,8 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): for flag_index, weight in get_flag_indices_and_weights(): - if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): - epoch_participation[index] = add_flag(epoch_participation[index], flag_index) + if flag_index in participation_flag_indices and not epoch_participation[index][flag_index]: + epoch_participation[index][flag_index] = True proposer_reward_numerator += get_base_reward(state, index) * weight # Reward proposer From df6bd1b6c33833c264561da361436453e5e55f4c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 12:26:37 +0100 Subject: [PATCH 21/90] BitVector -> Bitvector --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 66b069cc5..e023db1f7 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -69,7 +69,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | +| `ParticipationFlags` | `Bitvector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | ## Constants From d383a1421338e51908fc389519d5bc2e3603e173 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 5 May 2021 19:47:26 +0800 Subject: [PATCH 22/90] Fix ToC --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 60bbd69d8..2272a3359 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -14,8 +14,8 @@ - [Misc](#misc) - [Configuration](#configuration) - [Updated penalty values](#updated-penalty-values) + - [Sync committee](#sync-committee) - [Misc](#misc-1) - - [Time parameters](#time-parameters) - [Domain types](#domain-types) - [Containers](#containers) - [Modified containers](#modified-containers) From 6a9b3671f04f568a314e9d33f5e4b8060d9481a2 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 5 May 2021 19:53:37 +0800 Subject: [PATCH 23/90] Fix the type of `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` and update the config files --- configs/mainnet/altair.yaml | 14 +++++++------- configs/minimal/altair.yaml | 15 ++++++++------- specs/altair/beacon-chain.md | 2 +- 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..477a4e7f7 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -10,22 +10,22 @@ MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR: 64 PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 -# Misc +# Sync committee # --------------------------------------------------------------- # 2**10 (= 1,024) SYNC_COMMITTEE_SIZE: 1024 # 2**6 (= 64) SYNC_PUBKEYS_PER_AGGREGATE: 64 -# 2**2 (= 4) -INACTIVITY_SCORE_BIAS: 4 - - -# Time parameters -# --------------------------------------------------------------- # 2**8 (= 256) EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 256 +# Misc +# --------------------------------------------------------------- +# 2**2 (= 4) +INACTIVITY_SCORE_BIAS: 4 + + # Signature domains # --------------------------------------------------------------- DOMAIN_SYNC_COMMITTEE: 0x07000000 diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index f9b30eea2..36e7c5021 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -10,22 +10,22 @@ MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR: 64 PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 -# Misc +# Sync committee # --------------------------------------------------------------- # [customized] SYNC_COMMITTEE_SIZE: 32 # [customized] SYNC_PUBKEYS_PER_AGGREGATE: 16 -# 2**2 (= 4) -INACTIVITY_SCORE_BIAS: 4 - - -# Time parameters -# --------------------------------------------------------------- # [customized] EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 8 +# Misc +# --------------------------------------------------------------- +# 2**2 (= 4) +INACTIVITY_SCORE_BIAS: 4 + + # Signature domains # --------------------------------------------------------------- DOMAIN_SYNC_COMMITTEE: 0x07000000 @@ -50,6 +50,7 @@ MAX_VALID_LIGHT_CLIENT_UPDATES: 32 # [customized] LIGHT_CLIENT_UPDATE_TIMEOUT: 32 + # Validator # --------------------------------------------------------------- # 2**2 (= 4) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 2272a3359..cbad9413d 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -120,7 +120,7 @@ This patch updates a few configuration values to move penalty parameters toward | - | - | - | - | | `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | | | | `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `uint64(2**8)` (= 256) | epochs | ~27 hours | ### Misc From e2be7614cc2224c75ac6c0356dff48a9b31563e0 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 15:35:36 +0200 Subject: [PATCH 24/90] introduce merge fork version --- specs/merge/beacon-chain.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index d665f1151..50b9551b0 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -13,8 +13,8 @@ - [Introduction](#introduction) - [Custom types](#custom-types) - [Constants](#constants) - - [Transition](#transition) - [Execution](#execution) +- [Configuration](#configuration) - [Containers](#containers) - [Extended containers](#extended-containers) - [`BeaconBlockBody`](#beaconblockbody) @@ -50,12 +50,6 @@ We define the following Python custom types for type hinting and readability: ## Constants -### Transition - -| Name | Value | -| - | - | -| `TRANSITION_TOTAL_DIFFICULTY` | **TBD** | - ### Execution | Name | Value | @@ -64,6 +58,16 @@ We define the following Python custom types for type hinting and readability: | `MAX_EXECUTION_TRANSACTIONS` | `uint64(2**14)` (= 16,384) | | `BYTES_PER_LOGS_BLOOM` | `uint64(2**8)` (= 256) | +## Configuration + +Warning: this configuration is not definitive. + +| Name | Value | +| - | - | +| `MERGE_FORK_VERSION` | `Version('0x02000000')` | +| `MERGE_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | +| `TRANSITION_TOTAL_DIFFICULTY` | **TBD** | + ## Containers ### Extended containers From 470c6dcc6f38093a6991ee8bb78d1385d11d0d08 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 15:35:52 +0200 Subject: [PATCH 25/90] update test runner to handle merge phase --- tests/core/pyspec/eth2spec/test/context.py | 9 +++++++-- tests/core/pyspec/eth2spec/test/helpers/genesis.py | 3 +++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 8699c65ad..65201c0c4 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -7,7 +7,7 @@ from eth2spec.utils import bls from .exceptions import SkippedTest from .helpers.constants import ( - PHASE0, ALTAIR, + PHASE0, ALTAIR, MERGE, ALL_PHASES, FORKS_BEFORE_ALTAIR, FORKS_BEFORE_MERGE, ) from .helpers.genesis import create_genesis_state @@ -312,7 +312,7 @@ def with_phases(phases, other_phases=None): return None run_phases = [phase] - if PHASE0 not in run_phases and ALTAIR not in run_phases: + if PHASE0 not in run_phases and ALTAIR not in run_phases and MERGE not in run_phases: dump_skipping_message("none of the recognized phases are executable, skipping test.") return None @@ -331,12 +331,17 @@ def with_phases(phases, other_phases=None): if ALTAIR in available_phases: phase_dir[ALTAIR] = spec_altair + if MERGE in available_phases: + phase_dir[MERGE] = spec_merge + # return is ignored whenever multiple phases are ran. # This return is for test generators to emit python generators (yielding test vector outputs) if PHASE0 in run_phases: ret = fn(spec=spec_phase0, phases=phase_dir, *args, **kw) if ALTAIR in run_phases: ret = fn(spec=spec_altair, phases=phase_dir, *args, **kw) + if MERGE in run_phases: + ret = fn(spec=spec_merge, phases=phase_dir, *args, **kw) # TODO: merge, sharding, custody_game and das are not executable yet. # Tests that specify these features will not run, and get ignored for these specific phases. diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 49af43ec1..4a34a5eb3 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -1,6 +1,7 @@ from eth2spec.test.helpers.constants import ( ALTAIR, FORKS_BEFORE_ALTAIR, + MERGE, ) from eth2spec.test.helpers.keys import pubkeys @@ -28,6 +29,8 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if spec.fork == ALTAIR: current_version = spec.ALTAIR_FORK_VERSION + elif spec.fork == MERGE: + current_version = spec.MERGE_FORK_VERSION state = spec.BeaconState( genesis_time=0, From ff3a82e0f38de19559bbba52e2693e01d4dad44f Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 15:40:56 +0200 Subject: [PATCH 26/90] fix transactions field in exec payload helper --- tests/core/pyspec/eth2spec/test/helpers/execution_payload.py | 2 +- .../merge/block_processing/test_process_execution_payload.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index e829e07c7..0d035350d 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -18,7 +18,7 @@ def build_empty_execution_payload(spec, state): timestamp=timestamp, receipt_root=b"no receipts here" + b"\x00"*16, # TODO: root of empty MPT may be better. logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? - transactions_root=empty_txs, + transactions=empty_txs, ) # TODO: real RLP + block hash logic would be nice, requires RLP and keccak256 dependency however. payload.block_hash = spec.Hash32(spec.hash(payload.hash_tree_root() + b"FAKE RLP HASH")) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index 0cf5cef5f..f3f521c4b 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,4 +1,5 @@ from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.helpers.execution_payload import build_empty_execution_payload from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases_except with_merge_and_later = with_all_phases_except([PHASE0, ALTAIR]) @@ -36,7 +37,7 @@ def test_success_first_payload(spec, state): assert not spec.is_transition_completed(state) # TODO: execution payload - execution_payload = spec.ExecutionPayload() + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) From 865d7db5cad448e8d2ac37859a74eff9d8a2d633 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:03:52 +0200 Subject: [PATCH 27/90] update altair tests to not collide with Merge + fix merge test triggers --- .../test_process_sync_committee.py | 20 +++++++++---------- .../test_process_sync_committee_updates.py | 8 ++++---- .../test/altair/sanity/test_blocks.py | 17 ++++++++-------- .../unittests/validator/test_validator.py | 8 ++++---- tests/core/pyspec/eth2spec/test/context.py | 8 ++++++++ .../pyspec/eth2spec/test/helpers/block.py | 2 ++ .../test_process_execution_payload.py | 5 +---- .../eth2spec/test/merge/sanity/test_blocks.py | 4 ++-- 8 files changed, 39 insertions(+), 33 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index e0faf3d0d..9b415cfbd 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -17,7 +17,7 @@ from eth2spec.test.helpers.sync_committee import ( ) from eth2spec.test.context import ( expect_assertion_error, - with_all_phases_except, + with_altair_and_later, with_configs, spec_state_test, always_bls, @@ -62,7 +62,7 @@ def get_committee_indices(spec, state, duplicates=False): state.randao_mixes[randao_index] = hash(state.randao_mixes[randao_index]) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_missing_participant(spec, state): @@ -84,7 +84,7 @@ def test_invalid_signature_missing_participant(spec, state): yield from run_sync_committee_processing(spec, state, block, expect_exception=True) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_extra_participant(spec, state): @@ -197,7 +197,7 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): ) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_configs([MINIMAL], reason="to create nonduplicate committee") @spec_state_test def test_sync_committee_rewards_nonduplicate_committee(spec, state): @@ -213,7 +213,7 @@ def test_sync_committee_rewards_nonduplicate_committee(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_configs([MAINNET], reason="to create duplicate committee") @spec_state_test def test_sync_committee_rewards_duplicate_committee(spec, state): @@ -229,7 +229,7 @@ def test_sync_committee_rewards_duplicate_committee(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_not_full_participants(spec, state): @@ -240,7 +240,7 @@ def test_sync_committee_rewards_not_full_participants(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_empty_participants(spec, state): @@ -250,7 +250,7 @@ def test_sync_committee_rewards_empty_participants(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_past_block(spec, state): @@ -289,7 +289,7 @@ def test_invalid_signature_past_block(spec, state): yield from run_sync_committee_processing(spec, state, invalid_block, expect_exception=True) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_configs([MINIMAL], reason="to produce different committee sets") @spec_state_test @always_bls @@ -326,7 +326,7 @@ def test_invalid_signature_previous_committee(spec, state): yield from run_sync_committee_processing(spec, state, block, expect_exception=True) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls @with_configs([MINIMAL], reason="too slow") diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index 8ffc51507..c914c476d 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -2,7 +2,7 @@ from eth2spec.test.context import ( always_bls, spec_state_test, spec_test, - with_all_phases_except, + with_altair_and_later, with_configs, with_custom_state, single_phase, @@ -49,7 +49,7 @@ def run_sync_committees_progress_test(spec, state): assert state.next_sync_committee == third_sync_committee -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls @with_configs([MINIMAL], reason="too slow") @@ -60,7 +60,7 @@ def test_sync_committees_progress_genesis(spec, state): yield from run_sync_committees_progress_test(spec, state) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls @with_configs([MINIMAL], reason="too slow") @@ -73,7 +73,7 @@ def test_sync_committees_progress_not_genesis(spec, state): yield from run_sync_committees_progress_test(spec, state) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_custom_state(balances_fn=misc_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) @spec_test @single_phase diff --git a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py index 48ab6956b..ffe743531 100644 --- a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py @@ -11,9 +11,8 @@ from eth2spec.test.helpers.block import ( from eth2spec.test.helpers.sync_committee import ( compute_aggregate_sync_committee_signature, ) -from eth2spec.test.helpers.constants import PHASE0 from eth2spec.test.context import ( - with_all_phases_except, + with_altair_and_later, spec_state_test, ) @@ -40,46 +39,46 @@ def run_sync_committee_sanity_test(spec, state, fraction_full=1.0): yield 'post', state -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_full_sync_committee_committee(spec, state): next_epoch(spec, state) yield from run_sync_committee_sanity_test(spec, state, fraction_full=1.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_half_sync_committee_committee(spec, state): next_epoch(spec, state) yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.5) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_empty_sync_committee_committee(spec, state): next_epoch(spec, state) yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_full_sync_committee_committee_genesis(spec, state): yield from run_sync_committee_sanity_test(spec, state, fraction_full=1.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_half_sync_committee_committee_genesis(spec, state): yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.5) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_empty_sync_committee_committee_genesis(spec, state): yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_inactivity_scores(spec, state): for _ in range(spec.MIN_EPOCHS_TO_INACTIVITY_PENALTY + 2): diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index 244e1aa7e..e1f5d3860 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -8,7 +8,7 @@ from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( PHASE0, - with_all_phases_except, + with_altair_and_later, with_state, ) @@ -25,7 +25,7 @@ def ensure_assignments_in_sync_committee( assert spec.is_assigned_to_sync_committee(state, epoch, validator_index) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_state def test_is_assigned_to_sync_committee(phases, spec, state): epoch = spec.get_current_epoch(state) @@ -91,7 +91,7 @@ def _get_sync_committee_signature( @only_with_bls() -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_state def test_process_sync_committee_contributions(phases, spec, state): # skip over slots at genesis @@ -144,7 +144,7 @@ def _subnet_for_sync_committee_index(spec, i): return i // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): some_sync_committee_members = list( diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 65201c0c4..775efdcf3 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -367,12 +367,20 @@ def with_configs(configs, reason=None): def is_post_altair(spec): + if spec.fork == MERGE: # TODO: remove parallel Altair-Merge condition after rebase. + return False if spec.fork in FORKS_BEFORE_ALTAIR: return False return True def is_post_merge(spec): + if spec.fork == ALTAIR: # TODO: remove parallel Altair-Merge condition after rebase. + return False if spec.fork in FORKS_BEFORE_MERGE: return False return True + + +with_altair_and_later = with_phases([ALTAIR]) # TODO: include Merge, but not until Merge work is rebased. +with_merge_and_later = with_phases([MERGE]) diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 64f406633..41d0a9131 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -96,6 +96,8 @@ def build_empty_block(spec, state, slot=None): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY if is_post_merge(spec): + if not hasattr(state, 'latest_execution_payload_header'): + raise Exception("panic!!!") empty_block.body.execution_payload = build_empty_execution_payload(spec, state) apply_randao_reveal(spec, state, empty_block) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index f3f521c4b..96a657d7a 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,8 +1,5 @@ -from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.test.helpers.execution_payload import build_empty_execution_payload -from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases_except - -with_merge_and_later = with_all_phases_except([PHASE0, ALTAIR]) +from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_merge_and_later def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): diff --git a/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py index 1f9e69167..4a6db4106 100644 --- a/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py @@ -5,11 +5,11 @@ from eth2spec.test.helpers.block import ( build_empty_block_for_next_slot ) from eth2spec.test.context import ( - with_all_phases, spec_state_test + with_merge_and_later, spec_state_test ) -@with_all_phases +@with_merge_and_later @spec_state_test def test_empty_block_transition(spec, state): yield 'pre', state From cc11328f74d77bf3a914f71c6cbc606620bf0c24 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:24:44 +0200 Subject: [PATCH 28/90] fix merge forkchoice tests with mock get_pow_block --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 37fbd7795..9e5a546fa 100644 --- a/setup.py +++ b/setup.py @@ -441,7 +441,7 @@ ExecutionState = Any def get_pow_block(hash: Bytes32) -> PowBlock: - pass + return PowBlock(block_hash=hash, is_valid=True, is_processed=True, total_difficulty=TRANSITION_TOTAL_DIFFICULTY) def get_execution_state(execution_state_root: Bytes32) -> ExecutionState: From 00cd1c3db76c77fe468721075347df683adb8aa3 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:31:28 +0200 Subject: [PATCH 29/90] fix forkchoice unittest not recognizing merge spec --- .../test/phase0/unittests/fork_choice/test_on_attestation.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py index fcc5e2d21..8ff6bbb38 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py @@ -1,7 +1,7 @@ from eth2spec.test.context import with_all_phases, spec_state_test from eth2spec.test.helpers.block import build_empty_block_for_next_slot from eth2spec.test.helpers.attestations import get_valid_attestation, sign_attestation -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE from eth2spec.test.helpers.state import transition_to, state_transition_and_sign_block, next_epoch, next_slot from eth2spec.test.helpers.fork_choice import get_genesis_forkchoice_store @@ -19,7 +19,7 @@ def run_on_attestation(spec, state, store, attestation, valid=True): spec.on_attestation(store, attestation) sample_index = indexed_attestation.attesting_indices[0] - if spec.fork in (PHASE0, ALTAIR): + if spec.fork in (PHASE0, ALTAIR, MERGE): latest_message = spec.LatestMessage( epoch=attestation.data.target.epoch, root=attestation.data.beacon_block_root, From 2ef6291cbc74237c364c22c4f29f8debafe1d4b0 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:41:59 +0200 Subject: [PATCH 30/90] Minimal execution payload test, more merge-specific testing in later PR --- .../test_process_execution_payload.py | 109 +----------------- 1 file changed, 2 insertions(+), 107 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index 96a657d7a..acb88b944 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,6 +1,6 @@ from eth2spec.test.helpers.execution_payload import build_empty_execution_payload from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_merge_and_later - +from eth2spec.test.helpers.state import next_slot def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): """ @@ -31,115 +31,10 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, @with_merge_and_later @spec_state_test def test_success_first_payload(spec, state): + next_slot(spec, state) assert not spec.is_transition_completed(state) - # TODO: execution payload execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) - -@with_merge_and_later -@spec_state_test -def test_success_regular_payload(spec, state): - # TODO: setup state - assert spec.is_transition_completed(state) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload) - - -@with_merge_and_later -@spec_state_test -def test_success_first_payload_with_gap_slot(spec, state): - # TODO: transition gap slot - - assert not spec.is_transition_completed(state) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload) - - -@with_merge_and_later -@spec_state_test -def test_success_regular_payload_with_gap_slot(spec, state): - # TODO: setup state - assert spec.is_transition_completed(state) - # TODO: transition gap slot - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload) - - -@with_merge_and_later -@spec_state_test -def test_bad_execution_first_payload(spec, state): - # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) - - # TODO: execution payload. - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_execution_regular_payload(spec, state): - # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_parent_hash_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_number_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_everything_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_timestamp_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_timestamp_regular_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - From 56bcb630dbfa487a4797c718e5a0b5c3ac04abc2 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 17:03:29 +0200 Subject: [PATCH 31/90] Lint fixes for merge testing update --- .../altair/block_processing/test_process_sync_committee.py | 1 - .../epoch_processing/test_process_sync_committee_updates.py | 5 +---- .../test/altair/unittests/validator/test_validator.py | 1 - tests/core/pyspec/eth2spec/test/helpers/execution_payload.py | 2 +- .../merge/block_processing/test_process_execution_payload.py | 4 ++-- 5 files changed, 4 insertions(+), 9 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 9b415cfbd..db2664b21 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -9,7 +9,6 @@ from eth2spec.test.helpers.state import ( transition_to, ) from eth2spec.test.helpers.constants import ( - PHASE0, MAINNET, MINIMAL, ) from eth2spec.test.helpers.sync_committee import ( diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c914c476d..c909c791c 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -8,10 +8,7 @@ from eth2spec.test.context import ( single_phase, misc_balances, ) -from eth2spec.test.helpers.constants import ( - PHASE0, - MINIMAL, -) +from eth2spec.test.helpers.constants import MINIMAL from eth2spec.test.helpers.state import transition_to from eth2spec.test.helpers.epoch_processing import ( run_epoch_processing_with, diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index e1f5d3860..c8a894da6 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -7,7 +7,6 @@ from eth2spec.test.helpers.state import transition_to from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( - PHASE0, with_altair_and_later, with_state, ) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index 0d035350d..093b7cf2e 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -16,7 +16,7 @@ def build_empty_execution_payload(spec, state): gas_limit=latest.gas_limit, # retain same limit gas_used=0, # empty block, 0 gas timestamp=timestamp, - receipt_root=b"no receipts here" + b"\x00"*16, # TODO: root of empty MPT may be better. + receipt_root=b"no receipts here" + b"\x00" * 16, # TODO: root of empty MPT may be better. logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? transactions=empty_txs, ) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index acb88b944..d45a2689b 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,7 +1,8 @@ from eth2spec.test.helpers.execution_payload import build_empty_execution_payload -from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_merge_and_later +from eth2spec.test.context import spec_state_test, expect_assertion_error, with_merge_and_later from eth2spec.test.helpers.state import next_slot + def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): """ Run ``process_execution_payload``, yielding: @@ -37,4 +38,3 @@ def test_success_first_payload(spec, state): execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) - From a2cf833437dd5ceba194de702a00ff6243313fe9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 5 May 2021 09:32:28 -0700 Subject: [PATCH 32/90] Update specs/altair/beacon-chain.md Co-authored-by: Hsiao-Wei Wang --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index aa961d956..62f6174cf 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -126,7 +126,7 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**9)` (= 512) | epochs | ~54 hours | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `uint64(2**9)` (= 512) | epochs | ~54 hours | ### Domain types From 238a9b03fc794bfe322ebec4b67290df4159891c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 18:09:36 +0100 Subject: [PATCH 33/90] Correct confusing comments in "get_sync_committee_indices" --- specs/altair/beacon-chain.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e023db1f7..719f93033 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -205,6 +205,7 @@ class BeaconState(Container): ```python class SyncAggregate(Container): sync_committee_bits: Bitvector[SYNC_COMMITTEE_SIZE] + # TODO! Need multiple signatures as discussed between Justin and Vitalik May 3 2021 (see Telegram) sync_committee_signature: BLSSignature ``` @@ -255,8 +256,9 @@ def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: ```python def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices (which may include duplicate indices) + Return the sequence of sync committee indices for a given ``state`` and ``epoch``. + Can contain duplicate indices for small validator sets (< 2 * SYNC_COMMITTEE_SIZE) """ MAX_RANDOM_BYTE = 2**8 - 1 base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) @@ -270,7 +272,7 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val candidate_index = active_validator_indices[shuffled_index] random_byte = hash(seed + uint_to_bytes(uint64(i // 32)))[i % 32] effective_balance = state.validators[candidate_index].effective_balance - if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: # Sample with replacement + if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: sync_committee_indices.append(candidate_index) i += 1 return sync_committee_indices From 2aef63be01bd1a5feadde55aece356e6c6914cbf Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 5 May 2021 20:17:09 +0300 Subject: [PATCH 34/90] Fix back the EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION constant calculation --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index e13ca6c5b..2e2aba8c0 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -406,7 +406,7 @@ def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard # Proposer must have sufficient balance to pay for worst case fee burn EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION = ( - (EFFECTIVE_BALANCE_INCREMENT - EFFECTIVE_BALANCE_INCREMENT) + EFFECTIVE_BALANCE_INCREMENT - EFFECTIVE_BALANCE_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER // HYSTERESIS_QUOTIENT ) min_effective_balance = ( From a382cf6d5c97409868a3da558351522b8b710d95 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 20:04:06 +0100 Subject: [PATCH 35/90] Refactor participation flag list --- specs/altair/beacon-chain.md | 34 ++++++++++------------------------ 1 file changed, 10 insertions(+), 24 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 719f93033..e311734d9 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -100,6 +100,9 @@ Altair is the first beacon chain hard fork. Its main features are: | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | | `PARTICIPATION_FLAGS_LENGTH` | `8` | +| `PARTICIPATION_FLAGS` | `3` | +| `TIMELY_HEAD_WEIGHT` | `uint64(12)` | +| `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | ## Configuration @@ -233,22 +236,6 @@ def eth2_fast_aggregate_verify(pubkeys: Sequence[BLSPubkey], message: Bytes32, s return bls.FastAggregateVerify(pubkeys, message, signature) ``` -### Misc - -#### `get_flag_indices_and_weights` - -```python -def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: - """ - Return paired tuples of participation flag indices along with associated incentivization weights. - """ - return ( - (TIMELY_HEAD_FLAG_INDEX, TIMELY_HEAD_WEIGHT), - (TIMELY_SOURCE_FLAG_INDEX, TIMELY_SOURCE_WEIGHT), - (TIMELY_TARGET_FLAG_INDEX, TIMELY_TARGET_WEIGHT), - ) -``` - ### Beacon state accessors #### `get_sync_committee_indices` @@ -336,16 +323,16 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo #### `get_flag_index_deltas` ```python -def get_flag_index_deltas(state: BeaconState, flag_index: int, weight: uint64) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: +def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return the deltas for a given ``flag_index`` scaled by ``weight`` by scanning through the participation flags. """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, get_previous_epoch(state)) - increment = EFFECTIVE_BALANCE_INCREMENT # Factored out from balances to avoid uint64 overflow - unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // increment - active_increments = get_total_active_balance(state) // increment + weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] + unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT + active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: @@ -376,7 +363,7 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S previous_epoch = get_previous_epoch(state) matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) for index in get_eligible_validator_indices(state): - for (_, weight) in get_flag_indices_and_weights(): + for weight in PARTICIPATION_FLAG_WEIGHTS: # This inactivity penalty cancels the flag reward corresponding to the flag index penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) if index not in matching_target_indices: @@ -472,7 +459,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Update epoch participation flags proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): - for flag_index, weight in get_flag_indices_and_weights(): + for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): if flag_index in participation_flag_indices and not epoch_participation[index][flag_index]: epoch_participation[index][flag_index] = True proposer_reward_numerator += get_base_reward(state, index) * weight @@ -613,8 +600,7 @@ def process_rewards_and_penalties(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: return - flag_indices_and_numerators = get_flag_indices_and_weights() - flag_deltas = [get_flag_index_deltas(state, index, numerator) for (index, numerator) in flag_indices_and_numerators] + flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(PARTICIPATION_FLAGS)] deltas = flag_deltas + [get_inactivity_penalty_deltas(state)] for (rewards, penalties) in deltas: for index in range(len(state.validators)): From 227d1007e67558e7b7a419723efe932470976441 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 20:09:14 +0100 Subject: [PATCH 36/90] Update toc --- specs/altair/beacon-chain.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e311734d9..bd9c23bf3 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -27,10 +27,6 @@ - [Helper functions](#helper-functions) - [`Predicates`](#predicates) - [`eth2_fast_aggregate_verify`](#eth2_fast_aggregate_verify) - - [Misc](#misc-2) - - [`get_flag_indices_and_weights`](#get_flag_indices_and_weights) - - [`add_flag`](#add_flag) - - [`has_flag`](#has_flag) - [Beacon state accessors](#beacon-state-accessors) - [`get_sync_committee_indices`](#get_sync_committee_indices) - [`get_sync_committee`](#get_sync_committee) From 8ac59b73170d452c5a8e16f65b8bb14bbe8c5c75 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 22:38:16 +0200 Subject: [PATCH 37/90] fix old ssz-static todo comment --- tests/generators/ssz_static/main.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/generators/ssz_static/main.py b/tests/generators/ssz_static/main.py index 1e810f71e..d86636e85 100644 --- a/tests/generators/ssz_static/main.py +++ b/tests/generators/ssz_static/main.py @@ -93,7 +93,6 @@ if __name__ == "__main__": seed += 1 settings.append((seed, MAINNET, random_value.RandomizationMode.mode_random, False, 5)) seed += 1 - # TODO: enable testing for the whole merge spec. for fork in TESTGEN_FORKS: gen_runner.run_generator("ssz_static", [ create_provider(fork, config_name, seed, mode, chaos, cases_if_random) From 687641a79bbe9db18de358f6c230e05a59cb905e Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 21:55:36 +0100 Subject: [PATCH 38/90] Remove extra variable --- specs/altair/beacon-chain.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index bd9c23bf3..1a50e7337 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -97,7 +97,6 @@ Altair is the first beacon chain hard fork. Its main features are: | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | | `PARTICIPATION_FLAGS_LENGTH` | `8` | | `PARTICIPATION_FLAGS` | `3` | -| `TIMELY_HEAD_WEIGHT` | `uint64(12)` | | `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | ## Configuration From 55471bc5d4f69678baf6950e6197b5206b2cbb8a Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 21:58:03 +0100 Subject: [PATCH 39/90] Revert "BitVector -> Bitvector" This reverts commit df6bd1b6c33833c264561da361436453e5e55f4c. --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 1a50e7337..9e868fbaa 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -65,7 +65,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `Bitvector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | +| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | ## Constants From 77524036f56c71d4c8199c1393af524b7841290f Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 22:02:37 +0100 Subject: [PATCH 40/90] Revert "Flags to BitVector" This reverts commit 786e611c71d5c62bf30c988c692ab88d1a54d8f3. # Conflicts: # specs/altair/beacon-chain.md --- specs/altair/beacon-chain.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9e868fbaa..5bce9ba4e 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -27,6 +27,9 @@ - [Helper functions](#helper-functions) - [`Predicates`](#predicates) - [`eth2_fast_aggregate_verify`](#eth2_fast_aggregate_verify) + - [Misc](#misc-2) + - [`add_flag`](#add_flag) + - [`has_flag`](#has_flag) - [Beacon state accessors](#beacon-state-accessors) - [`get_sync_committee_indices`](#get_sync_committee_indices) - [`get_sync_committee`](#get_sync_committee) @@ -65,7 +68,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | +| `ParticipationFlags` | `uint8` | a succinct representation of 8 boolean participation flags | ## Constants @@ -95,7 +98,6 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | -| `PARTICIPATION_FLAGS_LENGTH` | `8` | | `PARTICIPATION_FLAGS` | `3` | | `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | @@ -231,6 +233,30 @@ def eth2_fast_aggregate_verify(pubkeys: Sequence[BLSPubkey], message: Bytes32, s return bls.FastAggregateVerify(pubkeys, message, signature) ``` +### Misc + +#### `add_flag` + +```python +def add_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: + """ + Return a new ``ParticipationFlags`` adding ``flag_index`` to ``flags``. + """ + flag = ParticipationFlags(2**flag_index) + return flags | flag +``` + +#### `has_flag` + +```python +def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: + """ + Return whether ``flags`` has ``flag_index`` set. + """ + flag = ParticipationFlags(2**flag_index) + return flags & flag == flag +``` + ### Beacon state accessors #### `get_sync_committee_indices` @@ -311,7 +337,7 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo else: epoch_participation = state.previous_epoch_participation active_validator_indices = get_active_validator_indices(state, epoch) - participating_indices = [i for i in active_validator_indices if epoch_participation[i][flag_index]] + participating_indices = [i for i in active_validator_indices if has_flag(epoch_participation[i], flag_index)] return set(filter(lambda index: not state.validators[index].slashed, participating_indices)) ``` From b041a9b0d68cdee2de0d0d36590f07b49ea12dff Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 22:22:32 +0100 Subject: [PATCH 41/90] Further flag_index revert --- specs/altair/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5bce9ba4e..5a8dcce24 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -481,8 +481,8 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): - if flag_index in participation_flag_indices and not epoch_participation[index][flag_index]: - epoch_participation[index][flag_index] = True + if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): + epoch_participation[index] = add_flag(epoch_participation[index], flag_index) proposer_reward_numerator += get_base_reward(state, index) * weight # Reward proposer From 76b5974d11692326cbbf513e9c04130aceabeba4 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 6 May 2021 02:21:52 +0200 Subject: [PATCH 42/90] is_execution_enabled function + misc review fixes Co-Authored-By: Danny Ryan --- specs/merge/beacon-chain.md | 10 +++++++++- tests/core/pyspec/eth2spec/test/context.py | 1 - .../block_processing/test_process_execution_payload.py | 5 ++++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 50b9551b0..626a86724 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -24,6 +24,7 @@ - [`ExecutionPayloadHeader`](#executionpayloadheader) - [Helper functions](#helper-functions) - [Misc](#misc) + - [`is_execution_enabled`](#is_execution_enabled) - [`is_transition_completed`](#is_transition_completed) - [`is_transition_block`](#is_transition_block) - [`compute_time_at_slot`](#compute_time_at_slot) @@ -140,6 +141,13 @@ class ExecutionPayloadHeader(Container): ### Misc +#### `is_execution_enabled` + +```python +def is_execution_enabled(state: BeaconState, block: BeaconBlock) -> bool: + return is_transition_completed(state) or is_transition_block(state, block) +``` + #### `is_transition_completed` ```python @@ -173,7 +181,7 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_eth1_data(state, block.body) process_operations(state, block.body) # Pre-merge, skip execution payload processing - if is_transition_completed(state) or is_transition_block(state, block): + if is_execution_enabled(state, block): process_execution_payload(state, block.body.execution_payload) # [New in Merge] ``` diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 775efdcf3..7a2e61c22 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -330,7 +330,6 @@ def with_phases(phases, other_phases=None): phase_dir[PHASE0] = spec_phase0 if ALTAIR in available_phases: phase_dir[ALTAIR] = spec_altair - if MERGE in available_phases: phase_dir[MERGE] = spec_merge diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index d45a2689b..fb1da8758 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -13,6 +13,8 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, If ``valid == False``, run expecting ``AssertionError`` """ + pre_exec_header = state.latest_execution_payload_header.copy() + yield 'pre', state yield 'execution', {'execution_valid': execution_valid} yield 'execution_payload', execution_payload @@ -26,7 +28,8 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, yield 'post', state - # TODO: any assertions to make? + assert pre_exec_header != state.latest_execution_payload_header + # TODO: any more assertions to make? @with_merge_and_later From 42733b7e34fa9f0e7fbd9eda0415171e843d0de7 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 6 May 2021 02:27:05 +0200 Subject: [PATCH 43/90] remove merge-test exec-payload trigger debug helper --- tests/core/pyspec/eth2spec/test/helpers/block.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 41d0a9131..64f406633 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -96,8 +96,6 @@ def build_empty_block(spec, state, slot=None): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY if is_post_merge(spec): - if not hasattr(state, 'latest_execution_payload_header'): - raise Exception("panic!!!") empty_block.body.execution_payload = build_empty_execution_payload(spec, state) apply_randao_reveal(spec, state, empty_block) From cbf9f85537c3c91c38ac51b1e3ac4efd2d5f240c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 12:28:18 +0100 Subject: [PATCH 44/90] Remove duplicate inactivity leak --- specs/altair/beacon-chain.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5a8dcce24..9ad2c0089 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -357,10 +357,7 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: - if is_in_inactivity_leak(state): - # This flag reward cancels the inactivity penalty corresponding to the flag index - rewards[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) - else: + if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) else: @@ -384,9 +381,6 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S previous_epoch = get_previous_epoch(state) matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) for index in get_eligible_validator_indices(state): - for weight in PARTICIPATION_FLAG_WEIGHTS: - # This inactivity penalty cancels the flag reward corresponding to the flag index - penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) if index not in matching_target_indices: penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR From cf724daa7a07092a02522879a0b1cb088db7c3b5 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 12:42:05 +0100 Subject: [PATCH 45/90] No inactivity penalty for untimely head --- specs/altair/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9ad2c0089..e90c587be 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -354,13 +354,14 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT + in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: - if not is_in_inactivity_leak(state): + if not in_inactivity_leak: reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - else: + elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) return rewards, penalties ``` From 81a8c2748fed22b6b5753840f228f00b212c8072 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 13:03:26 +0100 Subject: [PATCH 46/90] Integrate get_inactivity_penalty_deltas into reward computation for better readability --- specs/altair/beacon-chain.md | 34 ++++++++-------------------------- 1 file changed, 8 insertions(+), 26 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e90c587be..f464bd640 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -350,7 +350,8 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) - unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, get_previous_epoch(state)) + previous_epoch = get_previous_epoch(state) + unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, previous_epoch) weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT @@ -363,29 +364,11 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) - return rewards, penalties -``` - -#### Modified `get_inactivity_penalty_deltas` - -*Note*: The function `get_inactivity_penalty_deltas` is modified in the selection of matching target indices -and the removal of `BASE_REWARDS_PER_EPOCH`. - -```python -def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: - """ - Return the inactivity penalty deltas by considering timely target participation flags and inactivity scores. - """ - rewards = [Gwei(0) for _ in range(len(state.validators))] - penalties = [Gwei(0) for _ in range(len(state.validators))] - if is_in_inactivity_leak(state): - previous_epoch = get_previous_epoch(state) - matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) - for index in get_eligible_validator_indices(state): - if index not in matching_target_indices: - penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] - penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR - penalties[index] += Gwei(penalty_numerator // penalty_denominator) + # Quadratic inactivity leak + if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index not in unslashed_participating_indices: + penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] + penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR + penalties[index] += Gwei(penalty_numerator // penalty_denominator) return rewards, penalties ``` @@ -617,8 +600,7 @@ def process_rewards_and_penalties(state: BeaconState) -> None: return flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(PARTICIPATION_FLAGS)] - deltas = flag_deltas + [get_inactivity_penalty_deltas(state)] - for (rewards, penalties) in deltas: + for (rewards, penalties) in flag_deltas: for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) decrease_balance(state, ValidatorIndex(index), penalties[index]) From 4c73fec88e61e83240d0d45270e7afafb0d17d0f Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 6 May 2021 10:00:04 -0700 Subject: [PATCH 47/90] convert `LightClientStore` to python object instead of SSZ object this avoids the type overhead of having to define a max size for the object's data and skips the overhead of serialization/consensus for a type that does not need it --- configs/mainnet/altair.yaml | 2 -- configs/minimal/altair.yaml | 3 +-- setup.py | 2 +- specs/altair/sync-protocol.md | 12 ++++++------ .../test/altair/unittests/test_sync_protocol.py | 8 ++++---- 5 files changed, 12 insertions(+), 15 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index b7b7603a2..eac4e1c14 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -45,8 +45,6 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# 2**64 - 1 -MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 # Validator diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index 5618af8ff..0300ff1b8 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -45,8 +45,7 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# 2**64 - 1 -MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 + # Validator # --------------------------------------------------------------- diff --git a/setup.py b/setup.py index 37fbd7795..aad834e2c 100644 --- a/setup.py +++ b/setup.py @@ -548,7 +548,7 @@ ignored_dependencies = [ 'Bytes1', 'Bytes4', 'Bytes20', 'Bytes32', 'Bytes48', 'Bytes96', 'Bitlist', 'Bitvector', 'uint8', 'uint16', 'uint32', 'uint64', 'uint128', 'uint256', 'bytes', 'byte', 'ByteList', 'ByteVector', - 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', + 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', 'Set' ] diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index ca55c92f7..784ea63b2 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -50,7 +50,6 @@ uses sync committees introduced in [this beacon chain extension](./beacon-chain. | Name | Value | | - | - | | `MIN_SYNC_COMMITTEE_PARTICIPANTS` | `1` | -| `MAX_VALID_LIGHT_CLIENT_UPDATES` | `uint64(2**64 - 1)` | ## Containers @@ -87,9 +86,10 @@ class LightClientUpdate(Container): ### `LightClientStore` ```python -class LightClientStore(Container): +@dataclass +class LightClientStore(object): snapshot: LightClientSnapshot - valid_updates: List[LightClientUpdate, MAX_VALID_LIGHT_CLIENT_UPDATES] + valid_updates: Set[LightClientUpdate] ``` ## Helper functions @@ -175,7 +175,7 @@ def apply_light_client_update(snapshot: LightClientSnapshot, update: LightClient def process_light_client_update(store: LightClientStore, update: LightClientUpdate, current_slot: Slot, genesis_validators_root: Root) -> None: validate_light_client_update(store.snapshot, update, genesis_validators_root) - store.valid_updates.append(update) + store.valid_updates.add(update) update_timeout = SLOTS_PER_EPOCH * EPOCHS_PER_SYNC_COMMITTEE_PERIOD if ( @@ -186,10 +186,10 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda # Note that (2) means that the current light client design needs finality. # It may be changed to re-organizable light client design. See the on-going issue eth2.0-specs#2182. apply_light_client_update(store.snapshot, update) - store.valid_updates = [] + store.valid_updates = set() elif current_slot > store.snapshot.header.slot + update_timeout: # Forced best update when the update timeout has elapsed apply_light_client_update(store.snapshot, max(store.valid_updates, key=lambda update: sum(update.sync_committee_bits))) - store.valid_updates = [] + store.valid_updates = set() ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py index 9b8c35e76..932a46ca5 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py @@ -32,7 +32,7 @@ def test_process_light_client_update_not_updated(spec, state): ) store = spec.LightClientStore( snapshot=pre_snapshot, - valid_updates=[] + valid_updates=set(), ) # Block at slot 1 doesn't increase sync committee period, so it won't update snapshot @@ -76,7 +76,7 @@ def test_process_light_client_update_not_updated(spec, state): spec.process_light_client_update(store, update, state.slot, state.genesis_validators_root) assert len(store.valid_updates) == 1 - assert store.valid_updates[0] == update + assert store.valid_updates.pop() == update assert store.snapshot == pre_snapshot @@ -91,7 +91,7 @@ def test_process_light_client_update_timeout(spec, state): ) store = spec.LightClientStore( snapshot=pre_snapshot, - valid_updates=[] + valid_updates=set(), ) # Forward to next sync committee period @@ -156,7 +156,7 @@ def test_process_light_client_update_finality_updated(spec, state): ) store = spec.LightClientStore( snapshot=pre_snapshot, - valid_updates=[] + valid_updates=set(), ) # Change finality From 953b0278a10bbf98ed1f6e921a7accf9b87b62c8 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 6 May 2021 10:23:50 -0700 Subject: [PATCH 48/90] whitespace --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index aad834e2c..aaed891c4 100644 --- a/setup.py +++ b/setup.py @@ -167,7 +167,7 @@ def get_spec(file_name: str) -> SpecObject: comment = _get_eth2_spec_comment(child) if comment == "skip": should_skip = True - + return SpecObject( functions=functions, custom_types=custom_types, From 4b27b076f6c2b3e90b69f6ace83e89c13f3f30fe Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 6 May 2021 12:36:08 -0600 Subject: [PATCH 49/90] add missing comma --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index aaed891c4..039e8ab63 100644 --- a/setup.py +++ b/setup.py @@ -548,7 +548,7 @@ ignored_dependencies = [ 'Bytes1', 'Bytes4', 'Bytes20', 'Bytes32', 'Bytes48', 'Bytes96', 'Bitlist', 'Bitvector', 'uint8', 'uint16', 'uint32', 'uint64', 'uint128', 'uint256', 'bytes', 'byte', 'ByteList', 'ByteVector', - 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', 'Set' + 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', 'Set', ] From ba2c717bf112bb496afc80cedbd1aeac7d59049a Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 22:55:17 +0100 Subject: [PATCH 50/90] Remove PARTICIPATION_FLAGS --- specs/altair/beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f464bd640..f559a55ef 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -98,7 +98,6 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | -| `PARTICIPATION_FLAGS` | `3` | | `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | ## Configuration @@ -599,7 +598,7 @@ def process_rewards_and_penalties(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: return - flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(PARTICIPATION_FLAGS)] + flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(len(PARTICIPATION_FLAG_WEIGHTS))] for (rewards, penalties) in flag_deltas: for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) From 2fc68c451e8168b2c2cc85182e77e41550c15302 Mon Sep 17 00:00:00 2001 From: dankrad Date: Thu, 6 May 2021 22:55:55 +0100 Subject: [PATCH 51/90] Update specs/altair/beacon-chain.md Co-authored-by: Alex Stokes --- specs/altair/beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f559a55ef..f21584f30 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -263,8 +263,7 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ```python def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices - for a given ``state`` and ``epoch``. + Return the sequence of sync committee indices for a given ``state`` and ``epoch``. Can contain duplicate indices for small validator sets (< 2 * SYNC_COMMITTEE_SIZE) """ MAX_RANDOM_BYTE = 2**8 - 1 From 86104ea361921cd6b952bd13a22c6a198648404a Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 09:55:21 -0700 Subject: [PATCH 52/90] Use stable sync committee indices when processing block rewards --- specs/altair/beacon-chain.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index b2ee93557..59b11c8e4 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -571,7 +571,11 @@ def process_sync_committee(state: BeaconState, aggregate: SyncAggregate) -> None proposer_reward = Gwei(participant_reward * PROPOSER_WEIGHT // (WEIGHT_DENOMINATOR - PROPOSER_WEIGHT)) # Apply participant and proposer rewards - committee_indices = get_sync_committee_indices(state, get_current_epoch(state)) + committee_indices = [] + pubkeys = [v.pubkey for v in state.validators] + for pubkey in state.current_sync_committee.pubkeys: + index = pubkeys.index(pubkey) + committee_indices.append(ValidatorIndex(index)) participant_indices = [index for index, bit in zip(committee_indices, aggregate.sync_committee_bits) if bit] for participant_index in participant_indices: increase_balance(state, participant_index, participant_reward) From 04a9595415678013442460ef4a0ee90d0799924b Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 10:06:44 -0700 Subject: [PATCH 53/90] Add notes about sync committee stability --- specs/altair/beacon-chain.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 59b11c8e4..0855d6f11 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -277,6 +277,9 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val """ Return the sequence of sync committee indices (which may include duplicate indices) for a given ``state`` and ``epoch``. + + Note: This function is not stable during a sync committee period as + a validator's effective balance may change enough to affect the sampling. """ MAX_RANDOM_BYTE = 2**8 - 1 base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) @@ -310,6 +313,9 @@ def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_sync_committee_indices`` returns duplicate indices. Implementations must take care when handling optimizations relating to aggregation and verification in the presence of duplicates. + + Note: This function should only be called at sync committee period boundaries, as + ``get_sync_committee_indices`` is not stable within a given period. """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] From b336b710e9bd08594af87a16f53932bbe4672fe9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 15:54:01 -0700 Subject: [PATCH 54/90] Update specs/altair/beacon-chain.md Co-authored-by: vbuterin --- specs/altair/beacon-chain.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 0855d6f11..ac4457b22 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -577,11 +577,8 @@ def process_sync_committee(state: BeaconState, aggregate: SyncAggregate) -> None proposer_reward = Gwei(participant_reward * PROPOSER_WEIGHT // (WEIGHT_DENOMINATOR - PROPOSER_WEIGHT)) # Apply participant and proposer rewards - committee_indices = [] - pubkeys = [v.pubkey for v in state.validators] - for pubkey in state.current_sync_committee.pubkeys: - index = pubkeys.index(pubkey) - committee_indices.append(ValidatorIndex(index)) + all_pubkeys = [v.pubkey for v in state.validators] + committee_indices = [ValidatorIndex(all_pubkeys.index(pubkey)) for pubkey in state.current_sync_committee.pubkeys] participant_indices = [index for index, bit in zip(committee_indices, aggregate.sync_committee_bits) if bit] for participant_index in participant_indices: increase_balance(state, participant_index, participant_reward) From 72a4ff803b3be39da3a29730a273b04ec6f35a87 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 17:02:52 -0700 Subject: [PATCH 55/90] add test to ensure sync committees are referenced from the state --- .../test_process_sync_committee.py | 55 ++++++++++++++++--- 1 file changed, 48 insertions(+), 7 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index e0faf3d0d..3e4d77c0f 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -7,6 +7,7 @@ from eth2spec.test.helpers.block_processing import run_block_processing_to from eth2spec.test.helpers.state import ( state_transition_and_sign_block, transition_to, + next_epoch, ) from eth2spec.test.helpers.constants import ( PHASE0, @@ -160,13 +161,13 @@ def validate_sync_committee_rewards(spec, pre_state, post_state, committee, comm committee_bits, ) - if proposer_index == index: - reward += compute_sync_committee_proposer_reward( - spec, - pre_state, - committee, - committee_bits, - ) + if proposer_index == index: + reward += compute_sync_committee_proposer_reward( + spec, + pre_state, + committee, + committee_bits, + ) assert post_state.balances[index] == pre_state.balances[index] + reward @@ -367,3 +368,43 @@ def test_valid_signature_future_committee(spec, state): ) yield from run_sync_committee_processing(spec, state, block) + + +@with_all_phases_except([PHASE0]) +@spec_state_test +def test_sync_committee_is_only_computed_at_epoch_boundary(spec, state): + """ + Sync committees can only be computed at sync committee period boundaries. + Ensure a client respects the committee in the state (assumed to be derived + in the correct way). + """ + current_epoch = spec.get_current_epoch(state) + + # use a "synthetic" committee to simulate the situation + # where ``spec.get_sync_committee`` at the sync committee + # period epoch boundary would have diverged some epochs into the + # period; ``aggregate_pubkey`` is not relevant to this test + pubkeys = [] + committee_indices = [] + i = 0 + active_validator_count = len(spec.get_active_validator_indices(state, current_epoch)) + while len(pubkeys) < spec.SYNC_COMMITTEE_SIZE: + v = state.validators[i % active_validator_count] + if spec.is_active_validator(v, current_epoch): + pubkeys.append(v.pubkey) + committee_indices.append(i) + i += 1 + + synthetic_committee = spec.SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=spec.BLSPubkey()) + state.current_sync_committee = synthetic_committee + + assert spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD > 3 + for _ in range(3): + next_epoch(spec, state) + + committee = get_committee_indices(spec, state) + assert committee != committee_indices + committee_size = len(committee_indices) + committee_bits = [True] * committee_size + + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) From e38f758d2168b9811f7463998d0b71ea58480f83 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sun, 9 May 2021 17:08:54 -0700 Subject: [PATCH 56/90] `adjustment_quotient` to use `previous_epoch` --- specs/sharding/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index e13ca6c5b..65a0e95b7 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -714,11 +714,11 @@ def process_pending_headers(state: BeaconState) -> None: ```python def charge_confirmed_header_fees(state: BeaconState) -> None: new_gasprice = state.shard_gasprice + previous_epoch = get_previous_epoch(state) adjustment_quotient = ( - get_active_shard_count(state, get_current_epoch(state)) + get_active_shard_count(state, previous_epoch) * SLOTS_PER_EPOCH * GASPRICE_ADJUSTMENT_COEFFICIENT ) - previous_epoch = get_previous_epoch(state) previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): for shard_index in range(get_active_shard_count(state, previous_epoch)): From cd7842557054845462e2f209751c8f61d14c0111 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 10:30:47 -0600 Subject: [PATCH 57/90] lint --- specs/altair/beacon-chain.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5c21573fb..5a4b37a83 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -362,19 +362,22 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence previous_epoch = get_previous_epoch(state) unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, previous_epoch) weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] - unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT + unslashed_participating_balance = get_total_balance(state, unslashed_participating_indices) + unslashed_participating_increments = unslashed_participating_balance // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) - if index in unslashed_participating_indices: + index_participated = index in unslashed_participating_indices + if index_participated: if not in_inactivity_leak: reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) + # Quadratic inactivity leak - if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index not in unslashed_participating_indices: + if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index_participated: penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR penalties[index] += Gwei(penalty_numerator // penalty_denominator) From 1494fe6ace0c54231d8e7d080da7eb5134a829df Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 10:59:33 -0600 Subject: [PATCH 58/90] add get_inactivity_penalty_deltas back in --- specs/altair/beacon-chain.md | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5a4b37a83..265bbfa53 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -37,6 +37,7 @@ - [`get_base_reward`](#get_base_reward) - [`get_unslashed_participating_indices`](#get_unslashed_participating_indices) - [`get_flag_index_deltas`](#get_flag_index_deltas) + - [Modified `get_inactivity_penalty_deltas`](#modified-get_inactivity_penalty_deltas) - [Beacon state mutators](#beacon-state-mutators) - [Modified `slash_validator`](#modified-slash_validator) - [Block processing](#block-processing) @@ -368,19 +369,32 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) - index_participated = index in unslashed_participating_indices - if index_participated: + if index in unslashed_participating_indices: if not in_inactivity_leak: reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): + else: penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) + return rewards, penalties +``` - # Quadratic inactivity leak - if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index_participated: - penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] - penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR - penalties[index] += Gwei(penalty_numerator // penalty_denominator) +#### Modified `get_inactivity_penalty_deltas` + +```python +def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: + """ + Return the inactivity penalty deltas by considering timely target participation flags and inactivity scores. + """ + rewards = [Gwei(0) for _ in range(len(state.validators))] + penalties = [Gwei(0) for _ in range(len(state.validators))] + if is_in_inactivity_leak(state): + previous_epoch = get_previous_epoch(state) + matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) + for index in get_eligible_validator_indices(state): + if index not in matching_target_indices: + penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] + penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR + penalties[index] += Gwei(penalty_numerator // penalty_denominator) return rewards, penalties ``` @@ -613,7 +627,8 @@ def process_rewards_and_penalties(state: BeaconState) -> None: return flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(len(PARTICIPATION_FLAG_WEIGHTS))] - for (rewards, penalties) in flag_deltas: + deltas = flag_deltas + [get_inactivity_penalty_deltas(state)] + for (rewards, penalties) in deltas: for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) decrease_balance(state, ValidatorIndex(index), penalties[index]) From ff706e5c7a8bc68422595e7d3c2c55c9ca2183db Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 12:57:11 -0600 Subject: [PATCH 59/90] add logic for handling sync committee off by one issue --- specs/altair/p2p-interface.md | 11 ++++++++++- specs/altair/validator.md | 6 +++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 85e859191..02a159f7a 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -106,9 +106,18 @@ The following validations MUST pass before forwarding the `signed_contribution_a ```python def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: + # Committees assigned to `slot` sign for `slot - 1` + # This creates the exceptional logic below when transitioning between sync comittee periods + next_slot_epoch = compute_epoch_at_slot(state.slot + 1) + if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): + sync_committee = state.current_sync_committee + else: + sync_committee = state.next_sync_committee + + # Return pubkeys for the subcommittee index sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT i = subcommittee_index * sync_subcommittee_size - return state.current_sync_committee.pubkeys[i:i + sync_subcommittee_size] + return sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` - _[IGNORE]_ The contribution's slot is for the current slot, i.e. `contribution.slot == current_slot`. diff --git a/specs/altair/validator.md b/specs/altair/validator.md index dbcc73486..3a96adb46 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -143,6 +143,10 @@ A validator determines beacon committee assignments and beacon block proposal du To determine sync committee assignments, a validator can run the following function: `is_assigned_to_sync_committee(state, epoch, validator_index)` where `epoch` is an epoch number within the current or next sync committee period. This function is a predicate indicating the presence or absence of the validator in the corresponding sync committee for the queried sync committee period. +*Note*: Being assigned to a sync committee for a given `slot` means that the validator produces and broadcasts signatures for `slot - 1` for inclusion in `slot`. +This means that when assigned to an `epoch` sync committee signatures must be produced and broadcast for slots on range `[compute_start_slot_at_epoch(epoch) - 1, compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH - 1)` +rather than for the range `[compute_start_slot_at_epoch(epoch), compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH)`. + ```python def compute_sync_committee_period(epoch: Epoch) -> uint64: return epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD @@ -261,7 +265,7 @@ This process occurs each slot. ##### Prepare sync committee signature -If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every slot in the current sync committee period, the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. +If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every `slot` in the current sync committee period, the validator should prepare a `SyncCommitteeSignature` for the previous slot (`slot - 1`) according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of `slot - 1`. This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. From d8e2d19ecc510df32ba3e86471e8c760f14dcb68 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:01:31 -0600 Subject: [PATCH 60/90] spelling --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 02a159f7a..df2955f13 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -107,7 +107,7 @@ The following validations MUST pass before forwarding the `signed_contribution_a ```python def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: # Committees assigned to `slot` sign for `slot - 1` - # This creates the exceptional logic below when transitioning between sync comittee periods + # This creates the exceptional logic below when transitioning between sync committee periods next_slot_epoch = compute_epoch_at_slot(state.slot + 1) if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): sync_committee = state.current_sync_committee From 77d607a760865d9dd621da5b1f3e39dfe0e7161e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:24:14 -0600 Subject: [PATCH 61/90] fix test --- tests/core/pyspec/eth2spec/test/helpers/rewards.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index f81c1fc2a..7cf0206b3 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -62,13 +62,13 @@ def run_deltas(spec, state): if is_post_altair(spec): def get_source_deltas(state): - return spec.get_flag_index_deltas(state, spec.TIMELY_SOURCE_FLAG_INDEX, spec.TIMELY_SOURCE_WEIGHT) + return spec.get_flag_index_deltas(state, spec.TIMELY_SOURCE_FLAG_INDEX) def get_head_deltas(state): - return spec.get_flag_index_deltas(state, spec.TIMELY_HEAD_FLAG_INDEX, spec.TIMELY_HEAD_WEIGHT) + return spec.get_flag_index_deltas(state, spec.TIMELY_HEAD_FLAG_INDEX) def get_target_deltas(state): - return spec.get_flag_index_deltas(state, spec.TIMELY_TARGET_FLAG_INDEX, spec.TIMELY_TARGET_WEIGHT) + return spec.get_flag_index_deltas(state, spec.TIMELY_TARGET_FLAG_INDEX) yield from run_attestation_component_deltas( spec, @@ -228,7 +228,7 @@ def run_get_inactivity_penalty_deltas(spec, state): else: base_penalty = sum( base_reward * numerator // spec.WEIGHT_DENOMINATOR - for (_, numerator) in spec.get_flag_indices_and_weights() + for (_, numerator) in spec.PARTICIPATION_FLAG_WEIGHTS ) if not has_enough_for_reward(spec, state, index): From 85198fabfa15d01832484a8e5f4386bbe2964967 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:26:43 -0600 Subject: [PATCH 62/90] lint --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index df2955f13..6f250b57e 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -108,7 +108,7 @@ The following validations MUST pass before forwarding the `signed_contribution_a def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: # Committees assigned to `slot` sign for `slot - 1` # This creates the exceptional logic below when transitioning between sync committee periods - next_slot_epoch = compute_epoch_at_slot(state.slot + 1) + next_slot_epoch = compute_epoch_at_slot(Slot(state.slot + 1)) if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): sync_committee = state.current_sync_committee else: From a6b8574962dc03f6f36b56a744f3542e97ffeed8 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:38:45 -0600 Subject: [PATCH 63/90] test --- specs/altair/beacon-chain.md | 3 +- .../pyspec/eth2spec/test/helpers/rewards.py | 28 ++++++++++++------- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 265bbfa53..90b878cc1 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -366,11 +366,10 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence unslashed_participating_balance = get_total_balance(state, unslashed_participating_indices) unslashed_participating_increments = unslashed_participating_balance // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT - in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: - if not in_inactivity_leak: + if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) else: diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index 7cf0206b3..eb47dac98 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -133,10 +133,17 @@ def run_attestation_component_deltas(spec, state, component_delta_fn, matching_a validator = state.validators[index] enough_for_reward = has_enough_for_reward(spec, state, index) if index in matching_indices and not validator.slashed: - if enough_for_reward: - assert rewards[index] > 0 + if is_post_altair(spec): + if not spec.is_in_inactivity_leak(state) and enough_for_reward: + assert rewards[index] > 0 + else: + assert rewards[index] == 0 else: - assert rewards[index] == 0 + if enough_for_reward: + assert rewards[index] > 0 + else: + assert rewards[index] == 0 + assert penalties[index] == 0 else: assert rewards[index] == 0 @@ -225,18 +232,19 @@ def run_get_inactivity_penalty_deltas(spec, state): if not is_post_altair(spec): cancel_base_rewards_per_epoch = spec.BASE_REWARDS_PER_EPOCH base_penalty = cancel_base_rewards_per_epoch * base_reward - spec.get_proposer_reward(state, index) - else: - base_penalty = sum( - base_reward * numerator // spec.WEIGHT_DENOMINATOR - for (_, numerator) in spec.PARTICIPATION_FLAG_WEIGHTS - ) if not has_enough_for_reward(spec, state, index): assert penalties[index] == 0 elif index in matching_attesting_indices or not has_enough_for_leak_penalty(spec, state, index): - assert penalties[index] == base_penalty + if is_post_altair(spec): + assert penalties[index] == 0 + else: + assert penalties[index] == base_penalty else: - assert penalties[index] > base_penalty + if is_post_altair(spec): + assert penalties[index] > 0 + else: + assert penalties[index] > base_penalty else: assert penalties[index] == 0 From 3c609e02eac16e060ec3934cbe1807c5a2b7e5cf Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 07:28:24 -0600 Subject: [PATCH 64/90] pr feedback --- specs/altair/validator.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 3a96adb46..92e3a75c0 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -146,6 +146,7 @@ This function is a predicate indicating the presence or absence of the validator *Note*: Being assigned to a sync committee for a given `slot` means that the validator produces and broadcasts signatures for `slot - 1` for inclusion in `slot`. This means that when assigned to an `epoch` sync committee signatures must be produced and broadcast for slots on range `[compute_start_slot_at_epoch(epoch) - 1, compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH - 1)` rather than for the range `[compute_start_slot_at_epoch(epoch), compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH)`. +To reduce complexity during the Altair fork, sync committees are not expected to produce signatures for `compute_epoch_at_slot(ALTAIR_FORK_EPOCH) - 1`. ```python def compute_sync_committee_period(epoch: Epoch) -> uint64: @@ -296,11 +297,14 @@ The `subnet_id` is derived from the position in the sync committee such that the ```python def compute_subnets_for_sync_committee(state: BeaconState, validator_index: ValidatorIndex) -> Sequence[uint64]: + next_slot_epoch = compute_epoch_at_slot(Slot(state.slot + 1)) + if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): + sync_committee = state.current_sync_committee + else: + sync_committee = state.next_sync_committee + target_pubkey = state.validators[validator_index].pubkey - sync_committee_indices = [ - index for index, pubkey in enumerate(state.current_sync_committee.pubkeys) - if pubkey == target_pubkey - ] + sync_committee_indices = [index for index, pubkey in enumerate(sync_committee.pubkeys) if pubkey == target_pubkey] return [ uint64(index // (SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT)) for index in sync_committee_indices From b71aa3fb5623589868f9d11a71b664ac46e74405 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 27 Apr 2021 17:24:15 -0700 Subject: [PATCH 65/90] add `transition` spec test format --- .../test/altair/transition/__init__.py | 0 .../test/altair/transition/test_transition.py | 51 +++++++++++++ tests/core/pyspec/eth2spec/test/context.py | 37 +++++++++- tests/core/pyspec/eth2spec/test/utils.py | 48 ++++++++++++- tests/formats/transition/README.md | 72 +++++++++++++++++++ tests/generators/transition/main.py | 42 +++++++++++ tests/generators/transition/requirements.txt | 2 + 7 files changed, 250 insertions(+), 2 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/altair/transition/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py create mode 100644 tests/formats/transition/README.md create mode 100644 tests/generators/transition/main.py create mode 100644 tests/generators/transition/requirements.txt diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/__init__.py b/tests/core/pyspec/eth2spec/test/altair/transition/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py new file mode 100644 index 000000000..a6e06b1d2 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -0,0 +1,51 @@ +from eth2spec.test.context import ( + fork_transition_test, + single_phase, + with_custom_state, + default_activation_threshold, + low_balances, +) +from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.helpers.state import state_transition_and_sign_block +from eth2spec.test.helpers.block import build_empty_block_for_next_slot + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + yield "pre", state + + blocks = [] + for slot in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH): + block = build_empty_block_for_next_slot(spec, state) + state_transition_and_sign_block(spec, state, block) + blocks.append(pre_tag(block)) + + state = post_spec.upgrade_to_altair(state) + + assert state.fork.epoch == fork_epoch + assert state.fork.previous_version == post_spec.GENESIS_FORK_VERSION + assert state.fork.current_version == post_spec.ALTAIR_FORK_VERSION + + block = build_empty_block_for_next_slot(post_spec, state) + state_transition_and_sign_block(post_spec, state, block) + blocks.append(post_tag(block)) + + yield "blocks", blocks + yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR) +def test_normal_transition_with_manual_fork_epoch(state, spec, post_spec, pre_tag, post_tag): + fork_epoch = 2 + yield "fork_epoch", "meta", fork_epoch + + # run test with computed fork_epoch... + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +@with_custom_state(low_balances, default_activation_threshold) +@single_phase +def test_normal_transition_with_low_balances(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + yield "pre", state + + # run test with custom state... diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 7a2e61c22..57071169f 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -11,7 +11,7 @@ from .helpers.constants import ( ALL_PHASES, FORKS_BEFORE_ALTAIR, FORKS_BEFORE_MERGE, ) from .helpers.genesis import create_genesis_state -from .utils import vector_test, with_meta_tags +from .utils import vector_test, with_meta_tags, build_transition_test from random import Random from typing import Any, Callable, Sequence, TypedDict, Protocol @@ -383,3 +383,38 @@ def is_post_merge(spec): with_altair_and_later = with_phases([ALTAIR]) # TODO: include Merge, but not until Merge work is rebased. with_merge_and_later = with_phases([MERGE]) + + +def fork_transition_test(pre_fork_name, post_fork_name, fork_epoch=None): + """ + A decorator to construct a "transition" test from one fork of the eth2 spec + to another. + + Decorator assumes a transition from the `pre_fork_name` fork to the + `post_fork_name` fork. The user can supply a `fork_epoch` at which the + fork occurs or they must compute one (yielding to the generator) during the test + if more custom behavior is desired. + + A test using this decorator should expect to receive as parameters: + `state`: the default state constructed for the `pre_fork_name` fork + according to the `with_state` decorator. + `fork_epoch`: the `fork_epoch` provided to this decorator, if given. + `spec`: the version of the eth2 spec corresponding to `pre_fork_name`. + `post_spec`: the version of the eth2 spec corresponding to `post_fork_name`. + `pre_tag`: a function to tag data as belonging to `pre_fork_name` fork. + Used to discriminate data during consumption of the generated spec tests. + `post_tag`: a function to tag data as belonging to `post_fork_name` fork. + Used to discriminate data during consumption of the generated spec tests. + """ + def _wrapper(fn): + @with_phases([pre_fork_name], other_phases=[post_fork_name]) + @spec_test + @with_state + def _adapter(*args, **kwargs): + wrapped = build_transition_test(fn, + pre_fork_name, + post_fork_name, + fork_epoch=fork_epoch) + return wrapped(*args, **kwargs) + return _adapter + return _wrapper diff --git a/tests/core/pyspec/eth2spec/test/utils.py b/tests/core/pyspec/eth2spec/test/utils.py index bad6c867b..d94aeb5aa 100644 --- a/tests/core/pyspec/eth2spec/test/utils.py +++ b/tests/core/pyspec/eth2spec/test/utils.py @@ -1,5 +1,6 @@ +import inspect from typing import Dict, Any -from eth2spec.utils.ssz.ssz_typing import View +from eth2spec.utils.ssz.ssz_typing import View, boolean, Container from eth2spec.utils.ssz.ssz_impl import serialize @@ -93,3 +94,48 @@ def with_meta_tags(tags: Dict[str, Any]): yield k, 'meta', v return entry return runner + + +class FlaggedContainer(Container): + flag: boolean + obj: Container + + +def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): + """ + Handles the inner plumbing to generate `transition_test`s. + See that decorator in `context.py` for more information. + """ + def _adapter(*args, **kwargs): + post_spec = kwargs["phases"][post_fork_name] + + def pre_tag(obj): + return FlaggedContainer(flag=False, obj=obj) + + def post_tag(obj): + return FlaggedContainer(flag=True, obj=obj) + + yield "post_fork", "meta", post_fork_name + + has_fork_epoch = False + if fork_epoch: + kwargs["fork_epoch"] = fork_epoch + has_fork_epoch = True + yield "fork_epoch", "meta", fork_epoch + + # massage args to handle an optional custom state using + # `with_custom_state` decorator + expected_args = inspect.getfullargspec(fn) + if "phases" not in expected_args.kwonlyargs: + kwargs.pop("phases", None) + + for part in fn(*args, + post_spec=post_spec, + pre_tag=pre_tag, + post_tag=post_tag, + **kwargs): + if part[0] == "fork_epoch": + has_fork_epoch = True + yield part + assert has_fork_epoch + return _adapter diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md new file mode 100644 index 000000000..bd5985e50 --- /dev/null +++ b/tests/formats/transition/README.md @@ -0,0 +1,72 @@ +# Transition testing + +Transition tests to cover processing the chain across a fork boundary. + +Each test case contains a `post_fork` key in the `meta.yaml` that indicates the target fork which also fixes the fork the test begins in. + +Clients should assume forks happen sequentially in the following manner: + +0. `phase0` +1. `altair` + +For example, if a test case has `post_fork` of `altair`, the test consumer should assume the test begins in `phase0` and use that specification to process the initial state and any blocks up until the fork epoch. After the fork happens, the test consumer should use the specification according to the `altair` fork to process the remaining data. + +## Encoding notes + +This test type contains objects that span fork boundaries. +In general, it may not be clear which objects belong to which fork so each +object is prefixed with a SSZ `boolean` to indicate if the object belongs to the post fork or if it belongs to the initial fork. +This "flagged" data should be used to select the appropriate version of the spec when interpreting the enclosed object. + +```python +class FlaggedContainer(Container): + flag: boolean + obj: Container +``` + +If `flag` is `False`, then the `obj` belongs to the **initial** fork. +If `flag` is `True`, then the `obj` belongs to the **post** fork. + +Unless stated otherwise, all references to spec types below refer to SSZ-snappy +encoded data `obj` with the relevant `flag` set: +`FlaggedContainer(flag=flag, obj=obj)`. + +For example, when testing the fork from Phase 0 to Altair, an Altair block is given +as the encoding of `FlaggedContainer(flag=True, obj=SignedBeaconBlock())` where +`SignedBeaconBlock` is the type defined in the Altair spec. + +## Test case format + +### `meta.yaml` + +```yaml +post_fork: string -- String name of the spec after the fork. +fork_epoch: int -- The epoch at which the fork takes place. +blocks_count: int -- The number of blocks processed in this test. +``` + +*Note*: There may be a fork transition function to run at the `fork_epoch`. Refer to the specs for the relevant fork for further details. + +### `pre.ssz_snappy` + +A SSZ-snappy encoded `BeaconState` according to the specification of the initial fork, the state before running the block transitions. + +*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. + +### `blocks_.ssz_snappy` + +A series of files, with `` in range `[0, blocks_count)`. +Blocks must be processed in order, following the main transition function +(i.e. process slot and epoch transitions in between blocks as normal). + +Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version indicated by flag data as described in the `Encoding notes`. + +### `post.ssz_snappy` + +A SSZ-snappy encoded `BeaconState` according to the specification of the post fork, the state after running the block transitions. + +*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. + +## Condition + +The resulting state should match the expected `post` state. diff --git a/tests/generators/transition/main.py b/tests/generators/transition/main.py new file mode 100644 index 000000000..b7fd7b0a8 --- /dev/null +++ b/tests/generators/transition/main.py @@ -0,0 +1,42 @@ +from importlib import reload +from typing import Iterable + +from eth2spec.test.helpers.constants import ALTAIR, MINIMAL, MAINNET, PHASE0 +from eth2spec.config import config_util +from eth2spec.test.altair.transition import test_transition as test_altair_transition +from eth2spec.phase0 import spec as spec_phase0 +from eth2spec.altair import spec as spec_altair + +from eth2spec.gen_helpers.gen_base import gen_runner, gen_typing +from eth2spec.gen_helpers.gen_from_tests.gen import generate_from_tests + + +def create_provider(tests_src, config_name: str, pre_fork_name: str, post_fork_name: str) -> gen_typing.TestProvider: + + def prepare_fn(configs_path: str) -> str: + config_util.prepare_config(configs_path, config_name) + reload(spec_phase0) + reload(spec_altair) + return config_name + + def cases_fn() -> Iterable[gen_typing.TestCase]: + return generate_from_tests( + runner_name='transition', + handler_name='core', + src=tests_src, + fork_name=post_fork_name, + phase=pre_fork_name, + ) + + return gen_typing.TestProvider(prepare=prepare_fn, make_cases=cases_fn) + + +TRANSITION_TESTS = ((PHASE0, ALTAIR, test_altair_transition),) + + +if __name__ == "__main__": + for pre_fork, post_fork, transition_test_module in TRANSITION_TESTS: + gen_runner.run_generator("transition", [ + create_provider(transition_test_module, MINIMAL, pre_fork, post_fork), + create_provider(transition_test_module, MAINNET, pre_fork, post_fork), + ]) diff --git a/tests/generators/transition/requirements.txt b/tests/generators/transition/requirements.txt new file mode 100644 index 000000000..735f863fa --- /dev/null +++ b/tests/generators/transition/requirements.txt @@ -0,0 +1,2 @@ +pytest>=4.4 +../../../[generator] \ No newline at end of file From 0cc6e15b44f075f9258e4bf9524517e0fd4471c1 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 09:57:03 -0700 Subject: [PATCH 66/90] Update tests/formats/transition/README.md Co-authored-by: Adrian Sutton --- tests/formats/transition/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md index bd5985e50..fb8b3b5ea 100644 --- a/tests/formats/transition/README.md +++ b/tests/formats/transition/README.md @@ -51,7 +51,7 @@ blocks_count: int -- The number of blocks processed in this test. A SSZ-snappy encoded `BeaconState` according to the specification of the initial fork, the state before running the block transitions. -*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. +*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the pre fork. ### `blocks_.ssz_snappy` From d34b2a08d5b19e499190673d6385d47802ce3f17 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 11:35:18 -0700 Subject: [PATCH 67/90] Use `fork_block` index in lieu of fork flag --- tests/core/pyspec/eth2spec/test/utils.py | 18 ++++--- tests/formats/transition/README.md | 67 ++++++++++++------------ 2 files changed, 44 insertions(+), 41 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/utils.py b/tests/core/pyspec/eth2spec/test/utils.py index d94aeb5aa..61fc75040 100644 --- a/tests/core/pyspec/eth2spec/test/utils.py +++ b/tests/core/pyspec/eth2spec/test/utils.py @@ -1,6 +1,6 @@ import inspect from typing import Dict, Any -from eth2spec.utils.ssz.ssz_typing import View, boolean, Container +from eth2spec.utils.ssz.ssz_typing import View from eth2spec.utils.ssz.ssz_impl import serialize @@ -96,11 +96,6 @@ def with_meta_tags(tags: Dict[str, Any]): return runner -class FlaggedContainer(Container): - flag: boolean - obj: Container - - def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): """ Handles the inner plumbing to generate `transition_test`s. @@ -109,11 +104,15 @@ def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): def _adapter(*args, **kwargs): post_spec = kwargs["phases"][post_fork_name] + pre_fork_counter = 0 + def pre_tag(obj): - return FlaggedContainer(flag=False, obj=obj) + nonlocal pre_fork_counter + pre_fork_counter += 1 + return obj def post_tag(obj): - return FlaggedContainer(flag=True, obj=obj) + return obj yield "post_fork", "meta", post_fork_name @@ -138,4 +137,7 @@ def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): has_fork_epoch = True yield part assert has_fork_epoch + + if pre_fork_counter > 0: + yield "fork_block", "meta", pre_fork_counter - 1 return _adapter diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md index fb8b3b5ea..832f38ca2 100644 --- a/tests/formats/transition/README.md +++ b/tests/formats/transition/README.md @@ -11,30 +11,6 @@ Clients should assume forks happen sequentially in the following manner: For example, if a test case has `post_fork` of `altair`, the test consumer should assume the test begins in `phase0` and use that specification to process the initial state and any blocks up until the fork epoch. After the fork happens, the test consumer should use the specification according to the `altair` fork to process the remaining data. -## Encoding notes - -This test type contains objects that span fork boundaries. -In general, it may not be clear which objects belong to which fork so each -object is prefixed with a SSZ `boolean` to indicate if the object belongs to the post fork or if it belongs to the initial fork. -This "flagged" data should be used to select the appropriate version of the spec when interpreting the enclosed object. - -```python -class FlaggedContainer(Container): - flag: boolean - obj: Container -``` - -If `flag` is `False`, then the `obj` belongs to the **initial** fork. -If `flag` is `True`, then the `obj` belongs to the **post** fork. - -Unless stated otherwise, all references to spec types below refer to SSZ-snappy -encoded data `obj` with the relevant `flag` set: -`FlaggedContainer(flag=flag, obj=obj)`. - -For example, when testing the fork from Phase 0 to Altair, an Altair block is given -as the encoding of `FlaggedContainer(flag=True, obj=SignedBeaconBlock())` where -`SignedBeaconBlock` is the type defined in the Altair spec. - ## Test case format ### `meta.yaml` @@ -42,16 +18,17 @@ as the encoding of `FlaggedContainer(flag=True, obj=SignedBeaconBlock())` where ```yaml post_fork: string -- String name of the spec after the fork. fork_epoch: int -- The epoch at which the fork takes place. -blocks_count: int -- The number of blocks processed in this test. +fork_block: int -- Optional. The `` of the last block on the initial fork. +blocks_count: int -- Optional. The number of blocks processed in this test. ``` -*Note*: There may be a fork transition function to run at the `fork_epoch`. Refer to the specs for the relevant fork for further details. +*Note*: There may be a fork transition function to run at the `fork_epoch`. +Refer to the specs for the relevant fork for further details. ### `pre.ssz_snappy` -A SSZ-snappy encoded `BeaconState` according to the specification of the initial fork, the state before running the block transitions. - -*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the pre fork. +A SSZ-snappy encoded `BeaconState` according to the specification of +the initial fork, the state before running the block transitions. ### `blocks_.ssz_snappy` @@ -59,13 +36,37 @@ A series of files, with `` in range `[0, blocks_count)`. Blocks must be processed in order, following the main transition function (i.e. process slot and epoch transitions in between blocks as normal). -Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version indicated by flag data as described in the `Encoding notes`. +*Note*: `blocks_count` will be missing if there are no blocks in this test. + +Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version +as indicated by the `post_fork` and `fork_block` data in the `meta.yaml`. + +As blocks span fork boundaires, a `fork_block` number is given in +the `meta.yaml` to help resolve which blocks belong to which fork. + +The `fork_block` is the index in the test data of the **last** block +of the **initial** fork. + +To demonstrate, the following diagram shows slots with `_` and blocks +in those slots as `x`. The fork happens at the epoch delineated by the `|`. + +``` +x x x x +_ _ _ _ | _ _ _ _ +``` + +The `blocks_count` value in the `meta.yaml` in this case is `4` where the +`fork_block` value in the `meta.yaml` is `1`. If this particular example were +testing the fork from Phase 0 to Altair, blocks with indices `0, 1` represent +`SignedBeaconBlock`s defined in the Phase 0 spec and blocks with indices `2, 3` +represent `SignedBeaconBlock`s defined in the Altair spec. + +*Note*: `fork_block` will be missing if `blocks_count` is also missing. ### `post.ssz_snappy` -A SSZ-snappy encoded `BeaconState` according to the specification of the post fork, the state after running the block transitions. - -*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. +A SSZ-snappy encoded `BeaconState` according to the specification of +the post fork, the state after running the block transitions. ## Condition From 0e71496eb5e54ae60e06514853d74d01e53ffc65 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 11:46:46 -0700 Subject: [PATCH 68/90] add "normal" transition test --- .../test/altair/transition/test_transition.py | 77 +++++++++++-------- 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index a6e06b1d2..b9b67d55a 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -1,24 +1,44 @@ -from eth2spec.test.context import ( - fork_transition_test, - single_phase, - with_custom_state, - default_activation_threshold, - low_balances, -) +from eth2spec.test.context import fork_transition_test from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.test.helpers.state import state_transition_and_sign_block -from eth2spec.test.helpers.block import build_empty_block_for_next_slot +from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_empty_block, sign_block + + +def _state_transition_and_sign_block_at_slot(spec, state): + """ + Cribbed from `transition_unsigned_block` helper + where the early parts of the state transition have already + been applied to `state`. + + Used to produce a block during an irregular state transition. + """ + block = build_empty_block(spec, state) + + assert state.latest_block_header.slot < block.slot + assert state.slot == block.slot + spec.process_block(state, block) + block.state_root = state.hash_tree_root() + return sign_block(spec, state, block) @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): yield "pre", state + assert spec.get_current_epoch(state) < fork_epoch + blocks = [] - for slot in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH): + # regular state transition until fork: + for _ in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH - 1): block = build_empty_block_for_next_slot(spec, state) - state_transition_and_sign_block(spec, state, block) - blocks.append(pre_tag(block)) + signed_block = state_transition_and_sign_block(spec, state, block) + blocks.append(pre_tag(signed_block)) + + # irregular state transition to handle fork: + spec.process_slots(state, state.slot + 1) + + assert state.slot % spec.SLOTS_PER_EPOCH == 0 + assert spec.compute_epoch_at_slot(state.slot) == fork_epoch state = post_spec.upgrade_to_altair(state) @@ -26,26 +46,21 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag assert state.fork.previous_version == post_spec.GENESIS_FORK_VERSION assert state.fork.current_version == post_spec.ALTAIR_FORK_VERSION - block = build_empty_block_for_next_slot(post_spec, state) - state_transition_and_sign_block(post_spec, state, block) - blocks.append(post_tag(block)) + signed_block = _state_transition_and_sign_block_at_slot(post_spec, state) + blocks.append(post_tag(signed_block)) + + # continue regular state transition with new spec into next epoch + for _ in range(post_spec.SLOTS_PER_EPOCH): + block = build_empty_block_for_next_slot(post_spec, state) + signed_block = state_transition_and_sign_block(post_spec, state, block) + blocks.append(post_tag(signed_block)) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(set(slots_with_blocks)) == len(slots_with_blocks) + assert set(range(1, state.slot + 1)) == set(slots_with_blocks) yield "blocks", blocks yield "post", state - - -@fork_transition_test(PHASE0, ALTAIR) -def test_normal_transition_with_manual_fork_epoch(state, spec, post_spec, pre_tag, post_tag): - fork_epoch = 2 - yield "fork_epoch", "meta", fork_epoch - - # run test with computed fork_epoch... - - -@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) -@with_custom_state(low_balances, default_activation_threshold) -@single_phase -def test_normal_transition_with_low_balances(state, fork_epoch, spec, post_spec, pre_tag, post_tag): - yield "pre", state - - # run test with custom state... From 3f3aa4fb105298d4126980bc235ec8ef7ad674e4 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 16:21:46 -0700 Subject: [PATCH 69/90] add some altair tests --- .../test/altair/transition/test_transition.py | 158 ++++++++++++++++-- 1 file changed, 140 insertions(+), 18 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index b9b67d55a..b522c1f94 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -1,6 +1,6 @@ from eth2spec.test.context import fork_transition_test from eth2spec.test.helpers.constants import PHASE0, ALTAIR -from eth2spec.test.helpers.state import state_transition_and_sign_block +from eth2spec.test.helpers.state import state_transition_and_sign_block, next_slot from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_empty_block, sign_block @@ -21,20 +21,32 @@ def _state_transition_and_sign_block_at_slot(spec, state): return sign_block(spec, state, block) -@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) -def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): - yield "pre", state +def _all_blocks(_): + return True - assert spec.get_current_epoch(state) < fork_epoch - blocks = [] - # regular state transition until fork: - for _ in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH - 1): - block = build_empty_block_for_next_slot(spec, state) - signed_block = state_transition_and_sign_block(spec, state, block) - blocks.append(pre_tag(signed_block)) +def _skip_slots(*slots): + """ + Skip making a block if its slot is + passed as an argument to this filter + """ + def f(state_at_prior_slot): + return state_at_prior_slot.slot + 1 not in slots + return f - # irregular state transition to handle fork: + +def _state_transition_across_slots(spec, state, slot_count, block_filter=_all_blocks): + for _ in range(slot_count): + should_make_block = block_filter(state) + if should_make_block: + block = build_empty_block_for_next_slot(spec, state) + signed_block = state_transition_and_sign_block(spec, state, block) + yield signed_block + else: + next_slot(spec, state) + + +def _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=True): spec.process_slots(state, state.slot + 1) assert state.slot % spec.SLOTS_PER_EPOCH == 0 @@ -46,14 +58,40 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag assert state.fork.previous_version == post_spec.GENESIS_FORK_VERSION assert state.fork.current_version == post_spec.ALTAIR_FORK_VERSION - signed_block = _state_transition_and_sign_block_at_slot(post_spec, state) - blocks.append(post_tag(signed_block)) + if with_block: + return state, _state_transition_and_sign_block_at_slot(post_spec, state) + else: + return state, None + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count) + ]) + + # irregular state transition to handle fork: + state, block = _do_altair_fork(state, spec, post_spec, fork_epoch) + blocks.append(post_tag(block)) # continue regular state transition with new spec into next epoch - for _ in range(post_spec.SLOTS_PER_EPOCH): - block = build_empty_block_for_next_slot(post_spec, state) - signed_block = state_transition_and_sign_block(post_spec, state, block) - blocks.append(post_tag(signed_block)) + slot_count = post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count) + ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 @@ -64,3 +102,87 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag yield "blocks", blocks yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way except for the first block + of the new fork. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count) + ]) + + # irregular state transition to handle fork: + state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) + + # continue regular state transition with new spec into next epoch + slot_count = post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count) + ]) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(set(slots_with_blocks)) == len(slots_with_blocks) + expected_slots = set(range(1, state.slot + 1)).difference(set([fork_epoch * spec.SLOTS_PER_EPOCH])) + assert expected_slots == set(slots_with_blocks) + + yield "blocks", blocks + yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way except for the first block + of the new fork. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 + slot_count = last_slot_of_pre_fork - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count, block_filter=_skip_slots(last_slot_of_pre_fork)) + ]) + + # irregular state transition to handle fork: + state, block = _do_altair_fork(state, spec, post_spec, fork_epoch) + blocks.append(post_tag(block)) + + # continue regular state transition with new spec into next epoch + slot_count = post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count) + ]) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(set(slots_with_blocks)) == len(slots_with_blocks) + expected_slots = set(range(1, state.slot + 1)).difference(set([last_slot_of_pre_fork])) + assert expected_slots == set(slots_with_blocks) + + yield "blocks", blocks + yield "post", state From c08fb7714c4ba2e2a5d3c65ba84936493e8a88ce Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 16:34:19 -0700 Subject: [PATCH 70/90] More altair fork tests with varied block conditions --- .../test/altair/transition/test_transition.py | 55 +++++++++++++++++++ tests/formats/transition/README.md | 7 +-- 2 files changed, 58 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index b522c1f94..401a781e1 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -35,6 +35,19 @@ def _skip_slots(*slots): return f +def _no_blocks(_): + return False + + +def _only_at(slot): + """ + Only produce a block if its slot is `slot`. + """ + def f(state_at_prior_slot): + return state_at_prior_slot.slot + 1 == slot + return f + + def _state_transition_across_slots(spec, state, slot_count, block_filter=_all_blocks): for _ in range(slot_count): should_make_block = block_filter(state) @@ -186,3 +199,45 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t yield "blocks", blocks yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way except for the first block + of the new fork. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 + slot_count = last_slot_of_pre_fork - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count, block_filter=_no_blocks) + ]) + + # irregular state transition to handle fork: + state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) + + # continue regular state transition with new spec into next epoch + slot_count = post_spec.SLOTS_PER_EPOCH + last_slot = (fork_epoch + 1) * post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count, block_filter=_only_at(last_slot)) + ]) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(slots_with_blocks) == 1 + assert slots_with_blocks[0] == last_slot + + yield "blocks", blocks + yield "post", state diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md index 832f38ca2..37df65539 100644 --- a/tests/formats/transition/README.md +++ b/tests/formats/transition/README.md @@ -19,7 +19,7 @@ For example, if a test case has `post_fork` of `altair`, the test consumer shoul post_fork: string -- String name of the spec after the fork. fork_epoch: int -- The epoch at which the fork takes place. fork_block: int -- Optional. The `` of the last block on the initial fork. -blocks_count: int -- Optional. The number of blocks processed in this test. +blocks_count: int -- The number of blocks processed in this test. ``` *Note*: There may be a fork transition function to run at the `fork_epoch`. @@ -36,8 +36,6 @@ A series of files, with `` in range `[0, blocks_count)`. Blocks must be processed in order, following the main transition function (i.e. process slot and epoch transitions in between blocks as normal). -*Note*: `blocks_count` will be missing if there are no blocks in this test. - Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version as indicated by the `post_fork` and `fork_block` data in the `meta.yaml`. @@ -61,7 +59,8 @@ testing the fork from Phase 0 to Altair, blocks with indices `0, 1` represent `SignedBeaconBlock`s defined in the Phase 0 spec and blocks with indices `2, 3` represent `SignedBeaconBlock`s defined in the Altair spec. -*Note*: `fork_block` will be missing if `blocks_count` is also missing. +*Note*: If `fork_block` is missing, then all block data should be +interpreted as belonging to the post fork. ### `post.ssz_snappy` From d7448255833fbd0880501cbe070d5c2b43d48ca1 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 14:36:58 -0700 Subject: [PATCH 71/90] update docs --- .../eth2spec/test/altair/transition/test_transition.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index 401a781e1..48fdd64c8 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -162,8 +162,8 @@ def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ Transition from the initial `state` to the epoch after the `fork_epoch`, - producing blocks for every slot along the way except for the first block - of the new fork. + producing blocks for every slot along the way except for the last block + of the old fork. """ yield "pre", state @@ -205,8 +205,8 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ Transition from the initial `state` to the epoch after the `fork_epoch`, - producing blocks for every slot along the way except for the first block - of the new fork. + skipping blocks for every slot along the way except for the first block + in the ending epoch. """ yield "pre", state From e2aa595d5fc5f241ea271f8260a8d433bb7a11a9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 11 May 2021 10:06:18 -0700 Subject: [PATCH 72/90] PR feedback --- .../test/altair/transition/test_transition.py | 53 ++++++++++--------- 1 file changed, 27 insertions(+), 26 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index 48fdd64c8..c3b03d663 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -6,9 +6,9 @@ from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_e def _state_transition_and_sign_block_at_slot(spec, state): """ - Cribbed from `transition_unsigned_block` helper + Cribbed from ``transition_unsigned_block`` helper where the early parts of the state transition have already - been applied to `state`. + been applied to ``state``. Used to produce a block during an irregular state transition. """ @@ -41,15 +41,16 @@ def _no_blocks(_): def _only_at(slot): """ - Only produce a block if its slot is `slot`. + Only produce a block if its slot is ``slot``. """ def f(state_at_prior_slot): return state_at_prior_slot.slot + 1 == slot return f -def _state_transition_across_slots(spec, state, slot_count, block_filter=_all_blocks): - for _ in range(slot_count): +def _state_transition_across_slots(spec, state, to_slot, block_filter=_all_blocks): + assert state.slot < to_slot + while state.slot < to_slot: should_make_block = block_filter(state) if should_make_block: block = build_empty_block_for_next_slot(spec, state) @@ -80,7 +81,7 @@ def _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=True): @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, producing blocks for every slot along the way. """ yield "pre", state @@ -88,11 +89,11 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag assert spec.get_current_epoch(state) < fork_epoch # regular state transition until fork: - slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + to_slot = fork_epoch * spec.SLOTS_PER_EPOCH - 1 blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count) + _state_transition_across_slots(spec, state, to_slot) ]) # irregular state transition to handle fork: @@ -100,10 +101,10 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag blocks.append(post_tag(block)) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count) + _state_transition_across_slots(post_spec, state, to_slot) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 @@ -120,7 +121,7 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, producing blocks for every slot along the way except for the first block of the new fork. """ @@ -129,21 +130,21 @@ def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, assert spec.get_current_epoch(state) < fork_epoch # regular state transition until fork: - slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + to_slot = fork_epoch * spec.SLOTS_PER_EPOCH - 1 blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count) + _state_transition_across_slots(spec, state, to_slot) ]) # irregular state transition to handle fork: state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count) + _state_transition_across_slots(post_spec, state, to_slot) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 @@ -159,9 +160,9 @@ def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) -def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): +def test_transition_missing_last_pre_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, producing blocks for every slot along the way except for the last block of the old fork. """ @@ -171,11 +172,11 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t # regular state transition until fork: last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - slot_count = last_slot_of_pre_fork - state.slot + to_slot = last_slot_of_pre_fork blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count, block_filter=_skip_slots(last_slot_of_pre_fork)) + _state_transition_across_slots(spec, state, to_slot, block_filter=_skip_slots(last_slot_of_pre_fork)) ]) # irregular state transition to handle fork: @@ -183,10 +184,10 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t blocks.append(post_tag(block)) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count) + _state_transition_across_slots(post_spec, state, to_slot) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 @@ -204,7 +205,7 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, skipping blocks for every slot along the way except for the first block in the ending epoch. """ @@ -214,22 +215,22 @@ def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pr # regular state transition until fork: last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - slot_count = last_slot_of_pre_fork - state.slot + to_slot = last_slot_of_pre_fork blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count, block_filter=_no_blocks) + _state_transition_across_slots(spec, state, to_slot, block_filter=_no_blocks) ]) # irregular state transition to handle fork: state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot last_slot = (fork_epoch + 1) * post_spec.SLOTS_PER_EPOCH blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count, block_filter=_only_at(last_slot)) + _state_transition_across_slots(post_spec, state, to_slot, block_filter=_only_at(last_slot)) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 From 488ceed4f9de63898d8ebc490d6b044b5a27fec4 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 11:29:37 -0600 Subject: [PATCH 73/90] add notes about repeatedly failing tos erve blocks as being disconncetable --- specs/phase0/p2p-interface.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index a0bd2d4e2..f2c7d8610 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -752,18 +752,20 @@ Clients MUST keep a record of signed blocks seen on the epoch range where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blocks on this range. -Synced clients unable to reply to Block requests within the +Peers are *repeatedly* unable to reply to Block requests within the `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epoch range MAY get descored or disconnected at any time. -Note, due to this it is risky behaviour to begin participating as a full node at the head if having -not yet backfilled on this range. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` -to be compliant with `BlocksByRange` requests. To safely perform such a +to be fully compliant with `BlocksByRange` requests. To safely perform such a backfill of blocks to the recent state, the node MUST validate both (1) the proposer signatures and (2) that the blocks form a valid chain up to the most recent block referenced in the weak subjectivity state. +*Note*: Although clients that bootstrap from a weak subjectivity checkpoint can begin +participating in the networking immediately, other peers that are actively block syncing MAY +disconnect and/or temporarily ban such an un-synced or semi-synced client. + Clients MUST respond with at least the first block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOCKS` blocks. From a9cc036184dedde472e22a3ae9ba45db7a094695 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 11:48:26 -0600 Subject: [PATCH 74/90] remove timely_head penalty --- specs/altair/beacon-chain.md | 2 +- tests/core/pyspec/eth2spec/test/helpers/rewards.py | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 90b878cc1..57fec33df 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -372,7 +372,7 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - else: + elif not flag_index == TIMELY_HEAD_FLAG_INDEX: penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) return rewards, penalties ``` diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index eb47dac98..f563c70a2 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -147,7 +147,9 @@ def run_attestation_component_deltas(spec, state, component_delta_fn, matching_a assert penalties[index] == 0 else: assert rewards[index] == 0 - if enough_for_reward: + if is_post_altair(spec) and 'head' in deltas_name: + assert penalties[index] == 0 + elif enough_for_reward: assert penalties[index] > 0 else: assert penalties[index] == 0 From f328f77e653ad95b552e59517fa7990106044677 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 11 May 2021 10:58:45 -0700 Subject: [PATCH 75/90] clarify fork upgrade conditions --- specs/altair/fork.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..d789e7a8e 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -38,7 +38,11 @@ Note that for the pure Altair networks, we don't apply `upgrade_to_altair` since ### Upgrading the state -After `process_slots` of Phase 0 finishes, if `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. +If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. + +The upgrade occurs after the inner loop of `process_slots` that sets `state.slot` equal to `ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH` finishes. +Care must be taken when transitioning through the fork boundary as implementations will need a modified state transition function that deviates from the Phase 0 spec. +In particular, the `state_transition` function defined in the Phase 0 spec will not expose the correct time to execute the upgrade in the presence of skipped slots at the fork boundary. ```python def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: From 859a7d743e48b2ac24b43f084af847801fc4e1ce Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 11 May 2021 11:18:03 -0700 Subject: [PATCH 76/90] Only allow sync committee period calculation at period boundaries --- specs/altair/beacon-chain.md | 12 +++--- specs/altair/fork.md | 6 ++- .../test_process_sync_committee.py | 41 ------------------- .../test_process_sync_committee_updates.py | 3 +- .../test/altair/sanity/test_blocks.py | 3 +- .../altair/unittests/test_sync_protocol.py | 9 ++-- 6 files changed, 18 insertions(+), 56 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 34a33469e..fac8b3064 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -276,16 +276,14 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ Return the sequence of sync committee indices (which may include duplicate indices) - for a given ``state`` and ``epoch``. - - Note: This function is not stable during a sync committee period as - a validator's effective balance may change enough to affect the sampling. + for a given ``state`` and ``epoch`` at a sync committee period boundary. """ + assert epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0 + MAX_RANDOM_BYTE = 2**8 - 1 - base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) - active_validator_indices = get_active_validator_indices(state, base_epoch) + active_validator_indices = get_active_validator_indices(state, epoch) active_validator_count = uint64(len(active_validator_indices)) - seed = get_seed(state, base_epoch, DOMAIN_SYNC_COMMITTEE) + seed = get_seed(state, epoch, DOMAIN_SYNC_COMMITTEE) i = 0 sync_committee_indices: List[ValidatorIndex] = [] while len(sync_committee_indices) < SYNC_COMMITTEE_SIZE: diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..4498749d3 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -81,7 +81,9 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: inactivity_scores=[uint64(0) for _ in range(len(pre.validators))], ) # Fill in sync committees - post.current_sync_committee = get_sync_committee(post, get_current_epoch(post)) - post.next_sync_committee = get_sync_committee(post, get_current_epoch(post) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_period = epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD + base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + post.current_sync_committee = get_sync_committee(post, base_epoch) + post.next_sync_committee = get_sync_committee(post, base_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) return post ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 2e5d62d11..834f51956 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -7,7 +7,6 @@ from eth2spec.test.helpers.block_processing import run_block_processing_to from eth2spec.test.helpers.state import ( state_transition_and_sign_block, transition_to, - next_epoch, ) from eth2spec.test.helpers.constants import ( MAINNET, MINIMAL, @@ -367,43 +366,3 @@ def test_valid_signature_future_committee(spec, state): ) yield from run_sync_committee_processing(spec, state, block) - - -@with_altair_and_later -@spec_state_test -def test_sync_committee_is_only_computed_at_epoch_boundary(spec, state): - """ - Sync committees can only be computed at sync committee period boundaries. - Ensure a client respects the committee in the state (assumed to be derived - in the correct way). - """ - current_epoch = spec.get_current_epoch(state) - - # use a "synthetic" committee to simulate the situation - # where ``spec.get_sync_committee`` at the sync committee - # period epoch boundary would have diverged some epochs into the - # period; ``aggregate_pubkey`` is not relevant to this test - pubkeys = [] - committee_indices = [] - i = 0 - active_validator_count = len(spec.get_active_validator_indices(state, current_epoch)) - while len(pubkeys) < spec.SYNC_COMMITTEE_SIZE: - v = state.validators[i % active_validator_count] - if spec.is_active_validator(v, current_epoch): - pubkeys.append(v.pubkey) - committee_indices.append(i) - i += 1 - - synthetic_committee = spec.SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=spec.BLSPubkey()) - state.current_sync_committee = synthetic_committee - - assert spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD > 3 - for _ in range(3): - next_epoch(spec, state) - - committee = get_committee_indices(spec, state) - assert committee != committee_indices - committee_size = len(committee_indices) - committee_bits = [True] * committee_size - - yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c909c791c..2910145c9 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -39,8 +39,7 @@ def run_sync_committees_progress_test(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` - current_epoch = spec.get_current_epoch(state) - third_sync_committee = spec.get_sync_committee(state, current_epoch + 2 * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + third_sync_committee = spec.get_sync_committee(state, (next_period + 1) * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py index ffe743531..407455a0a 100644 --- a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py @@ -18,7 +18,8 @@ from eth2spec.test.context import ( def run_sync_committee_sanity_test(spec, state, fraction_full=1.0): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] participants = random.sample(committee, int(len(committee) * fraction_full)) yield 'pre', state diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py index 932a46ca5..554cebda8 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py @@ -46,7 +46,8 @@ def test_process_light_client_update_not_updated(spec, state): body_root=signed_block.message.body.hash_tree_root(), ) # Sync committee signing the header - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] sync_committee_bits = [True] * len(committee) sync_committee_signature = compute_aggregate_sync_committee_signature( spec, @@ -111,7 +112,8 @@ def test_process_light_client_update_timeout(spec, state): ) # Sync committee signing the finalized_block_header - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] sync_committee_bits = [True] * len(committee) sync_committee_signature = compute_aggregate_sync_committee_signature( spec, @@ -190,7 +192,8 @@ def test_process_light_client_update_finality_updated(spec, state): ) # Sync committee signing the finalized_block_header - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] sync_committee_bits = [True] * len(committee) sync_committee_signature = compute_aggregate_sync_committee_signature( spec, From 43ba615b7579ff1ac2e616ed1dd12cccec14118f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 14:13:22 -0600 Subject: [PATCH 77/90] Apply suggestions from code review --- specs/altair/fork.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/fork.md b/specs/altair/fork.md index d789e7a8e..ac2258def 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -40,9 +40,9 @@ Note that for the pure Altair networks, we don't apply `upgrade_to_altair` since If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. -The upgrade occurs after the inner loop of `process_slots` that sets `state.slot` equal to `ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH` finishes. +The upgrade occurs after the completion of the inner loop of `process_slots` that sets `state.slot` equal to `ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH`. Care must be taken when transitioning through the fork boundary as implementations will need a modified state transition function that deviates from the Phase 0 spec. -In particular, the `state_transition` function defined in the Phase 0 spec will not expose the correct time to execute the upgrade in the presence of skipped slots at the fork boundary. +In particular, the outer `state_transition` function defined in the Phase 0 spec will not expose the precise fork slot to execute the upgrade in the presence of skipped slots at the fork boundary. Instead the logic must be within `process_slots`. ```python def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: From 274788277636b0e0e319dcacbcd9a03bad21a1ca Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 15:18:18 -0600 Subject: [PATCH 78/90] use current_Epoch seed when calculating next_sync_committee --- specs/altair/beacon-chain.md | 4 ++-- specs/altair/fork.md | 10 ++++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 34a33469e..aa1509460 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -304,7 +304,7 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val ```python def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ - Return the sync committee for a given ``state`` and ``epoch``. + Return the *next* sync committee for a given ``state`` and ``epoch``. ``SyncCommittee`` contains an aggregate pubkey that enables resource-constrained clients to save some computation when verifying @@ -690,7 +690,7 @@ def process_sync_committee_updates(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + Epoch(1) if next_epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0: state.current_sync_committee = state.next_sync_committee - state.next_sync_committee = get_sync_committee(state, next_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + state.next_sync_committee = get_sync_committee(state, next_epoch) ``` ## Initialize state for pure Altair testnets and test vectors diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..dcd319745 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -80,8 +80,14 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: # Inactivity inactivity_scores=[uint64(0) for _ in range(len(pre.validators))], ) + # Fill in sync committees - post.current_sync_committee = get_sync_committee(post, get_current_epoch(post)) - post.next_sync_committee = get_sync_committee(post, get_current_epoch(post) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_period = epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_period = current_period - min(1, current_period) + current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + + post.current_sync_committee = get_sync_committee(post, previous_base_epoch) + post.next_sync_committee = get_sync_committee(post, current_base_epoch) return post ``` From 200c049778ade1d28064cb837c8ebdd6ca2356fa Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 15:55:33 -0600 Subject: [PATCH 79/90] fix seed calc issue --- specs/altair/beacon-chain.md | 17 ++++++++++++----- .../test_process_sync_committee.py | 6 +++--- .../test_process_sync_committee_updates.py | 2 +- .../pyspec/eth2spec/test/helpers/genesis.py | 11 +++++++---- 4 files changed, 23 insertions(+), 13 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index aa1509460..a69412c31 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -281,11 +281,12 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val Note: This function is not stable during a sync committee period as a validator's effective balance may change enough to affect the sampling. """ + assert epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0 + MAX_RANDOM_BYTE = 2**8 - 1 - base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) - active_validator_indices = get_active_validator_indices(state, base_epoch) + active_validator_indices = get_active_validator_indices(state, epoch) active_validator_count = uint64(len(active_validator_indices)) - seed = get_seed(state, base_epoch, DOMAIN_SYNC_COMMITTEE) + seed = get_seed(state, epoch, DOMAIN_SYNC_COMMITTEE) i = 0 sync_committee_indices: List[ValidatorIndex] = [] while len(sync_committee_indices) < SYNC_COMMITTEE_SIZE: @@ -689,6 +690,7 @@ def process_participation_flag_updates(state: BeaconState) -> None: def process_sync_committee_updates(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + Epoch(1) if next_epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0: + print("HEEEREEE") state.current_sync_committee = state.next_sync_committee state.next_sync_committee = get_sync_committee(state, next_epoch) ``` @@ -735,8 +737,13 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, state.genesis_validators_root = hash_tree_root(state.validators) # [New in Altair] Fill in sync committees - state.current_sync_committee = get_sync_committee(state, get_current_epoch(state)) - state.next_sync_committee = get_sync_committee(state, get_current_epoch(state) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_period = get_current_epoch(state) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_period = current_period - min(1, current_period) + current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + + state.current_sync_committee = get_sync_committee(state, previous_base_epoch) + state.next_sync_committee = get_sync_committee(state, current_base_epoch) return state ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 2e5d62d11..48e556f50 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -345,10 +345,10 @@ def test_valid_signature_future_committee(spec, state): transition_to(spec, state, slot_in_future_sync_committee_period) sync_committee = state.current_sync_committee + next_sync_committee = state.next_sync_committee + expected_next_sync_committee = spec.get_sync_committee(state, epoch_in_future_sync_committee_period) - expected_sync_committee = spec.get_sync_committee(state, epoch_in_future_sync_committee_period) - - assert sync_committee == expected_sync_committee + assert next_sync_committee == expected_next_sync_committee assert sync_committee != old_current_sync_committee assert sync_committee != old_next_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c909c791c..ca25797c4 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -40,7 +40,7 @@ def run_sync_committees_progress_test(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` current_epoch = spec.get_current_epoch(state) - third_sync_committee = spec.get_sync_committee(state, current_epoch + 2 * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + third_sync_committee = spec.get_sync_committee(state, current_epoch + 1) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 4a34a5eb3..23cd04c93 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -69,9 +69,12 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if spec.fork not in FORKS_BEFORE_ALTAIR: # Fill in sync committees - state.current_sync_committee = spec.get_sync_committee(state, spec.get_current_epoch(state)) - state.next_sync_committee = ( - spec.get_sync_committee(state, spec.get_current_epoch(state) + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) - ) + current_period = spec.get_current_epoch(state) // spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_period = current_period - min(1, current_period) + current_base_epoch = current_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_base_epoch = previous_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + + state.current_sync_committee = spec.get_sync_committee(state, previous_base_epoch) + state.next_sync_committee = spec.get_sync_committee(state, current_base_epoch) return state From 09cefa03f3895c2a319ee03bc06c66a6b8d5e11b Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 16:07:24 -0600 Subject: [PATCH 80/90] remov sync signature todo --- specs/altair/beacon-chain.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 57fec33df..6d97c14b5 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -203,7 +203,6 @@ class BeaconState(Container): ```python class SyncAggregate(Container): sync_committee_bits: Bitvector[SYNC_COMMITTEE_SIZE] - # TODO! Need multiple signatures as discussed between Justin and Vitalik May 3 2021 (see Telegram) sync_committee_signature: BLSSignature ``` From dad698f97aec8f6e80ecc74d093e050b9ae72163 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 12 May 2021 12:35:47 +0800 Subject: [PATCH 81/90] Update unit tests: add `test_compute_subnets_for_sync_committee_slot_period_boundary` --- .../unittests/validator/test_validator.py | 49 +++++++++++++++++-- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index c8a894da6..fd1b82c4b 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -143,20 +143,61 @@ def _subnet_for_sync_committee_index(spec, i): return i // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) +def _get_expected_subnets_by_pubkey(sync_committee_members): + expected_subnets_by_pubkey = defaultdict(list) + for (subnet, pubkey) in sync_committee_members: + expected_subnets_by_pubkey[pubkey].append(subnet) + return expected_subnets_by_pubkey + + @with_altair_and_later @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): + # Transition to the head of the next period + transition_to(spec, state, spec.SLOTS_PER_EPOCH * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + + next_slot_epoch = spec.compute_epoch_at_slot(state.slot + 1) + assert ( + spec.compute_sync_committee_period(spec.get_current_epoch(state)) + == spec.compute_sync_committee_period(next_slot_epoch) + ) some_sync_committee_members = list( ( _subnet_for_sync_committee_index(spec, i), pubkey, ) + # use current_sync_committee for i, pubkey in enumerate(state.current_sync_committee.pubkeys) ) - - expected_subnets_by_pubkey = defaultdict(list) - for (subnet, pubkey) in some_sync_committee_members: - expected_subnets_by_pubkey[pubkey].append(subnet) + expected_subnets_by_pubkey = _get_expected_subnets_by_pubkey(some_sync_committee_members) + + for _, pubkey in some_sync_committee_members: + validator_index = _validator_index_for_pubkey(state, pubkey) + subnets = spec.compute_subnets_for_sync_committee(state, validator_index) + expected_subnets = expected_subnets_by_pubkey[pubkey] + assert subnets == expected_subnets + + +@with_altair_and_later +@with_state +def test_compute_subnets_for_sync_committee_slot_period_boundary(state, spec, phases): + # Transition to the end of the period + transition_to(spec, state, spec.SLOTS_PER_EPOCH * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - 1) + + next_slot_epoch = spec.compute_epoch_at_slot(state.slot + 1) + assert ( + spec.compute_sync_committee_period(spec.get_current_epoch(state)) + != spec.compute_sync_committee_period(next_slot_epoch) + ) + some_sync_committee_members = list( + ( + _subnet_for_sync_committee_index(spec, i), + pubkey, + ) + # use next_sync_committee + for i, pubkey in enumerate(state.next_sync_committee.pubkeys) + ) + expected_subnets_by_pubkey = _get_expected_subnets_by_pubkey(some_sync_committee_members) for _, pubkey in some_sync_committee_members: validator_index = _validator_index_for_pubkey(state, pubkey) From 17820e371165c4f6ee56163e20cff7da36ba1033 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 12 May 2021 13:02:15 +0800 Subject: [PATCH 82/90] Skip the mainnet config slow tests --- .../test/altair/unittests/validator/test_validator.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index fd1b82c4b..cefaaf694 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -8,8 +8,12 @@ from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( with_altair_and_later, + with_configs, with_state, ) +from eth2spec.test.helpers.constants import ( + MINIMAL, +) rng = random.Random(1337) @@ -91,6 +95,7 @@ def _get_sync_committee_signature( @only_with_bls() @with_altair_and_later +@with_configs([MINIMAL], reason="too slow") @with_state def test_process_sync_committee_contributions(phases, spec, state): # skip over slots at genesis @@ -151,6 +156,7 @@ def _get_expected_subnets_by_pubkey(sync_committee_members): @with_altair_and_later +@with_configs([MINIMAL], reason="too slow") @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): # Transition to the head of the next period @@ -179,6 +185,7 @@ def test_compute_subnets_for_sync_committee(state, spec, phases): @with_altair_and_later +@with_configs([MINIMAL], reason="too slow") @with_state def test_compute_subnets_for_sync_committee_slot_period_boundary(state, spec, phases): # Transition to the end of the period From 8e07ece4927bbc3d7965b92b656abd7e09a73fc1 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 12 May 2021 14:04:49 +0800 Subject: [PATCH 83/90] Minor rephrase --- specs/altair/validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 92e3a75c0..67e5914d1 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -271,7 +271,7 @@ If a validator is in the current sync committee (i.e. `is_assigned_to_sync_commi This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. -`get_sync_committee_signature()` assumes `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice (including any empty slots up to the current slot processed with `process_slots` on top of the latest block), `block_root` is the root of the head block, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. +`get_sync_committee_signature(state, block_root, validator_index, privkey)` assumes the parameter `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice (including any empty slots up to the current slot processed with `process_slots` on top of the latest block), `block_root` is the root of the head block, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. ```python def get_sync_committee_signature(state: BeaconState, @@ -291,7 +291,7 @@ def get_sync_committee_signature(state: BeaconState, The validator broadcasts the assembled signature to the assigned subnet, the `sync_committee_{subnet_id}` pubsub topic. The `subnet_id` is derived from the position in the sync committee such that the sync committee is divided into "subcommittees". -`subnet_id` can be computed via `compute_subnets_for_sync_committee()` where `state` is a `BeaconState` during the matching sync committee period. +`subnet_id` can be computed via `compute_subnets_for_sync_committee(state, validator_index)` where `state` is a `BeaconState` during the matching sync committee period. *Note*: This function returns multiple subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. From 82b7a7be3b5cf6046ad046d4c499bb2721931bf2 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 08:29:42 -0600 Subject: [PATCH 84/90] Apply suggestions from code review Co-authored-by: Alex Stokes Co-authored-by: Jacek Sieka --- README.md | 2 +- specs/phase0/p2p-interface.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8a340e9ad..b74102c30 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ The current features are: * [Deposit Contract](specs/phase0/deposit-contract.md) * [Honest Validator](specs/phase0/validator.md) * [P2P Networking](specs/phase0/p2p-interface.md) -* [Weak Subjectivity](specs/phase0/weak-subjectivity.md.md) +* [Weak Subjectivity](specs/phase0/weak-subjectivity.md) ### Altair diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index f2c7d8610..69ff7e120 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -752,7 +752,7 @@ Clients MUST keep a record of signed blocks seen on the epoch range where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blocks on this range. -Peers are *repeatedly* unable to reply to Block requests within the +Peers that are unable to reply to block requests within the `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epoch range MAY get descored or disconnected at any time. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint @@ -763,7 +763,7 @@ proposer signatures and (2) that the blocks form a valid chain up to the most recent block referenced in the weak subjectivity state. *Note*: Although clients that bootstrap from a weak subjectivity checkpoint can begin -participating in the networking immediately, other peers that are actively block syncing MAY +participating in the networking immediately, other peers MAY disconnect and/or temporarily ban such an un-synced or semi-synced client. Clients MUST respond with at least the first block that exists in the range, if they have it, From f52f067b8ea3f8adbebc936207b06459d1956e72 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 08:36:27 -0600 Subject: [PATCH 85/90] add resourceunavailable error code --- specs/phase0/p2p-interface.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 69ff7e120..297d93948 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -567,6 +567,8 @@ The response code can have one of the following values, encoded as a single unsi The response payload adheres to the `ErrorMessage` schema (described below). - 2: **ServerError** -- the responder encountered an error while processing the request. The response payload adheres to the `ErrorMessage` schema (described below). +- 3: **ResourceUnavailable** -- the responder does not have requested resource. + The response payload adheres to the `ErrorMessage` schema (described below). Clients MAY use response codes above `128` to indicate alternative, erroneous request-specific responses. From 6371707779791a687886f6f7c3b2d1ebd45ae345 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 08:40:34 -0600 Subject: [PATCH 86/90] Apply suggestions from code review Co-authored-by: Hsiao-Wei Wang --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 6d97c14b5..9593d13a9 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -371,7 +371,7 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - elif not flag_index == TIMELY_HEAD_FLAG_INDEX: + elif flag_index != TIMELY_HEAD_FLAG_INDEX: penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) return rewards, penalties ``` From a8791f04c7016fdbc6a577826ab193296392e2ad Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 09:44:13 -0600 Subject: [PATCH 87/90] 'get_sync_committee -> get_next_sync_committee --- specs/altair/beacon-chain.md | 36 +++--- specs/altair/fork.md | 10 +- .../test_process_sync_committee.py | 119 ++++++++++-------- .../pyspec/eth2spec/test/helpers/genesis.py | 10 +- 4 files changed, 88 insertions(+), 87 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 39e4a2371..367858340 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -32,8 +32,8 @@ - [`add_flag`](#add_flag) - [`has_flag`](#has_flag) - [Beacon state accessors](#beacon-state-accessors) - - [`get_sync_committee_indices`](#get_sync_committee_indices) - - [`get_sync_committee`](#get_sync_committee) + - [`get_next_sync_committee_indices`](#get_next_sync_committee_indices) + - [`get_next_sync_committee`](#get_next_sync_committee) - [`get_base_reward_per_increment`](#get_base_reward_per_increment) - [`get_base_reward`](#get_base_reward) - [`get_unslashed_participating_indices`](#get_unslashed_participating_indices) @@ -270,15 +270,15 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ### Beacon state accessors -#### `get_sync_committee_indices` +#### `get_next_sync_committee_indices` ```python -def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: +def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorIndex]: """ Return the sequence of sync committee indices (which may include duplicate indices) - for a given ``state`` and ``epoch`` at a sync committee period boundary. + for the next sync committee, given a ``state`` at a sync committee period boundary. """ - assert epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0 + epoch = Epoch(get_current_epoch(state) + 1) MAX_RANDOM_BYTE = 2**8 - 1 active_validator_indices = get_active_validator_indices(state, epoch) @@ -297,25 +297,25 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val return sync_committee_indices ``` -#### `get_sync_committee` +#### `get_next_sync_committee` ```python -def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: +def get_next_sync_committee(state: BeaconState) -> SyncCommittee: """ - Return the *next* sync committee for a given ``state`` and ``epoch``. + Return the *next* sync committee for a given ``state``. ``SyncCommittee`` contains an aggregate pubkey that enables resource-constrained clients to save some computation when verifying the sync committee's signature. - ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_sync_committee_indices`` + ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_next_sync_committee_indices`` returns duplicate indices. Implementations must take care when handling optimizations relating to aggregation and verification in the presence of duplicates. Note: This function should only be called at sync committee period boundaries, as - ``get_sync_committee_indices`` is not stable within a given period. + ``get_next_sync_committee_indices`` is not stable within a given period. """ - indices = get_sync_committee_indices(state, epoch) + indices = get_next_sync_committee_indices(state) pubkeys = [state.validators[index].pubkey for index in indices] aggregate_pubkey = bls.AggregatePKs(pubkeys) return SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=aggregate_pubkey) @@ -688,7 +688,7 @@ def process_sync_committee_updates(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + Epoch(1) if next_epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0: state.current_sync_committee = state.next_sync_committee - state.next_sync_committee = get_sync_committee(state, next_epoch) + state.next_sync_committee = get_next_sync_committee(state) ``` ## Initialize state for pure Altair testnets and test vectors @@ -733,13 +733,9 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, state.genesis_validators_root = hash_tree_root(state.validators) # [New in Altair] Fill in sync committees - current_period = get_current_epoch(state) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_period = current_period - min(1, current_period) - current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - - state.current_sync_committee = get_sync_committee(state, previous_base_epoch) - state.next_sync_committee = get_sync_committee(state, current_base_epoch) + # Note: A duplicate committee is assigned for the current and next committee at genesis + state.current_sync_committee = get_next_sync_committee(state) + state.next_sync_committee = get_next_sync_committee(state) return state ``` diff --git a/specs/altair/fork.md b/specs/altair/fork.md index ea7af898f..2c8b9f855 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -86,12 +86,8 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: ) # Fill in sync committees - current_period = epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_period = current_period - min(1, current_period) - current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - - post.current_sync_committee = get_sync_committee(post, previous_base_epoch) - post.next_sync_committee = get_sync_committee(post, current_base_epoch) + # Note: A duplicate committee is assigned for the current and next committee at the fork boundary + post.current_sync_committee = get_next_sync_committee(post) + post.next_sync_committee = get_next_sync_committee(post) return post ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index d80e650af..e3ee32a93 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -49,9 +49,9 @@ def get_committee_indices(spec, state, duplicates=False): """ state = state.copy() current_epoch = spec.get_current_epoch(state) - randao_index = current_epoch % spec.EPOCHS_PER_HISTORICAL_VECTOR + randao_index = (current_epoch + 1) % spec.EPOCHS_PER_HISTORICAL_VECTOR while True: - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee = spec.get_next_sync_committee_indices(state) if duplicates: if len(committee) != len(set(committee)): return committee @@ -61,23 +61,32 @@ def get_committee_indices(spec, state, duplicates=False): state.randao_mixes[randao_index] = hash(state.randao_mixes[randao_index]) +def compute_committee_indices(spec, state, committee): + """ + Given a ``committee``, calculate and return the related indices + """ + all_pubkeys = [v.pubkey for v in state.validators] + committee_indices = [all_pubkeys.index(pubkey) for pubkey in committee.pubkeys] + return committee_indices + + @with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_missing_participant(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) rng = random.Random(2020) - random_participant = rng.choice(committee) + random_participant = rng.choice(committee_indices) block = build_empty_block_for_next_slot(spec, state) # Exclude one participant whose signature was included. block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[index != random_participant for index in committee], + sync_committee_bits=[index != random_participant for index in committee_indices], sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - committee, # full committee signs + committee_indices, # full committee signs ) ) yield from run_sync_committee_processing(spec, state, block, expect_exception=True) @@ -87,31 +96,38 @@ def test_invalid_signature_missing_participant(spec, state): @spec_state_test @always_bls def test_invalid_signature_extra_participant(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) rng = random.Random(3030) - random_participant = rng.choice(committee) + random_participant = rng.choice(committee_indices) block = build_empty_block_for_next_slot(spec, state) # Exclude one signature even though the block claims the entire committee participated. block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - [index for index in committee if index != random_participant], + [index for index in committee_indices if index != random_participant], ) ) yield from run_sync_committee_processing(spec, state, block, expect_exception=True) -def compute_sync_committee_inclusion_reward(spec, state, participant_index, committee, committee_bits): +def compute_sync_committee_inclusion_reward(spec, + state, + participant_index, + committee_indices, + committee_bits): total_active_increments = spec.get_total_active_balance(state) // spec.EFFECTIVE_BALANCE_INCREMENT total_base_rewards = spec.Gwei(spec.get_base_reward_per_increment(state) * total_active_increments) max_epoch_rewards = spec.Gwei(total_base_rewards * spec.SYNC_REWARD_WEIGHT // spec.WEIGHT_DENOMINATOR) - included_indices = [index for index, bit in zip(committee, committee_bits) if bit] - max_slot_rewards = spec.Gwei(max_epoch_rewards * len(included_indices) // len(committee) // spec.SLOTS_PER_EPOCH) + included_indices = [index for index, bit in zip(committee_indices, committee_bits) if bit] + max_slot_rewards = spec.Gwei( + max_epoch_rewards * len(included_indices) + // len(committee_indices) // spec.SLOTS_PER_EPOCH + ) # Compute the participant and proposer sync rewards committee_effective_balance = sum([state.validators[index].effective_balance for index in included_indices]) @@ -120,23 +136,23 @@ def compute_sync_committee_inclusion_reward(spec, state, participant_index, comm return spec.Gwei(max_slot_rewards * effective_balance // committee_effective_balance) -def compute_sync_committee_participant_reward(spec, state, participant_index, committee, committee_bits): - included_indices = [index for index, bit in zip(committee, committee_bits) if bit] +def compute_sync_committee_participant_reward(spec, state, participant_index, committee_indices, committee_bits): + included_indices = [index for index, bit in zip(committee_indices, committee_bits) if bit] multiplicities = Counter(included_indices) inclusion_reward = compute_sync_committee_inclusion_reward( - spec, state, participant_index, committee, committee_bits, + spec, state, participant_index, committee_indices, committee_bits, ) return spec.Gwei(inclusion_reward * multiplicities[participant_index]) -def compute_sync_committee_proposer_reward(spec, state, committee, committee_bits): +def compute_sync_committee_proposer_reward(spec, state, committee_indices, committee_bits): proposer_reward = 0 - for index, bit in zip(committee, committee_bits): + for index, bit in zip(committee_indices, committee_bits): if not bit: continue inclusion_reward = compute_sync_committee_inclusion_reward( - spec, state, index, committee, committee_bits, + spec, state, index, committee_indices, committee_bits, ) proposer_reward_denominator = ( (spec.WEIGHT_DENOMINATOR - spec.PROPOSER_WEIGHT) @@ -147,15 +163,15 @@ def compute_sync_committee_proposer_reward(spec, state, committee, committee_bit return proposer_reward -def validate_sync_committee_rewards(spec, pre_state, post_state, committee, committee_bits, proposer_index): +def validate_sync_committee_rewards(spec, pre_state, post_state, committee_indices, committee_bits, proposer_index): for index in range(len(post_state.validators)): reward = 0 - if index in committee: + if index in committee_indices: reward += compute_sync_committee_participant_reward( spec, pre_state, index, - committee, + committee_indices, committee_bits, ) @@ -163,14 +179,14 @@ def validate_sync_committee_rewards(spec, pre_state, post_state, committee, comm reward += compute_sync_committee_proposer_reward( spec, pre_state, - committee, + committee_indices, committee_bits, ) assert post_state.balances[index] == pre_state.balances[index] + reward -def run_successful_sync_committee_test(spec, state, committee, committee_bits): +def run_successful_sync_committee_test(spec, state, committee_indices, committee_bits): pre_state = state.copy() block = build_empty_block_for_next_slot(spec, state) @@ -180,7 +196,7 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): spec, state, block.slot - 1, - [index for index, bit in zip(committee, committee_bits) if bit], + [index for index, bit in zip(committee_indices, committee_bits) if bit], ) ) @@ -190,7 +206,7 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): spec, pre_state, state, - committee, + committee_indices, committee_bits, block.proposer_index, ) @@ -200,60 +216,60 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): @with_configs([MINIMAL], reason="to create nonduplicate committee") @spec_state_test def test_sync_committee_rewards_nonduplicate_committee(spec, state): - committee = get_committee_indices(spec, state, duplicates=False) - committee_size = len(committee) + committee_indices = get_committee_indices(spec, state, duplicates=False) + committee_size = len(committee_indices) committee_bits = [True] * committee_size active_validator_count = len(spec.get_active_validator_indices(state, spec.get_current_epoch(state))) # Preconditions of this test case assert active_validator_count >= spec.SYNC_COMMITTEE_SIZE - assert committee_size == len(set(committee)) + assert committee_size == len(set(committee_indices)) - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @with_configs([MAINNET], reason="to create duplicate committee") @spec_state_test def test_sync_committee_rewards_duplicate_committee(spec, state): - committee = get_committee_indices(spec, state, duplicates=True) - committee_size = len(committee) + committee_indices = get_committee_indices(spec, state, duplicates=True) + committee_size = len(committee_indices) committee_bits = [True] * committee_size active_validator_count = len(spec.get_active_validator_indices(state, spec.get_current_epoch(state))) # Preconditions of this test case assert active_validator_count < spec.SYNC_COMMITTEE_SIZE - assert committee_size > len(set(committee)) + assert committee_size > len(set(committee_indices)) - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_not_full_participants(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) rng = random.Random(1010) - committee_bits = [rng.choice([True, False]) for _ in committee] + committee_bits = [rng.choice([True, False]) for _ in committee_indices] - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_empty_participants(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) - committee_bits = [False for _ in committee] + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) + committee_bits = [False for _ in committee_indices] - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_past_block(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) blocks = [] for _ in range(2): @@ -261,12 +277,12 @@ def test_invalid_signature_past_block(spec, state): block = build_empty_block_for_next_slot(spec, state) # Valid sync committee signature here... block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - committee, + committee_indices, ) ) @@ -276,12 +292,12 @@ def test_invalid_signature_past_block(spec, state): invalid_block = build_empty_block_for_next_slot(spec, state) # Invalid signature from a slot other than the previous invalid_block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, invalid_block.slot - 2, - committee, + committee_indices, ) ) @@ -306,19 +322,18 @@ def test_invalid_signature_previous_committee(spec, state): transition_to(spec, state, slot_in_future_sync_committee_period) # Use the previous sync committee to produce the signature. - pubkeys = [validator.pubkey for validator in state.validators] # Ensure that the pubkey sets are different. assert set(old_sync_committee.pubkeys) != set(state.current_sync_committee.pubkeys) - committee = [pubkeys.index(pubkey) for pubkey in old_sync_committee.pubkeys] + committee_indices = compute_committee_indices(spec, state, old_sync_committee) block = build_empty_block_for_next_slot(spec, state) block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - committee, + committee_indices, ) ) @@ -345,14 +360,12 @@ def test_valid_signature_future_committee(spec, state): sync_committee = state.current_sync_committee next_sync_committee = state.next_sync_committee - expected_next_sync_committee = spec.get_sync_committee(state, epoch_in_future_sync_committee_period) - assert next_sync_committee == expected_next_sync_committee + assert next_sync_committee != sync_committee assert sync_committee != old_current_sync_committee assert sync_committee != old_next_sync_committee - pubkeys = [validator.pubkey for validator in state.validators] - committee_indices = [pubkeys.index(pubkey) for pubkey in sync_committee.pubkeys] + committee_indices = compute_committee_indices(spec, state, sync_committee) block = build_empty_block_for_next_slot(spec, state) block.body.sync_aggregate = spec.SyncAggregate( diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 23cd04c93..c57442766 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -69,12 +69,8 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if spec.fork not in FORKS_BEFORE_ALTAIR: # Fill in sync committees - current_period = spec.get_current_epoch(state) // spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_period = current_period - min(1, current_period) - current_base_epoch = current_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_base_epoch = previous_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - - state.current_sync_committee = spec.get_sync_committee(state, previous_base_epoch) - state.next_sync_committee = spec.get_sync_committee(state, current_base_epoch) + # Note: A duplicate committee is assigned for the current and next committee at genesis + state.current_sync_committee = spec.get_next_sync_committee(state) + state.next_sync_committee = spec.get_next_sync_committee(state) return state From 133875a6d6ce5a0e085e310cca3ca254db8d0dd1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 09:56:13 -0600 Subject: [PATCH 88/90] fix sync_committe_update tests --- .../epoch_processing/test_process_sync_committee_updates.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index ca25797c4..6af16287c 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -39,8 +39,7 @@ def run_sync_committees_progress_test(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` - current_epoch = spec.get_current_epoch(state) - third_sync_committee = spec.get_sync_committee(state, current_epoch + 1) + third_sync_committee = spec.get_next_sync_committee(state) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee From 4286f85a68a7b2abdca249cd02482616840e5f86 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 12 May 2021 11:05:41 -0700 Subject: [PATCH 89/90] Update validator guide with restricted sync committee computation --- specs/altair/validator.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 67e5914d1..4aee98ad4 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -177,7 +177,6 @@ At any given `epoch`, the `BeaconState` contains the current `SyncCommittee` and Once every `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs, the next `SyncCommittee` becomes the current `SyncCommittee` and the next committee is computed and stored. *Note*: The data required to compute a given committee is not cached in the `BeaconState` after committees are calculated at the period boundaries. -This means that calling `get_sync_commitee()` in a given `epoch` can return a different result than what was computed during the relevant epoch transition. For this reason, *always* get committee assignments via the fields of the `BeaconState` (`current_sync_committee` and `next_sync_committee`) or use the above reference code. A validator should plan for future sync committee assignments by noting which sync committee periods they are selected for participation. @@ -229,12 +228,12 @@ def process_sync_committee_contributions(block: BeaconBlock, contributions: Set[SyncCommitteeContribution]) -> None: sync_aggregate = SyncAggregate() signatures = [] + sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT for contribution in contributions: subcommittee_index = contribution.subcommittee_index for index, participated in enumerate(contribution.aggregation_bits): if participated: - sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT participant_index = sync_subcommittee_size * subcommittee_index + index sync_aggregate.sync_committee_bits[participant_index] = True signatures.append(contribution.signature) @@ -367,7 +366,7 @@ Set `contribution.subcommittee_index` to the index for the subcommittee index co ###### Aggregation bits Let `contribution.aggregation_bits` be a `Bitvector[SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT]`, where the `index`th bit is set in the `Bitvector` for each corresponding validator included in this aggregate from the corresponding subcommittee. -An aggregator finds the index in the sync committee (as returned by `get_sync_committee_indices()`) for a given validator referenced by `sync_committee_signature.validator_index` and maps the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is set in `contribution.aggegration_bits`. +An aggregator finds the index in the sync committee (as determined by a reverse pubkey lookup on `state.current_sync_committee.pubkeys`) for a given validator referenced by `sync_committee_signature.validator_index` and maps the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is set in `contribution.aggegration_bits`. For example, if a validator with index `2044` is pseudo-randomly sampled to sync committee index `135`. This sync committee index maps to `subcommittee_index` `1` with position `7` in the `Bitvector` for the contribution. From 5188671816b3a846b1fa5ea781d3321f0e2c5f04 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 12:18:35 -0600 Subject: [PATCH 90/90] Update specs/altair/beacon-chain.md Co-authored-by: dankrad --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9593d13a9..492a146ea 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -262,7 +262,7 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ Return the sequence of sync committee indices for a given ``state`` and ``epoch``. - Note: Committee can contain duplicate indices for small validator sets (< 2 * SYNC_COMMITTEE_SIZE) + Note: Committee can contain duplicate indices for small validator sets (< SYNC_COMMITTEE_SIZE + 128) Note: This function is not stable during a sync committee period as a validator's effective balance may change enough to affect the sampling.