From 17221c806501e11943fd8359754f79b65713468d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 12 Nov 2020 10:26:25 -0700 Subject: [PATCH 001/227] more clearly define min epoch range for blocksbyrange requests --- README.md | 1 + specs/phase0/p2p-interface.md | 41 ++++++++++++++++++++++++++++--- specs/phase0/weak-subjectivity.md | 5 +++- 3 files changed, 42 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index a3ca7c586..8104ec7cc 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ Core specifications for Eth2 clients be found in [specs](specs/). These are divi * [Deposit Contract](specs/phase0/deposit-contract.md) * [Honest Validator](specs/phase0/validator.md) * [P2P Networking](specs/phase0/p2p-interface.md) +* [Weak Subjectivity](specs/phase0/weak-subjectivity.md.md) ### Phase 1 * [From Phase 0 to Phase 1](specs/phase1/phase1-fork.md) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 5dc892991..8104b54c7 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -93,6 +93,7 @@ It consists of four main sections: - [Why is it called Req/Resp and not RPC?](#why-is-it-called-reqresp-and-not-rpc) - [Why do we allow empty responses in block requests?](#why-do-we-allow-empty-responses-in-block-requests) - [Why does `BeaconBlocksByRange` let the server choose which branch to send blocks from?](#why-does-beaconblocksbyrange-let-the-server-choose-which-branch-to-send-blocks-from) + - [Why are `BlocksByRange` requests only required to be served for the latest `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs?](#why-are-blocksbyrange-requests-only-required-to-be-served-for-the-latest-min_epochs_for_block_requests-epochs) - [What's the effect of empty slots on the sync algorithm?](#whats-the-effect-of-empty-slots-on-the-sync-algorithm) - [Discovery](#discovery) - [Why are we using discv5 and not libp2p Kademlia DHT?](#why-are-we-using-discv5-and-not-libp2p-kademlia-dht) @@ -171,6 +172,7 @@ This section outlines constants that are used in this spec. |---|---|---| | `GOSSIP_MAX_SIZE` | `2**20` (= 1048576, 1 MiB) | The maximum allowed size of uncompressed gossip messages. | | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request | +| `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks | | `MAX_CHUNK_SIZE` | `2**20` (1048576, 1 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. | | `TTFB_TIMEOUT` | `5s` | The maximum time to wait for first byte of request response (time-to-first-byte). | | `RESP_TIMEOUT` | `10s` | The maximum time for complete response transfer. | @@ -179,7 +181,6 @@ This section outlines constants that are used in this spec. | `MESSAGE_DOMAIN_INVALID_SNAPPY` | `0x00000000` | 4-byte domain for gossip message-id isolation of *invalid* snappy messages | | `MESSAGE_DOMAIN_VALID_SNAPPY` | `0x01000000` | 4-byte domain for gossip message-id isolation of *valid* snappy messages | - ## MetaData Clients MUST locally store the following `MetaData`: @@ -746,10 +747,17 @@ The request MUST be encoded as an SSZ-container. The response MUST consist of zero or more `response_chunk`. Each _successful_ `response_chunk` MUST contain a single `SignedBeaconBlock` payload. -Clients MUST keep a record of signed blocks seen since the start of the weak subjectivity period -and MUST support serving requests of blocks up to their own `head_block_root`. +Clients MUST keep a record of signed blocks seen on the epoch range +`[max(GENESIS_EPOCH, current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS), current_epoch]` +where `current_epoch` is defined by the current wall-clock time, +and clients MUST support serving requests of blocks on this range. -Clients MUST respond with at least the first block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOCKS` blocks. +*Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint +MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` +to be compliant with `BlocksByRange` requests. + +Clients MUST respond with at least the first block that exists in the range, if they have it, +and no more than `MAX_REQUEST_BLOCKS` blocks. The following blocks, where they exist, MUST be sent in consecutive order. @@ -1394,6 +1402,31 @@ To avoid this race condition, we allow the responding side to choose which branc The requesting client then goes on to validate the blocks and incorporate them in their own database -- because they follow the same rules, they should at this point arrive at the same canonical chain. +### Why are `BlocksByRange` requests only required to be served for the latest `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs? + +Due to economic finality and weak subjectivity requirements of a proof-of-stake blockchain, for a new node to safely join the network +the node must provide a recent checkpoint found out-of-band. This checkpoint can be in the form of a `root` & `epoch` or it can be the entire +beacon state and then a simple block sync from there to the head. We expect the latter to be the dominant UX strategy. + +These checkpoints *in the worst case* (i.e. very large validator set and maximal allowed safety decay) must be from the +most recent `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, and thus a user must be able to block sync to the head from this starting point. +Thus, this defines the epoch range outside which nodes may prune blocks, and +the epoch range that a new node syncing from a checkpoint must backfill. + +`MIN_EPOCHS_FOR_BLOCK_REQUESTS` is calculated using the arithmetic from `compute_weak_subjectivity_period` found in the +[weak subjectivity guide](./weak-subjectivity.md). Specifically to find this max epoch range, we use the worst case event of a very large validator size +(`>= MIN_PER_EPOCH_CHURN_LIMIT * CHURN_LIMIT_QUOTIENT`). + +```python +MIN_EPOCHS_FOR_BLOCK_REQUESTS = ( + MIN_VALIDATOR_WITHDRAWABILITY_DELAY + + MAX_SAFETY_DECAY * CHURN_LIMIT_QUOTIENT // (2 * 100) +) +``` + +Where `MAX_SAFETY_DECAY = 100` and thus `MIN_EPOCHS_FOR_BLOCK_REQUESTS = 33024` (~5 months). + + ### What's the effect of empty slots on the sync algorithm? When syncing one can only tell that a slot has been skipped on a particular branch diff --git a/specs/phase0/weak-subjectivity.md b/specs/phase0/weak-subjectivity.md index b4d78cb12..797f972a1 100644 --- a/specs/phase0/weak-subjectivity.md +++ b/specs/phase0/weak-subjectivity.md @@ -94,7 +94,9 @@ A brief reference for what these values look like in practice: ## Weak Subjectivity Sync -Clients should allow users to input a Weak Subjectivity Checkpoint at startup, and guarantee that any successful sync leads to the given Weak Subjectivity Checkpoint along the canonical chain. If such a sync is not possible, the client should treat this as a critical and irrecoverable failure. +Clients should allow users to input a Weak Subjectivity Checkpoint at startup, +and guarantee that any successful sync leads to the given Weak Subjectivity Checkpoint along the canonical chain. +If such a sync is not possible, the client should treat this as a critical and irrecoverable failure. ### Weak Subjectivity Sync Procedure @@ -112,6 +114,7 @@ Clients should allow users to input a Weak Subjectivity Checkpoint at startup, a Emit descriptive critical error if this assert fails, then exit client process. ### Checking for Stale Weak Subjectivity Checkpoint + Clients may choose to validate that the input Weak Subjectivity Checkpoint is not stale at the time of startup. To support this mechanism, the client needs to take the state at the Weak Subjectivity Checkpoint as a CLI parameter input (or fetch the state associated with the input Weak Subjectivity Checkpoint from some source). From 56aafbe533c6f5ed7881a65b22c41c1d7367c50e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 9 Dec 2020 12:37:56 -0700 Subject: [PATCH 002/227] add note about signature check when backfilling beaconblocks --- specs/phase0/p2p-interface.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 8104b54c7..df17258ab 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -94,6 +94,7 @@ It consists of four main sections: - [Why do we allow empty responses in block requests?](#why-do-we-allow-empty-responses-in-block-requests) - [Why does `BeaconBlocksByRange` let the server choose which branch to send blocks from?](#why-does-beaconblocksbyrange-let-the-server-choose-which-branch-to-send-blocks-from) - [Why are `BlocksByRange` requests only required to be served for the latest `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs?](#why-are-blocksbyrange-requests-only-required-to-be-served-for-the-latest-min_epochs_for_block_requests-epochs) + - [Why must the proposer signature be checked when backfilling blocks in the database?](#why-must-the-proposer-signature-be-checked-when-backfilling-blocks-in-the-database) - [What's the effect of empty slots on the sync algorithm?](#whats-the-effect-of-empty-slots-on-the-sync-algorithm) - [Discovery](#discovery) - [Why are we using discv5 and not libp2p Kademlia DHT?](#why-are-we-using-discv5-and-not-libp2p-kademlia-dht) @@ -754,7 +755,10 @@ and clients MUST support serving requests of blocks on this range. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` -to be compliant with `BlocksByRange` requests. +to be compliant with `BlocksByRange` requests. To safely perform such a +backfill of blocks to the recent state, the node MUST validate both (1) the +proposer signatures and (2) that the blocks form a valid chain up to the most +recent block referenced in the weak subjectivity state. Clients MUST respond with at least the first block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOCKS` blocks. @@ -1426,6 +1430,20 @@ MIN_EPOCHS_FOR_BLOCK_REQUESTS = ( Where `MAX_SAFETY_DECAY = 100` and thus `MIN_EPOCHS_FOR_BLOCK_REQUESTS = 33024` (~5 months). +### Why must the proposer signature be checked when backfilling blocks in the database? + +When backfilling blocks in a database from a know safe block/state (e.g. when starting from a weak subjectivity state), +the node not only must ensure the `BeaconBlock`s form a chain to the known safe block, +but also must check that the proposer signature is valid in the `SignedBeaconBlock` wrapper. + +This is because the signature is not part of the `BeaconBlock` hash chain, and +thus could be corrupted by an attacker serving valid `BeaconBlock`s but invalid +signatures contained in `SignedBeaconBlock`. + +Although in this particular use case this does not represent a decay in safety +(due to the assumptions of starting at a weak subjectivity checkpoint), it +would represent invalid historic data and could be unwittingly transmitted to +additional nodes. ### What's the effect of empty slots on the sync algorithm? From 2ad8fdb818f58778391d93540892d35570f33237 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 13 Jan 2021 18:03:40 -0700 Subject: [PATCH 003/227] add ability for node to randomly request and descore if not serving blocks on WS period --- specs/phase0/p2p-interface.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index df17258ab..4a412ac19 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -753,6 +753,11 @@ Clients MUST keep a record of signed blocks seen on the epoch range where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blocks on this range. +Synced clients unable to reply to Block requests within the +`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epoch range MAY get descored or disconnected at any time. +Note, due to this it is risky behaviour to begin participating as a full node at the head if having +not yet backfilled on this range. + *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` to be compliant with `BlocksByRange` requests. To safely perform such a From 9eb662786adebce7c3e41852a525458895e8bedc Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Thu, 1 Apr 2021 06:29:01 -0700 Subject: [PATCH 004/227] Add consistency checks in on_block tests --- .../test/phase0/unittests/fork_choice/test_on_block.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index 7fa32b86c..bb5aec578 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -234,6 +234,9 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): just_block.slot = spec.compute_start_slot_at_epoch(store.justified_checkpoint.epoch) store.blocks[just_block.hash_tree_root()] = just_block + just_block_parent = store.blocks[just_block.parent_root] + assert just_block_parent.slot < just_block.slot, f"just_block_parent.slot: {just_block_parent.slot}, just_block.slot: {just_block.slot}" + # Step time past safe slots spec.on_tick(store, store.time + spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED * spec.SECONDS_PER_SLOT) assert spec.get_current_slot(store) % spec.SLOTS_PER_EPOCH >= spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED @@ -284,6 +287,9 @@ def test_on_block_outside_safe_slots_but_finality(spec, state): just_block.slot = spec.compute_start_slot_at_epoch(store.justified_checkpoint.epoch) store.blocks[just_block.hash_tree_root()] = just_block + just_block_parent = store.blocks[just_block.parent_root] + assert just_block_parent.slot < just_block.slot, f"just_block_parent.slot: {just_block_parent.slot}, just_block.slot: {just_block.slot}" + # Step time past safe slots spec.on_tick(store, store.time + spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED * spec.SECONDS_PER_SLOT) assert spec.get_current_slot(store) % spec.SLOTS_PER_EPOCH >= spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED From 2478e1526a464dd9d93dbce88d98452452ec01dc Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Sat, 3 Apr 2021 16:50:43 -0700 Subject: [PATCH 005/227] Fix block hierarchy consistency in tests --- .../unittests/fork_choice/test_on_block.py | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index bb5aec578..79c55b2eb 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -224,14 +224,19 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): next_epoch(spec, state) spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) state, store, last_signed_block = apply_next_epoch_with_attestations(spec, state, store) + last_block_root = hash_tree_root(last_signed_block.message) + + # Mock ficticious justified checkpoint in store + store.justified_checkpoint = spec.Checkpoint( + epoch=spec.compute_epoch_at_slot(last_signed_block.message.slot), + root=spec.Root("0x4a55535449464945440000000000000000000000000000000000000000000000") + ) + next_epoch(spec, state) spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) - last_block_root = hash_tree_root(last_signed_block.message) - # Mock justified block in store + # Create new higher justified checkpoint not in branch of store's justified checkpoint just_block = build_empty_block_for_next_slot(spec, state) - # Slot is same as justified checkpoint so does not trigger an override in the store - just_block.slot = spec.compute_start_slot_at_epoch(store.justified_checkpoint.epoch) store.blocks[just_block.hash_tree_root()] = just_block just_block_parent = store.blocks[just_block.parent_root] @@ -271,20 +276,25 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): def test_on_block_outside_safe_slots_but_finality(spec, state): # Initialization store = get_genesis_forkchoice_store(spec, state) - time = 100 + time = 0 spec.on_tick(store, time) next_epoch(spec, state) spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) state, store, last_signed_block = apply_next_epoch_with_attestations(spec, state, store) + last_block_root = hash_tree_root(last_signed_block.message) + + # Mock ficticious justified checkpoint in store + store.justified_checkpoint = spec.Checkpoint( + epoch=spec.compute_epoch_at_slot(last_signed_block.message.slot), + root=spec.Root("0x4a55535449464945440000000000000000000000000000000000000000000000") + ) + next_epoch(spec, state) spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) - last_block_root = hash_tree_root(last_signed_block.message) - # Mock justified block in store + # Create new higher justified checkpoint not in branch of store's justified checkpoint just_block = build_empty_block_for_next_slot(spec, state) - # Slot is same as justified checkpoint so does not trigger an override in the store - just_block.slot = spec.compute_start_slot_at_epoch(store.justified_checkpoint.epoch) store.blocks[just_block.hash_tree_root()] = just_block just_block_parent = store.blocks[just_block.parent_root] From 21b878364a615ad0a5938ae566a416b10e98393d Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Sat, 3 Apr 2021 17:09:08 -0700 Subject: [PATCH 006/227] Fix checkpoint hierarchy consistency in tests --- .../test/phase0/unittests/fork_choice/test_on_block.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index 79c55b2eb..536241bf8 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -307,13 +307,15 @@ def test_on_block_outside_safe_slots_but_finality(spec, state): # Mock justified and finalized update in state just_fin_state = store.block_states[last_block_root] new_justified = spec.Checkpoint( - epoch=store.justified_checkpoint.epoch + 1, + epoch=spec.compute_epoch_at_slot(just_block.slot) + 1, root=just_block.hash_tree_root(), ) + assert new_justified.epoch > store.justified_checkpoint.epoch new_finalized = spec.Checkpoint( - epoch=store.finalized_checkpoint.epoch + 1, + epoch=spec.compute_epoch_at_slot(just_block.slot), root=just_block.parent_root, ) + assert new_finalized.epoch > store.finalized_checkpoint.epoch just_fin_state.current_justified_checkpoint = new_justified just_fin_state.finalized_checkpoint = new_finalized From 6e6afac86f819d2237cb349c8ee51af3a4503c9f Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Sat, 3 Apr 2021 17:17:43 -0700 Subject: [PATCH 007/227] Remove unnecessary asserts --- .../test/phase0/unittests/fork_choice/test_on_block.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index 536241bf8..00ea5f172 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -239,9 +239,6 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): just_block = build_empty_block_for_next_slot(spec, state) store.blocks[just_block.hash_tree_root()] = just_block - just_block_parent = store.blocks[just_block.parent_root] - assert just_block_parent.slot < just_block.slot, f"just_block_parent.slot: {just_block_parent.slot}, just_block.slot: {just_block.slot}" - # Step time past safe slots spec.on_tick(store, store.time + spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED * spec.SECONDS_PER_SLOT) assert spec.get_current_slot(store) % spec.SLOTS_PER_EPOCH >= spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED @@ -297,9 +294,6 @@ def test_on_block_outside_safe_slots_but_finality(spec, state): just_block = build_empty_block_for_next_slot(spec, state) store.blocks[just_block.hash_tree_root()] = just_block - just_block_parent = store.blocks[just_block.parent_root] - assert just_block_parent.slot < just_block.slot, f"just_block_parent.slot: {just_block_parent.slot}, just_block.slot: {just_block.slot}" - # Step time past safe slots spec.on_tick(store, store.time + spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED * spec.SECONDS_PER_SLOT) assert spec.get_current_slot(store) % spec.SLOTS_PER_EPOCH >= spec.SAFE_SLOTS_TO_UPDATE_JUSTIFIED From 9ec252e6f2244e6f0722335869250df6bf87f64d Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Sat, 3 Apr 2021 17:22:33 -0700 Subject: [PATCH 008/227] Fix linter --- .../test/phase0/unittests/fork_choice/test_on_block.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index 00ea5f172..3c1802ca2 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -225,13 +225,13 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) state, store, last_signed_block = apply_next_epoch_with_attestations(spec, state, store) last_block_root = hash_tree_root(last_signed_block.message) - + # Mock ficticious justified checkpoint in store store.justified_checkpoint = spec.Checkpoint( epoch=spec.compute_epoch_at_slot(last_signed_block.message.slot), root=spec.Root("0x4a55535449464945440000000000000000000000000000000000000000000000") ) - + next_epoch(spec, state) spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) @@ -280,13 +280,13 @@ def test_on_block_outside_safe_slots_but_finality(spec, state): spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) state, store, last_signed_block = apply_next_epoch_with_attestations(spec, state, store) last_block_root = hash_tree_root(last_signed_block.message) - + # Mock ficticious justified checkpoint in store store.justified_checkpoint = spec.Checkpoint( epoch=spec.compute_epoch_at_slot(last_signed_block.message.slot), root=spec.Root("0x4a55535449464945440000000000000000000000000000000000000000000000") ) - + next_epoch(spec, state) spec.on_tick(store, store.time + state.slot * spec.SECONDS_PER_SLOT) From 5194dd123cdb52c1616cd11a0eeee39ed7330f27 Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Sat, 3 Apr 2021 17:26:12 -0700 Subject: [PATCH 009/227] TIL fictitious --- .../test/phase0/unittests/fork_choice/test_on_block.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index 3c1802ca2..0cc8c33c9 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -226,7 +226,7 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): state, store, last_signed_block = apply_next_epoch_with_attestations(spec, state, store) last_block_root = hash_tree_root(last_signed_block.message) - # Mock ficticious justified checkpoint in store + # Mock fictitious justified checkpoint in store store.justified_checkpoint = spec.Checkpoint( epoch=spec.compute_epoch_at_slot(last_signed_block.message.slot), root=spec.Root("0x4a55535449464945440000000000000000000000000000000000000000000000") @@ -281,7 +281,7 @@ def test_on_block_outside_safe_slots_but_finality(spec, state): state, store, last_signed_block = apply_next_epoch_with_attestations(spec, state, store) last_block_root = hash_tree_root(last_signed_block.message) - # Mock ficticious justified checkpoint in store + # Mock fictitious justified checkpoint in store store.justified_checkpoint = spec.Checkpoint( epoch=spec.compute_epoch_at_slot(last_signed_block.message.slot), root=spec.Root("0x4a55535449464945440000000000000000000000000000000000000000000000") From 9e5ee0a083c9df9c4643e2220e7de26dbe5c6cd5 Mon Sep 17 00:00:00 2001 From: Aditya Asgaonkar Date: Sat, 3 Apr 2021 17:43:18 -0700 Subject: [PATCH 010/227] Remove unnecesssary change --- .../eth2spec/test/phase0/unittests/fork_choice/test_on_block.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py index 0cc8c33c9..48ea78e39 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_block.py @@ -273,7 +273,7 @@ def test_on_block_outside_safe_slots_and_multiple_better_justified(spec, state): def test_on_block_outside_safe_slots_but_finality(spec, state): # Initialization store = get_genesis_forkchoice_store(spec, state) - time = 0 + time = 100 spec.on_tick(store, time) next_epoch(spec, state) From 08f9f81c1f738acaa7ca1aa6d373c0a4e11d9c77 Mon Sep 17 00:00:00 2001 From: Adrian Sutton Date: Wed, 7 Apr 2021 11:44:24 +1000 Subject: [PATCH 011/227] Update note about changes to slash_validator. --- specs/altair/beacon-chain.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 72d511b04..9b476bce1 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -391,7 +391,8 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S #### Modified `slash_validator` -*Note*: The function `slash_validator` is modified to use `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR`. +*Note*: The function `slash_validator` is modified to use `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` +and use `PROPOSER_WEIGHT` when calculating the proposer reward. ```python def slash_validator(state: BeaconState, From 6e8b4b3ea96ad8add6dc605a20cba0f51e7cd79d Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 5 Apr 2021 22:59:09 +0800 Subject: [PATCH 012/227] Add eth2spec.merge.spec --- Makefile | 3 +- setup.py | 94 ++++++++++++++++++- specs/merge/beacon-chain.md | 11 ++- tests/core/pyspec/eth2spec/test/context.py | 7 ++ .../pyspec/eth2spec/utils/ssz/ssz_typing.py | 3 + 5 files changed, 110 insertions(+), 8 deletions(-) diff --git a/Makefile b/Makefile index 4ecb3e2ed..8f053e7a7 100644 --- a/Makefile +++ b/Makefile @@ -55,6 +55,7 @@ partial_clean: rm -rf $(DEPOSIT_CONTRACT_TESTER_DIR)/.pytest_cache rm -rf $(PY_SPEC_DIR)/phase0 rm -rf $(PY_SPEC_DIR)/altair + rm -rf $(PY_SPEC_DIR)/merge rm -rf $(PY_SPEC_DIR)/$(COV_HTML_OUT) rm -rf $(PY_SPEC_DIR)/.coverage rm -rf $(PY_SPEC_DIR)/test-reports @@ -119,7 +120,7 @@ codespell: lint: pyspec . venv/bin/activate; cd $(PY_SPEC_DIR); \ flake8 --config $(LINTER_CONFIG_FILE) ./eth2spec \ - && mypy --config-file $(LINTER_CONFIG_FILE) -p eth2spec.phase0 -p eth2spec.altair + && mypy --config-file $(LINTER_CONFIG_FILE) -p eth2spec.phase0 -p eth2spec.altair -p eth2spec.merge lint_generators: pyspec . venv/bin/activate; cd $(TEST_GENERATORS_DIR); \ diff --git a/setup.py b/setup.py index 6d9147661..762e282eb 100644 --- a/setup.py +++ b/setup.py @@ -13,7 +13,7 @@ FUNCTION_REGEX = r'^def [\w_]*' # Definitions in context.py PHASE0 = 'phase0' ALTAIR = 'altair' - +MERGE = 'merge' class SpecObject(NamedTuple): functions: Dict[str, str] @@ -99,7 +99,7 @@ def get_spec(file_name: str) -> SpecObject: ssz_dep_constants[row[0]] = row[1] else: constants[row[0]] = row[1].replace('**TBD**', '2**32') - elif row[1].startswith('uint') or row[1].startswith('Bytes'): + elif row[1].startswith('uint') or row[1].startswith('Bytes') or row[1].startswith('ByteList'): custom_types[row[0]] = row[1] return SpecObject( functions=functions, @@ -176,6 +176,34 @@ SSZObject = TypeVar('SSZObject', bound=View) CONFIG_NAME = 'mainnet' ''' +MERGE_IMPORTS = '''from eth2spec.phase0 import spec as phase0 +from eth2spec.config.config_util import apply_constants_config +from typing import ( + Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar +) + +from dataclasses import ( + dataclass, + field, +) + +from lru import LRU + +from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes +from eth2spec.utils.ssz.ssz_typing import ( + View, boolean, Container, List, Vector, uint8, uint32, uint64, uint256, + Bytes1, Bytes4, Bytes20, Bytes32, Bytes48, Bytes96, Bitlist, + ByteList, ByteVector +) +from eth2spec.utils import bls + +from eth2spec.utils.hash_function import hash + +SSZObject = TypeVar('SSZObject', bound=View) + +CONFIG_NAME = 'mainnet' +''' + SUNDRY_CONSTANTS_FUNCTIONS = ''' def ceillog2(x: int) -> uint64: if x < 1: @@ -269,6 +297,30 @@ def get_generalized_index(ssz_class: Any, *path: Sequence[Union[int, SSZVariable return GeneralizedIndex(ssz_path.gindex())''' +MERGE_SUNDRY_FUNCTIONS = """ +ApplicationState = Any + + +def get_pow_block(hash: Bytes32) -> PowBlock: + pass + + +def get_application_state(application_state_root: Bytes32) -> ApplicationState: + pass + + +def get_pow_chain_head() -> PowBlock: + pass + + +def application_state_transition(application_state: ApplicationState, application_payload: ApplicationPayload) -> None: + pass + + +def produce_application_payload(parent_hash: Bytes32) -> ApplicationPayload: + pass""" + + # The constants that depend on SSZ objects # Will verify the value at the end of the spec ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS = { @@ -283,6 +335,11 @@ assert ( ) == WEIGHT_DENOMINATOR''' +MERGE_HARDCODED_CUSTOM_TYPE_DEP_CONSTANTS = { + 'MAX_BYTES_PER_OPAQUE_TRANSACTION': 'uint64(2**20)', +} + + def is_phase0(fork): return fork == PHASE0 @@ -291,6 +348,10 @@ def is_altair(fork): return fork == ALTAIR +def is_merge(fork): + return fork == MERGE + + def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_class_objects: Dict[str, str]) -> str: """ Given all the objects that constitute a spec, combine them into a single pyfile. @@ -300,6 +361,15 @@ def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_cl [ f"class {key}({value}):\n pass\n" for key, value in spec_object.custom_types.items() + if not value.startswith('ByteList') + ] + ) + + ('\n\n' if len([key for key, value in spec_object.custom_types.items() if value.startswith('ByteList')]) > 0 else '') + + '\n\n'.join( + [ + f"{key} = {value}\n" + for key, value in spec_object.custom_types.items() + if value.startswith('ByteList') ] ) ) @@ -316,9 +386,15 @@ def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_cl if is_altair(fork): altair_ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS[x]), ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS)) + if is_merge(fork): + merge_custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, MERGE_HARDCODED_CUSTOM_TYPE_DEP_CONSTANTS[x]), MERGE_HARDCODED_CUSTOM_TYPE_DEP_CONSTANTS)) + + spec = ( imports + '\n\n' + f"fork = \'{fork}\'\n" + # The constants that some SSZ containers require. Need to be defined before `new_type_definitions` + + ('\n\n' + merge_custom_type_dep_constants + '\n' if is_merge(fork) else '') + '\n\n' + new_type_definitions + '\n' + SUNDRY_CONSTANTS_FUNCTIONS # The constants that some SSZ containers require. Need to be defined before `constants_spec` @@ -330,6 +406,7 @@ def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_cl # Functions to make pyspec work + '\n' + PHASE0_SUNDRY_FUNCTIONS + ('\n' + ALTAIR_SUNDRY_FUNCTIONS if is_altair(fork) else '') + + ('\n' + MERGE_SUNDRY_FUNCTIONS if is_merge(fork) else '') ) # Since some constants are hardcoded in setup.py, the following assertions verify that the hardcoded constants are @@ -357,7 +434,7 @@ def combine_constants(old_constants: Dict[str, str], new_constants: Dict[str, st ignored_dependencies = [ 'bit', 'boolean', 'Vector', 'List', 'Container', 'BLSPubkey', 'BLSSignature', - 'Bytes1', 'Bytes4', 'Bytes32', 'Bytes48', 'Bytes96', 'Bitlist', 'Bitvector', + 'Bytes1', 'Bytes4', 'Bytes20', 'Bytes32', 'Bytes48', 'Bytes96', 'Bitlist', 'Bitvector', 'uint8', 'uint16', 'uint32', 'uint64', 'uint128', 'uint256', 'bytes', 'byte', 'ByteList', 'ByteVector', 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', @@ -422,6 +499,7 @@ def combine_spec_objects(spec0: SpecObject, spec1: SpecObject) -> SpecObject: fork_imports = { 'phase0': PHASE0_IMPORTS, 'altair': ALTAIR_IMPORTS, + 'merge': MERGE_IMPORTS, } @@ -485,6 +563,16 @@ class PySpecCommand(Command): specs/altair/validator.md specs/altair/sync-protocol.md """ + elif is_merge(self.spec_fork): + self.md_doc_paths = """ + specs/phase0/beacon-chain.md + specs/phase0/fork-choice.md + specs/phase0/validator.md + specs/phase0/weak-subjectivity.md + specs/merge/beacon-chain.md + specs/merge/fork-choice.md + specs/merge/validator.md + """ else: raise Exception('no markdown files specified, and spec fork "%s" is unknown', self.spec_fork) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 86bb0bc95..53fbb6730 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -145,7 +145,9 @@ def is_transition_completed(state: BeaconState) -> boolean: ```python def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> boolean: - return state.latest_application_block_header.block_hash == Bytes32() and block_body.application_payload.block_hash != Bytes32() + is_empty_latest_application_block_header = state.latest_application_block_header.block_hash == Bytes32() + is_empty_application_payload_block_hash = block_body.application_payload.block_hash == Bytes32() + return is_empty_latest_application_block_header and not is_empty_application_payload_block_hash ``` ### Block processing @@ -182,14 +184,15 @@ def process_application_payload(state: BeaconState, body: BeaconBlockBody) -> No """ Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ + application_payload = body.application_payload if not is_transition_completed(state): - assert body.application_payload == ApplicationPayload() + assert application_payload == ApplicationPayload() return if not is_transition_block(state, body): - assert body.application_payload.parent_hash == state.latest_application_block_header.block_hash - assert body.application_payload.number == state.latest_application_block_header.number + 1 + assert application_payload.parent_hash == state.latest_application_block_header.block_hash + assert application_payload.number == state.latest_application_block_header.number + 1 application_state = get_application_state(state.latest_application_block_header.state_root) application_state_transition(application_state, body.application_payload) diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 02968a266..7ae1b9541 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -2,6 +2,7 @@ import pytest from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair +from eth2spec.merge import spec as spec_merge from eth2spec.utils import bls from .exceptions import SkippedTest @@ -19,6 +20,7 @@ from importlib import reload def reload_specs(): reload(spec_phase0) reload(spec_altair) + reload(spec_merge) # Some of the Spec module functionality is exposed here to deal with phase-specific changes. @@ -61,9 +63,14 @@ class SpecAltair(Spec): ... +class SpecMerge(Spec): + ... + + class SpecForks(TypedDict, total=False): PHASE0: SpecPhase0 ALTAIR: SpecAltair + MERGE: SpecMerge def _prepare_state(balances_fn: Callable[[Any], Sequence[int]], threshold_fn: Callable[[Any], int], diff --git a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py index 6626e26d6..9b18f8bda 100644 --- a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py +++ b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py @@ -6,3 +6,6 @@ from remerkleable.basic import boolean, bit, uint, byte, uint8, uint16, uint32, from remerkleable.bitfields import Bitvector, Bitlist from remerkleable.byte_arrays import ByteVector, Bytes1, Bytes4, Bytes8, Bytes32, Bytes48, Bytes96, ByteList from remerkleable.core import BasicView, View, Path + + +Bytes20 = ByteVector[20] # type: ignore From c1101a81813c470c6768e000f0fd34f07bfecbe9 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Thu, 8 Apr 2021 14:29:05 +0600 Subject: [PATCH 013/227] Apply new terminology to the merge spec --- specs/merge/beacon-chain.md | 98 ++++++++++++++++++------------------- specs/merge/fork-choice.md | 2 +- specs/merge/validator.md | 24 ++++----- 3 files changed, 62 insertions(+), 62 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 86bb0bc95..a170e4ed0 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -20,17 +20,17 @@ - [`BeaconBlockBody`](#beaconblockbody) - [`BeaconState`](#beaconstate) - [New containers](#new-containers) - - [`ApplicationPayload`](#applicationpayload) - - [`ApplicationBlockHeader`](#applicationblockheader) + - [`ExecutionPayload`](#executionpayload) + - [`ExecutionPayloadHeader`](#executionpayloadheader) - [Helper functions](#helper-functions) - [Misc](#misc) - [`is_transition_completed`](#is_transition_completed) - [`is_transition_block`](#is_transition_block) - [Block processing](#block-processing) - - [Application payload processing](#application-payload-processing) - - [`get_application_state`](#get_application_state) - - [`application_state_transition`](#application_state_transition) - - [`process_application_payload`](#process_application_payload) + - [Execution payload processing](#execution-payload-processing) + - [`get_execution_state`](#get_execution_state) + - [`execution_state_transition`](#execution_state_transition) + - [`process_execution_payload`](#process_execution_payload) @@ -38,7 +38,7 @@ ## Introduction This is a patch implementing the executable beacon chain proposal. -It enshrines application-layer execution and validity as a first class citizen at the core of the beacon chain. +It enshrines transaction execution and validity as a first class citizen at the core of the beacon chain. ## Custom types @@ -73,32 +73,32 @@ order and append any additional fields to the end. #### `BeaconBlockBody` -*Note*: `BeaconBlockBody` fields remain unchanged other than the addition of `application_payload`. +*Note*: `BeaconBlockBody` fields remain unchanged other than the addition of `execution_payload`. ```python class BeaconBlockBody(phase0.BeaconBlockBody): - application_payload: ApplicationPayload # [New in Merge] application payload + execution_payload: ExecutionPayload # [New in Merge] ``` #### `BeaconState` -*Note*: `BeaconState` fields remain unchanged other than addition of `latest_application_block_header`. +*Note*: `BeaconState` fields remain unchanged other than addition of `latest_execution_payload_header`. ```python class BeaconState(phase0.BeaconState): - # Application-layer - latest_application_block_header: ApplicationBlockHeader # [New in Merge] + # Execution-layer + latest_execution_payload_header: ExecutionPayloadHeader # [New in Merge] ``` ### New containers -#### `ApplicationPayload` +#### `ExecutionPayload` -The application payload included in a `BeaconBlockBody`. +The execution payload included in a `BeaconBlockBody`. ```python -class ApplicationPayload(Container): - block_hash: Bytes32 # Hash of application block +class ExecutionPayload(Container): + block_hash: Bytes32 # Hash of execution block parent_hash: Bytes32 coinbase: Bytes20 state_root: Bytes32 @@ -110,15 +110,15 @@ class ApplicationPayload(Container): transactions: List[OpaqueTransaction, MAX_APPLICATION_TRANSACTIONS] ``` -#### `ApplicationBlockHeader` +#### `ExecutionPayloadHeader` -The application block header included in a `BeaconState`. +The execution payload header included in a `BeaconState`. -*Note:* Holds application payload data without transaction list. +*Note:* Holds execution payload data without transaction bodies. ```python -class ApplicationBlockHeader(Container): - block_hash: Bytes32 # Hash of application block +class ExecutionPayloadHeader(Container): + block_hash: Bytes32 # Hash of execution block parent_hash: Bytes32 coinbase: Bytes20 state_root: Bytes32 @@ -138,14 +138,14 @@ class ApplicationBlockHeader(Container): ```python def is_transition_completed(state: BeaconState) -> boolean: - return state.latest_application_block_header.block_hash != Bytes32() + return state.latest_execution_payload_header.block_hash != Bytes32() ``` #### `is_transition_block` ```python def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> boolean: - return state.latest_application_block_header.block_hash == Bytes32() and block_body.application_payload.block_hash != Bytes32() + return state.latest_execution_payload_header.block_hash == Bytes32() and block_body.execution_payload.block_hash != Bytes32() ``` ### Block processing @@ -156,54 +156,54 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) - process_application_payload(state, block.body) # [New in Merge] + process_execution_payload(state, block.body) # [New in Merge] ``` -#### Application payload processing +#### Execution payload processing -##### `get_application_state` +##### `get_execution_state` -*Note*: `ApplicationState` class is an abstract class representing ethereum application state. +*Note*: `ExecutionState` class is an abstract class representing ethereum execution state. -Let `get_application_state(application_state_root: Bytes32) -> ApplicationState` be the function that given the root hash returns a copy of ethereum application state. +Let `get_execution_state(execution_state_root: Bytes32) -> ExecutionState` be the function that given the root hash returns a copy of ethereum execution state. The body of the function is implementation dependent. -##### `application_state_transition` +##### `execution_state_transition` -Let `application_state_transition(application_state: ApplicationState, application_payload: ApplicationPayload) -> None` be the transition function of ethereum application state. +Let `execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload) -> None` be the transition function of ethereum execution state. The body of the function is implementation dependent. -*Note*: `application_state_transition` must throw `AssertionError` if either the transition itself or one of the post-transition verifications has failed. +*Note*: `execution_state_transition` must throw `AssertionError` if either the transition itself or one of the pre or post conditions has failed. -##### `process_application_payload` +##### `process_execution_payload` ```python -def process_application_payload(state: BeaconState, body: BeaconBlockBody) -> None: +def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None: """ Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ if not is_transition_completed(state): - assert body.application_payload == ApplicationPayload() + assert body.execution_payload == ExecutionPayload() return if not is_transition_block(state, body): - assert body.application_payload.parent_hash == state.latest_application_block_header.block_hash - assert body.application_payload.number == state.latest_application_block_header.number + 1 + assert body.execution_payload.parent_hash == state.latest_execution_payload_header.block_hash + assert body.execution_payload.number == state.latest_execution_payload_header.number + 1 - application_state = get_application_state(state.latest_application_block_header.state_root) - application_state_transition(application_state, body.application_payload) + execution_state = get_execution_state(state.latest_execution_payload_header.state_root) + execution_state_transition(execution_state, body.execution_payload) - state.latest_application_block_header = ApplicationBlockHeader( - block_hash=application_payload.block_hash, - parent_hash=application_payload.parent_hash, - coinbase=application_payload.coinbase, - state_root=application_payload.state_root, - number=application_payload.number, - gas_limit=application_payload.gas_limit, - gas_used=application_payload.gas_used, - receipt_root=application_payload.receipt_root, - logs_bloom=application_payload.logs_bloom, - transactions_root=hash_tree_root(application_payload.transactions), + state.latest_execution_payload_header = ExecutionPayloadHeader( + block_hash=execution_payload.block_hash, + parent_hash=execution_payload.parent_hash, + coinbase=execution_payload.coinbase, + state_root=execution_payload.state_root, + number=execution_payload.number, + gas_limit=execution_payload.gas_limit, + gas_used=execution_payload.gas_used, + receipt_root=execution_payload.receipt_root, + logs_bloom=execution_payload.logs_bloom, + transactions_root=hash_tree_root(execution_payload.transactions), ) ``` diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index 8425358bf..647a663c2 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -77,7 +77,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: # [New in Merge] if is_transition_block(pre_state, block.body): # Delay consideration of block until PoW block is processed by the PoW node - pow_block = get_pow_block(block.body.application_payload.parent_hash) + pow_block = get_pow_block(block.body.execution_payload.parent_hash) assert pow_block.is_processed assert is_valid_transition_block(pow_block) diff --git a/specs/merge/validator.md b/specs/merge/validator.md index 1c43c9dd6..5b26a21dd 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -15,9 +15,9 @@ - [Beacon chain responsibilities](#beacon-chain-responsibilities) - [Block proposal](#block-proposal) - [Constructing the `BeaconBlockBody`](#constructing-the-beaconblockbody) - - [Application Payload](#application-payload) + - [Execution Payload](#execution-payload) - [`get_pow_chain_head`](#get_pow_chain_head) - - [`produce_application_payload`](#produce_application_payload) + - [`produce_execution_payload`](#produce_execution_payload) @@ -34,37 +34,37 @@ All terminology, constants, functions, and protocol mechanics defined in the upd ## Beacon chain responsibilities -All validator responsibilities remain unchanged other than those noted below. Namely, the transition block handling and the addition of `ApplicationPayload`. +All validator responsibilities remain unchanged other than those noted below. Namely, the transition block handling and the addition of `ExecutionPayload`. ### Block proposal #### Constructing the `BeaconBlockBody` -##### Application Payload +##### Execution Payload ###### `get_pow_chain_head` Let `get_pow_chain_head() -> PowBlock` be the function that returns the head of the PoW chain. The body of the function is implementation specific. -###### `produce_application_payload` +###### `produce_execution_payload` -Let `produce_application_payload(parent_hash: Bytes32) -> ApplicationPayload` be the function that produces new instance of application payload. +Let `produce_execution_payload(parent_hash: Bytes32) -> ExecutionPayload` be the function that produces new instance of execution payload. The body of this function is implementation dependent. -* Set `block.body.application_payload = get_application_payload(state)` where: +* Set `block.body.execution_payload = get_execution_payload(state)` where: ```python -def get_application_payload(state: BeaconState) -> ApplicationPayload: +def get_execution_payload(state: BeaconState) -> ExecutionPayload: if not is_transition_completed(state): pow_block = get_pow_chain_head() if not is_valid_transition_block(pow_block): # Pre-merge, empty payload - return ApplicationPayload() + return ExecutionPayload() else: # Signify merge via producing on top of the last PoW block - return produce_application_payload(pow_block.block_hash) + return produce_execution_payload(pow_block.block_hash) # Post-merge, normal payload - application_parent_hash = state.latest_application_block_header.block_hash - return produce_application_payload(application_parent_hash) + execution_parent_hash = state.latest_execution_payload_header.block_hash + return produce_execution_payload(execution_parent_hash) ``` From 13586cd9f88b1e58573eebf0028cd3c54991f667 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Thu, 8 Apr 2021 20:14:01 +0600 Subject: [PATCH 014/227] Polishing and fixes according to github comments --- specs/merge/beacon-chain.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index a170e4ed0..4e05c04ec 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -138,14 +138,14 @@ class ExecutionPayloadHeader(Container): ```python def is_transition_completed(state: BeaconState) -> boolean: - return state.latest_execution_payload_header.block_hash != Bytes32() + return state.latest_execution_payload_header != ExecutionPayloadHeader() ``` #### `is_transition_block` ```python def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> boolean: - return state.latest_execution_payload_header.block_hash == Bytes32() and block_body.execution_payload.block_hash != Bytes32() + return not is_transition_completed(state) and block_body.execution_payload != ExecutionPayload() ``` ### Block processing @@ -163,14 +163,14 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: ##### `get_execution_state` -*Note*: `ExecutionState` class is an abstract class representing ethereum execution state. +*Note*: `ExecutionState` class is an abstract class representing Ethereum execution state. -Let `get_execution_state(execution_state_root: Bytes32) -> ExecutionState` be the function that given the root hash returns a copy of ethereum execution state. +Let `get_execution_state(execution_state_root: Bytes32) -> ExecutionState` be the function that given the root hash returns a copy of Ethereum execution state. The body of the function is implementation dependent. ##### `execution_state_transition` -Let `execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload) -> None` be the transition function of ethereum execution state. +Let `execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload) -> None` be the transition function of Ethereum execution state. The body of the function is implementation dependent. *Note*: `execution_state_transition` must throw `AssertionError` if either the transition itself or one of the pre or post conditions has failed. @@ -183,16 +183,18 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ + execution_payload = body.execution_payload + if not is_transition_completed(state): - assert body.execution_payload == ExecutionPayload() + assert execution_payload == ExecutionPayload() return if not is_transition_block(state, body): - assert body.execution_payload.parent_hash == state.latest_execution_payload_header.block_hash - assert body.execution_payload.number == state.latest_execution_payload_header.number + 1 + assert execution_payload.parent_hash == state.latest_execution_payload_header.block_hash + assert execution_payload.number == state.latest_execution_payload_header.number + 1 execution_state = get_execution_state(state.latest_execution_payload_header.state_root) - execution_state_transition(execution_state, body.execution_payload) + execution_state_transition(execution_state, execution_payload) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, From 68b11ad4ad1c175e8f5e682707734427f18ff8fd Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Thu, 8 Apr 2021 20:14:34 +0600 Subject: [PATCH 015/227] Fix pre-condition checks in process_execution_payload --- specs/merge/beacon-chain.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 4e05c04ec..26ce63a08 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -185,11 +185,7 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None execution_payload = body.execution_payload - if not is_transition_completed(state): - assert execution_payload == ExecutionPayload() - return - - if not is_transition_block(state, body): + if is_transition_completed(state): assert execution_payload.parent_hash == state.latest_execution_payload_header.block_hash assert execution_payload.number == state.latest_execution_payload_header.number + 1 From 5e43ad69cfd2a51f0a08459761ca82570dfa58d5 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 5 Apr 2021 18:31:05 -0700 Subject: [PATCH 016/227] Add some basic tests for Altair validator guide --- .../test/altair/validator/__init__.py | 0 .../test/altair/validator/test_validator.py | 155 ++++++++++++++++++ 2 files changed, 155 insertions(+) create mode 100644 tests/core/pyspec/eth2spec/test/altair/validator/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py diff --git a/tests/core/pyspec/eth2spec/test/altair/validator/__init__.py b/tests/core/pyspec/eth2spec/test/altair/validator/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py new file mode 100644 index 000000000..da7b7a131 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py @@ -0,0 +1,155 @@ +import random +from eth2spec.utils.ssz.ssz_typing import Bitvector +from eth2spec.test.helpers.block import build_empty_block +from eth2spec.test.helpers.keys import pubkey_to_privkey +from eth2spec.test.helpers.state import transition_to +from eth2spec.utils import bls +from eth2spec.utils.bls import only_with_bls +from eth2spec.test.context import ( + PHASE0, + with_all_phases_except, + with_state, +) + +rng = random.Random(1337) + + +def ensure_assignments_in_sync_committee( + spec, state, epoch, sync_committee, active_pubkeys +): + assert len(sync_committee.pubkeys) >= 3 + some_pubkeys = rng.sample(sync_committee.pubkeys, 3) + for pubkey in some_pubkeys: + validator_index = active_pubkeys.index(pubkey) + assert spec.is_assigned_to_sync_committee(state, epoch, validator_index) + + +@with_all_phases_except([PHASE0]) +@with_state +def test_is_assigned_to_sync_committee(phases, spec, state): + epoch = spec.get_current_epoch(state) + validator_indices = spec.get_active_validator_indices(state, epoch) + validator_count = len(validator_indices) + + query_epoch = epoch + 1 + next_query_epoch = query_epoch + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + active_pubkeys = [state.validators[index].pubkey for index in validator_indices] + + ensure_assignments_in_sync_committee( + spec, state, query_epoch, state.current_sync_committee, active_pubkeys + ) + ensure_assignments_in_sync_committee( + spec, state, next_query_epoch, state.next_sync_committee, active_pubkeys + ) + + assert validator_count >= 3 + sync_committee_pubkeys = set( + list(state.current_sync_committee.pubkeys) + + list(state.next_sync_committee.pubkeys) + ) + disqualified_pubkeys = list( + filter(lambda key: key not in sync_committee_pubkeys, active_pubkeys) + ) + some_pubkeys = rng.sample(disqualified_pubkeys, 3) + for pubkey in some_pubkeys: + validator_index = active_pubkeys.index(pubkey) + is_current = spec.is_assigned_to_sync_committee( + state, query_epoch, validator_index + ) + is_next = spec.is_assigned_to_sync_committee( + state, next_query_epoch, validator_index + ) + is_current_or_next = is_current or is_next + assert not is_current_or_next + + +def _get_sync_committee_signature( + spec, + state, + target_slot, + target_block_root, + subcommittee_index, + index_in_subcommittee, +): + subcommittee_size = spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT + sync_committee_index = ( + subcommittee_index * subcommittee_size + index_in_subcommittee + ) + pubkey = state.current_sync_committee.pubkeys[sync_committee_index] + privkey = pubkey_to_privkey[pubkey] + + domain = spec.get_domain( + state, + spec.DOMAIN_SYNC_COMMITTEE, + ) + signing_data = spec.compute_signing_root(target_block_root, domain) + return bls.Sign(privkey, spec.hash_tree_root(signing_data)) + + +@only_with_bls() +@with_all_phases_except([PHASE0]) +@with_state +def test_process_sync_committee_contributions(phases, spec, state): + # skip over slots at genesis + transition_to(spec, state, state.slot + 3) + + # build a block and attempt to assemble a sync aggregate + # from some sync committee contributions + block = build_empty_block(spec, state) + previous_slot = state.slot - 1 + target_block_root = spec.get_block_root_at_slot(state, previous_slot) + aggregation_bits = Bitvector[ + spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT + ]() + aggregation_index = 0 + aggregation_bits[aggregation_index] = True + + contributions = [ + spec.SyncCommitteeContribution( + slot=block.slot, + beacon_block_root=target_block_root, + subcommittee_index=i, + aggregation_bits=aggregation_bits, + signature=_get_sync_committee_signature( + spec, state, previous_slot, target_block_root, i, aggregation_index + ), + ) + for i in range(spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) + ] + + # ensure the block has an empty sync aggregate... + empty_sync_aggregate = spec.SyncAggregate() + empty_sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY + assert block.body.sync_aggregate == empty_sync_aggregate + spec.process_sync_committee_contributions(block, set(contributions)) + + # and that after processing, it is no longer empty + assert len(block.body.sync_aggregate.sync_committee_bits) != 0 + assert ( + block.body.sync_aggregate.sync_committee_signature != spec.G2_POINT_AT_INFINITY + ) + # moreover, ensure the sync aggregate is valid if the block is accepted + spec.process_block(state, block) + + +@with_all_phases_except([PHASE0]) +@with_state +def test_compute_subnets_for_sync_committee(state, spec, phases): + k = max(spec.SYNC_COMMITTEE_SIZE // 4, 1) + some_sync_committee_members = rng.sample( + list( + (i, pubkey) for i, pubkey in enumerate(state.current_sync_committee.pubkeys) + ), + k, + ) + validator_indices = [ + list(map(lambda v: v.pubkey, state.validators)).index(pubkey) + for pubkey in map(lambda t: t[1], some_sync_committee_members) + ] + expected_subnets = [ + index // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) + for index in map(lambda t: t[0], some_sync_committee_members) + ] + for validator_index, expected_subnet in zip(validator_indices, expected_subnets): + subnet = spec.compute_subnets_for_sync_committee(state, validator_index) + assert subnet == [expected_subnet] From 4678ffc7942e3e7a4ce13632de858cbdd2778036 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 8 Apr 2021 10:50:21 -0700 Subject: [PATCH 017/227] Update tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py Co-authored-by: Hsiao-Wei Wang --- .../pyspec/eth2spec/test/altair/validator/test_validator.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py index da7b7a131..673184b18 100644 --- a/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py @@ -151,5 +151,5 @@ def test_compute_subnets_for_sync_committee(state, spec, phases): for index in map(lambda t: t[0], some_sync_committee_members) ] for validator_index, expected_subnet in zip(validator_indices, expected_subnets): - subnet = spec.compute_subnets_for_sync_committee(state, validator_index) - assert subnet == [expected_subnet] + subnets = spec.compute_subnets_for_sync_committee(state, validator_index) + assert subnets == [expected_subnet] From 749b49898a15bb5d895f33700643a5f0b17897e2 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 8 Apr 2021 11:22:59 -0700 Subject: [PATCH 018/227] file reorg --- .../eth2spec/test/altair/{ => unittests}/validator/__init__.py | 0 .../test/altair/{ => unittests}/validator/test_validator.py | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename tests/core/pyspec/eth2spec/test/altair/{ => unittests}/validator/__init__.py (100%) rename tests/core/pyspec/eth2spec/test/altair/{ => unittests}/validator/test_validator.py (100%) diff --git a/tests/core/pyspec/eth2spec/test/altair/validator/__init__.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/__init__.py similarity index 100% rename from tests/core/pyspec/eth2spec/test/altair/validator/__init__.py rename to tests/core/pyspec/eth2spec/test/altair/unittests/validator/__init__.py diff --git a/tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py similarity index 100% rename from tests/core/pyspec/eth2spec/test/altair/validator/test_validator.py rename to tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py From 3fd49744304f193436694bd3df2a7adeb7d75c5c Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 8 Apr 2021 17:44:10 -0700 Subject: [PATCH 019/227] iterate over the correct number of subcommittees --- .../eth2spec/test/altair/unittests/validator/test_validator.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index 673184b18..016234541 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -114,7 +114,7 @@ def test_process_sync_committee_contributions(phases, spec, state): spec, state, previous_slot, target_block_root, i, aggregation_index ), ) - for i in range(spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) + for i in range(spec.SYNC_COMMITTEE_SUBNET_COUNT) ] # ensure the block has an empty sync aggregate... From e4e65295b6db09ce1b299b0c8fe91974c6af5c75 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 8 Apr 2021 17:57:43 -0700 Subject: [PATCH 020/227] modify assignment test when sync committee size >= validator count --- .../unittests/validator/test_validator.py | 29 ++++++++++--------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index 016234541..bb26dff1c 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -42,25 +42,28 @@ def test_is_assigned_to_sync_committee(phases, spec, state): spec, state, next_query_epoch, state.next_sync_committee, active_pubkeys ) - assert validator_count >= 3 sync_committee_pubkeys = set( list(state.current_sync_committee.pubkeys) + list(state.next_sync_committee.pubkeys) ) - disqualified_pubkeys = list( + disqualified_pubkeys = set( filter(lambda key: key not in sync_committee_pubkeys, active_pubkeys) ) - some_pubkeys = rng.sample(disqualified_pubkeys, 3) - for pubkey in some_pubkeys: - validator_index = active_pubkeys.index(pubkey) - is_current = spec.is_assigned_to_sync_committee( - state, query_epoch, validator_index - ) - is_next = spec.is_assigned_to_sync_committee( - state, next_query_epoch, validator_index - ) - is_current_or_next = is_current or is_next - assert not is_current_or_next + # NOTE: only check `disqualified_pubkeys` if SYNC_COMMITEE_SIZE < validator count + if disqualified_pubkeys: + sample_size = 3 + assert validator_count >= sample_size + some_pubkeys = rng.sample(disqualified_pubkeys, sample_size) + for pubkey in some_pubkeys: + validator_index = active_pubkeys.index(pubkey) + is_current = spec.is_assigned_to_sync_committee( + state, query_epoch, validator_index + ) + is_next = spec.is_assigned_to_sync_committee( + state, next_query_epoch, validator_index + ) + is_current_or_next = is_current or is_next + assert not is_current_or_next def _get_sync_committee_signature( From 66905f4fa621c8204e5948d0277ce20aaad199e0 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 8 Apr 2021 18:25:58 -0700 Subject: [PATCH 021/227] update test to work for both minimal and mainnet config --- .../unittests/validator/test_validator.py | 40 +++++++++++-------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index bb26dff1c..244e1aa7e 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -1,4 +1,5 @@ import random +from collections import defaultdict from eth2spec.utils.ssz.ssz_typing import Bitvector from eth2spec.test.helpers.block import build_empty_block from eth2spec.test.helpers.keys import pubkey_to_privkey @@ -135,24 +136,31 @@ def test_process_sync_committee_contributions(phases, spec, state): spec.process_block(state, block) +def _validator_index_for_pubkey(state, pubkey): + return list(map(lambda v: v.pubkey, state.validators)).index(pubkey) + + +def _subnet_for_sync_committee_index(spec, i): + return i // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) + + @with_all_phases_except([PHASE0]) @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): - k = max(spec.SYNC_COMMITTEE_SIZE // 4, 1) - some_sync_committee_members = rng.sample( - list( - (i, pubkey) for i, pubkey in enumerate(state.current_sync_committee.pubkeys) - ), - k, + some_sync_committee_members = list( + ( + _subnet_for_sync_committee_index(spec, i), + pubkey, + ) + for i, pubkey in enumerate(state.current_sync_committee.pubkeys) ) - validator_indices = [ - list(map(lambda v: v.pubkey, state.validators)).index(pubkey) - for pubkey in map(lambda t: t[1], some_sync_committee_members) - ] - expected_subnets = [ - index // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) - for index in map(lambda t: t[0], some_sync_committee_members) - ] - for validator_index, expected_subnet in zip(validator_indices, expected_subnets): + + expected_subnets_by_pubkey = defaultdict(list) + for (subnet, pubkey) in some_sync_committee_members: + expected_subnets_by_pubkey[pubkey].append(subnet) + + for _, pubkey in some_sync_committee_members: + validator_index = _validator_index_for_pubkey(state, pubkey) subnets = spec.compute_subnets_for_sync_committee(state, validator_index) - assert subnets == [expected_subnet] + expected_subnets = expected_subnets_by_pubkey[pubkey] + assert subnets == expected_subnets From ffe7d6db6a4d15a355ef9b5d6072d440c2786fac Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Fri, 9 Apr 2021 15:36:45 +0600 Subject: [PATCH 022/227] Add pre-merge path to the process_execution_payload --- specs/merge/beacon-chain.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 26ce63a08..fbdab682e 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -183,6 +183,10 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ + # Pre-merge, skip processing + if not is_transition_completed(state) and not is_transition_block(state, body): + return + execution_payload = body.execution_payload if is_transition_completed(state): From a1ded22b3ab4fa08bf45c12fb945f9adfc5b37c8 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Thu, 8 Apr 2021 14:48:03 +0600 Subject: [PATCH 023/227] Introduce Hash32 custom type --- specs/merge/beacon-chain.md | 9 +++++---- specs/merge/fork-choice.md | 4 ++-- specs/merge/validator.md | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index be79cf5b5..a93076001 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -47,6 +47,7 @@ We define the following Python custom types for type hinting and readability: | Name | SSZ equivalent | Description | | - | - | - | | `OpaqueTransaction` | `ByteList[MAX_BYTES_PER_OPAQUE_TRANSACTION]` | a byte-list containing a single [typed transaction envelope](https://eips.ethereum.org/EIPS/eip-2718#opaque-byte-array-rather-than-an-rlp-array) structured as `TransactionType \|\| TransactionPayload` | +| `Hash32` | `Bytes32` | a 256-bit hash | ## Constants @@ -98,8 +99,8 @@ The execution payload included in a `BeaconBlockBody`. ```python class ExecutionPayload(Container): - block_hash: Bytes32 # Hash of execution block - parent_hash: Bytes32 + block_hash: Hash32 # Hash of execution block + parent_hash: Hash32 coinbase: Bytes20 state_root: Bytes32 number: uint64 @@ -118,8 +119,8 @@ The execution payload header included in a `BeaconState`. ```python class ExecutionPayloadHeader(Container): - block_hash: Bytes32 # Hash of execution block - parent_hash: Bytes32 + block_hash: Hash32 # Hash of execution block + parent_hash: Hash32 coinbase: Bytes20 state_root: Bytes32 number: uint64 diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index 647a663c2..df45bedd1 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -30,7 +30,7 @@ This is the modification of the fork choice according to the executable beacon c ```python class PowBlock(Container): - block_hash: Bytes32 + block_hash: Hash32 is_processed: boolean is_valid: boolean total_difficulty: uint256 @@ -38,7 +38,7 @@ class PowBlock(Container): #### `get_pow_block` -Let `get_pow_block(hash: Bytes32) -> PowBlock` be the function that given the hash of the PoW block returns its data. +Let `get_pow_block(hash: Hash32) -> PowBlock` be the function that given the hash of the PoW block returns its data. *Note*: The `eth_getBlockByHash` JSON-RPC method does not distinguish invalid blocks from blocks that haven't been processed yet. Either extending this existing method or implementing a new one is required. diff --git a/specs/merge/validator.md b/specs/merge/validator.md index 5b26a21dd..42fe7adcf 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -48,7 +48,7 @@ Let `get_pow_chain_head() -> PowBlock` be the function that returns the head of ###### `produce_execution_payload` -Let `produce_execution_payload(parent_hash: Bytes32) -> ExecutionPayload` be the function that produces new instance of execution payload. +Let `produce_execution_payload(parent_hash: Hash32) -> ExecutionPayload` be the function that produces new instance of execution payload. The body of this function is implementation dependent. * Set `block.body.execution_payload = get_execution_payload(state)` where: From 658ede2191e7da57a37a9c12f5c21cc4bc4c7020 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 9 Apr 2021 20:34:51 +0800 Subject: [PATCH 024/227] Refactor pyspec builder with `SpecAdjustment` classes --- setup.py | 339 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 206 insertions(+), 133 deletions(-) diff --git a/setup.py b/setup.py index 514f75c50..65b04fcf2 100644 --- a/setup.py +++ b/setup.py @@ -6,15 +6,35 @@ from distutils.util import convert_path import os import re from typing import Dict, NamedTuple, List +from abc import ABC, abstractmethod + FUNCTION_REGEX = r'^def [\w_]*' - # Definitions in context.py PHASE0 = 'phase0' ALTAIR = 'altair' MERGE = 'merge' +CONFIG_LOADER = ''' +apply_constants_config(globals()) +''' + +# The helper functions that are used when defining constants +CONSTANT_DEP_SUNDRY_CONSTANTS_FUNCTIONS = ''' +def ceillog2(x: int) -> uint64: + if x < 1: + raise ValueError(f"ceillog2 accepts only positive values, x={x}") + return uint64((x - 1).bit_length()) + + +def floorlog2(x: int) -> uint64: + if x < 1: + raise ValueError(f"floorlog2 accepts only positive values, x={x}") + return uint64(x.bit_length() - 1) +''' + + class SpecObject(NamedTuple): functions: Dict[str, str] custom_types: Dict[str, str] @@ -111,11 +131,55 @@ def get_spec(file_name: str) -> SpecObject: ) -CONFIG_LOADER = ''' -apply_constants_config(globals()) -''' +class SpecAdjustment(ABC): + @classmethod + @abstractmethod + def imports_and_predefinitions(cls) -> str: + """ + Importing functions and defining special types/constants for building pyspec. + """ + raise NotImplementedError() -PHASE0_IMPORTS = '''from eth2spec.config.config_util import apply_constants_config + @classmethod + @abstractmethod + def sundry_functions(cls) -> str: + """ + The functions that are (1) defined abstractly in specs or (2) adjusted for getting better performance. + """ + raise NotImplementedError() + + @classmethod + @abstractmethod + def hardcoded_ssz_dep_constants(cls) -> Dict[str, str]: + """ + The constants that are required for SSZ objects. + """ + raise NotImplementedError() + + @classmethod + @abstractmethod + def hardcoded_custom_type_dep_constants(cls) -> Dict[str, str]: + """ + The constants that are required for custom types. + """ + raise NotImplementedError() + + @classmethod + @abstractmethod + def invariant_checks(cls) -> str: + """ + The invariant checks + """ + raise NotImplementedError() + + +# +# Phase0SpecAdjustment +# +class Phase0SpecAdjustment(SpecAdjustment): + @classmethod + def imports_and_predefinitions(cls) -> str: + return '''from eth2spec.config.config_util import apply_constants_config from typing import ( Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar ) @@ -141,82 +205,9 @@ SSZObject = TypeVar('SSZObject', bound=View) CONFIG_NAME = 'mainnet' ''' -ALTAIR_IMPORTS = '''from eth2spec.phase0 import spec as phase0 -from eth2spec.config.config_util import apply_constants_config -from typing import ( - Any, Dict, Set, Sequence, NewType, Tuple, TypeVar, Callable, Optional, Union -) - -from dataclasses import ( - dataclass, - field, -) - -from lru import LRU - -from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes -from eth2spec.utils.ssz.ssz_typing import ( - View, boolean, Container, List, Vector, uint8, uint32, uint64, - Bytes1, Bytes4, Bytes32, Bytes48, Bytes96, Bitlist, Bitvector, - Path, -) -from eth2spec.utils import bls - -from eth2spec.utils.hash_function import hash - -# Whenever altair is loaded, make sure we have the latest phase0 -from importlib import reload -reload(phase0) - - -SSZVariableName = str -GeneralizedIndex = NewType('GeneralizedIndex', int) -SSZObject = TypeVar('SSZObject', bound=View) - -CONFIG_NAME = 'mainnet' -''' - -MERGE_IMPORTS = '''from eth2spec.phase0 import spec as phase0 -from eth2spec.config.config_util import apply_constants_config -from typing import ( - Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar -) - -from dataclasses import ( - dataclass, - field, -) - -from lru import LRU - -from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes -from eth2spec.utils.ssz.ssz_typing import ( - View, boolean, Container, List, Vector, uint8, uint32, uint64, uint256, - Bytes1, Bytes4, Bytes20, Bytes32, Bytes48, Bytes96, Bitlist, - ByteList, ByteVector -) -from eth2spec.utils import bls - -from eth2spec.utils.hash_function import hash - -SSZObject = TypeVar('SSZObject', bound=View) - -CONFIG_NAME = 'mainnet' -''' - -SUNDRY_CONSTANTS_FUNCTIONS = ''' -def ceillog2(x: int) -> uint64: - if x < 1: - raise ValueError(f"ceillog2 accepts only positive values, x={x}") - return uint64((x - 1).bit_length()) - - -def floorlog2(x: int) -> uint64: - if x < 1: - raise ValueError(f"floorlog2 accepts only positive values, x={x}") - return uint64(x.bit_length() - 1) -''' -PHASE0_SUNDRY_FUNCTIONS = ''' + @classmethod + def sundry_functions(cls) -> str: + return ''' def get_eth1_data(block: Eth1Block) -> Eth1Data: """ A stub function return mocking Eth1Data. @@ -287,9 +278,62 @@ get_attesting_indices = cache_this( ), _get_attesting_indices, lru_size=SLOTS_PER_EPOCH * MAX_COMMITTEES_PER_SLOT * 3)''' + @classmethod + def hardcoded_ssz_dep_constants(cls) -> Dict[str, str]: + return {} -ALTAIR_SUNDRY_FUNCTIONS = ''' + @classmethod + def hardcoded_custom_type_dep_constants(cls) -> Dict[str, str]: + return {} + @classmethod + def invariant_checks(cls) -> str: + return '' + + +# +# AltairSpecAdjustment +# +class AltairSpecAdjustment(Phase0SpecAdjustment): + @classmethod + def imports_and_predefinitions(cls) -> str: + return '''from eth2spec.phase0 import spec as phase0 +from eth2spec.config.config_util import apply_constants_config +from typing import ( + Any, Dict, Set, Sequence, NewType, Tuple, TypeVar, Callable, Optional, Union +) + +from dataclasses import ( + dataclass, + field, +) + +from lru import LRU + +from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes +from eth2spec.utils.ssz.ssz_typing import ( + View, boolean, Container, List, Vector, uint8, uint32, uint64, + Bytes1, Bytes4, Bytes32, Bytes48, Bytes96, Bitlist, Bitvector, + Path, +) +from eth2spec.utils import bls + +from eth2spec.utils.hash_function import hash + +# Whenever altair is loaded, make sure we have the latest phase0 +from importlib import reload +reload(phase0) + + +SSZVariableName = str +GeneralizedIndex = NewType('GeneralizedIndex', int) +SSZObject = TypeVar('SSZObject', bound=View) + +CONFIG_NAME = 'mainnet' +''' + @classmethod + def sundry_functions(cls) -> str: + return super().sundry_functions() + '\n\n' + ''' def get_generalized_index(ssz_class: Any, *path: Sequence[Union[int, SSZVariableName]]) -> GeneralizedIndex: ssz_path = Path(ssz_class) for item in path: @@ -297,7 +341,59 @@ def get_generalized_index(ssz_class: Any, *path: Sequence[Union[int, SSZVariable return GeneralizedIndex(ssz_path.gindex())''' -MERGE_SUNDRY_FUNCTIONS = """ + @classmethod + def hardcoded_ssz_dep_constants(cls) -> Dict[str, str]: + constants = { + 'FINALIZED_ROOT_INDEX': 'GeneralizedIndex(105)', + 'NEXT_SYNC_COMMITTEE_INDEX': 'GeneralizedIndex(55)', + } + return {**super().hardcoded_ssz_dep_constants(), **constants} + + @classmethod + def invariant_checks(cls) -> str: + return ''' +assert ( + TIMELY_HEAD_WEIGHT + TIMELY_SOURCE_WEIGHT + TIMELY_TARGET_WEIGHT + SYNC_REWARD_WEIGHT + PROPOSER_WEIGHT +) == WEIGHT_DENOMINATOR''' + + +# +# MergeSpecAdjustment +# +class MergeSpecAdjustment(Phase0SpecAdjustment): + @classmethod + def imports_and_predefinitions(cls): + return '''from eth2spec.phase0 import spec as phase0 +from eth2spec.config.config_util import apply_constants_config +from typing import ( + Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar +) + +from dataclasses import ( + dataclass, + field, +) + +from lru import LRU + +from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes +from eth2spec.utils.ssz.ssz_typing import ( + View, boolean, Container, List, Vector, uint8, uint32, uint64, uint256, + Bytes1, Bytes4, Bytes20, Bytes32, Bytes48, Bytes96, Bitlist, + ByteList, ByteVector +) +from eth2spec.utils import bls + +from eth2spec.utils.hash_function import hash + +SSZObject = TypeVar('SSZObject', bound=View) + +CONFIG_NAME = 'mainnet' +''' + + @classmethod + def sundry_functions(cls) -> str: + return super().sundry_functions() + '\n\n' + """ ExecutionState = Any @@ -321,22 +417,18 @@ def produce_execution_payload(parent_hash: Bytes32) -> ExecutionPayload: pass""" -# The constants that depend on SSZ objects -# Will verify the value at the end of the spec -ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS = { - 'FINALIZED_ROOT_INDEX': 'GeneralizedIndex(105)', - 'NEXT_SYNC_COMMITTEE_INDEX': 'GeneralizedIndex(55)', -} + @classmethod + def hardcoded_custom_type_dep_constants(cls) -> str: + constants = { + 'MAX_BYTES_PER_OPAQUE_TRANSACTION': 'uint64(2**20)', + } + return {**super().hardcoded_custom_type_dep_constants(), **constants} -ALTAIR_INVAIANT_CHECKS = ''' -assert ( - TIMELY_HEAD_WEIGHT + TIMELY_SOURCE_WEIGHT + TIMELY_TARGET_WEIGHT + SYNC_REWARD_WEIGHT + PROPOSER_WEIGHT -) == WEIGHT_DENOMINATOR''' - - -MERGE_HARDCODED_CUSTOM_TYPE_DEP_CONSTANTS = { - 'MAX_BYTES_PER_OPAQUE_TRANSACTION': 'uint64(2**20)', +spec_adjustments = { + PHASE0: Phase0SpecAdjustment, + ALTAIR: AltairSpecAdjustment, + MERGE: MergeSpecAdjustment, } @@ -352,7 +444,7 @@ def is_merge(fork): return fork == MERGE -def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_class_objects: Dict[str, str]) -> str: +def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, fork: str, ordered_class_objects: Dict[str, str]) -> str: """ Given all the objects that constitute a spec, combine them into a single pyfile. """ @@ -382,41 +474,29 @@ def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_cl spec_object.constants[k] += " # noqa: E501" constants_spec = '\n'.join(map(lambda x: '%s = %s' % (x, spec_object.constants[x]), spec_object.constants)) ordered_class_objects_spec = '\n\n'.join(ordered_class_objects.values()) - - if is_altair(fork): - altair_ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS[x]), ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS)) - - if is_merge(fork): - merge_custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, MERGE_HARDCODED_CUSTOM_TYPE_DEP_CONSTANTS[x]), MERGE_HARDCODED_CUSTOM_TYPE_DEP_CONSTANTS)) - - + ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, adjustment.hardcoded_ssz_dep_constants()[x]), adjustment.hardcoded_ssz_dep_constants())) + ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), adjustment.hardcoded_ssz_dep_constants())) + custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, adjustment.hardcoded_custom_type_dep_constants()[x]), adjustment.hardcoded_custom_type_dep_constants())) spec = ( - imports + adjustment.imports_and_predefinitions() + '\n\n' + f"fork = \'{fork}\'\n" # The constants that some SSZ containers require. Need to be defined before `new_type_definitions` - + ('\n\n' + merge_custom_type_dep_constants + '\n' if is_merge(fork) else '') + + ('\n\n' + custom_type_dep_constants + '\n' if custom_type_dep_constants != '' else '') + '\n\n' + new_type_definitions - + '\n' + SUNDRY_CONSTANTS_FUNCTIONS + + '\n' + CONSTANT_DEP_SUNDRY_CONSTANTS_FUNCTIONS # The constants that some SSZ containers require. Need to be defined before `constants_spec` - + ('\n\n' + altair_ssz_dep_constants if is_altair(fork) else '') + + ('\n\n' + ssz_dep_constants if ssz_dep_constants != '' else '') + '\n\n' + constants_spec + '\n\n' + CONFIG_LOADER + '\n\n' + ordered_class_objects_spec + '\n\n' + functions_spec - # Functions to make pyspec work - + '\n' + PHASE0_SUNDRY_FUNCTIONS - + ('\n' + ALTAIR_SUNDRY_FUNCTIONS if is_altair(fork) else '') - + ('\n' + MERGE_SUNDRY_FUNCTIONS if is_merge(fork) else '') + + '\n' + adjustment.sundry_functions() + # Since some constants are hardcoded in setup.py, the following assertions verify that the hardcoded constants are + # as same as the spec definition. + + ('\n\n\n' + ssz_dep_constants_verification if ssz_dep_constants_verification != '' else '') + + ('\n' + adjustment.invariant_checks() if adjustment.invariant_checks() != '' else '') + + '\n' ) - - # Since some constants are hardcoded in setup.py, the following assertions verify that the hardcoded constants are - # as same as the spec definition. - if is_altair(fork): - altair_ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS)) - spec += '\n\n\n' + altair_ssz_dep_constants_verification - spec += '\n' + ALTAIR_INVAIANT_CHECKS - - spec += '\n' return spec @@ -496,13 +576,6 @@ def combine_spec_objects(spec0: SpecObject, spec1: SpecObject) -> SpecObject: ) -fork_imports = { - 'phase0': PHASE0_IMPORTS, - 'altair': ALTAIR_IMPORTS, - 'merge': MERGE_IMPORTS, -} - - def build_spec(fork: str, source_files: List[str]) -> str: all_specs = [get_spec(spec) for spec in source_files] @@ -513,7 +586,7 @@ def build_spec(fork: str, source_files: List[str]) -> str: class_objects = {**spec_object.ssz_objects, **spec_object.dataclasses} dependency_order_class_objects(class_objects, spec_object.custom_types) - return objects_to_spec(spec_object, fork_imports[fork], fork, class_objects) + return objects_to_spec(spec_object, spec_adjustments[fork], fork, class_objects) class PySpecCommand(Command): @@ -611,7 +684,7 @@ class BuildPyCommand(build_py): self.run_command('pyspec') def run(self): - for spec_fork in fork_imports: + for spec_fork in spec_adjustments: self.run_pyspec_cmd(spec_fork=spec_fork) super(BuildPyCommand, self).run() @@ -639,7 +712,7 @@ class PyspecDevCommand(Command): def run(self): print("running build_py command") - for spec_fork in fork_imports: + for spec_fork in spec_adjustments: self.run_pyspec_cmd(spec_fork=spec_fork) commands = { From 334a9b2434f70fe6bb01672f5a8d6c3886b7220f Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Thu, 8 Apr 2021 15:26:32 +0600 Subject: [PATCH 025/227] Pass timestamp to execution state transition and payload production --- specs/merge/beacon-chain.md | 14 ++++++++++++-- specs/merge/validator.md | 5 +++-- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index a93076001..2793a4579 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -26,6 +26,7 @@ - [Misc](#misc) - [`is_transition_completed`](#is_transition_completed) - [`is_transition_block`](#is_transition_block) + - [`compute_time_at_slot`](#compute_time_at_slot) - [Block processing](#block-processing) - [Execution payload processing](#execution-payload-processing) - [`get_execution_state`](#get_execution_state) @@ -149,6 +150,14 @@ def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> bool return not is_transition_completed(state) and block_body.execution_payload != ExecutionPayload() ``` +#### `compute_time_at_slot` + +```python +def compute_time_at_slot(state: BeaconState, slot: Slot) -> uint64: + slots_since_genesis = slot - GENESIS_SLOT + return uint64(state.genesis_time + slots_since_genesis * SECONDS_PER_SLOT) +``` + ### Block processing ```python @@ -171,7 +180,7 @@ The body of the function is implementation dependent. ##### `execution_state_transition` -Let `execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload) -> None` be the transition function of Ethereum execution state. +Let `execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload, timestamp: uint64) -> None` be the transition function of Ethereum execution state. The body of the function is implementation dependent. *Note*: `execution_state_transition` must throw `AssertionError` if either the transition itself or one of the pre or post conditions has failed. @@ -193,8 +202,9 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None assert execution_payload.parent_hash == state.latest_execution_payload_header.block_hash assert execution_payload.number == state.latest_execution_payload_header.number + 1 + timestamp = compute_time_at_slot(state, state.slot) execution_state = get_execution_state(state.latest_execution_payload_header.state_root) - execution_state_transition(execution_state, execution_payload) + execution_state_transition(execution_state, body.execution_payload, timestamp) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, diff --git a/specs/merge/validator.md b/specs/merge/validator.md index 42fe7adcf..2a23fd3e8 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -48,7 +48,7 @@ Let `get_pow_chain_head() -> PowBlock` be the function that returns the head of ###### `produce_execution_payload` -Let `produce_execution_payload(parent_hash: Hash32) -> ExecutionPayload` be the function that produces new instance of execution payload. +Let `produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> ExecutionPayload` be the function that produces new instance of execution payload. The body of this function is implementation dependent. * Set `block.body.execution_payload = get_execution_payload(state)` where: @@ -66,5 +66,6 @@ def get_execution_payload(state: BeaconState) -> ExecutionPayload: # Post-merge, normal payload execution_parent_hash = state.latest_execution_payload_header.block_hash - return produce_execution_payload(execution_parent_hash) + timestamp = compute_time_at_slot(state, state.slot) + return produce_execution_payload(execution_parent_hash, timestamp) ``` From ac175fd3255cefbe02c677ca67abb3b329cc573d Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Thu, 8 Apr 2021 15:29:42 +0600 Subject: [PATCH 026/227] Replace state with its root in execution_state_transition params --- specs/merge/beacon-chain.md | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 2793a4579..98312d246 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -29,7 +29,6 @@ - [`compute_time_at_slot`](#compute_time_at_slot) - [Block processing](#block-processing) - [Execution payload processing](#execution-payload-processing) - - [`get_execution_state`](#get_execution_state) - [`execution_state_transition`](#execution_state_transition) - [`process_execution_payload`](#process_execution_payload) @@ -171,16 +170,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: #### Execution payload processing -##### `get_execution_state` - -*Note*: `ExecutionState` class is an abstract class representing Ethereum execution state. - -Let `get_execution_state(execution_state_root: Bytes32) -> ExecutionState` be the function that given the root hash returns a copy of Ethereum execution state. -The body of the function is implementation dependent. - ##### `execution_state_transition` -Let `execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload, timestamp: uint64) -> None` be the transition function of Ethereum execution state. +Let `execution_state_transition(execution_state_root: Bytes32, execution_payload: ExecutionPayload, timestamp: uint64) -> None` be the transition function of Ethereum execution state. The body of the function is implementation dependent. *Note*: `execution_state_transition` must throw `AssertionError` if either the transition itself or one of the pre or post conditions has failed. @@ -203,8 +195,8 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None assert execution_payload.number == state.latest_execution_payload_header.number + 1 timestamp = compute_time_at_slot(state, state.slot) - execution_state = get_execution_state(state.latest_execution_payload_header.state_root) - execution_state_transition(execution_state, body.execution_payload, timestamp) + execution_state_root = state.latest_execution_payload_header.state_root + execution_state_transition(execution_state_root, body.execution_payload, timestamp) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, From 1ffa436836c9117a0157290ff40b86478ddefa66 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 9 Apr 2021 21:28:58 +0800 Subject: [PATCH 027/227] Update `imports_and_predefinitions` --- setup.py | 48 +++++++++++++++++++++++------------------------- 1 file changed, 23 insertions(+), 25 deletions(-) diff --git a/setup.py b/setup.py index 65b04fcf2..5498f2803 100644 --- a/setup.py +++ b/setup.py @@ -179,25 +179,22 @@ class SpecAdjustment(ABC): class Phase0SpecAdjustment(SpecAdjustment): @classmethod def imports_and_predefinitions(cls) -> str: - return '''from eth2spec.config.config_util import apply_constants_config -from typing import ( - Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar -) - + return '''from lru import LRU from dataclasses import ( dataclass, field, ) +from typing import ( + Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar +) -from lru import LRU - +from eth2spec.config.config_util import apply_constants_config from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes from eth2spec.utils.ssz.ssz_typing import ( View, boolean, Container, List, Vector, uint8, uint32, uint64, Bytes1, Bytes4, Bytes32, Bytes48, Bytes96, Bitlist, Bitvector, ) from eth2spec.utils import bls - from eth2spec.utils.hash_function import hash SSZObject = TypeVar('SSZObject', bound=View) @@ -297,19 +294,18 @@ get_attesting_indices = cache_this( class AltairSpecAdjustment(Phase0SpecAdjustment): @classmethod def imports_and_predefinitions(cls) -> str: - return '''from eth2spec.phase0 import spec as phase0 -from eth2spec.config.config_util import apply_constants_config -from typing import ( - Any, Dict, Set, Sequence, NewType, Tuple, TypeVar, Callable, Optional, Union -) - + return '''from lru import LRU from dataclasses import ( dataclass, field, ) +from typing import ( + Any, Dict, Set, Sequence, NewType, Tuple, TypeVar, Callable, Optional, Union +) -from lru import LRU +from eth2spec.config.config_util import apply_constants_config +from eth2spec.phase0 import spec as phase0 from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes from eth2spec.utils.ssz.ssz_typing import ( View, boolean, Container, List, Vector, uint8, uint32, uint64, @@ -320,7 +316,7 @@ from eth2spec.utils import bls from eth2spec.utils.hash_function import hash -# Whenever altair is loaded, make sure we have the latest phase0 +# Whenever this spec version is loaded, make sure we have the latest phase0 from importlib import reload reload(phase0) @@ -363,19 +359,17 @@ assert ( class MergeSpecAdjustment(Phase0SpecAdjustment): @classmethod def imports_and_predefinitions(cls): - return '''from eth2spec.phase0 import spec as phase0 -from eth2spec.config.config_util import apply_constants_config -from typing import ( - Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar -) - + return '''from lru import LRU from dataclasses import ( dataclass, field, ) +from typing import ( + Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar +) -from lru import LRU - +from eth2spec.phase0 import spec as phase0 +from eth2spec.config.config_util import apply_constants_config from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes from eth2spec.utils.ssz.ssz_typing import ( View, boolean, Container, List, Vector, uint8, uint32, uint64, uint256, @@ -383,9 +377,13 @@ from eth2spec.utils.ssz.ssz_typing import ( ByteList, ByteVector ) from eth2spec.utils import bls - from eth2spec.utils.hash_function import hash + +# Whenever this spec version is loaded, make sure we have the latest phase0 +from importlib import reload +reload(phase0) + SSZObject = TypeVar('SSZObject', bound=View) CONFIG_NAME = 'mainnet' From ceb352be12fbca77e710574c0961db237b67652c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 9 Apr 2021 22:17:01 +0800 Subject: [PATCH 028/227] Refactor `imports_and_predefinitions` into `imports` and `preparations` --- setup.py | 101 ++++++++++++++++++++++--------------------------------- 1 file changed, 40 insertions(+), 61 deletions(-) diff --git a/setup.py b/setup.py index 5498f2803..8201df574 100644 --- a/setup.py +++ b/setup.py @@ -134,9 +134,17 @@ def get_spec(file_name: str) -> SpecObject: class SpecAdjustment(ABC): @classmethod @abstractmethod - def imports_and_predefinitions(cls) -> str: + def imports(cls) -> str: """ - Importing functions and defining special types/constants for building pyspec. + Import objects from other libaries. + """ + raise NotImplementedError() + + @classmethod + @abstractmethod + def preparations(cls) -> str: + """ + Define special types/constants for building pyspec or call functions. """ raise NotImplementedError() @@ -178,7 +186,7 @@ class SpecAdjustment(ABC): # class Phase0SpecAdjustment(SpecAdjustment): @classmethod - def imports_and_predefinitions(cls) -> str: + def imports(cls) -> str: return '''from lru import LRU from dataclasses import ( dataclass, @@ -192,12 +200,15 @@ from eth2spec.config.config_util import apply_constants_config from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes from eth2spec.utils.ssz.ssz_typing import ( View, boolean, Container, List, Vector, uint8, uint32, uint64, - Bytes1, Bytes4, Bytes32, Bytes48, Bytes96, Bitlist, Bitvector, -) + Bytes1, Bytes4, Bytes32, Bytes48, Bytes96, Bitlist) +from eth2spec.utils.ssz.ssz_typing import Bitvector # noqa: F401 from eth2spec.utils import bls from eth2spec.utils.hash_function import hash +''' -SSZObject = TypeVar('SSZObject', bound=View) + @classmethod + def preparations(cls) -> str: + return '''SSZObject = TypeVar('SSZObject', bound=View) CONFIG_NAME = 'mainnet' ''' @@ -293,40 +304,25 @@ get_attesting_indices = cache_this( # class AltairSpecAdjustment(Phase0SpecAdjustment): @classmethod - def imports_and_predefinitions(cls) -> str: - return '''from lru import LRU -from dataclasses import ( - dataclass, - field, -) -from typing import ( - Any, Dict, Set, Sequence, NewType, Tuple, TypeVar, Callable, Optional, Union -) - - -from eth2spec.config.config_util import apply_constants_config -from eth2spec.phase0 import spec as phase0 -from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes -from eth2spec.utils.ssz.ssz_typing import ( - View, boolean, Container, List, Vector, uint8, uint32, uint64, - Bytes1, Bytes4, Bytes32, Bytes48, Bytes96, Bitlist, Bitvector, - Path, -) -from eth2spec.utils import bls - -from eth2spec.utils.hash_function import hash - -# Whenever this spec version is loaded, make sure we have the latest phase0 + def imports(cls) -> str: + return super().imports() + '\n' + ''' +from typing import NewType, Union from importlib import reload -reload(phase0) +from eth2spec.phase0 import spec as phase0 +from eth2spec.utils.ssz.ssz_typing import Path +''' + + @classmethod + def preparations(cls): + return super().preparations() + '\n' + ''' +# Whenever this spec version is loaded, make sure we have the latest phase0 +reload(phase0) SSZVariableName = str GeneralizedIndex = NewType('GeneralizedIndex', int) -SSZObject = TypeVar('SSZObject', bound=View) - -CONFIG_NAME = 'mainnet' ''' + @classmethod def sundry_functions(cls) -> str: return super().sundry_functions() + '\n\n' + ''' @@ -358,35 +354,17 @@ assert ( # class MergeSpecAdjustment(Phase0SpecAdjustment): @classmethod - def imports_and_predefinitions(cls): - return '''from lru import LRU -from dataclasses import ( - dataclass, - field, -) -from typing import ( - Any, Callable, Dict, Set, Sequence, Tuple, Optional, TypeVar -) - + def imports(cls): + return super().imports() + '\n' + ''' from eth2spec.phase0 import spec as phase0 -from eth2spec.config.config_util import apply_constants_config -from eth2spec.utils.ssz.ssz_impl import hash_tree_root, copy, uint_to_bytes -from eth2spec.utils.ssz.ssz_typing import ( - View, boolean, Container, List, Vector, uint8, uint32, uint64, uint256, - Bytes1, Bytes4, Bytes20, Bytes32, Bytes48, Bytes96, Bitlist, - ByteList, ByteVector -) -from eth2spec.utils import bls -from eth2spec.utils.hash_function import hash - - -# Whenever this spec version is loaded, make sure we have the latest phase0 +from eth2spec.utils.ssz.ssz_typing import Bytes20, ByteList, ByteVector, uint256 from importlib import reload +''' + + @classmethod + def preparations(cls): + return super().preparations() + '\n' + ''' reload(phase0) - -SSZObject = TypeVar('SSZObject', bound=View) - -CONFIG_NAME = 'mainnet' ''' @classmethod @@ -476,7 +454,8 @@ def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, fork: s ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), adjustment.hardcoded_ssz_dep_constants())) custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, adjustment.hardcoded_custom_type_dep_constants()[x]), adjustment.hardcoded_custom_type_dep_constants())) spec = ( - adjustment.imports_and_predefinitions() + adjustment.imports() + + adjustment.preparations() + '\n\n' + f"fork = \'{fork}\'\n" # The constants that some SSZ containers require. Need to be defined before `new_type_definitions` + ('\n\n' + custom_type_dep_constants + '\n' if custom_type_dep_constants != '' else '') From 3320ebb8654b419d6b38129162ad2accf783572c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 9 Apr 2021 22:26:32 +0800 Subject: [PATCH 029/227] Fix typo and add `SpecAdjustment.fork` property --- setup.py | 46 ++++++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/setup.py b/setup.py index 8201df574..2101eebc4 100644 --- a/setup.py +++ b/setup.py @@ -132,11 +132,16 @@ def get_spec(file_name: str) -> SpecObject: class SpecAdjustment(ABC): + @property + @abstractmethod + def fork(self) -> str: + raise NotImplementedError() + @classmethod @abstractmethod def imports(cls) -> str: """ - Import objects from other libaries. + Import objects from other libraries. """ raise NotImplementedError() @@ -185,6 +190,8 @@ class SpecAdjustment(ABC): # Phase0SpecAdjustment # class Phase0SpecAdjustment(SpecAdjustment): + fork: str = PHASE0 + @classmethod def imports(cls) -> str: return '''from lru import LRU @@ -208,8 +215,8 @@ from eth2spec.utils.hash_function import hash @classmethod def preparations(cls) -> str: - return '''SSZObject = TypeVar('SSZObject', bound=View) - + return ''' +SSZObject = TypeVar('SSZObject', bound=View) CONFIG_NAME = 'mainnet' ''' @@ -303,6 +310,8 @@ get_attesting_indices = cache_this( # AltairSpecAdjustment # class AltairSpecAdjustment(Phase0SpecAdjustment): + fork: str = ALTAIR + @classmethod def imports(cls) -> str: return super().imports() + '\n' + ''' @@ -353,6 +362,8 @@ assert ( # MergeSpecAdjustment # class MergeSpecAdjustment(Phase0SpecAdjustment): + fork: str = MERGE + @classmethod def imports(cls): return super().imports() + '\n' + ''' @@ -402,25 +413,12 @@ def produce_execution_payload(parent_hash: Bytes32) -> ExecutionPayload: spec_adjustments = { - PHASE0: Phase0SpecAdjustment, - ALTAIR: AltairSpecAdjustment, - MERGE: MergeSpecAdjustment, + adjustment.fork: adjustment + for adjustment in (Phase0SpecAdjustment, AltairSpecAdjustment, MergeSpecAdjustment) } -def is_phase0(fork): - return fork == PHASE0 - - -def is_altair(fork): - return fork == ALTAIR - - -def is_merge(fork): - return fork == MERGE - - -def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, fork: str, ordered_class_objects: Dict[str, str]) -> str: +def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, ordered_class_objects: Dict[str, str]) -> str: """ Given all the objects that constitute a spec, combine them into a single pyfile. """ @@ -456,7 +454,7 @@ def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, fork: s spec = ( adjustment.imports() + adjustment.preparations() - + '\n\n' + f"fork = \'{fork}\'\n" + + '\n\n' + f"fork = \'{adjustment.fork}\'\n" # The constants that some SSZ containers require. Need to be defined before `new_type_definitions` + ('\n\n' + custom_type_dep_constants + '\n' if custom_type_dep_constants != '' else '') + '\n\n' + new_type_definitions @@ -563,7 +561,7 @@ def build_spec(fork: str, source_files: List[str]) -> str: class_objects = {**spec_object.ssz_objects, **spec_object.dataclasses} dependency_order_class_objects(class_objects, spec_object.custom_types) - return objects_to_spec(spec_object, spec_adjustments[fork], fork, class_objects) + return objects_to_spec(spec_object, spec_adjustments[fork], class_objects) class PySpecCommand(Command): @@ -595,14 +593,14 @@ class PySpecCommand(Command): if len(self.md_doc_paths) == 0: print("no paths were specified, using default markdown file paths for pyspec" " build (spec fork: %s)" % self.spec_fork) - if is_phase0(self.spec_fork): + if self.spec_fork == PHASE0: self.md_doc_paths = """ specs/phase0/beacon-chain.md specs/phase0/fork-choice.md specs/phase0/validator.md specs/phase0/weak-subjectivity.md """ - elif is_altair(self.spec_fork): + elif self.spec_fork == ALTAIR: self.md_doc_paths = """ specs/phase0/beacon-chain.md specs/phase0/fork-choice.md @@ -613,7 +611,7 @@ class PySpecCommand(Command): specs/altair/validator.md specs/altair/sync-protocol.md """ - elif is_merge(self.spec_fork): + elif self.spec_fork == MERGE: self.md_doc_paths = """ specs/phase0/beacon-chain.md specs/phase0/fork-choice.md From c4c4dade27ff08bad81171132b4717dd6e3762b7 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Fri, 9 Apr 2021 22:50:37 +0600 Subject: [PATCH 030/227] Add missing timestamp in validator.md/get_execution_payload --- specs/merge/validator.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/merge/validator.md b/specs/merge/validator.md index 2a23fd3e8..dccc5727b 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -62,7 +62,8 @@ def get_execution_payload(state: BeaconState) -> ExecutionPayload: return ExecutionPayload() else: # Signify merge via producing on top of the last PoW block - return produce_execution_payload(pow_block.block_hash) + timestamp = compute_time_at_slot(state, state.slot) + return produce_execution_payload(pow_block.block_hash, timestamp) # Post-merge, normal payload execution_parent_hash = state.latest_execution_payload_header.block_hash From a8160f1634eab48644a44d75ae22edb126d5f654 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 10 Apr 2021 00:53:37 +0800 Subject: [PATCH 031/227] Try to initialize state with pure Altair spec in tests --- specs/altair/beacon-chain.md | 47 +++++++++++++++++++ tests/core/pyspec/eth2spec/test/context.py | 14 ++---- .../eth2spec/test/helpers/fork_choice.py | 5 +- .../pyspec/eth2spec/test/helpers/genesis.py | 16 ++++++- 4 files changed, 67 insertions(+), 15 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9b476bce1..9f9af9de7 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -52,6 +52,7 @@ - [Slashings](#slashings) - [Participation flags updates](#participation-flags-updates) - [Sync committee updates](#sync-committee-updates) +- [Initialize state for testnets](#initialize-state-for-testnets) @@ -661,3 +662,49 @@ def process_sync_committee_updates(state: BeaconState) -> None: state.current_sync_committee = state.next_sync_committee state.next_sync_committee = get_sync_committee(state, next_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) ``` + +## Initialize state for testnets + +This helper function is only for initialize Altair testnets and tests. + +```python +def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, + eth1_timestamp: uint64, + deposits: Sequence[Deposit]) -> BeaconState: + fork = Fork( + previous_version=GENESIS_FORK_VERSION, + current_version=ALTAIR_FORK_VERSION, + epoch=GENESIS_EPOCH, + ) + state = BeaconState( + genesis_time=eth1_timestamp + GENESIS_DELAY, + fork=fork, + eth1_data=Eth1Data(block_hash=eth1_block_hash, deposit_count=uint64(len(deposits))), + latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), + randao_mixes=[eth1_block_hash] * EPOCHS_PER_HISTORICAL_VECTOR, # Seed RANDAO with Eth1 entropy + ) + + # Process deposits + leaves = list(map(lambda deposit: deposit.data, deposits)) + for index, deposit in enumerate(deposits): + deposit_data_list = List[DepositData, 2**DEPOSIT_CONTRACT_TREE_DEPTH](*leaves[:index + 1]) + state.eth1_data.deposit_root = hash_tree_root(deposit_data_list) + process_deposit(state, deposit) + + # Process activations + for index, validator in enumerate(state.validators): + balance = state.balances[index] + validator.effective_balance = min(balance - balance % EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE) + if validator.effective_balance == MAX_EFFECTIVE_BALANCE: + validator.activation_eligibility_epoch = GENESIS_EPOCH + validator.activation_epoch = GENESIS_EPOCH + + # Set genesis validators root for domain separation and chain versioning + state.genesis_validators_root = hash_tree_root(state.validators) + + # Fill in sync committees + state.current_sync_committee = get_sync_committee(state, get_current_epoch(state)) + state.next_sync_committee = get_sync_committee(state, get_current_epoch(state) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + + return state +``` diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 7ae1b9541..34b6e79a1 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -75,17 +75,11 @@ class SpecForks(TypedDict, total=False): def _prepare_state(balances_fn: Callable[[Any], Sequence[int]], threshold_fn: Callable[[Any], int], spec: Spec, phases: SpecForks): - - p0 = phases[PHASE0] - balances = balances_fn(p0) - activation_threshold = threshold_fn(p0) - - state = create_genesis_state(spec=p0, validator_balances=balances, + phase = phases[spec.fork] + balances = balances_fn(phase) + activation_threshold = threshold_fn(phase) + state = create_genesis_state(spec=phase, validator_balances=balances, activation_threshold=activation_threshold) - # TODO: upgrade to merge spec, and later sharding. - if spec.fork == ALTAIR: - state = phases[ALTAIR].upgrade_to_altair(state) - return state diff --git a/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py b/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py index 2ca37768b..f3b80b2ac 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py +++ b/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py @@ -1,7 +1,5 @@ from eth_utils import encode_hex -from eth2spec.phase0 import spec as phase0_spec - def get_anchor_root(spec, state): anchor_block_header = state.latest_block_header.copy() @@ -58,8 +56,7 @@ def get_genesis_forkchoice_store(spec, genesis_state): def get_genesis_forkchoice_store_and_block(spec, genesis_state): assert genesis_state.slot == spec.GENESIS_SLOT - # The genesis block must be a Phase 0 `BeaconBlock` - genesis_block = phase0_spec.BeaconBlock(state_root=genesis_state.hash_tree_root()) + genesis_block = spec.BeaconBlock(state_root=genesis_state.hash_tree_root()) return spec.get_forkchoice_store(genesis_state, genesis_block), genesis_block diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 14fd9aa47..5f2cd5281 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -20,6 +20,9 @@ def create_genesis_state(spec, validator_balances, activation_threshold): deposit_root = b'\x42' * 32 eth1_block_hash = b'\xda' * 32 + current_version = spec.GENESIS_FORK_VERSION + if spec.fork == 'altair': # TODO: fix `context.py` dependency + current_version = spec.ALTAIR_FORK_VERSION state = spec.BeaconState( genesis_time=0, eth1_deposit_index=len(validator_balances), @@ -30,7 +33,7 @@ def create_genesis_state(spec, validator_balances, activation_threshold): ), fork=spec.Fork( previous_version=spec.GENESIS_FORK_VERSION, - current_version=spec.GENESIS_FORK_VERSION, + current_version=current_version, epoch=spec.GENESIS_EPOCH, ), latest_block_header=spec.BeaconBlockHeader(body_root=spec.hash_tree_root(spec.BeaconBlockBody())), @@ -47,8 +50,19 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if validator.effective_balance >= activation_threshold: validator.activation_eligibility_epoch = spec.GENESIS_EPOCH validator.activation_epoch = spec.GENESIS_EPOCH + if spec.fork != 'phase0': # TODO: fix `context.py` dependency + state.previous_epoch_participation.append(spec.ParticipationFlags(0b0000_0000)) + state.current_epoch_participation.append(spec.ParticipationFlags(0b0000_0000)) + state.inactivity_scores.append(spec.uint64(0)) # Set genesis validators root for domain separation and chain versioning state.genesis_validators_root = spec.hash_tree_root(state.validators) + if spec.fork != 'phase0': # TODO: fix `context.py` dependency + # Fill in sync committees + state.current_sync_committee = spec.get_sync_committee(state, spec.get_current_epoch(state)) + state.next_sync_committee = ( + spec.get_sync_committee(state, spec.get_current_epoch(state) + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + ) + return state From 79230c5f68f975a4765dcd4c0bd9f28a11e427b8 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 10 Apr 2021 03:04:27 +0800 Subject: [PATCH 032/227] Update pyspec builder --- setup.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 514f75c50..b337d70ff 100644 --- a/setup.py +++ b/setup.py @@ -313,11 +313,13 @@ def get_pow_chain_head() -> PowBlock: pass -def execution_state_transition(execution_state: ExecutionState, execution_payload: ExecutionPayload) -> None: +def execution_state_transition(execution_state_root: Bytes32, + execution_payload: ExecutionPayload, + timestamp: uint64) -> None: pass -def produce_execution_payload(parent_hash: Bytes32) -> ExecutionPayload: +def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> ExecutionPayload: pass""" From 8708ec4bb0e2e59ef8ac050b51ced5f4867afc1a Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 12 Apr 2021 22:02:04 +0800 Subject: [PATCH 033/227] Update doc --- specs/altair/beacon-chain.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9f9af9de7..e14a0f5c0 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -663,9 +663,11 @@ def process_sync_committee_updates(state: BeaconState) -> None: state.next_sync_committee = get_sync_committee(state, next_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) ``` -## Initialize state for testnets +## Initialize state for Altair testnets -This helper function is only for initialize Altair testnets and tests. +This helper function is only for initializing the pure Altair testnets and tests, where we use the dummy where the `ALTAIR_FORK_SLOT` GENESIS_SLOT`. + +*Note*: The function `initialize_beacon_state_from_eth1` is modified with `ALTAIR_FORK_VERSION` fork version and initial sync committees. ```python def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, From cd43d64809acb37344fc13ef48b2b03a60c7c48f Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 12 Apr 2021 22:20:27 +0800 Subject: [PATCH 034/227] Move constants to `eth2spec.test.helpers.constants` --- .../gen_helpers/gen_from_tests/gen.py | 3 +- .../test_process_sync_committee.py | 6 ++-- .../test_process_sync_committee_updates.py | 6 ++-- .../eth2spec/test/altair/fork/test_fork.py | 6 ++-- .../test/altair/sanity/test_blocks.py | 2 +- .../test/altair/unittests/test_helpers.py | 2 +- .../altair/unittests/test_sync_protocol.py | 6 ++-- tests/core/pyspec/eth2spec/test/context.py | 30 ++++--------------- .../test_process_attestation.py | 2 +- .../test_process_chunk_challenge.py | 6 ++-- .../test_process_custody_key_reveal.py | 2 +- .../test_process_custody_slashing.py | 6 ++-- ...est_process_early_derived_secret_reveal.py | 2 +- .../test_process_challenge_deadlines.py | 6 ++-- .../test_process_custody_final_updates.py | 2 +- .../test_process_reveal_deadlines.py | 6 ++-- .../test/custody_game/sanity/test_blocks.py | 6 ++-- .../pyspec/eth2spec/test/helpers/constants.py | 29 ++++++++++++++++++ .../pyspec/eth2spec/test/helpers/typing.py | 4 +++ ...st_process_participation_record_updates.py | 3 +- .../test/phase0/fork_choice/test_get_head.py | 2 +- .../phase0/genesis/test_initialization.py | 2 +- .../test/phase0/genesis/test_validity.py | 2 +- .../test/phase0/rewards/test_basic.py | 3 +- .../eth2spec/test/phase0/rewards/test_leak.py | 3 +- .../test/phase0/sanity/test_blocks.py | 3 +- .../fork_choice/test_on_attestation.py | 3 +- .../validator/test_validator_unittest.py | 2 +- .../unittests/test_get_start_shard.py | 2 +- tests/generators/README.md | 2 +- tests/generators/bls/main.py | 2 +- tests/generators/epoch_processing/main.py | 2 +- tests/generators/finality/main.py | 2 +- tests/generators/fork_choice/main.py | 2 +- tests/generators/forks/main.py | 2 +- tests/generators/genesis/main.py | 2 +- tests/generators/operations/main.py | 2 +- tests/generators/rewards/main.py | 2 +- tests/generators/sanity/main.py | 2 +- tests/generators/shuffling/main.py | 2 +- tests/generators/ssz_generic/main.py | 2 +- tests/generators/ssz_static/main.py | 2 +- 42 files changed, 109 insertions(+), 74 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/helpers/constants.py create mode 100644 tests/core/pyspec/eth2spec/test/helpers/typing.py diff --git a/tests/core/pyspec/eth2spec/gen_helpers/gen_from_tests/gen.py b/tests/core/pyspec/eth2spec/gen_helpers/gen_from_tests/gen.py index c090869d5..bc63b212b 100644 --- a/tests/core/pyspec/eth2spec/gen_helpers/gen_from_tests/gen.py +++ b/tests/core/pyspec/eth2spec/gen_helpers/gen_from_tests/gen.py @@ -4,7 +4,8 @@ from typing import Any, Callable, Dict, Iterable, Optional from eth2spec.config import config_util from eth2spec.utils import bls -from eth2spec.test.context import ALL_CONFIGS, TESTGEN_FORKS, SpecForkName, ConfigName +from eth2spec.test.helpers.constants import ALL_CONFIGS, TESTGEN_FORKS +from eth2spec.test.helpers.typing import SpecForkName, ConfigName from eth2spec.gen_helpers.gen_base import gen_runner from eth2spec.gen_helpers.gen_base.gen_typing import TestCase, TestProvider diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index ed9bcac9c..307d0f82d 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -8,12 +8,14 @@ from eth2spec.test.helpers.state import ( state_transition_and_sign_block, transition_to, ) +from eth2spec.test.helpers.constants import ( + PHASE0, + MAINNET, MINIMAL, +) from eth2spec.test.helpers.sync_committee import ( compute_aggregate_sync_committee_signature, ) from eth2spec.test.context import ( - PHASE0, - MAINNET, MINIMAL, expect_assertion_error, with_all_phases_except, with_configs, diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index 4c5ac0633..7ba2645a2 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -1,10 +1,12 @@ from eth2spec.test.context import ( - PHASE0, - MINIMAL, spec_state_test, with_all_phases_except, with_configs, ) +from eth2spec.test.helpers.constants import ( + PHASE0, + MINIMAL, +) from eth2spec.test.helpers.state import transition_to from eth2spec.test.helpers.epoch_processing import ( run_epoch_processing_with, diff --git a/tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py b/tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py index fb056b72a..1ad39209c 100644 --- a/tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py +++ b/tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py @@ -1,6 +1,4 @@ from eth2spec.test.context import ( - PHASE0, ALTAIR, - MINIMAL, with_phases, with_custom_state, with_configs, @@ -8,6 +6,10 @@ from eth2spec.test.context import ( low_balances, misc_balances, large_validator_set, ) from eth2spec.test.utils import with_meta_tags +from eth2spec.test.helpers.constants import ( + PHASE0, ALTAIR, + MINIMAL, +) from eth2spec.test.helpers.state import ( next_epoch, next_epoch_via_block, diff --git a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py index 9f28926d4..48ab6956b 100644 --- a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py @@ -11,8 +11,8 @@ from eth2spec.test.helpers.block import ( from eth2spec.test.helpers.sync_committee import ( compute_aggregate_sync_committee_signature, ) +from eth2spec.test.helpers.constants import PHASE0 from eth2spec.test.context import ( - PHASE0, with_all_phases_except, spec_state_test, ) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/test_helpers.py b/tests/core/pyspec/eth2spec/test/altair/unittests/test_helpers.py index d89ff6d84..c837f06c3 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/test_helpers.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/test_helpers.py @@ -1,8 +1,8 @@ from eth2spec.test.context import ( spec_state_test, with_phases, - ALTAIR, ) +from eth2spec.test.helpers.constants import ALTAIR from eth2spec.test.helpers.merkle import build_proof diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py index 4c9b98e0a..9b8c35e76 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py @@ -1,6 +1,4 @@ from eth2spec.test.context import ( - ALTAIR, - MINIMAL, spec_state_test, with_configs, with_phases, @@ -10,6 +8,10 @@ from eth2spec.test.helpers.block import ( build_empty_block, build_empty_block_for_next_slot, ) +from eth2spec.test.helpers.constants import ( + ALTAIR, + MINIMAL, +) from eth2spec.test.helpers.state import ( next_slots, state_transition_and_sign_block, diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 7ae1b9541..002e84136 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -6,11 +6,15 @@ from eth2spec.merge import spec as spec_merge from eth2spec.utils import bls from .exceptions import SkippedTest +from .helpers.constants import ( + PHASE0, ALTAIR, MERGE, SHARDING, CUSTODY_GAME, DAS, + ALL_PHASES, +) from .helpers.genesis import create_genesis_state from .utils import vector_test, with_meta_tags from random import Random -from typing import Any, Callable, NewType, Sequence, TypedDict, Protocol +from typing import Any, Callable, Sequence, TypedDict, Protocol from lru import LRU @@ -23,30 +27,6 @@ def reload_specs(): reload(spec_merge) -# Some of the Spec module functionality is exposed here to deal with phase-specific changes. - -SpecForkName = NewType("SpecForkName", str) -ConfigName = NewType("ConfigName", str) - -PHASE0 = SpecForkName('phase0') -ALTAIR = SpecForkName('altair') - -# Experimental phases (not included in default "ALL_PHASES"): -MERGE = SpecForkName('merge') -SHARDING = SpecForkName('sharding') -CUSTODY_GAME = SpecForkName('custody_game') -DAS = SpecForkName('das') - -ALL_PHASES = (PHASE0, ALTAIR) - -MAINNET = ConfigName('mainnet') -MINIMAL = ConfigName('minimal') - -ALL_CONFIGS = (MINIMAL, MAINNET) - -# The forks that output to the test vectors. -TESTGEN_FORKS = (PHASE0, ALTAIR) - # TODO: currently phases are defined as python modules. # It would be better if they would be more well-defined interfaces for stronger typing. diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py index 92633a8c5..707ac0b2e 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py @@ -1,9 +1,9 @@ from eth2spec.test.context import ( - CUSTODY_GAME, with_phases, spec_state_test, always_bls, ) +from eth2spec.test.helpers.constants import CUSTODY_GAME from eth2spec.test.helpers.state import transition_to from eth2spec.test.helpers.attestations import ( run_attestation_processing, diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py index 69143812f..5a939c108 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py @@ -6,10 +6,12 @@ from eth2spec.test.helpers.custody import ( from eth2spec.test.helpers.attestations import ( get_valid_on_time_attestation, ) -from eth2spec.test.helpers.state import transition_to, transition_to_valid_shard_slot -from eth2spec.test.context import ( +from eth2spec.test.helpers.constants import ( CUSTODY_GAME, MINIMAL, +) +from eth2spec.test.helpers.state import transition_to, transition_to_valid_shard_slot +from eth2spec.test.context import ( expect_assertion_error, disable_process_reveal_deadlines, spec_state_test, diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_key_reveal.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_key_reveal.py index a825d7c75..fa8401228 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_key_reveal.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_key_reveal.py @@ -1,6 +1,6 @@ +from eth2spec.test.helpers.constants import CUSTODY_GAME from eth2spec.test.helpers.custody import get_valid_custody_key_reveal from eth2spec.test.context import ( - CUSTODY_GAME, with_phases, spec_state_test, expect_assertion_error, diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py index 12ac708aa..f554761f3 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py @@ -5,12 +5,14 @@ from eth2spec.test.helpers.custody import ( from eth2spec.test.helpers.attestations import ( get_valid_on_time_attestation, ) +from eth2spec.test.helpers.constants import ( + CUSTODY_GAME, + MINIMAL, +) from eth2spec.test.helpers.keys import privkeys from eth2spec.utils.ssz.ssz_typing import ByteList from eth2spec.test.helpers.state import get_balance, transition_to from eth2spec.test.context import ( - MINIMAL, - CUSTODY_GAME, with_phases, spec_state_test, expect_assertion_error, diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_early_derived_secret_reveal.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_early_derived_secret_reveal.py index d3da6c580..904c9af4e 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_early_derived_secret_reveal.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_early_derived_secret_reveal.py @@ -1,7 +1,7 @@ +from eth2spec.test.helpers.constants import CUSTODY_GAME from eth2spec.test.helpers.custody import get_valid_early_derived_secret_reveal from eth2spec.test.helpers.state import next_epoch_via_block, get_balance from eth2spec.test.context import ( - CUSTODY_GAME, with_phases, spec_state_test, expect_assertion_error, diff --git a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py index f0e353cec..42ff54303 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py @@ -7,13 +7,15 @@ from eth2spec.test.helpers.attestations import ( ) from eth2spec.test.helpers.state import transition_to, transition_to_valid_shard_slot from eth2spec.test.context import ( - CUSTODY_GAME, - MINIMAL, spec_state_test, with_phases, with_configs, ) from eth2spec.test.phase0.block_processing.test_process_attestation import run_attestation_processing +from eth2spec.test.helpers.constants import ( + CUSTODY_GAME, + MINIMAL, +) from eth2spec.test.helpers.epoch_processing import run_epoch_processing_with from eth2spec.test.custody_game.block_processing.test_process_chunk_challenge import ( diff --git a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py index acc076bce..92c311a29 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py @@ -1,4 +1,4 @@ -from eth2spec.test.context import ( +from eth2spec.test.helpers.constants import ( CUSTODY_GAME, ) from eth2spec.test.helpers.custody import ( diff --git a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_reveal_deadlines.py b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_reveal_deadlines.py index c82ebd8bb..aec24bca8 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_reveal_deadlines.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_reveal_deadlines.py @@ -3,12 +3,14 @@ from eth2spec.test.helpers.custody import ( ) from eth2spec.test.helpers.state import transition_to from eth2spec.test.context import ( - CUSTODY_GAME, - MINIMAL, with_phases, with_configs, spec_state_test, ) +from eth2spec.test.helpers.constants import ( + CUSTODY_GAME, + MINIMAL, +) from eth2spec.test.helpers.epoch_processing import run_epoch_processing_with from eth2spec.test.custody_game.block_processing.test_process_custody_key_reveal import ( run_custody_key_reveal_processing, diff --git a/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py index 88dd54bda..5f5eadef3 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py @@ -1,14 +1,16 @@ from typing import Dict, Sequence from eth2spec.test.context import ( - CUSTODY_GAME, - MINIMAL, with_phases, spec_state_test, with_configs, ) from eth2spec.test.helpers.attestations import get_valid_on_time_attestation from eth2spec.test.helpers.block import build_empty_block +from eth2spec.test.helpers.constants import ( + CUSTODY_GAME, + MINIMAL, +) from eth2spec.test.helpers.custody import ( get_custody_slashable_test_vector, get_valid_chunk_challenge, diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py new file mode 100644 index 000000000..bffffe348 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -0,0 +1,29 @@ +from .typing import SpecForkName, ConfigName + + +# +# SpecForkName +# +# Some of the Spec module functionality is exposed here to deal with phase-specific changes. +PHASE0 = SpecForkName('phase0') +ALTAIR = SpecForkName('altair') + +# Experimental phases (not included in default "ALL_PHASES"): +MERGE = SpecForkName('merge') +SHARDING = SpecForkName('sharding') +CUSTODY_GAME = SpecForkName('custody_game') +DAS = SpecForkName('das') + +# The forks that pytest runs with. +ALL_PHASES = (PHASE0, ALTAIR) +# The forks that output to the test vectors. +TESTGEN_FORKS = (PHASE0, ALTAIR) + + +# +# Config +# +MAINNET = ConfigName('mainnet') +MINIMAL = ConfigName('minimal') + +ALL_CONFIGS = (MINIMAL, MAINNET) diff --git a/tests/core/pyspec/eth2spec/test/helpers/typing.py b/tests/core/pyspec/eth2spec/test/helpers/typing.py new file mode 100644 index 000000000..04578f64c --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/typing.py @@ -0,0 +1,4 @@ +from typing import NewType + +SpecForkName = NewType("SpecForkName", str) +ConfigName = NewType("ConfigName", str) diff --git a/tests/core/pyspec/eth2spec/test/phase0/epoch_processing/test_process_participation_record_updates.py b/tests/core/pyspec/eth2spec/test/phase0/epoch_processing/test_process_participation_record_updates.py index 978ef5739..c4d4332bc 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/epoch_processing/test_process_participation_record_updates.py +++ b/tests/core/pyspec/eth2spec/test/phase0/epoch_processing/test_process_participation_record_updates.py @@ -1,4 +1,5 @@ -from eth2spec.test.context import PHASE0, spec_state_test, with_phases +from eth2spec.test.context import spec_state_test, with_phases +from eth2spec.test.helpers.constants import PHASE0 from eth2spec.test.helpers.epoch_processing import ( run_epoch_processing_with ) diff --git a/tests/core/pyspec/eth2spec/test/phase0/fork_choice/test_get_head.py b/tests/core/pyspec/eth2spec/test/phase0/fork_choice/test_get_head.py index c598a3a7f..0822d44c2 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/fork_choice/test_get_head.py +++ b/tests/core/pyspec/eth2spec/test/phase0/fork_choice/test_get_head.py @@ -1,7 +1,6 @@ from eth_utils import encode_hex from eth2spec.test.context import ( - MINIMAL, is_post_altair, spec_state_test, with_all_phases, @@ -9,6 +8,7 @@ from eth2spec.test.context import ( ) from eth2spec.test.helpers.attestations import get_valid_attestation, next_epoch_with_attestations from eth2spec.test.helpers.block import build_empty_block_for_next_slot +from eth2spec.test.helpers.constants import MINIMAL from eth2spec.test.helpers.fork_choice import ( tick_and_run_on_attestation, tick_and_run_on_block, diff --git a/tests/core/pyspec/eth2spec/test/phase0/genesis/test_initialization.py b/tests/core/pyspec/eth2spec/test/phase0/genesis/test_initialization.py index 41fbd7726..be9d120b4 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/genesis/test_initialization.py +++ b/tests/core/pyspec/eth2spec/test/phase0/genesis/test_initialization.py @@ -1,11 +1,11 @@ from eth2spec.test.context import ( - MINIMAL, is_post_altair, single_phase, spec_test, with_configs, with_all_phases, ) +from eth2spec.test.helpers.constants import MINIMAL from eth2spec.test.helpers.deposits import ( prepare_full_genesis_deposits, prepare_random_genesis_deposits, diff --git a/tests/core/pyspec/eth2spec/test/phase0/genesis/test_validity.py b/tests/core/pyspec/eth2spec/test/phase0/genesis/test_validity.py index a7bcaa6d0..91148da1d 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/genesis/test_validity.py +++ b/tests/core/pyspec/eth2spec/test/phase0/genesis/test_validity.py @@ -1,11 +1,11 @@ from eth2spec.test.context import ( - MINIMAL, is_post_altair, spec_test, single_phase, with_configs, with_all_phases, ) +from eth2spec.test.helpers.constants import MINIMAL from eth2spec.test.helpers.deposits import ( prepare_full_genesis_deposits, ) diff --git a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_basic.py b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_basic.py index 82dbccc52..511849f8f 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_basic.py +++ b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_basic.py @@ -1,4 +1,5 @@ -from eth2spec.test.context import PHASE0, with_all_phases, with_phases, spec_state_test +from eth2spec.test.context import with_all_phases, with_phases, spec_state_test +from eth2spec.test.helpers.constants import PHASE0 import eth2spec.test.helpers.rewards as rewards_helpers diff --git a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_leak.py b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_leak.py index f11b2734f..0ad2491f0 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_leak.py +++ b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_leak.py @@ -1,4 +1,5 @@ -from eth2spec.test.context import PHASE0, with_all_phases, with_phases, spec_state_test +from eth2spec.test.context import with_all_phases, with_phases, spec_state_test +from eth2spec.test.helpers.constants import PHASE0 from eth2spec.test.helpers.rewards import leaking import eth2spec.test.helpers.rewards as rewards_helpers diff --git a/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py index 3efe07b01..9b184b018 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py @@ -24,9 +24,8 @@ from eth2spec.test.helpers.multi_operations import ( run_slash_and_exit, run_test_full_random_operations, ) - +from eth2spec.test.helpers.constants import PHASE0, MINIMAL from eth2spec.test.context import ( - PHASE0, MINIMAL, spec_test, spec_state_test, dump_skipping_message, with_phases, with_all_phases, single_phase, expect_assertion_error, always_bls, diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py index 4d5f1ca16..fcc5e2d21 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py @@ -1,6 +1,7 @@ -from eth2spec.test.context import PHASE0, ALTAIR, with_all_phases, spec_state_test +from eth2spec.test.context import with_all_phases, spec_state_test from eth2spec.test.helpers.block import build_empty_block_for_next_slot from eth2spec.test.helpers.attestations import get_valid_attestation, sign_attestation +from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.test.helpers.state import transition_to, state_transition_and_sign_block, next_epoch, next_slot from eth2spec.test.helpers.fork_choice import get_genesis_forkchoice_store diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/validator/test_validator_unittest.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/validator/test_validator_unittest.py index d793db789..fb63839d6 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/validator/test_validator_unittest.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/validator/test_validator_unittest.py @@ -1,8 +1,8 @@ from eth2spec.test.context import ( spec_state_test, always_bls, with_phases, with_all_phases, - PHASE0, ) +from eth2spec.test.helpers.constants import PHASE0 from eth2spec.test.helpers.attestations import build_attestation_data, get_valid_attestation from eth2spec.test.helpers.block import build_empty_block from eth2spec.test.helpers.deposits import prepare_state_and_deposit diff --git a/tests/core/pyspec/eth2spec/test/sharding/unittests/test_get_start_shard.py b/tests/core/pyspec/eth2spec/test/sharding/unittests/test_get_start_shard.py index 41ef754b3..c38f28b2b 100644 --- a/tests/core/pyspec/eth2spec/test/sharding/unittests/test_get_start_shard.py +++ b/tests/core/pyspec/eth2spec/test/sharding/unittests/test_get_start_shard.py @@ -1,8 +1,8 @@ from eth2spec.test.context import ( - SHARDING, with_phases, spec_state_test, ) +from eth2spec.test.helpers.constants import SHARDING from eth2spec.test.helpers.state import next_epoch diff --git a/tests/generators/README.md b/tests/generators/README.md index 5e12e6bfe..731184326 100644 --- a/tests/generators/README.md +++ b/tests/generators/README.md @@ -164,7 +164,7 @@ Another example, to generate tests from pytests: ```python from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators diff --git a/tests/generators/bls/main.py b/tests/generators/bls/main.py index fecc2df7d..3d67c9db3 100644 --- a/tests/generators/bls/main.py +++ b/tests/generators/bls/main.py @@ -12,7 +12,7 @@ from eth_utils import ( import milagro_bls_binding as milagro_bls from eth2spec.utils import bls -from eth2spec.test.context import PHASE0 +from eth2spec.test.helpers.constants import PHASE0 from eth2spec.gen_helpers.gen_base import gen_runner, gen_typing diff --git a/tests/generators/epoch_processing/main.py b/tests/generators/epoch_processing/main.py index e3caf69fa..9efd96534 100644 --- a/tests/generators/epoch_processing/main.py +++ b/tests/generators/epoch_processing/main.py @@ -1,7 +1,7 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR specs = (spec_phase0, spec_altair) diff --git a/tests/generators/finality/main.py b/tests/generators/finality/main.py index 29ab46c39..e50425da4 100644 --- a/tests/generators/finality/main.py +++ b/tests/generators/finality/main.py @@ -1,7 +1,7 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR specs = (spec_phase0, spec_altair) diff --git a/tests/generators/fork_choice/main.py b/tests/generators/fork_choice/main.py index f09bbcc0a..445ee629c 100644 --- a/tests/generators/fork_choice/main.py +++ b/tests/generators/fork_choice/main.py @@ -1,7 +1,7 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR specs = (spec_phase0, spec_altair) diff --git a/tests/generators/forks/main.py b/tests/generators/forks/main.py index 3bec19735..71a4a54de 100644 --- a/tests/generators/forks/main.py +++ b/tests/generators/forks/main.py @@ -1,7 +1,7 @@ from importlib import reload from typing import Iterable -from eth2spec.test.context import PHASE0, ALTAIR, MINIMAL, MAINNET +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MINIMAL, MAINNET from eth2spec.config import config_util from eth2spec.test.altair.fork import test_fork as test_altair_forks from eth2spec.phase0 import spec as spec_phase0 diff --git a/tests/generators/genesis/main.py b/tests/generators/genesis/main.py index 362677c24..3b4d045a3 100644 --- a/tests/generators/genesis/main.py +++ b/tests/generators/genesis/main.py @@ -1,7 +1,7 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR specs = (spec_phase0, spec_altair) diff --git a/tests/generators/operations/main.py b/tests/generators/operations/main.py index f00c4f894..0c1b84c24 100644 --- a/tests/generators/operations/main.py +++ b/tests/generators/operations/main.py @@ -1,7 +1,7 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR specs = (spec_phase0, spec_altair) diff --git a/tests/generators/rewards/main.py b/tests/generators/rewards/main.py index d8115f4cb..2a0f14863 100644 --- a/tests/generators/rewards/main.py +++ b/tests/generators/rewards/main.py @@ -1,7 +1,7 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR specs = (spec_phase0, spec_altair) diff --git a/tests/generators/sanity/main.py b/tests/generators/sanity/main.py index 90f9a00df..7f7fecc67 100644 --- a/tests/generators/sanity/main.py +++ b/tests/generators/sanity/main.py @@ -1,6 +1,6 @@ from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators diff --git a/tests/generators/shuffling/main.py b/tests/generators/shuffling/main.py index 091207bca..73b714760 100644 --- a/tests/generators/shuffling/main.py +++ b/tests/generators/shuffling/main.py @@ -6,7 +6,7 @@ from eth2spec.gen_helpers.gen_base import gen_runner, gen_typing from eth2spec.config import config_util from eth2spec.phase0 import spec as spec -from eth2spec.test.context import PHASE0 +from eth2spec.test.helpers.constants import PHASE0 def shuffling_case_fn(seed, count): diff --git a/tests/generators/ssz_generic/main.py b/tests/generators/ssz_generic/main.py index 737f8cda6..e6cd3d976 100644 --- a/tests/generators/ssz_generic/main.py +++ b/tests/generators/ssz_generic/main.py @@ -6,7 +6,7 @@ import ssz_bitvector import ssz_boolean import ssz_uints import ssz_container -from eth2spec.test.context import PHASE0 +from eth2spec.test.helpers.constants import PHASE0 def create_provider(handler_name: str, suite_name: str, case_maker) -> gen_typing.TestProvider: diff --git a/tests/generators/ssz_static/main.py b/tests/generators/ssz_static/main.py index 77a88a8cd..617fb2f02 100644 --- a/tests/generators/ssz_static/main.py +++ b/tests/generators/ssz_static/main.py @@ -9,7 +9,7 @@ from eth2spec.debug import random_value, encode from eth2spec.config import config_util from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.context import ALTAIR, TESTGEN_FORKS, MINIMAL, MAINNET +from eth2spec.test.helpers.constants import ALTAIR, TESTGEN_FORKS, MINIMAL, MAINNET from eth2spec.utils.ssz.ssz_typing import Container from eth2spec.utils.ssz.ssz_impl import ( hash_tree_root, From 4bf6331c219b09ca7dc633afee167b1f50c7a5d8 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 12 Apr 2021 15:11:21 -0700 Subject: [PATCH 035/227] update remerkleable --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 6d9147661..74a96eea8 100644 --- a/setup.py +++ b/setup.py @@ -607,7 +607,7 @@ setup( "py_ecc==5.2.0", "milagro_bls_binding==1.6.3", "dataclasses==0.6", - "remerkleable==0.1.18", + "remerkleable==0.1.19", "ruamel.yaml==0.16.5", "lru-dict==1.1.6", ] From 3ea897d531cdf1b3446b0724dc7db03db42120ed Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 12 Apr 2021 15:27:04 -0700 Subject: [PATCH 036/227] Add sync committee aggregation constant to config --- configs/mainnet/altair.yaml | 6 ++++++ configs/minimal/altair.yaml | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 2a2552da4..475cea732 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -51,3 +51,9 @@ MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 MAX_VALID_LIGHT_CLIENT_UPDATES: 8192 # 2**13 (=8192) LIGHT_CLIENT_UPDATE_TIMEOUT: 8192 + + +# Validator +# --------------------------------------------------------------- +# 2**2 (= 4) +TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE: 4 diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index b852e13c0..096941721 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -51,3 +51,8 @@ MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 MAX_VALID_LIGHT_CLIENT_UPDATES: 32 # [customized] LIGHT_CLIENT_UPDATE_TIMEOUT: 32 + +# Validator +# --------------------------------------------------------------- +# 2**2 (= 4) +TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE: 4 From 44de07fee924fd010f73a1b4f83086cb1782bb23 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Tue, 13 Apr 2021 15:20:45 +0600 Subject: [PATCH 037/227] Replace hash: Hash32 with block_hash: Hash32 Co-authored-by: Hsiao-Wei Wang --- specs/merge/fork-choice.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index df45bedd1..ae3917814 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -38,7 +38,7 @@ class PowBlock(Container): #### `get_pow_block` -Let `get_pow_block(hash: Hash32) -> PowBlock` be the function that given the hash of the PoW block returns its data. +Let `get_pow_block(block_hash: Hash32) -> PowBlock` be the function that given the hash of the PoW block returns its data. *Note*: The `eth_getBlockByHash` JSON-RPC method does not distinguish invalid blocks from blocks that haven't been processed yet. Either extending this existing method or implementing a new one is required. @@ -113,4 +113,3 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: if ancestor_at_finalized_slot != store.finalized_checkpoint.root: store.justified_checkpoint = state.current_justified_checkpoint ``` - From 13edd20a365d8ac8f018864b806d1552fd2fcef5 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Tue, 13 Apr 2021 15:29:07 +0600 Subject: [PATCH 038/227] Change Eth1Data.block_hash type to Hash32 --- specs/merge/beacon-chain.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 98312d246..3a13c1a3d 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -16,6 +16,8 @@ - [Transition](#transition) - [Execution](#execution) - [Containers](#containers) + - [Updated containers](#updated-containers) + - [`Eth1Data`](#eth1data) - [Extended containers](#extended-containers) - [`BeaconBlockBody`](#beaconblockbody) - [`BeaconState`](#beaconstate) @@ -67,6 +69,19 @@ We define the following Python custom types for type hinting and readability: ## Containers +### Modified containers + +#### `Eth1Data` + +*Note*: The only modification is the type of `block_hash` field that is changed to `Hash32`. + +```python +class Eth1Data(Container): + deposit_root: Root + deposit_count: uint64 + block_hash: Hash32 # [Modified in Merge] +``` + ### Extended containers *Note*: Extended SSZ containers inherit all fields from the parent in the original From ad0f1e56207b9170d6609f57aa4cf552f8e3bc49 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Tue, 13 Apr 2021 19:08:47 +0600 Subject: [PATCH 039/227] Add timestamp field into ExecutionPayload --- specs/merge/beacon-chain.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 3a13c1a3d..ade05d4b8 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -121,6 +121,7 @@ class ExecutionPayload(Container): number: uint64 gas_limit: uint64 gas_used: uint64 + timestamp: uint64 receipt_root: Bytes32 logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] transactions: List[OpaqueTransaction, MAX_APPLICATION_TRANSACTIONS] @@ -141,6 +142,7 @@ class ExecutionPayloadHeader(Container): number: uint64 gas_limit: uint64 gas_used: uint64 + timestamp: uint64 receipt_root: Bytes32 logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] transactions_root: Root @@ -187,7 +189,7 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: ##### `execution_state_transition` -Let `execution_state_transition(execution_state_root: Bytes32, execution_payload: ExecutionPayload, timestamp: uint64) -> None` be the transition function of Ethereum execution state. +Let `execution_state_transition(execution_state_root: Bytes32, execution_payload: ExecutionPayload) -> None` be the transition function of Ethereum execution state. The body of the function is implementation dependent. *Note*: `execution_state_transition` must throw `AssertionError` if either the transition itself or one of the pre or post conditions has failed. @@ -209,9 +211,10 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None assert execution_payload.parent_hash == state.latest_execution_payload_header.block_hash assert execution_payload.number == state.latest_execution_payload_header.number + 1 - timestamp = compute_time_at_slot(state, state.slot) + assert execution_payload.timestamp == compute_time_at_slot(state, state.slot) + execution_state_root = state.latest_execution_payload_header.state_root - execution_state_transition(execution_state_root, body.execution_payload, timestamp) + execution_state_transition(execution_state_root, body.execution_payload) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, @@ -221,6 +224,7 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None number=execution_payload.number, gas_limit=execution_payload.gas_limit, gas_used=execution_payload.gas_used, + timestamp=execution_payload.timestamp, receipt_root=execution_payload.receipt_root, logs_bloom=execution_payload.logs_bloom, transactions_root=hash_tree_root(execution_payload.transactions), From dbbc63b7a3857fbda74b194b238b6ef7a5179f40 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Tue, 13 Apr 2021 19:13:54 +0600 Subject: [PATCH 040/227] Replace execution_state_transition with validate_execution_payload --- specs/merge/beacon-chain.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index ade05d4b8..2dc1bbc84 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -16,7 +16,7 @@ - [Transition](#transition) - [Execution](#execution) - [Containers](#containers) - - [Updated containers](#updated-containers) + - [Modified containers](#modified-containers) - [`Eth1Data`](#eth1data) - [Extended containers](#extended-containers) - [`BeaconBlockBody`](#beaconblockbody) @@ -31,7 +31,7 @@ - [`compute_time_at_slot`](#compute_time_at_slot) - [Block processing](#block-processing) - [Execution payload processing](#execution-payload-processing) - - [`execution_state_transition`](#execution_state_transition) + - [`validate_execution_payload`](#validate_execution_payload) - [`process_execution_payload`](#process_execution_payload) @@ -187,13 +187,11 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: #### Execution payload processing -##### `execution_state_transition` +##### `validate_execution_payload` -Let `execution_state_transition(execution_state_root: Bytes32, execution_payload: ExecutionPayload) -> None` be the transition function of Ethereum execution state. +Let `validate_execution_payload(execution_payload: ExecutionPayload) -> boolean` be the function checking whether given `ExecutionPayload` is valid or not. The body of the function is implementation dependent. -*Note*: `execution_state_transition` must throw `AssertionError` if either the transition itself or one of the pre or post conditions has failed. - ##### `process_execution_payload` ```python @@ -213,8 +211,7 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None assert execution_payload.timestamp == compute_time_at_slot(state, state.slot) - execution_state_root = state.latest_execution_payload_header.state_root - execution_state_transition(execution_state_root, body.execution_payload) + assert validate_execution_payload(execution_payload) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, From 7d8570d48871b82e260ca09454299fc5edf7ef2a Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 14 Apr 2021 12:53:30 +0600 Subject: [PATCH 041/227] Warn about potential overflows in compute_time_at_slot --- specs/merge/beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 2dc1bbc84..028eb5b84 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -168,6 +168,8 @@ def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> bool #### `compute_time_at_slot` +*Note*: This function is unsafe with respect to overflows and underflows. + ```python def compute_time_at_slot(state: BeaconState, slot: Slot) -> uint64: slots_since_genesis = slot - GENESIS_SLOT From 292fd604f8f6073c891694fa3794c98634b9e6b4 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 14 Apr 2021 12:54:49 +0600 Subject: [PATCH 042/227] Replace boolean with bool whenever make sense --- specs/merge/beacon-chain.md | 6 +++--- specs/merge/fork-choice.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 028eb5b84..688f36b47 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -155,14 +155,14 @@ class ExecutionPayloadHeader(Container): #### `is_transition_completed` ```python -def is_transition_completed(state: BeaconState) -> boolean: +def is_transition_completed(state: BeaconState) -> bool: return state.latest_execution_payload_header != ExecutionPayloadHeader() ``` #### `is_transition_block` ```python -def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> boolean: +def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> bool: return not is_transition_completed(state) and block_body.execution_payload != ExecutionPayload() ``` @@ -191,7 +191,7 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: ##### `validate_execution_payload` -Let `validate_execution_payload(execution_payload: ExecutionPayload) -> boolean` be the function checking whether given `ExecutionPayload` is valid or not. +Let `validate_execution_payload(execution_payload: ExecutionPayload) -> bool` be the function checking whether given `ExecutionPayload` is valid or not. The body of the function is implementation dependent. ##### `process_execution_payload` diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index ae3917814..34647f45d 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -47,7 +47,7 @@ Let `get_pow_block(block_hash: Hash32) -> PowBlock` be the function that given t Used by fork-choice handler, `on_block`. ```python -def is_valid_transition_block(block: PowBlock) -> boolean: +def is_valid_transition_block(block: PowBlock) -> bool: is_total_difficulty_reached = block.total_difficulty >= TRANSITION_TOTAL_DIFFICULTY return block.is_valid and is_total_difficulty_reached ``` From 9d79831b568ace7e06b9b52db23bb083632a5980 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 14 Apr 2021 13:03:58 +0600 Subject: [PATCH 043/227] Rename validate_execution_payload to verify_execution_state_transition --- specs/merge/beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 688f36b47..4e0c554d2 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -31,7 +31,7 @@ - [`compute_time_at_slot`](#compute_time_at_slot) - [Block processing](#block-processing) - [Execution payload processing](#execution-payload-processing) - - [`validate_execution_payload`](#validate_execution_payload) + - [`verify_execution_state_transition`](#verify_execution_state_transition) - [`process_execution_payload`](#process_execution_payload) @@ -189,9 +189,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: #### Execution payload processing -##### `validate_execution_payload` +##### `verify_execution_state_transition` -Let `validate_execution_payload(execution_payload: ExecutionPayload) -> bool` be the function checking whether given `ExecutionPayload` is valid or not. +Let `verify_execution_state_transition(execution_payload: ExecutionPayload) -> bool` be the function that verifies given `ExecutionPayload` with respect to execution state transition. The body of the function is implementation dependent. ##### `process_execution_payload` @@ -213,7 +213,7 @@ def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None assert execution_payload.timestamp == compute_time_at_slot(state, state.slot) - assert validate_execution_payload(execution_payload) + assert verify_execution_state_transition(execution_payload) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, From 320172fb08a0dad653fa3a4a67dce92bce5747b7 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 14 Apr 2021 08:02:09 -0500 Subject: [PATCH 044/227] fix lint --- setup.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/setup.py b/setup.py index b337d70ff..6490f4ffa 100644 --- a/setup.py +++ b/setup.py @@ -313,10 +313,8 @@ def get_pow_chain_head() -> PowBlock: pass -def execution_state_transition(execution_state_root: Bytes32, - execution_payload: ExecutionPayload, - timestamp: uint64) -> None: - pass +def verify_execution_state_transition(execution_payload: ExecutionPayload) -> bool: + return True def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> ExecutionPayload: From 35fea7a11b9239e23491f9abfc050d887c8dbd92 Mon Sep 17 00:00:00 2001 From: BenSchZA Date: Wed, 14 Apr 2021 21:37:25 +0200 Subject: [PATCH 045/227] Update copy INVAIANT to INVARIANT --- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 0a64307c0..0a95fbb5a 100644 --- a/setup.py +++ b/setup.py @@ -329,7 +329,7 @@ ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS = { } -ALTAIR_INVAIANT_CHECKS = ''' +ALTAIR_INVARIANT_CHECKS = ''' assert ( TIMELY_HEAD_WEIGHT + TIMELY_SOURCE_WEIGHT + TIMELY_TARGET_WEIGHT + SYNC_REWARD_WEIGHT + PROPOSER_WEIGHT ) == WEIGHT_DENOMINATOR''' @@ -414,7 +414,7 @@ def objects_to_spec(spec_object: SpecObject, imports: str, fork: str, ordered_cl if is_altair(fork): altair_ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), ALTAIR_HARDCODED_SSZ_DEP_CONSTANTS)) spec += '\n\n\n' + altair_ssz_dep_constants_verification - spec += '\n' + ALTAIR_INVAIANT_CHECKS + spec += '\n' + ALTAIR_INVARIANT_CHECKS spec += '\n' return spec From bb63af53c9d1acd66eb5ab13ace6bb3fb71d320c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 13 Apr 2021 23:21:40 +0800 Subject: [PATCH 046/227] Refactor genesis.py --- tests/core/pyspec/eth2spec/test/context.py | 8 +++----- tests/core/pyspec/eth2spec/test/helpers/constants.py | 4 +++- tests/core/pyspec/eth2spec/test/helpers/genesis.py | 12 +++++++++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 147793afe..438e611cf 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -7,8 +7,8 @@ from eth2spec.utils import bls from .exceptions import SkippedTest from .helpers.constants import ( - PHASE0, ALTAIR, MERGE, SHARDING, CUSTODY_GAME, DAS, - ALL_PHASES, + PHASE0, ALTAIR, + ALL_PHASES, FORKS_BEFORE_ALTAIR, ) from .helpers.genesis import create_genesis_state from .utils import vector_test, with_meta_tags @@ -362,8 +362,6 @@ def with_configs(configs, reason=None): def is_post_altair(spec): - # TODO: everything runs in parallel to Altair. - # After features are rebased on the Altair fork, this can be reduced to just PHASE0. - if spec.fork in [PHASE0, MERGE, SHARDING, CUSTODY_GAME, DAS]: + if spec.fork in FORKS_BEFORE_ALTAIR: return False return True diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index bffffe348..ccd7b20a2 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -18,7 +18,9 @@ DAS = SpecForkName('das') ALL_PHASES = (PHASE0, ALTAIR) # The forks that output to the test vectors. TESTGEN_FORKS = (PHASE0, ALTAIR) - +# TODO: everything runs in parallel to Altair. +# After features are rebased on the Altair fork, this can be reduced to just PHASE0. +FORKS_BEFORE_ALTAIR = (PHASE0, MERGE, SHARDING, CUSTODY_GAME, DAS) # # Config diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 5f2cd5281..49af43ec1 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -1,3 +1,7 @@ +from eth2spec.test.helpers.constants import ( + ALTAIR, + FORKS_BEFORE_ALTAIR, +) from eth2spec.test.helpers.keys import pubkeys @@ -21,8 +25,10 @@ def create_genesis_state(spec, validator_balances, activation_threshold): eth1_block_hash = b'\xda' * 32 current_version = spec.GENESIS_FORK_VERSION - if spec.fork == 'altair': # TODO: fix `context.py` dependency + + if spec.fork == ALTAIR: current_version = spec.ALTAIR_FORK_VERSION + state = spec.BeaconState( genesis_time=0, eth1_deposit_index=len(validator_balances), @@ -50,7 +56,7 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if validator.effective_balance >= activation_threshold: validator.activation_eligibility_epoch = spec.GENESIS_EPOCH validator.activation_epoch = spec.GENESIS_EPOCH - if spec.fork != 'phase0': # TODO: fix `context.py` dependency + if spec.fork not in FORKS_BEFORE_ALTAIR: state.previous_epoch_participation.append(spec.ParticipationFlags(0b0000_0000)) state.current_epoch_participation.append(spec.ParticipationFlags(0b0000_0000)) state.inactivity_scores.append(spec.uint64(0)) @@ -58,7 +64,7 @@ def create_genesis_state(spec, validator_balances, activation_threshold): # Set genesis validators root for domain separation and chain versioning state.genesis_validators_root = spec.hash_tree_root(state.validators) - if spec.fork != 'phase0': # TODO: fix `context.py` dependency + if spec.fork not in FORKS_BEFORE_ALTAIR: # Fill in sync committees state.current_sync_committee = spec.get_sync_committee(state, spec.get_current_epoch(state)) state.next_sync_committee = ( From b65566f184f3ed84381f3e2581c246a84480b1ea Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 15 Apr 2021 15:41:12 +0800 Subject: [PATCH 047/227] Fix ToC --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e14a0f5c0..f46df6bb2 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -52,7 +52,7 @@ - [Slashings](#slashings) - [Participation flags updates](#participation-flags-updates) - [Sync committee updates](#sync-committee-updates) -- [Initialize state for testnets](#initialize-state-for-testnets) +- [Initialize state for Altair testnets](#initialize-state-for-altair-testnets) From 26f54d6f22cb561afcb1fa830d47eb4efb441d7e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 15 Apr 2021 15:49:29 +0800 Subject: [PATCH 048/227] Remove `CONFIG_NAME` from the config files. And add it back to pyspec context for testing --- configs/mainnet/altair.yaml | 2 -- configs/mainnet/custody_game.yaml | 2 -- configs/mainnet/merge.yaml | 2 -- configs/mainnet/phase0.yaml | 2 -- configs/mainnet/sharding.yaml | 2 -- configs/minimal/altair.yaml | 2 -- configs/minimal/custody_game.yaml | 2 -- configs/minimal/merge.yaml | 2 -- configs/minimal/phase0.yaml | 2 -- configs/minimal/sharding.yaml | 2 -- tests/core/pyspec/eth2spec/config/config_util.py | 3 +-- 11 files changed, 1 insertion(+), 22 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 475cea732..44490f982 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -1,7 +1,5 @@ # Mainnet preset - Altair -CONFIG_NAME: "mainnet" - # Updated penalty values # --------------------------------------------------------------- # 3 * 2**24 (= 50,331,648) diff --git a/configs/mainnet/custody_game.yaml b/configs/mainnet/custody_game.yaml index 8039d839b..ecb2dc377 100644 --- a/configs/mainnet/custody_game.yaml +++ b/configs/mainnet/custody_game.yaml @@ -1,7 +1,5 @@ # Mainnet preset - Custody Game -CONFIG_NAME: "mainnet" - # Time parameters # --------------------------------------------------------------- # 2**1 (= 2) epochs, 12.8 minutes diff --git a/configs/mainnet/merge.yaml b/configs/mainnet/merge.yaml index 582e7d642..b4667f5b5 100644 --- a/configs/mainnet/merge.yaml +++ b/configs/mainnet/merge.yaml @@ -1,7 +1,5 @@ # Mainnet preset - The Merge -CONFIG_NAME: "mainnet" - # Fork # --------------------------------------------------------------- MERGE_FORK_VERSION: 0x02000000 diff --git a/configs/mainnet/phase0.yaml b/configs/mainnet/phase0.yaml index ace44dd23..8b902f1c3 100644 --- a/configs/mainnet/phase0.yaml +++ b/configs/mainnet/phase0.yaml @@ -1,7 +1,5 @@ # Mainnet preset -CONFIG_NAME: "mainnet" - # Misc # --------------------------------------------------------------- # 2**6 (= 64) diff --git a/configs/mainnet/sharding.yaml b/configs/mainnet/sharding.yaml index 0e59a674b..4773aa5f5 100644 --- a/configs/mainnet/sharding.yaml +++ b/configs/mainnet/sharding.yaml @@ -1,7 +1,5 @@ # Mainnet preset - Sharding -CONFIG_NAME: "mainnet" - # Fork # --------------------------------------------------------------- SHARDING_FORK_VERSION: 0x03000000 diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index 096941721..10bdf318b 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -1,7 +1,5 @@ # Minimal preset - Altair -CONFIG_NAME: "minimal" - # Updated penalty values # --------------------------------------------------------------- # 3 * 2**24 (= 50,331,648) diff --git a/configs/minimal/custody_game.yaml b/configs/minimal/custody_game.yaml index 1d9393e80..8b8992fb6 100644 --- a/configs/minimal/custody_game.yaml +++ b/configs/minimal/custody_game.yaml @@ -1,7 +1,5 @@ # Minimal preset - Custody Game -CONFIG_NAME: "minimal" - # Time parameters # --------------------------------------------------------------- # 2**1 (= 2) epochs, 12.8 minutes diff --git a/configs/minimal/merge.yaml b/configs/minimal/merge.yaml index e2857917f..394595d02 100644 --- a/configs/minimal/merge.yaml +++ b/configs/minimal/merge.yaml @@ -1,7 +1,5 @@ # Minimal preset - The Merge -CONFIG_NAME: "minimal" - # Fork # --------------------------------------------------------------- MERGE_FORK_VERSION: 0x02000001 diff --git a/configs/minimal/phase0.yaml b/configs/minimal/phase0.yaml index 93337adf8..3624a2702 100644 --- a/configs/minimal/phase0.yaml +++ b/configs/minimal/phase0.yaml @@ -1,7 +1,5 @@ # Minimal preset -CONFIG_NAME: "minimal" - # Misc # --------------------------------------------------------------- diff --git a/configs/minimal/sharding.yaml b/configs/minimal/sharding.yaml index ca1cc1d6b..f32e2827d 100644 --- a/configs/minimal/sharding.yaml +++ b/configs/minimal/sharding.yaml @@ -1,7 +1,5 @@ # Minimal preset - Sharding -CONFIG_NAME: "minimal" - # Fork # --------------------------------------------------------------- SHARDING_FORK_VERSION: 0x03000001 diff --git a/tests/core/pyspec/eth2spec/config/config_util.py b/tests/core/pyspec/eth2spec/config/config_util.py index ee2007600..917cf3a60 100644 --- a/tests/core/pyspec/eth2spec/config/config_util.py +++ b/tests/core/pyspec/eth2spec/config/config_util.py @@ -54,8 +54,7 @@ def load_config_file(configs_dir: str, presets_name: str) -> Dict[str, Any]: out[k] = [int(item) if item.isdigit() else item for item in v] elif isinstance(v, str) and v.startswith("0x"): out[k] = bytes.fromhex(v[2:]) - elif k == "CONFIG_NAME": - out[k] = str(v) else: out[k] = int(v) + out['CONFIG_NAME'] = presets_name return out From c7166a37af400ba174c70db88ceeaa690685f873 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 15 Apr 2021 06:47:11 -0500 Subject: [PATCH 049/227] change eth1data block_hash type to Hash32 in phase 0 --- specs/merge/beacon-chain.md | 16 ---------------- specs/phase0/beacon-chain.md | 4 +++- 2 files changed, 3 insertions(+), 17 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 4e0c554d2..a508cb670 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -16,8 +16,6 @@ - [Transition](#transition) - [Execution](#execution) - [Containers](#containers) - - [Modified containers](#modified-containers) - - [`Eth1Data`](#eth1data) - [Extended containers](#extended-containers) - [`BeaconBlockBody`](#beaconblockbody) - [`BeaconState`](#beaconstate) @@ -49,7 +47,6 @@ We define the following Python custom types for type hinting and readability: | Name | SSZ equivalent | Description | | - | - | - | | `OpaqueTransaction` | `ByteList[MAX_BYTES_PER_OPAQUE_TRANSACTION]` | a byte-list containing a single [typed transaction envelope](https://eips.ethereum.org/EIPS/eip-2718#opaque-byte-array-rather-than-an-rlp-array) structured as `TransactionType \|\| TransactionPayload` | -| `Hash32` | `Bytes32` | a 256-bit hash | ## Constants @@ -69,19 +66,6 @@ We define the following Python custom types for type hinting and readability: ## Containers -### Modified containers - -#### `Eth1Data` - -*Note*: The only modification is the type of `block_hash` field that is changed to `Hash32`. - -```python -class Eth1Data(Container): - deposit_root: Root - deposit_count: uint64 - block_hash: Hash32 # [Modified in Merge] -``` - ### Extended containers *Note*: Extended SSZ containers inherit all fields from the parent in the original diff --git a/specs/phase0/beacon-chain.md b/specs/phase0/beacon-chain.md index cbd085bd3..3176d543c 100644 --- a/specs/phase0/beacon-chain.md +++ b/specs/phase0/beacon-chain.md @@ -157,6 +157,7 @@ We define the following Python custom types for type hinting and readability: | `ValidatorIndex` | `uint64` | a validator registry index | | `Gwei` | `uint64` | an amount in Gwei | | `Root` | `Bytes32` | a Merkle root | +| `Hash32` | `Bytes32` | a 256-bit hash | | `Version` | `Bytes4` | a fork version number | | `DomainType` | `Bytes4` | a domain type | | `ForkDigest` | `Bytes4` | a digest of the current fork data | @@ -164,6 +165,7 @@ We define the following Python custom types for type hinting and readability: | `BLSPubkey` | `Bytes48` | a BLS12-381 public key | | `BLSSignature` | `Bytes96` | a BLS12-381 signature | + ## Constants The following values are (non-configurable) constants used throughout the specification. @@ -374,7 +376,7 @@ class PendingAttestation(Container): class Eth1Data(Container): deposit_root: Root deposit_count: uint64 - block_hash: Bytes32 + block_hash: Hash32 ``` #### `HistoricalBatch` From 81a83898cf29e25ce536f2fadcf19c0362ab893e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 15 Apr 2021 12:19:51 -0500 Subject: [PATCH 050/227] add committee progress tests for non genesis case --- .../test_process_sync_committee_updates.py | 52 ++++++++++++++----- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index 7ba2645a2..c5bd9443e 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -1,7 +1,11 @@ from eth2spec.test.context import ( spec_state_test, + spec_test, with_all_phases_except, with_configs, + with_custom_state, + single_phase, + misc_balances, ) from eth2spec.test.helpers.constants import ( PHASE0, @@ -13,20 +17,12 @@ from eth2spec.test.helpers.epoch_processing import ( ) -@with_all_phases_except([PHASE0]) -@spec_state_test -@with_configs([MINIMAL], reason="too slow") -def test_sync_committees_progress(spec, state): - current_epoch = spec.get_current_epoch(state) - # NOTE: if not in the genesis epoch, period math below needs to be - # adjusted relative to the current epoch - assert current_epoch == 0 - +def run_sync_committees_progress_test(spec, state): first_sync_committee = state.current_sync_committee second_sync_committee = state.next_sync_committee - slot_at_end_of_current_period = spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD * spec.SLOTS_PER_EPOCH - 1 - transition_to(spec, state, slot_at_end_of_current_period) + end_slot_of_current_period = state.slot + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD * spec.SLOTS_PER_EPOCH - 1 + transition_to(spec, state, end_slot_of_current_period) # Ensure assignments have not changed: assert state.current_sync_committee == first_sync_committee @@ -36,7 +32,39 @@ def test_sync_committees_progress(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` - third_sync_committee = spec.get_sync_committee(state, 2 * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_epoch = spec.get_current_epoch(state) + third_sync_committee = spec.get_sync_committee(state, current_epoch + 2 * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee + + +@with_all_phases_except([PHASE0]) +@spec_state_test +@with_configs([MINIMAL], reason="too slow") +def test_sync_committees_progress_genesis(spec, state): + # Genesis epoch period has an exceptional case + spec.get_current_epoch(state) + assert spec.get_current_epoch(state) == spec.GENESIS_EPOCH + + yield from run_sync_committees_progress_test(spec, state) + + +@with_all_phases_except([PHASE0]) +@spec_state_test +@with_configs([MINIMAL], reason="too slow") +def test_sync_committees_progress_not_genesis(spec, state): + # Transition out of the genesis epoch period to test non-exceptional case + start_slot_of_next_period = state.slot + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD * spec.SLOTS_PER_EPOCH + transition_to(spec, state, start_slot_of_next_period) + + yield from run_sync_committees_progress_test(spec, state) + + +@with_all_phases_except([PHASE0]) +@with_custom_state(balances_fn=misc_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) +@spec_test +@single_phase +@with_configs([MINIMAL], reason="too slow") +def test_sync_committees_progress_misc_balances(spec, state): + yield from run_sync_committees_progress_test(spec, state) From 5349645a8f012c38dc7abe2e060a593c4ab409d0 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 16 Apr 2021 11:29:10 +0800 Subject: [PATCH 051/227] Rename `SpecAdjustment` to `SpecBuilder` and add `build_spec` interface --- setup.py | 59 ++++++++++++++++++++++++++++++++------------------------ 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/setup.py b/setup.py index c6e9f55ae..66a1b7cfe 100644 --- a/setup.py +++ b/setup.py @@ -5,7 +5,7 @@ from distutils import dir_util from distutils.util import convert_path import os import re -from typing import Dict, NamedTuple, List +from typing import Dict, NamedTuple, List, Sequence from abc import ABC, abstractmethod @@ -131,7 +131,7 @@ def get_spec(file_name: str) -> SpecObject: ) -class SpecAdjustment(ABC): +class SpecBuilder(ABC): @property @abstractmethod def fork(self) -> str: @@ -185,11 +185,16 @@ class SpecAdjustment(ABC): """ raise NotImplementedError() + @classmethod + @abstractmethod + def build_spec(cls, source_files: List[str]) -> str: + raise NotImplementedError() + # -# Phase0SpecAdjustment +# Phase0SpecBuilder # -class Phase0SpecAdjustment(SpecAdjustment): +class Phase0SpecBuilder(SpecBuilder): fork: str = PHASE0 @classmethod @@ -305,11 +310,15 @@ get_attesting_indices = cache_this( def invariant_checks(cls) -> str: return '' + @classmethod + def build_spec(cls, source_files: Sequence[str]) -> str: + return _build_spec(cls.fork, source_files) + # -# AltairSpecAdjustment +# AltairSpecBuilder # -class AltairSpecAdjustment(Phase0SpecAdjustment): +class AltairSpecBuilder(Phase0SpecBuilder): fork: str = ALTAIR @classmethod @@ -359,9 +368,9 @@ assert ( # -# MergeSpecAdjustment +# MergeSpecBuilder # -class MergeSpecAdjustment(Phase0SpecAdjustment): +class MergeSpecBuilder(Phase0SpecBuilder): fork: str = MERGE @classmethod @@ -412,13 +421,13 @@ def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> Executi return {**super().hardcoded_custom_type_dep_constants(), **constants} -spec_adjustments = { - adjustment.fork: adjustment - for adjustment in (Phase0SpecAdjustment, AltairSpecAdjustment, MergeSpecAdjustment) +spec_builders = { + builder.fork: builder + for builder in (Phase0SpecBuilder, AltairSpecBuilder, MergeSpecBuilder) } -def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, ordered_class_objects: Dict[str, str]) -> str: +def objects_to_spec(spec_object: SpecObject, builder: SpecBuilder, ordered_class_objects: Dict[str, str]) -> str: """ Given all the objects that constitute a spec, combine them into a single pyfile. """ @@ -448,13 +457,13 @@ def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, ordered spec_object.constants[k] += " # noqa: E501" constants_spec = '\n'.join(map(lambda x: '%s = %s' % (x, spec_object.constants[x]), spec_object.constants)) ordered_class_objects_spec = '\n\n'.join(ordered_class_objects.values()) - ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, adjustment.hardcoded_ssz_dep_constants()[x]), adjustment.hardcoded_ssz_dep_constants())) - ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), adjustment.hardcoded_ssz_dep_constants())) - custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, adjustment.hardcoded_custom_type_dep_constants()[x]), adjustment.hardcoded_custom_type_dep_constants())) + ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, builder.hardcoded_ssz_dep_constants()[x]), builder.hardcoded_ssz_dep_constants())) + ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), builder.hardcoded_ssz_dep_constants())) + custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, builder.hardcoded_custom_type_dep_constants()[x]), builder.hardcoded_custom_type_dep_constants())) spec = ( - adjustment.imports() - + adjustment.preparations() - + '\n\n' + f"fork = \'{adjustment.fork}\'\n" + builder.imports() + + builder.preparations() + + '\n\n' + f"fork = \'{builder.fork}\'\n" # The constants that some SSZ containers require. Need to be defined before `new_type_definitions` + ('\n\n' + custom_type_dep_constants + '\n' if custom_type_dep_constants != '' else '') + '\n\n' + new_type_definitions @@ -465,11 +474,11 @@ def objects_to_spec(spec_object: SpecObject, adjustment: SpecAdjustment, ordered + '\n\n' + CONFIG_LOADER + '\n\n' + ordered_class_objects_spec + '\n\n' + functions_spec - + '\n' + adjustment.sundry_functions() + + '\n' + builder.sundry_functions() # Since some constants are hardcoded in setup.py, the following assertions verify that the hardcoded constants are # as same as the spec definition. + ('\n\n\n' + ssz_dep_constants_verification if ssz_dep_constants_verification != '' else '') - + ('\n' + adjustment.invariant_checks() if adjustment.invariant_checks() != '' else '') + + ('\n' + builder.invariant_checks() if builder.invariant_checks() != '' else '') + '\n' ) return spec @@ -551,7 +560,7 @@ def combine_spec_objects(spec0: SpecObject, spec1: SpecObject) -> SpecObject: ) -def build_spec(fork: str, source_files: List[str]) -> str: +def _build_spec(fork: str, source_files: Sequence[str]) -> str: all_specs = [get_spec(spec) for spec in source_files] spec_object = all_specs[0] @@ -561,7 +570,7 @@ def build_spec(fork: str, source_files: List[str]) -> str: class_objects = {**spec_object.ssz_objects, **spec_object.dataclasses} dependency_order_class_objects(class_objects, spec_object.custom_types) - return objects_to_spec(spec_object, spec_adjustments[fork], class_objects) + return objects_to_spec(spec_object, spec_builders[fork], class_objects) class PySpecCommand(Command): @@ -631,7 +640,7 @@ class PySpecCommand(Command): raise Exception('Pyspec markdown input file "%s" does not exist.' % filename) def run(self): - spec_str = build_spec(self.spec_fork, self.parsed_md_doc_paths) + spec_str = spec_builders[self.spec_fork].build_spec(self.parsed_md_doc_paths) if self.dry_run: self.announce('dry run successfully prepared contents for spec.' f' out dir: "{self.out_dir}", spec fork: "{self.spec_fork}"') @@ -659,7 +668,7 @@ class BuildPyCommand(build_py): self.run_command('pyspec') def run(self): - for spec_fork in spec_adjustments: + for spec_fork in spec_builders: self.run_pyspec_cmd(spec_fork=spec_fork) super(BuildPyCommand, self).run() @@ -687,7 +696,7 @@ class PyspecDevCommand(Command): def run(self): print("running build_py command") - for spec_fork in spec_adjustments: + for spec_fork in spec_builders: self.run_pyspec_cmd(spec_fork=spec_fork) commands = { From 57ac8c3f281ff3cf0ad2b585995564aee817c320 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 16 Apr 2021 11:34:50 +0800 Subject: [PATCH 052/227] Fix doc --- specs/altair/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f46df6bb2..978a7eadd 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -665,7 +665,7 @@ def process_sync_committee_updates(state: BeaconState) -> None: ## Initialize state for Altair testnets -This helper function is only for initializing the pure Altair testnets and tests, where we use the dummy where the `ALTAIR_FORK_SLOT` GENESIS_SLOT`. +This helper function is only for initializing the pure Altair testnets and tests, where we set `ALTAIR_FORK_SLOT = GENESIS_SLOT`. *Note*: The function `initialize_beacon_state_from_eth1` is modified with `ALTAIR_FORK_VERSION` fork version and initial sync committees. @@ -704,7 +704,7 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, # Set genesis validators root for domain separation and chain versioning state.genesis_validators_root = hash_tree_root(state.validators) - # Fill in sync committees + # [New in Altair] Fill in sync committees state.current_sync_committee = get_sync_committee(state, get_current_epoch(state)) state.next_sync_committee = get_sync_committee(state, get_current_epoch(state) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) From 7167c5a9d9b05ed1b57e356206930a15bce9c5e7 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 16 Apr 2021 11:32:27 -0500 Subject: [PATCH 053/227] generate sync committee update tests with always_bls --- .../test_process_sync_committee_updates.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c5bd9443e..e88d3ef1a 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -1,4 +1,5 @@ from eth2spec.test.context import ( + always_bls, spec_state_test, spec_test, with_all_phases_except, @@ -17,6 +18,11 @@ from eth2spec.test.helpers.epoch_processing import ( ) +# +# Note: +# Calculating sync committees requires pubkey aggregation, thus all tests are generated with `always_bls` +# + def run_sync_committees_progress_test(spec, state): first_sync_committee = state.current_sync_committee second_sync_committee = state.next_sync_committee @@ -41,6 +47,7 @@ def run_sync_committees_progress_test(spec, state): @with_all_phases_except([PHASE0]) @spec_state_test +@always_bls @with_configs([MINIMAL], reason="too slow") def test_sync_committees_progress_genesis(spec, state): # Genesis epoch period has an exceptional case @@ -52,6 +59,7 @@ def test_sync_committees_progress_genesis(spec, state): @with_all_phases_except([PHASE0]) @spec_state_test +@always_bls @with_configs([MINIMAL], reason="too slow") def test_sync_committees_progress_not_genesis(spec, state): # Transition out of the genesis epoch period to test non-exceptional case @@ -65,6 +73,7 @@ def test_sync_committees_progress_not_genesis(spec, state): @with_custom_state(balances_fn=misc_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) @spec_test @single_phase +@always_bls @with_configs([MINIMAL], reason="too slow") def test_sync_committees_progress_misc_balances(spec, state): yield from run_sync_committees_progress_test(spec, state) From 55f2cc6e414f9ea26bfe5856f329d84b522dbb39 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 16 Apr 2021 11:42:26 -0500 Subject: [PATCH 054/227] address @ralexstokes PR comments --- .../test_process_sync_committee_updates.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index e88d3ef1a..8ffc51507 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -27,7 +27,11 @@ def run_sync_committees_progress_test(spec, state): first_sync_committee = state.current_sync_committee second_sync_committee = state.next_sync_committee - end_slot_of_current_period = state.slot + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD * spec.SLOTS_PER_EPOCH - 1 + current_period = spec.get_current_epoch(state) // spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + next_period = current_period + 1 + next_period_start_epoch = next_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + next_period_start_slot = next_period_start_epoch * spec.SLOTS_PER_EPOCH + end_slot_of_current_period = next_period_start_slot - 1 transition_to(spec, state, end_slot_of_current_period) # Ensure assignments have not changed: @@ -51,7 +55,6 @@ def run_sync_committees_progress_test(spec, state): @with_configs([MINIMAL], reason="too slow") def test_sync_committees_progress_genesis(spec, state): # Genesis epoch period has an exceptional case - spec.get_current_epoch(state) assert spec.get_current_epoch(state) == spec.GENESIS_EPOCH yield from run_sync_committees_progress_test(spec, state) @@ -63,8 +66,9 @@ def test_sync_committees_progress_genesis(spec, state): @with_configs([MINIMAL], reason="too slow") def test_sync_committees_progress_not_genesis(spec, state): # Transition out of the genesis epoch period to test non-exceptional case - start_slot_of_next_period = state.slot + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD * spec.SLOTS_PER_EPOCH - transition_to(spec, state, start_slot_of_next_period) + assert spec.get_current_epoch(state) == spec.GENESIS_EPOCH + slot_in_next_period = state.slot + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD * spec.SLOTS_PER_EPOCH + transition_to(spec, state, slot_in_next_period) yield from run_sync_committees_progress_test(spec, state) From c0f12315220258006ef171addc78f7cafa1afff3 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Mon, 19 Apr 2021 15:24:06 +1000 Subject: [PATCH 055/227] Rename MAX_APPLICATION_TRANSACTIONS --- specs/merge/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index a508cb670..b9c443962 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -61,7 +61,7 @@ We define the following Python custom types for type hinting and readability: | Name | Value | | - | - | | `MAX_BYTES_PER_OPAQUE_TRANSACTION` | `uint64(2**20)` (= 1,048,576) | -| `MAX_APPLICATION_TRANSACTIONS` | `uint64(2**14)` (= 16,384) | +| `MAX_EXECUTION_TRANSACTIONS` | `uint64(2**14)` (= 16,384) | | `BYTES_PER_LOGS_BLOOM` | `uint64(2**8)` (= 256) | ## Containers @@ -108,7 +108,7 @@ class ExecutionPayload(Container): timestamp: uint64 receipt_root: Bytes32 logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] - transactions: List[OpaqueTransaction, MAX_APPLICATION_TRANSACTIONS] + transactions: List[OpaqueTransaction, MAX_EXECUTION_TRANSACTIONS] ``` #### `ExecutionPayloadHeader` From 9e4f5c18796a8097fd5f4995122df337a19f55be Mon Sep 17 00:00:00 2001 From: "Sam.An" <56215891+sammiee5311@users.noreply.github.com> Date: Mon, 19 Apr 2021 22:41:45 +0900 Subject: [PATCH 056/227] changed alphabet string to string module. changed alphabet string to string module. --- setup.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 70cf5176e..0fb56e708 100644 --- a/setup.py +++ b/setup.py @@ -5,6 +5,7 @@ from distutils import dir_util from distutils.util import convert_path import os import re +import string from typing import Dict, NamedTuple, List FUNCTION_REGEX = r'^def [\w_]*' @@ -89,10 +90,10 @@ def get_spec(file_name: str) -> SpecObject: if '`' in row[i]: row[i] = row[i][:row[i].find('`')] is_constant_def = True - if row[0][0] not in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ_': + if row[0][0] not in string.ascii_uppercase + '_': is_constant_def = False for c in row[0]: - if c not in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ_0123456789': + if c not in string.ascii_uppercase + '_' + string.digits: is_constant_def = False if is_constant_def: if row[1].startswith('get_generalized_index'): From b49869a78420610ba57daf5f96dc9a7ee31d08b8 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 19 Apr 2021 21:19:38 -0700 Subject: [PATCH 057/227] Use `process_execution_payload` in sharding doc --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index e02229838..bbcc57211 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -466,7 +466,7 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) # [Modified in Sharding] - process_application_payload(state, block.body) # [New in Merge] + process_execution_payload(state, block.body) # [New in Merge] ``` #### Operations From a83a2f1bc7456c1db0ed3bce9fa84745405918cd Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 20 Apr 2021 20:18:00 +0800 Subject: [PATCH 058/227] Add note for ByteList and ByteVector --- ssz/simple-serialize.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index d36e444ec..d97b8ea1c 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -75,7 +75,8 @@ For convenience we alias: * `bit` to `boolean` * `byte` to `uint8` (this is a basic type) -* `BytesN` to `Vector[byte, N]` (this is *not* a basic type) +* `BytesN` and `ByteVector[N]` to `Vector[byte, N]` (this is *not* a basic type) +* `ByteList[N]` to `List[byte, N]` * `null`: `{}` ### Default values From 5e8304b4c6eea304cbea856d06325821b6f1375d Mon Sep 17 00:00:00 2001 From: protolambda Date: Tue, 20 Apr 2021 17:26:11 +0200 Subject: [PATCH 059/227] Merge SSZ-static tests --- tests/generators/ssz_static/main.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tests/generators/ssz_static/main.py b/tests/generators/ssz_static/main.py index 617fb2f02..94ce02335 100644 --- a/tests/generators/ssz_static/main.py +++ b/tests/generators/ssz_static/main.py @@ -9,7 +9,8 @@ from eth2spec.debug import random_value, encode from eth2spec.config import config_util from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import ALTAIR, TESTGEN_FORKS, MINIMAL, MAINNET +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import ALTAIR, MERGE, TESTGEN_FORKS, MINIMAL, MAINNET from eth2spec.utils.ssz.ssz_typing import Container from eth2spec.utils.ssz.ssz_impl import ( hash_tree_root, @@ -64,6 +65,7 @@ def create_provider(fork_name, config_name: str, seed: int, mode: random_value.R config_util.prepare_config(configs_path, config_name) reload(spec_phase0) reload(spec_altair) + reload(spec_merge) return config_name def cases_fn() -> Iterable[gen_typing.TestCase]: @@ -71,6 +73,8 @@ def create_provider(fork_name, config_name: str, seed: int, mode: random_value.R spec = spec_phase0 if fork_name == ALTAIR: spec = spec_altair + elif fork_name == MERGE: + spec = spec_merge for (i, (name, ssz_type)) in enumerate(get_spec_ssz_types(spec)): yield from ssz_static_cases(fork_name, seed * 1000 + i, name, ssz_type, mode, chaos, count) @@ -89,8 +93,8 @@ if __name__ == "__main__": seed += 1 settings.append((seed, MAINNET, random_value.RandomizationMode.mode_random, False, 5)) seed += 1 - - for fork in TESTGEN_FORKS: + # TODO: enable testing for the whole merge spec. + for fork in TESTGEN_FORKS + (MERGE,): gen_runner.run_generator("ssz_static", [ create_provider(fork, config_name, seed, mode, chaos, cases_if_random) for (seed, config_name, mode, chaos, cases_if_random) in settings From 666f847354c4e513960290d8ad2ff4dcfdc43de8 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 20 Apr 2021 23:37:53 +0800 Subject: [PATCH 060/227] Update specs and test format note. --- specs/altair/beacon-chain.md | 6 +++--- specs/altair/fork.md | 2 ++ tests/formats/README.md | 2 ++ 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 978a7eadd..7f6556556 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -663,11 +663,11 @@ def process_sync_committee_updates(state: BeaconState) -> None: state.next_sync_committee = get_sync_committee(state, next_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) ``` -## Initialize state for Altair testnets +## Initialize state for pure Altair testnets and test vectors -This helper function is only for initializing the pure Altair testnets and tests, where we set `ALTAIR_FORK_SLOT = GENESIS_SLOT`. +This helper function is only for initializing the state for pure Altair testnets and tests. -*Note*: The function `initialize_beacon_state_from_eth1` is modified with `ALTAIR_FORK_VERSION` fork version and initial sync committees. +*Note*: The function `initialize_beacon_state_from_eth1` is modified with `ALTAIR_FORK_VERSION` fork version and initial sync committees. ```python def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, diff --git a/specs/altair/fork.md b/specs/altair/fork.md index b51466c1e..6312eb09f 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -34,6 +34,8 @@ Warning: this configuration is not definitive. TBD. Social consensus, along with state conditions such as epoch boundary, finality, deposits, active validator count, etc. may be part of the decision process to trigger the fork. For now we assume the condition will be triggered at slot `ALTAIR_FORK_SLOT`, where `ALTAIR_FORK_SLOT % SLOTS_PER_EPOCH == 0`. +Note that for the pure Altair testnets, we don't apply `upgrade_to_altair` since it starts with Altair version logic. + ### Upgrading the state After `process_slots` of Phase 0 finishes, if `state.slot == ALTAIR_FORK_SLOT`, an irregular state change is made to upgrade to Altair. diff --git a/tests/formats/README.md b/tests/formats/README.md index 8f817220e..16f756fc1 100644 --- a/tests/formats/README.md +++ b/tests/formats/README.md @@ -139,6 +139,8 @@ E.g. `pre.ssz_snappy`, `deposit.ssz_snappy`, `post.ssz_snappy`. Diffing a `pre.ssz_snappy` and `post.ssz_snappy` provides all the information for testing, when decompressed and decoded. Then the difference between pre and post can be compared to anything that changes the pre state, e.g. `deposit.ssz_snappy` +Note that by default, the SSZ data is in the given test case's version, e.g., if it's `altair` test case, use `altair.BeaconState` container to deserialize the given state. + YAML is generally used for test metadata, and for tests that do not use SSZ: e.g. shuffling and BLS tests. In this case, there is no point in adding special SSZ types. And the size and efficiency of YAML is acceptable. From baf4b73c1804763c926e18489027c0726431d5a5 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 20 Apr 2021 23:38:48 +0800 Subject: [PATCH 061/227] Fix ToC --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 7f6556556..a2138103d 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -52,7 +52,7 @@ - [Slashings](#slashings) - [Participation flags updates](#participation-flags-updates) - [Sync committee updates](#sync-committee-updates) -- [Initialize state for Altair testnets](#initialize-state-for-altair-testnets) +- [Initialize state for pure Altair testnets and test vectors](#initialize-state-for-pure-altair-testnets-and-test-vectors) From 66e1a2858f9fbebf5e00539d1a34b78025673d37 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 21 Apr 2021 00:24:44 +0800 Subject: [PATCH 062/227] Use `ALTAIR_FORK_EPOCH` instead of `ALTAIR_FORK_SLOT` --- configs/README.md | 2 +- configs/mainnet/altair.yaml | 2 +- configs/mainnet/merge.yaml | 2 +- configs/mainnet/sharding.yaml | 2 +- configs/minimal/altair.yaml | 2 +- configs/minimal/merge.yaml | 2 +- configs/minimal/sharding.yaml | 2 +- specs/altair/fork.md | 6 +++--- specs/das/fork-choice.md | 2 +- tests/core/pyspec/eth2spec/test/helpers/state.py | 5 +++-- 10 files changed, 14 insertions(+), 13 deletions(-) diff --git a/configs/README.md b/configs/README.md index 15529e590..2cf4e3f60 100644 --- a/configs/README.md +++ b/configs/README.md @@ -15,7 +15,7 @@ Over time, the need to sync an older state may be deprecated. In this case, the prefix on the new constant may be removed, and the old constant will keep a special name before completely being removed. A previous iteration of forking made use of "timelines", but this collides with the definitions used in the spec (constants for special forking slots, etc.), and was not integrated sufficiently in any of the spec tools or implementations. -Instead, the config essentially doubles as fork definition now, e.g. changing the value for `ALTAIR_FORK_SLOT` changes the fork. +Instead, the config essentially doubles as fork definition now, e.g. changing the value for `ALTAIR_FORK_EPOCH` changes the fork. Another reason to prefer forking through constants is the ability to program a forking moment based on context, instead of being limited to a static slot number. diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 44490f982..3cd4b8419 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -38,7 +38,7 @@ DOMAIN_CONTRIBUTION_AND_PROOF: 0x09000000 # 0x01000000 ALTAIR_FORK_VERSION: 0x01000000 # TBD -ALTAIR_FORK_SLOT: 18446744073709551615 +ALTAIR_FORK_EPOCH: 18446744073709551615 # Sync protocol diff --git a/configs/mainnet/merge.yaml b/configs/mainnet/merge.yaml index b4667f5b5..4e012ac05 100644 --- a/configs/mainnet/merge.yaml +++ b/configs/mainnet/merge.yaml @@ -4,4 +4,4 @@ # --------------------------------------------------------------- MERGE_FORK_VERSION: 0x02000000 # TBD, temporarily max uint64 value: 2**64 - 1 -MERGE_FORK_SLOT: 18446744073709551615 +MERGE_FORK_EPOCH: 18446744073709551615 diff --git a/configs/mainnet/sharding.yaml b/configs/mainnet/sharding.yaml index 4773aa5f5..b3c22c354 100644 --- a/configs/mainnet/sharding.yaml +++ b/configs/mainnet/sharding.yaml @@ -4,7 +4,7 @@ # --------------------------------------------------------------- SHARDING_FORK_VERSION: 0x03000000 # TBD, temporarily max uint64 value: 2**64 - 1 -SHARDING_FORK_SLOT: 18446744073709551615 +SHARDING_FORK_EPOCH: 18446744073709551615 # Beacon-chain diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index 10bdf318b..f9b30eea2 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -38,7 +38,7 @@ DOMAIN_CONTRIBUTION_AND_PROOF: 0x09000000 # [customized] Highest byte set to 0x01 to avoid collisions with mainnet versioning ALTAIR_FORK_VERSION: 0x01000001 # [customized] -ALTAIR_FORK_SLOT: 18446744073709551615 +ALTAIR_FORK_EPOCH: 18446744073709551615 # Sync protocol diff --git a/configs/minimal/merge.yaml b/configs/minimal/merge.yaml index 394595d02..3b50cd5ca 100644 --- a/configs/minimal/merge.yaml +++ b/configs/minimal/merge.yaml @@ -4,4 +4,4 @@ # --------------------------------------------------------------- MERGE_FORK_VERSION: 0x02000001 # TBD, temporarily max uint64 value: 2**64 - 1 -MERGE_FORK_SLOT: 18446744073709551615 +MERGE_FORK_EPOCH: 18446744073709551615 diff --git a/configs/minimal/sharding.yaml b/configs/minimal/sharding.yaml index f32e2827d..c6ca8b560 100644 --- a/configs/minimal/sharding.yaml +++ b/configs/minimal/sharding.yaml @@ -4,7 +4,7 @@ # --------------------------------------------------------------- SHARDING_FORK_VERSION: 0x03000001 # TBD, temporarily max uint64 value: 2**64 - 1 -MERGE_FORK_SLOT: 18446744073709551615 +MERGE_FORK_EPOCH: 18446744073709551615 # Beacon-chain diff --git a/specs/altair/fork.md b/specs/altair/fork.md index b51466c1e..739584309 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -26,17 +26,17 @@ Warning: this configuration is not definitive. | Name | Value | | - | - | | `ALTAIR_FORK_VERSION` | `Version('0x01000000')` | -| `ALTAIR_FORK_SLOT` | `Slot(18446744073709551615)` **TBD** | +| `ALTAIR_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | ## Fork to Altair ### Fork trigger -TBD. Social consensus, along with state conditions such as epoch boundary, finality, deposits, active validator count, etc. may be part of the decision process to trigger the fork. For now we assume the condition will be triggered at slot `ALTAIR_FORK_SLOT`, where `ALTAIR_FORK_SLOT % SLOTS_PER_EPOCH == 0`. +TBD. Social consensus, along with state conditions such as epoch boundary, finality, deposits, active validator count, etc. may be part of the decision process to trigger the fork. For now we assume the condition will be triggered at epoch `ALTAIR_FORK_EPOCH`. ### Upgrading the state -After `process_slots` of Phase 0 finishes, if `state.slot == ALTAIR_FORK_SLOT`, an irregular state change is made to upgrade to Altair. +After `process_slots` of Phase 0 finishes, if `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. ```python def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: diff --git a/specs/das/fork-choice.md b/specs/das/fork-choice.md index 329c033a7..ec9f3ab59 100644 --- a/specs/das/fork-choice.md +++ b/specs/das/fork-choice.md @@ -37,7 +37,7 @@ def get_new_dependencies(state: BeaconState) -> Set[DataCommitment]: ```python def get_all_dependencies(store: Store, block: BeaconBlock) -> Set[DataCommitment]: - if block.slot < SHARDING_FORK_SLOT: + if compute_epoch_at_slot(block.slot) < SHARDING_FORK_EPOCH: return set() else: latest = get_new_dependencies(store.block_states[hash_tree_root(block)]) diff --git a/tests/core/pyspec/eth2spec/test/helpers/state.py b/tests/core/pyspec/eth2spec/test/helpers/state.py index 8980053f5..d61df7610 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/state.py +++ b/tests/core/pyspec/eth2spec/test/helpers/state.py @@ -42,9 +42,10 @@ def transition_to_slot_via_block(spec, state, slot): def transition_to_valid_shard_slot(spec, state): """ - Transition to slot `spec.SHARDING_FORK_SLOT + 1` and fork at `spec.SHARDING_FORK_SLOT`. + Transition to slot `compute_epoch_at_slot(spec.SHARDING_FORK_EPOCH) + 1` + and fork at `compute_epoch_at_slot(spec.SHARDING_FORK_EPOCH)`. """ - transition_to(spec, state, spec.SHARDING_FORK_SLOT) + transition_to(spec, state, spec.compute_epoch_at_slot(spec.SHARDING_FORK_EPOCH)) next_slot(spec, state) From 1694cdbd0293b3187bc8c978830539527c642a4c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 20 Apr 2021 13:40:25 -0600 Subject: [PATCH 063/227] add always_bls for historical batch sanity test --- tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py index 9b184b018..15f64ac79 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py @@ -930,8 +930,11 @@ def test_balance_driven_status_transitions(spec, state): assert state.validators[validator_index].exit_epoch < spec.FAR_FUTURE_EPOCH +# Requires always_bls because historical root period and sync committee period is same length +# so this epoch transition also computes new sync committees which requires aggregation @with_all_phases @spec_state_test +@always_bls def test_historical_batch(spec, state): state.slot += spec.SLOTS_PER_HISTORICAL_ROOT - (state.slot % spec.SLOTS_PER_HISTORICAL_ROOT) - 1 pre_historical_roots_len = len(state.historical_roots) From c99d72d54197a93c164a6b1df355e42aced072f7 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 21 Apr 2021 03:17:33 +0200 Subject: [PATCH 064/227] update pyspec dev usage docs, improve makefile --- Makefile | 20 ++++++----- tests/core/pyspec/README.md | 67 ++++++++++++++++++++++++------------- 2 files changed, 56 insertions(+), 31 deletions(-) diff --git a/Makefile b/Makefile index 8f053e7a7..0522c8842 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,10 @@ SPEC_DIR = ./specs SSZ_DIR = ./ssz TEST_LIBS_DIR = ./tests/core TEST_GENERATORS_DIR = ./tests/generators +# The working dir during testing PY_SPEC_DIR = $(TEST_LIBS_DIR)/pyspec +ETH2SPEC_MODULE_DIR = $(PY_SPEC_DIR)/eth2spec +TEST_REPORT_DIR = $(PY_SPEC_DIR)/test-reports TEST_VECTOR_DIR = ../eth2.0-spec-tests/tests GENERATOR_DIR = ./tests/generators SOLIDITY_DEPOSIT_CONTRACT_DIR = ./solidity_deposit_contract @@ -27,7 +30,8 @@ MARKDOWN_FILES = $(wildcard $(SPEC_DIR)/phase0/*.md) $(wildcard $(SPEC_DIR)/alta $(wildcard $(SPEC_DIR)/sharding/*.md) COV_HTML_OUT=.htmlcov -COV_INDEX_FILE=$(PY_SPEC_DIR)/$(COV_HTML_OUT)/index.html +COV_HTML_OUT_DIR=$(PY_SPEC_DIR)/$(COV_HTML_OUT) +COV_INDEX_FILE=$(COV_HTML_OUT_DIR)/index.html CURRENT_DIR = ${CURDIR} LINTER_CONFIG_FILE = $(CURRENT_DIR)/linter.ini @@ -53,17 +57,17 @@ partial_clean: rm -f .coverage rm -rf $(PY_SPEC_DIR)/.pytest_cache rm -rf $(DEPOSIT_CONTRACT_TESTER_DIR)/.pytest_cache - rm -rf $(PY_SPEC_DIR)/phase0 - rm -rf $(PY_SPEC_DIR)/altair - rm -rf $(PY_SPEC_DIR)/merge - rm -rf $(PY_SPEC_DIR)/$(COV_HTML_OUT) - rm -rf $(PY_SPEC_DIR)/.coverage - rm -rf $(PY_SPEC_DIR)/test-reports + rm -rf $(ETH2SPEC_MODULE_DIR)/phase0 + rm -rf $(ETH2SPEC_MODULE_DIR)/altair + rm -rf $(ETH2SPEC_MODULE_DIR)/merge + rm -rf $(COV_HTML_OUT_DIR) + rm -rf $(TEST_REPORT_DIR) rm -rf eth2spec.egg-info dist build rm -rf build clean: partial_clean rm -rf venv + # legacy cleanup. The pyspec venv should be located at the repository root rm -rf $(PY_SPEC_DIR)/venv rm -rf $(DEPOSIT_CONTRACT_COMPILER_DIR)/venv rm -rf $(DEPOSIT_CONTRACT_TESTER_DIR)/venv @@ -88,7 +92,7 @@ pyspec: # installs the packages to run pyspec tests install_test: - python3 -m venv venv; . venv/bin/activate; python3 -m pip install .[lint]; python3 -m pip install -e .[test] + python3 -m venv venv; . venv/bin/activate; python3 -m pip install -e .[lint]; python3 -m pip install -e .[test] test: pyspec . venv/bin/activate; cd $(PY_SPEC_DIR); \ diff --git a/tests/core/pyspec/README.md b/tests/core/pyspec/README.md index d84054f11..33695a223 100644 --- a/tests/core/pyspec/README.md +++ b/tests/core/pyspec/README.md @@ -7,27 +7,30 @@ With this executable spec, test-generators can easily create test-vectors for client implementations, and the spec itself can be verified to be consistent and coherent through sanity tests implemented with pytest. -## Building - -To build the pyspec: `python setup.py build` - (or `pip install .`, but beware that ignored files will still be copied over to a temporary dir, due to pip issue 2195). -This outputs the build files to the `./build/lib/eth2spec/...` dir, and can't be used for local test running. Instead, use the dev-install as described below. - ## Dev Install -All the dynamic parts of the spec are automatically built with `python setup.py pyspecdev`. -Unlike the regular install, this outputs spec files to their original source location, instead of build output only. +First, create a `venv` and install the developer dependencies (`test` and `lint` extras): -Alternatively, you can build a sub-set of the pyspec with the distutil command: -```bash -python setup.py pyspec --spec-fork=phase0 --md-doc-paths="specs/phase0/beacon-chain.md specs/phase0/fork-choice.md" --out-dir=my_spec_dir +```shell +make install_test ``` -## Py-tests +All the dynamic parts of the spec are built with: -After installing, you can install the optional dependencies for testing and linting. -With makefile: `make install_test`. -Or manually: run `pip install .[test]` and `pip install .[lint]`. +```shell +(venv) python setup.py pyspecdev +``` + +Unlike the regular install, this outputs spec files to their intended source location, +to enable debuggers to navigate between packages and generated code, without fragile directory linking. + +By default, when installing the `eth2spec` as package in non-develop mode, +the distutils implementation of the `setup` runs `build`, which is extended to run the same `pyspec` work, +but outputs into the standard `./build/lib` output. +This enables the `eth2.0-specs` repository to be installed like any other python package. + + +## Py-tests These tests are not intended for client-consumption. These tests are testing the spec itself, to verify consistency and provide feedback on modifications of the spec. @@ -39,20 +42,32 @@ However, most of the tests can be run in generator-mode, to output test vectors Run `make test` from the root of the specs repository (after running `make install_test` if have not before). +Note that the `make` commands run through the build steps: it runs the `build` output, not the local package source files. + #### Manual -From the repository root: +See `Dev install` for test pre-requisites. -Install venv and install: -```bash -python3 -m venv venv -. venv/bin/activate -python setup.py pyspecdev +Tests are built for `pytest`. + +Caveats: +- Working directory must be `./tests/core/pyspec`. The work-directory is important to locate eth2 configuration files. +- Run `pytest` as module. It avoids environment differences, and the behavior is different too: + `pytest` as module adds the current directory to the `sys.path` + +Full test usage, with explicit configuration for illustration of options usage: +```shell +(venv) python -m pytest --config=minimal eth2spec ``` -Run the test command from the `tests/core/pyspec` directory: +Or, to run a specific test file, specify the full path: +```shell +(venv) python -m pytest --config=minimal ./eth2spec/test/phase0/block_processing/test_process_attestation.py ``` -pytest --config=minimal eth2spec + +Or, to run a specific test function (specify the `eth2spec` module, or the script path if the keyword is ambiguous): +```shell +(venv) python -m pytest --config=minimal -k test_success_multi_proposer_index_iterations eth2spec ``` Options: @@ -64,6 +79,12 @@ Options: Run `make open_cov` from the root of the specs repository after running `make test` to open the html code coverage report. +### Advanced + +Building spec files from any markdown sources, to a custom location: +```bash +(venv) python setup.py pyspec --spec-fork=phase0 --md-doc-paths="specs/phase0/beacon-chain.md specs/phase0/fork-choice.md" --out-dir=my_spec_dir +``` ## Contributing From 93f6a541adf8116cb86f072fcc7ab5620e7ced52 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 21 Apr 2021 16:42:54 +0800 Subject: [PATCH 065/227] PR feedback from @djrtwo --- specs/altair/beacon-chain.md | 2 +- specs/altair/fork.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index a2138103d..438c892c6 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -667,7 +667,7 @@ def process_sync_committee_updates(state: BeaconState) -> None: This helper function is only for initializing the state for pure Altair testnets and tests. -*Note*: The function `initialize_beacon_state_from_eth1` is modified with `ALTAIR_FORK_VERSION` fork version and initial sync committees. +*Note*: The function `initialize_beacon_state_from_eth1` is modified: (1) using `ALTAIR_FORK_VERSION` as the current fork version, (2) utilizing the Altair `BeaconBlockBody` when constructing the initial `latest_block_header`, and (3) adding initial sync committees. ```python def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, diff --git a/specs/altair/fork.md b/specs/altair/fork.md index b5dac5b22..be06e183e 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -34,7 +34,7 @@ Warning: this configuration is not definitive. TBD. Social consensus, along with state conditions such as epoch boundary, finality, deposits, active validator count, etc. may be part of the decision process to trigger the fork. For now we assume the condition will be triggered at epoch `ALTAIR_FORK_EPOCH`. -Note that for the pure Altair testnets, we don't apply `upgrade_to_altair` since it starts with Altair version logic. +Note that for the pure Altair networks, we don't apply `upgrade_to_altair` since it starts with Altair version logic. ### Upgrading the state From d09a0c2bce8c663571d855e743b6a24c98604707 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 21 Apr 2021 18:55:05 +0300 Subject: [PATCH 066/227] get_shard_proposer_index: Use slot instead of beacon_state.slot --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index bbcc57211..250f8b7ed 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -402,7 +402,7 @@ def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard """ epoch = compute_epoch_at_slot(slot) committee = get_shard_committee(beacon_state, epoch, shard) - seed = hash(get_seed(beacon_state, epoch, DOMAIN_BEACON_PROPOSER) + uint_to_bytes(beacon_state.slot)) + seed = hash(get_seed(beacon_state, epoch, DOMAIN_BEACON_PROPOSER) + uint_to_bytes(slot)) # Proposer must have sufficient balance to pay for worst case fee burn EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION = ( From b870f928013e6ca8fbc560493641068d06db38e5 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 21 Apr 2021 17:08:38 -0700 Subject: [PATCH 067/227] Update validator.md --- specs/altair/validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 1649eabc8..69ae34ec8 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -413,7 +413,7 @@ ENR advertisement is indicated by setting the appropriate bit(s) of the bitfield Any bits modified for the sync committee responsibilities are unset in the ENR after any validators have left the sync committee. *Note*: The first sync committee from phase 0 to the Altair fork will not be known until the fork happens which implies subnet assignments are not known until then. -Early sync committee members should listen for topic subscriptions from peers and employ discovery via the ENR advertisements near the fork boundary to form initial subnets +Early sync committee members should listen for topic subscriptions from peers and employ discovery via the ENR advertisements near the fork boundary to form initial subnets. Some early sync committee rewards may be missed while the initial subnets form. * To join a sync committee subnet, select a random number of epochs before the end of the current sync committee period between 1 and `SYNC_COMMITTEE_SUBNET_COUNT`, inclusive. From 1c8f56c84b27820fca67051041518c57a9fa2c37 Mon Sep 17 00:00:00 2001 From: Diederik Loerakker Date: Thu, 22 Apr 2021 02:35:54 +0200 Subject: [PATCH 068/227] Fix indentation of comment Co-authored-by: Alex Stokes --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 0522c8842..e4969ff2f 100644 --- a/Makefile +++ b/Makefile @@ -67,7 +67,7 @@ partial_clean: clean: partial_clean rm -rf venv - # legacy cleanup. The pyspec venv should be located at the repository root + # legacy cleanup. The pyspec venv should be located at the repository root rm -rf $(PY_SPEC_DIR)/venv rm -rf $(DEPOSIT_CONTRACT_COMPILER_DIR)/venv rm -rf $(DEPOSIT_CONTRACT_TESTER_DIR)/venv From de3ac15d9c08d3a2f300d53799db5d5013ea1955 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 20 Apr 2021 14:55:46 -0700 Subject: [PATCH 069/227] introduce alternate spec parser with spec comments --- setup.py | 195 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 122 insertions(+), 73 deletions(-) diff --git a/setup.py b/setup.py index 2b6d0cd83..d9dd5f8e3 100644 --- a/setup.py +++ b/setup.py @@ -1,4 +1,3 @@ -from enum import Enum, auto from setuptools import setup, find_packages, Command from setuptools.command.build_py import build_py from distutils import dir_util @@ -6,11 +5,24 @@ from distutils.util import convert_path import os import re import string -from typing import Dict, NamedTuple, List, Sequence +from typing import Dict, NamedTuple, List, Sequence, Optional from abc import ABC, abstractmethod +import ast -FUNCTION_REGEX = r'^def [\w_]*' +# NOTE: have to programmatically include third-party dependencies in `setup.py`. +MARKO_VERSION = "marko==1.0.2" +try: + import marko +except ImportError: + import pip + pip.main(["install", MARKO_VERSION]) + +from marko.block import Heading, FencedCode, LinkRefDef, BlankLine +from marko.inline import CodeSpan +from marko.ext.gfm import gfm +from marko.ext.gfm.elements import Table, Paragraph + # Definitions in context.py PHASE0 = 'phase0' @@ -45,83 +57,118 @@ class SpecObject(NamedTuple): dataclasses: Dict[str, str] -class CodeBlockType(Enum): - SSZ = auto() - DATACLASS = auto() - FUNCTION = auto() +def _get_name_from_heading(heading: Heading) -> Optional[str]: + last_child = heading.children[-1] + if isinstance(last_child, CodeSpan): + return last_child.children + return None + + +def _get_source_from_code_block(block: FencedCode) -> str: + return block.children[0].children.strip() + + +def _get_function_name_from_source(source: str) -> str: + fn = ast.parse(source).body[0] + return fn.name + + +def _get_class_info_from_source(source: str) -> (str, Optional[str]): + class_def = ast.parse(source).body[0] + base = class_def.bases[0] + if isinstance(base, ast.Name): + parent_class = base.id + else: + # NOTE: SSZ definition derives from earlier phase... + # e.g. `phase0.SignedBeaconBlock` + # TODO: check for consistency with other phases + parent_class = None + return class_def.name, parent_class + + +def _is_constant_id(name: str) -> bool: + if name[0] not in string.ascii_uppercase + '_': + return False + return all(map(lambda c: c in string.ascii_uppercase + '_' + string.digits, name[1:])) + + +ETH2_SPEC_COMMENT_PREFIX = "eth2spec:" + + +def _get_eth2_spec_comment(child: LinkRefDef) -> Optional[str]: + _, _, title = child._parse_info + if not (title[0] == "(" and title[len(title)-1] == ")"): + return None + title = title[1:len(title)-1] + if not title.startswith(ETH2_SPEC_COMMENT_PREFIX): + return None + return title[len(ETH2_SPEC_COMMENT_PREFIX):].strip() def get_spec(file_name: str) -> SpecObject: - """ - Takes in the file name of a spec.md file, opens it and returns a parsed spec object. - - Note: This function makes heavy use of the inherent ordering of dicts, - if this is not supported by your python version, it will not work. - """ - pulling_from = None # line number of start of latest object - current_name = None # most recent section title functions: Dict[str, str] = {} constants: Dict[str, str] = {} ssz_dep_constants: Dict[str, str] = {} ssz_objects: Dict[str, str] = {} dataclasses: Dict[str, str] = {} - function_matcher = re.compile(FUNCTION_REGEX) - block_type = CodeBlockType.FUNCTION custom_types: Dict[str, str] = {} - for linenum, line in enumerate(open(file_name).readlines()): - line = line.rstrip() - if pulling_from is None and len(line) > 0 and line[0] == '#' and line[-1] == '`': - current_name = line[line[:-1].rfind('`') + 1: -1] - if line[:9] == '```python': - assert pulling_from is None - pulling_from = linenum + 1 - elif line[:3] == '```': - pulling_from = None - else: - # Handle function definitions & ssz_objects - if pulling_from is not None: - if len(line) > 18 and line[:6] == 'class ' and (line[-12:] == '(Container):' or '(phase' in line): - end = -12 if line[-12:] == '(Container):' else line.find('(') - name = line[6:end] - # Check consistency with markdown header - assert name == current_name - block_type = CodeBlockType.SSZ - elif line[:10] == '@dataclass': - block_type = CodeBlockType.DATACLASS - elif function_matcher.match(line) is not None: - current_name = function_matcher.match(line).group(0) - block_type = CodeBlockType.FUNCTION - if block_type == CodeBlockType.SSZ: - ssz_objects[current_name] = ssz_objects.get(current_name, '') + line + '\n' - elif block_type == CodeBlockType.DATACLASS: - dataclasses[current_name] = dataclasses.get(current_name, '') + line + '\n' - elif block_type == CodeBlockType.FUNCTION: - functions[current_name] = functions.get(current_name, '') + line + '\n' - else: - pass + with open(file_name) as source_file: + document = gfm.parse(source_file.read()) - # Handle constant and custom types table entries - elif pulling_from is None and len(line) > 0 and line[0] == '|': - row = line[1:].split('|') - if len(row) >= 2: - for i in range(2): - row[i] = row[i].strip().strip('`') - if '`' in row[i]: - row[i] = row[i][:row[i].find('`')] - is_constant_def = True - if row[0][0] not in string.ascii_uppercase + '_': - is_constant_def = False - for c in row[0]: - if c not in string.ascii_uppercase + '_' + string.digits: - is_constant_def = False - if is_constant_def: - if row[1].startswith('get_generalized_index'): - ssz_dep_constants[row[0]] = row[1] + current_name = None + should_skip = False + for child in document.children: + if isinstance(child, BlankLine): + continue + if should_skip: + should_skip = False + continue + if isinstance(child, Heading): + current_name = _get_name_from_heading(child) + elif isinstance(child, FencedCode): + if child.lang != "python": + continue + source = _get_source_from_code_block(child) + if source.startswith("def"): + current_name = _get_function_name_from_source(source) + functions[current_name] = "\n".join(line.rstrip() for line in source.splitlines()) + elif source.startswith("@dataclass"): + dataclasses[current_name] = "\n".join(line.rstrip() for line in source.splitlines()) + elif source.startswith("class"): + class_name, parent_class = _get_class_info_from_source(source) + # check consistency with spec + assert class_name == current_name + if parent_class: + assert parent_class == "Container" + # NOTE: trim whitespace from spec + ssz_objects[current_name] = "\n".join(line.rstrip() for line in source.splitlines()) + else: + raise Exception("unrecognized python code element") + elif isinstance(child, Table): + for row in child.children: + cells = row.children + if len(cells) >= 2: + name_cell = cells[0] + name = name_cell.children[0].children + value_cell = cells[1] + value = value_cell.children[0].children + if isinstance(value, list): + # marko parses `**X**` as a list containing a X + value = value[0].children + if _is_constant_id(name): + if value.startswith("get_generalized_index"): + ssz_dep_constants[name] = value else: - constants[row[0]] = row[1].replace('**TBD**', '2**32') - elif row[1].startswith('uint') or row[1].startswith('Bytes') or row[1].startswith('ByteList'): - custom_types[row[0]] = row[1] + constants[name] = value.replace("TBD", "2**32") + elif value.startswith("uint") or value.startswith("Bytes") or value.startswith("ByteList"): + custom_types[name] = value + elif isinstance(child, LinkRefDef): + comment = _get_eth2_spec_comment(child) + if comment: + if comment == "skip": + should_skip = True + return SpecObject( functions=functions, custom_types=custom_types, @@ -424,7 +471,7 @@ def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> Executi spec_builders = { builder.fork: builder - for builder in (Phase0SpecBuilder, AltairSpecBuilder, MergeSpecBuilder) + for builder in (AltairSpecBuilder, ) } @@ -452,12 +499,12 @@ def objects_to_spec(spec_object: SpecObject, builder: SpecBuilder, ordered_class for k in list(spec_object.functions): if "ceillog2" in k or "floorlog2" in k: del spec_object.functions[k] - functions_spec = '\n\n'.join(spec_object.functions.values()) + functions_spec = '\n\n\n'.join(spec_object.functions.values()) for k in list(spec_object.constants.keys()): if k == "BLS12_381_Q": spec_object.constants[k] += " # noqa: E501" constants_spec = '\n'.join(map(lambda x: '%s = %s' % (x, spec_object.constants[x]), spec_object.constants)) - ordered_class_objects_spec = '\n\n'.join(ordered_class_objects.values()) + ordered_class_objects_spec = '\n\n\n'.join(ordered_class_objects.values()) ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, builder.hardcoded_ssz_dep_constants()[x]), builder.hardcoded_ssz_dep_constants())) ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), builder.hardcoded_ssz_dep_constants())) custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, builder.hardcoded_custom_type_dep_constants()[x]), builder.hardcoded_custom_type_dep_constants())) @@ -474,8 +521,8 @@ def objects_to_spec(spec_object: SpecObject, builder: SpecBuilder, ordered_class + '\n\n' + constants_spec + '\n\n' + CONFIG_LOADER + '\n\n' + ordered_class_objects_spec - + '\n\n' + functions_spec - + '\n' + builder.sundry_functions() + + '\n\n\n' + functions_spec + + '\n\n' + builder.sundry_functions() # Since some constants are hardcoded in setup.py, the following assertions verify that the hardcoded constants are # as same as the spec definition. + ('\n\n\n' + ssz_dep_constants_verification if ssz_dep_constants_verification != '' else '') @@ -619,6 +666,7 @@ class PySpecCommand(Command): specs/altair/beacon-chain.md specs/altair/fork.md specs/altair/validator.md + specs/altair/p2p-interface.md specs/altair/sync-protocol.md """ elif self.spec_fork == MERGE: @@ -756,5 +804,6 @@ setup( "remerkleable==0.1.19", "ruamel.yaml==0.16.5", "lru-dict==1.1.6", + "marko==1.0.2", ] ) From a9e3ecabbd0705d648f496c9bf441423aa1ab402 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 20 Apr 2021 14:56:07 -0700 Subject: [PATCH 070/227] demo spec comment feature --- setup.py | 2 +- specs/altair/p2p-interface.md | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index d9dd5f8e3..2b1b23cd7 100644 --- a/setup.py +++ b/setup.py @@ -471,7 +471,7 @@ def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> Executi spec_builders = { builder.fork: builder - for builder in (AltairSpecBuilder, ) + for builder in (Phase0SpecBuilder, AltairSpecBuilder, MergeSpecBuilder) } diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index f76c3bce9..a32137d1f 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -182,6 +182,8 @@ Request and Response remain unchanged. A `ForkDigest`-context is used to select Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: +[0]: # (eth2spec: skip) + | `fork_version` | Chunk SSZ type | | ------------------------ | -------------------------- | | `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` | @@ -195,6 +197,8 @@ Request and Response remain unchanged. A `ForkDigest`-context is used to select Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: +[1]: # (eth2spec: skip) + | `fork_version` | Chunk SSZ type | | ------------------------ | -------------------------- | | `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` | From 360a1dd598c046aa400a87e69ec81919525fcb51 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 13 Apr 2021 14:33:07 -0700 Subject: [PATCH 071/227] Respect subcommittees in gossip validations for Altair --- specs/altair/p2p-interface.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index a32137d1f..e12b8e01d 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -83,15 +83,23 @@ See the [state transition document](./beacon-chain.md#beaconblockbody) for Altai This topic is used to propagate partially aggregated sync committee signatures to be included in future blocks. -The following validations MUST pass before forwarding the `signed_contribution_and_proof` on the network; define `contribution_and_proof = signed_contribution_and_proof.message` and `contribution = contribution_and_proof.contribution` for convenience: +The following validations MUST pass before forwarding the `signed_contribution_and_proof` on the network; define `contribution_and_proof = signed_contribution_and_proof.message`, `contribution = contribution_and_proof.contribution`, and the following function `get_sync_subcommittee_pubkeys` for convenience: + +```python +SYNC_SUBCOMMITTEE_SIZE = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT + +def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: + i = subcommittee_index * SYNC_SUBCOMMITTEE_SIZE + return state.current_sync_committee.pubkeys[i:i+SYNC_SUBCOMMITTEE_SIZE] +``` - _[IGNORE]_ The contribution's slot is for the current slot, i.e. `contribution.slot == current_slot`. - _[IGNORE]_ The block being signed over (`contribution.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[REJECT]_ The subcommittee index is in the allowed range, i.e. `contribution.subcommittee_index < SYNC_COMMITTEE_SUBNET_COUNT`. -- _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot`. -- _[REJECT]_ `contribution_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_sync_committee_aggregator(state, contribution.slot, contribution_and_proof.selection_proof)` returns `True`. -- _[REJECT]_ The aggregator's validator index is within the current sync committee -- - i.e. `state.validators[contribution_and_proof.aggregator_index].pubkey in state.current_sync_committee.pubkeys`. +- _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`. +- _[REJECT]_ `contribution_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_sync_committee_aggregator(contribution_and_proof.selection_proof)` returns `True`. +- _[REJECT]_ The aggregator's validator index is in the declared subcommittee of the current sync committee -- + i.e. `state.validators[contribution_and_proof.aggregator_index].pubkey in get_sync_subcommittee_pubkeys(state, contribution.subcommittee_index)`. - _[REJECT]_ The `contribution_and_proof.selection_proof` is a valid signature of the `SyncCommitteeSigningData` derived from the `contribution` by the validator with index `contribution_and_proof.aggregator_index`. - _[REJECT]_ The aggregator signature, `signed_contribution_and_proof.signature`, is valid. - _[REJECT]_ The aggregate signature is valid for the message `beacon_block_root` and aggregate pubkey derived from the participation info in `aggregation_bits` for the subcommittee specified by the `subcommittee_index`. From f992a9ae651cdb03722f7368ee62d2c8dc74f058 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 13 Apr 2021 14:45:04 -0700 Subject: [PATCH 072/227] Remove duplicate validation --- specs/altair/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index e12b8e01d..08c1682f2 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -117,8 +117,8 @@ The following validations MUST pass before forwarding the `sync_committee_signat - _[IGNORE]_ The signature's slot is for the current slot, i.e. `sync_committee_signature.slot == current_slot`. - _[IGNORE]_ The block being signed over (`sync_committee_signature.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_signature.validator_index`. -- _[REJECT]_ The validator producing this `sync_committee_signature` is in the current sync committee, i.e. `state.validators[sync_committee_signature.validator_index].pubkey in state.current_sync_committee.pubkeys`. -- _[REJECT]_ The `subnet_id` is correct, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. +- _[REJECT]_ The `subnet_id` is correct, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. + Note this validation implies the validator is part of the broader current sync committee along with the correct subcommittee. - _[REJECT]_ The `signature` is valid for the message `beacon_block_root` for the validator referenced by `validator_index`. #### Sync committees and aggregation From 99b2cc2f3ec01387eea010e6f4c530381e176fd4 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 13 Apr 2021 14:51:20 -0700 Subject: [PATCH 073/227] Clarify usage of field in data --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 08c1682f2..738260cff 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -102,7 +102,7 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64 i.e. `state.validators[contribution_and_proof.aggregator_index].pubkey in get_sync_subcommittee_pubkeys(state, contribution.subcommittee_index)`. - _[REJECT]_ The `contribution_and_proof.selection_proof` is a valid signature of the `SyncCommitteeSigningData` derived from the `contribution` by the validator with index `contribution_and_proof.aggregator_index`. - _[REJECT]_ The aggregator signature, `signed_contribution_and_proof.signature`, is valid. -- _[REJECT]_ The aggregate signature is valid for the message `beacon_block_root` and aggregate pubkey derived from the participation info in `aggregation_bits` for the subcommittee specified by the `subcommittee_index`. +- _[REJECT]_ The aggregate signature is valid for the message `beacon_block_root` and aggregate pubkey derived from the participation info in `aggregation_bits` for the subcommittee specified by the `contribution.subcommittee_index`. #### Sync committee subnets From 3c07303c5caee4d21619231a74bd561224d869b2 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 15 Apr 2021 11:28:13 -0700 Subject: [PATCH 074/227] Update specs/altair/p2p-interface.md Co-authored-by: Danny Ryan --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 738260cff..47a5826c5 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -117,7 +117,7 @@ The following validations MUST pass before forwarding the `sync_committee_signat - _[IGNORE]_ The signature's slot is for the current slot, i.e. `sync_committee_signature.slot == current_slot`. - _[IGNORE]_ The block being signed over (`sync_committee_signature.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_signature.validator_index`. -- _[REJECT]_ The `subnet_id` is correct, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. +- _[REJECT]_ The `subnet_id` is valid for the given validator, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. Note this validation implies the validator is part of the broader current sync committee along with the correct subcommittee. - _[REJECT]_ The `signature` is valid for the message `beacon_block_root` for the validator referenced by `validator_index`. From 917d40b4d31d4ed8597fa6520415f25d1cb0ac00 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Thu, 22 Apr 2021 20:38:21 +0300 Subject: [PATCH 075/227] Fix missing argument in get_active_shard_count() call --- specs/sharding/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 250f8b7ed..55a47a09e 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -668,9 +668,10 @@ def process_pending_headers(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: return - previous_epoch_start_slot = compute_start_slot_at_epoch(get_previous_epoch(state)) + previous_epoch = get_previous_epoch(state) + previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard in range(get_active_shard_count(state)): + for shard in range(get_active_shard_count(state, previous_epoch)): # Pending headers for this (slot, shard) combo candidates = [ c for c in state.previous_epoch_pending_shard_headers From ebd16e1b799aad5242283652d822700fe3b0584d Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 15 Apr 2021 11:46:44 -0700 Subject: [PATCH 076/227] add helper in p2p document to executable spec --- specs/altair/p2p-interface.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 47a5826c5..bcd366c66 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -86,11 +86,10 @@ This topic is used to propagate partially aggregated sync committee signatures t The following validations MUST pass before forwarding the `signed_contribution_and_proof` on the network; define `contribution_and_proof = signed_contribution_and_proof.message`, `contribution = contribution_and_proof.contribution`, and the following function `get_sync_subcommittee_pubkeys` for convenience: ```python -SYNC_SUBCOMMITTEE_SIZE = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT - def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: - i = subcommittee_index * SYNC_SUBCOMMITTEE_SIZE - return state.current_sync_committee.pubkeys[i:i+SYNC_SUBCOMMITTEE_SIZE] + sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT + i = subcommittee_index * sync_subcommittee_size + return state.current_sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` - _[IGNORE]_ The contribution's slot is for the current slot, i.e. `contribution.slot == current_slot`. From d7c276bcd58e854ad22377914b337e748a0d9cdb Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Thu, 22 Apr 2021 22:06:24 +0300 Subject: [PATCH 077/227] Remove obsolete var and calculation --- specs/sharding/beacon-chain.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 55a47a09e..bd3871c6a 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -679,7 +679,6 @@ def process_pending_headers(state: BeaconState) -> None: ] # The entire committee (and its balance) full_committee = get_beacon_committee(state, slot, shard) - full_committee_balance = get_total_balance(state, full_committee) # If any candidates already confirmed, skip if True in [c.confirmed for c in candidates]: continue From 8f371f56277dbb81a4f8f0350004390cde25cbd9 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Thu, 22 Apr 2021 22:08:44 +0300 Subject: [PATCH 078/227] Cleanup: move var calculation down right before its usage --- specs/sharding/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index bd3871c6a..3cd262ed6 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -677,12 +677,12 @@ def process_pending_headers(state: BeaconState) -> None: c for c in state.previous_epoch_pending_shard_headers if (c.slot, c.shard) == (slot, shard) ] - # The entire committee (and its balance) - full_committee = get_beacon_committee(state, slot, shard) # If any candidates already confirmed, skip if True in [c.confirmed for c in candidates]: continue + # The entire committee (and its balance) + full_committee = get_beacon_committee(state, slot, shard) # The set of voters who voted for each header (and their total balances) voting_sets = [ [v for i, v in enumerate(full_committee) if c.votes[i]] From 58e768392e7c6d91f80cb74f6cfc600f610223b3 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 23 Apr 2021 08:03:49 -0700 Subject: [PATCH 079/227] Simplify spec comment processing --- setup.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup.py b/setup.py index 2b1b23cd7..37fbd7795 100644 --- a/setup.py +++ b/setup.py @@ -165,9 +165,8 @@ def get_spec(file_name: str) -> SpecObject: custom_types[name] = value elif isinstance(child, LinkRefDef): comment = _get_eth2_spec_comment(child) - if comment: - if comment == "skip": - should_skip = True + if comment == "skip": + should_skip = True return SpecObject( functions=functions, From 93378afcd05bfd70cfc228d39ebbec85085dd5d8 Mon Sep 17 00:00:00 2001 From: ericsson Date: Fri, 23 Apr 2021 18:21:29 +0300 Subject: [PATCH 080/227] Fix typing problem: `upgrade_to_altair` should use `phase0.get_current_epoch` --- specs/altair/fork.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/fork.md b/specs/altair/fork.md index 739584309..2d9f2e26c 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -40,7 +40,7 @@ After `process_slots` of Phase 0 finishes, if `state.slot % SLOTS_PER_EPOCH == 0 ```python def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: - epoch = get_current_epoch(pre) + epoch = phase0.get_current_epoch(pre) post = BeaconState( # Versioning genesis_time=pre.genesis_time, From f55b327c5160139562352b844a1f276148ecec62 Mon Sep 17 00:00:00 2001 From: ericsson Date: Fri, 23 Apr 2021 18:47:25 +0300 Subject: [PATCH 081/227] fix typos in sharding --- specs/sharding/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 3cd262ed6..a2427a2b1 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -280,7 +280,7 @@ class ShardProposerSlashing(Container): #### `next_power_of_two` ```python -def next_power_of_two(x): +def next_power_of_two(x: int) -> int: return 2 ** ((x - 1).bit_length()) ``` @@ -374,7 +374,7 @@ ensuring that the balance is always sufficient to cover gas costs. def compute_proposer_index(beacon_state: BeaconState, indices: Sequence[ValidatorIndex], seed: Bytes32, - min_effective_balance: GWei = GWei(0)) -> ValidatorIndex: + min_effective_balance: Gwei = Gwei(0)) -> ValidatorIndex: """ Return from ``indices`` a random index sampled by effective balance. """ From ee360df689af6bf4ec3edad4a45d0bb2c2dd07d2 Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 00:12:28 +0300 Subject: [PATCH 082/227] missed `body_summary` attribute when accessing`ShardBlobBodySummary.beacon_block_root` --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 3cd262ed6..c1fee19bd 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -559,7 +559,7 @@ def process_shard_header(state: BeaconState, assert header.shard < get_active_shard_count(state, header_epoch) # Verify that the block root matches, # to ensure the header will only be included in this specific Beacon Chain sub-tree. - assert header.beacon_block_root == get_block_root_at_slot(state, header.slot - 1) + assert header.body_summary.beacon_block_root == get_block_root_at_slot(state, header.slot - 1) # Verify proposer assert header.proposer_index == get_shard_proposer_index(state, header.slot, header.shard) # Verify signature From 47e52a636122b707399c81f3b0241462e9bb6073 Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 17:32:05 +0300 Subject: [PATCH 083/227] fix typo: `get_shard_proposer_index` vs `get_shard_proposer` --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index c1fee19bd..8ae66b5e6 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -728,7 +728,7 @@ def charge_confirmed_header_fees(state: BeaconState) -> None: candidate = confirmed_candidates[0] # Charge EIP 1559 fee - proposer = get_shard_proposer(state, slot, shard) + proposer = get_shard_proposer_index(state, slot, shard) fee = ( (state.shard_gasprice * candidate.commitment.length) // TARGET_SAMPLES_PER_BLOCK From 6814efac2e8934f15409bbf976aec671d1356c62 Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 17:32:59 +0300 Subject: [PATCH 084/227] fix typo: `state` missed in `get_committee_count_per_slot` call --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 8ae66b5e6..f577ab1df 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -752,7 +752,7 @@ def reset_pending_headers(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + 1 next_epoch_start_slot = compute_start_slot_at_epoch(next_epoch) for slot in range(next_epoch_start_slot, next_epoch_start_slot + SLOTS_IN_EPOCH): - for index in range(get_committee_count_per_slot(next_epoch)): + for index in range(get_committee_count_per_slot(state, next_epoch)): shard = compute_shard_from_committee_index(state, slot, index) committee_length = len(get_beacon_committee(state, slot, shard)) state.current_epoch_pending_shard_headers.append(PendingShardHeader( From 77bceeae690f8d41cbe0b08812ae8f073a216554 Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 17:33:53 +0300 Subject: [PATCH 085/227] fix typo: `SLOTS_IN_EPOCH` used instead of `SLOTS_PER_EPOCH` --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index f577ab1df..41dcca410 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -751,7 +751,7 @@ def reset_pending_headers(state: BeaconState) -> None: # Add dummy "empty" PendingShardHeader (default vote for if no shard header available) next_epoch = get_current_epoch(state) + 1 next_epoch_start_slot = compute_start_slot_at_epoch(next_epoch) - for slot in range(next_epoch_start_slot, next_epoch_start_slot + SLOTS_IN_EPOCH): + for slot in range(next_epoch_start_slot, next_epoch_start_slot + SLOTS_PER_EPOCH): for index in range(get_committee_count_per_slot(state, next_epoch)): shard = compute_shard_from_committee_index(state, slot, index) committee_length = len(get_beacon_committee(state, slot, shard)) From e05356893f9e769739ba3e61435027daffaa88d0 Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 17:38:09 +0300 Subject: [PATCH 086/227] fix typo: `c` var name used instad of `header` in `process_pending_headers` --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 41dcca410..28491731e 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -706,7 +706,7 @@ def process_pending_headers(state: BeaconState) -> None: state.grandparent_epoch_confirmed_commitments[shard][slot_index] = DataCommitment() confirmed_headers = [candidate for candidate in state.previous_epoch_pending_shard_headers if candidate.confirmed] for header in confirmed_headers: - state.grandparent_epoch_confirmed_commitments[c.shard][c.slot % SLOTS_PER_EPOCH] = c.commitment + state.grandparent_epoch_confirmed_commitments[header.shard][header.slot % SLOTS_PER_EPOCH] = header.commitment ``` ```python From 6288252d44445ce5cbdb685da3fcd0f6d119c785 Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 17:46:55 +0300 Subject: [PATCH 087/227] fix typo: `process_confirmed_header_fees` called instead of `charge_confirmed_header_fees` --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 28491731e..04a469a60 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -644,7 +644,7 @@ def process_epoch(state: BeaconState) -> None: # Sharding process_pending_headers(state) - process_confirmed_header_fees(state) + charge_confirmed_header_fees(state) reset_pending_headers(state) # Final updates From d99cfee853498f4e72a3a786fcea775a09c83b5b Mon Sep 17 00:00:00 2001 From: ericsson Date: Sat, 24 Apr 2021 18:21:59 +0300 Subject: [PATCH 088/227] typing problem fixed: `get_total_balance` expected second argument to be of `Set[ValidatorIndex]` type, however, `Sequence` is passed --- specs/sharding/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 3cd262ed6..4b53c94ea 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -539,7 +539,7 @@ def update_pending_votes(state: BeaconState, attestation: Attestation) -> None: participants = get_attesting_indices(state, attestation.data, pending_header.votes) participants_balance = get_total_balance(state, participants) full_committee = get_beacon_committee(state, attestation.data.slot, attestation.data.index) - full_committee_balance = get_total_balance(state, full_committee) + full_committee_balance = get_total_balance(state, set(full_committee)) if participants_balance * 3 >= full_committee_balance * 2: pending_header.confirmed = True ``` @@ -685,7 +685,7 @@ def process_pending_headers(state: BeaconState) -> None: full_committee = get_beacon_committee(state, slot, shard) # The set of voters who voted for each header (and their total balances) voting_sets = [ - [v for i, v in enumerate(full_committee) if c.votes[i]] + set(v for i, v in enumerate(full_committee) if c.votes[i]) for c in candidates ] voting_balances = [ From 17bc3c1c72ff2cad680f96109d0fff966e2d18f9 Mon Sep 17 00:00:00 2001 From: ericsson Date: Tue, 27 Apr 2021 13:44:22 +0300 Subject: [PATCH 089/227] convert `shard` to `CommitteeIndex` when passing to `get_beacon_committee` --- specs/sharding/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index fb687e1ca..afa04ba41 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -671,7 +671,8 @@ def process_pending_headers(state: BeaconState) -> None: previous_epoch = get_previous_epoch(state) previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard in range(get_active_shard_count(state, previous_epoch)): + for shard_index in range(get_active_shard_count(state, previous_epoch)): + shard = Shard(shard_index) # Pending headers for this (slot, shard) combo candidates = [ c for c in state.previous_epoch_pending_shard_headers @@ -682,7 +683,7 @@ def process_pending_headers(state: BeaconState) -> None: continue # The entire committee (and its balance) - full_committee = get_beacon_committee(state, slot, shard) + full_committee = get_beacon_committee(state, slot, CommitteeIndex(shard)) # The set of voters who voted for each header (and their total balances) voting_sets = [ set(v for i, v in enumerate(full_committee) if c.votes[i]) From 865a077aa3b63ded8dc6ad6c5950b73c3b121203 Mon Sep 17 00:00:00 2001 From: ericsson Date: Tue, 27 Apr 2021 14:14:25 +0300 Subject: [PATCH 090/227] convert `shard` and `index` to `CommitteeIndex` in `reset_pending_headers` --- specs/sharding/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index afa04ba41..75033a508 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -754,8 +754,8 @@ def reset_pending_headers(state: BeaconState) -> None: next_epoch_start_slot = compute_start_slot_at_epoch(next_epoch) for slot in range(next_epoch_start_slot, next_epoch_start_slot + SLOTS_PER_EPOCH): for index in range(get_committee_count_per_slot(state, next_epoch)): - shard = compute_shard_from_committee_index(state, slot, index) - committee_length = len(get_beacon_committee(state, slot, shard)) + shard = compute_shard_from_committee_index(state, slot, CommitteeIndex(index)) + committee_length = len(get_beacon_committee(state, slot, CommitteeIndex(shard))) state.current_epoch_pending_shard_headers.append(PendingShardHeader( slot=slot, shard=shard, From 4d684bd8d959414c5fef6f764f7c7a7b34053a67 Mon Sep 17 00:00:00 2001 From: ericsson Date: Tue, 27 Apr 2021 15:44:25 +0300 Subject: [PATCH 091/227] convert `shard` to `CommitteeIndex` in `charge_confirmed_header_fees` --- specs/sharding/beacon-chain.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 75033a508..eb6959f47 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -719,7 +719,8 @@ def charge_confirmed_header_fees(state: BeaconState) -> None: ) previous_epoch_start_slot = compute_start_slot_at_epoch(get_previous_epoch(state)) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard in range(SHARD_COUNT): + for shard_index in range(SHARD_COUNT): + shard = Shard(shard_index) confirmed_candidates = [ c for c in state.previous_epoch_pending_shard_headers if (c.slot, c.shard, c.confirmed) == (slot, shard, True) From a0c30313fa5a8b48eb41aba7df10022ccb70b890 Mon Sep 17 00:00:00 2001 From: ericsson Date: Tue, 27 Apr 2021 17:07:56 +0300 Subject: [PATCH 092/227] convert `shard` to `CommitteeIndex` using `compute_committee_index_from_shard`, based on @djrtwo comment --- specs/sharding/beacon-chain.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index eb6959f47..7916c0364 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -683,7 +683,8 @@ def process_pending_headers(state: BeaconState) -> None: continue # The entire committee (and its balance) - full_committee = get_beacon_committee(state, slot, CommitteeIndex(shard)) + index = compute_committee_index_from_shard(state, slot, shard) + full_committee = get_beacon_committee(state, slot, index) # The set of voters who voted for each header (and their total balances) voting_sets = [ set(v for i, v in enumerate(full_committee) if c.votes[i]) From e3c95e967cdbe57babcc86c1dcda9f68801c4e46 Mon Sep 17 00:00:00 2001 From: ericsson Date: Tue, 27 Apr 2021 17:09:16 +0300 Subject: [PATCH 093/227] use `committee_index` instead of `shard` --- specs/sharding/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 7916c0364..2ddcc6a82 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -756,8 +756,9 @@ def reset_pending_headers(state: BeaconState) -> None: next_epoch_start_slot = compute_start_slot_at_epoch(next_epoch) for slot in range(next_epoch_start_slot, next_epoch_start_slot + SLOTS_PER_EPOCH): for index in range(get_committee_count_per_slot(state, next_epoch)): - shard = compute_shard_from_committee_index(state, slot, CommitteeIndex(index)) - committee_length = len(get_beacon_committee(state, slot, CommitteeIndex(shard))) + committee_index = CommitteeIndex(index) + shard = compute_shard_from_committee_index(state, slot, committee_index) + committee_length = len(get_beacon_committee(state, slot, committee_index)) state.current_epoch_pending_shard_headers.append(PendingShardHeader( slot=slot, shard=shard, From 9bb3444c890737a981451217f66fc9c298ba930c Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 22 Apr 2021 17:18:50 -0700 Subject: [PATCH 094/227] Add `syncnets` data to Altair `MetaData`. --- specs/altair/p2p-interface.md | 37 +++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index bcd366c66..edc165c6d 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -15,6 +15,7 @@ Altair adds new messages, topics and data to the Req-Resp, Gossip and Discovery - [Warning](#warning) - [Modifications in Altair](#modifications-in-altair) + - [MetaData](#metadata) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) @@ -30,6 +31,7 @@ Altair adds new messages, topics and data to the Req-Resp, Gossip and Discovery - [Messages](#messages) - [BeaconBlocksByRange v2](#beaconblocksbyrange-v2) - [BeaconBlocksByRoot v2](#beaconblocksbyroot-v2) + - [GetMetaData v2](#getmetadata-v2) - [Transitioning from v1 to v2](#transitioning-from-v1-to-v2) - [The discovery domain: discv5](#the-discovery-domain-discv5) @@ -43,6 +45,23 @@ Refer to the note in the [validator guide](./validator.md) for further details. # Modifications in Altair +## MetaData + +The `MetaData` stored locally by clients is updated with an additional field to communicate the sync committee subnet subscriptions: + +``` +( + seq_number: uint64 + attnets: Bitvector[ATTESTATION_SUBNET_COUNT] + syncnets: Bitvector[SYNC_COMMITTEE_SUBNET_COUNT] +) +``` + +Where + +- `seq_number` and `attnets` have the same meaning defined in the Phase 0 document. +- `syncnets` is a `Bitvector` representing the node's sync committee subnet subscriptions. This field should mirror the data in the node's ENR as outlined in the [validator guide](./validator.md#sync-committee-subnet-stability). + ## The gossip domain: gossipsub Gossip meshes are added in Altair to support the consensus activities of the sync committees. @@ -211,6 +230,24 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` | | `ALTAIR_FORK_VERSION` | `altair.SignedBeaconBlock` | +#### GetMetaData v2 + +**Protocol ID:** `/eth2/beacon_chain/req/metadata/2/` + +No Request Content. + +Response Content: + +``` +( + MetaData +) +``` + +Requests the MetaData of a peer, using the new `MetaData` definition given above +that is extended from phase 0 in Altair. Other conditions for the `GetMetaData` +protocol are unchanged from the phase 0 p2p networking document. + ### Transitioning from v1 to v2 In advance of the fork, implementations can opt in to both run the v1 and v2 for a smooth transition. From ae429a38a0dc4dc060f53c4aad41146f17056c07 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 27 Apr 2021 12:11:15 -0600 Subject: [PATCH 095/227] add Modified comment --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 438c892c6..f498c40c5 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -675,7 +675,7 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, deposits: Sequence[Deposit]) -> BeaconState: fork = Fork( previous_version=GENESIS_FORK_VERSION, - current_version=ALTAIR_FORK_VERSION, + current_version=ALTAIR_FORK_VERSION, # [Modified in Altair] epoch=GENESIS_EPOCH, ) state = BeaconState( From eb9f7adaa7e87bf5148d5fcc8845e7edb270117c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 27 Apr 2021 12:42:39 -0600 Subject: [PATCH 096/227] add altair p2p to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 36c8ad510..d05834b9f 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ The current features are: * [Altair fork](specs/altair/fork.md) * [Light client sync protocol](specs/altair/sync-protocol.md) * [Honest Validator guide changes](specs/altair/validator.md) +* [P2P Networking](specs/altair/p2p-interface.md) ### Merge From 9f74f1f9e513575e53b7dfec8c0b92ccf835639f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 27 Apr 2021 15:06:45 -0600 Subject: [PATCH 097/227] minor sync committee cleanups --- specs/altair/p2p-interface.md | 2 +- specs/altair/validator.md | 81 +++++++++++++++++++---------------- 2 files changed, 46 insertions(+), 37 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index edc165c6d..4f2fd6aa8 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -118,7 +118,7 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64 - _[REJECT]_ `contribution_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_sync_committee_aggregator(contribution_and_proof.selection_proof)` returns `True`. - _[REJECT]_ The aggregator's validator index is in the declared subcommittee of the current sync committee -- i.e. `state.validators[contribution_and_proof.aggregator_index].pubkey in get_sync_subcommittee_pubkeys(state, contribution.subcommittee_index)`. -- _[REJECT]_ The `contribution_and_proof.selection_proof` is a valid signature of the `SyncCommitteeSigningData` derived from the `contribution` by the validator with index `contribution_and_proof.aggregator_index`. +- _[REJECT]_ The `contribution_and_proof.selection_proof` is a valid signature of the `SyncAggregatorSelectionData` derived from the `contribution` by the validator with index `contribution_and_proof.aggregator_index`. - _[REJECT]_ The aggregator signature, `signed_contribution_and_proof.signature`, is valid. - _[REJECT]_ The aggregate signature is valid for the message `beacon_block_root` and aggregate pubkey derived from the participation info in `aggregation_bits` for the subcommittee specified by the `contribution.subcommittee_index`. diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 69ae34ec8..3d8258df9 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -18,7 +18,7 @@ This is an accompanying document to [Ethereum 2.0 Altair -- The Beacon Chain](./ - [`SyncCommitteeContribution`](#synccommitteecontribution) - [`ContributionAndProof`](#contributionandproof) - [`SignedContributionAndProof`](#signedcontributionandproof) - - [`SyncCommitteeSigningData`](#synccommitteesigningdata) + - [`SyncAggregatorSelectionData`](#syncaggregatorselectiondata) - [Validator assignments](#validator-assignments) - [Sync Committee](#sync-committee) - [Lookahead](#lookahead) @@ -49,7 +49,7 @@ This is an accompanying document to [Ethereum 2.0 Altair -- The Beacon Chain](./ ## Introduction -This document represents the expected behavior of an "honest validator" with respect to Altair of the Ethereum 2.0 protocol. +This document represents the expected behavior of an "honest validator" with respect to the Altair upgrade of the Ethereum 2.0 protocol. It builds on the [previous document for the behavior of an "honest validator" from Phase 0](../phase0/validator.md) of the Ethereum 2.0 protocol. This previous document is referred to below as the "Phase 0 document". @@ -126,10 +126,10 @@ class SignedContributionAndProof(Container): signature: BLSSignature ``` -### `SyncCommitteeSigningData` +### `SyncAggregatorSelectionData` ```python -class SyncCommitteeSigningData(Container): +class SyncAggregatorSelectionData(Container): slot: Slot subcommittee_index: uint64 ``` @@ -178,7 +178,7 @@ For this reason, *always* get committee assignments via the fields of the `Beaco A validator should plan for future sync committee assignments by noting which sync committee periods they are selected for participation. Specifically, a validator should: * Upon (re)syncing the chain and upon sync committee period boundaries, check for assignments in the current and next sync committee periods. -* If the validator is in the current sync committee period, they can perform the responsibilities below for sync committee rewards. +* If the validator is in the current sync committee period, then they perform the responsibilities below for sync committee rewards. * If the validator is in the next sync committee period, they should wait until the next `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` boundary and then perform the responsibilities throughout that period. ## Beacon chain responsibilities @@ -187,7 +187,7 @@ A validator maintains the responsibilities given in the Phase 0 document. Block proposals are modified to incorporate the sync committee signatures as detailed below. -When assigned to a sync committee, validators have a new responsibility to sign beacon block roots during each slot of the sync committee period. +When assigned to a sync committee, validators have a new responsibility to sign and broadcast beacon block roots during each slot of the sync committee period. These signatures are aggregated and routed to the proposer over gossip for inclusion into a beacon block. Assignments to a particular sync committee are infrequent at normal validator counts; however, an action every slot is required when in the current active sync committee. @@ -209,8 +209,8 @@ After constructing the `BeaconBlockBody` as per that section, the proposer has a The proposer receives a number of `SyncCommitteeContribution`s (wrapped in `SignedContributionAndProof`s on the wire) from validators in the sync committee who are selected to partially aggregate signatures from independent subcommittees formed by breaking the full sync committee into `SYNC_COMMITTEE_SUBNET_COUNT` pieces (see below for details). -The proposer collects these contributions for further aggregation when preparing a block. -Proposers should select the best contribution seen across all aggregators for each subnet/subcommittee when preparing a block. +The proposer collects the contributions that match their local view of the chain (i.e. `contribution.beacon_block_root == block.parent_root`) for further aggregation when preparing a block. +Of these contributions, proposers should select the best contribution seen across all aggregators for each subnet/subcommitte. A contribution with more valid signatures is better than a contribution with fewer signatures. Recall `block.body.sync_aggregate.sync_committee_bits` is a `Bitvector` where the `i`th bit is `True` if the corresponding validator in the sync committee has produced a valid signature, @@ -229,7 +229,8 @@ def process_sync_committee_contributions(block: BeaconBlock, subcommittee_index = contribution.subcommittee_index for index, participated in enumerate(contribution.aggregation_bits): if participated: - participant_index = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT * subcommittee_index + index + sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT + participant_index = sync_subcommittee_size * subcommittee_index + index sync_aggregate.sync_committee_bits[participant_index] = True signatures.append(contribution.signature) @@ -252,20 +253,20 @@ There is no change compared to the phase 0 document. ### Sync committees -Sync committee members employ an aggregation scheme to reduce load on the global proposer channel that is monitored by all potential proposers to be able to include the full output of the sync committee every slot. -Sync committee members produce individual signatures on subnets (similar to the attestation subnets) via `SyncCommitteeSignature`s which are then collected by aggregators sampled from the sync subcommittees to produce a `SyncCommitteeContribution` which is gossiped to proposers. +Sync committee members employ an aggregation scheme to reduce load on the global proposer channel that is monitored by all potential proposers to be able to include the full output of the sync committee every slot. +Sync committee members produce individual signatures on subnets (similar to the attestation subnets) via `SyncCommitteeSignature`s which are then collected by aggregators sampled from the sync subcommittees to produce a `SyncCommitteeContribution` which is gossiped to proposers. This process occurs each slot. #### Sync committee signatures ##### Prepare sync committee signature -If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee` above returns `True`), then for every slot in the current sync committee period the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. +If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every slot in the current sync committee period the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. -`get_sync_committee_signature` assumes `state` is the head state corresponding to processing the block at the current slot as determined by the fork choice (including any empty slots processed with `process_slots`), `block_root` is the root of the head block whose processing results in `state`, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. +`get_sync_committee_signature()` assumes `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice (including any empty slots up to the current slot processed with `process_slots` on top of the latest block), `block_root` is the root of the head block, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. ```python def get_sync_committee_signature(state: BeaconState, @@ -285,14 +286,17 @@ def get_sync_committee_signature(state: BeaconState, The validator broadcasts the assembled signature to the assigned subnet, the `sync_committee_{subnet_id}` pubsub topic. The `subnet_id` is derived from the position in the sync committee such that the sync committee is divided into "subcommittees". -It can be computed via `compute_subnets_for_sync_committee` where `state` is a `BeaconState` during the matching sync committee period. -This function returns multiple subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. +`subnet_id` can be computed via `compute_subnets_for_sync_committee()` where `state` is a `BeaconState` during the matching sync committee period. + +*Note*: This function returns multiple subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. ```python def compute_subnets_for_sync_committee(state: BeaconState, validator_index: ValidatorIndex) -> Sequence[uint64]: target_pubkey = state.validators[validator_index].pubkey - sync_committee_indices = [index for index, pubkey in enumerate(state.current_sync_committee.pubkeys) - if pubkey == target_pubkey] + sync_committee_indices = [ + index for index, pubkey in enumerate(state.current_sync_committee.pubkeys) + if pubkey == target_pubkey + ] return [ uint64(index // (SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT)) for index in sync_committee_indices @@ -305,18 +309,20 @@ def compute_subnets_for_sync_committee(state: BeaconState, validator_index: Vali #### Sync committee contributions -Each slot some sync committee members in each subcommittee are selected to aggregate the `SyncCommitteeSignature`s into a `SyncCommitteeContribution` which is broadcast on a global channel for inclusion into the next block. +Each slot, some sync committee members in each subcommittee are selected to aggregate the `SyncCommitteeSignature`s into a `SyncCommitteeContribution` which is broadcast on a global channel for inclusion into the next block. ##### Aggregation selection -A validator is selected to aggregate based on the computation in `is_sync_committee_aggregator` where `signature` is the BLS signature returned by `get_sync_committee_selection_proof`. +A validator is selected to aggregate based on the value returned by `is_sync_committee_aggregator()` where `signature` is the BLS signature returned by `get_sync_committee_selection_proof()`. The signature function takes a `BeaconState` with the relevant sync committees for the queried `slot` (i.e. `state.slot` is within the span covered by the current or next sync committee period), the `subcommittee_index` equal to the `subnet_id`, and the `privkey` is the BLS private key associated with the validator. ```python -def get_sync_committee_selection_proof(state: BeaconState, slot: Slot, - subcommittee_index: uint64, privkey: int) -> BLSSignature: +def get_sync_committee_selection_proof(state: BeaconState, + slot: Slot, + subcommittee_index: uint64, + privkey: int) -> BLSSignature: domain = get_domain(state, DOMAIN_SYNC_COMMITTEE_SELECTION_PROOF, compute_epoch_at_slot(slot)) - signing_data = SyncCommitteeSigningData( + signing_data = SyncAggregatorSelectionData( slot=slot, subcommittee_index=subcommittee_index, ) @@ -330,7 +336,7 @@ def is_sync_committee_aggregator(signature: BLSSignature) -> bool: return bytes_to_uint64(hash(signature)[0:8]) % modulo == 0 ``` -*NOTE*: the set of aggregators generally changes every slot; however, the assignments can be computed ahead of time as soon as the committee is known +*NOTE*: The set of aggregators generally changes every slot; however, the assignments can be computed ahead of time as soon as the committee is known. ##### Construct sync committee contribution @@ -353,18 +359,21 @@ Set `contribution.subcommittee_index` to the index for the subcommittee index co ###### Aggregation bits Let `contribution.aggregation_bits` be a `Bitvector[SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT]`, where the `index`th bit is set in the `Bitvector` for each corresponding validator included in this aggregate from the corresponding subcommittee. -An aggregator needs to find the index in the sync committee (as returned by `get_sync_committee_indices`) for a given validator referenced by `sync_committee_signature.validator_index` and map the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is the one set in the `Bitvector`. +An aggregator finds the index in the sync committee (as returned by `get_sync_committee_indices()`) for a given validator referenced by `sync_committee_signature.validator_index` and maps the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is set in `contribution.aggegration_bits`. + For example, if a validator with index `2044` is pseudo-randomly sampled to sync committee index `135`. This sync committee index maps to `subcommittee_index` `1` with position `7` in the `Bitvector` for the contribution. -Also note that a validator **could be included multiple times** in a given subcommittee such that multiple bits are set for a single `SyncCommitteeSignature`. + +*Note*: A validator **could be included multiple times** in a given subcommittee such that multiple bits are set for a single `SyncCommitteeSignature`. ###### Signature -Set `contribution.signature = aggregate_signature` where `aggregate_signature` is obtained by assembling the appropriate collection of `BLSSignature`s from the set of `sync_committee_signatures` and using the `bls.Aggregate` function to produce an aggregate `BLSSignature`. +Set `contribution.signature = aggregate_signature` where `aggregate_signature` is obtained by assembling the appropriate collection of `BLSSignature`s from the set of `sync_committee_signatures` and using the `bls.Aggregate()` function to produce an aggregate `BLSSignature`. + The collection of input signatures should include one signature per validator who had a bit set in the `aggregation_bits` bitfield, with repeated signatures if one validator maps to multiple indices within the subcommittee. ##### Broadcast sync committee contribution -If the validator is selected to aggregate (`is_sync_committee_aggregator`), then they broadcast their best aggregate as a `SignedContributionAndProof` to the global aggregate channel (`sync_committee_contribution_and_proof` topic) two-thirds of the way through the `slot`-that is, `SECONDS_PER_SLOT * 2 / 3` seconds after the start of `slot`. +If the validator is selected to aggregate (`is_sync_committee_aggregator()`), then they broadcast their best aggregate as a `SignedContributionAndProof` to the global aggregate channel (`sync_committee_contribution_and_proof` topic) two-thirds of the way through the `slot`-that is, `SECONDS_PER_SLOT * 2 / 3` seconds after the start of `slot`. Selection proofs are provided in `ContributionAndProof` to prove to the gossip channel that the validator has been selected as an aggregator. @@ -404,19 +413,19 @@ def get_contribution_and_proof_signature(state: BeaconState, ## Sync committee subnet stability -The sync committee subnets need special care to ensure stability given the relatively low number of validators involved in the sync committee at any particular time. +The sync committee subnets need special care to ensure stability given the relatively low number of validators involved in the sync committee at any particular time. To provide this stability, a validator must do the following: -* Maintain advertisement of the subnet the validator in the sync committee is assigned to in their node's ENR as soon as they have joined the subnet. -Subnet assignments are known `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs in advance and can be computed with `compute_subnets_for_sync_committee` defined above. -ENR advertisement is indicated by setting the appropriate bit(s) of the bitfield found under the `syncnets` key in the ENR corresponding to the derived `subnet_id`(s). -Any bits modified for the sync committee responsibilities are unset in the ENR after any validators have left the sync committee. +* Maintain advertisement of the subnet the validator in the sync committee is assigned to in their node's ENR as soon as they have joined the subnet. +Subnet assignments are known `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs in advance and can be computed with `compute_subnets_for_sync_committee` defined above. +ENR advertisement is indicated by setting the appropriate bit(s) of the bitfield found under the `syncnets` key in the ENR corresponding to the derived `subnet_id`(s). +Any bits modified for the sync committee responsibilities are unset in the ENR once the node no longer has any validators in the subcommittee. *Note*: The first sync committee from phase 0 to the Altair fork will not be known until the fork happens which implies subnet assignments are not known until then. -Early sync committee members should listen for topic subscriptions from peers and employ discovery via the ENR advertisements near the fork boundary to form initial subnets. +Early sync committee members should listen for topic subscriptions from peers and employ discovery via the ENR advertisements near the fork boundary to form initial subnets. Some early sync committee rewards may be missed while the initial subnets form. -* To join a sync committee subnet, select a random number of epochs before the end of the current sync committee period between 1 and `SYNC_COMMITTEE_SUBNET_COUNT`, inclusive. -Validators should join their member subnet at the beginning of the epoch they have randomly selected. -For example, if the next sync committee period starts at epoch `853,248` and the validator randomly selects an offset of `3`, they should join the subnet at the beginning of epoch `853,245`. +* To join a sync committee subnet, select a random number of epochs before the end of the current sync committee period between 1 and `SYNC_COMMITTEE_SUBNET_COUNT`, inclusive. +Validators should join their member subnet at the beginning of the epoch they have randomly selected. +For example, if the next sync committee period starts at epoch `853,248` and the validator randomly selects an offset of `3`, they should join the subnet at the beginning of epoch `853,245`. Validators should leverage the lookahead period on sync committee assignments so that they can join the appropriate subnets ahead of their assigned sync committee period. From e89fae86a6bda811caa1e099c1ef366ca77a9fd3 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 28 Apr 2021 06:52:16 -0600 Subject: [PATCH 098/227] Apply suggestions from code review Co-authored-by: Hsiao-Wei Wang --- specs/altair/validator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 3d8258df9..c69e490b1 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -210,7 +210,7 @@ After constructing the `BeaconBlockBody` as per that section, the proposer has a The proposer receives a number of `SyncCommitteeContribution`s (wrapped in `SignedContributionAndProof`s on the wire) from validators in the sync committee who are selected to partially aggregate signatures from independent subcommittees formed by breaking the full sync committee into `SYNC_COMMITTEE_SUBNET_COUNT` pieces (see below for details). The proposer collects the contributions that match their local view of the chain (i.e. `contribution.beacon_block_root == block.parent_root`) for further aggregation when preparing a block. -Of these contributions, proposers should select the best contribution seen across all aggregators for each subnet/subcommitte. +Of these contributions, proposers should select the best contribution seen across all aggregators for each subnet/subcommittee. A contribution with more valid signatures is better than a contribution with fewer signatures. Recall `block.body.sync_aggregate.sync_committee_bits` is a `Bitvector` where the `i`th bit is `True` if the corresponding validator in the sync committee has produced a valid signature, @@ -261,7 +261,7 @@ This process occurs each slot. ##### Prepare sync committee signature -If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every slot in the current sync committee period the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. +If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every slot in the current sync committee period, the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. @@ -421,7 +421,7 @@ Subnet assignments are known `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs in advanc ENR advertisement is indicated by setting the appropriate bit(s) of the bitfield found under the `syncnets` key in the ENR corresponding to the derived `subnet_id`(s). Any bits modified for the sync committee responsibilities are unset in the ENR once the node no longer has any validators in the subcommittee. - *Note*: The first sync committee from phase 0 to the Altair fork will not be known until the fork happens which implies subnet assignments are not known until then. + *Note*: The first sync committee from phase 0 to the Altair fork will not be known until the fork happens, which implies subnet assignments are not known until then. Early sync committee members should listen for topic subscriptions from peers and employ discovery via the ENR advertisements near the fork boundary to form initial subnets. Some early sync committee rewards may be missed while the initial subnets form. From c5986106ca987c6e46163d0bfbf92732fc6170ba Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 29 Apr 2021 15:39:57 +0200 Subject: [PATCH 099/227] fix note about sharding beacon state extension --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 2ddcc6a82..6fd83c9cf 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -166,7 +166,7 @@ class BeaconBlockBody(merge.BeaconBlockBody): # [extends The Merge block body] ### `BeaconState` ```python -class BeaconState(merge.BeaconState): # [extends The Merge block body] +class BeaconState(merge.BeaconState): # [extends The Merge state] # [Updated fields] previous_epoch_attestations: List[PendingAttestation, MAX_ATTESTATIONS * SLOTS_PER_EPOCH] current_epoch_attestations: List[PendingAttestation, MAX_ATTESTATIONS * SLOTS_PER_EPOCH] From 86d8a10495967671c192de6fe93dfb76c6ae7f52 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 30 Apr 2021 14:09:09 +0300 Subject: [PATCH 100/227] Replace unspecified SHARD_COUNT const with get_active_shard_count(previous_epoch) --- specs/sharding/beacon-chain.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 6fd83c9cf..ef7dadfc2 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -670,8 +670,9 @@ def process_pending_headers(state: BeaconState) -> None: previous_epoch = get_previous_epoch(state) previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) + active_shard_count = get_active_shard_count(state, previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard_index in range(get_active_shard_count(state, previous_epoch)): + for shard_index in range(active_shard_count): shard = Shard(shard_index) # Pending headers for this (slot, shard) combo candidates = [ @@ -704,7 +705,7 @@ def process_pending_headers(state: BeaconState) -> None: winning_index = [c.root for c in candidates].index(Root()) candidates[winning_index].confirmed = True for slot_index in range(SLOTS_PER_EPOCH): - for shard in range(SHARD_COUNT): + for shard in range(active_shard_count): state.grandparent_epoch_confirmed_commitments[shard][slot_index] = DataCommitment() confirmed_headers = [candidate for candidate in state.previous_epoch_pending_shard_headers if candidate.confirmed] for header in confirmed_headers: @@ -718,9 +719,10 @@ def charge_confirmed_header_fees(state: BeaconState) -> None: get_active_shard_count(state, get_current_epoch(state)) * SLOTS_PER_EPOCH * GASPRICE_ADJUSTMENT_COEFFICIENT ) - previous_epoch_start_slot = compute_start_slot_at_epoch(get_previous_epoch(state)) + previous_epoch = get_previous_epoch(state) + previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard_index in range(SHARD_COUNT): + for shard_index in range(get_active_shard_count(state, previous_epoch)): shard = Shard(shard_index) confirmed_candidates = [ c for c in state.previous_epoch_pending_shard_headers From 36fd9195f3302fa30a29e3a51ece287566e75069 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 30 Apr 2021 14:14:09 +0300 Subject: [PATCH 101/227] Replace unspecified DOMAIN_SHARD_HEADER const with DOMAIN_SHARD_PROPOSER --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index ef7dadfc2..9f945270d 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -563,7 +563,7 @@ def process_shard_header(state: BeaconState, # Verify proposer assert header.proposer_index == get_shard_proposer_index(state, header.slot, header.shard) # Verify signature - signing_root = compute_signing_root(header, get_domain(state, DOMAIN_SHARD_HEADER)) + signing_root = compute_signing_root(header, get_domain(state, DOMAIN_SHARD_PROPOSER)) assert bls.Verify(state.validators[header.proposer_index].pubkey, signing_root, signed_header.signature) # Verify the length by verifying the degree. From 637f232ab3957a0f32454eefe1608fdadddb1e4a Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 30 Apr 2021 14:35:34 +0300 Subject: [PATCH 102/227] Need to specify MAX_SHARDS since we are filling the Vector of MAX_SHARDS length --- specs/sharding/beacon-chain.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 9f945270d..7082b453b 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -670,9 +670,8 @@ def process_pending_headers(state: BeaconState) -> None: previous_epoch = get_previous_epoch(state) previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) - active_shard_count = get_active_shard_count(state, previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard_index in range(active_shard_count): + for shard_index in range(get_active_shard_count(state, previous_epoch)): shard = Shard(shard_index) # Pending headers for this (slot, shard) combo candidates = [ @@ -705,7 +704,7 @@ def process_pending_headers(state: BeaconState) -> None: winning_index = [c.root for c in candidates].index(Root()) candidates[winning_index].confirmed = True for slot_index in range(SLOTS_PER_EPOCH): - for shard in range(active_shard_count): + for shard in range(MAX_SHARDS): state.grandparent_epoch_confirmed_commitments[shard][slot_index] = DataCommitment() confirmed_headers = [candidate for candidate in state.previous_epoch_pending_shard_headers if candidate.confirmed] for header in confirmed_headers: @@ -719,10 +718,9 @@ def charge_confirmed_header_fees(state: BeaconState) -> None: get_active_shard_count(state, get_current_epoch(state)) * SLOTS_PER_EPOCH * GASPRICE_ADJUSTMENT_COEFFICIENT ) - previous_epoch = get_previous_epoch(state) - previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) + previous_epoch_start_slot = compute_start_slot_at_epoch(get_previous_epoch(state)) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard_index in range(get_active_shard_count(state, previous_epoch)): + for shard_index in range(SHARD_COUNT): shard = Shard(shard_index) confirmed_candidates = [ c for c in state.previous_epoch_pending_shard_headers From 99d50108d0f3d0a867dab7785371d7cdd9b47065 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 30 Apr 2021 14:39:37 +0300 Subject: [PATCH 103/227] Replace unspecified SHARD_COUNT const with get_active_shard_count(previous_epoch) --- specs/sharding/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 7082b453b..bddc24a40 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -718,9 +718,10 @@ def charge_confirmed_header_fees(state: BeaconState) -> None: get_active_shard_count(state, get_current_epoch(state)) * SLOTS_PER_EPOCH * GASPRICE_ADJUSTMENT_COEFFICIENT ) - previous_epoch_start_slot = compute_start_slot_at_epoch(get_previous_epoch(state)) + previous_epoch = get_previous_epoch(state) + previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - for shard_index in range(SHARD_COUNT): + for shard_index in range(get_active_shard_count(state, previous_epoch)): shard = Shard(shard_index) confirmed_candidates = [ c for c in state.previous_epoch_pending_shard_headers From 2199b8e0f08651edd2c7af7b26f75b01bd57868a Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 30 Apr 2021 14:58:30 +0300 Subject: [PATCH 104/227] Calc the right root with respect to (slot, shard) for an empty PendingShardHeader --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 6fd83c9cf..df52cce41 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -763,7 +763,7 @@ def reset_pending_headers(state: BeaconState) -> None: slot=slot, shard=shard, commitment=DataCommitment(), - root=Root(), + root=hash_tree_root(ShardBlobHeader(slot = slot, shard = shard)), votes=Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length), confirmed=False, )) From dca6d3370a0c0095133dd0fddc32b3052fb61d39 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 30 Apr 2021 15:21:51 +0300 Subject: [PATCH 105/227] Search winning 'empty' PendingShardHeader index by the empty DataCommitment instead of zero Root --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index df52cce41..68f6ccaa2 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -701,7 +701,7 @@ def process_pending_headers(state: BeaconState) -> None: winning_index = voting_balances.index(max(voting_balances)) else: # If no votes, zero wins - winning_index = [c.root for c in candidates].index(Root()) + winning_index = [c.commitment for c in candidates].index(DataCommitment()) candidates[winning_index].confirmed = True for slot_index in range(SLOTS_PER_EPOCH): for shard in range(SHARD_COUNT): From 68d6e4319a5a7292653606b9ff2dcaac2c3c0c5c Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Sat, 1 May 2021 16:30:23 -0700 Subject: [PATCH 106/227] Adjust sync committee size and duration --- configs/mainnet/altair.yaml | 8 ++++---- specs/altair/beacon-chain.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..32b75fc9c 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -12,8 +12,8 @@ PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 # Misc # --------------------------------------------------------------- -# 2**10 (= 1,024) -SYNC_COMMITTEE_SIZE: 1024 +# 2**9 (= 512) +SYNC_COMMITTEE_SIZE: 512 # 2**6 (= 64) SYNC_PUBKEYS_PER_AGGREGATE: 64 # 2**2 (= 4) @@ -22,8 +22,8 @@ INACTIVITY_SCORE_BIAS: 4 # Time parameters # --------------------------------------------------------------- -# 2**8 (= 256) -EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 256 +# 2**9 (= 512) +EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 512 # Signature domains diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f498c40c5..aa961d956 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -118,7 +118,7 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | | - | - | -| `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | +| `SYNC_COMMITTEE_SIZE` | `uint64(2**9)` (= 512) | | `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | @@ -126,7 +126,7 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**9)` (= 512) | epochs | ~54 hours | ### Domain types From 780121f9b042edf58bc4010e6f9dd4934ca761c0 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sun, 2 May 2021 20:31:52 -0700 Subject: [PATCH 107/227] Add backticks to the arguments in functional comments --- specs/altair/beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f498c40c5..2e1d2d070 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -267,7 +267,7 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ```python def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices (which may include duplicate indices) for a given state and epoch. + Return the sequence of sync committee indices (which may include duplicate indices) for a given ``state`` and ``epoch``. """ MAX_RANDOM_BYTE = 2**8 - 1 base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) @@ -292,7 +292,7 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val ```python def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ - Return the sync committee for a given state and epoch. + Return the sync committee for a given ``state`` and ``epoch``. """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] @@ -323,7 +323,7 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: ```python def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epoch: Epoch) -> Set[ValidatorIndex]: """ - Return the active and unslashed validator indices for the given epoch and flag index. + Return the active and unslashed validator indices for the given ``epoch`` and ``flag_index``. """ assert epoch in (get_previous_epoch(state), get_current_epoch(state)) if epoch == get_current_epoch(state): @@ -340,7 +340,7 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo ```python def get_flag_index_deltas(state: BeaconState, flag_index: int, weight: uint64) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ - Return the deltas for a given flag index by scanning through the participation flags. + Return the deltas for a given ``flag_index`` by scanning through the participation flags. """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) From 930ca61690aa5c57cb1242ea290d831a3020eaa5 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sun, 2 May 2021 20:47:57 -0700 Subject: [PATCH 108/227] Fix lint (124 > 120 characters) --- specs/altair/beacon-chain.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 2e1d2d070..0e7be47e3 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -267,7 +267,8 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ```python def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices (which may include duplicate indices) for a given ``state`` and ``epoch``. + Return the sequence of sync committee indices (which may include duplicate indices) + for a given ``state`` and ``epoch``. """ MAX_RANDOM_BYTE = 2**8 - 1 base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) From 053179cee9343b9b87fac362e04d856778b2e07f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 4 May 2021 07:39:22 -0600 Subject: [PATCH 109/227] comments and tests on altair --- specs/altair/beacon-chain.md | 31 ++++++++++++++----- specs/altair/sync-protocol.md | 3 +- .../test_process_sync_committee.py | 10 ++++++ 3 files changed, 35 insertions(+), 9 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f498c40c5..5426b9cf0 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -237,6 +237,9 @@ def eth2_fast_aggregate_verify(pubkeys: Sequence[BLSPubkey], message: Bytes32, s ```python def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: + """ + Return paired tuples of participation flag indices along with associated incentivization weights. + """ return ( (TIMELY_HEAD_FLAG_INDEX, TIMELY_HEAD_WEIGHT), (TIMELY_SOURCE_FLAG_INDEX, TIMELY_SOURCE_WEIGHT), @@ -248,6 +251,9 @@ def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: ```python def add_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: + """ + Return a new ``ParticipationFlags`` adding ``flag_index`` to ``flags``. + """ flag = ParticipationFlags(2**flag_index) return flags | flag ``` @@ -256,6 +262,9 @@ def add_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: ```python def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: + """ + Return whether ``flags`` has ``flag_index`` set. + """ flag = ParticipationFlags(2**flag_index) return flags & flag == flag ``` @@ -310,10 +319,16 @@ def get_base_reward_per_increment(state: BeaconState) -> Gwei: #### `get_base_reward` -*Note*: The function `get_base_reward` is modified with the removal of `BASE_REWARDS_PER_EPOCH`. +*Note*: The function `get_base_reward` is modified with the removal of `BASE_REWARDS_PER_EPOCH` and the use of increment based accounting. ```python def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: + """ + Return the base reward for the validator defined by ``index`` with respect to the current ``state``. + + Note: A validator can optimally earn one base reward per epoch over a long time horizon. + This takes into account both per-epoch (e.g. attestation) and intermittent duties (e.g. block proposal and sync committees) + """ increments = state.validators[index].effective_balance // EFFECTIVE_BALANCE_INCREMENT return Gwei(increments * get_base_reward_per_increment(state)) ``` @@ -323,7 +338,7 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: ```python def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epoch: Epoch) -> Set[ValidatorIndex]: """ - Return the active and unslashed validator indices for the given epoch and flag index. + Return the set of validator indicies that are both active and unslashed for the given ``flag_index`` and ``epoch``. """ assert epoch in (get_previous_epoch(state), get_current_epoch(state)) if epoch == get_current_epoch(state): @@ -340,7 +355,7 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo ```python def get_flag_index_deltas(state: BeaconState, flag_index: int, weight: uint64) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ - Return the deltas for a given flag index by scanning through the participation flags. + Return the deltas for a given ``flag_index`` scaled by ``weight`` by scanning through the participation flags. """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) @@ -454,9 +469,9 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: justified_checkpoint = state.previous_justified_checkpoint # Matching roots - is_matching_head = data.beacon_block_root == get_block_root_at_slot(state, data.slot) is_matching_source = data.source == justified_checkpoint - is_matching_target = data.target.root == get_block_root(state, data.target.epoch) + is_matching_target = is_matching_source and data.target.root == get_block_root(state, data.target.epoch) + is_matching_head = is_matching_target and data.beacon_block_root == get_block_root_at_slot(state, data.slot) assert is_matching_source # Verify signature @@ -464,12 +479,12 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Participation flag indices participation_flag_indices = [] - if is_matching_head and is_matching_target and state.slot == data.slot + MIN_ATTESTATION_INCLUSION_DELAY: - participation_flag_indices.append(TIMELY_HEAD_FLAG_INDEX) if is_matching_source and state.slot <= data.slot + integer_squareroot(SLOTS_PER_EPOCH): participation_flag_indices.append(TIMELY_SOURCE_FLAG_INDEX) if is_matching_target and state.slot <= data.slot + SLOTS_PER_EPOCH: participation_flag_indices.append(TIMELY_TARGET_FLAG_INDEX) + if is_matching_head and state.slot == data.slot + MIN_ATTESTATION_INCLUSION_DELAY: + participation_flag_indices.append(TIMELY_HEAD_FLAG_INDEX) # Update epoch participation flags proposer_reward_numerator = 0 @@ -487,7 +502,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: #### Modified `process_deposit` -*Note*: The function `process_deposit` is modified to initialize `inactivity_scores`, `previous_epoch_participation`, `current_epoch_participation`. +*Note*: The function `process_deposit` is modified to initialize `inactivity_scores`, `previous_epoch_participation`, and `current_epoch_participation`. ```python def process_deposit(state: BeaconState, deposit: Deposit) -> None: diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index 28705803b..925c882ef 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -115,7 +115,8 @@ A light client maintains its state in a `store` object of type `LightClientStore #### `validate_light_client_update` ```python -def validate_light_client_update(snapshot: LightClientSnapshot, update: LightClientUpdate, +def validate_light_client_update(snapshot: LightClientSnapshot, + update: LightClientUpdate, genesis_validators_root: Root) -> None: # Verify update slot is larger than snapshot slot assert update.header.slot > snapshot.header.slot diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 307d0f82d..e0faf3d0d 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -240,6 +240,16 @@ def test_sync_committee_rewards_not_full_participants(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) +@with_all_phases_except([PHASE0]) +@spec_state_test +@always_bls +def test_sync_committee_rewards_empty_participants(spec, state): + committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_bits = [False for _ in committee] + + yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + + @with_all_phases_except([PHASE0]) @spec_state_test @always_bls From 9980d2098dad7f14342d781eb8389b9dfb6a1136 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 4 May 2021 22:21:58 +0800 Subject: [PATCH 110/227] Fix lint --- specs/altair/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 26fce7fdd..726747c59 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -328,7 +328,8 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: Return the base reward for the validator defined by ``index`` with respect to the current ``state``. Note: A validator can optimally earn one base reward per epoch over a long time horizon. - This takes into account both per-epoch (e.g. attestation) and intermittent duties (e.g. block proposal and sync committees) + This takes into account both per-epoch (e.g. attestation) and intermittent duties (e.g. block proposal + and sync committees). """ increments = state.validators[index].effective_balance // EFFECTIVE_BALANCE_INCREMENT return Gwei(increments * get_base_reward_per_increment(state)) @@ -339,7 +340,7 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: ```python def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epoch: Epoch) -> Set[ValidatorIndex]: """ - Return the set of validator indicies that are both active and unslashed for the given ``flag_index`` and ``epoch``. + Return the set of validator indices that are both active and unslashed for the given ``flag_index`` and ``epoch``. """ assert epoch in (get_previous_epoch(state), get_current_epoch(state)) if epoch == get_current_epoch(state): From 0438f2f27c5c3eb1b43c940962fec702361ee1b2 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Sat, 1 May 2021 16:07:43 -0700 Subject: [PATCH 111/227] whitespace --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 726747c59..25b8bae54 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -409,7 +409,7 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S #### Modified `slash_validator` -*Note*: The function `slash_validator` is modified to use `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` +*Note*: The function `slash_validator` is modified to use `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` and use `PROPOSER_WEIGHT` when calculating the proposer reward. ```python From 7b33c1119a383ea693ada3793699e0bb1d43c8de Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Sat, 1 May 2021 16:08:36 -0700 Subject: [PATCH 112/227] simplify sync committee pubkey aggregation in altair --- configs/mainnet/altair.yaml | 2 -- configs/minimal/altair.yaml | 2 -- specs/altair/beacon-chain.md | 8 +++----- 3 files changed, 3 insertions(+), 9 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..46b74b3aa 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -14,8 +14,6 @@ PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 # --------------------------------------------------------------- # 2**10 (= 1,024) SYNC_COMMITTEE_SIZE: 1024 -# 2**6 (= 64) -SYNC_PUBKEYS_PER_AGGREGATE: 64 # 2**2 (= 4) INACTIVITY_SCORE_BIAS: 4 diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index f9b30eea2..42e464ab4 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -14,8 +14,6 @@ PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 # --------------------------------------------------------------- # [customized] SYNC_COMMITTEE_SIZE: 32 -# [customized] -SYNC_PUBKEYS_PER_AGGREGATE: 16 # 2**2 (= 4) INACTIVITY_SCORE_BIAS: 4 diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 25b8bae54..96a14fe21 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -119,7 +119,6 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | | - | - | | `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | -| `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | ### Time parameters @@ -212,7 +211,7 @@ class SyncAggregate(Container): ```python class SyncCommittee(Container): pubkeys: Vector[BLSPubkey, SYNC_COMMITTEE_SIZE] - pubkey_aggregates: Vector[BLSPubkey, SYNC_COMMITTEE_SIZE // SYNC_PUBKEYS_PER_AGGREGATE] + aggregate_pubkey: BLSPubkey ``` ## Helper functions @@ -306,9 +305,8 @@ def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] - partition = [pubkeys[i:i + SYNC_PUBKEYS_PER_AGGREGATE] for i in range(0, len(pubkeys), SYNC_PUBKEYS_PER_AGGREGATE)] - pubkey_aggregates = [bls.AggregatePKs(preaggregate) for preaggregate in partition] - return SyncCommittee(pubkeys=pubkeys, pubkey_aggregates=pubkey_aggregates) + aggregate_pubkey = bls.AggregatePKs(pubkeys) + return SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=aggregate_pubkey) ``` #### `get_base_reward_per_increment` From 9c3d5982cfbe9a52b02e2bd028a873c9226a34c9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 12:16:39 -0700 Subject: [PATCH 113/227] add documentation about duplicate pubkeys --- specs/altair/beacon-chain.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 96a14fe21..b75835d55 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -302,6 +302,14 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ Return the sync committee for a given ``state`` and ``epoch``. + + ``SyncCommittee`` contains an aggregate pubkey that enables + resource-constrained clients to save some computation when verifying + the sync committee's signature. + + ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_sync_committee_indices`` + returns duplicate indices. Implementations must take care when handling + optimizations relating to aggregation and verification in the presence of duplicates. """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] From 7d236561bd39bb92f3d0c7662c5d4d2762a5fe45 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 12:41:19 -0700 Subject: [PATCH 114/227] adjust subnet count to reflect smaller sync committees --- specs/altair/validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index c69e490b1..c58086b8e 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -74,7 +74,7 @@ This document is currently illustrative for early Altair testnets and some parts | Name | Value | Unit | | - | - | :-: | | `TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE` | `2**2` (= 4) | validators | -| `SYNC_COMMITTEE_SUBNET_COUNT` | `8` | The number of sync committee subnets used in the gossipsub aggregation protocol. | +| `SYNC_COMMITTEE_SUBNET_COUNT` | `4` | The number of sync committee subnets used in the gossipsub aggregation protocol. | ## Containers From eae64fd18c94cfa74fad5edf09bc72ba09fe8cf6 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 12:43:21 -0700 Subject: [PATCH 115/227] clean up whitespace on altair files --- specs/altair/p2p-interface.md | 28 ++++++++++---------- specs/altair/validator.md | 50 +++++++++++++++++------------------ 2 files changed, 39 insertions(+), 39 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 4f2fd6aa8..85e859191 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -40,8 +40,8 @@ Altair adds new messages, topics and data to the Req-Resp, Gossip and Discovery ## Warning -This document is currently illustrative for early Altair testnets and some parts are subject to change. -Refer to the note in the [validator guide](./validator.md) for further details. +This document is currently illustrative for early Altair testnets and some parts are subject to change. +Refer to the note in the [validator guide](./validator.md) for further details. # Modifications in Altair @@ -64,12 +64,12 @@ Where ## The gossip domain: gossipsub -Gossip meshes are added in Altair to support the consensus activities of the sync committees. +Gossip meshes are added in Altair to support the consensus activities of the sync committees. Validators use an aggregation scheme to balance the processing and networking load across all of the relevant actors. ### Topics and messages -Topics follow the same specification as in the Phase 0 document. +Topics follow the same specification as in the Phase 0 document. New topics are added in Altair to support the sync committees and the beacon block topic is updated with the modified type. The specification around the creation, validation, and dissemination of messages has not changed from the Phase 0 document. @@ -93,13 +93,13 @@ Altair changes the type of the global beacon block topic and adds one global top ##### `beacon_block` The existing specification for this topic does not change from the Phase 0 document, -but the type of the payload does change to the (modified) `SignedBeaconBlock`. +but the type of the payload does change to the (modified) `SignedBeaconBlock`. This type changes due to the inclusion of the inner `BeaconBlockBody` that is modified in Altair. See the [state transition document](./beacon-chain.md#beaconblockbody) for Altair for further details. ##### `sync_committee_contribution_and_proof` - + This topic is used to propagate partially aggregated sync committee signatures to be included in future blocks. The following validations MUST pass before forwarding the `signed_contribution_and_proof` on the network; define `contribution_and_proof = signed_contribution_and_proof.message`, `contribution = contribution_and_proof.contribution`, and the following function `get_sync_subcommittee_pubkeys` for convenience: @@ -135,16 +135,16 @@ The following validations MUST pass before forwarding the `sync_committee_signat - _[IGNORE]_ The signature's slot is for the current slot, i.e. `sync_committee_signature.slot == current_slot`. - _[IGNORE]_ The block being signed over (`sync_committee_signature.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_signature.validator_index`. -- _[REJECT]_ The `subnet_id` is valid for the given validator, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. +- _[REJECT]_ The `subnet_id` is valid for the given validator, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. Note this validation implies the validator is part of the broader current sync committee along with the correct subcommittee. - _[REJECT]_ The `signature` is valid for the message `beacon_block_root` for the validator referenced by `validator_index`. #### Sync committees and aggregation -The aggregation scheme closely follows the design of the attestation aggregation scheme. -Sync committee signatures are broadcast into "subnets" defined by a topic. -The number of subnets is defined by `SYNC_COMMITTEE_SUBNET_COUNT` in the [Altair validator guide](./validator.md#constants). -Sync committee members are divided into "subcommittees" which are then assigned to a subnet for the duration of tenure in the sync committee. +The aggregation scheme closely follows the design of the attestation aggregation scheme. +Sync committee signatures are broadcast into "subnets" defined by a topic. +The number of subnets is defined by `SYNC_COMMITTEE_SUBNET_COUNT` in the [Altair validator guide](./validator.md#constants). +Sync committee members are divided into "subcommittees" which are then assigned to a subnet for the duration of tenure in the sync committee. Individual validators can be duplicated in the broader sync committee such that they are included multiple times in a given subcommittee or across multiple subcommittees. Unaggregated signatures (along with metadata) are sent as `SyncCommitteeSignature`s on the `sync_committee_{subnet_id}` topics. @@ -244,7 +244,7 @@ Response Content: ) ``` -Requests the MetaData of a peer, using the new `MetaData` definition given above +Requests the MetaData of a peer, using the new `MetaData` definition given above that is extended from phase 0 in Altair. Other conditions for the `GetMetaData` protocol are unchanged from the phase 0 p2p networking document. @@ -253,7 +253,7 @@ protocol are unchanged from the phase 0 p2p networking document. In advance of the fork, implementations can opt in to both run the v1 and v2 for a smooth transition. This is non-breaking, and is recommended as soon as the fork specification is stable. -The v1 variants will be deprecated, and implementations should use v2 when available +The v1 variants will be deprecated, and implementations should use v2 when available (as negotiated with peers via LibP2P multistream-select). The v1 method MAY be unregistered at the fork boundary. @@ -265,7 +265,7 @@ the responder MUST return the **InvalidRequest** response code. The `attnets` key of the ENR is used as defined in the Phase 0 document. An additional bitfield is added to the ENR under the key `syncnets` to facilitate sync committee subnet discovery. -The length of this bitfield is `SYNC_COMMITTEE_SUBNET_COUNT` where each bit corresponds to a distinct `subnet_id` for a specific sync committee subnet. +The length of this bitfield is `SYNC_COMMITTEE_SUBNET_COUNT` where each bit corresponds to a distinct `subnet_id` for a specific sync committee subnet. The `i`th bit is set in this bitfield if the validator is currently subscribed to the `sync_committee_{i}` topic. See the [validator document](./validator.md#sync-committee-subnet-stability) for further details on how the new bits are used. diff --git a/specs/altair/validator.md b/specs/altair/validator.md index c69e490b1..8cd9fc404 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -49,18 +49,18 @@ This is an accompanying document to [Ethereum 2.0 Altair -- The Beacon Chain](./ ## Introduction -This document represents the expected behavior of an "honest validator" with respect to the Altair upgrade of the Ethereum 2.0 protocol. -It builds on the [previous document for the behavior of an "honest validator" from Phase 0](../phase0/validator.md) of the Ethereum 2.0 protocol. +This document represents the expected behavior of an "honest validator" with respect to the Altair upgrade of the Ethereum 2.0 protocol. +It builds on the [previous document for the behavior of an "honest validator" from Phase 0](../phase0/validator.md) of the Ethereum 2.0 protocol. This previous document is referred to below as the "Phase 0 document". -Altair introduces a new type of committee: the sync committee. Sync committees are responsible for signing each block of the canonical chain and there exists an efficient algorithm for light clients to sync the chain using the output of the sync committees. -See the [sync protocol](./sync-protocol.md) for further details on the light client sync. -Under this network upgrade, validators track their participation in this new committee type and produce the relevant signatures as required. +Altair introduces a new type of committee: the sync committee. Sync committees are responsible for signing each block of the canonical chain and there exists an efficient algorithm for light clients to sync the chain using the output of the sync committees. +See the [sync protocol](./sync-protocol.md) for further details on the light client sync. +Under this network upgrade, validators track their participation in this new committee type and produce the relevant signatures as required. Block proposers incorporate the (aggregated) sync committee signatures into each block they produce. ## Prerequisites -All terminology, constants, functions, and protocol mechanics defined in the [Altair -- The Beacon Chain](./beacon-chain.md) doc are requisite for this document and used throughout. +All terminology, constants, functions, and protocol mechanics defined in the [Altair -- The Beacon Chain](./beacon-chain.md) doc are requisite for this document and used throughout. Please see this document before continuing and use as a reference throughout. ## Warning @@ -168,11 +168,11 @@ def is_assigned_to_sync_committee(state: BeaconState, ### Lookahead The sync committee shufflings give validators 1 sync committee period of lookahead which amounts to `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs. -At any given `epoch`, the `BeaconState` contains the current `SyncCommittee` and the next `SyncCommittee`. +At any given `epoch`, the `BeaconState` contains the current `SyncCommittee` and the next `SyncCommittee`. Once every `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs, the next `SyncCommittee` becomes the current `SyncCommittee` and the next committee is computed and stored. -*Note*: The data required to compute a given committee is not cached in the `BeaconState` after committees are calculated at the period boundaries. -This means that calling `get_sync_commitee()` in a given `epoch` can return a different result than what was computed during the relevant epoch transition. +*Note*: The data required to compute a given committee is not cached in the `BeaconState` after committees are calculated at the period boundaries. +This means that calling `get_sync_commitee()` in a given `epoch` can return a different result than what was computed during the relevant epoch transition. For this reason, *always* get committee assignments via the fields of the `BeaconState` (`current_sync_committee` and `next_sync_committee`) or use the above reference code. A validator should plan for future sync committee assignments by noting which sync committee periods they are selected for participation. @@ -188,7 +188,7 @@ A validator maintains the responsibilities given in the Phase 0 document. Block proposals are modified to incorporate the sync committee signatures as detailed below. When assigned to a sync committee, validators have a new responsibility to sign and broadcast beacon block roots during each slot of the sync committee period. -These signatures are aggregated and routed to the proposer over gossip for inclusion into a beacon block. +These signatures are aggregated and routed to the proposer over gossip for inclusion into a beacon block. Assignments to a particular sync committee are infrequent at normal validator counts; however, an action every slot is required when in the current active sync committee. ### Block proposal @@ -202,25 +202,25 @@ No change to [Preparing for a `BeaconBlock`](../phase0/validator.md#preparing-fo #### Constructing the `BeaconBlockBody` -Each section of [Constructing the `BeaconBlockBody`](../phase0/validator.md#constructing-the-beaconblockbody) should be followed. +Each section of [Constructing the `BeaconBlockBody`](../phase0/validator.md#constructing-the-beaconblockbody) should be followed. After constructing the `BeaconBlockBody` as per that section, the proposer has an additional task to include the sync committee signatures: ##### Sync committee The proposer receives a number of `SyncCommitteeContribution`s (wrapped in `SignedContributionAndProof`s on the wire) from validators in the sync committee who are selected to partially aggregate signatures from independent subcommittees formed by breaking the full sync committee into `SYNC_COMMITTEE_SUBNET_COUNT` pieces (see below for details). -The proposer collects the contributions that match their local view of the chain (i.e. `contribution.beacon_block_root == block.parent_root`) for further aggregation when preparing a block. +The proposer collects the contributions that match their local view of the chain (i.e. `contribution.beacon_block_root == block.parent_root`) for further aggregation when preparing a block. Of these contributions, proposers should select the best contribution seen across all aggregators for each subnet/subcommittee. A contribution with more valid signatures is better than a contribution with fewer signatures. -Recall `block.body.sync_aggregate.sync_committee_bits` is a `Bitvector` where the `i`th bit is `True` if the corresponding validator in the sync committee has produced a valid signature, +Recall `block.body.sync_aggregate.sync_committee_bits` is a `Bitvector` where the `i`th bit is `True` if the corresponding validator in the sync committee has produced a valid signature, and that `block.body.sync_aggregate.sync_committee_signature` is the aggregate BLS signature combining all of the valid signatures. -Given a collection of the best seen `contributions` (with no repeating `subcommittee_index` values) and the `BeaconBlock` under construction, +Given a collection of the best seen `contributions` (with no repeating `subcommittee_index` values) and the `BeaconBlock` under construction, the proposer processes them as follows: ```python -def process_sync_committee_contributions(block: BeaconBlock, +def process_sync_committee_contributions(block: BeaconBlock, contributions: Set[SyncCommitteeContribution]) -> None: sync_aggregate = SyncAggregate() signatures = [] @@ -248,7 +248,7 @@ No change to [Packaging into a `SignedBeaconBlock`](../phase0/validator.md#packa ### Attesting and attestation aggregation -Refer to the phase 0 document for the [attesting](../phase0/validator.md#attesting) and [attestation aggregation](../phase0/validator.md#attestation-aggregation) responsibilities. +Refer to the phase 0 document for the [attesting](../phase0/validator.md#attesting) and [attestation aggregation](../phase0/validator.md#attestation-aggregation) responsibilities. There is no change compared to the phase 0 document. ### Sync committees @@ -263,15 +263,15 @@ This process occurs each slot. If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every slot in the current sync committee period, the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. -This logic is triggered upon the same conditions as when producing an attestation. +This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. `get_sync_committee_signature()` assumes `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice (including any empty slots up to the current slot processed with `process_slots` on top of the latest block), `block_root` is the root of the head block, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. ```python -def get_sync_committee_signature(state: BeaconState, +def get_sync_committee_signature(state: BeaconState, block_root: Root, - validator_index: ValidatorIndex, + validator_index: ValidatorIndex, privkey: int) -> SyncCommitteeSignature: epoch = get_current_epoch(state) domain = get_domain(state, DOMAIN_SYNC_COMMITTEE, epoch) @@ -286,7 +286,7 @@ def get_sync_committee_signature(state: BeaconState, The validator broadcasts the assembled signature to the assigned subnet, the `sync_committee_{subnet_id}` pubsub topic. The `subnet_id` is derived from the position in the sync committee such that the sync committee is divided into "subcommittees". -`subnet_id` can be computed via `compute_subnets_for_sync_committee()` where `state` is a `BeaconState` during the matching sync committee period. +`subnet_id` can be computed via `compute_subnets_for_sync_committee()` where `state` is a `BeaconState` during the matching sync committee period. *Note*: This function returns multiple subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. @@ -340,7 +340,7 @@ def is_sync_committee_aggregator(signature: BLSSignature) -> bool: ##### Construct sync committee contribution -If a validator is selected to aggregate the `SyncCommitteeSignature`s produced on a subnet during a given `slot`, they construct an aggregated `SyncCommitteeContribution`. +If a validator is selected to aggregate the `SyncCommitteeSignature`s produced on a subnet during a given `slot`, they construct an aggregated `SyncCommitteeContribution`. Given all of the (valid) collected `sync_committee_signatures: Set[SyncCommitteeSignature]` from the `sync_committee_{subnet_id}` gossip during the selected `slot` with an equivalent `beacon_block_root` to that of the aggregator, the aggregator creates a `contribution: SyncCommitteeContribution` with the following fields: @@ -361,13 +361,13 @@ Set `contribution.subcommittee_index` to the index for the subcommittee index co Let `contribution.aggregation_bits` be a `Bitvector[SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT]`, where the `index`th bit is set in the `Bitvector` for each corresponding validator included in this aggregate from the corresponding subcommittee. An aggregator finds the index in the sync committee (as returned by `get_sync_committee_indices()`) for a given validator referenced by `sync_committee_signature.validator_index` and maps the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is set in `contribution.aggegration_bits`. -For example, if a validator with index `2044` is pseudo-randomly sampled to sync committee index `135`. This sync committee index maps to `subcommittee_index` `1` with position `7` in the `Bitvector` for the contribution. +For example, if a validator with index `2044` is pseudo-randomly sampled to sync committee index `135`. This sync committee index maps to `subcommittee_index` `1` with position `7` in the `Bitvector` for the contribution. *Note*: A validator **could be included multiple times** in a given subcommittee such that multiple bits are set for a single `SyncCommitteeSignature`. ###### Signature -Set `contribution.signature = aggregate_signature` where `aggregate_signature` is obtained by assembling the appropriate collection of `BLSSignature`s from the set of `sync_committee_signatures` and using the `bls.Aggregate()` function to produce an aggregate `BLSSignature`. +Set `contribution.signature = aggregate_signature` where `aggregate_signature` is obtained by assembling the appropriate collection of `BLSSignature`s from the set of `sync_committee_signatures` and using the `bls.Aggregate()` function to produce an aggregate `BLSSignature`. The collection of input signatures should include one signature per validator who had a bit set in the `aggregation_bits` bitfield, with repeated signatures if one validator maps to multiple indices within the subcommittee. @@ -402,8 +402,8 @@ def get_contribution_and_proof(state: BeaconState, Then `signed_contribution_and_proof = SignedContributionAndProof(message=contribution_and_proof, signature=signature)` is constructed and broadcast. Where `signature` is obtained from: ```python -def get_contribution_and_proof_signature(state: BeaconState, - contribution_and_proof: ContributionAndProof, +def get_contribution_and_proof_signature(state: BeaconState, + contribution_and_proof: ContributionAndProof, privkey: int) -> BLSSignature: contribution = contribution_and_proof.contribution domain = get_domain(state, DOMAIN_CONTRIBUTION_AND_PROOF, compute_epoch_at_slot(contribution.slot)) From 7a168be862294d580003126d881cc7f4ba4e2584 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 13:25:34 -0700 Subject: [PATCH 116/227] allow fault tolerance equal to threshold, not just above --- specs/altair/sync-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index 925c882ef..fba263bb1 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -185,7 +185,7 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda store.valid_updates.append(update) if ( - sum(update.sync_committee_bits) * 3 > len(update.sync_committee_bits) * 2 + sum(update.sync_committee_bits) * 3 >= len(update.sync_committee_bits) * 2 and update.finality_header != BeaconBlockHeader() ): # Apply update if (1) 2/3 quorum is reached and (2) we have a finality proof. From 3b803241193083696c2922f5983712a269d8d62b Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 13:42:02 -0700 Subject: [PATCH 117/227] Compute `LIGHT_CLIENT_UPDATE_TIMEOUT` in lieu of maintaining a constant --- configs/mainnet/altair.yaml | 2 -- configs/minimal/altair.yaml | 2 -- specs/altair/sync-protocol.md | 10 ++-------- 3 files changed, 2 insertions(+), 12 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..6db260cca 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -47,8 +47,6 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 # 2**13 MAX_VALID_LIGHT_CLIENT_UPDATES: 8192 -# 2**13 (=8192) -LIGHT_CLIENT_UPDATE_TIMEOUT: 8192 # Validator diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index f9b30eea2..5c5781010 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -47,8 +47,6 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 # [customized] MAX_VALID_LIGHT_CLIENT_UPDATES: 32 -# [customized] -LIGHT_CLIENT_UPDATE_TIMEOUT: 32 # Validator # --------------------------------------------------------------- diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index fba263bb1..ca55c92f7 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -12,7 +12,6 @@ - [Constants](#constants) - [Configuration](#configuration) - [Misc](#misc) - - [Time parameters](#time-parameters) - [Containers](#containers) - [`LightClientSnapshot`](#lightclientsnapshot) - [`LightClientUpdate`](#lightclientupdate) @@ -53,12 +52,6 @@ uses sync committees introduced in [this beacon chain extension](./beacon-chain. | `MIN_SYNC_COMMITTEE_PARTICIPANTS` | `1` | | `MAX_VALID_LIGHT_CLIENT_UPDATES` | `uint64(2**64 - 1)` | -### Time parameters - -| Name | Value | Unit | Duration | -| - | - | :-: | :-: | -| `LIGHT_CLIENT_UPDATE_TIMEOUT` | `Slot(2**13)` | slots | ~27 hours | - ## Containers ### `LightClientSnapshot` @@ -184,6 +177,7 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda validate_light_client_update(store.snapshot, update, genesis_validators_root) store.valid_updates.append(update) + update_timeout = SLOTS_PER_EPOCH * EPOCHS_PER_SYNC_COMMITTEE_PERIOD if ( sum(update.sync_committee_bits) * 3 >= len(update.sync_committee_bits) * 2 and update.finality_header != BeaconBlockHeader() @@ -193,7 +187,7 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda # It may be changed to re-organizable light client design. See the on-going issue eth2.0-specs#2182. apply_light_client_update(store.snapshot, update) store.valid_updates = [] - elif current_slot > store.snapshot.header.slot + LIGHT_CLIENT_UPDATE_TIMEOUT: + elif current_slot > store.snapshot.header.slot + update_timeout: # Forced best update when the update timeout has elapsed apply_light_client_update(store.snapshot, max(store.valid_updates, key=lambda update: sum(update.sync_committee_bits))) From 165c960cf15b63c30bed2dd2ce9b1a36c7734093 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 13:46:30 -0700 Subject: [PATCH 118/227] Update config value to reflect spec --- configs/mainnet/altair.yaml | 4 ++-- configs/minimal/altair.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 6db260cca..b7b7603a2 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -45,8 +45,8 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# 2**13 -MAX_VALID_LIGHT_CLIENT_UPDATES: 8192 +# 2**64 - 1 +MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 # Validator diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index 5c5781010..5618af8ff 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -45,8 +45,8 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# [customized] -MAX_VALID_LIGHT_CLIENT_UPDATES: 32 +# 2**64 - 1 +MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 # Validator # --------------------------------------------------------------- From f37f9a367d4f55957ab70c59e6502993cf43ad3a Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:15:15 +0200 Subject: [PATCH 119/227] include merge in generators --- tests/generators/epoch_processing/main.py | 10 ++++++++-- tests/generators/finality/main.py | 9 ++++++--- tests/generators/fork_choice/main.py | 8 ++++++-- tests/generators/operations/main.py | 13 +++++++++++-- tests/generators/rewards/main.py | 11 +++++++++-- tests/generators/sanity/main.py | 11 +++++++++-- tests/generators/ssz_static/main.py | 2 +- 7 files changed, 50 insertions(+), 14 deletions(-) diff --git a/tests/generators/epoch_processing/main.py b/tests/generators/epoch_processing/main.py index 9efd96534..2aa6381ff 100644 --- a/tests/generators/epoch_processing/main.py +++ b/tests/generators/epoch_processing/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -27,6 +28,10 @@ if __name__ == "__main__": **phase_0_mods, } # also run the previous phase 0 tests + # No epoch-processing changes in Merge and previous testing repeats with new types, so no additional tests required. + # TODO: rebase onto Altair testing later. + merge_mods = phase_0_mods + # TODO Custody Game testgen is disabled for now # custody_game_mods = {**{key: 'eth2spec.test.custody_game.epoch_processing.test_process_' + key for key in [ # 'reveal_deadlines', @@ -37,6 +42,7 @@ if __name__ == "__main__": all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="epoch_processing", specs=specs, all_mods=all_mods) diff --git a/tests/generators/finality/main.py b/tests/generators/finality/main.py index e50425da4..148ddef96 100644 --- a/tests/generators/finality/main.py +++ b/tests/generators/finality/main.py @@ -1,19 +1,22 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": phase_0_mods = {'finality': 'eth2spec.test.phase0.finality.test_finality'} - altair_mods = phase_0_mods # No additional altair specific finality tests + altair_mods = phase_0_mods # No additional Altair specific finality tests + merge_mods = phase_0_mods # No additional Merge specific finality tests all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: spec_merge, } run_state_test_generators(runner_name="finality", specs=specs, all_mods=all_mods) diff --git a/tests/generators/fork_choice/main.py b/tests/generators/fork_choice/main.py index 445ee629c..ae15caa1d 100644 --- a/tests/generators/fork_choice/main.py +++ b/tests/generators/fork_choice/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -13,10 +14,13 @@ if __name__ == "__main__": ]} # No additional Altair specific finality tests, yet. altair_mods = phase_0_mods + # No specific Merge tests yet. TODO: rebase onto Altair testing later. + merge_mods = phase_0_mods all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="fork_choice", specs=specs, all_mods=all_mods) diff --git a/tests/generators/operations/main.py b/tests/generators/operations/main.py index 0c1b84c24..316ccdf10 100644 --- a/tests/generators/operations/main.py +++ b/tests/generators/operations/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -23,6 +24,13 @@ if __name__ == "__main__": **phase_0_mods, } # also run the previous phase 0 tests + merge_mods = { + **{key: 'eth2spec.test.merge.block_processing.test_process_' + key for key in [ + 'execution_payload', + ]}, + **phase_0_mods, # TODO: runs phase0 tests. Rebase to include `altair_mods` testing later. + } + # TODO Custody Game testgen is disabled for now # custody_game_mods = {**{key: 'eth2spec.test.custody_game.block_processing.test_process_' + key for key in [ # 'attestation', @@ -35,6 +43,7 @@ if __name__ == "__main__": all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="operations", specs=specs, all_mods=all_mods) diff --git a/tests/generators/rewards/main.py b/tests/generators/rewards/main.py index 2a0f14863..8e50732e1 100644 --- a/tests/generators/rewards/main.py +++ b/tests/generators/rewards/main.py @@ -1,10 +1,11 @@ from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -16,9 +17,15 @@ if __name__ == "__main__": # No additional altair specific rewards tests, yet. altair_mods = phase_0_mods + # No additional merge specific rewards tests, yet. + # Note: Block rewards are non-epoch rewards and are tested as part of block processing tests. + # Transaction fees are part of the execution-layer. + merge_mods = phase_0_mods + all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="rewards", specs=specs, all_mods=all_mods) diff --git a/tests/generators/sanity/main.py b/tests/generators/sanity/main.py index 7f7fecc67..3bed6672d 100644 --- a/tests/generators/sanity/main.py +++ b/tests/generators/sanity/main.py @@ -1,11 +1,12 @@ from eth2spec.phase0 import spec as spec_phase0 from eth2spec.altair import spec as spec_altair -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.merge import spec as spec_merge +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators -specs = (spec_phase0, spec_altair) +specs = (spec_phase0, spec_altair, spec_merge) if __name__ == "__main__": @@ -17,9 +18,15 @@ if __name__ == "__main__": 'blocks', ]}, **phase_0_mods} # also run the previous phase 0 tests + # Altair-specific test cases are ignored, but should be included after the Merge is rebased onto Altair work. + merge_mods = {**{key: 'eth2spec.test.merge.sanity.test_' + key for key in [ + 'blocks', + ]}, **phase_0_mods} # TODO: Merge inherits phase0 tests for now. + all_mods = { PHASE0: phase_0_mods, ALTAIR: altair_mods, + MERGE: merge_mods, } run_state_test_generators(runner_name="sanity", specs=specs, all_mods=all_mods) diff --git a/tests/generators/ssz_static/main.py b/tests/generators/ssz_static/main.py index 94ce02335..1e810f71e 100644 --- a/tests/generators/ssz_static/main.py +++ b/tests/generators/ssz_static/main.py @@ -94,7 +94,7 @@ if __name__ == "__main__": settings.append((seed, MAINNET, random_value.RandomizationMode.mode_random, False, 5)) seed += 1 # TODO: enable testing for the whole merge spec. - for fork in TESTGEN_FORKS + (MERGE,): + for fork in TESTGEN_FORKS: gen_runner.run_generator("ssz_static", [ create_provider(fork, config_name, seed, mode, chaos, cases_if_random) for (seed, config_name, mode, chaos, cases_if_random) in settings From ab693c972445a7ba362d2f527447461f377ce82a Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:16:22 +0200 Subject: [PATCH 120/227] update spec test constants for merge --- tests/core/pyspec/eth2spec/test/helpers/constants.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index ccd7b20a2..d8f2a37ba 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -7,21 +7,24 @@ from .typing import SpecForkName, ConfigName # Some of the Spec module functionality is exposed here to deal with phase-specific changes. PHASE0 = SpecForkName('phase0') ALTAIR = SpecForkName('altair') +MERGE = SpecForkName('merge') # Experimental phases (not included in default "ALL_PHASES"): -MERGE = SpecForkName('merge') SHARDING = SpecForkName('sharding') CUSTODY_GAME = SpecForkName('custody_game') DAS = SpecForkName('das') # The forks that pytest runs with. -ALL_PHASES = (PHASE0, ALTAIR) +ALL_PHASES = (PHASE0, ALTAIR, MERGE) # The forks that output to the test vectors. -TESTGEN_FORKS = (PHASE0, ALTAIR) +TESTGEN_FORKS = (PHASE0, ALTAIR, MERGE) # TODO: everything runs in parallel to Altair. # After features are rebased on the Altair fork, this can be reduced to just PHASE0. FORKS_BEFORE_ALTAIR = (PHASE0, MERGE, SHARDING, CUSTODY_GAME, DAS) +# TODO: when rebasing Merge onto Altair, add ALTAIR to this tuple. +FORKS_BEFORE_MERGE = (PHASE0,) + # # Config # From 521cffc3e9c4de950489470eb1eefbf2871a5c33 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:17:10 +0200 Subject: [PATCH 121/227] update execution-payload processing to isolate payload from block body --- specs/merge/beacon-chain.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index b9c443962..d665f1151 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -146,8 +146,8 @@ def is_transition_completed(state: BeaconState) -> bool: #### `is_transition_block` ```python -def is_transition_block(state: BeaconState, block_body: BeaconBlockBody) -> bool: - return not is_transition_completed(state) and block_body.execution_payload != ExecutionPayload() +def is_transition_block(state: BeaconState, block: BeaconBlock) -> bool: + return not is_transition_completed(state) and block.body.execution_payload != ExecutionPayload() ``` #### `compute_time_at_slot` @@ -168,7 +168,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) - process_execution_payload(state, block.body) # [New in Merge] + # Pre-merge, skip execution payload processing + if is_transition_completed(state) or is_transition_block(state, block): + process_execution_payload(state, block.body.execution_payload) # [New in Merge] ``` #### Execution payload processing @@ -181,16 +183,10 @@ The body of the function is implementation dependent. ##### `process_execution_payload` ```python -def process_execution_payload(state: BeaconState, body: BeaconBlockBody) -> None: +def process_execution_payload(state: BeaconState, execution_payload: ExecutionPayload) -> None: """ Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ - # Pre-merge, skip processing - if not is_transition_completed(state) and not is_transition_block(state, body): - return - - execution_payload = body.execution_payload - if is_transition_completed(state): assert execution_payload.parent_hash == state.latest_execution_payload_header.block_hash assert execution_payload.number == state.latest_execution_payload_header.number + 1 From 25d0d673a9b0950c0aa2d0b2406f849c9dca0470 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:18:01 +0200 Subject: [PATCH 122/227] start testing of merge functionality --- tests/core/pyspec/eth2spec/test/context.py | 8 +- .../pyspec/eth2spec/test/helpers/block.py | 6 +- .../test/helpers/execution_payload.py | 26 ++++ .../pyspec/eth2spec/test/merge/__init__.py | 0 .../test/merge/block_processing/__init__.py | 0 .../test_process_execution_payload.py | 147 ++++++++++++++++++ .../eth2spec/test/merge/sanity/__init__.py | 0 .../eth2spec/test/merge/sanity/test_blocks.py | 25 +++ 8 files changed, 210 insertions(+), 2 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/helpers/execution_payload.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/block_processing/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/sanity/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 438e611cf..8699c65ad 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -8,7 +8,7 @@ from eth2spec.utils import bls from .exceptions import SkippedTest from .helpers.constants import ( PHASE0, ALTAIR, - ALL_PHASES, FORKS_BEFORE_ALTAIR, + ALL_PHASES, FORKS_BEFORE_ALTAIR, FORKS_BEFORE_MERGE, ) from .helpers.genesis import create_genesis_state from .utils import vector_test, with_meta_tags @@ -365,3 +365,9 @@ def is_post_altair(spec): if spec.fork in FORKS_BEFORE_ALTAIR: return False return True + + +def is_post_merge(spec): + if spec.fork in FORKS_BEFORE_MERGE: + return False + return True diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 6949b54bc..64f406633 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -1,4 +1,5 @@ -from eth2spec.test.context import is_post_altair +from eth2spec.test.context import is_post_altair, is_post_merge +from eth2spec.test.helpers.execution_payload import build_empty_execution_payload from eth2spec.test.helpers.keys import privkeys from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls @@ -94,6 +95,9 @@ def build_empty_block(spec, state, slot=None): if is_post_altair(spec): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY + if is_post_merge(spec): + empty_block.body.execution_payload = build_empty_execution_payload(spec, state) + apply_randao_reveal(spec, state, empty_block) return empty_block diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py new file mode 100644 index 000000000..e829e07c7 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -0,0 +1,26 @@ + +def build_empty_execution_payload(spec, state): + """ + Assuming a pre-state of the same slot, build a valid ExecutionPayload without any transactions. + """ + latest = state.latest_execution_payload_header + timestamp = spec.compute_time_at_slot(state, state.slot) + empty_txs = spec.List[spec.OpaqueTransaction, spec.MAX_EXECUTION_TRANSACTIONS]() + + payload = spec.ExecutionPayload( + block_hash=spec.Hash32(), + parent_hash=latest.block_hash, + coinbase=spec.Bytes20(), + state_root=latest.state_root, # no changes to the state + number=latest.number + 1, + gas_limit=latest.gas_limit, # retain same limit + gas_used=0, # empty block, 0 gas + timestamp=timestamp, + receipt_root=b"no receipts here" + b"\x00"*16, # TODO: root of empty MPT may be better. + logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? + transactions_root=empty_txs, + ) + # TODO: real RLP + block hash logic would be nice, requires RLP and keccak256 dependency however. + payload.block_hash = spec.Hash32(spec.hash(payload.hash_tree_root() + b"FAKE RLP HASH")) + + return payload diff --git a/tests/core/pyspec/eth2spec/test/merge/__init__.py b/tests/core/pyspec/eth2spec/test/merge/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/__init__.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py new file mode 100644 index 000000000..0cf5cef5f --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -0,0 +1,147 @@ +from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases_except + +with_merge_and_later = with_all_phases_except([PHASE0, ALTAIR]) + + +def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): + """ + Run ``process_execution_payload``, yielding: + - pre-state ('pre') + - execution payload ('execution_payload') + - execution details, to mock EVM execution ('execution.yml', a dict with 'execution_valid' key and boolean value) + - post-state ('post'). + If ``valid == False``, run expecting ``AssertionError`` + """ + + yield 'pre', state + yield 'execution', {'execution_valid': execution_valid} + yield 'execution_payload', execution_payload + + if not valid: + expect_assertion_error(lambda: spec.process_execution_payload(state, execution_payload)) + yield 'post', None + return + + spec.process_execution_payload(state, execution_payload) + + yield 'post', state + + # TODO: any assertions to make? + + +@with_merge_and_later +@spec_state_test +def test_success_first_payload(spec, state): + assert not spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_regular_payload(spec, state): + # TODO: setup state + assert spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_first_payload_with_gap_slot(spec, state): + # TODO: transition gap slot + + assert not spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_regular_payload_with_gap_slot(spec, state): + # TODO: setup state + assert spec.is_transition_completed(state) + # TODO: transition gap slot + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_bad_execution_first_payload(spec, state): + # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) + + # TODO: execution payload. + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_execution_regular_payload(spec, state): + # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_parent_hash_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_number_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_everything_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_timestamp_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_timestamp_regular_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + diff --git a/tests/core/pyspec/eth2spec/test/merge/sanity/__init__.py b/tests/core/pyspec/eth2spec/test/merge/sanity/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py new file mode 100644 index 000000000..1f9e69167 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py @@ -0,0 +1,25 @@ +from eth2spec.test.helpers.state import ( + state_transition_and_sign_block +) +from eth2spec.test.helpers.block import ( + build_empty_block_for_next_slot +) +from eth2spec.test.context import ( + with_all_phases, spec_state_test +) + + +@with_all_phases +@spec_state_test +def test_empty_block_transition(spec, state): + yield 'pre', state + + block = build_empty_block_for_next_slot(spec, state) + assert len(block.body.execution_payload.transactions) == 0 + + signed_block = state_transition_and_sign_block(spec, state, block) + + yield 'blocks', [signed_block] + yield 'post', state + +# TODO: tests with EVM, mock or replacement? From 36032fd115c1be0eef52ac4a68c3cc8672b5ed70 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 00:37:00 +0200 Subject: [PATCH 123/227] update doc about format --- tests/formats/operations/README.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/tests/formats/operations/README.md b/tests/formats/operations/README.md index 033c5fdef..f562a6f2a 100644 --- a/tests/formats/operations/README.md +++ b/tests/formats/operations/README.md @@ -33,17 +33,23 @@ This excludes the other parts of the block-transition. Operations: -| *`operation-name`* | *`operation-object`* | *`input name`* | *`processing call`* | -|-------------------------|-----------------------|----------------------|-----------------------------------------------------------------| -| `attestation` | `Attestation` | `attestation` | `process_attestation(state, attestation)` | -| `attester_slashing` | `AttesterSlashing` | `attester_slashing` | `process_attester_slashing(state, attester_slashing)` | -| `block_header` | `BeaconBlock` | **`block`** | `process_block_header(state, block)` | -| `deposit` | `Deposit` | `deposit` | `process_deposit(state, deposit)` | -| `proposer_slashing` | `ProposerSlashing` | `proposer_slashing` | `process_proposer_slashing(state, proposer_slashing)` | -| `voluntary_exit` | `SignedVoluntaryExit` | `voluntary_exit` | `process_voluntary_exit(state, voluntary_exit)` | -| `sync_aggregate` | `SyncAggregate` | `sync_aggregate` | `process_sync_committee(state, sync_aggregate)` (new in Altair) | +| *`operation-name`* | *`operation-object`* | *`input name`* | *`processing call`* | +|-------------------------|-----------------------|----------------------|----------------------------------------------------------------------| +| `attestation` | `Attestation` | `attestation` | `process_attestation(state, attestation)` | +| `attester_slashing` | `AttesterSlashing` | `attester_slashing` | `process_attester_slashing(state, attester_slashing)` | +| `block_header` | `BeaconBlock` | **`block`** | `process_block_header(state, block)` | +| `deposit` | `Deposit` | `deposit` | `process_deposit(state, deposit)` | +| `proposer_slashing` | `ProposerSlashing` | `proposer_slashing` | `process_proposer_slashing(state, proposer_slashing)` | +| `voluntary_exit` | `SignedVoluntaryExit` | `voluntary_exit` | `process_voluntary_exit(state, voluntary_exit)` | +| `sync_aggregate` | `SyncAggregate` | `sync_aggregate` | `process_sync_committee(state, sync_aggregate)` (new in Altair) | +| `execution_payload` | `ExecutionPayload` | `execution_payload` | `process_execution_payload(state, execution_payload)` (new in Merge) | Note that `block_header` is not strictly an operation (and is a full `Block`), but processed in the same manner, and hence included here. +The `execution_payload` processing normally requires a `verify_execution_state_transition(execution_payload)`, +a responsibility of an (external) execution engine. +During testing this execution is mocked, an `execution.yml` is provided instead: +a dict containing an `execution_valid` boolean field with the verification result. + The resulting state should match the expected `post` state, or if the `post` state is left blank, the handler should reject the input operation as invalid. From a562f2aeb45026902f6d9ae3e92e338ddcd87d6c Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 4 May 2021 17:05:15 -0700 Subject: [PATCH 124/227] "toward" -> "closer to" for penalty adjustments Makes it clearer that even the Altair values are not final --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 726747c59..7bf0c2d99 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -104,7 +104,7 @@ Altair is the first beacon chain hard fork. Its main features are: ### Updated penalty values -This patch updates a few configuration values to move penalty parameters toward their final, maximum security values. +This patch updates a few configuration values to move penalty parameters closer to their final, maximum security values. *Note*: The spec does *not* override previous configuration values but instead creates new values and replaces usage throughout. From 9dcdbafba318fe42333c66f51e703cec2adf1434 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 4 May 2021 17:19:15 -0700 Subject: [PATCH 125/227] Reorganization of config params to put sync committee in one section This may be marginally "cleaner" than the previous approach, keeping constants with the same topic together. --- specs/altair/beacon-chain.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 726747c59..49169eae8 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -114,20 +114,19 @@ This patch updates a few configuration values to move penalty parameters toward | `MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR` | `uint64(2**6)` (= 64) | | `PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR` | `uint64(2)` | +### Sync committee + +| Name | Value | Unit | Duration | +| - | - | - | - | +| `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | | | +| `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | + ### Misc | Name | Value | -| - | - | -| `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | -| `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | -### Time parameters - -| Name | Value | Unit | Duration | -| - | - | :-: | :-: | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | - ### Domain types | Name | Value | From b310482bce960f77271c1929c47235b63b4f3842 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 4 May 2021 17:19:51 -0700 Subject: [PATCH 126/227] Update specs/altair/beacon-chain.md --- specs/altair/beacon-chain.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 49169eae8..60bbd69d8 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -125,6 +125,7 @@ This patch updates a few configuration values to move penalty parameters toward ### Misc | Name | Value | +| - | - | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | ### Domain types From 79fc41146d898d80358ff6c746217d827587ee2c Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 5 May 2021 13:37:07 +0600 Subject: [PATCH 127/227] Adjust is_transition_block call in fork-choice --- specs/merge/fork-choice.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index 34647f45d..f478dd7e6 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -75,7 +75,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: assert get_ancestor(store, block.parent_root, finalized_slot) == store.finalized_checkpoint.root # [New in Merge] - if is_transition_block(pre_state, block.body): + if is_transition_block(pre_state, block): # Delay consideration of block until PoW block is processed by the PoW node pow_block = get_pow_block(block.body.execution_payload.parent_hash) assert pow_block.is_processed From ddc59422ba31e8ec164239280a69868ae65ce7ce Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 5 May 2021 13:21:15 +0300 Subject: [PATCH 128/227] Revert "Calc the right root with respect to (slot, shard) for an empty PendingShardHeader" This reverts commit 2199b8e0 --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 68f6ccaa2..decffdaa7 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -763,7 +763,7 @@ def reset_pending_headers(state: BeaconState) -> None: slot=slot, shard=shard, commitment=DataCommitment(), - root=hash_tree_root(ShardBlobHeader(slot = slot, shard = shard)), + root=Root(), votes=Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length), confirmed=False, )) From befe4c7db3a8d0bdda6719a220a9f010037a07b7 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 5 May 2021 13:21:25 +0300 Subject: [PATCH 129/227] Revert "Search winning 'empty' PendingShardHeader index by the empty DataCommitment instead of zero Root" This reverts commit dca6d337 --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index decffdaa7..6fd83c9cf 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -701,7 +701,7 @@ def process_pending_headers(state: BeaconState) -> None: winning_index = voting_balances.index(max(voting_balances)) else: # If no votes, zero wins - winning_index = [c.commitment for c in candidates].index(DataCommitment()) + winning_index = [c.root for c in candidates].index(Root()) candidates[winning_index].confirmed = True for slot_index in range(SLOTS_PER_EPOCH): for shard in range(SHARD_COUNT): From 3cc1256a723ea719f9a81b4dedacb15c439ae327 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 5 May 2021 13:31:19 +0300 Subject: [PATCH 130/227] update_pending_votes: search pending header by header.(root + slot + shard) --- specs/sharding/beacon-chain.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 6fd83c9cf..b93cd2e2a 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -512,17 +512,22 @@ def update_pending_votes(state: BeaconState, attestation: Attestation) -> None: pending_headers = state.current_epoch_pending_shard_headers else: pending_headers = state.previous_epoch_pending_shard_headers - pending_header = None - for header in pending_headers: - if header.root == attestation.data.shard_header_root: - pending_header = header - assert pending_header is not None - assert pending_header.slot == attestation.data.slot - assert pending_header.shard == compute_shard_from_committee_index( + + attestation_shard = compute_shard_from_committee_index( state, attestation.data.slot, attestation.data.index, ) + pending_header = None + for header in pending_headers: + if ( + header.root == attestation.data.shard_header_root + and header.slot == attestation.data.slot + and header.shard == attestation_shard + ): + pending_header = header + assert pending_header is not None + for i in range(len(pending_header.votes)): pending_header.votes[i] = pending_header.votes[i] or attestation.aggregation_bits[i] From 786e611c71d5c62bf30c988c692ab88d1a54d8f3 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 12:05:32 +0100 Subject: [PATCH 131/227] Flags to BitVector --- specs/altair/beacon-chain.md | 31 +++++-------------------------- 1 file changed, 5 insertions(+), 26 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 7bf0c2d99..66b069cc5 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -69,7 +69,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `uint8` | a succinct representation of 8 boolean participation flags | +| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | ## Constants @@ -99,6 +99,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | +| `PARTICIPATION_FLAGS_LENGTH` | `8` | ## Configuration @@ -247,28 +248,6 @@ def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: ) ``` -#### `add_flag` - -```python -def add_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: - """ - Return a new ``ParticipationFlags`` adding ``flag_index`` to ``flags``. - """ - flag = ParticipationFlags(2**flag_index) - return flags | flag -``` - -#### `has_flag` - -```python -def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: - """ - Return whether ``flags`` has ``flag_index`` set. - """ - flag = ParticipationFlags(2**flag_index) - return flags & flag == flag -``` - ### Beacon state accessors #### `get_sync_committee_indices` @@ -348,7 +327,7 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo else: epoch_participation = state.previous_epoch_participation active_validator_indices = get_active_validator_indices(state, epoch) - participating_indices = [i for i in active_validator_indices if has_flag(epoch_participation[i], flag_index)] + participating_indices = [i for i in active_validator_indices if epoch_participation[i][flag_index]] return set(filter(lambda index: not state.validators[index].slashed, participating_indices)) ``` @@ -492,8 +471,8 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): for flag_index, weight in get_flag_indices_and_weights(): - if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): - epoch_participation[index] = add_flag(epoch_participation[index], flag_index) + if flag_index in participation_flag_indices and not epoch_participation[index][flag_index]: + epoch_participation[index][flag_index] = True proposer_reward_numerator += get_base_reward(state, index) * weight # Reward proposer From df6bd1b6c33833c264561da361436453e5e55f4c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 12:26:37 +0100 Subject: [PATCH 132/227] BitVector -> Bitvector --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 66b069cc5..e023db1f7 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -69,7 +69,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | +| `ParticipationFlags` | `Bitvector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | ## Constants From d383a1421338e51908fc389519d5bc2e3603e173 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 5 May 2021 19:47:26 +0800 Subject: [PATCH 133/227] Fix ToC --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 60bbd69d8..2272a3359 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -14,8 +14,8 @@ - [Misc](#misc) - [Configuration](#configuration) - [Updated penalty values](#updated-penalty-values) + - [Sync committee](#sync-committee) - [Misc](#misc-1) - - [Time parameters](#time-parameters) - [Domain types](#domain-types) - [Containers](#containers) - [Modified containers](#modified-containers) From 6a9b3671f04f568a314e9d33f5e4b8060d9481a2 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 5 May 2021 19:53:37 +0800 Subject: [PATCH 134/227] Fix the type of `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` and update the config files --- configs/mainnet/altair.yaml | 14 +++++++------- configs/minimal/altair.yaml | 15 ++++++++------- specs/altair/beacon-chain.md | 2 +- 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index 3cd4b8419..477a4e7f7 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -10,22 +10,22 @@ MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR: 64 PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 -# Misc +# Sync committee # --------------------------------------------------------------- # 2**10 (= 1,024) SYNC_COMMITTEE_SIZE: 1024 # 2**6 (= 64) SYNC_PUBKEYS_PER_AGGREGATE: 64 -# 2**2 (= 4) -INACTIVITY_SCORE_BIAS: 4 - - -# Time parameters -# --------------------------------------------------------------- # 2**8 (= 256) EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 256 +# Misc +# --------------------------------------------------------------- +# 2**2 (= 4) +INACTIVITY_SCORE_BIAS: 4 + + # Signature domains # --------------------------------------------------------------- DOMAIN_SYNC_COMMITTEE: 0x07000000 diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index f9b30eea2..36e7c5021 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -10,22 +10,22 @@ MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR: 64 PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR: 2 -# Misc +# Sync committee # --------------------------------------------------------------- # [customized] SYNC_COMMITTEE_SIZE: 32 # [customized] SYNC_PUBKEYS_PER_AGGREGATE: 16 -# 2**2 (= 4) -INACTIVITY_SCORE_BIAS: 4 - - -# Time parameters -# --------------------------------------------------------------- # [customized] EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 8 +# Misc +# --------------------------------------------------------------- +# 2**2 (= 4) +INACTIVITY_SCORE_BIAS: 4 + + # Signature domains # --------------------------------------------------------------- DOMAIN_SYNC_COMMITTEE: 0x07000000 @@ -50,6 +50,7 @@ MAX_VALID_LIGHT_CLIENT_UPDATES: 32 # [customized] LIGHT_CLIENT_UPDATE_TIMEOUT: 32 + # Validator # --------------------------------------------------------------- # 2**2 (= 4) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 2272a3359..cbad9413d 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -120,7 +120,7 @@ This patch updates a few configuration values to move penalty parameters toward | - | - | - | - | | `SYNC_COMMITTEE_SIZE` | `uint64(2**10)` (= 1,024) | | | | `SYNC_PUBKEYS_PER_AGGREGATE` | `uint64(2**6)` (= 64) | | | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**8)` (= 256) | epochs | ~27 hours | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `uint64(2**8)` (= 256) | epochs | ~27 hours | ### Misc From e2be7614cc2224c75ac6c0356dff48a9b31563e0 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 15:35:36 +0200 Subject: [PATCH 135/227] introduce merge fork version --- specs/merge/beacon-chain.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index d665f1151..50b9551b0 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -13,8 +13,8 @@ - [Introduction](#introduction) - [Custom types](#custom-types) - [Constants](#constants) - - [Transition](#transition) - [Execution](#execution) +- [Configuration](#configuration) - [Containers](#containers) - [Extended containers](#extended-containers) - [`BeaconBlockBody`](#beaconblockbody) @@ -50,12 +50,6 @@ We define the following Python custom types for type hinting and readability: ## Constants -### Transition - -| Name | Value | -| - | - | -| `TRANSITION_TOTAL_DIFFICULTY` | **TBD** | - ### Execution | Name | Value | @@ -64,6 +58,16 @@ We define the following Python custom types for type hinting and readability: | `MAX_EXECUTION_TRANSACTIONS` | `uint64(2**14)` (= 16,384) | | `BYTES_PER_LOGS_BLOOM` | `uint64(2**8)` (= 256) | +## Configuration + +Warning: this configuration is not definitive. + +| Name | Value | +| - | - | +| `MERGE_FORK_VERSION` | `Version('0x02000000')` | +| `MERGE_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | +| `TRANSITION_TOTAL_DIFFICULTY` | **TBD** | + ## Containers ### Extended containers From 470c6dcc6f38093a6991ee8bb78d1385d11d0d08 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 15:35:52 +0200 Subject: [PATCH 136/227] update test runner to handle merge phase --- tests/core/pyspec/eth2spec/test/context.py | 9 +++++++-- tests/core/pyspec/eth2spec/test/helpers/genesis.py | 3 +++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 8699c65ad..65201c0c4 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -7,7 +7,7 @@ from eth2spec.utils import bls from .exceptions import SkippedTest from .helpers.constants import ( - PHASE0, ALTAIR, + PHASE0, ALTAIR, MERGE, ALL_PHASES, FORKS_BEFORE_ALTAIR, FORKS_BEFORE_MERGE, ) from .helpers.genesis import create_genesis_state @@ -312,7 +312,7 @@ def with_phases(phases, other_phases=None): return None run_phases = [phase] - if PHASE0 not in run_phases and ALTAIR not in run_phases: + if PHASE0 not in run_phases and ALTAIR not in run_phases and MERGE not in run_phases: dump_skipping_message("none of the recognized phases are executable, skipping test.") return None @@ -331,12 +331,17 @@ def with_phases(phases, other_phases=None): if ALTAIR in available_phases: phase_dir[ALTAIR] = spec_altair + if MERGE in available_phases: + phase_dir[MERGE] = spec_merge + # return is ignored whenever multiple phases are ran. # This return is for test generators to emit python generators (yielding test vector outputs) if PHASE0 in run_phases: ret = fn(spec=spec_phase0, phases=phase_dir, *args, **kw) if ALTAIR in run_phases: ret = fn(spec=spec_altair, phases=phase_dir, *args, **kw) + if MERGE in run_phases: + ret = fn(spec=spec_merge, phases=phase_dir, *args, **kw) # TODO: merge, sharding, custody_game and das are not executable yet. # Tests that specify these features will not run, and get ignored for these specific phases. diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 49af43ec1..4a34a5eb3 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -1,6 +1,7 @@ from eth2spec.test.helpers.constants import ( ALTAIR, FORKS_BEFORE_ALTAIR, + MERGE, ) from eth2spec.test.helpers.keys import pubkeys @@ -28,6 +29,8 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if spec.fork == ALTAIR: current_version = spec.ALTAIR_FORK_VERSION + elif spec.fork == MERGE: + current_version = spec.MERGE_FORK_VERSION state = spec.BeaconState( genesis_time=0, From ff3a82e0f38de19559bbba52e2693e01d4dad44f Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 15:40:56 +0200 Subject: [PATCH 137/227] fix transactions field in exec payload helper --- tests/core/pyspec/eth2spec/test/helpers/execution_payload.py | 2 +- .../merge/block_processing/test_process_execution_payload.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index e829e07c7..0d035350d 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -18,7 +18,7 @@ def build_empty_execution_payload(spec, state): timestamp=timestamp, receipt_root=b"no receipts here" + b"\x00"*16, # TODO: root of empty MPT may be better. logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? - transactions_root=empty_txs, + transactions=empty_txs, ) # TODO: real RLP + block hash logic would be nice, requires RLP and keccak256 dependency however. payload.block_hash = spec.Hash32(spec.hash(payload.hash_tree_root() + b"FAKE RLP HASH")) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index 0cf5cef5f..f3f521c4b 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,4 +1,5 @@ from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.helpers.execution_payload import build_empty_execution_payload from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases_except with_merge_and_later = with_all_phases_except([PHASE0, ALTAIR]) @@ -36,7 +37,7 @@ def test_success_first_payload(spec, state): assert not spec.is_transition_completed(state) # TODO: execution payload - execution_payload = spec.ExecutionPayload() + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) From 865d7db5cad448e8d2ac37859a74eff9d8a2d633 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:03:52 +0200 Subject: [PATCH 138/227] update altair tests to not collide with Merge + fix merge test triggers --- .../test_process_sync_committee.py | 20 +++++++++---------- .../test_process_sync_committee_updates.py | 8 ++++---- .../test/altair/sanity/test_blocks.py | 17 ++++++++-------- .../unittests/validator/test_validator.py | 8 ++++---- tests/core/pyspec/eth2spec/test/context.py | 8 ++++++++ .../pyspec/eth2spec/test/helpers/block.py | 2 ++ .../test_process_execution_payload.py | 5 +---- .../eth2spec/test/merge/sanity/test_blocks.py | 4 ++-- 8 files changed, 39 insertions(+), 33 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index e0faf3d0d..9b415cfbd 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -17,7 +17,7 @@ from eth2spec.test.helpers.sync_committee import ( ) from eth2spec.test.context import ( expect_assertion_error, - with_all_phases_except, + with_altair_and_later, with_configs, spec_state_test, always_bls, @@ -62,7 +62,7 @@ def get_committee_indices(spec, state, duplicates=False): state.randao_mixes[randao_index] = hash(state.randao_mixes[randao_index]) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_missing_participant(spec, state): @@ -84,7 +84,7 @@ def test_invalid_signature_missing_participant(spec, state): yield from run_sync_committee_processing(spec, state, block, expect_exception=True) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_extra_participant(spec, state): @@ -197,7 +197,7 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): ) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_configs([MINIMAL], reason="to create nonduplicate committee") @spec_state_test def test_sync_committee_rewards_nonduplicate_committee(spec, state): @@ -213,7 +213,7 @@ def test_sync_committee_rewards_nonduplicate_committee(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_configs([MAINNET], reason="to create duplicate committee") @spec_state_test def test_sync_committee_rewards_duplicate_committee(spec, state): @@ -229,7 +229,7 @@ def test_sync_committee_rewards_duplicate_committee(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_not_full_participants(spec, state): @@ -240,7 +240,7 @@ def test_sync_committee_rewards_not_full_participants(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_empty_participants(spec, state): @@ -250,7 +250,7 @@ def test_sync_committee_rewards_empty_participants(spec, state): yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_past_block(spec, state): @@ -289,7 +289,7 @@ def test_invalid_signature_past_block(spec, state): yield from run_sync_committee_processing(spec, state, invalid_block, expect_exception=True) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_configs([MINIMAL], reason="to produce different committee sets") @spec_state_test @always_bls @@ -326,7 +326,7 @@ def test_invalid_signature_previous_committee(spec, state): yield from run_sync_committee_processing(spec, state, block, expect_exception=True) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls @with_configs([MINIMAL], reason="too slow") diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index 8ffc51507..c914c476d 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -2,7 +2,7 @@ from eth2spec.test.context import ( always_bls, spec_state_test, spec_test, - with_all_phases_except, + with_altair_and_later, with_configs, with_custom_state, single_phase, @@ -49,7 +49,7 @@ def run_sync_committees_progress_test(spec, state): assert state.next_sync_committee == third_sync_committee -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls @with_configs([MINIMAL], reason="too slow") @@ -60,7 +60,7 @@ def test_sync_committees_progress_genesis(spec, state): yield from run_sync_committees_progress_test(spec, state) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test @always_bls @with_configs([MINIMAL], reason="too slow") @@ -73,7 +73,7 @@ def test_sync_committees_progress_not_genesis(spec, state): yield from run_sync_committees_progress_test(spec, state) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_custom_state(balances_fn=misc_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) @spec_test @single_phase diff --git a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py index 48ab6956b..ffe743531 100644 --- a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py @@ -11,9 +11,8 @@ from eth2spec.test.helpers.block import ( from eth2spec.test.helpers.sync_committee import ( compute_aggregate_sync_committee_signature, ) -from eth2spec.test.helpers.constants import PHASE0 from eth2spec.test.context import ( - with_all_phases_except, + with_altair_and_later, spec_state_test, ) @@ -40,46 +39,46 @@ def run_sync_committee_sanity_test(spec, state, fraction_full=1.0): yield 'post', state -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_full_sync_committee_committee(spec, state): next_epoch(spec, state) yield from run_sync_committee_sanity_test(spec, state, fraction_full=1.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_half_sync_committee_committee(spec, state): next_epoch(spec, state) yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.5) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_empty_sync_committee_committee(spec, state): next_epoch(spec, state) yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_full_sync_committee_committee_genesis(spec, state): yield from run_sync_committee_sanity_test(spec, state, fraction_full=1.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_half_sync_committee_committee_genesis(spec, state): yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.5) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_empty_sync_committee_committee_genesis(spec, state): yield from run_sync_committee_sanity_test(spec, state, fraction_full=0.0) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @spec_state_test def test_inactivity_scores(spec, state): for _ in range(spec.MIN_EPOCHS_TO_INACTIVITY_PENALTY + 2): diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index 244e1aa7e..e1f5d3860 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -8,7 +8,7 @@ from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( PHASE0, - with_all_phases_except, + with_altair_and_later, with_state, ) @@ -25,7 +25,7 @@ def ensure_assignments_in_sync_committee( assert spec.is_assigned_to_sync_committee(state, epoch, validator_index) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_state def test_is_assigned_to_sync_committee(phases, spec, state): epoch = spec.get_current_epoch(state) @@ -91,7 +91,7 @@ def _get_sync_committee_signature( @only_with_bls() -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_state def test_process_sync_committee_contributions(phases, spec, state): # skip over slots at genesis @@ -144,7 +144,7 @@ def _subnet_for_sync_committee_index(spec, i): return i // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) -@with_all_phases_except([PHASE0]) +@with_altair_and_later @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): some_sync_committee_members = list( diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 65201c0c4..775efdcf3 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -367,12 +367,20 @@ def with_configs(configs, reason=None): def is_post_altair(spec): + if spec.fork == MERGE: # TODO: remove parallel Altair-Merge condition after rebase. + return False if spec.fork in FORKS_BEFORE_ALTAIR: return False return True def is_post_merge(spec): + if spec.fork == ALTAIR: # TODO: remove parallel Altair-Merge condition after rebase. + return False if spec.fork in FORKS_BEFORE_MERGE: return False return True + + +with_altair_and_later = with_phases([ALTAIR]) # TODO: include Merge, but not until Merge work is rebased. +with_merge_and_later = with_phases([MERGE]) diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 64f406633..41d0a9131 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -96,6 +96,8 @@ def build_empty_block(spec, state, slot=None): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY if is_post_merge(spec): + if not hasattr(state, 'latest_execution_payload_header'): + raise Exception("panic!!!") empty_block.body.execution_payload = build_empty_execution_payload(spec, state) apply_randao_reveal(spec, state, empty_block) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index f3f521c4b..96a657d7a 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,8 +1,5 @@ -from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.test.helpers.execution_payload import build_empty_execution_payload -from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases_except - -with_merge_and_later = with_all_phases_except([PHASE0, ALTAIR]) +from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_merge_and_later def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): diff --git a/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py index 1f9e69167..4a6db4106 100644 --- a/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/merge/sanity/test_blocks.py @@ -5,11 +5,11 @@ from eth2spec.test.helpers.block import ( build_empty_block_for_next_slot ) from eth2spec.test.context import ( - with_all_phases, spec_state_test + with_merge_and_later, spec_state_test ) -@with_all_phases +@with_merge_and_later @spec_state_test def test_empty_block_transition(spec, state): yield 'pre', state From cc11328f74d77bf3a914f71c6cbc606620bf0c24 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:24:44 +0200 Subject: [PATCH 139/227] fix merge forkchoice tests with mock get_pow_block --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 37fbd7795..9e5a546fa 100644 --- a/setup.py +++ b/setup.py @@ -441,7 +441,7 @@ ExecutionState = Any def get_pow_block(hash: Bytes32) -> PowBlock: - pass + return PowBlock(block_hash=hash, is_valid=True, is_processed=True, total_difficulty=TRANSITION_TOTAL_DIFFICULTY) def get_execution_state(execution_state_root: Bytes32) -> ExecutionState: From 00cd1c3db76c77fe468721075347df683adb8aa3 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:31:28 +0200 Subject: [PATCH 140/227] fix forkchoice unittest not recognizing merge spec --- .../test/phase0/unittests/fork_choice/test_on_attestation.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py index fcc5e2d21..8ff6bbb38 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py +++ b/tests/core/pyspec/eth2spec/test/phase0/unittests/fork_choice/test_on_attestation.py @@ -1,7 +1,7 @@ from eth2spec.test.context import with_all_phases, spec_state_test from eth2spec.test.helpers.block import build_empty_block_for_next_slot from eth2spec.test.helpers.attestations import get_valid_attestation, sign_attestation -from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.helpers.constants import PHASE0, ALTAIR, MERGE from eth2spec.test.helpers.state import transition_to, state_transition_and_sign_block, next_epoch, next_slot from eth2spec.test.helpers.fork_choice import get_genesis_forkchoice_store @@ -19,7 +19,7 @@ def run_on_attestation(spec, state, store, attestation, valid=True): spec.on_attestation(store, attestation) sample_index = indexed_attestation.attesting_indices[0] - if spec.fork in (PHASE0, ALTAIR): + if spec.fork in (PHASE0, ALTAIR, MERGE): latest_message = spec.LatestMessage( epoch=attestation.data.target.epoch, root=attestation.data.beacon_block_root, From ed4b8d5f184d6eb38480899457f8b6ecd9200fb7 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Wed, 5 May 2021 07:40:23 -0700 Subject: [PATCH 141/227] `get_shard_proposer_index` to use `DOMAIN_SHARD_PROPOSER` --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index bddc24a40..e13ca6c5b 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -402,7 +402,7 @@ def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard """ epoch = compute_epoch_at_slot(slot) committee = get_shard_committee(beacon_state, epoch, shard) - seed = hash(get_seed(beacon_state, epoch, DOMAIN_BEACON_PROPOSER) + uint_to_bytes(slot)) + seed = hash(get_seed(beacon_state, epoch, DOMAIN_SHARD_PROPOSER) + uint_to_bytes(slot)) # Proposer must have sufficient balance to pay for worst case fee burn EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION = ( From 2ef6291cbc74237c364c22c4f29f8debafe1d4b0 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:41:59 +0200 Subject: [PATCH 142/227] Minimal execution payload test, more merge-specific testing in later PR --- .../test_process_execution_payload.py | 109 +----------------- 1 file changed, 2 insertions(+), 107 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index 96a657d7a..acb88b944 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,6 +1,6 @@ from eth2spec.test.helpers.execution_payload import build_empty_execution_payload from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_merge_and_later - +from eth2spec.test.helpers.state import next_slot def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): """ @@ -31,115 +31,10 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, @with_merge_and_later @spec_state_test def test_success_first_payload(spec, state): + next_slot(spec, state) assert not spec.is_transition_completed(state) - # TODO: execution payload execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) - -@with_merge_and_later -@spec_state_test -def test_success_regular_payload(spec, state): - # TODO: setup state - assert spec.is_transition_completed(state) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload) - - -@with_merge_and_later -@spec_state_test -def test_success_first_payload_with_gap_slot(spec, state): - # TODO: transition gap slot - - assert not spec.is_transition_completed(state) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload) - - -@with_merge_and_later -@spec_state_test -def test_success_regular_payload_with_gap_slot(spec, state): - # TODO: setup state - assert spec.is_transition_completed(state) - # TODO: transition gap slot - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload) - - -@with_merge_and_later -@spec_state_test -def test_bad_execution_first_payload(spec, state): - # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) - - # TODO: execution payload. - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_execution_regular_payload(spec, state): - # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_parent_hash_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_number_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_everything_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_timestamp_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - - -@with_merge_and_later -@spec_state_test -def test_bad_timestamp_regular_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() - - yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) - From 56bcb630dbfa487a4797c718e5a0b5c3ac04abc2 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 17:03:29 +0200 Subject: [PATCH 143/227] Lint fixes for merge testing update --- .../altair/block_processing/test_process_sync_committee.py | 1 - .../epoch_processing/test_process_sync_committee_updates.py | 5 +---- .../test/altair/unittests/validator/test_validator.py | 1 - tests/core/pyspec/eth2spec/test/helpers/execution_payload.py | 2 +- .../merge/block_processing/test_process_execution_payload.py | 4 ++-- 5 files changed, 4 insertions(+), 9 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 9b415cfbd..db2664b21 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -9,7 +9,6 @@ from eth2spec.test.helpers.state import ( transition_to, ) from eth2spec.test.helpers.constants import ( - PHASE0, MAINNET, MINIMAL, ) from eth2spec.test.helpers.sync_committee import ( diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c914c476d..c909c791c 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -8,10 +8,7 @@ from eth2spec.test.context import ( single_phase, misc_balances, ) -from eth2spec.test.helpers.constants import ( - PHASE0, - MINIMAL, -) +from eth2spec.test.helpers.constants import MINIMAL from eth2spec.test.helpers.state import transition_to from eth2spec.test.helpers.epoch_processing import ( run_epoch_processing_with, diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index e1f5d3860..c8a894da6 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -7,7 +7,6 @@ from eth2spec.test.helpers.state import transition_to from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( - PHASE0, with_altair_and_later, with_state, ) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index 0d035350d..093b7cf2e 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -16,7 +16,7 @@ def build_empty_execution_payload(spec, state): gas_limit=latest.gas_limit, # retain same limit gas_used=0, # empty block, 0 gas timestamp=timestamp, - receipt_root=b"no receipts here" + b"\x00"*16, # TODO: root of empty MPT may be better. + receipt_root=b"no receipts here" + b"\x00" * 16, # TODO: root of empty MPT may be better. logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? transactions=empty_txs, ) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index acb88b944..d45a2689b 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,7 +1,8 @@ from eth2spec.test.helpers.execution_payload import build_empty_execution_payload -from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_merge_and_later +from eth2spec.test.context import spec_state_test, expect_assertion_error, with_merge_and_later from eth2spec.test.helpers.state import next_slot + def run_execution_payload_processing(spec, state, execution_payload, valid=True, execution_valid=True): """ Run ``process_execution_payload``, yielding: @@ -37,4 +38,3 @@ def test_success_first_payload(spec, state): execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) - From a2cf833437dd5ceba194de702a00ff6243313fe9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 5 May 2021 09:32:28 -0700 Subject: [PATCH 144/227] Update specs/altair/beacon-chain.md Co-authored-by: Hsiao-Wei Wang --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index aa961d956..62f6174cf 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -126,7 +126,7 @@ This patch updates a few configuration values to move penalty parameters toward | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `Epoch(2**9)` (= 512) | epochs | ~54 hours | +| `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` | `uint64(2**9)` (= 512) | epochs | ~54 hours | ### Domain types From 238a9b03fc794bfe322ebec4b67290df4159891c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 18:09:36 +0100 Subject: [PATCH 145/227] Correct confusing comments in "get_sync_committee_indices" --- specs/altair/beacon-chain.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e023db1f7..719f93033 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -205,6 +205,7 @@ class BeaconState(Container): ```python class SyncAggregate(Container): sync_committee_bits: Bitvector[SYNC_COMMITTEE_SIZE] + # TODO! Need multiple signatures as discussed between Justin and Vitalik May 3 2021 (see Telegram) sync_committee_signature: BLSSignature ``` @@ -255,8 +256,9 @@ def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: ```python def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices (which may include duplicate indices) + Return the sequence of sync committee indices for a given ``state`` and ``epoch``. + Can contain duplicate indices for small validator sets (< 2 * SYNC_COMMITTEE_SIZE) """ MAX_RANDOM_BYTE = 2**8 - 1 base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) @@ -270,7 +272,7 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val candidate_index = active_validator_indices[shuffled_index] random_byte = hash(seed + uint_to_bytes(uint64(i // 32)))[i % 32] effective_balance = state.validators[candidate_index].effective_balance - if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: # Sample with replacement + if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: sync_committee_indices.append(candidate_index) i += 1 return sync_committee_indices From 2aef63be01bd1a5feadde55aece356e6c6914cbf Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 5 May 2021 20:17:09 +0300 Subject: [PATCH 146/227] Fix back the EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION constant calculation --- specs/sharding/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index e13ca6c5b..2e2aba8c0 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -406,7 +406,7 @@ def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard # Proposer must have sufficient balance to pay for worst case fee burn EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION = ( - (EFFECTIVE_BALANCE_INCREMENT - EFFECTIVE_BALANCE_INCREMENT) + EFFECTIVE_BALANCE_INCREMENT - EFFECTIVE_BALANCE_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER // HYSTERESIS_QUOTIENT ) min_effective_balance = ( From a382cf6d5c97409868a3da558351522b8b710d95 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 20:04:06 +0100 Subject: [PATCH 147/227] Refactor participation flag list --- specs/altair/beacon-chain.md | 34 ++++++++++------------------------ 1 file changed, 10 insertions(+), 24 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 719f93033..e311734d9 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -100,6 +100,9 @@ Altair is the first beacon chain hard fork. Its main features are: | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | | `PARTICIPATION_FLAGS_LENGTH` | `8` | +| `PARTICIPATION_FLAGS` | `3` | +| `TIMELY_HEAD_WEIGHT` | `uint64(12)` | +| `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | ## Configuration @@ -233,22 +236,6 @@ def eth2_fast_aggregate_verify(pubkeys: Sequence[BLSPubkey], message: Bytes32, s return bls.FastAggregateVerify(pubkeys, message, signature) ``` -### Misc - -#### `get_flag_indices_and_weights` - -```python -def get_flag_indices_and_weights() -> Sequence[Tuple[int, uint64]]: - """ - Return paired tuples of participation flag indices along with associated incentivization weights. - """ - return ( - (TIMELY_HEAD_FLAG_INDEX, TIMELY_HEAD_WEIGHT), - (TIMELY_SOURCE_FLAG_INDEX, TIMELY_SOURCE_WEIGHT), - (TIMELY_TARGET_FLAG_INDEX, TIMELY_TARGET_WEIGHT), - ) -``` - ### Beacon state accessors #### `get_sync_committee_indices` @@ -336,16 +323,16 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo #### `get_flag_index_deltas` ```python -def get_flag_index_deltas(state: BeaconState, flag_index: int, weight: uint64) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: +def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ Return the deltas for a given ``flag_index`` scaled by ``weight`` by scanning through the participation flags. """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, get_previous_epoch(state)) - increment = EFFECTIVE_BALANCE_INCREMENT # Factored out from balances to avoid uint64 overflow - unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // increment - active_increments = get_total_active_balance(state) // increment + weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] + unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT + active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: @@ -376,7 +363,7 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S previous_epoch = get_previous_epoch(state) matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) for index in get_eligible_validator_indices(state): - for (_, weight) in get_flag_indices_and_weights(): + for weight in PARTICIPATION_FLAG_WEIGHTS: # This inactivity penalty cancels the flag reward corresponding to the flag index penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) if index not in matching_target_indices: @@ -472,7 +459,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Update epoch participation flags proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): - for flag_index, weight in get_flag_indices_and_weights(): + for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): if flag_index in participation_flag_indices and not epoch_participation[index][flag_index]: epoch_participation[index][flag_index] = True proposer_reward_numerator += get_base_reward(state, index) * weight @@ -613,8 +600,7 @@ def process_rewards_and_penalties(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: return - flag_indices_and_numerators = get_flag_indices_and_weights() - flag_deltas = [get_flag_index_deltas(state, index, numerator) for (index, numerator) in flag_indices_and_numerators] + flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(PARTICIPATION_FLAGS)] deltas = flag_deltas + [get_inactivity_penalty_deltas(state)] for (rewards, penalties) in deltas: for index in range(len(state.validators)): From 227d1007e67558e7b7a419723efe932470976441 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 20:09:14 +0100 Subject: [PATCH 148/227] Update toc --- specs/altair/beacon-chain.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e311734d9..bd9c23bf3 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -27,10 +27,6 @@ - [Helper functions](#helper-functions) - [`Predicates`](#predicates) - [`eth2_fast_aggregate_verify`](#eth2_fast_aggregate_verify) - - [Misc](#misc-2) - - [`get_flag_indices_and_weights`](#get_flag_indices_and_weights) - - [`add_flag`](#add_flag) - - [`has_flag`](#has_flag) - [Beacon state accessors](#beacon-state-accessors) - [`get_sync_committee_indices`](#get_sync_committee_indices) - [`get_sync_committee`](#get_sync_committee) From 8ac59b73170d452c5a8e16f65b8bb14bbe8c5c75 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 22:38:16 +0200 Subject: [PATCH 149/227] fix old ssz-static todo comment --- tests/generators/ssz_static/main.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/generators/ssz_static/main.py b/tests/generators/ssz_static/main.py index 1e810f71e..d86636e85 100644 --- a/tests/generators/ssz_static/main.py +++ b/tests/generators/ssz_static/main.py @@ -93,7 +93,6 @@ if __name__ == "__main__": seed += 1 settings.append((seed, MAINNET, random_value.RandomizationMode.mode_random, False, 5)) seed += 1 - # TODO: enable testing for the whole merge spec. for fork in TESTGEN_FORKS: gen_runner.run_generator("ssz_static", [ create_provider(fork, config_name, seed, mode, chaos, cases_if_random) From 687641a79bbe9db18de358f6c230e05a59cb905e Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 21:55:36 +0100 Subject: [PATCH 150/227] Remove extra variable --- specs/altair/beacon-chain.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index bd9c23bf3..1a50e7337 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -97,7 +97,6 @@ Altair is the first beacon chain hard fork. Its main features are: | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | | `PARTICIPATION_FLAGS_LENGTH` | `8` | | `PARTICIPATION_FLAGS` | `3` | -| `TIMELY_HEAD_WEIGHT` | `uint64(12)` | | `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | ## Configuration From 55471bc5d4f69678baf6950e6197b5206b2cbb8a Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 21:58:03 +0100 Subject: [PATCH 151/227] Revert "BitVector -> Bitvector" This reverts commit df6bd1b6c33833c264561da361436453e5e55f4c. --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 1a50e7337..9e868fbaa 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -65,7 +65,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `Bitvector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | +| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | ## Constants From 77524036f56c71d4c8199c1393af524b7841290f Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 22:02:37 +0100 Subject: [PATCH 152/227] Revert "Flags to BitVector" This reverts commit 786e611c71d5c62bf30c988c692ab88d1a54d8f3. # Conflicts: # specs/altair/beacon-chain.md --- specs/altair/beacon-chain.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9e868fbaa..5bce9ba4e 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -27,6 +27,9 @@ - [Helper functions](#helper-functions) - [`Predicates`](#predicates) - [`eth2_fast_aggregate_verify`](#eth2_fast_aggregate_verify) + - [Misc](#misc-2) + - [`add_flag`](#add_flag) + - [`has_flag`](#has_flag) - [Beacon state accessors](#beacon-state-accessors) - [`get_sync_committee_indices`](#get_sync_committee_indices) - [`get_sync_committee`](#get_sync_committee) @@ -65,7 +68,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | SSZ equivalent | Description | | - | - | - | -| `ParticipationFlags` | `BitVector[PARTICIPATION_FLAGS_LENGTH]` | a succinct representation of up to 8 boolean participation flags | +| `ParticipationFlags` | `uint8` | a succinct representation of 8 boolean participation flags | ## Constants @@ -95,7 +98,6 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | -| `PARTICIPATION_FLAGS_LENGTH` | `8` | | `PARTICIPATION_FLAGS` | `3` | | `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | @@ -231,6 +233,30 @@ def eth2_fast_aggregate_verify(pubkeys: Sequence[BLSPubkey], message: Bytes32, s return bls.FastAggregateVerify(pubkeys, message, signature) ``` +### Misc + +#### `add_flag` + +```python +def add_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: + """ + Return a new ``ParticipationFlags`` adding ``flag_index`` to ``flags``. + """ + flag = ParticipationFlags(2**flag_index) + return flags | flag +``` + +#### `has_flag` + +```python +def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: + """ + Return whether ``flags`` has ``flag_index`` set. + """ + flag = ParticipationFlags(2**flag_index) + return flags & flag == flag +``` + ### Beacon state accessors #### `get_sync_committee_indices` @@ -311,7 +337,7 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo else: epoch_participation = state.previous_epoch_participation active_validator_indices = get_active_validator_indices(state, epoch) - participating_indices = [i for i in active_validator_indices if epoch_participation[i][flag_index]] + participating_indices = [i for i in active_validator_indices if has_flag(epoch_participation[i], flag_index)] return set(filter(lambda index: not state.validators[index].slashed, participating_indices)) ``` From b041a9b0d68cdee2de0d0d36590f07b49ea12dff Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 5 May 2021 22:22:32 +0100 Subject: [PATCH 153/227] Further flag_index revert --- specs/altair/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5bce9ba4e..5a8dcce24 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -481,8 +481,8 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): - if flag_index in participation_flag_indices and not epoch_participation[index][flag_index]: - epoch_participation[index][flag_index] = True + if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): + epoch_participation[index] = add_flag(epoch_participation[index], flag_index) proposer_reward_numerator += get_base_reward(state, index) * weight # Reward proposer From 76b5974d11692326cbbf513e9c04130aceabeba4 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 6 May 2021 02:21:52 +0200 Subject: [PATCH 154/227] is_execution_enabled function + misc review fixes Co-Authored-By: Danny Ryan --- specs/merge/beacon-chain.md | 10 +++++++++- tests/core/pyspec/eth2spec/test/context.py | 1 - .../block_processing/test_process_execution_payload.py | 5 ++++- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 50b9551b0..626a86724 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -24,6 +24,7 @@ - [`ExecutionPayloadHeader`](#executionpayloadheader) - [Helper functions](#helper-functions) - [Misc](#misc) + - [`is_execution_enabled`](#is_execution_enabled) - [`is_transition_completed`](#is_transition_completed) - [`is_transition_block`](#is_transition_block) - [`compute_time_at_slot`](#compute_time_at_slot) @@ -140,6 +141,13 @@ class ExecutionPayloadHeader(Container): ### Misc +#### `is_execution_enabled` + +```python +def is_execution_enabled(state: BeaconState, block: BeaconBlock) -> bool: + return is_transition_completed(state) or is_transition_block(state, block) +``` + #### `is_transition_completed` ```python @@ -173,7 +181,7 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_eth1_data(state, block.body) process_operations(state, block.body) # Pre-merge, skip execution payload processing - if is_transition_completed(state) or is_transition_block(state, block): + if is_execution_enabled(state, block): process_execution_payload(state, block.body.execution_payload) # [New in Merge] ``` diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 775efdcf3..7a2e61c22 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -330,7 +330,6 @@ def with_phases(phases, other_phases=None): phase_dir[PHASE0] = spec_phase0 if ALTAIR in available_phases: phase_dir[ALTAIR] = spec_altair - if MERGE in available_phases: phase_dir[MERGE] = spec_merge diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index d45a2689b..fb1da8758 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -13,6 +13,8 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, If ``valid == False``, run expecting ``AssertionError`` """ + pre_exec_header = state.latest_execution_payload_header.copy() + yield 'pre', state yield 'execution', {'execution_valid': execution_valid} yield 'execution_payload', execution_payload @@ -26,7 +28,8 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, yield 'post', state - # TODO: any assertions to make? + assert pre_exec_header != state.latest_execution_payload_header + # TODO: any more assertions to make? @with_merge_and_later From 42733b7e34fa9f0e7fbd9eda0415171e843d0de7 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 6 May 2021 02:27:05 +0200 Subject: [PATCH 155/227] remove merge-test exec-payload trigger debug helper --- tests/core/pyspec/eth2spec/test/helpers/block.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 41d0a9131..64f406633 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -96,8 +96,6 @@ def build_empty_block(spec, state, slot=None): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY if is_post_merge(spec): - if not hasattr(state, 'latest_execution_payload_header'): - raise Exception("panic!!!") empty_block.body.execution_payload = build_empty_execution_payload(spec, state) apply_randao_reveal(spec, state, empty_block) From ea6c0429185a2db2b34d8a38ad159cd93344f33b Mon Sep 17 00:00:00 2001 From: Michael Sproul Date: Thu, 6 May 2021 10:55:10 +1000 Subject: [PATCH 156/227] Altair: carry-over prev epoch participation --- specs/altair/beacon-chain.md | 60 +++++++++++++++++++++++------------- specs/altair/fork.md | 17 ++++++++++ 2 files changed, 56 insertions(+), 21 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 7bf0c2d99..374bac5cf 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -37,6 +37,7 @@ - [`get_base_reward_per_increment`](#get_base_reward_per_increment) - [`get_base_reward`](#get_base_reward) - [`get_unslashed_participating_indices`](#get_unslashed_participating_indices) + - [`get_attestation_participation_flag_indices`](#get_attestation_participation_flag_indices) - [`get_flag_index_deltas`](#get_flag_index_deltas) - [Modified `get_inactivity_penalty_deltas`](#modified-get_inactivity_penalty_deltas) - [Beacon state mutators](#beacon-state-mutators) @@ -352,6 +353,37 @@ def get_unslashed_participating_indices(state: BeaconState, flag_index: int, epo return set(filter(lambda index: not state.validators[index].slashed, participating_indices)) ``` +#### `get_attestation_participation_flag_indices` + +```python +def get_attestation_participation_flag_indices(state: BeaconState, + data: AttestationData, + inclusion_delay: uint64) -> Sequence[int]: + """ + Return the flag indices that are satisfied by an attestation. + """ + if data.target.epoch == get_current_epoch(state): + justified_checkpoint = state.current_justified_checkpoint + else: + justified_checkpoint = state.previous_justified_checkpoint + + # Matching roots + is_matching_source = data.source == justified_checkpoint + is_matching_target = is_matching_source and data.target.root == get_block_root(state, data.target.epoch) + is_matching_head = is_matching_target and data.beacon_block_root == get_block_root_at_slot(state, data.slot) + assert is_matching_source + + participation_flag_indices = [] + if is_matching_source and inclusion_delay <= integer_squareroot(SLOTS_PER_EPOCH): + participation_flag_indices.append(TIMELY_SOURCE_FLAG_INDEX) + if is_matching_target and inclusion_delay <= SLOTS_PER_EPOCH: + participation_flag_indices.append(TIMELY_TARGET_FLAG_INDEX) + if is_matching_head and inclusion_delay == MIN_ATTESTATION_INCLUSION_DELAY: + participation_flag_indices.append(TIMELY_HEAD_FLAG_INDEX) + + return participation_flag_indices +``` + #### `get_flag_index_deltas` ```python @@ -463,32 +495,18 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: committee = get_beacon_committee(state, data.slot, data.index) assert len(attestation.aggregation_bits) == len(committee) - if data.target.epoch == get_current_epoch(state): - epoch_participation = state.current_epoch_participation - justified_checkpoint = state.current_justified_checkpoint - else: - epoch_participation = state.previous_epoch_participation - justified_checkpoint = state.previous_justified_checkpoint - - # Matching roots - is_matching_source = data.source == justified_checkpoint - is_matching_target = is_matching_source and data.target.root == get_block_root(state, data.target.epoch) - is_matching_head = is_matching_target and data.beacon_block_root == get_block_root_at_slot(state, data.slot) - assert is_matching_source + # Participation flag indices + participation_flag_indices = get_attestation_participation_flag_indices(state, data, state.slot - data.slot) # Verify signature assert is_valid_indexed_attestation(state, get_indexed_attestation(state, attestation)) - # Participation flag indices - participation_flag_indices = [] - if is_matching_source and state.slot <= data.slot + integer_squareroot(SLOTS_PER_EPOCH): - participation_flag_indices.append(TIMELY_SOURCE_FLAG_INDEX) - if is_matching_target and state.slot <= data.slot + SLOTS_PER_EPOCH: - participation_flag_indices.append(TIMELY_TARGET_FLAG_INDEX) - if is_matching_head and state.slot == data.slot + MIN_ATTESTATION_INCLUSION_DELAY: - participation_flag_indices.append(TIMELY_HEAD_FLAG_INDEX) - # Update epoch participation flags + if data.target.epoch == get_current_epoch(state): + epoch_participation = state.current_epoch_participation + else: + epoch_participation = state.previous_epoch_participation + proposer_reward_numerator = 0 for index in get_attesting_indices(state, data, attestation.aggregation_bits): for flag_index, weight in get_flag_indices_and_weights(): diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..d8374d009 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -41,6 +41,21 @@ Note that for the pure Altair networks, we don't apply `upgrade_to_altair` since After `process_slots` of Phase 0 finishes, if `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. ```python +def translate_participation(state: BeaconState, pending_attestations: Sequence[PendingAttestation]) -> None: + for attestation in pending_attestations: + data = attestation.data + inclusion_delay = attestation.inclusion_delay + # Translate attestation inclusion info to flag indices + participation_flag_indices = get_attestation_participation_flag_indices(state, data, inclusion_delay) + + # Apply flags to all attesting validators + epoch_participation = state.previous_epoch_participation + for index in get_attesting_indices(state, data, attestation.aggregation_bits): + for flag_index, weight in get_flag_indices_and_weights(): + if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): + epoch_participation[index] = add_flag(epoch_participation[index], flag_index) + + def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: epoch = phase0.get_current_epoch(pre) post = BeaconState( @@ -80,6 +95,8 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: # Inactivity inactivity_scores=[uint64(0) for _ in range(len(pre.validators))], ) + # Fill in previous epoch participation from the pre state's pending attestations + translate_participation(post, pre.previous_epoch_attestations) # Fill in sync committees post.current_sync_committee = get_sync_committee(post, get_current_epoch(post)) post.next_sync_committee = get_sync_committee(post, get_current_epoch(post) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) From cbf9f85537c3c91c38ac51b1e3ac4efd2d5f240c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 12:28:18 +0100 Subject: [PATCH 157/227] Remove duplicate inactivity leak --- specs/altair/beacon-chain.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5a8dcce24..9ad2c0089 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -357,10 +357,7 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: - if is_in_inactivity_leak(state): - # This flag reward cancels the inactivity penalty corresponding to the flag index - rewards[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) - else: + if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) else: @@ -384,9 +381,6 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S previous_epoch = get_previous_epoch(state) matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) for index in get_eligible_validator_indices(state): - for weight in PARTICIPATION_FLAG_WEIGHTS: - # This inactivity penalty cancels the flag reward corresponding to the flag index - penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) if index not in matching_target_indices: penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR From cf724daa7a07092a02522879a0b1cb088db7c3b5 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 12:42:05 +0100 Subject: [PATCH 158/227] No inactivity penalty for untimely head --- specs/altair/beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9ad2c0089..e90c587be 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -354,13 +354,14 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT + in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: - if not is_in_inactivity_leak(state): + if not in_inactivity_leak: reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - else: + elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) return rewards, penalties ``` From 81a8c2748fed22b6b5753840f228f00b212c8072 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 13:03:26 +0100 Subject: [PATCH 159/227] Integrate get_inactivity_penalty_deltas into reward computation for better readability --- specs/altair/beacon-chain.md | 34 ++++++++-------------------------- 1 file changed, 8 insertions(+), 26 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index e90c587be..f464bd640 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -350,7 +350,8 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators) - unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, get_previous_epoch(state)) + previous_epoch = get_previous_epoch(state) + unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, previous_epoch) weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT @@ -363,29 +364,11 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) - return rewards, penalties -``` - -#### Modified `get_inactivity_penalty_deltas` - -*Note*: The function `get_inactivity_penalty_deltas` is modified in the selection of matching target indices -and the removal of `BASE_REWARDS_PER_EPOCH`. - -```python -def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: - """ - Return the inactivity penalty deltas by considering timely target participation flags and inactivity scores. - """ - rewards = [Gwei(0) for _ in range(len(state.validators))] - penalties = [Gwei(0) for _ in range(len(state.validators))] - if is_in_inactivity_leak(state): - previous_epoch = get_previous_epoch(state) - matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) - for index in get_eligible_validator_indices(state): - if index not in matching_target_indices: - penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] - penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR - penalties[index] += Gwei(penalty_numerator // penalty_denominator) + # Quadratic inactivity leak + if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index not in unslashed_participating_indices: + penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] + penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR + penalties[index] += Gwei(penalty_numerator // penalty_denominator) return rewards, penalties ``` @@ -617,8 +600,7 @@ def process_rewards_and_penalties(state: BeaconState) -> None: return flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(PARTICIPATION_FLAGS)] - deltas = flag_deltas + [get_inactivity_penalty_deltas(state)] - for (rewards, penalties) in deltas: + for (rewards, penalties) in flag_deltas: for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) decrease_balance(state, ValidatorIndex(index), penalties[index]) From 4c73fec88e61e83240d0d45270e7afafb0d17d0f Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 6 May 2021 10:00:04 -0700 Subject: [PATCH 160/227] convert `LightClientStore` to python object instead of SSZ object this avoids the type overhead of having to define a max size for the object's data and skips the overhead of serialization/consensus for a type that does not need it --- configs/mainnet/altair.yaml | 2 -- configs/minimal/altair.yaml | 3 +-- setup.py | 2 +- specs/altair/sync-protocol.md | 12 ++++++------ .../test/altair/unittests/test_sync_protocol.py | 8 ++++---- 5 files changed, 12 insertions(+), 15 deletions(-) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index b7b7603a2..eac4e1c14 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -45,8 +45,6 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# 2**64 - 1 -MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 # Validator diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index 5618af8ff..0300ff1b8 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -45,8 +45,7 @@ ALTAIR_FORK_EPOCH: 18446744073709551615 # --------------------------------------------------------------- # 1 MIN_SYNC_COMMITTEE_PARTICIPANTS: 1 -# 2**64 - 1 -MAX_VALID_LIGHT_CLIENT_UPDATES: 18446744073709551615 + # Validator # --------------------------------------------------------------- diff --git a/setup.py b/setup.py index 37fbd7795..aad834e2c 100644 --- a/setup.py +++ b/setup.py @@ -548,7 +548,7 @@ ignored_dependencies = [ 'Bytes1', 'Bytes4', 'Bytes20', 'Bytes32', 'Bytes48', 'Bytes96', 'Bitlist', 'Bitvector', 'uint8', 'uint16', 'uint32', 'uint64', 'uint128', 'uint256', 'bytes', 'byte', 'ByteList', 'ByteVector', - 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', + 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', 'Set' ] diff --git a/specs/altair/sync-protocol.md b/specs/altair/sync-protocol.md index ca55c92f7..784ea63b2 100644 --- a/specs/altair/sync-protocol.md +++ b/specs/altair/sync-protocol.md @@ -50,7 +50,6 @@ uses sync committees introduced in [this beacon chain extension](./beacon-chain. | Name | Value | | - | - | | `MIN_SYNC_COMMITTEE_PARTICIPANTS` | `1` | -| `MAX_VALID_LIGHT_CLIENT_UPDATES` | `uint64(2**64 - 1)` | ## Containers @@ -87,9 +86,10 @@ class LightClientUpdate(Container): ### `LightClientStore` ```python -class LightClientStore(Container): +@dataclass +class LightClientStore(object): snapshot: LightClientSnapshot - valid_updates: List[LightClientUpdate, MAX_VALID_LIGHT_CLIENT_UPDATES] + valid_updates: Set[LightClientUpdate] ``` ## Helper functions @@ -175,7 +175,7 @@ def apply_light_client_update(snapshot: LightClientSnapshot, update: LightClient def process_light_client_update(store: LightClientStore, update: LightClientUpdate, current_slot: Slot, genesis_validators_root: Root) -> None: validate_light_client_update(store.snapshot, update, genesis_validators_root) - store.valid_updates.append(update) + store.valid_updates.add(update) update_timeout = SLOTS_PER_EPOCH * EPOCHS_PER_SYNC_COMMITTEE_PERIOD if ( @@ -186,10 +186,10 @@ def process_light_client_update(store: LightClientStore, update: LightClientUpda # Note that (2) means that the current light client design needs finality. # It may be changed to re-organizable light client design. See the on-going issue eth2.0-specs#2182. apply_light_client_update(store.snapshot, update) - store.valid_updates = [] + store.valid_updates = set() elif current_slot > store.snapshot.header.slot + update_timeout: # Forced best update when the update timeout has elapsed apply_light_client_update(store.snapshot, max(store.valid_updates, key=lambda update: sum(update.sync_committee_bits))) - store.valid_updates = [] + store.valid_updates = set() ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py index 9b8c35e76..932a46ca5 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py @@ -32,7 +32,7 @@ def test_process_light_client_update_not_updated(spec, state): ) store = spec.LightClientStore( snapshot=pre_snapshot, - valid_updates=[] + valid_updates=set(), ) # Block at slot 1 doesn't increase sync committee period, so it won't update snapshot @@ -76,7 +76,7 @@ def test_process_light_client_update_not_updated(spec, state): spec.process_light_client_update(store, update, state.slot, state.genesis_validators_root) assert len(store.valid_updates) == 1 - assert store.valid_updates[0] == update + assert store.valid_updates.pop() == update assert store.snapshot == pre_snapshot @@ -91,7 +91,7 @@ def test_process_light_client_update_timeout(spec, state): ) store = spec.LightClientStore( snapshot=pre_snapshot, - valid_updates=[] + valid_updates=set(), ) # Forward to next sync committee period @@ -156,7 +156,7 @@ def test_process_light_client_update_finality_updated(spec, state): ) store = spec.LightClientStore( snapshot=pre_snapshot, - valid_updates=[] + valid_updates=set(), ) # Change finality From 953b0278a10bbf98ed1f6e921a7accf9b87b62c8 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 6 May 2021 10:23:50 -0700 Subject: [PATCH 161/227] whitespace --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index aad834e2c..aaed891c4 100644 --- a/setup.py +++ b/setup.py @@ -167,7 +167,7 @@ def get_spec(file_name: str) -> SpecObject: comment = _get_eth2_spec_comment(child) if comment == "skip": should_skip = True - + return SpecObject( functions=functions, custom_types=custom_types, From 4b27b076f6c2b3e90b69f6ace83e89c13f3f30fe Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 6 May 2021 12:36:08 -0600 Subject: [PATCH 162/227] add missing comma --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index aaed891c4..039e8ab63 100644 --- a/setup.py +++ b/setup.py @@ -548,7 +548,7 @@ ignored_dependencies = [ 'Bytes1', 'Bytes4', 'Bytes20', 'Bytes32', 'Bytes48', 'Bytes96', 'Bitlist', 'Bitvector', 'uint8', 'uint16', 'uint32', 'uint64', 'uint128', 'uint256', 'bytes', 'byte', 'ByteList', 'ByteVector', - 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', 'Set' + 'Dict', 'dict', 'field', 'ceillog2', 'floorlog2', 'Set', ] From ba2c717bf112bb496afc80cedbd1aeac7d59049a Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 6 May 2021 22:55:17 +0100 Subject: [PATCH 163/227] Remove PARTICIPATION_FLAGS --- specs/altair/beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f464bd640..f559a55ef 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -98,7 +98,6 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | -| `PARTICIPATION_FLAGS` | `3` | | `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | ## Configuration @@ -599,7 +598,7 @@ def process_rewards_and_penalties(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: return - flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(PARTICIPATION_FLAGS)] + flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(len(PARTICIPATION_FLAG_WEIGHTS))] for (rewards, penalties) in flag_deltas: for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) From 2fc68c451e8168b2c2cc85182e77e41550c15302 Mon Sep 17 00:00:00 2001 From: dankrad Date: Thu, 6 May 2021 22:55:55 +0100 Subject: [PATCH 164/227] Update specs/altair/beacon-chain.md Co-authored-by: Alex Stokes --- specs/altair/beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index f559a55ef..f21584f30 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -263,8 +263,7 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ```python def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices - for a given ``state`` and ``epoch``. + Return the sequence of sync committee indices for a given ``state`` and ``epoch``. Can contain duplicate indices for small validator sets (< 2 * SYNC_COMMITTEE_SIZE) """ MAX_RANDOM_BYTE = 2**8 - 1 From 2bc2a308873f263bbbcf6f57a2d1b6e6808f7ee5 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 5 May 2021 16:50:42 +0200 Subject: [PATCH 165/227] scaffold execution payload tests --- .../test_process_execution_payload.py | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index fb1da8758..6dbdc5e01 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -41,3 +41,108 @@ def test_success_first_payload(spec, state): execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_regular_payload(spec, state): + # TODO: setup state + assert spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_first_payload_with_gap_slot(spec, state): + # TODO: transition gap slot + + assert not spec.is_transition_completed(state) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_success_regular_payload_with_gap_slot(spec, state): + # TODO: setup state + assert spec.is_transition_completed(state) + # TODO: transition gap slot + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload) + + +@with_merge_and_later +@spec_state_test +def test_bad_execution_first_payload(spec, state): + # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) + + # TODO: execution payload. + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_execution_regular_payload(spec, state): + # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) + + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_parent_hash_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_number_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_everything_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_timestamp_first_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) + + +@with_merge_and_later +@spec_state_test +def test_bad_timestamp_regular_payload(spec, state): + # TODO: execution payload + execution_payload = spec.ExecutionPayload() + + yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) From 22b7c393cce25084d52935cb859a2d88026e0348 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 7 May 2021 16:52:44 +0300 Subject: [PATCH 166/227] Include {slot} param to shard_blob_ subnets --- specs/sharding/p2p-interface.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 42984dfe3..05da54749 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -15,7 +15,7 @@ - [SignedShardBlob](#signedshardblob) - [Gossip domain](#gossip-domain) - [Topics and messages](#topics-and-messages) - - [Shard blobs: `shard_blob_{shard}`](#shard-blobs-shard_blob_shard) + - [Shard blobs: `shard_blob_{shard}_{slot}`](#shard-blobs-shard_blob_shard) - [Shard header: `shard_header`](#shard-header-shard_header) - [Shard proposer slashing: `shard_proposer_slashing`](#shard-proposer-slashing-shard_proposer_slashing) @@ -77,18 +77,19 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| -| `shard_blob_{shard}` | `SignedShardBlob` | +| `shard_blob_{shard}_{slot}` | `SignedShardBlob` | | `shard_header` | `SignedShardHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. -#### Shard blobs: `shard_blob_{shard}` +#### Shard blobs: `shard_blob_{shard}_{slot}` -Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{shard}` subnets. +Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{shard}_{slot}` subnets. The following validations MUST pass before forwarding the `signed_blob` (with inner `message` as `blob`) on the horizontal subnet or creating samples for it. - _[REJECT]_ `blob.shard` MUST match the topic `{shard}` parameter. (And thus within valid shard index range) +- _[REJECT]_ `blob.slot - compute_start_slot_at_epoch(blob.slot)` MUST match the topic `{slot}` parameter. - _[IGNORE]_ The `blob` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that `blob.slot <= current_slot` (a client MAY queue future blobs for processing at the appropriate slot). From bd376fa649128bac18dc03d4e8474449df04a024 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 7 May 2021 17:09:03 +0300 Subject: [PATCH 167/227] Fix the table of contents --- specs/sharding/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 05da54749..f0e33dcf8 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -15,7 +15,7 @@ - [SignedShardBlob](#signedshardblob) - [Gossip domain](#gossip-domain) - [Topics and messages](#topics-and-messages) - - [Shard blobs: `shard_blob_{shard}_{slot}`](#shard-blobs-shard_blob_shard) + - [Shard blobs: `shard_blob_{shard}_{slot}`](#shard-blobs-shard_blob_shard_slot) - [Shard header: `shard_header`](#shard-header-shard_header) - [Shard proposer slashing: `shard_proposer_slashing`](#shard-proposer-slashing-shard_proposer_slashing) From 86104ea361921cd6b952bd13a22c6a198648404a Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 09:55:21 -0700 Subject: [PATCH 168/227] Use stable sync committee indices when processing block rewards --- specs/altair/beacon-chain.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index b2ee93557..59b11c8e4 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -571,7 +571,11 @@ def process_sync_committee(state: BeaconState, aggregate: SyncAggregate) -> None proposer_reward = Gwei(participant_reward * PROPOSER_WEIGHT // (WEIGHT_DENOMINATOR - PROPOSER_WEIGHT)) # Apply participant and proposer rewards - committee_indices = get_sync_committee_indices(state, get_current_epoch(state)) + committee_indices = [] + pubkeys = [v.pubkey for v in state.validators] + for pubkey in state.current_sync_committee.pubkeys: + index = pubkeys.index(pubkey) + committee_indices.append(ValidatorIndex(index)) participant_indices = [index for index, bit in zip(committee_indices, aggregate.sync_committee_bits) if bit] for participant_index in participant_indices: increase_balance(state, participant_index, participant_reward) From 04a9595415678013442460ef4a0ee90d0799924b Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 10:06:44 -0700 Subject: [PATCH 169/227] Add notes about sync committee stability --- specs/altair/beacon-chain.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 59b11c8e4..0855d6f11 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -277,6 +277,9 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val """ Return the sequence of sync committee indices (which may include duplicate indices) for a given ``state`` and ``epoch``. + + Note: This function is not stable during a sync committee period as + a validator's effective balance may change enough to affect the sampling. """ MAX_RANDOM_BYTE = 2**8 - 1 base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) @@ -310,6 +313,9 @@ def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_sync_committee_indices`` returns duplicate indices. Implementations must take care when handling optimizations relating to aggregation and verification in the presence of duplicates. + + Note: This function should only be called at sync committee period boundaries, as + ``get_sync_committee_indices`` is not stable within a given period. """ indices = get_sync_committee_indices(state, epoch) pubkeys = [state.validators[index].pubkey for index in indices] From d1bedbfbec74828eccdaee9ac9d9d91ddb0b1386 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 7 May 2021 11:05:27 -0700 Subject: [PATCH 170/227] Update inactivity penalty deltas processing Two main changes: 1. Inactivity scores are modified to decrease slowly for inactive validators when we are not in a leak, and quickly for active validators 2. The inactivity penalties are applied even not during a leak (note that inactivity _scores_ decrease when outside of a leak) This has the effect that the inactivity leak "overshoots" the target of finalizing again, and keeps leaking balances a bit more. For inactive validators, this PR sets post-leak recovery to happen 3x faster than the during-leak increase, so if a validator loses 3% during a leak, if they stay offline they should expect to lose another 1% until their score decreases back to zero. --- specs/altair/beacon-chain.md | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index b2ee93557..681d4aad0 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -120,6 +120,7 @@ This patch updates a few configuration values to move penalty parameters closer | - | - | | `SYNC_COMMITTEE_SIZE` | `uint64(2**9)` (= 512) | | `INACTIVITY_SCORE_BIAS` | `uint64(4)` | +| `INACTIVITY_SCORE_RECOVERY_RATE` | `uint64(16)` | ### Time parameters @@ -397,17 +398,16 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S """ rewards = [Gwei(0) for _ in range(len(state.validators))] penalties = [Gwei(0) for _ in range(len(state.validators))] - if is_in_inactivity_leak(state): - previous_epoch = get_previous_epoch(state) - matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) - for index in get_eligible_validator_indices(state): - for (_, weight) in get_flag_indices_and_weights(): - # This inactivity penalty cancels the flag reward corresponding to the flag index - penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) - if index not in matching_target_indices: - penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] - penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR - penalties[index] += Gwei(penalty_numerator // penalty_denominator) + previous_epoch = get_previous_epoch(state) + matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) + for index in get_eligible_validator_indices(state): + for (_, weight) in get_flag_indices_and_weights(): + # This inactivity penalty cancels the flag reward corresponding to the flag index + penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) + if index not in matching_target_indices: + penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] + penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR + penalties[index] += Gwei(penalty_numerator // penalty_denominator) return rewards, penalties ``` @@ -621,11 +621,14 @@ def process_justification_and_finalization(state: BeaconState) -> None: ```python def process_inactivity_updates(state: BeaconState) -> None: for index in get_eligible_validator_indices(state): + # Increase inactivity score of inactive validators if index in get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, get_previous_epoch(state)): - if state.inactivity_scores[index] > 0: - state.inactivity_scores[index] -= 1 - elif is_in_inactivity_leak(state): + state.inactivity_scores[index] -= min(1, state.inactivity_scores[index]) + else: state.inactivity_scores[index] += INACTIVITY_SCORE_BIAS + # Decrease the score of all validators for forgiveness when not during a leak + if not if is_in_inactivity_leak(state): + state.inactivity_scores[index] -= min(INACTIVITY_SCORE_RECOVERY_RATE, state.inactivity_scores[index]) ``` #### Rewards and penalties From b336b710e9bd08594af87a16f53932bbe4672fe9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 15:54:01 -0700 Subject: [PATCH 171/227] Update specs/altair/beacon-chain.md Co-authored-by: vbuterin --- specs/altair/beacon-chain.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 0855d6f11..ac4457b22 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -577,11 +577,8 @@ def process_sync_committee(state: BeaconState, aggregate: SyncAggregate) -> None proposer_reward = Gwei(participant_reward * PROPOSER_WEIGHT // (WEIGHT_DENOMINATOR - PROPOSER_WEIGHT)) # Apply participant and proposer rewards - committee_indices = [] - pubkeys = [v.pubkey for v in state.validators] - for pubkey in state.current_sync_committee.pubkeys: - index = pubkeys.index(pubkey) - committee_indices.append(ValidatorIndex(index)) + all_pubkeys = [v.pubkey for v in state.validators] + committee_indices = [ValidatorIndex(all_pubkeys.index(pubkey)) for pubkey in state.current_sync_committee.pubkeys] participant_indices = [index for index, bit in zip(committee_indices, aggregate.sync_committee_bits) if bit] for participant_index in participant_indices: increase_balance(state, participant_index, participant_reward) From 72a4ff803b3be39da3a29730a273b04ec6f35a87 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 7 May 2021 17:02:52 -0700 Subject: [PATCH 172/227] add test to ensure sync committees are referenced from the state --- .../test_process_sync_committee.py | 55 ++++++++++++++++--- 1 file changed, 48 insertions(+), 7 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index e0faf3d0d..3e4d77c0f 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -7,6 +7,7 @@ from eth2spec.test.helpers.block_processing import run_block_processing_to from eth2spec.test.helpers.state import ( state_transition_and_sign_block, transition_to, + next_epoch, ) from eth2spec.test.helpers.constants import ( PHASE0, @@ -160,13 +161,13 @@ def validate_sync_committee_rewards(spec, pre_state, post_state, committee, comm committee_bits, ) - if proposer_index == index: - reward += compute_sync_committee_proposer_reward( - spec, - pre_state, - committee, - committee_bits, - ) + if proposer_index == index: + reward += compute_sync_committee_proposer_reward( + spec, + pre_state, + committee, + committee_bits, + ) assert post_state.balances[index] == pre_state.balances[index] + reward @@ -367,3 +368,43 @@ def test_valid_signature_future_committee(spec, state): ) yield from run_sync_committee_processing(spec, state, block) + + +@with_all_phases_except([PHASE0]) +@spec_state_test +def test_sync_committee_is_only_computed_at_epoch_boundary(spec, state): + """ + Sync committees can only be computed at sync committee period boundaries. + Ensure a client respects the committee in the state (assumed to be derived + in the correct way). + """ + current_epoch = spec.get_current_epoch(state) + + # use a "synthetic" committee to simulate the situation + # where ``spec.get_sync_committee`` at the sync committee + # period epoch boundary would have diverged some epochs into the + # period; ``aggregate_pubkey`` is not relevant to this test + pubkeys = [] + committee_indices = [] + i = 0 + active_validator_count = len(spec.get_active_validator_indices(state, current_epoch)) + while len(pubkeys) < spec.SYNC_COMMITTEE_SIZE: + v = state.validators[i % active_validator_count] + if spec.is_active_validator(v, current_epoch): + pubkeys.append(v.pubkey) + committee_indices.append(i) + i += 1 + + synthetic_committee = spec.SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=spec.BLSPubkey()) + state.current_sync_committee = synthetic_committee + + assert spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD > 3 + for _ in range(3): + next_epoch(spec, state) + + committee = get_committee_indices(spec, state) + assert committee != committee_indices + committee_size = len(committee_indices) + committee_bits = [True] * committee_size + + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) From e38f758d2168b9811f7463998d0b71ea58480f83 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sun, 9 May 2021 17:08:54 -0700 Subject: [PATCH 173/227] `adjustment_quotient` to use `previous_epoch` --- specs/sharding/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index e13ca6c5b..65a0e95b7 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -714,11 +714,11 @@ def process_pending_headers(state: BeaconState) -> None: ```python def charge_confirmed_header_fees(state: BeaconState) -> None: new_gasprice = state.shard_gasprice + previous_epoch = get_previous_epoch(state) adjustment_quotient = ( - get_active_shard_count(state, get_current_epoch(state)) + get_active_shard_count(state, previous_epoch) * SLOTS_PER_EPOCH * GASPRICE_ADJUSTMENT_COEFFICIENT ) - previous_epoch = get_previous_epoch(state) previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): for shard_index in range(get_active_shard_count(state, previous_epoch)): From e78e0458470c17a89a38bca6b115c5e5b1c84f54 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Mon, 10 May 2021 16:12:23 +0600 Subject: [PATCH 174/227] Implement execution payload tests --- setup.py | 4 +- .../test/helpers/execution_payload.py | 31 ++++- .../test_process_execution_payload.py | 121 ++++++++++++------ 3 files changed, 116 insertions(+), 40 deletions(-) diff --git a/setup.py b/setup.py index 9e5a546fa..f0befc8c7 100644 --- a/setup.py +++ b/setup.py @@ -451,9 +451,9 @@ def get_execution_state(execution_state_root: Bytes32) -> ExecutionState: def get_pow_chain_head() -> PowBlock: pass - +verify_execution_state_transition_ret_value = True def verify_execution_state_transition(execution_payload: ExecutionPayload) -> bool: - return True + return verify_execution_state_transition_ret_value def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> ExecutionPayload: diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index 093b7cf2e..36e63dd33 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -1,4 +1,3 @@ - def build_empty_execution_payload(spec, state): """ Assuming a pre-state of the same slot, build a valid ExecutionPayload without any transactions. @@ -24,3 +23,33 @@ def build_empty_execution_payload(spec, state): payload.block_hash = spec.Hash32(spec.hash(payload.hash_tree_root() + b"FAKE RLP HASH")) return payload + +def get_execution_payload_header(spec, execution_payload): + return spec.ExecutionPayloadHeader( + block_hash=execution_payload.block_hash, + parent_hash=execution_payload.parent_hash, + coinbase=execution_payload.coinbase, + state_root=execution_payload.state_root, + number=execution_payload.number, + gas_limit=execution_payload.gas_limit, + gas_used=execution_payload.gas_used, + timestamp=execution_payload.timestamp, + receipt_root=execution_payload.receipt_root, + logs_bloom=execution_payload.logs_bloom, + transactions_root=spec.hash_tree_root(execution_payload.transactions) + ) + +def build_state_with_incomplete_transition(spec, state): + return build_state_with_execution_payload_header(spec, state, spec.ExecutionPayloadHeader()) + +def build_state_with_complete_transition(spec, state): + pre_state_payload = build_empty_execution_payload(spec, state) + payload_header = get_execution_payload_header(spec, pre_state_payload) + + return build_state_with_execution_payload_header(spec, state, payload_header) + +def build_state_with_execution_payload_header(spec, state, execution_payload_header): + pre_state = state.copy() + pre_state.latest_execution_payload_header = execution_payload_header + + return pre_state diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index 6dbdc5e01..f1cd8ff25 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -1,4 +1,9 @@ -from eth2spec.test.helpers.execution_payload import build_empty_execution_payload +from eth2spec.test.helpers.execution_payload import ( + build_empty_execution_payload, + get_execution_payload_header, + build_state_with_incomplete_transition, + build_state_with_complete_transition, +) from eth2spec.test.context import spec_state_test, expect_assertion_error, with_merge_and_later from eth2spec.test.helpers.state import next_slot @@ -13,31 +18,36 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, If ``valid == False``, run expecting ``AssertionError`` """ - pre_exec_header = state.latest_execution_payload_header.copy() - yield 'pre', state yield 'execution', {'execution_valid': execution_valid} yield 'execution_payload', execution_payload + + spec.verify_execution_state_transition_ret_value = execution_valid + if not valid: expect_assertion_error(lambda: spec.process_execution_payload(state, execution_payload)) yield 'post', None + spec.verify_execution_state_transition_ret_value = True return spec.process_execution_payload(state, execution_payload) yield 'post', state - assert pre_exec_header != state.latest_execution_payload_header - # TODO: any more assertions to make? + assert state.latest_execution_payload_header == get_execution_payload_header(spec, execution_payload) + + spec.verify_execution_state_transition_ret_value = True @with_merge_and_later @spec_state_test def test_success_first_payload(spec, state): + # pre-state + state = build_state_with_incomplete_transition(spec, state) next_slot(spec, state) - assert not spec.is_transition_completed(state) + # execution payload execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) @@ -46,11 +56,12 @@ def test_success_first_payload(spec, state): @with_merge_and_later @spec_state_test def test_success_regular_payload(spec, state): - # TODO: setup state - assert spec.is_transition_completed(state) + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) - # TODO: execution payload - execution_payload = spec.ExecutionPayload() + # execution payload + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) @@ -58,12 +69,13 @@ def test_success_regular_payload(spec, state): @with_merge_and_later @spec_state_test def test_success_first_payload_with_gap_slot(spec, state): - # TODO: transition gap slot + # pre-state + state = build_state_with_incomplete_transition(spec, state) + next_slot(spec, state) + next_slot(spec, state) - assert not spec.is_transition_completed(state) - - # TODO: execution payload - execution_payload = spec.ExecutionPayload() + # execution payload + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) @@ -71,12 +83,13 @@ def test_success_first_payload_with_gap_slot(spec, state): @with_merge_and_later @spec_state_test def test_success_regular_payload_with_gap_slot(spec, state): - # TODO: setup state - assert spec.is_transition_completed(state) - # TODO: transition gap slot + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) + next_slot(spec, state) - # TODO: execution payload - execution_payload = spec.ExecutionPayload() + # execution payload + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload) @@ -86,8 +99,12 @@ def test_success_regular_payload_with_gap_slot(spec, state): def test_bad_execution_first_payload(spec, state): # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) - # TODO: execution payload. - execution_payload = spec.ExecutionPayload() + # pre-state + state = build_state_with_incomplete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) @@ -97,35 +114,55 @@ def test_bad_execution_first_payload(spec, state): def test_bad_execution_regular_payload(spec, state): # completely valid payload, but execution itself fails (e.g. block exceeds gas limit) - # TODO: execution payload - execution_payload = spec.ExecutionPayload() + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False, execution_valid=False) @with_merge_and_later @spec_state_test -def test_bad_parent_hash_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() +def test_bad_parent_hash_regular_payload(spec, state): + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) + execution_payload.parent_hash = spec.Hash32() yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @with_merge_and_later @spec_state_test -def test_bad_number_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() +def test_bad_number_regular_payload(spec, state): + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) + execution_payload.number = execution_payload.number + 1 yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @with_merge_and_later @spec_state_test -def test_bad_everything_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() +def test_bad_everything_regular_payload(spec, state): + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) + execution_payload.parent_hash = spec.Hash32() + execution_payload.number = execution_payload.number + 1 yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @@ -133,8 +170,13 @@ def test_bad_everything_first_payload(spec, state): @with_merge_and_later @spec_state_test def test_bad_timestamp_first_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() + # pre-state + state = build_state_with_incomplete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) + execution_payload.timestamp = execution_payload.timestamp + 1 yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @@ -142,7 +184,12 @@ def test_bad_timestamp_first_payload(spec, state): @with_merge_and_later @spec_state_test def test_bad_timestamp_regular_payload(spec, state): - # TODO: execution payload - execution_payload = spec.ExecutionPayload() + # pre-state + state = build_state_with_complete_transition(spec, state) + next_slot(spec, state) + + # execution payload + execution_payload = build_empty_execution_payload(spec, state) + execution_payload.timestamp = execution_payload.timestamp + 1 yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) From f58ba8f5b200d158c0323e5b986456252bacbe6e Mon Sep 17 00:00:00 2001 From: protolambda Date: Mon, 10 May 2021 15:48:37 +0200 Subject: [PATCH 175/227] define execution engine protocol --- specs/merge/beacon-chain.md | 39 +++++++++++++++++++++------- specs/merge/fork-choice.md | 52 ++++++++++++++++++++++++++++++++++--- specs/merge/validator.md | 28 +++++++++++++++++--- 3 files changed, 103 insertions(+), 16 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index 626a86724..bca58eb7f 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -22,6 +22,9 @@ - [New containers](#new-containers) - [`ExecutionPayload`](#executionpayload) - [`ExecutionPayloadHeader`](#executionpayloadheader) +- [Protocols](#protocols) + - [`ExecutionEngine`](#executionengine) + - [`new_block`](#new_block) - [Helper functions](#helper-functions) - [Misc](#misc) - [`is_execution_enabled`](#is_execution_enabled) @@ -30,7 +33,6 @@ - [`compute_time_at_slot`](#compute_time_at_slot) - [Block processing](#block-processing) - [Execution payload processing](#execution-payload-processing) - - [`verify_execution_state_transition`](#verify_execution_state_transition) - [`process_execution_payload`](#process_execution_payload) @@ -137,6 +139,30 @@ class ExecutionPayloadHeader(Container): transactions_root: Root ``` +## Protocols + +### `ExecutionEngine` + +The `ExecutionEngine` protocol separates the consensus and execution sub-systems. +The consensus implementation references an instance of this sub-system with `EXECUTION_ENGINE`. + +The following methods are added to the `ExecutionEngine` protocol for use in the state transition: + +#### `new_block` + +Verifies the given `ExecutionPayload` with respect to execution state transition, and persists changes if valid. + +The body of this function is implementation dependent. +The Consensus API may be used to implement this with an external execution engine. + +```python +def new_block(self: ExecutionEngine, execution_payload: ExecutionPayload) -> bool: + """ + Returns True if the ``execution_payload`` was verified and processed successfully, False otherwise. + """ + ... +``` + ## Helper functions ### Misc @@ -182,20 +208,15 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_operations(state, block.body) # Pre-merge, skip execution payload processing if is_execution_enabled(state, block): - process_execution_payload(state, block.body.execution_payload) # [New in Merge] + process_execution_payload(state, block.body.execution_payload, EXECUTION_ENGINE) # [New in Merge] ``` #### Execution payload processing -##### `verify_execution_state_transition` - -Let `verify_execution_state_transition(execution_payload: ExecutionPayload) -> bool` be the function that verifies given `ExecutionPayload` with respect to execution state transition. -The body of the function is implementation dependent. - ##### `process_execution_payload` ```python -def process_execution_payload(state: BeaconState, execution_payload: ExecutionPayload) -> None: +def process_execution_payload(state: BeaconState, execution_payload: ExecutionPayload, execution_engine: ExecutionEngine) -> None: """ Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ @@ -205,7 +226,7 @@ def process_execution_payload(state: BeaconState, execution_payload: ExecutionPa assert execution_payload.timestamp == compute_time_at_slot(state, state.slot) - assert verify_execution_state_transition(execution_payload) + assert execution_engine.new_block(execution_payload) state.latest_execution_payload_header = ExecutionPayloadHeader( block_hash=execution_payload.block_hash, diff --git a/specs/merge/fork-choice.md b/specs/merge/fork-choice.md index f478dd7e6..9e6c341bc 100644 --- a/specs/merge/fork-choice.md +++ b/specs/merge/fork-choice.md @@ -8,11 +8,16 @@ - [Introduction](#introduction) - - [Helpers](#helpers) +- [Protocols](#protocols) + - [`ExecutionEngine`](#executionengine) + - [`set_head`](#set_head) + - [`finalize_block`](#finalize_block) +- [Containers](#containers) - [`PowBlock`](#powblock) +- [Helper functions](#helper-functions) - [`get_pow_block`](#get_pow_block) - [`is_valid_transition_block`](#is_valid_transition_block) - - [Updated fork-choice handlers](#updated-fork-choice-handlers) +- [Updated fork-choice handlers](#updated-fork-choice-handlers) - [`on_block`](#on_block) @@ -24,7 +29,44 @@ This is the modification of the fork choice according to the executable beacon c *Note*: It introduces the process of transition from the last PoW block to the first PoS block. -### Helpers +## Protocols + +### `ExecutionEngine` + +The following methods are added to the `ExecutionEngine` protocol for use in the fork choice: + +#### `set_head` + +Re-organizes the execution payload chain and corresponding state to make `block_hash` the head. + +The body of this function is implementation dependent. +The Consensus API may be used to implement this with an external execution engine. + +```python +def set_head(self: ExecutionEngine, block_hash: Hash32) -> bool: + """ + Returns True if the ``block_hash`` was successfully set as head of the execution payload chain. + """ + ... +``` + +#### `finalize_block` + +Applies finality to the execution state: it irreversibly persists the chain of all execution payloads +and corresponding state, up to and including `block_hash`. + +The body of this function is implementation dependent. +The Consensus API may be used to implement this with an external execution engine. + +```python +def finalize_block(self: ExecutionEngine, block_hash: Hash32) -> bool: + """ + Returns True if the data up to and including ``block_hash`` was successfully finalized. + """ + ... +``` + +## Containers #### `PowBlock` @@ -36,6 +78,8 @@ class PowBlock(Container): total_difficulty: uint256 ``` +## Helper functions + #### `get_pow_block` Let `get_pow_block(block_hash: Hash32) -> PowBlock` be the function that given the hash of the PoW block returns its data. @@ -52,7 +96,7 @@ def is_valid_transition_block(block: PowBlock) -> bool: return block.is_valid and is_total_difficulty_reached ``` -### Updated fork-choice handlers +## Updated fork-choice handlers #### `on_block` diff --git a/specs/merge/validator.md b/specs/merge/validator.md index dccc5727b..21fc49a36 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -12,6 +12,9 @@ - [Introduction](#introduction) - [Prerequisites](#prerequisites) +- [Protocols](#protocols) + - [`ExecutionEngine`](#executionengine) + - [`assemble_block`](#assemble_block) - [Beacon chain responsibilities](#beacon-chain-responsibilities) - [Block proposal](#block-proposal) - [Constructing the `BeaconBlockBody`](#constructing-the-beaconblockbody) @@ -32,6 +35,25 @@ This document is an extension of the [Phase 0 -- Validator](../phase0/validator. All terminology, constants, functions, and protocol mechanics defined in the updated Beacon Chain doc of [The Merge](./beacon-chain.md) are requisite for this document and used throughout. Please see related Beacon Chain doc before continuing and use them as a reference throughout. +## Protocols + +### `ExecutionEngine` + +The following methods are added to the `ExecutionEngine` protocol for use as a validator: + +#### `assemble_block` + +Produces a new instance of an execution payload, with the specified timestamp, +on top of the execution payload chain tip identified by `block_head`. + +The body of this function is implementation dependent. +The Consensus API may be used to implement this with an external execution engine. + +```python +def assemble_block(self: ExecutionEngine, block_hash: Hash32, timestamp: uint64) -> ExecutionPayload: + ... +``` + ## Beacon chain responsibilities All validator responsibilities remain unchanged other than those noted below. Namely, the transition block handling and the addition of `ExecutionPayload`. @@ -49,12 +71,12 @@ Let `get_pow_chain_head() -> PowBlock` be the function that returns the head of ###### `produce_execution_payload` Let `produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> ExecutionPayload` be the function that produces new instance of execution payload. -The body of this function is implementation dependent. +The `ExecutionEngine` protocol is used for the implementation specific part of execution payload proposals. * Set `block.body.execution_payload = get_execution_payload(state)` where: ```python -def get_execution_payload(state: BeaconState) -> ExecutionPayload: +def get_execution_payload(state: BeaconState, execution_engine: ExecutionEngine) -> ExecutionPayload: if not is_transition_completed(state): pow_block = get_pow_chain_head() if not is_valid_transition_block(pow_block): @@ -63,7 +85,7 @@ def get_execution_payload(state: BeaconState) -> ExecutionPayload: else: # Signify merge via producing on top of the last PoW block timestamp = compute_time_at_slot(state, state.slot) - return produce_execution_payload(pow_block.block_hash, timestamp) + return execution_engine.assemble_block(pow_block.block_hash, timestamp) # Post-merge, normal payload execution_parent_hash = state.latest_execution_payload_header.block_hash From cd7842557054845462e2f209751c8f61d14c0111 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 10:30:47 -0600 Subject: [PATCH 176/227] lint --- specs/altair/beacon-chain.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5c21573fb..5a4b37a83 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -362,19 +362,22 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence previous_epoch = get_previous_epoch(state) unslashed_participating_indices = get_unslashed_participating_indices(state, flag_index, previous_epoch) weight = PARTICIPATION_FLAG_WEIGHTS[flag_index] - unslashed_participating_increments = get_total_balance(state, unslashed_participating_indices) // EFFECTIVE_BALANCE_INCREMENT + unslashed_participating_balance = get_total_balance(state, unslashed_participating_indices) + unslashed_participating_increments = unslashed_participating_balance // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) - if index in unslashed_participating_indices: + index_participated = index in unslashed_participating_indices + if index_participated: if not in_inactivity_leak: reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) + # Quadratic inactivity leak - if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index not in unslashed_participating_indices: + if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index_participated: penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR penalties[index] += Gwei(penalty_numerator // penalty_denominator) From 1494fe6ace0c54231d8e7d080da7eb5134a829df Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 10:59:33 -0600 Subject: [PATCH 177/227] add get_inactivity_penalty_deltas back in --- specs/altair/beacon-chain.md | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 5a4b37a83..265bbfa53 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -37,6 +37,7 @@ - [`get_base_reward`](#get_base_reward) - [`get_unslashed_participating_indices`](#get_unslashed_participating_indices) - [`get_flag_index_deltas`](#get_flag_index_deltas) + - [Modified `get_inactivity_penalty_deltas`](#modified-get_inactivity_penalty_deltas) - [Beacon state mutators](#beacon-state-mutators) - [Modified `slash_validator`](#modified-slash_validator) - [Block processing](#block-processing) @@ -368,19 +369,32 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) - index_participated = index in unslashed_participating_indices - if index_participated: + if index in unslashed_participating_indices: if not in_inactivity_leak: reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - elif not (in_inactivity_leak and flag_index == TIMELY_HEAD_FLAG_INDEX): + else: penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) + return rewards, penalties +``` - # Quadratic inactivity leak - if flag_index == TIMELY_TARGET_FLAG_INDEX and in_inactivity_leak and index_participated: - penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] - penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR - penalties[index] += Gwei(penalty_numerator // penalty_denominator) +#### Modified `get_inactivity_penalty_deltas` + +```python +def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: + """ + Return the inactivity penalty deltas by considering timely target participation flags and inactivity scores. + """ + rewards = [Gwei(0) for _ in range(len(state.validators))] + penalties = [Gwei(0) for _ in range(len(state.validators))] + if is_in_inactivity_leak(state): + previous_epoch = get_previous_epoch(state) + matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) + for index in get_eligible_validator_indices(state): + if index not in matching_target_indices: + penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] + penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR + penalties[index] += Gwei(penalty_numerator // penalty_denominator) return rewards, penalties ``` @@ -613,7 +627,8 @@ def process_rewards_and_penalties(state: BeaconState) -> None: return flag_deltas = [get_flag_index_deltas(state, flag_index) for flag_index in range(len(PARTICIPATION_FLAG_WEIGHTS))] - for (rewards, penalties) in flag_deltas: + deltas = flag_deltas + [get_inactivity_penalty_deltas(state)] + for (rewards, penalties) in deltas: for index in range(len(state.validators)): increase_balance(state, ValidatorIndex(index), rewards[index]) decrease_balance(state, ValidatorIndex(index), penalties[index]) From ff706e5c7a8bc68422595e7d3c2c55c9ca2183db Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 12:57:11 -0600 Subject: [PATCH 178/227] add logic for handling sync committee off by one issue --- specs/altair/p2p-interface.md | 11 ++++++++++- specs/altair/validator.md | 6 +++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 85e859191..02a159f7a 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -106,9 +106,18 @@ The following validations MUST pass before forwarding the `signed_contribution_a ```python def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: + # Committees assigned to `slot` sign for `slot - 1` + # This creates the exceptional logic below when transitioning between sync comittee periods + next_slot_epoch = compute_epoch_at_slot(state.slot + 1) + if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): + sync_committee = state.current_sync_committee + else: + sync_committee = state.next_sync_committee + + # Return pubkeys for the subcommittee index sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT i = subcommittee_index * sync_subcommittee_size - return state.current_sync_committee.pubkeys[i:i + sync_subcommittee_size] + return sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` - _[IGNORE]_ The contribution's slot is for the current slot, i.e. `contribution.slot == current_slot`. diff --git a/specs/altair/validator.md b/specs/altair/validator.md index dbcc73486..3a96adb46 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -143,6 +143,10 @@ A validator determines beacon committee assignments and beacon block proposal du To determine sync committee assignments, a validator can run the following function: `is_assigned_to_sync_committee(state, epoch, validator_index)` where `epoch` is an epoch number within the current or next sync committee period. This function is a predicate indicating the presence or absence of the validator in the corresponding sync committee for the queried sync committee period. +*Note*: Being assigned to a sync committee for a given `slot` means that the validator produces and broadcasts signatures for `slot - 1` for inclusion in `slot`. +This means that when assigned to an `epoch` sync committee signatures must be produced and broadcast for slots on range `[compute_start_slot_at_epoch(epoch) - 1, compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH - 1)` +rather than for the range `[compute_start_slot_at_epoch(epoch), compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH)`. + ```python def compute_sync_committee_period(epoch: Epoch) -> uint64: return epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD @@ -261,7 +265,7 @@ This process occurs each slot. ##### Prepare sync committee signature -If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every slot in the current sync committee period, the validator should prepare a `SyncCommitteeSignature` according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of the current slot. +If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every `slot` in the current sync committee period, the validator should prepare a `SyncCommitteeSignature` for the previous slot (`slot - 1`) according to the logic in `get_sync_committee_signature` as soon as they have determined the head block of `slot - 1`. This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. From d8e2d19ecc510df32ba3e86471e8c760f14dcb68 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:01:31 -0600 Subject: [PATCH 179/227] spelling --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 02a159f7a..df2955f13 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -107,7 +107,7 @@ The following validations MUST pass before forwarding the `signed_contribution_a ```python def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: # Committees assigned to `slot` sign for `slot - 1` - # This creates the exceptional logic below when transitioning between sync comittee periods + # This creates the exceptional logic below when transitioning between sync committee periods next_slot_epoch = compute_epoch_at_slot(state.slot + 1) if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): sync_committee = state.current_sync_committee From 77d607a760865d9dd621da5b1f3e39dfe0e7161e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:24:14 -0600 Subject: [PATCH 180/227] fix test --- tests/core/pyspec/eth2spec/test/helpers/rewards.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index f81c1fc2a..7cf0206b3 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -62,13 +62,13 @@ def run_deltas(spec, state): if is_post_altair(spec): def get_source_deltas(state): - return spec.get_flag_index_deltas(state, spec.TIMELY_SOURCE_FLAG_INDEX, spec.TIMELY_SOURCE_WEIGHT) + return spec.get_flag_index_deltas(state, spec.TIMELY_SOURCE_FLAG_INDEX) def get_head_deltas(state): - return spec.get_flag_index_deltas(state, spec.TIMELY_HEAD_FLAG_INDEX, spec.TIMELY_HEAD_WEIGHT) + return spec.get_flag_index_deltas(state, spec.TIMELY_HEAD_FLAG_INDEX) def get_target_deltas(state): - return spec.get_flag_index_deltas(state, spec.TIMELY_TARGET_FLAG_INDEX, spec.TIMELY_TARGET_WEIGHT) + return spec.get_flag_index_deltas(state, spec.TIMELY_TARGET_FLAG_INDEX) yield from run_attestation_component_deltas( spec, @@ -228,7 +228,7 @@ def run_get_inactivity_penalty_deltas(spec, state): else: base_penalty = sum( base_reward * numerator // spec.WEIGHT_DENOMINATOR - for (_, numerator) in spec.get_flag_indices_and_weights() + for (_, numerator) in spec.PARTICIPATION_FLAG_WEIGHTS ) if not has_enough_for_reward(spec, state, index): From 85198fabfa15d01832484a8e5f4386bbe2964967 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:26:43 -0600 Subject: [PATCH 181/227] lint --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index df2955f13..6f250b57e 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -108,7 +108,7 @@ The following validations MUST pass before forwarding the `signed_contribution_a def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64) -> Sequence[BLSPubkey]: # Committees assigned to `slot` sign for `slot - 1` # This creates the exceptional logic below when transitioning between sync committee periods - next_slot_epoch = compute_epoch_at_slot(state.slot + 1) + next_slot_epoch = compute_epoch_at_slot(Slot(state.slot + 1)) if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): sync_committee = state.current_sync_committee else: From a6b8574962dc03f6f36b56a744f3542e97ffeed8 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 10 May 2021 13:38:45 -0600 Subject: [PATCH 182/227] test --- specs/altair/beacon-chain.md | 3 +- .../pyspec/eth2spec/test/helpers/rewards.py | 28 ++++++++++++------- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 265bbfa53..90b878cc1 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -366,11 +366,10 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence unslashed_participating_balance = get_total_balance(state, unslashed_participating_indices) unslashed_participating_increments = unslashed_participating_balance // EFFECTIVE_BALANCE_INCREMENT active_increments = get_total_active_balance(state) // EFFECTIVE_BALANCE_INCREMENT - in_inactivity_leak = is_in_inactivity_leak(state) for index in get_eligible_validator_indices(state): base_reward = get_base_reward(state, index) if index in unslashed_participating_indices: - if not in_inactivity_leak: + if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) else: diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index 7cf0206b3..eb47dac98 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -133,10 +133,17 @@ def run_attestation_component_deltas(spec, state, component_delta_fn, matching_a validator = state.validators[index] enough_for_reward = has_enough_for_reward(spec, state, index) if index in matching_indices and not validator.slashed: - if enough_for_reward: - assert rewards[index] > 0 + if is_post_altair(spec): + if not spec.is_in_inactivity_leak(state) and enough_for_reward: + assert rewards[index] > 0 + else: + assert rewards[index] == 0 else: - assert rewards[index] == 0 + if enough_for_reward: + assert rewards[index] > 0 + else: + assert rewards[index] == 0 + assert penalties[index] == 0 else: assert rewards[index] == 0 @@ -225,18 +232,19 @@ def run_get_inactivity_penalty_deltas(spec, state): if not is_post_altair(spec): cancel_base_rewards_per_epoch = spec.BASE_REWARDS_PER_EPOCH base_penalty = cancel_base_rewards_per_epoch * base_reward - spec.get_proposer_reward(state, index) - else: - base_penalty = sum( - base_reward * numerator // spec.WEIGHT_DENOMINATOR - for (_, numerator) in spec.PARTICIPATION_FLAG_WEIGHTS - ) if not has_enough_for_reward(spec, state, index): assert penalties[index] == 0 elif index in matching_attesting_indices or not has_enough_for_leak_penalty(spec, state, index): - assert penalties[index] == base_penalty + if is_post_altair(spec): + assert penalties[index] == 0 + else: + assert penalties[index] == base_penalty else: - assert penalties[index] > base_penalty + if is_post_altair(spec): + assert penalties[index] > 0 + else: + assert penalties[index] > base_penalty else: assert penalties[index] == 0 From 3c609e02eac16e060ec3934cbe1807c5a2b7e5cf Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 07:28:24 -0600 Subject: [PATCH 183/227] pr feedback --- specs/altair/validator.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 3a96adb46..92e3a75c0 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -146,6 +146,7 @@ This function is a predicate indicating the presence or absence of the validator *Note*: Being assigned to a sync committee for a given `slot` means that the validator produces and broadcasts signatures for `slot - 1` for inclusion in `slot`. This means that when assigned to an `epoch` sync committee signatures must be produced and broadcast for slots on range `[compute_start_slot_at_epoch(epoch) - 1, compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH - 1)` rather than for the range `[compute_start_slot_at_epoch(epoch), compute_start_slot_at_epoch(epoch) + SLOTS_PER_EPOCH)`. +To reduce complexity during the Altair fork, sync committees are not expected to produce signatures for `compute_epoch_at_slot(ALTAIR_FORK_EPOCH) - 1`. ```python def compute_sync_committee_period(epoch: Epoch) -> uint64: @@ -296,11 +297,14 @@ The `subnet_id` is derived from the position in the sync committee such that the ```python def compute_subnets_for_sync_committee(state: BeaconState, validator_index: ValidatorIndex) -> Sequence[uint64]: + next_slot_epoch = compute_epoch_at_slot(Slot(state.slot + 1)) + if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): + sync_committee = state.current_sync_committee + else: + sync_committee = state.next_sync_committee + target_pubkey = state.validators[validator_index].pubkey - sync_committee_indices = [ - index for index, pubkey in enumerate(state.current_sync_committee.pubkeys) - if pubkey == target_pubkey - ] + sync_committee_indices = [index for index, pubkey in enumerate(sync_committee.pubkeys) if pubkey == target_pubkey] return [ uint64(index // (SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT)) for index in sync_committee_indices From e31a2af87b33c7b2063386d527cbde5208894948 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 08:15:04 -0600 Subject: [PATCH 184/227] remove extra if --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 210b7e4b7..4c6f47590 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -634,7 +634,7 @@ def process_inactivity_updates(state: BeaconState) -> None: else: state.inactivity_scores[index] += INACTIVITY_SCORE_BIAS # Decrease the score of all validators for forgiveness when not during a leak - if not if is_in_inactivity_leak(state): + if not is_in_inactivity_leak(state): state.inactivity_scores[index] -= min(INACTIVITY_SCORE_RECOVERY_RATE, state.inactivity_scores[index]) ``` From b71aa3fb5623589868f9d11a71b664ac46e74405 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 27 Apr 2021 17:24:15 -0700 Subject: [PATCH 185/227] add `transition` spec test format --- .../test/altair/transition/__init__.py | 0 .../test/altair/transition/test_transition.py | 51 +++++++++++++ tests/core/pyspec/eth2spec/test/context.py | 37 +++++++++- tests/core/pyspec/eth2spec/test/utils.py | 48 ++++++++++++- tests/formats/transition/README.md | 72 +++++++++++++++++++ tests/generators/transition/main.py | 42 +++++++++++ tests/generators/transition/requirements.txt | 2 + 7 files changed, 250 insertions(+), 2 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/altair/transition/__init__.py create mode 100644 tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py create mode 100644 tests/formats/transition/README.md create mode 100644 tests/generators/transition/main.py create mode 100644 tests/generators/transition/requirements.txt diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/__init__.py b/tests/core/pyspec/eth2spec/test/altair/transition/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py new file mode 100644 index 000000000..a6e06b1d2 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -0,0 +1,51 @@ +from eth2spec.test.context import ( + fork_transition_test, + single_phase, + with_custom_state, + default_activation_threshold, + low_balances, +) +from eth2spec.test.helpers.constants import PHASE0, ALTAIR +from eth2spec.test.helpers.state import state_transition_and_sign_block +from eth2spec.test.helpers.block import build_empty_block_for_next_slot + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + yield "pre", state + + blocks = [] + for slot in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH): + block = build_empty_block_for_next_slot(spec, state) + state_transition_and_sign_block(spec, state, block) + blocks.append(pre_tag(block)) + + state = post_spec.upgrade_to_altair(state) + + assert state.fork.epoch == fork_epoch + assert state.fork.previous_version == post_spec.GENESIS_FORK_VERSION + assert state.fork.current_version == post_spec.ALTAIR_FORK_VERSION + + block = build_empty_block_for_next_slot(post_spec, state) + state_transition_and_sign_block(post_spec, state, block) + blocks.append(post_tag(block)) + + yield "blocks", blocks + yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR) +def test_normal_transition_with_manual_fork_epoch(state, spec, post_spec, pre_tag, post_tag): + fork_epoch = 2 + yield "fork_epoch", "meta", fork_epoch + + # run test with computed fork_epoch... + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +@with_custom_state(low_balances, default_activation_threshold) +@single_phase +def test_normal_transition_with_low_balances(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + yield "pre", state + + # run test with custom state... diff --git a/tests/core/pyspec/eth2spec/test/context.py b/tests/core/pyspec/eth2spec/test/context.py index 7a2e61c22..57071169f 100644 --- a/tests/core/pyspec/eth2spec/test/context.py +++ b/tests/core/pyspec/eth2spec/test/context.py @@ -11,7 +11,7 @@ from .helpers.constants import ( ALL_PHASES, FORKS_BEFORE_ALTAIR, FORKS_BEFORE_MERGE, ) from .helpers.genesis import create_genesis_state -from .utils import vector_test, with_meta_tags +from .utils import vector_test, with_meta_tags, build_transition_test from random import Random from typing import Any, Callable, Sequence, TypedDict, Protocol @@ -383,3 +383,38 @@ def is_post_merge(spec): with_altair_and_later = with_phases([ALTAIR]) # TODO: include Merge, but not until Merge work is rebased. with_merge_and_later = with_phases([MERGE]) + + +def fork_transition_test(pre_fork_name, post_fork_name, fork_epoch=None): + """ + A decorator to construct a "transition" test from one fork of the eth2 spec + to another. + + Decorator assumes a transition from the `pre_fork_name` fork to the + `post_fork_name` fork. The user can supply a `fork_epoch` at which the + fork occurs or they must compute one (yielding to the generator) during the test + if more custom behavior is desired. + + A test using this decorator should expect to receive as parameters: + `state`: the default state constructed for the `pre_fork_name` fork + according to the `with_state` decorator. + `fork_epoch`: the `fork_epoch` provided to this decorator, if given. + `spec`: the version of the eth2 spec corresponding to `pre_fork_name`. + `post_spec`: the version of the eth2 spec corresponding to `post_fork_name`. + `pre_tag`: a function to tag data as belonging to `pre_fork_name` fork. + Used to discriminate data during consumption of the generated spec tests. + `post_tag`: a function to tag data as belonging to `post_fork_name` fork. + Used to discriminate data during consumption of the generated spec tests. + """ + def _wrapper(fn): + @with_phases([pre_fork_name], other_phases=[post_fork_name]) + @spec_test + @with_state + def _adapter(*args, **kwargs): + wrapped = build_transition_test(fn, + pre_fork_name, + post_fork_name, + fork_epoch=fork_epoch) + return wrapped(*args, **kwargs) + return _adapter + return _wrapper diff --git a/tests/core/pyspec/eth2spec/test/utils.py b/tests/core/pyspec/eth2spec/test/utils.py index bad6c867b..d94aeb5aa 100644 --- a/tests/core/pyspec/eth2spec/test/utils.py +++ b/tests/core/pyspec/eth2spec/test/utils.py @@ -1,5 +1,6 @@ +import inspect from typing import Dict, Any -from eth2spec.utils.ssz.ssz_typing import View +from eth2spec.utils.ssz.ssz_typing import View, boolean, Container from eth2spec.utils.ssz.ssz_impl import serialize @@ -93,3 +94,48 @@ def with_meta_tags(tags: Dict[str, Any]): yield k, 'meta', v return entry return runner + + +class FlaggedContainer(Container): + flag: boolean + obj: Container + + +def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): + """ + Handles the inner plumbing to generate `transition_test`s. + See that decorator in `context.py` for more information. + """ + def _adapter(*args, **kwargs): + post_spec = kwargs["phases"][post_fork_name] + + def pre_tag(obj): + return FlaggedContainer(flag=False, obj=obj) + + def post_tag(obj): + return FlaggedContainer(flag=True, obj=obj) + + yield "post_fork", "meta", post_fork_name + + has_fork_epoch = False + if fork_epoch: + kwargs["fork_epoch"] = fork_epoch + has_fork_epoch = True + yield "fork_epoch", "meta", fork_epoch + + # massage args to handle an optional custom state using + # `with_custom_state` decorator + expected_args = inspect.getfullargspec(fn) + if "phases" not in expected_args.kwonlyargs: + kwargs.pop("phases", None) + + for part in fn(*args, + post_spec=post_spec, + pre_tag=pre_tag, + post_tag=post_tag, + **kwargs): + if part[0] == "fork_epoch": + has_fork_epoch = True + yield part + assert has_fork_epoch + return _adapter diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md new file mode 100644 index 000000000..bd5985e50 --- /dev/null +++ b/tests/formats/transition/README.md @@ -0,0 +1,72 @@ +# Transition testing + +Transition tests to cover processing the chain across a fork boundary. + +Each test case contains a `post_fork` key in the `meta.yaml` that indicates the target fork which also fixes the fork the test begins in. + +Clients should assume forks happen sequentially in the following manner: + +0. `phase0` +1. `altair` + +For example, if a test case has `post_fork` of `altair`, the test consumer should assume the test begins in `phase0` and use that specification to process the initial state and any blocks up until the fork epoch. After the fork happens, the test consumer should use the specification according to the `altair` fork to process the remaining data. + +## Encoding notes + +This test type contains objects that span fork boundaries. +In general, it may not be clear which objects belong to which fork so each +object is prefixed with a SSZ `boolean` to indicate if the object belongs to the post fork or if it belongs to the initial fork. +This "flagged" data should be used to select the appropriate version of the spec when interpreting the enclosed object. + +```python +class FlaggedContainer(Container): + flag: boolean + obj: Container +``` + +If `flag` is `False`, then the `obj` belongs to the **initial** fork. +If `flag` is `True`, then the `obj` belongs to the **post** fork. + +Unless stated otherwise, all references to spec types below refer to SSZ-snappy +encoded data `obj` with the relevant `flag` set: +`FlaggedContainer(flag=flag, obj=obj)`. + +For example, when testing the fork from Phase 0 to Altair, an Altair block is given +as the encoding of `FlaggedContainer(flag=True, obj=SignedBeaconBlock())` where +`SignedBeaconBlock` is the type defined in the Altair spec. + +## Test case format + +### `meta.yaml` + +```yaml +post_fork: string -- String name of the spec after the fork. +fork_epoch: int -- The epoch at which the fork takes place. +blocks_count: int -- The number of blocks processed in this test. +``` + +*Note*: There may be a fork transition function to run at the `fork_epoch`. Refer to the specs for the relevant fork for further details. + +### `pre.ssz_snappy` + +A SSZ-snappy encoded `BeaconState` according to the specification of the initial fork, the state before running the block transitions. + +*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. + +### `blocks_.ssz_snappy` + +A series of files, with `` in range `[0, blocks_count)`. +Blocks must be processed in order, following the main transition function +(i.e. process slot and epoch transitions in between blocks as normal). + +Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version indicated by flag data as described in the `Encoding notes`. + +### `post.ssz_snappy` + +A SSZ-snappy encoded `BeaconState` according to the specification of the post fork, the state after running the block transitions. + +*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. + +## Condition + +The resulting state should match the expected `post` state. diff --git a/tests/generators/transition/main.py b/tests/generators/transition/main.py new file mode 100644 index 000000000..b7fd7b0a8 --- /dev/null +++ b/tests/generators/transition/main.py @@ -0,0 +1,42 @@ +from importlib import reload +from typing import Iterable + +from eth2spec.test.helpers.constants import ALTAIR, MINIMAL, MAINNET, PHASE0 +from eth2spec.config import config_util +from eth2spec.test.altair.transition import test_transition as test_altair_transition +from eth2spec.phase0 import spec as spec_phase0 +from eth2spec.altair import spec as spec_altair + +from eth2spec.gen_helpers.gen_base import gen_runner, gen_typing +from eth2spec.gen_helpers.gen_from_tests.gen import generate_from_tests + + +def create_provider(tests_src, config_name: str, pre_fork_name: str, post_fork_name: str) -> gen_typing.TestProvider: + + def prepare_fn(configs_path: str) -> str: + config_util.prepare_config(configs_path, config_name) + reload(spec_phase0) + reload(spec_altair) + return config_name + + def cases_fn() -> Iterable[gen_typing.TestCase]: + return generate_from_tests( + runner_name='transition', + handler_name='core', + src=tests_src, + fork_name=post_fork_name, + phase=pre_fork_name, + ) + + return gen_typing.TestProvider(prepare=prepare_fn, make_cases=cases_fn) + + +TRANSITION_TESTS = ((PHASE0, ALTAIR, test_altair_transition),) + + +if __name__ == "__main__": + for pre_fork, post_fork, transition_test_module in TRANSITION_TESTS: + gen_runner.run_generator("transition", [ + create_provider(transition_test_module, MINIMAL, pre_fork, post_fork), + create_provider(transition_test_module, MAINNET, pre_fork, post_fork), + ]) diff --git a/tests/generators/transition/requirements.txt b/tests/generators/transition/requirements.txt new file mode 100644 index 000000000..735f863fa --- /dev/null +++ b/tests/generators/transition/requirements.txt @@ -0,0 +1,2 @@ +pytest>=4.4 +../../../[generator] \ No newline at end of file From 0cc6e15b44f075f9258e4bf9524517e0fd4471c1 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 09:57:03 -0700 Subject: [PATCH 186/227] Update tests/formats/transition/README.md Co-authored-by: Adrian Sutton --- tests/formats/transition/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md index bd5985e50..fb8b3b5ea 100644 --- a/tests/formats/transition/README.md +++ b/tests/formats/transition/README.md @@ -51,7 +51,7 @@ blocks_count: int -- The number of blocks processed in this test. A SSZ-snappy encoded `BeaconState` according to the specification of the initial fork, the state before running the block transitions. -*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. +*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the pre fork. ### `blocks_.ssz_snappy` From d34b2a08d5b19e499190673d6385d47802ce3f17 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 11:35:18 -0700 Subject: [PATCH 187/227] Use `fork_block` index in lieu of fork flag --- tests/core/pyspec/eth2spec/test/utils.py | 18 ++++--- tests/formats/transition/README.md | 67 ++++++++++++------------ 2 files changed, 44 insertions(+), 41 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/utils.py b/tests/core/pyspec/eth2spec/test/utils.py index d94aeb5aa..61fc75040 100644 --- a/tests/core/pyspec/eth2spec/test/utils.py +++ b/tests/core/pyspec/eth2spec/test/utils.py @@ -1,6 +1,6 @@ import inspect from typing import Dict, Any -from eth2spec.utils.ssz.ssz_typing import View, boolean, Container +from eth2spec.utils.ssz.ssz_typing import View from eth2spec.utils.ssz.ssz_impl import serialize @@ -96,11 +96,6 @@ def with_meta_tags(tags: Dict[str, Any]): return runner -class FlaggedContainer(Container): - flag: boolean - obj: Container - - def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): """ Handles the inner plumbing to generate `transition_test`s. @@ -109,11 +104,15 @@ def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): def _adapter(*args, **kwargs): post_spec = kwargs["phases"][post_fork_name] + pre_fork_counter = 0 + def pre_tag(obj): - return FlaggedContainer(flag=False, obj=obj) + nonlocal pre_fork_counter + pre_fork_counter += 1 + return obj def post_tag(obj): - return FlaggedContainer(flag=True, obj=obj) + return obj yield "post_fork", "meta", post_fork_name @@ -138,4 +137,7 @@ def build_transition_test(fn, pre_fork_name, post_fork_name, fork_epoch=None): has_fork_epoch = True yield part assert has_fork_epoch + + if pre_fork_counter > 0: + yield "fork_block", "meta", pre_fork_counter - 1 return _adapter diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md index fb8b3b5ea..832f38ca2 100644 --- a/tests/formats/transition/README.md +++ b/tests/formats/transition/README.md @@ -11,30 +11,6 @@ Clients should assume forks happen sequentially in the following manner: For example, if a test case has `post_fork` of `altair`, the test consumer should assume the test begins in `phase0` and use that specification to process the initial state and any blocks up until the fork epoch. After the fork happens, the test consumer should use the specification according to the `altair` fork to process the remaining data. -## Encoding notes - -This test type contains objects that span fork boundaries. -In general, it may not be clear which objects belong to which fork so each -object is prefixed with a SSZ `boolean` to indicate if the object belongs to the post fork or if it belongs to the initial fork. -This "flagged" data should be used to select the appropriate version of the spec when interpreting the enclosed object. - -```python -class FlaggedContainer(Container): - flag: boolean - obj: Container -``` - -If `flag` is `False`, then the `obj` belongs to the **initial** fork. -If `flag` is `True`, then the `obj` belongs to the **post** fork. - -Unless stated otherwise, all references to spec types below refer to SSZ-snappy -encoded data `obj` with the relevant `flag` set: -`FlaggedContainer(flag=flag, obj=obj)`. - -For example, when testing the fork from Phase 0 to Altair, an Altair block is given -as the encoding of `FlaggedContainer(flag=True, obj=SignedBeaconBlock())` where -`SignedBeaconBlock` is the type defined in the Altair spec. - ## Test case format ### `meta.yaml` @@ -42,16 +18,17 @@ as the encoding of `FlaggedContainer(flag=True, obj=SignedBeaconBlock())` where ```yaml post_fork: string -- String name of the spec after the fork. fork_epoch: int -- The epoch at which the fork takes place. -blocks_count: int -- The number of blocks processed in this test. +fork_block: int -- Optional. The `` of the last block on the initial fork. +blocks_count: int -- Optional. The number of blocks processed in this test. ``` -*Note*: There may be a fork transition function to run at the `fork_epoch`. Refer to the specs for the relevant fork for further details. +*Note*: There may be a fork transition function to run at the `fork_epoch`. +Refer to the specs for the relevant fork for further details. ### `pre.ssz_snappy` -A SSZ-snappy encoded `BeaconState` according to the specification of the initial fork, the state before running the block transitions. - -*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the pre fork. +A SSZ-snappy encoded `BeaconState` according to the specification of +the initial fork, the state before running the block transitions. ### `blocks_.ssz_snappy` @@ -59,13 +36,37 @@ A series of files, with `` in range `[0, blocks_count)`. Blocks must be processed in order, following the main transition function (i.e. process slot and epoch transitions in between blocks as normal). -Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version indicated by flag data as described in the `Encoding notes`. +*Note*: `blocks_count` will be missing if there are no blocks in this test. + +Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version +as indicated by the `post_fork` and `fork_block` data in the `meta.yaml`. + +As blocks span fork boundaires, a `fork_block` number is given in +the `meta.yaml` to help resolve which blocks belong to which fork. + +The `fork_block` is the index in the test data of the **last** block +of the **initial** fork. + +To demonstrate, the following diagram shows slots with `_` and blocks +in those slots as `x`. The fork happens at the epoch delineated by the `|`. + +``` +x x x x +_ _ _ _ | _ _ _ _ +``` + +The `blocks_count` value in the `meta.yaml` in this case is `4` where the +`fork_block` value in the `meta.yaml` is `1`. If this particular example were +testing the fork from Phase 0 to Altair, blocks with indices `0, 1` represent +`SignedBeaconBlock`s defined in the Phase 0 spec and blocks with indices `2, 3` +represent `SignedBeaconBlock`s defined in the Altair spec. + +*Note*: `fork_block` will be missing if `blocks_count` is also missing. ### `post.ssz_snappy` -A SSZ-snappy encoded `BeaconState` according to the specification of the post fork, the state after running the block transitions. - -*NOTE*: This object is _not_ "flagged" as it is assumed to always belong to the post fork. +A SSZ-snappy encoded `BeaconState` according to the specification of +the post fork, the state after running the block transitions. ## Condition From 0e71496eb5e54ae60e06514853d74d01e53ffc65 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 11:46:46 -0700 Subject: [PATCH 188/227] add "normal" transition test --- .../test/altair/transition/test_transition.py | 77 +++++++++++-------- 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index a6e06b1d2..b9b67d55a 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -1,24 +1,44 @@ -from eth2spec.test.context import ( - fork_transition_test, - single_phase, - with_custom_state, - default_activation_threshold, - low_balances, -) +from eth2spec.test.context import fork_transition_test from eth2spec.test.helpers.constants import PHASE0, ALTAIR from eth2spec.test.helpers.state import state_transition_and_sign_block -from eth2spec.test.helpers.block import build_empty_block_for_next_slot +from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_empty_block, sign_block + + +def _state_transition_and_sign_block_at_slot(spec, state): + """ + Cribbed from `transition_unsigned_block` helper + where the early parts of the state transition have already + been applied to `state`. + + Used to produce a block during an irregular state transition. + """ + block = build_empty_block(spec, state) + + assert state.latest_block_header.slot < block.slot + assert state.slot == block.slot + spec.process_block(state, block) + block.state_root = state.hash_tree_root() + return sign_block(spec, state, block) @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): yield "pre", state + assert spec.get_current_epoch(state) < fork_epoch + blocks = [] - for slot in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH): + # regular state transition until fork: + for _ in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH - 1): block = build_empty_block_for_next_slot(spec, state) - state_transition_and_sign_block(spec, state, block) - blocks.append(pre_tag(block)) + signed_block = state_transition_and_sign_block(spec, state, block) + blocks.append(pre_tag(signed_block)) + + # irregular state transition to handle fork: + spec.process_slots(state, state.slot + 1) + + assert state.slot % spec.SLOTS_PER_EPOCH == 0 + assert spec.compute_epoch_at_slot(state.slot) == fork_epoch state = post_spec.upgrade_to_altair(state) @@ -26,26 +46,21 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag assert state.fork.previous_version == post_spec.GENESIS_FORK_VERSION assert state.fork.current_version == post_spec.ALTAIR_FORK_VERSION - block = build_empty_block_for_next_slot(post_spec, state) - state_transition_and_sign_block(post_spec, state, block) - blocks.append(post_tag(block)) + signed_block = _state_transition_and_sign_block_at_slot(post_spec, state) + blocks.append(post_tag(signed_block)) + + # continue regular state transition with new spec into next epoch + for _ in range(post_spec.SLOTS_PER_EPOCH): + block = build_empty_block_for_next_slot(post_spec, state) + signed_block = state_transition_and_sign_block(post_spec, state, block) + blocks.append(post_tag(signed_block)) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(set(slots_with_blocks)) == len(slots_with_blocks) + assert set(range(1, state.slot + 1)) == set(slots_with_blocks) yield "blocks", blocks yield "post", state - - -@fork_transition_test(PHASE0, ALTAIR) -def test_normal_transition_with_manual_fork_epoch(state, spec, post_spec, pre_tag, post_tag): - fork_epoch = 2 - yield "fork_epoch", "meta", fork_epoch - - # run test with computed fork_epoch... - - -@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) -@with_custom_state(low_balances, default_activation_threshold) -@single_phase -def test_normal_transition_with_low_balances(state, fork_epoch, spec, post_spec, pre_tag, post_tag): - yield "pre", state - - # run test with custom state... From 3f3aa4fb105298d4126980bc235ec8ef7ad674e4 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 16:21:46 -0700 Subject: [PATCH 189/227] add some altair tests --- .../test/altair/transition/test_transition.py | 158 ++++++++++++++++-- 1 file changed, 140 insertions(+), 18 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index b9b67d55a..b522c1f94 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -1,6 +1,6 @@ from eth2spec.test.context import fork_transition_test from eth2spec.test.helpers.constants import PHASE0, ALTAIR -from eth2spec.test.helpers.state import state_transition_and_sign_block +from eth2spec.test.helpers.state import state_transition_and_sign_block, next_slot from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_empty_block, sign_block @@ -21,20 +21,32 @@ def _state_transition_and_sign_block_at_slot(spec, state): return sign_block(spec, state, block) -@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) -def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): - yield "pre", state +def _all_blocks(_): + return True - assert spec.get_current_epoch(state) < fork_epoch - blocks = [] - # regular state transition until fork: - for _ in range(state.slot, fork_epoch * spec.SLOTS_PER_EPOCH - 1): - block = build_empty_block_for_next_slot(spec, state) - signed_block = state_transition_and_sign_block(spec, state, block) - blocks.append(pre_tag(signed_block)) +def _skip_slots(*slots): + """ + Skip making a block if its slot is + passed as an argument to this filter + """ + def f(state_at_prior_slot): + return state_at_prior_slot.slot + 1 not in slots + return f - # irregular state transition to handle fork: + +def _state_transition_across_slots(spec, state, slot_count, block_filter=_all_blocks): + for _ in range(slot_count): + should_make_block = block_filter(state) + if should_make_block: + block = build_empty_block_for_next_slot(spec, state) + signed_block = state_transition_and_sign_block(spec, state, block) + yield signed_block + else: + next_slot(spec, state) + + +def _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=True): spec.process_slots(state, state.slot + 1) assert state.slot % spec.SLOTS_PER_EPOCH == 0 @@ -46,14 +58,40 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag assert state.fork.previous_version == post_spec.GENESIS_FORK_VERSION assert state.fork.current_version == post_spec.ALTAIR_FORK_VERSION - signed_block = _state_transition_and_sign_block_at_slot(post_spec, state) - blocks.append(post_tag(signed_block)) + if with_block: + return state, _state_transition_and_sign_block_at_slot(post_spec, state) + else: + return state, None + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count) + ]) + + # irregular state transition to handle fork: + state, block = _do_altair_fork(state, spec, post_spec, fork_epoch) + blocks.append(post_tag(block)) # continue regular state transition with new spec into next epoch - for _ in range(post_spec.SLOTS_PER_EPOCH): - block = build_empty_block_for_next_slot(post_spec, state) - signed_block = state_transition_and_sign_block(post_spec, state, block) - blocks.append(post_tag(signed_block)) + slot_count = post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count) + ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 @@ -64,3 +102,87 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag yield "blocks", blocks yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way except for the first block + of the new fork. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count) + ]) + + # irregular state transition to handle fork: + state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) + + # continue regular state transition with new spec into next epoch + slot_count = post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count) + ]) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(set(slots_with_blocks)) == len(slots_with_blocks) + expected_slots = set(range(1, state.slot + 1)).difference(set([fork_epoch * spec.SLOTS_PER_EPOCH])) + assert expected_slots == set(slots_with_blocks) + + yield "blocks", blocks + yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way except for the first block + of the new fork. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 + slot_count = last_slot_of_pre_fork - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count, block_filter=_skip_slots(last_slot_of_pre_fork)) + ]) + + # irregular state transition to handle fork: + state, block = _do_altair_fork(state, spec, post_spec, fork_epoch) + blocks.append(post_tag(block)) + + # continue regular state transition with new spec into next epoch + slot_count = post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count) + ]) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(set(slots_with_blocks)) == len(slots_with_blocks) + expected_slots = set(range(1, state.slot + 1)).difference(set([last_slot_of_pre_fork])) + assert expected_slots == set(slots_with_blocks) + + yield "blocks", blocks + yield "post", state From c08fb7714c4ba2e2a5d3c65ba84936493e8a88ce Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 30 Apr 2021 16:34:19 -0700 Subject: [PATCH 190/227] More altair fork tests with varied block conditions --- .../test/altair/transition/test_transition.py | 55 +++++++++++++++++++ tests/formats/transition/README.md | 7 +-- 2 files changed, 58 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index b522c1f94..401a781e1 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -35,6 +35,19 @@ def _skip_slots(*slots): return f +def _no_blocks(_): + return False + + +def _only_at(slot): + """ + Only produce a block if its slot is `slot`. + """ + def f(state_at_prior_slot): + return state_at_prior_slot.slot + 1 == slot + return f + + def _state_transition_across_slots(spec, state, slot_count, block_filter=_all_blocks): for _ in range(slot_count): should_make_block = block_filter(state) @@ -186,3 +199,45 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t yield "blocks", blocks yield "post", state + + +@fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) +def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pre_tag, post_tag): + """ + Transition from the initial `state` to the epoch after the `fork_epoch`, + producing blocks for every slot along the way except for the first block + of the new fork. + """ + yield "pre", state + + assert spec.get_current_epoch(state) < fork_epoch + + # regular state transition until fork: + last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 + slot_count = last_slot_of_pre_fork - state.slot + blocks = [] + blocks.extend([ + pre_tag(block) for block in + _state_transition_across_slots(spec, state, slot_count, block_filter=_no_blocks) + ]) + + # irregular state transition to handle fork: + state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) + + # continue regular state transition with new spec into next epoch + slot_count = post_spec.SLOTS_PER_EPOCH + last_slot = (fork_epoch + 1) * post_spec.SLOTS_PER_EPOCH + blocks.extend([ + post_tag(block) for block in + _state_transition_across_slots(post_spec, state, slot_count, block_filter=_only_at(last_slot)) + ]) + + assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 + assert post_spec.compute_epoch_at_slot(state.slot) == fork_epoch + 1 + + slots_with_blocks = [block.message.slot for block in blocks] + assert len(slots_with_blocks) == 1 + assert slots_with_blocks[0] == last_slot + + yield "blocks", blocks + yield "post", state diff --git a/tests/formats/transition/README.md b/tests/formats/transition/README.md index 832f38ca2..37df65539 100644 --- a/tests/formats/transition/README.md +++ b/tests/formats/transition/README.md @@ -19,7 +19,7 @@ For example, if a test case has `post_fork` of `altair`, the test consumer shoul post_fork: string -- String name of the spec after the fork. fork_epoch: int -- The epoch at which the fork takes place. fork_block: int -- Optional. The `` of the last block on the initial fork. -blocks_count: int -- Optional. The number of blocks processed in this test. +blocks_count: int -- The number of blocks processed in this test. ``` *Note*: There may be a fork transition function to run at the `fork_epoch`. @@ -36,8 +36,6 @@ A series of files, with `` in range `[0, blocks_count)`. Blocks must be processed in order, following the main transition function (i.e. process slot and epoch transitions in between blocks as normal). -*Note*: `blocks_count` will be missing if there are no blocks in this test. - Blocks are encoded as `SignedBeaconBlock`s from the relevant spec version as indicated by the `post_fork` and `fork_block` data in the `meta.yaml`. @@ -61,7 +59,8 @@ testing the fork from Phase 0 to Altair, blocks with indices `0, 1` represent `SignedBeaconBlock`s defined in the Phase 0 spec and blocks with indices `2, 3` represent `SignedBeaconBlock`s defined in the Altair spec. -*Note*: `fork_block` will be missing if `blocks_count` is also missing. +*Note*: If `fork_block` is missing, then all block data should be +interpreted as belonging to the post fork. ### `post.ssz_snappy` From d7448255833fbd0880501cbe070d5c2b43d48ca1 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 4 May 2021 14:36:58 -0700 Subject: [PATCH 191/227] update docs --- .../eth2spec/test/altair/transition/test_transition.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index 401a781e1..48fdd64c8 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -162,8 +162,8 @@ def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ Transition from the initial `state` to the epoch after the `fork_epoch`, - producing blocks for every slot along the way except for the first block - of the new fork. + producing blocks for every slot along the way except for the last block + of the old fork. """ yield "pre", state @@ -205,8 +205,8 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ Transition from the initial `state` to the epoch after the `fork_epoch`, - producing blocks for every slot along the way except for the first block - of the new fork. + skipping blocks for every slot along the way except for the first block + in the ending epoch. """ yield "pre", state From e2aa595d5fc5f241ea271f8260a8d433bb7a11a9 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 11 May 2021 10:06:18 -0700 Subject: [PATCH 192/227] PR feedback --- .../test/altair/transition/test_transition.py | 53 ++++++++++--------- 1 file changed, 27 insertions(+), 26 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index 48fdd64c8..c3b03d663 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -6,9 +6,9 @@ from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_e def _state_transition_and_sign_block_at_slot(spec, state): """ - Cribbed from `transition_unsigned_block` helper + Cribbed from ``transition_unsigned_block`` helper where the early parts of the state transition have already - been applied to `state`. + been applied to ``state``. Used to produce a block during an irregular state transition. """ @@ -41,15 +41,16 @@ def _no_blocks(_): def _only_at(slot): """ - Only produce a block if its slot is `slot`. + Only produce a block if its slot is ``slot``. """ def f(state_at_prior_slot): return state_at_prior_slot.slot + 1 == slot return f -def _state_transition_across_slots(spec, state, slot_count, block_filter=_all_blocks): - for _ in range(slot_count): +def _state_transition_across_slots(spec, state, to_slot, block_filter=_all_blocks): + assert state.slot < to_slot + while state.slot < to_slot: should_make_block = block_filter(state) if should_make_block: block = build_empty_block_for_next_slot(spec, state) @@ -80,7 +81,7 @@ def _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=True): @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, producing blocks for every slot along the way. """ yield "pre", state @@ -88,11 +89,11 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag assert spec.get_current_epoch(state) < fork_epoch # regular state transition until fork: - slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + to_slot = fork_epoch * spec.SLOTS_PER_EPOCH - 1 blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count) + _state_transition_across_slots(spec, state, to_slot) ]) # irregular state transition to handle fork: @@ -100,10 +101,10 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag blocks.append(post_tag(block)) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count) + _state_transition_across_slots(post_spec, state, to_slot) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 @@ -120,7 +121,7 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, producing blocks for every slot along the way except for the first block of the new fork. """ @@ -129,21 +130,21 @@ def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, assert spec.get_current_epoch(state) < fork_epoch # regular state transition until fork: - slot_count = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - state.slot + to_slot = fork_epoch * spec.SLOTS_PER_EPOCH - 1 blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count) + _state_transition_across_slots(spec, state, to_slot) ]) # irregular state transition to handle fork: state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count) + _state_transition_across_slots(post_spec, state, to_slot) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 @@ -159,9 +160,9 @@ def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) -def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): +def test_transition_missing_last_pre_fork_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, producing blocks for every slot along the way except for the last block of the old fork. """ @@ -171,11 +172,11 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t # regular state transition until fork: last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - slot_count = last_slot_of_pre_fork - state.slot + to_slot = last_slot_of_pre_fork blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count, block_filter=_skip_slots(last_slot_of_pre_fork)) + _state_transition_across_slots(spec, state, to_slot, block_filter=_skip_slots(last_slot_of_pre_fork)) ]) # irregular state transition to handle fork: @@ -183,10 +184,10 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t blocks.append(post_tag(block)) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count) + _state_transition_across_slots(post_spec, state, to_slot) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 @@ -204,7 +205,7 @@ def test_transition_missing_fork_block(state, fork_epoch, spec, post_spec, pre_t @fork_transition_test(PHASE0, ALTAIR, fork_epoch=2) def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pre_tag, post_tag): """ - Transition from the initial `state` to the epoch after the `fork_epoch`, + Transition from the initial ``state`` to the epoch after the ``fork_epoch``, skipping blocks for every slot along the way except for the first block in the ending epoch. """ @@ -214,22 +215,22 @@ def test_transition_only_blocks_post_fork(state, fork_epoch, spec, post_spec, pr # regular state transition until fork: last_slot_of_pre_fork = fork_epoch * spec.SLOTS_PER_EPOCH - 1 - slot_count = last_slot_of_pre_fork - state.slot + to_slot = last_slot_of_pre_fork blocks = [] blocks.extend([ pre_tag(block) for block in - _state_transition_across_slots(spec, state, slot_count, block_filter=_no_blocks) + _state_transition_across_slots(spec, state, to_slot, block_filter=_no_blocks) ]) # irregular state transition to handle fork: state, _ = _do_altair_fork(state, spec, post_spec, fork_epoch, with_block=False) # continue regular state transition with new spec into next epoch - slot_count = post_spec.SLOTS_PER_EPOCH + to_slot = post_spec.SLOTS_PER_EPOCH + state.slot last_slot = (fork_epoch + 1) * post_spec.SLOTS_PER_EPOCH blocks.extend([ post_tag(block) for block in - _state_transition_across_slots(post_spec, state, slot_count, block_filter=_only_at(last_slot)) + _state_transition_across_slots(post_spec, state, to_slot, block_filter=_only_at(last_slot)) ]) assert state.slot % post_spec.SLOTS_PER_EPOCH == 0 From 554f5141d375dd5e14a89f20121f7d527b1c5233 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Tue, 11 May 2021 20:16:58 +0300 Subject: [PATCH 193/227] Replace the {shard}_{slot} with {subnet_id}. Add {subnet_id} calculation function --- specs/sharding/p2p-interface.md | 36 +++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index f0e33dcf8..47ed52970 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -9,13 +9,15 @@ - [Introduction](#introduction) +- [Constants](#constants) + - [Misc](#misc) - [New containers](#new-containers) - [ShardBlobBody](#shardblobbody) - [ShardBlob](#shardblob) - [SignedShardBlob](#signedshardblob) - [Gossip domain](#gossip-domain) - [Topics and messages](#topics-and-messages) - - [Shard blobs: `shard_blob_{shard}_{slot}`](#shard-blobs-shard_blob_shard_slot) + - [Shard blobs: `shard_blob_{subnet_id}`](#shard-blobs-shard_blob_subnet_id) - [Shard header: `shard_header`](#shard-header-shard_header) - [Shard proposer slashing: `shard_proposer_slashing`](#shard-proposer-slashing-shard_proposer_slashing) @@ -29,6 +31,14 @@ The specification of these changes continues in the same format as the [Phase0]( [Altair](../altair/p2p-interface.md) network specifications, and assumes them as pre-requisite. The adjustments and additions for Shards are outlined in this document. +## Constants + +### Misc + +| Name | Value | Description | +| ---- | ----- | ----------- | +| `SHARD_BLOB_SUBNET_COUNT` | `64` | The number of `shard_blob_{subnet_id}` subnets used in the gossipsub protocol. | + ## New containers ### ShardBlobBody @@ -77,24 +87,38 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| -| `shard_blob_{shard}_{slot}` | `SignedShardBlob` | +| `shard_blob_{subnet_id}` | `SignedShardBlob` | | `shard_header` | `SignedShardHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. -#### Shard blobs: `shard_blob_{shard}_{slot}` +#### Shard blobs: `shard_blob_{subnet_id}` -Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{shard}_{slot}` subnets. +Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{subnet_id}` subnets. + +```python +def compute_subnet_for_shard_blob(state: BeaconState, slot: Slot, shard: Shard) -> uint64: + """ + Compute the correct subnet for a shard blob publication. + Note, this mimics compute_subnet_for_attestation(). + """ + committee_index = compute_committee_index_from_shard(state, slot, shard) + committees_per_slot = get_committee_count_per_slot(state, compute_epoch_at_slot(slot)) + slots_since_epoch_start = Slot(slot % SLOTS_PER_EPOCH) + committees_since_epoch_start = committees_per_slot * slots_since_epoch_start + + return uint64((committees_since_epoch_start + committee_index) % SHARD_BLOB_SUBNET_COUNT) +``` The following validations MUST pass before forwarding the `signed_blob` (with inner `message` as `blob`) on the horizontal subnet or creating samples for it. -- _[REJECT]_ `blob.shard` MUST match the topic `{shard}` parameter. (And thus within valid shard index range) -- _[REJECT]_ `blob.slot - compute_start_slot_at_epoch(blob.slot)` MUST match the topic `{slot}` parameter. - _[IGNORE]_ The `blob` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that `blob.slot <= current_slot` (a client MAY queue future blobs for processing at the appropriate slot). - _[IGNORE]_ The `blob` is new enough to be still be processed -- i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)` +- _[REJECT]_ The shard blob is for the correct subnet -- + i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id` - _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination. - _[REJECT]_ As already limited by the SSZ list-limit, it is important the blob is well-formatted and not too large. - _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid. From 488ceed4f9de63898d8ebc490d6b044b5a27fec4 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 11:29:37 -0600 Subject: [PATCH 194/227] add notes about repeatedly failing tos erve blocks as being disconncetable --- specs/phase0/p2p-interface.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index a0bd2d4e2..f2c7d8610 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -752,18 +752,20 @@ Clients MUST keep a record of signed blocks seen on the epoch range where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blocks on this range. -Synced clients unable to reply to Block requests within the +Peers are *repeatedly* unable to reply to Block requests within the `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epoch range MAY get descored or disconnected at any time. -Note, due to this it is risky behaviour to begin participating as a full node at the head if having -not yet backfilled on this range. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint MUST backfill the local block database to at least epoch `current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS` -to be compliant with `BlocksByRange` requests. To safely perform such a +to be fully compliant with `BlocksByRange` requests. To safely perform such a backfill of blocks to the recent state, the node MUST validate both (1) the proposer signatures and (2) that the blocks form a valid chain up to the most recent block referenced in the weak subjectivity state. +*Note*: Although clients that bootstrap from a weak subjectivity checkpoint can begin +participating in the networking immediately, other peers that are actively block syncing MAY +disconnect and/or temporarily ban such an un-synced or semi-synced client. + Clients MUST respond with at least the first block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOCKS` blocks. From a9cc036184dedde472e22a3ae9ba45db7a094695 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 11:48:26 -0600 Subject: [PATCH 195/227] remove timely_head penalty --- specs/altair/beacon-chain.md | 2 +- tests/core/pyspec/eth2spec/test/helpers/rewards.py | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 90b878cc1..57fec33df 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -372,7 +372,7 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - else: + elif not flag_index == TIMELY_HEAD_FLAG_INDEX: penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) return rewards, penalties ``` diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index eb47dac98..f563c70a2 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -147,7 +147,9 @@ def run_attestation_component_deltas(spec, state, component_delta_fn, matching_a assert penalties[index] == 0 else: assert rewards[index] == 0 - if enough_for_reward: + if is_post_altair(spec) and 'head' in deltas_name: + assert penalties[index] == 0 + elif enough_for_reward: assert penalties[index] > 0 else: assert penalties[index] == 0 From f328f77e653ad95b552e59517fa7990106044677 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 11 May 2021 10:58:45 -0700 Subject: [PATCH 196/227] clarify fork upgrade conditions --- specs/altair/fork.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..d789e7a8e 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -38,7 +38,11 @@ Note that for the pure Altair networks, we don't apply `upgrade_to_altair` since ### Upgrading the state -After `process_slots` of Phase 0 finishes, if `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. +If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. + +The upgrade occurs after the inner loop of `process_slots` that sets `state.slot` equal to `ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH` finishes. +Care must be taken when transitioning through the fork boundary as implementations will need a modified state transition function that deviates from the Phase 0 spec. +In particular, the `state_transition` function defined in the Phase 0 spec will not expose the correct time to execute the upgrade in the presence of skipped slots at the fork boundary. ```python def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: From 859a7d743e48b2ac24b43f084af847801fc4e1ce Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 11 May 2021 11:18:03 -0700 Subject: [PATCH 197/227] Only allow sync committee period calculation at period boundaries --- specs/altair/beacon-chain.md | 12 +++--- specs/altair/fork.md | 6 ++- .../test_process_sync_committee.py | 41 ------------------- .../test_process_sync_committee_updates.py | 3 +- .../test/altair/sanity/test_blocks.py | 3 +- .../altair/unittests/test_sync_protocol.py | 9 ++-- 6 files changed, 18 insertions(+), 56 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 34a33469e..fac8b3064 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -276,16 +276,14 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ Return the sequence of sync committee indices (which may include duplicate indices) - for a given ``state`` and ``epoch``. - - Note: This function is not stable during a sync committee period as - a validator's effective balance may change enough to affect the sampling. + for a given ``state`` and ``epoch`` at a sync committee period boundary. """ + assert epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0 + MAX_RANDOM_BYTE = 2**8 - 1 - base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) - active_validator_indices = get_active_validator_indices(state, base_epoch) + active_validator_indices = get_active_validator_indices(state, epoch) active_validator_count = uint64(len(active_validator_indices)) - seed = get_seed(state, base_epoch, DOMAIN_SYNC_COMMITTEE) + seed = get_seed(state, epoch, DOMAIN_SYNC_COMMITTEE) i = 0 sync_committee_indices: List[ValidatorIndex] = [] while len(sync_committee_indices) < SYNC_COMMITTEE_SIZE: diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..4498749d3 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -81,7 +81,9 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: inactivity_scores=[uint64(0) for _ in range(len(pre.validators))], ) # Fill in sync committees - post.current_sync_committee = get_sync_committee(post, get_current_epoch(post)) - post.next_sync_committee = get_sync_committee(post, get_current_epoch(post) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_period = epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD + base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + post.current_sync_committee = get_sync_committee(post, base_epoch) + post.next_sync_committee = get_sync_committee(post, base_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) return post ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 2e5d62d11..834f51956 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -7,7 +7,6 @@ from eth2spec.test.helpers.block_processing import run_block_processing_to from eth2spec.test.helpers.state import ( state_transition_and_sign_block, transition_to, - next_epoch, ) from eth2spec.test.helpers.constants import ( MAINNET, MINIMAL, @@ -367,43 +366,3 @@ def test_valid_signature_future_committee(spec, state): ) yield from run_sync_committee_processing(spec, state, block) - - -@with_altair_and_later -@spec_state_test -def test_sync_committee_is_only_computed_at_epoch_boundary(spec, state): - """ - Sync committees can only be computed at sync committee period boundaries. - Ensure a client respects the committee in the state (assumed to be derived - in the correct way). - """ - current_epoch = spec.get_current_epoch(state) - - # use a "synthetic" committee to simulate the situation - # where ``spec.get_sync_committee`` at the sync committee - # period epoch boundary would have diverged some epochs into the - # period; ``aggregate_pubkey`` is not relevant to this test - pubkeys = [] - committee_indices = [] - i = 0 - active_validator_count = len(spec.get_active_validator_indices(state, current_epoch)) - while len(pubkeys) < spec.SYNC_COMMITTEE_SIZE: - v = state.validators[i % active_validator_count] - if spec.is_active_validator(v, current_epoch): - pubkeys.append(v.pubkey) - committee_indices.append(i) - i += 1 - - synthetic_committee = spec.SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=spec.BLSPubkey()) - state.current_sync_committee = synthetic_committee - - assert spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD > 3 - for _ in range(3): - next_epoch(spec, state) - - committee = get_committee_indices(spec, state) - assert committee != committee_indices - committee_size = len(committee_indices) - committee_bits = [True] * committee_size - - yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c909c791c..2910145c9 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -39,8 +39,7 @@ def run_sync_committees_progress_test(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` - current_epoch = spec.get_current_epoch(state) - third_sync_committee = spec.get_sync_committee(state, current_epoch + 2 * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + third_sync_committee = spec.get_sync_committee(state, (next_period + 1) * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py index ffe743531..407455a0a 100644 --- a/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/altair/sanity/test_blocks.py @@ -18,7 +18,8 @@ from eth2spec.test.context import ( def run_sync_committee_sanity_test(spec, state, fraction_full=1.0): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] participants = random.sample(committee, int(len(committee) * fraction_full)) yield 'pre', state diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py index 932a46ca5..554cebda8 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/test_sync_protocol.py @@ -46,7 +46,8 @@ def test_process_light_client_update_not_updated(spec, state): body_root=signed_block.message.body.hash_tree_root(), ) # Sync committee signing the header - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] sync_committee_bits = [True] * len(committee) sync_committee_signature = compute_aggregate_sync_committee_signature( spec, @@ -111,7 +112,8 @@ def test_process_light_client_update_timeout(spec, state): ) # Sync committee signing the finalized_block_header - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] sync_committee_bits = [True] * len(committee) sync_committee_signature = compute_aggregate_sync_committee_signature( spec, @@ -190,7 +192,8 @@ def test_process_light_client_update_finality_updated(spec, state): ) # Sync committee signing the finalized_block_header - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + all_pubkeys = [v.pubkey for v in state.validators] + committee = [all_pubkeys.index(pubkey) for pubkey in state.current_sync_committee.pubkeys] sync_committee_bits = [True] * len(committee) sync_committee_signature = compute_aggregate_sync_committee_signature( spec, From 43ba615b7579ff1ac2e616ed1dd12cccec14118f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 14:13:22 -0600 Subject: [PATCH 198/227] Apply suggestions from code review --- specs/altair/fork.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/fork.md b/specs/altair/fork.md index d789e7a8e..ac2258def 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -40,9 +40,9 @@ Note that for the pure Altair networks, we don't apply `upgrade_to_altair` since If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == ALTAIR_FORK_EPOCH`, an irregular state change is made to upgrade to Altair. -The upgrade occurs after the inner loop of `process_slots` that sets `state.slot` equal to `ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH` finishes. +The upgrade occurs after the completion of the inner loop of `process_slots` that sets `state.slot` equal to `ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH`. Care must be taken when transitioning through the fork boundary as implementations will need a modified state transition function that deviates from the Phase 0 spec. -In particular, the `state_transition` function defined in the Phase 0 spec will not expose the correct time to execute the upgrade in the presence of skipped slots at the fork boundary. +In particular, the outer `state_transition` function defined in the Phase 0 spec will not expose the precise fork slot to execute the upgrade in the presence of skipped slots at the fork boundary. Instead the logic must be within `process_slots`. ```python def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: From 274788277636b0e0e319dcacbcd9a03bad21a1ca Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 15:18:18 -0600 Subject: [PATCH 199/227] use current_Epoch seed when calculating next_sync_committee --- specs/altair/beacon-chain.md | 4 ++-- specs/altair/fork.md | 10 ++++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 34a33469e..aa1509460 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -304,7 +304,7 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val ```python def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: """ - Return the sync committee for a given ``state`` and ``epoch``. + Return the *next* sync committee for a given ``state`` and ``epoch``. ``SyncCommittee`` contains an aggregate pubkey that enables resource-constrained clients to save some computation when verifying @@ -690,7 +690,7 @@ def process_sync_committee_updates(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + Epoch(1) if next_epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0: state.current_sync_committee = state.next_sync_committee - state.next_sync_committee = get_sync_committee(state, next_epoch + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + state.next_sync_committee = get_sync_committee(state, next_epoch) ``` ## Initialize state for pure Altair testnets and test vectors diff --git a/specs/altair/fork.md b/specs/altair/fork.md index be562b82b..dcd319745 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -80,8 +80,14 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: # Inactivity inactivity_scores=[uint64(0) for _ in range(len(pre.validators))], ) + # Fill in sync committees - post.current_sync_committee = get_sync_committee(post, get_current_epoch(post)) - post.next_sync_committee = get_sync_committee(post, get_current_epoch(post) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_period = epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_period = current_period - min(1, current_period) + current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + + post.current_sync_committee = get_sync_committee(post, previous_base_epoch) + post.next_sync_committee = get_sync_committee(post, current_base_epoch) return post ``` From 200c049778ade1d28064cb837c8ebdd6ca2356fa Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 15:55:33 -0600 Subject: [PATCH 200/227] fix seed calc issue --- specs/altair/beacon-chain.md | 17 ++++++++++++----- .../test_process_sync_committee.py | 6 +++--- .../test_process_sync_committee_updates.py | 2 +- .../pyspec/eth2spec/test/helpers/genesis.py | 11 +++++++---- 4 files changed, 23 insertions(+), 13 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index aa1509460..a69412c31 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -281,11 +281,12 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val Note: This function is not stable during a sync committee period as a validator's effective balance may change enough to affect the sampling. """ + assert epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0 + MAX_RANDOM_BYTE = 2**8 - 1 - base_epoch = Epoch((max(epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD, 1) - 1) * EPOCHS_PER_SYNC_COMMITTEE_PERIOD) - active_validator_indices = get_active_validator_indices(state, base_epoch) + active_validator_indices = get_active_validator_indices(state, epoch) active_validator_count = uint64(len(active_validator_indices)) - seed = get_seed(state, base_epoch, DOMAIN_SYNC_COMMITTEE) + seed = get_seed(state, epoch, DOMAIN_SYNC_COMMITTEE) i = 0 sync_committee_indices: List[ValidatorIndex] = [] while len(sync_committee_indices) < SYNC_COMMITTEE_SIZE: @@ -689,6 +690,7 @@ def process_participation_flag_updates(state: BeaconState) -> None: def process_sync_committee_updates(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + Epoch(1) if next_epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0: + print("HEEEREEE") state.current_sync_committee = state.next_sync_committee state.next_sync_committee = get_sync_committee(state, next_epoch) ``` @@ -735,8 +737,13 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, state.genesis_validators_root = hash_tree_root(state.validators) # [New in Altair] Fill in sync committees - state.current_sync_committee = get_sync_committee(state, get_current_epoch(state)) - state.next_sync_committee = get_sync_committee(state, get_current_epoch(state) + EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + current_period = get_current_epoch(state) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_period = current_period - min(1, current_period) + current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD + + state.current_sync_committee = get_sync_committee(state, previous_base_epoch) + state.next_sync_committee = get_sync_committee(state, current_base_epoch) return state ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index 2e5d62d11..48e556f50 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -345,10 +345,10 @@ def test_valid_signature_future_committee(spec, state): transition_to(spec, state, slot_in_future_sync_committee_period) sync_committee = state.current_sync_committee + next_sync_committee = state.next_sync_committee + expected_next_sync_committee = spec.get_sync_committee(state, epoch_in_future_sync_committee_period) - expected_sync_committee = spec.get_sync_committee(state, epoch_in_future_sync_committee_period) - - assert sync_committee == expected_sync_committee + assert next_sync_committee == expected_next_sync_committee assert sync_committee != old_current_sync_committee assert sync_committee != old_next_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index c909c791c..ca25797c4 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -40,7 +40,7 @@ def run_sync_committees_progress_test(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` current_epoch = spec.get_current_epoch(state) - third_sync_committee = spec.get_sync_committee(state, current_epoch + 2 * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + third_sync_committee = spec.get_sync_committee(state, current_epoch + 1) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 4a34a5eb3..23cd04c93 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -69,9 +69,12 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if spec.fork not in FORKS_BEFORE_ALTAIR: # Fill in sync committees - state.current_sync_committee = spec.get_sync_committee(state, spec.get_current_epoch(state)) - state.next_sync_committee = ( - spec.get_sync_committee(state, spec.get_current_epoch(state) + spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) - ) + current_period = spec.get_current_epoch(state) // spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_period = current_period - min(1, current_period) + current_base_epoch = current_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + previous_base_epoch = previous_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD + + state.current_sync_committee = spec.get_sync_committee(state, previous_base_epoch) + state.next_sync_committee = spec.get_sync_committee(state, current_base_epoch) return state From 09cefa03f3895c2a319ee03bc06c66a6b8d5e11b Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 16:07:24 -0600 Subject: [PATCH 201/227] remov sync signature todo --- specs/altair/beacon-chain.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 57fec33df..6d97c14b5 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -203,7 +203,6 @@ class BeaconState(Container): ```python class SyncAggregate(Container): sync_committee_bits: Bitvector[SYNC_COMMITTEE_SIZE] - # TODO! Need multiple signatures as discussed between Justin and Vitalik May 3 2021 (see Telegram) sync_committee_signature: BLSSignature ``` From 8bb0531f584756f248e52ab44eb23ec8c6ac9f5c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 11 May 2021 16:23:31 -0600 Subject: [PATCH 202/227] only give counter-weight penalty if leak. TMP --- specs/altair/beacon-chain.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 4c6f47590..56b6b0b79 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -407,9 +407,11 @@ def get_inactivity_penalty_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], S previous_epoch = get_previous_epoch(state) matching_target_indices = get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch) for index in get_eligible_validator_indices(state): - for (_, weight) in get_flag_indices_and_weights(): - # This inactivity penalty cancels the flag reward corresponding to the flag index - penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) + if is_in_inactivity_leak(state): + # TODO: to be removed in PR 2399 + for (_, weight) in get_flag_indices_and_weights(): + # This inactivity penalty cancels the flag reward corresponding to the flag index + penalties[index] += Gwei(get_base_reward(state, index) * weight // WEIGHT_DENOMINATOR) if index not in matching_target_indices: penalty_numerator = state.validators[index].effective_balance * state.inactivity_scores[index] penalty_denominator = INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR From 0390ab819a6628ff582a4eba3b0f56ced22ce268 Mon Sep 17 00:00:00 2001 From: protolambda Date: Wed, 12 May 2021 02:40:23 +0200 Subject: [PATCH 203/227] Protocols pyspec support + execution payload tests cleanup --- setup.py | 80 ++++++++++++++++--- specs/merge/beacon-chain.md | 4 +- specs/merge/validator.md | 2 +- .../test/helpers/execution_payload.py | 4 + .../test_process_execution_payload.py | 18 +++-- 5 files changed, 91 insertions(+), 17 deletions(-) diff --git a/setup.py b/setup.py index f0befc8c7..285d2d3ac 100644 --- a/setup.py +++ b/setup.py @@ -5,6 +5,7 @@ from distutils.util import convert_path import os import re import string +import textwrap from typing import Dict, NamedTuple, List, Sequence, Optional from abc import ABC, abstractmethod import ast @@ -48,8 +49,14 @@ def floorlog2(x: int) -> uint64: ''' +class ProtocolDefinition(NamedTuple): + # just function definitions currently. May expand with configuration vars in future. + functions: Dict[str, str] + + class SpecObject(NamedTuple): functions: Dict[str, str] + protocols: Dict[str, ProtocolDefinition] custom_types: Dict[str, str] constants: Dict[str, str] ssz_dep_constants: Dict[str, str] # the constants that depend on ssz_objects @@ -73,6 +80,18 @@ def _get_function_name_from_source(source: str) -> str: return fn.name +def _get_self_type_from_source(source: str) -> Optional[str]: + fn = ast.parse(source).body[0] + args = fn.args.args + if len(args) == 0: + return None + if args[0].arg != 'self': + return None + if args[0].annotation is None: + return None + return args[0].annotation.id + + def _get_class_info_from_source(source: str) -> (str, Optional[str]): class_def = ast.parse(source).body[0] base = class_def.bases[0] @@ -107,6 +126,7 @@ def _get_eth2_spec_comment(child: LinkRefDef) -> Optional[str]: def get_spec(file_name: str) -> SpecObject: functions: Dict[str, str] = {} + protocols: Dict[str, ProtocolDefinition] = {} constants: Dict[str, str] = {} ssz_dep_constants: Dict[str, str] = {} ssz_objects: Dict[str, str] = {} @@ -132,7 +152,14 @@ def get_spec(file_name: str) -> SpecObject: source = _get_source_from_code_block(child) if source.startswith("def"): current_name = _get_function_name_from_source(source) - functions[current_name] = "\n".join(line.rstrip() for line in source.splitlines()) + self_type_name = _get_self_type_from_source(source) + function_def = "\n".join(line.rstrip() for line in source.splitlines()) + if self_type_name is None: + functions[current_name] = function_def + else: + if self_type_name not in protocols: + protocols[self_type_name] = ProtocolDefinition(functions={}) + protocols[self_type_name].functions[current_name] = function_def elif source.startswith("@dataclass"): dataclasses[current_name] = "\n".join(line.rstrip() for line in source.splitlines()) elif source.startswith("class"): @@ -170,6 +197,7 @@ def get_spec(file_name: str) -> SpecObject: return SpecObject( functions=functions, + protocols=protocols, custom_types=custom_types, constants=constants, ssz_dep_constants=ssz_dep_constants, @@ -422,7 +450,8 @@ class MergeSpecBuilder(Phase0SpecBuilder): @classmethod def imports(cls): - return super().imports() + '\n' + ''' + return super().imports() + ''' +from typing import Protocol from eth2spec.phase0 import spec as phase0 from eth2spec.utils.ssz.ssz_typing import Bytes20, ByteList, ByteVector, uint256 from importlib import reload @@ -451,13 +480,23 @@ def get_execution_state(execution_state_root: Bytes32) -> ExecutionState: def get_pow_chain_head() -> PowBlock: pass -verify_execution_state_transition_ret_value = True -def verify_execution_state_transition(execution_payload: ExecutionPayload) -> bool: - return verify_execution_state_transition_ret_value + +class NoopExecutionEngine(ExecutionEngine): + + def new_block(self, execution_payload: ExecutionPayload) -> bool: + return True + + def set_head(self, block_hash: Hash32) -> bool: + return True + + def finalize_block(self, block_hash: Hash32) -> bool: + return True + + def assemble_block(self, block_hash: Hash32, timestamp: uint64) -> ExecutionPayload: + raise NotImplementedError("no default block production") -def produce_execution_payload(parent_hash: Hash32, timestamp: uint64) -> ExecutionPayload: - pass""" +EXECUTION_ENGINE = NoopExecutionEngine()""" @classmethod @@ -495,6 +534,15 @@ def objects_to_spec(spec_object: SpecObject, builder: SpecBuilder, ordered_class ] ) ) + + def format_protocol(protocol_name: str, protocol_def: ProtocolDefinition) -> str: + protocol = f"class {protocol_name}(Protocol):" + for fn_source in protocol_def.functions.values(): + fn_source = fn_source.replace("self: "+protocol_name, "self") + protocol += "\n\n" + textwrap.indent(fn_source, " ") + return protocol + + protocols_spec = '\n\n\n'.join(format_protocol(k, v) for k, v in spec_object.protocols.items()) for k in list(spec_object.functions): if "ceillog2" in k or "floorlog2" in k: del spec_object.functions[k] @@ -520,6 +568,7 @@ def objects_to_spec(spec_object: SpecObject, builder: SpecBuilder, ordered_class + '\n\n' + constants_spec + '\n\n' + CONFIG_LOADER + '\n\n' + ordered_class_objects_spec + + ('\n\n\n' + protocols_spec if protocols_spec != '' else '') + '\n\n\n' + functions_spec + '\n\n' + builder.sundry_functions() # Since some constants are hardcoded in setup.py, the following assertions verify that the hardcoded constants are @@ -531,6 +580,17 @@ def objects_to_spec(spec_object: SpecObject, builder: SpecBuilder, ordered_class return spec +def combine_protocols(old_protocols: Dict[str, ProtocolDefinition], + new_protocols: Dict[str, ProtocolDefinition]) -> Dict[str, ProtocolDefinition]: + for key, value in new_protocols.items(): + if key not in old_protocols: + old_protocols[key] = value + else: + functions = combine_functions(old_protocols[key].functions, value.functions) + old_protocols[key] = ProtocolDefinition(functions=functions) + return old_protocols + + def combine_functions(old_functions: Dict[str, str], new_functions: Dict[str, str]) -> Dict[str, str]: for key, value in new_functions.items(): old_functions[key] = value @@ -589,8 +649,9 @@ def combine_spec_objects(spec0: SpecObject, spec1: SpecObject) -> SpecObject: """ Takes in two spec variants (as tuples of their objects) and combines them using the appropriate combiner function. """ - functions0, custom_types0, constants0, ssz_dep_constants0, ssz_objects0, dataclasses0 = spec0 - functions1, custom_types1, constants1, ssz_dep_constants1, ssz_objects1, dataclasses1 = spec1 + functions0, protocols0, custom_types0, constants0, ssz_dep_constants0, ssz_objects0, dataclasses0 = spec0 + functions1, protocols1, custom_types1, constants1, ssz_dep_constants1, ssz_objects1, dataclasses1 = spec1 + protocols = combine_protocols(protocols0, protocols1) functions = combine_functions(functions0, functions1) custom_types = combine_constants(custom_types0, custom_types1) constants = combine_constants(constants0, constants1) @@ -599,6 +660,7 @@ def combine_spec_objects(spec0: SpecObject, spec1: SpecObject) -> SpecObject: dataclasses = combine_functions(dataclasses0, dataclasses1) return SpecObject( functions=functions, + protocols=protocols, custom_types=custom_types, constants=constants, ssz_dep_constants=ssz_dep_constants, diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index bca58eb7f..af633e18a 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -216,7 +216,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: ##### `process_execution_payload` ```python -def process_execution_payload(state: BeaconState, execution_payload: ExecutionPayload, execution_engine: ExecutionEngine) -> None: +def process_execution_payload(state: BeaconState, + execution_payload: ExecutionPayload, + execution_engine: ExecutionEngine) -> None: """ Note: This function is designed to be able to be run in parallel with the other `process_block` sub-functions """ diff --git a/specs/merge/validator.md b/specs/merge/validator.md index 21fc49a36..7a1b61439 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -90,5 +90,5 @@ def get_execution_payload(state: BeaconState, execution_engine: ExecutionEngine) # Post-merge, normal payload execution_parent_hash = state.latest_execution_payload_header.block_hash timestamp = compute_time_at_slot(state, state.slot) - return produce_execution_payload(execution_parent_hash, timestamp) + return execution_engine.assemble_block(execution_parent_hash, timestamp) ``` diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index 36e63dd33..7774aa4d9 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -24,6 +24,7 @@ def build_empty_execution_payload(spec, state): return payload + def get_execution_payload_header(spec, execution_payload): return spec.ExecutionPayloadHeader( block_hash=execution_payload.block_hash, @@ -39,15 +40,18 @@ def get_execution_payload_header(spec, execution_payload): transactions_root=spec.hash_tree_root(execution_payload.transactions) ) + def build_state_with_incomplete_transition(spec, state): return build_state_with_execution_payload_header(spec, state, spec.ExecutionPayloadHeader()) + def build_state_with_complete_transition(spec, state): pre_state_payload = build_empty_execution_payload(spec, state) payload_header = get_execution_payload_header(spec, pre_state_payload) return build_state_with_execution_payload_header(spec, state, payload_header) + def build_state_with_execution_payload_header(spec, state, execution_payload_header): pre_state = state.copy() pre_state.latest_execution_payload_header = execution_payload_header diff --git a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py index f1cd8ff25..5edd31960 100644 --- a/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/merge/block_processing/test_process_execution_payload.py @@ -22,23 +22,29 @@ def run_execution_payload_processing(spec, state, execution_payload, valid=True, yield 'execution', {'execution_valid': execution_valid} yield 'execution_payload', execution_payload + called_new_block = False - spec.verify_execution_state_transition_ret_value = execution_valid + class TestEngine(spec.NoopExecutionEngine): + def new_block(self, payload) -> bool: + nonlocal called_new_block, execution_valid + called_new_block = True + assert payload == execution_payload + return execution_valid if not valid: - expect_assertion_error(lambda: spec.process_execution_payload(state, execution_payload)) + expect_assertion_error(lambda: spec.process_execution_payload(state, execution_payload, TestEngine())) yield 'post', None - spec.verify_execution_state_transition_ret_value = True return - spec.process_execution_payload(state, execution_payload) + spec.process_execution_payload(state, execution_payload, TestEngine()) + + # Make sure we called the engine + assert called_new_block yield 'post', state assert state.latest_execution_payload_header == get_execution_payload_header(spec, execution_payload) - spec.verify_execution_state_transition_ret_value = True - @with_merge_and_later @spec_state_test From dad698f97aec8f6e80ecc74d093e050b9ae72163 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 12 May 2021 12:35:47 +0800 Subject: [PATCH 204/227] Update unit tests: add `test_compute_subnets_for_sync_committee_slot_period_boundary` --- .../unittests/validator/test_validator.py | 49 +++++++++++++++++-- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index c8a894da6..fd1b82c4b 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -143,20 +143,61 @@ def _subnet_for_sync_committee_index(spec, i): return i // (spec.SYNC_COMMITTEE_SIZE // spec.SYNC_COMMITTEE_SUBNET_COUNT) +def _get_expected_subnets_by_pubkey(sync_committee_members): + expected_subnets_by_pubkey = defaultdict(list) + for (subnet, pubkey) in sync_committee_members: + expected_subnets_by_pubkey[pubkey].append(subnet) + return expected_subnets_by_pubkey + + @with_altair_and_later @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): + # Transition to the head of the next period + transition_to(spec, state, spec.SLOTS_PER_EPOCH * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) + + next_slot_epoch = spec.compute_epoch_at_slot(state.slot + 1) + assert ( + spec.compute_sync_committee_period(spec.get_current_epoch(state)) + == spec.compute_sync_committee_period(next_slot_epoch) + ) some_sync_committee_members = list( ( _subnet_for_sync_committee_index(spec, i), pubkey, ) + # use current_sync_committee for i, pubkey in enumerate(state.current_sync_committee.pubkeys) ) - - expected_subnets_by_pubkey = defaultdict(list) - for (subnet, pubkey) in some_sync_committee_members: - expected_subnets_by_pubkey[pubkey].append(subnet) + expected_subnets_by_pubkey = _get_expected_subnets_by_pubkey(some_sync_committee_members) + + for _, pubkey in some_sync_committee_members: + validator_index = _validator_index_for_pubkey(state, pubkey) + subnets = spec.compute_subnets_for_sync_committee(state, validator_index) + expected_subnets = expected_subnets_by_pubkey[pubkey] + assert subnets == expected_subnets + + +@with_altair_and_later +@with_state +def test_compute_subnets_for_sync_committee_slot_period_boundary(state, spec, phases): + # Transition to the end of the period + transition_to(spec, state, spec.SLOTS_PER_EPOCH * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - 1) + + next_slot_epoch = spec.compute_epoch_at_slot(state.slot + 1) + assert ( + spec.compute_sync_committee_period(spec.get_current_epoch(state)) + != spec.compute_sync_committee_period(next_slot_epoch) + ) + some_sync_committee_members = list( + ( + _subnet_for_sync_committee_index(spec, i), + pubkey, + ) + # use next_sync_committee + for i, pubkey in enumerate(state.next_sync_committee.pubkeys) + ) + expected_subnets_by_pubkey = _get_expected_subnets_by_pubkey(some_sync_committee_members) for _, pubkey in some_sync_committee_members: validator_index = _validator_index_for_pubkey(state, pubkey) From 17820e371165c4f6ee56163e20cff7da36ba1033 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 12 May 2021 13:02:15 +0800 Subject: [PATCH 205/227] Skip the mainnet config slow tests --- .../test/altair/unittests/validator/test_validator.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index fd1b82c4b..cefaaf694 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -8,8 +8,12 @@ from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( with_altair_and_later, + with_configs, with_state, ) +from eth2spec.test.helpers.constants import ( + MINIMAL, +) rng = random.Random(1337) @@ -91,6 +95,7 @@ def _get_sync_committee_signature( @only_with_bls() @with_altair_and_later +@with_configs([MINIMAL], reason="too slow") @with_state def test_process_sync_committee_contributions(phases, spec, state): # skip over slots at genesis @@ -151,6 +156,7 @@ def _get_expected_subnets_by_pubkey(sync_committee_members): @with_altair_and_later +@with_configs([MINIMAL], reason="too slow") @with_state def test_compute_subnets_for_sync_committee(state, spec, phases): # Transition to the head of the next period @@ -179,6 +185,7 @@ def test_compute_subnets_for_sync_committee(state, spec, phases): @with_altair_and_later +@with_configs([MINIMAL], reason="too slow") @with_state def test_compute_subnets_for_sync_committee_slot_period_boundary(state, spec, phases): # Transition to the end of the period From 8e07ece4927bbc3d7965b92b656abd7e09a73fc1 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 12 May 2021 14:04:49 +0800 Subject: [PATCH 206/227] Minor rephrase --- specs/altair/validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 92e3a75c0..67e5914d1 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -271,7 +271,7 @@ If a validator is in the current sync committee (i.e. `is_assigned_to_sync_commi This logic is triggered upon the same conditions as when producing an attestation. Meaning, a sync committee member should produce and broadcast a `SyncCommitteeSignature` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first. -`get_sync_committee_signature()` assumes `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice (including any empty slots up to the current slot processed with `process_slots` on top of the latest block), `block_root` is the root of the head block, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. +`get_sync_committee_signature(state, block_root, validator_index, privkey)` assumes the parameter `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice (including any empty slots up to the current slot processed with `process_slots` on top of the latest block), `block_root` is the root of the head block, `validator_index` is the index of the validator in the registry `state.validators` controlled by `privkey`, and `privkey` is the BLS private key for the validator. ```python def get_sync_committee_signature(state: BeaconState, @@ -291,7 +291,7 @@ def get_sync_committee_signature(state: BeaconState, The validator broadcasts the assembled signature to the assigned subnet, the `sync_committee_{subnet_id}` pubsub topic. The `subnet_id` is derived from the position in the sync committee such that the sync committee is divided into "subcommittees". -`subnet_id` can be computed via `compute_subnets_for_sync_committee()` where `state` is a `BeaconState` during the matching sync committee period. +`subnet_id` can be computed via `compute_subnets_for_sync_committee(state, validator_index)` where `state` is a `BeaconState` during the matching sync committee period. *Note*: This function returns multiple subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. From 82b7a7be3b5cf6046ad046d4c499bb2721931bf2 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 08:29:42 -0600 Subject: [PATCH 207/227] Apply suggestions from code review Co-authored-by: Alex Stokes Co-authored-by: Jacek Sieka --- README.md | 2 +- specs/phase0/p2p-interface.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8a340e9ad..b74102c30 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ The current features are: * [Deposit Contract](specs/phase0/deposit-contract.md) * [Honest Validator](specs/phase0/validator.md) * [P2P Networking](specs/phase0/p2p-interface.md) -* [Weak Subjectivity](specs/phase0/weak-subjectivity.md.md) +* [Weak Subjectivity](specs/phase0/weak-subjectivity.md) ### Altair diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index f2c7d8610..69ff7e120 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -752,7 +752,7 @@ Clients MUST keep a record of signed blocks seen on the epoch range where `current_epoch` is defined by the current wall-clock time, and clients MUST support serving requests of blocks on this range. -Peers are *repeatedly* unable to reply to Block requests within the +Peers that are unable to reply to block requests within the `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epoch range MAY get descored or disconnected at any time. *Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint @@ -763,7 +763,7 @@ proposer signatures and (2) that the blocks form a valid chain up to the most recent block referenced in the weak subjectivity state. *Note*: Although clients that bootstrap from a weak subjectivity checkpoint can begin -participating in the networking immediately, other peers that are actively block syncing MAY +participating in the networking immediately, other peers MAY disconnect and/or temporarily ban such an un-synced or semi-synced client. Clients MUST respond with at least the first block that exists in the range, if they have it, From f52f067b8ea3f8adbebc936207b06459d1956e72 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 08:36:27 -0600 Subject: [PATCH 208/227] add resourceunavailable error code --- specs/phase0/p2p-interface.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 69ff7e120..297d93948 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -567,6 +567,8 @@ The response code can have one of the following values, encoded as a single unsi The response payload adheres to the `ErrorMessage` schema (described below). - 2: **ServerError** -- the responder encountered an error while processing the request. The response payload adheres to the `ErrorMessage` schema (described below). +- 3: **ResourceUnavailable** -- the responder does not have requested resource. + The response payload adheres to the `ErrorMessage` schema (described below). Clients MAY use response codes above `128` to indicate alternative, erroneous request-specific responses. From 6371707779791a687886f6f7c3b2d1ebd45ae345 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 08:40:34 -0600 Subject: [PATCH 209/227] Apply suggestions from code review Co-authored-by: Hsiao-Wei Wang --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 6d97c14b5..9593d13a9 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -371,7 +371,7 @@ def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence if not is_in_inactivity_leak(state): reward_numerator = base_reward * weight * unslashed_participating_increments rewards[index] += Gwei(reward_numerator // (active_increments * WEIGHT_DENOMINATOR)) - elif not flag_index == TIMELY_HEAD_FLAG_INDEX: + elif flag_index != TIMELY_HEAD_FLAG_INDEX: penalties[index] += Gwei(base_reward * weight // WEIGHT_DENOMINATOR) return rewards, penalties ``` From a8791f04c7016fdbc6a577826ab193296392e2ad Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 09:44:13 -0600 Subject: [PATCH 210/227] 'get_sync_committee -> get_next_sync_committee --- specs/altair/beacon-chain.md | 36 +++--- specs/altair/fork.md | 10 +- .../test_process_sync_committee.py | 119 ++++++++++-------- .../pyspec/eth2spec/test/helpers/genesis.py | 10 +- 4 files changed, 88 insertions(+), 87 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 39e4a2371..367858340 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -32,8 +32,8 @@ - [`add_flag`](#add_flag) - [`has_flag`](#has_flag) - [Beacon state accessors](#beacon-state-accessors) - - [`get_sync_committee_indices`](#get_sync_committee_indices) - - [`get_sync_committee`](#get_sync_committee) + - [`get_next_sync_committee_indices`](#get_next_sync_committee_indices) + - [`get_next_sync_committee`](#get_next_sync_committee) - [`get_base_reward_per_increment`](#get_base_reward_per_increment) - [`get_base_reward`](#get_base_reward) - [`get_unslashed_participating_indices`](#get_unslashed_participating_indices) @@ -270,15 +270,15 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ### Beacon state accessors -#### `get_sync_committee_indices` +#### `get_next_sync_committee_indices` ```python -def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: +def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorIndex]: """ Return the sequence of sync committee indices (which may include duplicate indices) - for a given ``state`` and ``epoch`` at a sync committee period boundary. + for the next sync committee, given a ``state`` at a sync committee period boundary. """ - assert epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0 + epoch = Epoch(get_current_epoch(state) + 1) MAX_RANDOM_BYTE = 2**8 - 1 active_validator_indices = get_active_validator_indices(state, epoch) @@ -297,25 +297,25 @@ def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[Val return sync_committee_indices ``` -#### `get_sync_committee` +#### `get_next_sync_committee` ```python -def get_sync_committee(state: BeaconState, epoch: Epoch) -> SyncCommittee: +def get_next_sync_committee(state: BeaconState) -> SyncCommittee: """ - Return the *next* sync committee for a given ``state`` and ``epoch``. + Return the *next* sync committee for a given ``state``. ``SyncCommittee`` contains an aggregate pubkey that enables resource-constrained clients to save some computation when verifying the sync committee's signature. - ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_sync_committee_indices`` + ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_next_sync_committee_indices`` returns duplicate indices. Implementations must take care when handling optimizations relating to aggregation and verification in the presence of duplicates. Note: This function should only be called at sync committee period boundaries, as - ``get_sync_committee_indices`` is not stable within a given period. + ``get_next_sync_committee_indices`` is not stable within a given period. """ - indices = get_sync_committee_indices(state, epoch) + indices = get_next_sync_committee_indices(state) pubkeys = [state.validators[index].pubkey for index in indices] aggregate_pubkey = bls.AggregatePKs(pubkeys) return SyncCommittee(pubkeys=pubkeys, aggregate_pubkey=aggregate_pubkey) @@ -688,7 +688,7 @@ def process_sync_committee_updates(state: BeaconState) -> None: next_epoch = get_current_epoch(state) + Epoch(1) if next_epoch % EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0: state.current_sync_committee = state.next_sync_committee - state.next_sync_committee = get_sync_committee(state, next_epoch) + state.next_sync_committee = get_next_sync_committee(state) ``` ## Initialize state for pure Altair testnets and test vectors @@ -733,13 +733,9 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, state.genesis_validators_root = hash_tree_root(state.validators) # [New in Altair] Fill in sync committees - current_period = get_current_epoch(state) // EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_period = current_period - min(1, current_period) - current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - - state.current_sync_committee = get_sync_committee(state, previous_base_epoch) - state.next_sync_committee = get_sync_committee(state, current_base_epoch) + # Note: A duplicate committee is assigned for the current and next committee at genesis + state.current_sync_committee = get_next_sync_committee(state) + state.next_sync_committee = get_next_sync_committee(state) return state ``` diff --git a/specs/altair/fork.md b/specs/altair/fork.md index ea7af898f..2c8b9f855 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -86,12 +86,8 @@ def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: ) # Fill in sync committees - current_period = epoch // EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_period = current_period - min(1, current_period) - current_base_epoch = current_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_base_epoch = previous_period * EPOCHS_PER_SYNC_COMMITTEE_PERIOD - - post.current_sync_committee = get_sync_committee(post, previous_base_epoch) - post.next_sync_committee = get_sync_committee(post, current_base_epoch) + # Note: A duplicate committee is assigned for the current and next committee at the fork boundary + post.current_sync_committee = get_next_sync_committee(post) + post.next_sync_committee = get_next_sync_committee(post) return post ``` diff --git a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py index d80e650af..e3ee32a93 100644 --- a/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py +++ b/tests/core/pyspec/eth2spec/test/altair/block_processing/test_process_sync_committee.py @@ -49,9 +49,9 @@ def get_committee_indices(spec, state, duplicates=False): """ state = state.copy() current_epoch = spec.get_current_epoch(state) - randao_index = current_epoch % spec.EPOCHS_PER_HISTORICAL_VECTOR + randao_index = (current_epoch + 1) % spec.EPOCHS_PER_HISTORICAL_VECTOR while True: - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee = spec.get_next_sync_committee_indices(state) if duplicates: if len(committee) != len(set(committee)): return committee @@ -61,23 +61,32 @@ def get_committee_indices(spec, state, duplicates=False): state.randao_mixes[randao_index] = hash(state.randao_mixes[randao_index]) +def compute_committee_indices(spec, state, committee): + """ + Given a ``committee``, calculate and return the related indices + """ + all_pubkeys = [v.pubkey for v in state.validators] + committee_indices = [all_pubkeys.index(pubkey) for pubkey in committee.pubkeys] + return committee_indices + + @with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_missing_participant(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) rng = random.Random(2020) - random_participant = rng.choice(committee) + random_participant = rng.choice(committee_indices) block = build_empty_block_for_next_slot(spec, state) # Exclude one participant whose signature was included. block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[index != random_participant for index in committee], + sync_committee_bits=[index != random_participant for index in committee_indices], sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - committee, # full committee signs + committee_indices, # full committee signs ) ) yield from run_sync_committee_processing(spec, state, block, expect_exception=True) @@ -87,31 +96,38 @@ def test_invalid_signature_missing_participant(spec, state): @spec_state_test @always_bls def test_invalid_signature_extra_participant(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) rng = random.Random(3030) - random_participant = rng.choice(committee) + random_participant = rng.choice(committee_indices) block = build_empty_block_for_next_slot(spec, state) # Exclude one signature even though the block claims the entire committee participated. block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - [index for index in committee if index != random_participant], + [index for index in committee_indices if index != random_participant], ) ) yield from run_sync_committee_processing(spec, state, block, expect_exception=True) -def compute_sync_committee_inclusion_reward(spec, state, participant_index, committee, committee_bits): +def compute_sync_committee_inclusion_reward(spec, + state, + participant_index, + committee_indices, + committee_bits): total_active_increments = spec.get_total_active_balance(state) // spec.EFFECTIVE_BALANCE_INCREMENT total_base_rewards = spec.Gwei(spec.get_base_reward_per_increment(state) * total_active_increments) max_epoch_rewards = spec.Gwei(total_base_rewards * spec.SYNC_REWARD_WEIGHT // spec.WEIGHT_DENOMINATOR) - included_indices = [index for index, bit in zip(committee, committee_bits) if bit] - max_slot_rewards = spec.Gwei(max_epoch_rewards * len(included_indices) // len(committee) // spec.SLOTS_PER_EPOCH) + included_indices = [index for index, bit in zip(committee_indices, committee_bits) if bit] + max_slot_rewards = spec.Gwei( + max_epoch_rewards * len(included_indices) + // len(committee_indices) // spec.SLOTS_PER_EPOCH + ) # Compute the participant and proposer sync rewards committee_effective_balance = sum([state.validators[index].effective_balance for index in included_indices]) @@ -120,23 +136,23 @@ def compute_sync_committee_inclusion_reward(spec, state, participant_index, comm return spec.Gwei(max_slot_rewards * effective_balance // committee_effective_balance) -def compute_sync_committee_participant_reward(spec, state, participant_index, committee, committee_bits): - included_indices = [index for index, bit in zip(committee, committee_bits) if bit] +def compute_sync_committee_participant_reward(spec, state, participant_index, committee_indices, committee_bits): + included_indices = [index for index, bit in zip(committee_indices, committee_bits) if bit] multiplicities = Counter(included_indices) inclusion_reward = compute_sync_committee_inclusion_reward( - spec, state, participant_index, committee, committee_bits, + spec, state, participant_index, committee_indices, committee_bits, ) return spec.Gwei(inclusion_reward * multiplicities[participant_index]) -def compute_sync_committee_proposer_reward(spec, state, committee, committee_bits): +def compute_sync_committee_proposer_reward(spec, state, committee_indices, committee_bits): proposer_reward = 0 - for index, bit in zip(committee, committee_bits): + for index, bit in zip(committee_indices, committee_bits): if not bit: continue inclusion_reward = compute_sync_committee_inclusion_reward( - spec, state, index, committee, committee_bits, + spec, state, index, committee_indices, committee_bits, ) proposer_reward_denominator = ( (spec.WEIGHT_DENOMINATOR - spec.PROPOSER_WEIGHT) @@ -147,15 +163,15 @@ def compute_sync_committee_proposer_reward(spec, state, committee, committee_bit return proposer_reward -def validate_sync_committee_rewards(spec, pre_state, post_state, committee, committee_bits, proposer_index): +def validate_sync_committee_rewards(spec, pre_state, post_state, committee_indices, committee_bits, proposer_index): for index in range(len(post_state.validators)): reward = 0 - if index in committee: + if index in committee_indices: reward += compute_sync_committee_participant_reward( spec, pre_state, index, - committee, + committee_indices, committee_bits, ) @@ -163,14 +179,14 @@ def validate_sync_committee_rewards(spec, pre_state, post_state, committee, comm reward += compute_sync_committee_proposer_reward( spec, pre_state, - committee, + committee_indices, committee_bits, ) assert post_state.balances[index] == pre_state.balances[index] + reward -def run_successful_sync_committee_test(spec, state, committee, committee_bits): +def run_successful_sync_committee_test(spec, state, committee_indices, committee_bits): pre_state = state.copy() block = build_empty_block_for_next_slot(spec, state) @@ -180,7 +196,7 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): spec, state, block.slot - 1, - [index for index, bit in zip(committee, committee_bits) if bit], + [index for index, bit in zip(committee_indices, committee_bits) if bit], ) ) @@ -190,7 +206,7 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): spec, pre_state, state, - committee, + committee_indices, committee_bits, block.proposer_index, ) @@ -200,60 +216,60 @@ def run_successful_sync_committee_test(spec, state, committee, committee_bits): @with_configs([MINIMAL], reason="to create nonduplicate committee") @spec_state_test def test_sync_committee_rewards_nonduplicate_committee(spec, state): - committee = get_committee_indices(spec, state, duplicates=False) - committee_size = len(committee) + committee_indices = get_committee_indices(spec, state, duplicates=False) + committee_size = len(committee_indices) committee_bits = [True] * committee_size active_validator_count = len(spec.get_active_validator_indices(state, spec.get_current_epoch(state))) # Preconditions of this test case assert active_validator_count >= spec.SYNC_COMMITTEE_SIZE - assert committee_size == len(set(committee)) + assert committee_size == len(set(committee_indices)) - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @with_configs([MAINNET], reason="to create duplicate committee") @spec_state_test def test_sync_committee_rewards_duplicate_committee(spec, state): - committee = get_committee_indices(spec, state, duplicates=True) - committee_size = len(committee) + committee_indices = get_committee_indices(spec, state, duplicates=True) + committee_size = len(committee_indices) committee_bits = [True] * committee_size active_validator_count = len(spec.get_active_validator_indices(state, spec.get_current_epoch(state))) # Preconditions of this test case assert active_validator_count < spec.SYNC_COMMITTEE_SIZE - assert committee_size > len(set(committee)) + assert committee_size > len(set(committee_indices)) - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_not_full_participants(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) rng = random.Random(1010) - committee_bits = [rng.choice([True, False]) for _ in committee] + committee_bits = [rng.choice([True, False]) for _ in committee_indices] - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @spec_state_test @always_bls def test_sync_committee_rewards_empty_participants(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) - committee_bits = [False for _ in committee] + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) + committee_bits = [False for _ in committee_indices] - yield from run_successful_sync_committee_test(spec, state, committee, committee_bits) + yield from run_successful_sync_committee_test(spec, state, committee_indices, committee_bits) @with_altair_and_later @spec_state_test @always_bls def test_invalid_signature_past_block(spec, state): - committee = spec.get_sync_committee_indices(state, spec.get_current_epoch(state)) + committee_indices = compute_committee_indices(spec, state, state.current_sync_committee) blocks = [] for _ in range(2): @@ -261,12 +277,12 @@ def test_invalid_signature_past_block(spec, state): block = build_empty_block_for_next_slot(spec, state) # Valid sync committee signature here... block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - committee, + committee_indices, ) ) @@ -276,12 +292,12 @@ def test_invalid_signature_past_block(spec, state): invalid_block = build_empty_block_for_next_slot(spec, state) # Invalid signature from a slot other than the previous invalid_block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, invalid_block.slot - 2, - committee, + committee_indices, ) ) @@ -306,19 +322,18 @@ def test_invalid_signature_previous_committee(spec, state): transition_to(spec, state, slot_in_future_sync_committee_period) # Use the previous sync committee to produce the signature. - pubkeys = [validator.pubkey for validator in state.validators] # Ensure that the pubkey sets are different. assert set(old_sync_committee.pubkeys) != set(state.current_sync_committee.pubkeys) - committee = [pubkeys.index(pubkey) for pubkey in old_sync_committee.pubkeys] + committee_indices = compute_committee_indices(spec, state, old_sync_committee) block = build_empty_block_for_next_slot(spec, state) block.body.sync_aggregate = spec.SyncAggregate( - sync_committee_bits=[True] * len(committee), + sync_committee_bits=[True] * len(committee_indices), sync_committee_signature=compute_aggregate_sync_committee_signature( spec, state, block.slot - 1, - committee, + committee_indices, ) ) @@ -345,14 +360,12 @@ def test_valid_signature_future_committee(spec, state): sync_committee = state.current_sync_committee next_sync_committee = state.next_sync_committee - expected_next_sync_committee = spec.get_sync_committee(state, epoch_in_future_sync_committee_period) - assert next_sync_committee == expected_next_sync_committee + assert next_sync_committee != sync_committee assert sync_committee != old_current_sync_committee assert sync_committee != old_next_sync_committee - pubkeys = [validator.pubkey for validator in state.validators] - committee_indices = [pubkeys.index(pubkey) for pubkey in sync_committee.pubkeys] + committee_indices = compute_committee_indices(spec, state, sync_committee) block = build_empty_block_for_next_slot(spec, state) block.body.sync_aggregate = spec.SyncAggregate( diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 23cd04c93..c57442766 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -69,12 +69,8 @@ def create_genesis_state(spec, validator_balances, activation_threshold): if spec.fork not in FORKS_BEFORE_ALTAIR: # Fill in sync committees - current_period = spec.get_current_epoch(state) // spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_period = current_period - min(1, current_period) - current_base_epoch = current_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - previous_base_epoch = previous_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD - - state.current_sync_committee = spec.get_sync_committee(state, previous_base_epoch) - state.next_sync_committee = spec.get_sync_committee(state, current_base_epoch) + # Note: A duplicate committee is assigned for the current and next committee at genesis + state.current_sync_committee = spec.get_next_sync_committee(state) + state.next_sync_committee = spec.get_next_sync_committee(state) return state From 133875a6d6ce5a0e085e310cca3ca254db8d0dd1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 09:56:13 -0600 Subject: [PATCH 211/227] fix sync_committe_update tests --- .../epoch_processing/test_process_sync_committee_updates.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py index ca25797c4..6af16287c 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_sync_committee_updates.py @@ -39,8 +39,7 @@ def run_sync_committees_progress_test(spec, state): # Can compute the third committee having computed final balances in the last epoch # of this `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` - current_epoch = spec.get_current_epoch(state) - third_sync_committee = spec.get_sync_committee(state, current_epoch + 1) + third_sync_committee = spec.get_next_sync_committee(state) assert state.current_sync_committee == second_sync_committee assert state.next_sync_committee == third_sync_committee From 4286f85a68a7b2abdca249cd02482616840e5f86 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 12 May 2021 11:05:41 -0700 Subject: [PATCH 212/227] Update validator guide with restricted sync committee computation --- specs/altair/validator.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 67e5914d1..4aee98ad4 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -177,7 +177,6 @@ At any given `epoch`, the `BeaconState` contains the current `SyncCommittee` and Once every `EPOCHS_PER_SYNC_COMMITTEE_PERIOD` epochs, the next `SyncCommittee` becomes the current `SyncCommittee` and the next committee is computed and stored. *Note*: The data required to compute a given committee is not cached in the `BeaconState` after committees are calculated at the period boundaries. -This means that calling `get_sync_commitee()` in a given `epoch` can return a different result than what was computed during the relevant epoch transition. For this reason, *always* get committee assignments via the fields of the `BeaconState` (`current_sync_committee` and `next_sync_committee`) or use the above reference code. A validator should plan for future sync committee assignments by noting which sync committee periods they are selected for participation. @@ -229,12 +228,12 @@ def process_sync_committee_contributions(block: BeaconBlock, contributions: Set[SyncCommitteeContribution]) -> None: sync_aggregate = SyncAggregate() signatures = [] + sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT for contribution in contributions: subcommittee_index = contribution.subcommittee_index for index, participated in enumerate(contribution.aggregation_bits): if participated: - sync_subcommittee_size = SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT participant_index = sync_subcommittee_size * subcommittee_index + index sync_aggregate.sync_committee_bits[participant_index] = True signatures.append(contribution.signature) @@ -367,7 +366,7 @@ Set `contribution.subcommittee_index` to the index for the subcommittee index co ###### Aggregation bits Let `contribution.aggregation_bits` be a `Bitvector[SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT]`, where the `index`th bit is set in the `Bitvector` for each corresponding validator included in this aggregate from the corresponding subcommittee. -An aggregator finds the index in the sync committee (as returned by `get_sync_committee_indices()`) for a given validator referenced by `sync_committee_signature.validator_index` and maps the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is set in `contribution.aggegration_bits`. +An aggregator finds the index in the sync committee (as determined by a reverse pubkey lookup on `state.current_sync_committee.pubkeys`) for a given validator referenced by `sync_committee_signature.validator_index` and maps the sync committee index to an index in the subcommittee (along with the prior `subcommittee_index`). This index within the subcommittee is set in `contribution.aggegration_bits`. For example, if a validator with index `2044` is pseudo-randomly sampled to sync committee index `135`. This sync committee index maps to `subcommittee_index` `1` with position `7` in the `Bitvector` for the contribution. From 5188671816b3a846b1fa5ea781d3321f0e2c5f04 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 12:18:35 -0600 Subject: [PATCH 213/227] Update specs/altair/beacon-chain.md Co-authored-by: dankrad --- specs/altair/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 9593d13a9..492a146ea 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -262,7 +262,7 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: def get_sync_committee_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]: """ Return the sequence of sync committee indices for a given ``state`` and ``epoch``. - Note: Committee can contain duplicate indices for small validator sets (< 2 * SYNC_COMMITTEE_SIZE) + Note: Committee can contain duplicate indices for small validator sets (< SYNC_COMMITTEE_SIZE + 128) Note: This function is not stable during a sync committee period as a validator's effective balance may change enough to affect the sampling. From fa1bdabaced8ccbce63f1e03f2cbba7e6a1d0860 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 14:23:11 -0600 Subject: [PATCH 214/227] add random inactivity scores tests --- .../altair/rewards/test_inactivity_scores.py | 121 ++++++++++++++++++ .../test/phase0/rewards/test_random.py | 10 +- 2 files changed, 130 insertions(+), 1 deletion(-) create mode 100644 tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py diff --git a/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py b/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py new file mode 100644 index 000000000..a523b5157 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py @@ -0,0 +1,121 @@ +from random import Random + +from eth2spec.test.context import ( + with_altair_and_later, + spec_test, + spec_state_test, + with_custom_state, + single_phase, + low_balances, misc_balances, +) +from eth2spec.test.helpers.rewards import leaking +import eth2spec.test.helpers.rewards as rewards_helpers + + +def randomize_inactivity_scores(spec, state, minimum=0, maximum=50000, rng=Random(4242)): + state.inactivity_scores = [rng.randint(minimum, maximum) for _ in range(len(state.validators))] + + +@with_altair_and_later +@spec_state_test +def test_random_inactivity_scores_0(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(9999)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(9999)) + + +@with_altair_and_later +@spec_state_test +def test_random_inactivity_scores_1(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(10000)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(10000)) + + +@with_altair_and_later +@spec_state_test +def test_half_zero_half_random_inactivity_scores(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(10101)) + half_val_point = len(state.validators) // 2 + state.inactivity_scores = [0] * half_val_point + state.inactivity_scores[half_val_point:] + + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(10101)) + + +@with_altair_and_later +@spec_state_test +def test_random_high_inactivity_scores(spec, state): + randomize_inactivity_scores(spec, state, minimum=500000, maximum=5000000, rng=Random(9998)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(9998)) + + +@with_altair_and_later +@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) +@spec_test +@single_phase +def test_random_inactivity_scores_low_balances_0(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(11111)) + yield from rewards_helpers.run_test_full_random(spec, state) + + +@with_altair_and_later +@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: 0) +@spec_test +@single_phase +def test_random_inactivity_scores_low_balances_1(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(22222)) + yield from rewards_helpers.run_test_full_random(spec, state) + + +@with_altair_and_later +@with_custom_state(balances_fn=misc_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) +@spec_test +@single_phase +def test_full_random_misc_balances(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(33333)) + yield from rewards_helpers.run_test_full_random(spec, state) + + +# +# Leaking variants +# + +@with_altair_and_later +@spec_state_test +@leaking() +def test_random_inactivity_scores_leaking_0(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(9999)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(9999)) + + +@with_altair_and_later +@spec_state_test +@leaking() +def test_random_inactivity_scores_leaking_1(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(10000)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(10000)) + + +@with_altair_and_later +@spec_state_test +@leaking() +def test_half_zero_half_random_inactivity_scores_leaking(spec, state): + randomize_inactivity_scores(spec, state, rng=Random(10101)) + half_val_point = len(state.validators) // 2 + state.inactivity_scores = [0] * half_val_point + state.inactivity_scores[half_val_point:] + + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(10101)) + + +@with_altair_and_later +@spec_state_test +@leaking() +def test_random_high_inactivity_scores_leaking(spec, state): + randomize_inactivity_scores(spec, state, minimum=500000, maximum=5000000, rng=Random(9998)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(9998)) + + +@with_altair_and_later +@spec_state_test +@leaking(epochs=5) +def test_random_high_inactivity_scores_leaking_5_epochs(spec, state): + randomize_inactivity_scores(spec, state, minimum=500000, maximum=5000000, rng=Random(9998)) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(9998)) diff --git a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py index ae44c6640..51bd27c14 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py +++ b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py @@ -39,7 +39,15 @@ def test_full_random_3(spec, state): @with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) @spec_test @single_phase -def test_full_random_low_balances(spec, state): +def test_full_random_low_balances_0(spec, state): + yield from rewards_helpers.run_test_full_random(spec, state) + + +@with_all_phases +@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: 0) +@spec_test +@single_phase +def test_full_random_low_balances_1(spec, state): yield from rewards_helpers.run_test_full_random(spec, state) From 4a91c939624c1f09026908d2af4f147dd8fca909 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 12 May 2021 14:58:32 -0600 Subject: [PATCH 215/227] add epoch processing tests for inactivity udpates --- specs/altair/beacon-chain.md | 4 + .../test_process_inactivity_updates.py | 113 ++++++++++++++++++ .../altair/rewards/test_inactivity_scores.py | 5 +- .../eth2spec/test/helpers/epoch_processing.py | 3 +- .../test/helpers/inactivity_scores.py | 5 + tests/formats/epoch_processing/README.md | 3 +- 6 files changed, 126 insertions(+), 7 deletions(-) create mode 100644 tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py create mode 100644 tests/core/pyspec/eth2spec/test/helpers/inactivity_scores.py diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index a0aa458fc..3a5cec869 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -606,6 +606,10 @@ def process_justification_and_finalization(state: BeaconState) -> None: ```python def process_inactivity_updates(state: BeaconState) -> None: + # Score updates Based on previous epoch participation, skip genesis epoch + if get_current_epoch(state) == GENESIS_EPOCH: + return + for index in get_eligible_validator_indices(state): # Increase inactivity score of inactive validators if index in get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, get_previous_epoch(state)): diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py new file mode 100644 index 000000000..0efdf3e10 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py @@ -0,0 +1,113 @@ +from random import Random + +from eth2spec.test.context import spec_state_test, with_altair_and_later +from eth2spec.test.helpers.inactivity_scores import randomize_inactivity_scores +from eth2spec.test.helpers.state import ( + next_epoch_via_block, +) +from eth2spec.test.helpers.epoch_processing import ( + run_epoch_processing_with +) + + +def set_full_participation(spec, state): + full_flags = spec.ParticipationFlags(0) + for flag_index in range(len(spec.PARTICIPATION_FLAG_WEIGHTS)): + full_flags = spec.add_flag(full_flags, flag_index) + + for index in range(len(state.validators)): + state.previous_epoch_participation[index] = full_flags + + +def randomize_flags(spec, state, rng=Random(2080)): + for index in range(len(state.validators)): + # ~1/3 have bad head or bad target or not timely enough + is_timely_correct_head = rng.randint(0, 2) != 0 + flags = state.previous_epoch_participation[index] + + def set_flag(index, value): + nonlocal flags + flag = spec.ParticipationFlags(2**index) + if value: + flags |= flag + else: + flags &= 0xff ^ flag + + set_flag(spec.TIMELY_HEAD_FLAG_INDEX, is_timely_correct_head) + if is_timely_correct_head: + # If timely head, then must be timely target + set_flag(spec.TIMELY_TARGET_FLAG_INDEX, True) + # If timely head, then must be timely source + set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, True) + else: + # ~50% of remaining have bad target or not timely enough + set_flag(spec.TIMELY_TARGET_FLAG_INDEX, rng.choice([True, False])) + # ~50% of remaining have bad source or not timely enough + set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, rng.choice([True, False])) + state.previous_epoch_participation[index] = flags + + +def run_process_inactivity_updates(spec, state): + yield from run_epoch_processing_with(spec, state, 'process_inactivity_updates') + + +@with_altair_and_later +@spec_state_test +def test_genesis(spec, state): + yield from run_process_inactivity_updates(spec, state) + + +# +# Genesis epoch processing is skipped +# Thus all of following tests all go past genesis epoch to test core functionality +# + +@with_altair_and_later +@spec_state_test +def test_all_zero_inactivity_scores_empty_participation(spec, state): + state.inactivity_scores = [0] * len(state.validators) + yield from run_process_inactivity_updates(spec, state) + + +@with_altair_and_later +@spec_state_test +def test_all_zero_inactivity_scores_random_participation(spec, state): + next_epoch_via_block(spec, state) + state.inactivity_scores = [0] * len(state.validators) + randomize_flags(spec, state) + yield from run_process_inactivity_updates(spec, state) + + +@with_altair_and_later +@spec_state_test +def test_all_zero_inactivity_scores_full_participation(spec, state): + next_epoch_via_block(spec, state) + state.inactivity_scores = [0] * len(state.validators) + set_full_participation(spec, state) + yield from run_process_inactivity_updates(spec, state) + + +@with_altair_and_later +@spec_state_test +def test_random_inactivity_scores_empty_participation(spec, state): + next_epoch_via_block(spec, state) + randomize_inactivity_scores(spec, state, rng=Random(9999)) + yield from run_process_inactivity_updates(spec, state) + + +@with_altair_and_later +@spec_state_test +def test_random_inactivity_scores_random_participation(spec, state): + next_epoch_via_block(spec, state) + randomize_inactivity_scores(spec, state, rng=Random(22222)) + randomize_flags(spec, state) + yield from run_process_inactivity_updates(spec, state) + + +@with_altair_and_later +@spec_state_test +def test_random_inactivity_scores_full_participation(spec, state): + next_epoch_via_block(spec, state) + randomize_inactivity_scores(spec, state, rng=Random(33333)) + set_full_participation(spec, state) + yield from run_process_inactivity_updates(spec, state) diff --git a/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py b/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py index a523b5157..a35095278 100644 --- a/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py +++ b/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py @@ -8,14 +8,11 @@ from eth2spec.test.context import ( single_phase, low_balances, misc_balances, ) +from eth2spec.test.helpers.inactivity_scores import randomize_inactivity_scores from eth2spec.test.helpers.rewards import leaking import eth2spec.test.helpers.rewards as rewards_helpers -def randomize_inactivity_scores(spec, state, minimum=0, maximum=50000, rng=Random(4242)): - state.inactivity_scores = [rng.randint(minimum, maximum) for _ in range(len(state.validators))] - - @with_altair_and_later @spec_state_test def test_random_inactivity_scores_0(spec, state): diff --git a/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py b/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py index 3c47c4895..c783692fc 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py +++ b/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py @@ -9,6 +9,7 @@ def get_process_calls(spec): # or the old function will stick around. return [ 'process_justification_and_finalization', + 'process_inactivity_updates', # altair 'process_rewards_and_penalties', 'process_registry_updates', 'process_reveal_deadlines', # custody game @@ -26,7 +27,7 @@ def get_process_calls(spec): 'process_participation_flag_updates' if is_post_altair(spec) else ( 'process_participation_record_updates' ), - 'process_sync_committee_updates', + 'process_sync_committee_updates', # altair 'process_shard_epoch_increment' # sharding ] diff --git a/tests/core/pyspec/eth2spec/test/helpers/inactivity_scores.py b/tests/core/pyspec/eth2spec/test/helpers/inactivity_scores.py new file mode 100644 index 000000000..5c28bfc24 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/inactivity_scores.py @@ -0,0 +1,5 @@ +from random import Random + + +def randomize_inactivity_scores(spec, state, minimum=0, maximum=50000, rng=Random(4242)): + state.inactivity_scores = [rng.randint(minimum, maximum) for _ in range(len(state.validators))] diff --git a/tests/formats/epoch_processing/README.md b/tests/formats/epoch_processing/README.md index 3ac2a28c4..c917e3f75 100644 --- a/tests/formats/epoch_processing/README.md +++ b/tests/formats/epoch_processing/README.md @@ -32,9 +32,8 @@ The provided pre-state is already transitioned to just before the specific sub-t Sub-transitions: -Sub-transitions: - - `justification_and_finalization` +- `inactivity_penalty_updates` - `rewards_and_penalties` - `registry_updates` - `slashings` From a6d5b2e215a5796bfd96713e8fbbb930b353e95b Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 13 May 2021 08:22:24 -0600 Subject: [PATCH 216/227] pull state randomization functions out of rewards testing --- .../pyspec/eth2spec/test/helpers/rewards.py | 96 +-------------- .../pyspec/eth2spec/test/helpers/state.py | 110 +++++++++++++++++- 2 files changed, 115 insertions(+), 91 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index f563c70a2..47a629ac5 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -4,8 +4,11 @@ from lru import LRU from eth2spec.phase0 import spec as spec_phase0 from eth2spec.test.context import is_post_altair from eth2spec.test.helpers.attestations import cached_prepare_state_with_attestations -from eth2spec.test.helpers.deposits import mock_deposit -from eth2spec.test.helpers.state import next_epoch +from eth2spec.test.helpers.state import ( + next_epoch, + set_some_new_deposits, exit_random_validators, slash_random_validators, + randomize_state, +) from eth2spec.utils.ssz.ssz_typing import Container, uint64, List @@ -285,49 +288,6 @@ def leaking(epochs=None): return deco -def set_some_new_deposits(spec, state, rng): - num_validators = len(state.validators) - # Set ~1/10 to just recently deposited - for index in range(num_validators): - # If not already active, skip - if not spec.is_active_validator(state.validators[index], spec.get_current_epoch(state)): - continue - if rng.randrange(num_validators) < num_validators // 10: - mock_deposit(spec, state, index) - # Set ~half of selected to eligible for activation - if rng.choice([True, False]): - state.validators[index].activation_eligibility_epoch = spec.get_current_epoch(state) - - -def exit_random_validators(spec, state, rng): - if spec.get_current_epoch(state) < 5: - # Move epochs forward to allow for some validators already exited/withdrawable - for _ in range(5): - next_epoch(spec, state) - - current_epoch = spec.get_current_epoch(state) - # Exit ~1/2 of validators - for index in spec.get_active_validator_indices(state, current_epoch): - if rng.choice([True, False]): - continue - - validator = state.validators[index] - validator.exit_epoch = rng.choice([current_epoch - 1, current_epoch - 2, current_epoch - 3]) - # ~1/2 are withdrawable - if rng.choice([True, False]): - validator.withdrawable_epoch = current_epoch - else: - validator.withdrawable_epoch = current_epoch + 1 - - -def slash_random_validators(spec, state, rng): - # Slash ~1/2 of validators - for index in range(len(state.validators)): - # slash at least one validator - if index == 0 or rng.choice([True, False]): - spec.slash_validator(state, index) - - def run_test_empty(spec, state): # Do not add any attestations to state @@ -531,49 +491,5 @@ def run_test_all_balances_too_low_for_reward(spec, state): def run_test_full_random(spec, state, rng=Random(8020)): - set_some_new_deposits(spec, state, rng) - exit_random_validators(spec, state, rng) - slash_random_validators(spec, state, rng) - - cached_prepare_state_with_attestations(spec, state) - - if not is_post_altair(spec): - for pending_attestation in state.previous_epoch_attestations: - # ~1/3 have bad target - if rng.randint(0, 2) == 0: - pending_attestation.data.target.root = b'\x55' * 32 - # ~1/3 have bad head - if rng.randint(0, 2) == 0: - pending_attestation.data.beacon_block_root = b'\x66' * 32 - # ~50% participation - pending_attestation.aggregation_bits = [rng.choice([True, False]) - for _ in pending_attestation.aggregation_bits] - # Random inclusion delay - pending_attestation.inclusion_delay = rng.randint(1, spec.SLOTS_PER_EPOCH) - else: - for index in range(len(state.validators)): - # ~1/3 have bad head or bad target or not timely enough - is_timely_correct_head = rng.randint(0, 2) != 0 - flags = state.previous_epoch_participation[index] - - def set_flag(index, value): - nonlocal flags - flag = spec.ParticipationFlags(2**index) - if value: - flags |= flag - else: - flags &= 0xff ^ flag - - set_flag(spec.TIMELY_HEAD_FLAG_INDEX, is_timely_correct_head) - if is_timely_correct_head: - # If timely head, then must be timely target - set_flag(spec.TIMELY_TARGET_FLAG_INDEX, True) - # If timely head, then must be timely source - set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, True) - else: - # ~50% of remaining have bad target or not timely enough - set_flag(spec.TIMELY_TARGET_FLAG_INDEX, rng.choice([True, False])) - # ~50% of remaining have bad source or not timely enough - set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, rng.choice([True, False])) - state.previous_epoch_participation[index] = flags + randomize_state(spec, state, rng) yield from run_deltas(spec, state) diff --git a/tests/core/pyspec/eth2spec/test/helpers/state.py b/tests/core/pyspec/eth2spec/test/helpers/state.py index d61df7610..610f4871e 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/state.py +++ b/tests/core/pyspec/eth2spec/test/helpers/state.py @@ -1,5 +1,9 @@ -from eth2spec.test.context import expect_assertion_error +from random import Random + +from eth2spec.test.context import expect_assertion_error, is_post_altair from eth2spec.test.helpers.block import apply_empty_block, sign_block, transition_unsigned_block +from eth2spec.test.helpers.deposits import mock_deposit +from eth2spec.test.helpers.attestations import cached_prepare_state_with_attestations def get_balance(state, index): @@ -84,3 +88,107 @@ def state_transition_and_sign_block(spec, state, block, expect_fail=False): transition_unsigned_block(spec, state, block) block.state_root = state.hash_tree_root() return sign_block(spec, state, block) + + +def set_some_new_deposits(spec, state, rng): + num_validators = len(state.validators) + # Set ~1/10 to just recently deposited + for index in range(num_validators): + # If not already active, skip + if not spec.is_active_validator(state.validators[index], spec.get_current_epoch(state)): + continue + if rng.randrange(num_validators) < num_validators // 10: + mock_deposit(spec, state, index) + # Set ~half of selected to eligible for activation + if rng.choice([True, False]): + state.validators[index].activation_eligibility_epoch = spec.get_current_epoch(state) + + +def exit_random_validators(spec, state, rng): + if spec.get_current_epoch(state) < 5: + # Move epochs forward to allow for some validators already exited/withdrawable + for _ in range(5): + next_epoch(spec, state) + + current_epoch = spec.get_current_epoch(state) + # Exit ~1/2 of validators + for index in spec.get_active_validator_indices(state, current_epoch): + if rng.choice([True, False]): + continue + + validator = state.validators[index] + validator.exit_epoch = rng.choice([current_epoch - 1, current_epoch - 2, current_epoch - 3]) + # ~1/2 are withdrawable + if rng.choice([True, False]): + validator.withdrawable_epoch = current_epoch + else: + validator.withdrawable_epoch = current_epoch + 1 + + +def slash_random_validators(spec, state, rng): + # Slash ~1/2 of validators + for index in range(len(state.validators)): + # slash at least one validator + if index == 0 or rng.choice([True, False]): + spec.slash_validator(state, index) + + +def randomize_epoch_participation(spec, state, epoch, rng): + assert epoch in (spec.get_current_epoch(state), spec.get_previous_epoch(state)) + if not is_post_altair(spec): + if epoch == spec.get_current_epoch(state): + pending_attestations = state.current_epoch_attestations + else: + pending_attestations = state.previous_epoch_attestations + for pending_attestation in pending_attestations: + # ~1/3 have bad target + if rng.randint(0, 2) == 0: + pending_attestation.data.target.root = b'\x55' * 32 + # ~1/3 have bad head + if rng.randint(0, 2) == 0: + pending_attestation.data.beacon_block_root = b'\x66' * 32 + # ~50% participation + pending_attestation.aggregation_bits = [rng.choice([True, False]) + for _ in pending_attestation.aggregation_bits] + # Random inclusion delay + pending_attestation.inclusion_delay = rng.randint(1, spec.SLOTS_PER_EPOCH) + else: + if epoch == spec.get_current_epoch(state): + epoch_participation = state.current_epoch_participation + else: + epoch_participation = state.previous_epoch_participation + for index in range(len(state.validators)): + # ~1/3 have bad head or bad target or not timely enough + is_timely_correct_head = rng.randint(0, 2) != 0 + flags = epoch_participation[index] + + def set_flag(index, value): + nonlocal flags + flag = spec.ParticipationFlags(2**index) + if value: + flags |= flag + else: + flags &= 0xff ^ flag + + set_flag(spec.TIMELY_HEAD_FLAG_INDEX, is_timely_correct_head) + if is_timely_correct_head: + # If timely head, then must be timely target + set_flag(spec.TIMELY_TARGET_FLAG_INDEX, True) + # If timely head, then must be timely source + set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, True) + else: + # ~50% of remaining have bad target or not timely enough + set_flag(spec.TIMELY_TARGET_FLAG_INDEX, rng.choice([True, False])) + # ~50% of remaining have bad source or not timely enough + set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, rng.choice([True, False])) + epoch_participation[index] = flags + + +def randomize_state(spec, state, rng=Random(8020)): + set_some_new_deposits(spec, state, rng) + exit_random_validators(spec, state, rng) + slash_random_validators(spec, state, rng) + + cached_prepare_state_with_attestations(spec, state) + randomize_epoch_participation(spec, state, spec.get_previous_epoch(state), rng) + randomize_epoch_participation(spec, state, spec.get_current_epoch(state), rng) From 57e13fd5b2cd0c57ab0e693207348d002684e726 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 13 May 2021 10:59:29 -0600 Subject: [PATCH 217/227] add INACTIVITY_SCORE_RECOVERY_RATE to configs --- configs/mainnet/altair.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/configs/mainnet/altair.yaml b/configs/mainnet/altair.yaml index e387fc87a..a6761b142 100644 --- a/configs/mainnet/altair.yaml +++ b/configs/mainnet/altair.yaml @@ -22,6 +22,8 @@ EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 512 # --------------------------------------------------------------- # 2**2 (= 4) INACTIVITY_SCORE_BIAS: 4 +# 2**4 (= 16) +INACTIVITY_SCORE_RECOVERY_RATE: 16 # Signature domains From a52565aa8141fed2ae4424436cd236130ff529c6 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 13 May 2021 13:21:20 -0600 Subject: [PATCH 218/227] add random altair fork tests for better translation coverage + pr review --- specs/altair/fork.md | 7 +- ...test_fork.py => test_altair_fork_basic.py} | 45 +------ .../altair/fork/test_altair_fork_random.py | 120 ++++++++++++++++++ .../eth2spec/test/helpers/altair/fork.py | 42 ++++++ .../pyspec/eth2spec/test/helpers/random.py | 113 +++++++++++++++++ .../pyspec/eth2spec/test/helpers/rewards.py | 6 +- .../pyspec/eth2spec/test/helpers/state.py | 110 +--------------- 7 files changed, 288 insertions(+), 155 deletions(-) rename tests/core/pyspec/eth2spec/test/altair/fork/{test_fork.py => test_altair_fork_basic.py} (66%) create mode 100644 tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_random.py create mode 100644 tests/core/pyspec/eth2spec/test/helpers/altair/fork.py create mode 100644 tests/core/pyspec/eth2spec/test/helpers/random.py diff --git a/specs/altair/fork.md b/specs/altair/fork.md index d14e8ae8d..b4c4e3a31 100644 --- a/specs/altair/fork.md +++ b/specs/altair/fork.md @@ -45,7 +45,7 @@ Care must be taken when transitioning through the fork boundary as implementatio In particular, the outer `state_transition` function defined in the Phase 0 spec will not expose the precise fork slot to execute the upgrade in the presence of skipped slots at the fork boundary. Instead the logic must be within `process_slots`. ```python -def translate_participation(state: BeaconState, pending_attestations: Sequence[PendingAttestation]) -> None: +def translate_participation(state: BeaconState, pending_attestations: Sequence[phase0.PendingAttestation]) -> None: for attestation in pending_attestations: data = attestation.data inclusion_delay = attestation.inclusion_delay @@ -55,9 +55,8 @@ def translate_participation(state: BeaconState, pending_attestations: Sequence[P # Apply flags to all attesting validators epoch_participation = state.previous_epoch_participation for index in get_attesting_indices(state, data, attestation.aggregation_bits): - for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): - if flag_index in participation_flag_indices and not has_flag(epoch_participation[index], flag_index): - epoch_participation[index] = add_flag(epoch_participation[index], flag_index) + for flag_index in participation_flag_indices: + epoch_participation[index] = add_flag(epoch_participation[index], flag_index) def upgrade_to_altair(pre: phase0.BeaconState) -> BeaconState: diff --git a/tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py b/tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_basic.py similarity index 66% rename from tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py rename to tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_basic.py index 1ad39209c..bc082026e 100644 --- a/tests/core/pyspec/eth2spec/test/altair/fork/test_fork.py +++ b/tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_basic.py @@ -14,47 +14,10 @@ from eth2spec.test.helpers.state import ( next_epoch, next_epoch_via_block, ) - - -ALTAIR_FORK_TEST_META_TAGS = { - 'fork': 'altair', -} - - -def run_fork_test(post_spec, pre_state): - yield 'pre', pre_state - - post_state = post_spec.upgrade_to_altair(pre_state) - - # Stable fields - stable_fields = [ - 'genesis_time', 'genesis_validators_root', 'slot', - # History - 'latest_block_header', 'block_roots', 'state_roots', 'historical_roots', - # Eth1 - 'eth1_data', 'eth1_data_votes', 'eth1_deposit_index', - # Registry - 'validators', 'balances', - # Randomness - 'randao_mixes', - # Slashings - 'slashings', - # Finality - 'justification_bits', 'previous_justified_checkpoint', 'current_justified_checkpoint', 'finalized_checkpoint', - ] - for field in stable_fields: - assert getattr(pre_state, field) == getattr(post_state, field) - - # Modified fields - modified_fields = ['fork'] - for field in modified_fields: - assert getattr(pre_state, field) != getattr(post_state, field) - - assert pre_state.fork.current_version == post_state.fork.previous_version - assert post_state.fork.current_version == post_spec.ALTAIR_FORK_VERSION - assert post_state.fork.epoch == post_spec.get_current_epoch(post_state) - - yield 'post', post_state +from eth2spec.test.helpers.altair.fork import ( + ALTAIR_FORK_TEST_META_TAGS, + run_fork_test, +) @with_phases(phases=[PHASE0], other_phases=[ALTAIR]) diff --git a/tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_random.py b/tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_random.py new file mode 100644 index 000000000..ba350bd68 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/altair/fork/test_altair_fork_random.py @@ -0,0 +1,120 @@ +from random import Random + +from eth2spec.test.context import ( + with_phases, + with_custom_state, + with_configs, + spec_test, with_state, + low_balances, misc_balances, large_validator_set, +) +from eth2spec.test.utils import with_meta_tags +from eth2spec.test.helpers.constants import ( + PHASE0, ALTAIR, + MINIMAL, +) +from eth2spec.test.helpers.altair.fork import ( + ALTAIR_FORK_TEST_META_TAGS, + run_fork_test, +) +from eth2spec.test.helpers.random import ( + randomize_state, + randomize_attestation_participation, +) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_state +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_0(spec, phases, state): + randomize_state(spec, state, rng=Random(1010)) + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_state +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_1(spec, phases, state): + randomize_state(spec, state, rng=Random(2020)) + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_state +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_2(spec, phases, state): + randomize_state(spec, state, rng=Random(3030)) + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_state +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_3(spec, phases, state): + randomize_state(spec, state, rng=Random(4040)) + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_state +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_duplicate_attestations(spec, phases, state): + randomize_state(spec, state, rng=Random(1111)) + # Note: `run_fork_test` empties `current_epoch_attestations` + state.previous_epoch_attestations = state.previous_epoch_attestations + state.previous_epoch_attestations + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_state +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_mismatched_attestations(spec, phases, state): + # Create a random state + randomize_state(spec, state, rng=Random(2222)) + + # Now make two copies + state_0 = state.copy() + state_1 = state.copy() + + # Randomize attestation participation of both + randomize_attestation_participation(spec, state_0, rng=Random(3333)) + randomize_attestation_participation(spec, state_1, rng=Random(4444)) + + # Note: `run_fork_test` empties `current_epoch_attestations` + # Use pending attestations from both random states in a single state for testing + state_0.previous_epoch_attestations = state_0.previous_epoch_attestations + state_1.previous_epoch_attestations + yield from run_fork_test(phases[ALTAIR], state_0) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_low_balances(spec, phases, state): + randomize_state(spec, state, rng=Random(5050)) + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@spec_test +@with_custom_state(balances_fn=misc_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_misc_balances(spec, phases, state): + randomize_state(spec, state, rng=Random(6060)) + yield from run_fork_test(phases[ALTAIR], state) + + +@with_phases(phases=[PHASE0], other_phases=[ALTAIR]) +@with_configs([MINIMAL], + reason="mainnet config leads to larger validator set than limit of public/private keys pre-generated") +@spec_test +@with_custom_state(balances_fn=large_validator_set, threshold_fn=lambda spec: spec.EJECTION_BALANCE) +@with_meta_tags(ALTAIR_FORK_TEST_META_TAGS) +def test_altair_fork_random_large_validator_set(spec, phases, state): + randomize_state(spec, state, rng=Random(7070)) + yield from run_fork_test(phases[ALTAIR], state) diff --git a/tests/core/pyspec/eth2spec/test/helpers/altair/fork.py b/tests/core/pyspec/eth2spec/test/helpers/altair/fork.py new file mode 100644 index 000000000..b1074c881 --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/altair/fork.py @@ -0,0 +1,42 @@ +ALTAIR_FORK_TEST_META_TAGS = { + 'fork': 'altair', +} + + +def run_fork_test(post_spec, pre_state): + # Clean up state to be more realistic + pre_state.current_epoch_attestations = [] + + yield 'pre', pre_state + + post_state = post_spec.upgrade_to_altair(pre_state) + + # Stable fields + stable_fields = [ + 'genesis_time', 'genesis_validators_root', 'slot', + # History + 'latest_block_header', 'block_roots', 'state_roots', 'historical_roots', + # Eth1 + 'eth1_data', 'eth1_data_votes', 'eth1_deposit_index', + # Registry + 'validators', 'balances', + # Randomness + 'randao_mixes', + # Slashings + 'slashings', + # Finality + 'justification_bits', 'previous_justified_checkpoint', 'current_justified_checkpoint', 'finalized_checkpoint', + ] + for field in stable_fields: + assert getattr(pre_state, field) == getattr(post_state, field) + + # Modified fields + modified_fields = ['fork'] + for field in modified_fields: + assert getattr(pre_state, field) != getattr(post_state, field) + + assert pre_state.fork.current_version == post_state.fork.previous_version + assert post_state.fork.current_version == post_spec.ALTAIR_FORK_VERSION + assert post_state.fork.epoch == post_spec.get_current_epoch(post_state) + + yield 'post', post_state diff --git a/tests/core/pyspec/eth2spec/test/helpers/random.py b/tests/core/pyspec/eth2spec/test/helpers/random.py new file mode 100644 index 000000000..5b5e419ba --- /dev/null +++ b/tests/core/pyspec/eth2spec/test/helpers/random.py @@ -0,0 +1,113 @@ +from random import Random + +from eth2spec.test.helpers.attestations import cached_prepare_state_with_attestations +from eth2spec.test.context import is_post_altair +from eth2spec.test.helpers.deposits import mock_deposit +from eth2spec.test.helpers.state import next_epoch + + +def set_some_new_deposits(spec, state, rng): + num_validators = len(state.validators) + # Set ~1/10 to just recently deposited + for index in range(num_validators): + # If not already active, skip + if not spec.is_active_validator(state.validators[index], spec.get_current_epoch(state)): + continue + if rng.randrange(num_validators) < num_validators // 10: + mock_deposit(spec, state, index) + # Set ~half of selected to eligible for activation + if rng.choice([True, False]): + state.validators[index].activation_eligibility_epoch = spec.get_current_epoch(state) + + +def exit_random_validators(spec, state, rng): + if spec.get_current_epoch(state) < 5: + # Move epochs forward to allow for some validators already exited/withdrawable + for _ in range(5): + next_epoch(spec, state) + + current_epoch = spec.get_current_epoch(state) + # Exit ~1/2 of validators + for index in spec.get_active_validator_indices(state, current_epoch): + if rng.choice([True, False]): + continue + + validator = state.validators[index] + validator.exit_epoch = rng.choice([current_epoch - 1, current_epoch - 2, current_epoch - 3]) + # ~1/2 are withdrawable + if rng.choice([True, False]): + validator.withdrawable_epoch = current_epoch + else: + validator.withdrawable_epoch = current_epoch + 1 + + +def slash_random_validators(spec, state, rng): + # Slash ~1/2 of validators + for index in range(len(state.validators)): + # slash at least one validator + if index == 0 or rng.choice([True, False]): + spec.slash_validator(state, index) + + +def randomize_epoch_participation(spec, state, epoch, rng): + assert epoch in (spec.get_current_epoch(state), spec.get_previous_epoch(state)) + if not is_post_altair(spec): + if epoch == spec.get_current_epoch(state): + pending_attestations = state.current_epoch_attestations + else: + pending_attestations = state.previous_epoch_attestations + for pending_attestation in pending_attestations: + # ~1/3 have bad target + if rng.randint(0, 2) == 0: + pending_attestation.data.target.root = b'\x55' * 32 + # ~1/3 have bad head + if rng.randint(0, 2) == 0: + pending_attestation.data.beacon_block_root = b'\x66' * 32 + # ~50% participation + pending_attestation.aggregation_bits = [rng.choice([True, False]) + for _ in pending_attestation.aggregation_bits] + # Random inclusion delay + pending_attestation.inclusion_delay = rng.randint(1, spec.SLOTS_PER_EPOCH) + else: + if epoch == spec.get_current_epoch(state): + epoch_participation = state.current_epoch_participation + else: + epoch_participation = state.previous_epoch_participation + for index in range(len(state.validators)): + # ~1/3 have bad head or bad target or not timely enough + is_timely_correct_head = rng.randint(0, 2) != 0 + flags = epoch_participation[index] + + def set_flag(index, value): + nonlocal flags + flag = spec.ParticipationFlags(2**index) + if value: + flags |= flag + else: + flags &= 0xff ^ flag + + set_flag(spec.TIMELY_HEAD_FLAG_INDEX, is_timely_correct_head) + if is_timely_correct_head: + # If timely head, then must be timely target + set_flag(spec.TIMELY_TARGET_FLAG_INDEX, True) + # If timely head, then must be timely source + set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, True) + else: + # ~50% of remaining have bad target or not timely enough + set_flag(spec.TIMELY_TARGET_FLAG_INDEX, rng.choice([True, False])) + # ~50% of remaining have bad source or not timely enough + set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, rng.choice([True, False])) + epoch_participation[index] = flags + + +def randomize_attestation_participation(spec, state, rng=Random(8020)): + cached_prepare_state_with_attestations(spec, state) + randomize_epoch_participation(spec, state, spec.get_previous_epoch(state), rng) + randomize_epoch_participation(spec, state, spec.get_current_epoch(state), rng) + + +def randomize_state(spec, state, rng=Random(8020)): + set_some_new_deposits(spec, state, rng) + exit_random_validators(spec, state, rng) + slash_random_validators(spec, state, rng) + randomize_attestation_participation(spec, state, rng) diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index 47a629ac5..6bf922750 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -3,12 +3,16 @@ from lru import LRU from eth2spec.phase0 import spec as spec_phase0 from eth2spec.test.context import is_post_altair -from eth2spec.test.helpers.attestations import cached_prepare_state_with_attestations from eth2spec.test.helpers.state import ( next_epoch, +) +from eth2spec.test.helpers.random import ( set_some_new_deposits, exit_random_validators, slash_random_validators, randomize_state, ) +from eth2spec.test.helpers.attestations import ( + cached_prepare_state_with_attestations, +) from eth2spec.utils.ssz.ssz_typing import Container, uint64, List diff --git a/tests/core/pyspec/eth2spec/test/helpers/state.py b/tests/core/pyspec/eth2spec/test/helpers/state.py index 610f4871e..d61df7610 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/state.py +++ b/tests/core/pyspec/eth2spec/test/helpers/state.py @@ -1,9 +1,5 @@ -from random import Random - -from eth2spec.test.context import expect_assertion_error, is_post_altair +from eth2spec.test.context import expect_assertion_error from eth2spec.test.helpers.block import apply_empty_block, sign_block, transition_unsigned_block -from eth2spec.test.helpers.deposits import mock_deposit -from eth2spec.test.helpers.attestations import cached_prepare_state_with_attestations def get_balance(state, index): @@ -88,107 +84,3 @@ def state_transition_and_sign_block(spec, state, block, expect_fail=False): transition_unsigned_block(spec, state, block) block.state_root = state.hash_tree_root() return sign_block(spec, state, block) - - -def set_some_new_deposits(spec, state, rng): - num_validators = len(state.validators) - # Set ~1/10 to just recently deposited - for index in range(num_validators): - # If not already active, skip - if not spec.is_active_validator(state.validators[index], spec.get_current_epoch(state)): - continue - if rng.randrange(num_validators) < num_validators // 10: - mock_deposit(spec, state, index) - # Set ~half of selected to eligible for activation - if rng.choice([True, False]): - state.validators[index].activation_eligibility_epoch = spec.get_current_epoch(state) - - -def exit_random_validators(spec, state, rng): - if spec.get_current_epoch(state) < 5: - # Move epochs forward to allow for some validators already exited/withdrawable - for _ in range(5): - next_epoch(spec, state) - - current_epoch = spec.get_current_epoch(state) - # Exit ~1/2 of validators - for index in spec.get_active_validator_indices(state, current_epoch): - if rng.choice([True, False]): - continue - - validator = state.validators[index] - validator.exit_epoch = rng.choice([current_epoch - 1, current_epoch - 2, current_epoch - 3]) - # ~1/2 are withdrawable - if rng.choice([True, False]): - validator.withdrawable_epoch = current_epoch - else: - validator.withdrawable_epoch = current_epoch + 1 - - -def slash_random_validators(spec, state, rng): - # Slash ~1/2 of validators - for index in range(len(state.validators)): - # slash at least one validator - if index == 0 or rng.choice([True, False]): - spec.slash_validator(state, index) - - -def randomize_epoch_participation(spec, state, epoch, rng): - assert epoch in (spec.get_current_epoch(state), spec.get_previous_epoch(state)) - if not is_post_altair(spec): - if epoch == spec.get_current_epoch(state): - pending_attestations = state.current_epoch_attestations - else: - pending_attestations = state.previous_epoch_attestations - for pending_attestation in pending_attestations: - # ~1/3 have bad target - if rng.randint(0, 2) == 0: - pending_attestation.data.target.root = b'\x55' * 32 - # ~1/3 have bad head - if rng.randint(0, 2) == 0: - pending_attestation.data.beacon_block_root = b'\x66' * 32 - # ~50% participation - pending_attestation.aggregation_bits = [rng.choice([True, False]) - for _ in pending_attestation.aggregation_bits] - # Random inclusion delay - pending_attestation.inclusion_delay = rng.randint(1, spec.SLOTS_PER_EPOCH) - else: - if epoch == spec.get_current_epoch(state): - epoch_participation = state.current_epoch_participation - else: - epoch_participation = state.previous_epoch_participation - for index in range(len(state.validators)): - # ~1/3 have bad head or bad target or not timely enough - is_timely_correct_head = rng.randint(0, 2) != 0 - flags = epoch_participation[index] - - def set_flag(index, value): - nonlocal flags - flag = spec.ParticipationFlags(2**index) - if value: - flags |= flag - else: - flags &= 0xff ^ flag - - set_flag(spec.TIMELY_HEAD_FLAG_INDEX, is_timely_correct_head) - if is_timely_correct_head: - # If timely head, then must be timely target - set_flag(spec.TIMELY_TARGET_FLAG_INDEX, True) - # If timely head, then must be timely source - set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, True) - else: - # ~50% of remaining have bad target or not timely enough - set_flag(spec.TIMELY_TARGET_FLAG_INDEX, rng.choice([True, False])) - # ~50% of remaining have bad source or not timely enough - set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, rng.choice([True, False])) - epoch_participation[index] = flags - - -def randomize_state(spec, state, rng=Random(8020)): - set_some_new_deposits(spec, state, rng) - exit_random_validators(spec, state, rng) - slash_random_validators(spec, state, rng) - - cached_prepare_state_with_attestations(spec, state) - randomize_epoch_participation(spec, state, spec.get_previous_epoch(state), rng) - randomize_epoch_participation(spec, state, spec.get_current_epoch(state), rng) From 699a3f837eea9774971b3a897de81b6ec7a680e5 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 13 May 2021 13:38:43 -0600 Subject: [PATCH 219/227] pr feedback --- configs/minimal/altair.yaml | 2 ++ specs/altair/beacon-chain.md | 2 +- .../epoch_processing/test_process_inactivity_updates.py | 1 + .../test/altair/rewards/test_inactivity_scores.py | 2 +- tests/core/pyspec/eth2spec/test/helpers/rewards.py | 1 - .../pyspec/eth2spec/test/phase0/rewards/test_random.py | 8 ++++---- tests/formats/epoch_processing/README.md | 1 + 7 files changed, 10 insertions(+), 7 deletions(-) diff --git a/configs/minimal/altair.yaml b/configs/minimal/altair.yaml index a66b5c7ca..f9b8401e1 100644 --- a/configs/minimal/altair.yaml +++ b/configs/minimal/altair.yaml @@ -22,6 +22,8 @@ EPOCHS_PER_SYNC_COMMITTEE_PERIOD: 8 # --------------------------------------------------------------- # 2**2 (= 4) INACTIVITY_SCORE_BIAS: 4 +# 2**4 (= 16) +INACTIVITY_SCORE_RECOVERY_RATE: 16 # Signature domains diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 3a5cec869..5ce01e6b0 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -606,7 +606,7 @@ def process_justification_and_finalization(state: BeaconState) -> None: ```python def process_inactivity_updates(state: BeaconState) -> None: - # Score updates Based on previous epoch participation, skip genesis epoch + # Score updates based on previous epoch participation, skip genesis epoch if get_current_epoch(state) == GENESIS_EPOCH: return diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py index 0efdf3e10..7e501dedc 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py @@ -65,6 +65,7 @@ def test_genesis(spec, state): @with_altair_and_later @spec_state_test def test_all_zero_inactivity_scores_empty_participation(spec, state): + next_epoch_via_block(spec, state) state.inactivity_scores = [0] * len(state.validators) yield from run_process_inactivity_updates(spec, state) diff --git a/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py b/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py index a35095278..9eca9a92a 100644 --- a/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py +++ b/tests/core/pyspec/eth2spec/test/altair/rewards/test_inactivity_scores.py @@ -54,7 +54,7 @@ def test_random_inactivity_scores_low_balances_0(spec, state): @with_altair_and_later -@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: 0) +@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) @spec_test @single_phase def test_random_inactivity_scores_low_balances_1(spec, state): diff --git a/tests/core/pyspec/eth2spec/test/helpers/rewards.py b/tests/core/pyspec/eth2spec/test/helpers/rewards.py index f563c70a2..fa412c974 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/rewards.py +++ b/tests/core/pyspec/eth2spec/test/helpers/rewards.py @@ -265,7 +265,6 @@ _cache_dict = LRU(size=10) def leaking(epochs=None): - def deco(fn): def entry(*args, spec, state, **kw): # If the pre-state is not already known in the LRU, then take it, diff --git a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py index 51bd27c14..b1af64a77 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py +++ b/tests/core/pyspec/eth2spec/test/phase0/rewards/test_random.py @@ -40,15 +40,15 @@ def test_full_random_3(spec, state): @spec_test @single_phase def test_full_random_low_balances_0(spec, state): - yield from rewards_helpers.run_test_full_random(spec, state) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(5050)) @with_all_phases -@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: 0) +@with_custom_state(balances_fn=low_balances, threshold_fn=lambda spec: spec.EJECTION_BALANCE) @spec_test @single_phase def test_full_random_low_balances_1(spec, state): - yield from rewards_helpers.run_test_full_random(spec, state) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(6060)) @with_all_phases @@ -56,4 +56,4 @@ def test_full_random_low_balances_1(spec, state): @spec_test @single_phase def test_full_random_misc_balances(spec, state): - yield from rewards_helpers.run_test_full_random(spec, state) + yield from rewards_helpers.run_test_full_random(spec, state, rng=Random(7070)) diff --git a/tests/formats/epoch_processing/README.md b/tests/formats/epoch_processing/README.md index c917e3f75..d9abcaf98 100644 --- a/tests/formats/epoch_processing/README.md +++ b/tests/formats/epoch_processing/README.md @@ -43,5 +43,6 @@ Sub-transitions: - `randao_mixes_reset` - `historical_roots_update` - `participation_record_updates` +- `sync_committee_updates` The resulting state should match the expected `post` state. From 049210bd8aea9c5015b2313534c0522fa7e845ea Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 13 May 2021 13:44:41 -0600 Subject: [PATCH 220/227] utilize new randomize functions in process_inactivity_updates testing --- .../test_process_inactivity_updates.py | 40 ++++--------------- 1 file changed, 8 insertions(+), 32 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py index 7e501dedc..1cc8cb9ae 100644 --- a/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py +++ b/tests/core/pyspec/eth2spec/test/altair/epoch_processing/test_process_inactivity_updates.py @@ -8,6 +8,9 @@ from eth2spec.test.helpers.state import ( from eth2spec.test.helpers.epoch_processing import ( run_epoch_processing_with ) +from eth2spec.test.helpers.random import ( + randomize_attestation_participation, +) def set_full_participation(spec, state): @@ -16,37 +19,10 @@ def set_full_participation(spec, state): full_flags = spec.add_flag(full_flags, flag_index) for index in range(len(state.validators)): + state.current_epoch_participation[index] = full_flags state.previous_epoch_participation[index] = full_flags -def randomize_flags(spec, state, rng=Random(2080)): - for index in range(len(state.validators)): - # ~1/3 have bad head or bad target or not timely enough - is_timely_correct_head = rng.randint(0, 2) != 0 - flags = state.previous_epoch_participation[index] - - def set_flag(index, value): - nonlocal flags - flag = spec.ParticipationFlags(2**index) - if value: - flags |= flag - else: - flags &= 0xff ^ flag - - set_flag(spec.TIMELY_HEAD_FLAG_INDEX, is_timely_correct_head) - if is_timely_correct_head: - # If timely head, then must be timely target - set_flag(spec.TIMELY_TARGET_FLAG_INDEX, True) - # If timely head, then must be timely source - set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, True) - else: - # ~50% of remaining have bad target or not timely enough - set_flag(spec.TIMELY_TARGET_FLAG_INDEX, rng.choice([True, False])) - # ~50% of remaining have bad source or not timely enough - set_flag(spec.TIMELY_SOURCE_FLAG_INDEX, rng.choice([True, False])) - state.previous_epoch_participation[index] = flags - - def run_process_inactivity_updates(spec, state): yield from run_epoch_processing_with(spec, state, 'process_inactivity_updates') @@ -75,7 +51,7 @@ def test_all_zero_inactivity_scores_empty_participation(spec, state): def test_all_zero_inactivity_scores_random_participation(spec, state): next_epoch_via_block(spec, state) state.inactivity_scores = [0] * len(state.validators) - randomize_flags(spec, state) + randomize_attestation_participation(spec, state, rng=Random(5555)) yield from run_process_inactivity_updates(spec, state) @@ -83,8 +59,8 @@ def test_all_zero_inactivity_scores_random_participation(spec, state): @spec_state_test def test_all_zero_inactivity_scores_full_participation(spec, state): next_epoch_via_block(spec, state) - state.inactivity_scores = [0] * len(state.validators) set_full_participation(spec, state) + state.inactivity_scores = [0] * len(state.validators) yield from run_process_inactivity_updates(spec, state) @@ -100,8 +76,8 @@ def test_random_inactivity_scores_empty_participation(spec, state): @spec_state_test def test_random_inactivity_scores_random_participation(spec, state): next_epoch_via_block(spec, state) + randomize_attestation_participation(spec, state, rng=Random(22222)) randomize_inactivity_scores(spec, state, rng=Random(22222)) - randomize_flags(spec, state) yield from run_process_inactivity_updates(spec, state) @@ -109,6 +85,6 @@ def test_random_inactivity_scores_random_participation(spec, state): @spec_state_test def test_random_inactivity_scores_full_participation(spec, state): next_epoch_via_block(spec, state) - randomize_inactivity_scores(spec, state, rng=Random(33333)) set_full_participation(spec, state) + randomize_inactivity_scores(spec, state, rng=Random(33333)) yield from run_process_inactivity_updates(spec, state) From d3160ba23a3fd27aa69004f381d37ff0db589522 Mon Sep 17 00:00:00 2001 From: protolambda Date: Fri, 14 May 2021 01:07:22 +0200 Subject: [PATCH 221/227] update ExecutionEngine protocol arg references --- specs/merge/beacon-chain.md | 2 +- specs/merge/validator.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/merge/beacon-chain.md b/specs/merge/beacon-chain.md index af633e18a..85953a547 100644 --- a/specs/merge/beacon-chain.md +++ b/specs/merge/beacon-chain.md @@ -150,7 +150,7 @@ The following methods are added to the `ExecutionEngine` protocol for use in the #### `new_block` -Verifies the given `ExecutionPayload` with respect to execution state transition, and persists changes if valid. +Verifies the given `execution_payload` with respect to execution state transition, and persists changes if valid. The body of this function is implementation dependent. The Consensus API may be used to implement this with an external execution engine. diff --git a/specs/merge/validator.md b/specs/merge/validator.md index 7a1b61439..c4c396059 100644 --- a/specs/merge/validator.md +++ b/specs/merge/validator.md @@ -43,8 +43,8 @@ The following methods are added to the `ExecutionEngine` protocol for use as a v #### `assemble_block` -Produces a new instance of an execution payload, with the specified timestamp, -on top of the execution payload chain tip identified by `block_head`. +Produces a new instance of an execution payload, with the specified `timestamp`, +on top of the execution payload chain tip identified by `block_hash`. The body of this function is implementation dependent. The Consensus API may be used to implement this with an external execution engine. From 1310105174368d10f21948be410ca0525e49366b Mon Sep 17 00:00:00 2001 From: terence tsao Date: Thu, 13 May 2021 16:22:28 -0700 Subject: [PATCH 222/227] Return non-duplicated sync committee subnets --- specs/altair/validator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 4aee98ad4..e6e00a674 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -292,7 +292,7 @@ The validator broadcasts the assembled signature to the assigned subnet, the `sy The `subnet_id` is derived from the position in the sync committee such that the sync committee is divided into "subcommittees". `subnet_id` can be computed via `compute_subnets_for_sync_committee(state, validator_index)` where `state` is a `BeaconState` during the matching sync committee period. -*Note*: This function returns multiple subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. +*Note*: This function returns multiple non-duplicated subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. ```python def compute_subnets_for_sync_committee(state: BeaconState, validator_index: ValidatorIndex) -> Sequence[uint64]: @@ -304,10 +304,10 @@ def compute_subnets_for_sync_committee(state: BeaconState, validator_index: Vali target_pubkey = state.validators[validator_index].pubkey sync_committee_indices = [index for index, pubkey in enumerate(sync_committee.pubkeys) if pubkey == target_pubkey] - return [ + return set([ uint64(index // (SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT)) for index in sync_committee_indices - ] + ]) ``` *Note*: Subnet assignment does not change during the duration of a validator's assignment to a given sync committee. From 5dd29b6659c0f4bc05b01390d577d8d9f908636a Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 14 May 2021 06:15:48 -0600 Subject: [PATCH 223/227] fix tests and minor copy edit --- specs/altair/validator.md | 4 ++-- .../test/altair/unittests/validator/test_validator.py | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index e6e00a674..3b3362b22 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -292,10 +292,10 @@ The validator broadcasts the assembled signature to the assigned subnet, the `sy The `subnet_id` is derived from the position in the sync committee such that the sync committee is divided into "subcommittees". `subnet_id` can be computed via `compute_subnets_for_sync_committee(state, validator_index)` where `state` is a `BeaconState` during the matching sync committee period. -*Note*: This function returns multiple non-duplicated subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. +*Note*: This function returns multiple deduplicated subnets if a given validator index is included multiple times in a given sync committee across multiple subcommittees. ```python -def compute_subnets_for_sync_committee(state: BeaconState, validator_index: ValidatorIndex) -> Sequence[uint64]: +def compute_subnets_for_sync_committee(state: BeaconState, validator_index: ValidatorIndex) -> Set[uint64]: next_slot_epoch = compute_epoch_at_slot(Slot(state.slot + 1)) if compute_sync_committee_period(get_current_epoch(state)) == compute_sync_committee_period(next_slot_epoch): sync_committee = state.current_sync_committee diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index cefaaf694..18fb730d7 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -149,9 +149,10 @@ def _subnet_for_sync_committee_index(spec, i): def _get_expected_subnets_by_pubkey(sync_committee_members): - expected_subnets_by_pubkey = defaultdict(list) + # Build deduplicated set for each pubkey + expected_subnets_by_pubkey = defaultdict(set) for (subnet, pubkey) in sync_committee_members: - expected_subnets_by_pubkey[pubkey].append(subnet) + expected_subnets_by_pubkey[pubkey].add(subnet) return expected_subnets_by_pubkey From 160b704f4a8619f8d503de43779793f43f6d7107 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 14 May 2021 06:19:29 -0600 Subject: [PATCH 224/227] ensure indices are ordered source, target, head everywhere --- specs/altair/beacon-chain.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 35909f23f..282b0f529 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -77,17 +77,17 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | -| `TIMELY_HEAD_FLAG_INDEX` | `0` | -| `TIMELY_SOURCE_FLAG_INDEX` | `1` | -| `TIMELY_TARGET_FLAG_INDEX` | `2` | +| `TIMELY_SOURCE_FLAG_INDEX` | `0` | +| `TIMELY_TARGET_FLAG_INDEX` | `1` | +| `TIMELY_HEAD_FLAG_INDEX` | `2` | ### Incentivization weights | Name | Value | | - | - | -| `TIMELY_HEAD_WEIGHT` | `uint64(12)` | | `TIMELY_SOURCE_WEIGHT` | `uint64(12)` | | `TIMELY_TARGET_WEIGHT` | `uint64(24)` | +| `TIMELY_HEAD_WEIGHT` | `uint64(12)` | | `SYNC_REWARD_WEIGHT` | `uint64(8)` | | `PROPOSER_WEIGHT` | `uint64(8)` | | `WEIGHT_DENOMINATOR` | `uint64(64)` | @@ -99,7 +99,7 @@ Altair is the first beacon chain hard fork. Its main features are: | Name | Value | | - | - | | `G2_POINT_AT_INFINITY` | `BLSSignature(b'\xc0' + b'\x00' * 95)` | -| `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_HEAD_WEIGHT, TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT]` | +| `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT, TIMELY_HEAD_FLAG_INDEX]` | ## Configuration From ccacc936de862004f25907b5d927a0758b112e27 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 14 May 2021 06:44:23 -0600 Subject: [PATCH 225/227] bmp VERSION.txt to 1.1.0-alpha.4 --- tests/core/pyspec/eth2spec/VERSION.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/VERSION.txt b/tests/core/pyspec/eth2spec/VERSION.txt index 57b7d37da..4b68ba4fa 100644 --- a/tests/core/pyspec/eth2spec/VERSION.txt +++ b/tests/core/pyspec/eth2spec/VERSION.txt @@ -1 +1 @@ -1.1.0-alpha.3 \ No newline at end of file +1.1.0-alpha.4 \ No newline at end of file From 110e6b38eee2b8832922d611f4f19f3dd43fbbbc Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 14 May 2021 08:07:08 -0600 Subject: [PATCH 226/227] remove resourceunavailable for more discussion --- specs/phase0/p2p-interface.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 297d93948..69ff7e120 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -567,8 +567,6 @@ The response code can have one of the following values, encoded as a single unsi The response payload adheres to the `ErrorMessage` schema (described below). - 2: **ServerError** -- the responder encountered an error while processing the request. The response payload adheres to the `ErrorMessage` schema (described below). -- 3: **ResourceUnavailable** -- the responder does not have requested resource. - The response payload adheres to the `ErrorMessage` schema (described below). Clients MAY use response codes above `128` to indicate alternative, erroneous request-specific responses. From c87333face105412e4ff059776dd770f2b216b1c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 14 May 2021 10:05:54 -0600 Subject: [PATCH 227/227] minor review from dankrad on release pr --- specs/altair/beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 282b0f529..bb2d9554e 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -327,7 +327,7 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: """ Return the base reward for the validator defined by ``index`` with respect to the current ``state``. - Note: A validator can optimally earn one base reward per epoch over a long time horizon. + Note: An optimally performing validator can earn one base reward per epoch over a long time horizon. This takes into account both per-epoch (e.g. attestation) and intermittent duties (e.g. block proposal and sync committees). """ @@ -388,7 +388,7 @@ def get_attestation_participation_flag_indices(state: BeaconState, ```python def get_flag_index_deltas(state: BeaconState, flag_index: int) -> Tuple[Sequence[Gwei], Sequence[Gwei]]: """ - Return the deltas for a given ``flag_index`` scaled by ``weight`` by scanning through the participation flags. + Return the deltas for a given ``flag_index`` by scanning through the participation flags. """ rewards = [Gwei(0)] * len(state.validators) penalties = [Gwei(0)] * len(state.validators)