From 6a92407702b8f3b21ec30fcfaa81794ae70f5f9c Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 25 Feb 2019 04:16:18 -0600 Subject: [PATCH 001/161] Developer notice --- specs/core/1_shard-data-chains.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 48eea7d2b..b68545c39 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -2,6 +2,8 @@ **NOTICE**: This document is a work-in-progress for researchers and implementers. It reflects recent spec changes and takes precedence over the [Python proof-of-concept implementation](https://github.com/ethereum/beacon_chain). +At the current stage, development teams are encouraged to treat the "Shard chains and crosslink data" section as reasonably close to the expected final form and worth starting development if they have spare resources, but the "Updates to the beacon chain" section, while fundamentally feature-complete, is still expected to undergo very considerable revision and review. + ## Table of contents @@ -15,26 +17,27 @@ - [Time parameters](#time-parameters) - [Max operations per block](#max-operations-per-block) - [Signature domains](#signature-domains) - - [Helper functions](#helper-functions) + - [Shard chains and crosslink data](#shard-chains-and-crosslink-data) + - [Helper functions](#helper-functions) - [`get_split_offset`](#get_split_offset) - [`get_shuffled_committee`](#get_shuffled_committee) - [`get_persistent_committee`](#get_persistent_committee) - [`get_shard_proposer_index`](#get_shard_proposer_index) - - [Data Structures](#data-structures) + - [Data Structures](#data-structures) - [Shard chain blocks](#shard-chain-blocks) - - [Shard block processing](#shard-block-processing) + - [Shard block processing](#shard-block-processing) - [Verifying shard block data](#verifying-shard-block-data) - [Verifying a crosslink](#verifying-a-crosslink) - [Shard block fork choice rule](#shard-block-fork-choice-rule) -- [Updates to the beacon chain](#updates-to-the-beacon-chain) - - [Data structures](#data-structures) + - [Updates to the beacon chain](#updates-to-the-beacon-chain) + - [Data structures](#data-structures) - [`Validator`](#validator) - [`BeaconBlockBody`](#beaconblockbody) - [`BranchChallenge`](#branchchallenge) - [`BranchResponse`](#branchresponse) - [`BranchChallengeRecord`](#branchchallengerecord) - [`SubkeyReveal`](#subkeyreveal) - - [Helpers](#helpers) + - [Helpers](#helpers) - [`get_attestation_merkle_depth`](#get_attestation_merkle_depth) - [`epoch_to_custody_period`](#epoch_to_custody_period) - [`slot_to_custody_period`](#slot_to_custody_period) @@ -42,13 +45,13 @@ - [`verify_custody_subkey_reveal`](#verify_custody_subkey_reveal) - [`prepare_validator_for_withdrawal`](#prepare_validator_for_withdrawal) - [`penalize_validator`](#penalize_validator) - - [Per-slot processing](#per-slot-processing) + - [Per-slot processing](#per-slot-processing) - [Operations](#operations) - [Branch challenges](#branch-challenges) - [Branch responses](#branch-responses) - [Subkey reveals](#subkey-reveals) - - [Per-epoch processing](#per-epoch-processing) - - [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition) + - [Per-epoch processing](#per-epoch-processing) + - [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition) @@ -98,6 +101,8 @@ Phase 1 depends upon all of the constants defined in [Phase 0](0_beacon-chain.md | `DOMAIN_SHARD_ATTESTER`| 130 | | `DOMAIN_CUSTODY_SUBKEY`| 131 | +# Shard chains and crosslink data + ## Helper functions #### `get_split_offset` From c9f1dfcbb3161a95a2f3d635039ec53a80e59341 Mon Sep 17 00:00:00 2001 From: Dean Eigenmann Date: Mon, 25 Feb 2019 21:39:23 +0100 Subject: [PATCH 002/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6bff0f705..4de77f3da 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1633,7 +1633,7 @@ For each `proposer_slashing` in `block.body.proposer_slashings`: * Verify that `proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot`. * Verify that `proposer_slashing.proposal_1.shard == proposer_slashing.proposal_2.shard`. * Verify that `proposer_slashing.proposal_1.block_root != proposer_slashing.proposal_2.block_root`. -* Verify that `proposer.slashed == False`. +* Verify that `not proposer.slashed`. * Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposer_slashing.proposal_1, "signature"), signature=proposer_slashing.proposal_1.signature, domain=get_domain(state.fork, slot_to_epoch(proposer_slashing.proposal_1.slot), DOMAIN_PROPOSAL))`. * Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposer_slashing.proposal_2, "signature"), signature=proposer_slashing.proposal_2.signature, domain=get_domain(state.fork, slot_to_epoch(proposer_slashing.proposal_2.slot), DOMAIN_PROPOSAL))`. * Run `slash_validator(state, proposer_slashing.proposer_index)`. @@ -1650,7 +1650,7 @@ For each `attester_slashing` in `block.body.attester_slashings`: * Verify that `is_double_vote(slashable_attestation_1.data, slashable_attestation_2.data)` or `is_surround_vote(slashable_attestation_1.data, slashable_attestation_2.data)`. * Verify that `verify_slashable_attestation(state, slashable_attestation_1)`. * Verify that `verify_slashable_attestation(state, slashable_attestation_2)`. -* Let `slashable_indices = [index for index in slashable_attestation_1.validator_indices if index in slashable_attestation_2.validator_indices and state.validator_registry[index].slashed == False]`. +* Let `slashable_indices = [index for index in slashable_attestation_1.validator_indices if index in slashable_attestation_2.validator_indices and not state.validator_registry[index].slashed]`. * Verify that `len(slashable_indices) >= 1`. * Run `slash_validator(state, index)` for each `index` in `slashable_indices`. @@ -1885,7 +1885,7 @@ Case 2: `epochs_since_finality > 4`: * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_boundary_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_head_attester_indices`, loses `base_reward(state, index)`. -* Any [active validator](#dfn-active-validator) `index` with `validator.slashed == True`, loses `2 * inactivity_penalty(state, index, epochs_since_finality) + base_reward(state, index)`. +* Any [active validator](#dfn-active-validator) `index` with `validator.slashed`, loses `2 * inactivity_penalty(state, index, epochs_since_finality) + base_reward(state, index)`. * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` loses `base_reward(state, index) - base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index)` ##### Attestation inclusion From 05f6f68028d760ced717832dd2863a6748086863 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 25 Feb 2019 16:45:57 -0600 Subject: [PATCH 003/161] Made developer notice less committal --- specs/core/1_shard-data-chains.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index b68545c39..6d292b421 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -2,7 +2,7 @@ **NOTICE**: This document is a work-in-progress for researchers and implementers. It reflects recent spec changes and takes precedence over the [Python proof-of-concept implementation](https://github.com/ethereum/beacon_chain). -At the current stage, development teams are encouraged to treat the "Shard chains and crosslink data" section as reasonably close to the expected final form and worth starting development if they have spare resources, but the "Updates to the beacon chain" section, while fundamentally feature-complete, is still expected to undergo very considerable revision and review. +At the current stage, Phase 1 is, while fundamentally feature-complete, is still subject to change. Development teams with spare resources may consider starting on the "Shard chains and crosslink data" section; at least basic properties, such as the fact that a shard block can get created every slot and is dependent on both a parent block in the same shard and a beacon chain block at or before that same slot, are unlikely to change, though details are likely to undergo similar kinds of changes to what Phase 0 has undergone since the start of the year. ## Table of contents From daa827531881ae9c89124ae1b651fa1983f5bdca Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 25 Feb 2019 21:19:28 -0700 Subject: [PATCH 004/161] minor bug in validator doc `get_committee_assignment` (#689) --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 1fd1c7ac3..1523fa507 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -353,7 +353,7 @@ def get_committee_assignment( a beacon block at the assigned slot. """ previous_epoch = get_previous_epoch(state) - next_epoch = get_current_epoch(state) + next_epoch = get_current_epoch(state) + 1 assert previous_epoch <= epoch <= next_epoch epoch_start_slot = get_epoch_start_slot(epoch) From 8b14a2bef5d872a719df9f7c58dd31efdd743d66 Mon Sep 17 00:00:00 2001 From: Dean Eigenmann Date: Tue, 26 Feb 2019 17:15:26 +0100 Subject: [PATCH 005/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4de77f3da..1bc07fb31 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1633,7 +1633,7 @@ For each `proposer_slashing` in `block.body.proposer_slashings`: * Verify that `proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot`. * Verify that `proposer_slashing.proposal_1.shard == proposer_slashing.proposal_2.shard`. * Verify that `proposer_slashing.proposal_1.block_root != proposer_slashing.proposal_2.block_root`. -* Verify that `not proposer.slashed`. +* Verify that `proposer.slashed is False`. * Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposer_slashing.proposal_1, "signature"), signature=proposer_slashing.proposal_1.signature, domain=get_domain(state.fork, slot_to_epoch(proposer_slashing.proposal_1.slot), DOMAIN_PROPOSAL))`. * Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposer_slashing.proposal_2, "signature"), signature=proposer_slashing.proposal_2.signature, domain=get_domain(state.fork, slot_to_epoch(proposer_slashing.proposal_2.slot), DOMAIN_PROPOSAL))`. * Run `slash_validator(state, proposer_slashing.proposer_index)`. From e7d908f1f1814791870158fc1fb7f4b82cfb7f4f Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 26 Feb 2019 14:45:25 -0800 Subject: [PATCH 006/161] formatting nitpick drop some parens in the spirit of consistency and parsimony --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6bff0f705..e8347704e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2044,7 +2044,7 @@ def process_exit_queue(state: BeaconState) -> None: #### Final updates * Set `state.latest_active_index_roots[(next_epoch + ACTIVATION_EXIT_DELAY) % LATEST_ACTIVE_INDEX_ROOTS_LENGTH] = hash_tree_root(get_active_validator_indices(state.validator_registry, next_epoch + ACTIVATION_EXIT_DELAY))`. -* Set `state.latest_slashed_balances[(next_epoch) % LATEST_SLASHED_EXIT_LENGTH] = state.latest_slashed_balances[current_epoch % LATEST_SLASHED_EXIT_LENGTH]`. +* Set `state.latest_slashed_balances[next_epoch % LATEST_SLASHED_EXIT_LENGTH] = state.latest_slashed_balances[current_epoch % LATEST_SLASHED_EXIT_LENGTH]`. * Set `state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch)`. * Remove any `attestation` in `state.latest_attestations` such that `slot_to_epoch(attestation.data.slot) < current_epoch`. From b2c53045fcb4879d6f7b01ab2cf72946bae0ee15 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 26 Feb 2019 15:55:27 -0700 Subject: [PATCH 007/161] make get_beacon_proposer_index safe for next epoch --- specs/core/0_beacon-chain.md | 12 ++++++++++-- specs/validator/0_beacon-chain-validator.md | 3 +-- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6bff0f705..0592919e4 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -967,11 +967,19 @@ def generate_seed(state: BeaconState, ```python def get_beacon_proposer_index(state: BeaconState, - slot: Slot) -> ValidatorIndex: + slot: Slot, + registry_change: bool=False)) -> ValidatorIndex: """ Return the beacon proposer index for the ``slot``. """ - first_committee, _ = get_crosslink_committees_at_slot(state, slot)[0] + epoch = slot_to_epoch(slot) + current_epoch = get_current_epoch(state) + previous_epoch = get_previous_epoch(state) + next_epoch = current_epoch + 1 + + assert previous_epoch <= epoch <= next_epoch + + first_committee, _ = get_crosslink_committees_at_slot(state, slot, registry_change)[0] return first_committee[slot % len(first_committee)] ``` diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 1523fa507..949a5e429 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -371,8 +371,7 @@ def get_committee_assignment( if len(selected_committees) > 0: validators = selected_committees[0][0] shard = selected_committees[0][1] - first_committee_at_slot = crosslink_committees[0][0] # List[ValidatorIndex] - is_proposer = first_committee_at_slot[slot % len(first_committee_at_slot)] == validator_index + is_proposer = validator_index == get_beacon_proposer_index(state, slot, registry_change) assignment = (validators, shard, slot, is_proposer) return assignment From e9c3af54288b3ce0b4ad3975c66ed43d5de08cb5 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 26 Feb 2019 18:00:20 -0600 Subject: [PATCH 008/161] Update 1_shard-data-chains.md --- specs/core/1_shard-data-chains.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 6d292b421..1cd5d89de 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -2,7 +2,7 @@ **NOTICE**: This document is a work-in-progress for researchers and implementers. It reflects recent spec changes and takes precedence over the [Python proof-of-concept implementation](https://github.com/ethereum/beacon_chain). -At the current stage, Phase 1 is, while fundamentally feature-complete, is still subject to change. Development teams with spare resources may consider starting on the "Shard chains and crosslink data" section; at least basic properties, such as the fact that a shard block can get created every slot and is dependent on both a parent block in the same shard and a beacon chain block at or before that same slot, are unlikely to change, though details are likely to undergo similar kinds of changes to what Phase 0 has undergone since the start of the year. +At the current stage, Phase 1, while fundamentally feature-complete, is still subject to change. Development teams with spare resources may consider starting on the "Shard chains and crosslink data" section; at least basic properties, such as the fact that a shard block can get created every slot and is dependent on both a parent block in the same shard and a beacon chain block at or before that same slot, are unlikely to change, though details are likely to undergo similar kinds of changes to what Phase 0 has undergone since the start of the year. ## Table of contents From 5c5f15c82c2e47af6b340da809c5f908ffe7ef2e Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 11:54:23 +0100 Subject: [PATCH 009/161] Rewrite SSZ spec * Implement tuples and a chunk-size reduction to 32 bytes (see #665 and #679) * Simplify presentation where appropriate. For example, deserialisation is implicit from serialisation (similar to `bls_sign` being implicit from `bls_verify`) and is left as an implementation exercise. * Dramatically reduce spec size and hopefully improve readability. --- specs/simple-serialize.md | 474 ++++++++------------------------------ 1 file changed, 95 insertions(+), 379 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 109ee289e..9cee8a507 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,424 +1,140 @@ -# [WIP] SimpleSerialize (SSZ) Spec +# [WIP] SimpleSerialiZe (SSZ) -This is the **work in progress** document to describe `SimpleSerialize`, the -current selected serialization method for Ethereum 2.0 using the Beacon Chain. +This is a **work in progress** describing typing, serialisation and Merkleisation of Ethereum 2.0 objects. -This document specifies the general information for serializing and -deserializing objects and data types. +## Table of contents -## ToC - -* [About](#about) -* [Variables and Functions](#variables-and-functions) -* [Constants](#constants) -* [Overview](#overview) - + [Serialize/Encode](#serializeencode) - - [uintN](#uintn) - - [bool](#bool) - - [bytesN](#bytesn) - - [List/Vectors](#listvectors) - - [Container](#container) - + [Deserialize/Decode](#deserializedecode) - - [uintN](#uintn-1) - - [bool](#bool-1) - - [bytesN](#bytesn-1) - - [List/Vectors](#listvectors-1) - - [Container](#container-1) - + [Tree Hash](#tree-hash) - - [`uint8`..`uint256`, `bool`, `bytes1`..`bytes32`](#uint8uint256-bool-bytes1bytes32) - - [`uint264`..`uintN`, `bytes33`..`bytesN`](#uint264uintn-bytes33bytesn) - - [List/Vectors](#listvectors-2) - - [Container](#container-2) - + [Signed Roots](#signed-roots) -* [Implementations](#implementations) - -## About - -`SimpleSerialize` was first proposed by Vitalik Buterin as the serialization -protocol for use in the Ethereum 2.0 Beacon Chain. - -The core feature of `ssz` is the simplicity of the serialization with low -overhead. - -## Variables and Functions - -| Term | Definition | -|:-------------|:-----------------------------------------------------------------------------------------------| -| `little` | Little endian. | -| `byteorder` | Specifies [endianness](https://en.wikipedia.org/wiki/Endianness): big endian or little endian. | -| `len` | Length/number of bytes. | -| `to_bytes` | Convert to bytes. Should take parameters ``size`` and ``byteorder``. | -| `from_bytes` | Convert from bytes to object. Should take ``bytes`` and ``byteorder``. | -| `value` | The value to serialize. | -| `rawbytes` | Raw serialized bytes. | -| `deserialized_object` | The deserialized data in the data structure of your programming language. | -| `new_index` | An index to keep track the latest position where the `rawbytes` have been deserialized. | +- [Constants](#constants) +- [Types](#types) + - [Primitive types](#primitive-types) + - [Composite types](#composite-types) + - [Notation](#notation) + - [Aliases](#aliases) +- [Serialization](#serialization) + - [`uintN`](#uintn) + - [`bool`](#bool) + - [Containers](#containers) + - [Tuples](#tuples) + - [Lists](#lists) +- [Deserialization](#deserialization) +- [Merkleization](#merkleization) +- [Signed containers](#signed-containers) +- [Implementations](#implementations) ## Constants -| Constant | Value | Definition | -|:------------------|:-----:|:--------------------------------------------------------------------------------------| -| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. | -| `SSZ_CHUNK_SIZE` | 128 | Number of bytes for the chunk size of the Merkle tree leaf. | +| Name | Value | Definition | +|-|:-:|-| +| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. | -## Overview +## Types -### Serialize/Encode +### Primitive types -#### uintN +* `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) +* `bool`: 1-bit unsigned integer -| uint Type | Usage | -|:---------:|:-----------------------------------------------------------| -| `uintN` | Type of `N` bits unsigned integer, where ``N % 8 == 0``. | +### Composite types -Convert directly to bytes the size of the int. (e.g. ``uint16 = 2 bytes``) +* **Containes**: ordered heterogenous collection of values +* **Tuple**: ordered fixed-size homogeneous collection of values +* **List**: ordered variable-size homogenous collection of values -All integers are serialized as **little endian**. +### Notation -| Check to perform | Code | -|:-----------------------|:----------------------| -| Size is a byte integer | ``int_size % 8 == 0`` | +* **Containes**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` +* **Tuple**: angle-braket notation `[]`, e.g. `uint64[]` +* **List**: angle-braket notation `[N]`, e.g. `uint64[N]` + +### Aliases + +For convenience we alias: + +* `byte` to `uint8` +* `bytes` to `byte[]` +* `bytesN` to `byte[N]` +* `bit` to `bool` + +## Serialization + +We reccursively define a `serialize` function. In the code below `value` refers to a value of the specified type. + +### `uintN` ```python -assert(int_size % 8 == 0) -buffer_size = int_size / 8 -return value.to_bytes(buffer_size, 'little') +assert N in [8, 16, 32, 64, 128, 256] +return value.to_bytes(N / 8, 'little') ``` -#### bool - -Convert directly to a single 0x00 or 0x01 byte. - -| Check to perform | Code | -|:------------------|:---------------------------| -| Value is boolean | ``value in (True, False)`` | +### `bool` ```python -assert(value in (True, False)) +assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -#### bytesN - -A fixed-size byte array. - -| Checks to perform | Code | -|:---------------------------------------|:---------------------| -| Length in bytes is correct for `bytesN` | ``len(value) == N`` | +### Containers ```python -assert(len(value) == N) - -return value +serialized_elements = [serialize(element) for element in value] +serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +assert len(serialized_bytes) < 2**32 +serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') +return serialized_length + serialized_bytes ``` -#### List/Vectors - -Lists are a collection of elements of the same homogeneous type. - -| Check to perform | Code | -|:--------------------------------------------|:----------------------------| -| Length of serialized list fits into 4 bytes | ``len(serialized) < 2**32`` | - -1. Serialize all list elements individually and concatenate them. -2. Prefix the concatenation with its length encoded as a `4-byte` **little-endian** unsigned integer. - -We define `bytes` to be a synonym of `List[bytes1]`. - -**Example in Python** +### Tuples ```python - -serialized_list_string = b'' - -for item in value: - serialized_list_string += serialize(item) - -assert(len(serialized_list_string) < 2**32) - -serialized_len = (len(serialized_list_string).to_bytes(LENGTH_BYTES, 'little')) - -return serialized_len + serialized_list_string +serialized_elements = [serialize(element) for element in value] +serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +return serialized_bytes ``` -#### Container - -A container represents a heterogenous, associative collection of key-value pairs. Each pair is referred to as a `field`. To get the value for a given field, you supply the key which is a symbol unique to the container referred to as the field's `name`. The container data type is analogous to the `struct` type found in many languages like C or Go. - -To serialize a container, obtain the list of its field's names in the specified order. For each field name in this list, obtain the corresponding value and serialize it. Tightly pack the complete set of serialized values in the same order as the field names into a buffer. Calculate the size of this buffer of serialized bytes and encode as a `4-byte` **little endian** `uint32`. Prepend the encoded length to the buffer. The result of this concatenation is the final serialized value of the container. - -| Check to perform | Code | -|:----------------------------------------------|:----------------------------| -| Length of serialized fields fits into 4 bytes | ``len(serialized) < 2**32`` | - -To serialize: - -1. Get the list of the container's fields. - -2. For each name in the list, obtain the corresponding value from the container and serialize it. Place this serialized value into a buffer. The serialized values should be tightly packed. - -3. Get the number of raw bytes in the serialized buffer. Encode that number as a `4-byte` **little endian** `uint32`. - -4. Prepend the length to the serialized buffer. - -**Example in Python** +### Lists ```python -def get_field_names(typ): - return typ.fields.keys() - -def get_value_for_field_name(value, field_name): - return getattr(value, field_name) - -def get_type_for_field_name(typ, field_name): - return typ.fields[field_name] - -serialized_buffer = b'' - -typ = type(value) -for field_name in get_field_names(typ): - field_value = get_value_for_field_name(value, field_name) - field_type = get_type_for_field_name(typ, field_name) - serialized_buffer += serialize(field_value, field_type) - -assert(len(serialized_buffer) < 2**32) - -serialized_len = (len(serialized_buffer).to_bytes(LENGTH_BYTES, 'little')) - -return serialized_len + serialized_buffer +serialized_elements = [serialize(element) for element in value] +serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +assert len(serialized_elements) < 2**32 +serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') +return serialized_length + serialized_bytes ``` -### Deserialize/Decode +## Deserialization -The decoding requires knowledge of the type of the item to be decoded. When -performing decoding on an entire serialized string, it also requires knowledge -of the order in which the objects have been serialized. +Given a type, serialisation is an injective function from objects of that type to byte strings. That is, deserialisation—the inverse function—is well-defined. -Note: Each return will provide: -- `deserialized_object` -- `new_index` +## Merkleization -At each step, the following checks should be made: +We first define helper functions: -| Check to perform | Check | -|:-------------------------|:-----------------------------------------------------------| -| Ensure sufficient length | ``len(rawbytes) >= current_index + deserialize_length`` | +* `pack`: Given ordered objects of the same basic type, serialise them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleise`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. +* `mix_in_length`: Given a Merkle root `r` and a length `l` (32-byte little-endian serialisation) return `hash(r + l)`. -At the final step, the following checks should be made: +Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` recursively: -| Check to perform | Check | -|:-------------------------|:-------------------------------------| -| Ensure no extra length | `new_index == len(rawbytes)` | +* `merkleize(pack(o))` if `o` is a basic object or a tuple of basic objects +* `mix_in_length(merkleize(pack(o)), len(o))` if `o` is a list of basic objects +* `merkleize([hash_tree_root(element) for element in o])` if `o` is a tuple of composite objects or a container +* `mix_in_length(merkleize([hash_tree_root(element) for element in o]), len(o))` if `o` is a list of composite objects -#### uintN +## Signed containers -Convert directly from bytes into integer utilising the number of bytes the same -size as the integer length. (e.g. ``uint16 == 2 bytes``) - -All integers are interpreted as **little endian**. - -```python -byte_length = int_size / 8 -new_index = current_index + byte_length -assert(len(rawbytes) >= new_index) -return int.from_bytes(rawbytes[current_index:current_index+byte_length], 'little'), new_index -``` - -#### bool - -Return True if 0x01, False if 0x00. - -```python -assert rawbytes in (b'\x00', b'\x01') -return True if rawbytes == b'\x01' else False -``` - -#### bytesN - -Return the `N` bytes. - -```python -assert(len(rawbytes) >= current_index + N) -new_index = current_index + N -return rawbytes[current_index:current_index+N], new_index -``` - -#### List/Vectors - -Deserialize each element in the list. -1. Get the length of the serialized list. -2. Loop through deserializing each item in the list until you reach the -entire length of the list. - -| Check to perform | code | -|:------------------------------------------|:----------------------------------------------------------------| -| ``rawbytes`` has enough left for length | ``len(rawbytes) > current_index + LENGTH_BYTES`` | -| list is not greater than serialized bytes | ``len(rawbytes) > current_index + LENGTH_BYTES + total_length`` | - -```python -assert(len(rawbytes) > current_index + LENGTH_BYTES) -total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'little') -new_index = current_index + LENGTH_BYTES + total_length -assert(len(rawbytes) >= new_index) -item_index = current_index + LENGTH_BYTES -deserialized_list = [] - -while item_index < new_index: - object, item_index = deserialize(rawbytes, item_index, item_type) - deserialized_list.append(object) - -return deserialized_list, new_index -``` - -#### Container - -Refer to the section on container encoding for some definitions. - -To deserialize a container, loop over each field in the container and use the type of that field to know what kind of deserialization to perform. Consume successive elements of the data stream for each successful deserialization. - -Instantiate a container with the full set of deserialized data, matching each member with the corresponding field. - -| Check to perform | code | -|:------------------------------------------|:----------------------------------------------------------------| -| ``rawbytes`` has enough left for length | ``len(rawbytes) > current_index + LENGTH_BYTES`` | -| list is not greater than serialized bytes | ``len(rawbytes) > current_index + LENGTH_BYTES + total_length`` | - -To deserialize: - -1. Get the list of the container's fields. -2. For each name in the list, attempt to deserialize a value for that type. Collect these values as they will be used to construct an instance of the container. -3. Construct a container instance after successfully consuming the entire subset of the stream for the serialized container. - -**Example in Python** - -```python -def get_field_names(typ): - return typ.fields.keys() - -def get_value_for_field_name(value, field_name): - return getattr(value, field_name) - -def get_type_for_field_name(typ, field_name): - return typ.fields[field_name] - -class Container: - # this is the container; here we will define an empty class for demonstration - pass - -# get a reference to the type in some way... -container = Container() -typ = type(container) - -assert(len(rawbytes) > current_index + LENGTH_BYTES) -total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'little') -new_index = current_index + LENGTH_BYTES + total_length -assert(len(rawbytes) >= new_index) -item_index = current_index + LENGTH_BYTES - -values = {} -for field_name in get_field_names(typ): - field_name_type = get_type_for_field_name(typ, field_name) - values[field_name], item_index = deserialize(data, item_index, field_name_type) -assert item_index == new_index -return typ(**values), item_index -``` - -### Tree Hash - -The below `hash_tree_root_internal` algorithm is defined recursively in the case of lists and containers, and it outputs a value equal to or less than 32 bytes in size. For use as a "final output" (eg. for signing), use `hash_tree_root(x) = zpad(hash_tree_root_internal(x), 32)`, where `zpad` is a helper that extends the given `bytes` value to the desired `length` by adding zero bytes on the right: - -```python -def zpad(input: bytes, length: int) -> bytes: - return input + b'\x00' * (length - len(input)) -``` - -Refer to [the helper function `hash`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#hash) of Phase 0 of the [Eth2.0 specs](https://github.com/ethereum/eth2.0-specs) for a definition of the hash function used below, `hash(x)`. - -#### `uint8`..`uint256`, `bool`, `bytes1`..`bytes32` - -Return the serialization of the value. - -#### `uint264`..`uintN`, `bytes33`..`bytesN` - -Return the hash of the serialization of the value. - -#### List/Vectors - -First, we define the Merkle tree function. - -```python -# Merkle tree hash of a list of homogenous, non-empty items -def merkle_hash(lst): - # Store length of list (to compensate for non-bijectiveness of padding) - datalen = len(lst).to_bytes(32, 'little') - - if len(lst) == 0: - # Handle empty list case - chunkz = [b'\x00' * SSZ_CHUNK_SIZE] - elif len(lst[0]) < SSZ_CHUNK_SIZE: - # See how many items fit in a chunk - items_per_chunk = SSZ_CHUNK_SIZE // len(lst[0]) - - # Build a list of chunks based on the number of items in the chunk - chunkz = [ - zpad(b''.join(lst[i:i + items_per_chunk]), SSZ_CHUNK_SIZE) - for i in range(0, len(lst), items_per_chunk) - ] - else: - # Leave large items alone - chunkz = lst - - # Merkleise - def next_power_of_2(x): - return 1 if x == 0 else 2**(x - 1).bit_length() - - for i in range(len(chunkz), next_power_of_2(len(chunkz))): - chunkz.append(b'\x00' * SSZ_CHUNK_SIZE) - while len(chunkz) > 1: - chunkz = [hash(chunkz[i] + chunkz[i+1]) for i in range(0, len(chunkz), 2)] - - # Return hash of root and data length - return hash(chunkz[0] + datalen) -``` - -To `hash_tree_root_internal` a list, we simply do: - -```python -return merkle_hash([hash_tree_root_internal(item) for item in value]) -``` - -Where the inner `hash_tree_root_internal` is a recursive application of the tree-hashing function (returning less than 32 bytes for short single values). - -#### Container - -Recursively tree hash the values in the container in the same order as the fields, and Merkle hash the results. - -```python -return merkle_hash([hash_tree_root_internal(getattr(x, field)) for field in value.fields]) -``` - -### Signed roots - -Let `field_name` be a field name in an SSZ container `container`. We define `truncate(container, field_name)` to be the `container` with the fields from `field_name` onwards truncated away. That is, `truncate(container, field_name) = [getattr(container, field)) for field in value.fields[:i]]` where `i = value.fields.index(field_name)`. - -When `field_name` maps to a signature (e.g. a BLS12-381 signature of type `Bytes96`) the convention is that the corresponding signed message be `signed_root(container, field_name) = hash_tree_root(truncate(container, field_name))`. For example if `container = {"foo": sub_object_1, "bar": sub_object_2, "signature": bytes96, "baz": sub_object_3}` then `signed_root(container, "signature") = merkle_hash([hash_tree_root(sub_object_1), hash_tree_root(sub_object_2)])`. - -Note that this convention means that fields after the signature are _not_ signed over. If there are multiple signatures in `container` then those are expected to be signing over the fields in the order specified. If multiple signatures of the same value are expected the convention is that the signature field be an array of signatures. +Let `container` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `container`. Further, the signed message for `container` is `signed_root(container) = hash_tree_root(truncate_last(container))` where `truncate_last` truncates the last element of `container`. ## Implementations -| Language | Implementation | Description | -|:--------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------| -| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ | -| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ. | -| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ. | -| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech. | -| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ | -| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite | -| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs | -| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ | -| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ | -| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ | - -## Copyright -Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). +| Language | Implementation | Description | +|:-:|-|-| +| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ | +| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ | +| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ | +| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech | +| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ | +| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite | +| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs | +| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ | +| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ | +| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ | From df50ac1adc76b0fe4cfecbd296c612db5f62df65 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:03:27 +0100 Subject: [PATCH 010/161] Update simple-serialize.md --- specs/simple-serialize.md | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 9cee8a507..4f83af4a4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -25,7 +25,8 @@ This is a **work in progress** describing typing, serialisation and Merkleisatio | Name | Value | Definition | |-|:-:|-| -| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. | +| `LENGTH_BYTES` | 4 | Number of bytes for the length of variable-length serialized objects. | +| `MAX_LENGTH` | 2**(8 * LENGTH_BYTES) | Maximum serialization length. | ## Types @@ -36,15 +37,15 @@ This is a **work in progress** describing typing, serialisation and Merkleisatio ### Composite types -* **Containes**: ordered heterogenous collection of values -* **Tuple**: ordered fixed-size homogeneous collection of values -* **List**: ordered variable-size homogenous collection of values +* **Container**: ordered heterogenous collection of values +* **Tuple**: ordered fixed-length homogeneous collection of values +* **List**: ordered variable-length homogenous collection of values ### Notation -* **Containes**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` -* **Tuple**: angle-braket notation `[]`, e.g. `uint64[]` -* **List**: angle-braket notation `[N]`, e.g. `uint64[N]` +* **Container**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` +* **Tuple**: angle-braket notation `[N]`, e.g. `uint64[N]` +* **List**: angle-braket notation `[]`, e.g. `uint64[]` ### Aliases @@ -57,28 +58,28 @@ For convenience we alias: ## Serialization -We reccursively define a `serialize` function. In the code below `value` refers to a value of the specified type. +We reccursively define the `serialize` function which consumes an object `o` (of the type specified) and returns a byte string `[]byte`. ### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] -return value.to_bytes(N / 8, 'little') +return o.to_bytes(N / 8, 'little') ``` ### `bool` ```python -assert value in (True, False) -return b'\x01' if value is True else b'\x00' +assert o in (True, False) +return b'\x01' if o is True else b'\x00' ``` ### Containers ```python -serialized_elements = [serialize(element) for element in value] +serialized_elements = [serialize(element) for element in o] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -assert len(serialized_bytes) < 2**32 +assert len(serialized_bytes) < MAX_LENGTH serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` @@ -86,7 +87,7 @@ return serialized_length + serialized_bytes ### Tuples ```python -serialized_elements = [serialize(element) for element in value] +serialized_elements = [serialize(element) for element in o] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) return serialized_bytes ``` @@ -94,9 +95,9 @@ return serialized_bytes ### Lists ```python -serialized_elements = [serialize(element) for element in value] +serialized_elements = [serialize(element) for element in o] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -assert len(serialized_elements) < 2**32 +assert len(serialized_elements) < MAX_LENGTH serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` From 313fe46a94f0790bfb1be024e4ebf8f921a0fbdc Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:06:06 +0100 Subject: [PATCH 011/161] Update simple-serialize.md --- specs/simple-serialize.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 4f83af4a4..7d3d69725 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,6 +1,6 @@ # [WIP] SimpleSerialiZe (SSZ) -This is a **work in progress** describing typing, serialisation and Merkleisation of Ethereum 2.0 objects. +This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects. ## Table of contents @@ -25,8 +25,8 @@ This is a **work in progress** describing typing, serialisation and Merkleisatio | Name | Value | Definition | |-|:-:|-| -| `LENGTH_BYTES` | 4 | Number of bytes for the length of variable-length serialized objects. | -| `MAX_LENGTH` | 2**(8 * LENGTH_BYTES) | Maximum serialization length. | +| `LENGTH_BYTES` | `4` | Number of bytes for the length of variable-length serialized objects. | +| `MAX_LENGTH` | `2**(8 * LENGTH_BYTES)` | Maximum serialization length. | ## Types @@ -104,15 +104,15 @@ return serialized_length + serialized_bytes ## Deserialization -Given a type, serialisation is an injective function from objects of that type to byte strings. That is, deserialisation—the inverse function—is well-defined. +Given a type, serialization is an injective function from objects of that type to byte strings. That is, deserialization—the inverse function—is well-defined. ## Merkleization We first define helper functions: -* `pack`: Given ordered objects of the same basic type, serialise them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleise`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. -* `mix_in_length`: Given a Merkle root `r` and a length `l` (32-byte little-endian serialisation) return `hash(r + l)`. +* `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. +* `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` recursively: From 5116645dfa802044d56c63a9f77220a7ec7b1e9e Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:40:08 +0100 Subject: [PATCH 012/161] Update simple-serialize.md --- specs/simple-serialize.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 7d3d69725..f972952d5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -18,7 +18,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Lists](#lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) -- [Signed containers](#signed-containers) +- [Self-signed containers](#self-signed-containers) - [Implementations](#implementations) ## Constants @@ -121,21 +121,21 @@ Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` rec * `merkleize([hash_tree_root(element) for element in o])` if `o` is a tuple of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in o]), len(o))` if `o` is a list of composite objects -## Signed containers +## Self-signed containers Let `container` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `container`. Further, the signed message for `container` is `signed_root(container) = hash_tree_root(truncate_last(container))` where `truncate_last` truncates the last element of `container`. ## Implementations -| Language | Implementation | Description | -|:-:|-|-| -| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ | -| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ | -| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ | -| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech | -| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ | -| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite | -| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs | -| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ | -| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ | -| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ | +| Language | Project | Maintainer | Implementation | +|-|-|-|-| +| Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) | +| Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | +| Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | +| Rust | Shasper | ParityTech | [https://github.com/paritytech/shasper/tree/master/util/ssz](https://github.com/paritytech/shasper/tree/master/util/ssz) | +| Javascript | Lodestart | Chain Safe Systems | [https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | +| Java | Cava | ConsenSys | [https://www.github.com/ConsenSys/cava/tree/master/ssz](https://www.github.com/ConsenSys/cava/tree/master/ssz) | +| Go | Prysm | Prysmatic Labs | [https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | +| Swift | Yeeth | Dean Eigenmann | [https://github.com/yeeth/SimpleSerialize.swift](https://github.com/yeeth/SimpleSerialize.swift) | +| C# | | Jordan Andrews | [https://github.com/codingupastorm/csharp-ssz](https://github.com/codingupastorm/csharp-ssz) | +| C++ | | | [https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | From 28cf5860ea139bb0953075083d3f5f638541946a Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:53:24 +0100 Subject: [PATCH 013/161] Update simple-serialize.md --- specs/simple-serialize.md | 42 +++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f972952d5..780f1cc90 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,11 +1,11 @@ -# [WIP] SimpleSerialiZe (SSZ) +# SimpleSerialiZe (SSZ) This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects. ## Table of contents - [Constants](#constants) -- [Types](#types) +- [Typing](#typing) - [Primitive types](#primitive-types) - [Composite types](#composite-types) - [Notation](#notation) @@ -23,12 +23,12 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Constants -| Name | Value | Definition | -|-|:-:|-| +| Name | Value | Description | +|-|-|-| | `LENGTH_BYTES` | `4` | Number of bytes for the length of variable-length serialized objects. | | `MAX_LENGTH` | `2**(8 * LENGTH_BYTES)` | Maximum serialization length. | -## Types +## Typing ### Primitive types @@ -43,9 +43,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Notation -* **Container**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` -* **Tuple**: angle-braket notation `[N]`, e.g. `uint64[N]` -* **List**: angle-braket notation `[]`, e.g. `uint64[]` +* **Container**: key-pair curly braken notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) +* **Tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) +* **List**: angle braket notation `[]` (e.g. `uint64[]`) ### Aliases @@ -58,26 +58,26 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `o` (of the type specified) and returns a byte string `[]byte`. +We reccursively define the `serialize` function which consumes an object `object` (of the type specified) and returns a byte string of type `bytes`. ### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] -return o.to_bytes(N / 8, 'little') +return object.to_bytes(N / 8, 'little') ``` ### `bool` ```python -assert o in (True, False) -return b'\x01' if o is True else b'\x00' +assert object in (True, False) +return b'\x01' if object is True else b'\x00' ``` ### Containers ```python -serialized_elements = [serialize(element) for element in o] +serialized_elements = [serialize(element) for element in object] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) assert len(serialized_bytes) < MAX_LENGTH serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') @@ -87,7 +87,7 @@ return serialized_length + serialized_bytes ### Tuples ```python -serialized_elements = [serialize(element) for element in o] +serialized_elements = [serialize(element) for element in object] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) return serialized_bytes ``` @@ -95,7 +95,7 @@ return serialized_bytes ### Lists ```python -serialized_elements = [serialize(element) for element in o] +serialized_elements = [serialize(element) for element in object] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) assert len(serialized_elements) < MAX_LENGTH serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') @@ -114,12 +114,12 @@ We first define helper functions: * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. -Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` recursively: +Let `object` be an object. We now define object Merkleization `hash_tree_root(object)` recursively: -* `merkleize(pack(o))` if `o` is a basic object or a tuple of basic objects -* `mix_in_length(merkleize(pack(o)), len(o))` if `o` is a list of basic objects -* `merkleize([hash_tree_root(element) for element in o])` if `o` is a tuple of composite objects or a container -* `mix_in_length(merkleize([hash_tree_root(element) for element in o]), len(o))` if `o` is a list of composite objects +* `merkleize(pack(object))` if `object` is a basic object or a tuple of basic objects +* `mix_in_length(merkleize(pack(object)), len(object))` if `object` is a list of basic objects +* `merkleize([hash_tree_root(element) for element in object])` if `object` is a tuple of composite objects or a container +* `mix_in_length(merkleize([hash_tree_root(element) for element in object]), len(object))` if `object` is a list of composite objects ## Self-signed containers @@ -128,7 +128,7 @@ Let `container` be a self-signed container object. The convention is that the si ## Implementations | Language | Project | Maintainer | Implementation | -|-|-|-|-| +|:-:|-|-| | Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) | | Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | | Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | From 07922a63fbb52cd3f42441cb78909e8a6bc4c0c9 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:54:56 +0100 Subject: [PATCH 014/161] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 780f1cc90..eb1d3cace 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -128,7 +128,7 @@ Let `container` be a self-signed container object. The convention is that the si ## Implementations | Language | Project | Maintainer | Implementation | -|:-:|-|-| +|-|-|-|-| | Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) | | Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | | Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | From e698d7e29bb85122d33290bb255457ca8bcc0de1 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:57:06 +0100 Subject: [PATCH 015/161] Update simple-serialize.md --- specs/simple-serialize.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index eb1d3cace..7da967a25 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,24 +30,24 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Typing -### Primitive types +#### Primitive types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `bool`: 1-bit unsigned integer -### Composite types +#### Composite types -* **Container**: ordered heterogenous collection of values -* **Tuple**: ordered fixed-length homogeneous collection of values -* **List**: ordered variable-length homogenous collection of values +* **container**: ordered heterogenous collection of values +* **tuple**: ordered fixed-length homogeneous collection of values +* **list**: ordered variable-length homogenous collection of values -### Notation +#### Notation -* **Container**: key-pair curly braken notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) -* **Tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) -* **List**: angle braket notation `[]` (e.g. `uint64[]`) +* **container**: key-pair curly braket notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) +* **tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) +* **list**: angle braket notation `[]` (e.g. `uint64[]`) -### Aliases +#### Aliases For convenience we alias: @@ -60,21 +60,21 @@ For convenience we alias: We reccursively define the `serialize` function which consumes an object `object` (of the type specified) and returns a byte string of type `bytes`. -### `uintN` +#### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] return object.to_bytes(N / 8, 'little') ``` -### `bool` +#### `bool` ```python assert object in (True, False) return b'\x01' if object is True else b'\x00' ``` -### Containers +#### Containers ```python serialized_elements = [serialize(element) for element in object] @@ -84,7 +84,7 @@ serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` -### Tuples +#### Tuples ```python serialized_elements = [serialize(element) for element in object] @@ -92,7 +92,7 @@ serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) return serialized_bytes ``` -### Lists +#### Lists ```python serialized_elements = [serialize(element) for element in object] From 0df8f8b3c68bc95647c004b89b7decf6d61a8863 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 13:04:36 +0100 Subject: [PATCH 016/161] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 7da967a25..41640aabe 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -6,7 +6,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Constants](#constants) - [Typing](#typing) - - [Primitive types](#primitive-types) + - [Basic types](#basic-types) - [Composite types](#composite-types) - [Notation](#notation) - [Aliases](#aliases) @@ -30,7 +30,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Typing -#### Primitive types +#### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `bool`: 1-bit unsigned integer From 828146cbf8b37fd4455df664c45c8a7cb5160889 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 27 Feb 2019 17:15:46 +0100 Subject: [PATCH 017/161] Apply suggestions from code review Co-Authored-By: JustinDrake --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 41640aabe..a87f477ae 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -64,7 +64,7 @@ We reccursively define the `serialize` function which consumes an object `object ```python assert N in [8, 16, 32, 64, 128, 256] -return object.to_bytes(N / 8, 'little') +return object.to_bytes(N // 8, 'little') ``` #### `bool` @@ -111,7 +111,7 @@ Given a type, serialization is an injective function from objects of that type t We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. +* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. Let `object` be an object. We now define object Merkleization `hash_tree_root(object)` recursively: From 95fa9d56b85d04e9153c7ec5774a8505ddb2e949 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:35:26 +0100 Subject: [PATCH 018/161] Update simple-serialize.md --- specs/simple-serialize.md | 45 +++++++++++---------------------------- 1 file changed, 12 insertions(+), 33 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index a87f477ae..3dc9b0b07 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -13,9 +13,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`uintN`](#uintn) - [`bool`](#bool) - - [Containers](#containers) - - [Tuples](#tuples) - - [Lists](#lists) + - [Containers, tuples, lists](#containers-tuples-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -58,50 +56,31 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `object` (of the type specified) and returns a byte string of type `bytes`. +We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. #### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] -return object.to_bytes(N // 8, 'little') +return value.to_bytes(N // 8, 'little') ``` #### `bool` ```python -assert object in (True, False) -return b'\x01' if object is True else b'\x00' +assert value in (True, False) +return b'\x01' if value is True else b'\x00' ``` -#### Containers +#### Containers, tuples, lists ```python -serialized_elements = [serialize(element) for element in object] -serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +serialized_bytes = ''.join([serialize(element) for element in value]) assert len(serialized_bytes) < MAX_LENGTH serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` -#### Tuples - -```python -serialized_elements = [serialize(element) for element in object] -serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -return serialized_bytes -``` - -#### Lists - -```python -serialized_elements = [serialize(element) for element in object] -serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -assert len(serialized_elements) < MAX_LENGTH -serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') -return serialized_length + serialized_bytes -``` - ## Deserialization Given a type, serialization is an injective function from objects of that type to byte strings. That is, deserialization—the inverse function—is well-defined. @@ -114,12 +93,12 @@ We first define helper functions: * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. -Let `object` be an object. We now define object Merkleization `hash_tree_root(object)` recursively: +Let `value` be an object. We now define object Merkleization `hash_tree_root(value)` recursively: -* `merkleize(pack(object))` if `object` is a basic object or a tuple of basic objects -* `mix_in_length(merkleize(pack(object)), len(object))` if `object` is a list of basic objects -* `merkleize([hash_tree_root(element) for element in object])` if `object` is a tuple of composite objects or a container -* `mix_in_length(merkleize([hash_tree_root(element) for element in object]), len(object))` if `object` is a list of composite objects +* `merkleize(pack(value))` if `value` is a basic object or a tuple of basic objects +* `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects +* `merkleize([hash_tree_root(element) for element in value])` if `value` is a tuple of composite objects or a container +* `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects ## Self-signed containers From aa676354a049b8a2812c237861b0d7e9c02a83c6 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:54:19 +0100 Subject: [PATCH 019/161] Update simple-serialize.md --- specs/simple-serialize.md | 29 +++++++---------------------- 1 file changed, 7 insertions(+), 22 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 3dc9b0b07..639231b30 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -4,11 +4,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Table of contents -- [Constants](#constants) - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) - - [Notation](#notation) - [Aliases](#aliases) - [Serialization](#serialization) - [`uintN`](#uintn) @@ -19,31 +17,18 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Self-signed containers](#self-signed-containers) - [Implementations](#implementations) -## Constants - -| Name | Value | Description | -|-|-|-| -| `LENGTH_BYTES` | `4` | Number of bytes for the length of variable-length serialized objects. | -| `MAX_LENGTH` | `2**(8 * LENGTH_BYTES)` | Maximum serialization length. | - ## Typing -#### Basic types +### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `bool`: 1-bit unsigned integer -#### Composite types +### Composite types -* **container**: ordered heterogenous collection of values -* **tuple**: ordered fixed-length homogeneous collection of values -* **list**: ordered variable-length homogenous collection of values - -#### Notation - -* **container**: key-pair curly braket notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) -* **tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) -* **list**: angle braket notation `[]` (e.g. `uint64[]`) +* **container**: ordered heterogenous collection of values (key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}`) +* **tuple**: ordered fixed-length homogeneous collection of values (angle braket notation `[N]`, e.g. `uint64[N]`) +* **list**: ordered variable-length homogenous collection of values (angle braket notation `[]`, e.g. `uint64[]`) #### Aliases @@ -52,7 +37,6 @@ For convenience we alias: * `byte` to `uint8` * `bytes` to `byte[]` * `bytesN` to `byte[N]` -* `bit` to `bool` ## Serialization @@ -75,8 +59,9 @@ return b'\x01' if value is True else b'\x00' #### Containers, tuples, lists ```python +LENGTH_BYTES = 4 serialized_bytes = ''.join([serialize(element) for element in value]) -assert len(serialized_bytes) < MAX_LENGTH +assert len(serialized_bytes) < 2**(8 * LENGTH_BYTES) serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` From 54a81a8ebf7f328f0d2183f9f822854a756e37ae Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:56:51 +0100 Subject: [PATCH 020/161] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 639231b30..2c0bb8f6a 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -59,8 +59,8 @@ return b'\x01' if value is True else b'\x00' #### Containers, tuples, lists ```python -LENGTH_BYTES = 4 serialized_bytes = ''.join([serialize(element) for element in value]) +LENGTH_BYTES = 4 assert len(serialized_bytes) < 2**(8 * LENGTH_BYTES) serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes @@ -78,7 +78,7 @@ We first define helper functions: * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. -Let `value` be an object. We now define object Merkleization `hash_tree_root(value)` recursively: +We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: * `merkleize(pack(value))` if `value` is a basic object or a tuple of basic objects * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects From e76619358828638cf2cdf495e5029611a16861d9 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:59:49 +0100 Subject: [PATCH 021/161] Update simple-serialize.md --- specs/simple-serialize.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 2c0bb8f6a..7f6866cb5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -26,9 +26,12 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types -* **container**: ordered heterogenous collection of values (key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}`) -* **tuple**: ordered fixed-length homogeneous collection of values (angle braket notation `[N]`, e.g. `uint64[N]`) -* **list**: ordered variable-length homogenous collection of values (angle braket notation `[]`, e.g. `uint64[]`) +* **container**: ordered heterogenous collection of values + * key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}` +* **tuple**: ordered fixed-length homogeneous collection of values + * angle braket notation `[N]`, e.g. `uint64[N]` +* **list**: ordered variable-length homogenous collection of values + * angle braket notation `[]`, e.g. `uint64[]` #### Aliases @@ -40,7 +43,7 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We reccursively define the serialisation `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. #### `uintN` From 99074d09ecba1c02182e23496fcc638f289b2722 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 18:00:49 +0100 Subject: [PATCH 022/161] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 7f6866cb5..565b0d788 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -33,7 +33,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **list**: ordered variable-length homogenous collection of values * angle braket notation `[]`, e.g. `uint64[]` -#### Aliases +### Aliases For convenience we alias: From bac352e70f825da9ac5a7155a5a8581033de15c3 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 18:02:53 +0100 Subject: [PATCH 023/161] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 565b0d788..e2ce4f208 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -43,7 +43,7 @@ For convenience we alias: ## Serialization -We reccursively define the serialisation `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. #### `uintN` From e2f4e32332de6ff28815b69dba861c292686222a Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 27 Feb 2019 11:27:23 -0700 Subject: [PATCH 024/161] fix committee start shard bug --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index e8347704e..05bb37f48 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1978,8 +1978,8 @@ def update_validator_registry(state: BeaconState) -> None: and perform the following updates: -* Set `state.current_shuffling_epoch = next_epoch` * Set `state.current_shuffling_start_shard = (state.current_shuffling_start_shard + get_current_epoch_committee_count(state)) % SHARD_COUNT` +* Set `state.current_shuffling_epoch = next_epoch` * Set `state.current_shuffling_seed = generate_seed(state, state.current_shuffling_epoch)` If a validator registry update does _not_ happen do the following: From c9ac8170f018be1390bf66a172ee45b794b760d4 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 27 Feb 2019 11:37:54 -0700 Subject: [PATCH 025/161] ensure that next committee assignemtns is stable when need to be --- specs/core/0_beacon-chain.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 05bb37f48..7d26c06f9 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -876,18 +876,21 @@ def get_crosslink_committees_at_slot(state: BeaconState, shuffling_epoch = state.previous_shuffling_epoch shuffling_start_shard = state.previous_shuffling_start_shard elif epoch == next_epoch: - current_committees_per_epoch = get_current_epoch_committee_count(state) - committees_per_epoch = get_next_epoch_committee_count(state) - shuffling_epoch = next_epoch - epochs_since_last_registry_update = current_epoch - state.validator_registry_update_epoch if registry_change: + committees_per_epoch = get_next_epoch_committee_count(state) + shuffling_epoch = next_epoch seed = generate_seed(state, next_epoch) + current_committees_per_epoch = get_current_epoch_committee_count(state) shuffling_start_shard = (state.current_shuffling_start_shard + current_committees_per_epoch) % SHARD_COUNT elif epochs_since_last_registry_update > 1 and is_power_of_two(epochs_since_last_registry_update): + committees_per_epoch = get_next_epoch_committee_count(state) + shuffling_epoch = next_epoch seed = generate_seed(state, next_epoch) shuffling_start_shard = state.current_shuffling_start_shard else: + committees_per_epoch = get_current_epoch_committee_count(state) + shuffling_epoch = state.current_shuffling_epoch seed = state.current_shuffling_seed shuffling_start_shard = state.current_shuffling_start_shard From 8176cc4cf0c9eaee68abbf24ca4344dd1fac90e1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 27 Feb 2019 22:23:42 +0100 Subject: [PATCH 026/161] Apply suggestions from code review Co-Authored-By: JustinDrake --- specs/simple-serialize.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index e2ce4f208..221a48144 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -43,23 +43,23 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. -#### `uintN` +### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] return value.to_bytes(N // 8, 'little') ``` -#### `bool` +### `bool` ```python assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -#### Containers, tuples, lists +### Containers, tuples, lists ```python serialized_bytes = ''.join([serialize(element) for element in value]) @@ -79,7 +79,7 @@ We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. -* `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. +* `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 74349eacd25c96d2678d4a0afbc2ce8f4436dcc4 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 22:33:36 +0100 Subject: [PATCH 027/161] Update simple-serialize.md --- specs/simple-serialize.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 221a48144..fb5ee23ae 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -4,6 +4,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Table of contents +- [Constants](#constants) - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) @@ -17,8 +18,14 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Self-signed containers](#self-signed-containers) - [Implementations](#implementations) -## Typing +## Constants +| Name | Value | Description | +|-|-|-| +| `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk. +| `BYTES_PER_LENGTH_PREFIX` | `4` | Number of bytes per serialized length prefix. | + +## Typing ### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) @@ -27,7 +34,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types * **container**: ordered heterogenous collection of values - * key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}` + * key-pair curly braket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` * **tuple**: ordered fixed-length homogeneous collection of values * angle braket notation `[N]`, e.g. `uint64[N]` * **list**: ordered variable-length homogenous collection of values @@ -63,9 +70,8 @@ return b'\x01' if value is True else b'\x00' ```python serialized_bytes = ''.join([serialize(element) for element in value]) -LENGTH_BYTES = 4 -assert len(serialized_bytes) < 2**(8 * LENGTH_BYTES) -serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') +assert len(serialized_bytes) < 2**(8 * BYTES_PER_LENGTH_PREFIX) +serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, 'little') return serialized_length + serialized_bytes ``` @@ -77,8 +83,8 @@ Given a type, serialization is an injective function from objects of that type t We first define helper functions: -* `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. +* `pack`: Given ordered objects of the same basic type, serialize them, pack them into BYTES_PER_CHUNK-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleize`: Given ordered BYTES_PER_CHUNK-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 3e6934b602d58656ffe4d5a31b0c238ce74cbb66 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 22:44:20 +0100 Subject: [PATCH 028/161] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index fb5ee23ae..6f3186aee 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,8 +83,8 @@ Given a type, serialization is an injective function from objects of that type t We first define helper functions: -* `pack`: Given ordered objects of the same basic type, serialize them, pack them into BYTES_PER_CHUNK-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered BYTES_PER_CHUNK-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. +* `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 5692b3483d820d32d899a97735b4f90d9132f1ee Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 27 Feb 2019 15:02:11 -0700 Subject: [PATCH 029/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 1bc07fb31..c2f7b7791 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1650,7 +1650,7 @@ For each `attester_slashing` in `block.body.attester_slashings`: * Verify that `is_double_vote(slashable_attestation_1.data, slashable_attestation_2.data)` or `is_surround_vote(slashable_attestation_1.data, slashable_attestation_2.data)`. * Verify that `verify_slashable_attestation(state, slashable_attestation_1)`. * Verify that `verify_slashable_attestation(state, slashable_attestation_2)`. -* Let `slashable_indices = [index for index in slashable_attestation_1.validator_indices if index in slashable_attestation_2.validator_indices and not state.validator_registry[index].slashed]`. +* Let `slashable_indices = [index for index in slashable_attestation_1.validator_indices if index in slashable_attestation_2.validator_indices and state.validator_registry[index].slashed is False]`. * Verify that `len(slashable_indices) >= 1`. * Run `slash_validator(state, index)` for each `index` in `slashable_indices`. @@ -1885,7 +1885,7 @@ Case 2: `epochs_since_finality > 4`: * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_boundary_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_head_attester_indices`, loses `base_reward(state, index)`. -* Any [active validator](#dfn-active-validator) `index` with `validator.slashed`, loses `2 * inactivity_penalty(state, index, epochs_since_finality) + base_reward(state, index)`. +* Any [active validator](#dfn-active-validator) `index` with `validator.slashed is True`, loses `2 * inactivity_penalty(state, index, epochs_since_finality) + base_reward(state, index)`. * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` loses `base_reward(state, index) - base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index)` ##### Attestation inclusion From 34091d70ec9bc2fae0eda3256b9bad3e30dff841 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 28 Feb 2019 16:30:46 +0800 Subject: [PATCH 030/161] Fix typo --- specs/core/0_beacon-chain.md | 3 ++- specs/validator/0_beacon-chain-validator.md | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0592919e4..ba22c716b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -55,6 +55,7 @@ - [Helper functions](#helper-functions) - [`hash`](#hash) - [`hash_tree_root`](#hash_tree_root) + - [`signed_root`](#signed_root) - [`slot_to_epoch`](#slot_to_epoch) - [`get_previous_epoch`](#get_previous_epoch) - [`get_current_epoch`](#get_current_epoch) @@ -968,7 +969,7 @@ def generate_seed(state: BeaconState, ```python def get_beacon_proposer_index(state: BeaconState, slot: Slot, - registry_change: bool=False)) -> ValidatorIndex: + registry_change: bool=False) -> ValidatorIndex: """ Return the beacon proposer index for the ``slot``. """ diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 949a5e429..e1083c5c5 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -50,7 +50,7 @@ __NOTICE__: This document is a work-in-progress for researchers and implementers - [Aggregation bitfield](#aggregation-bitfield) - [Custody bitfield](#custody-bitfield) - [Aggregate signature](#aggregate-signature) - - [Validator assigments](#validator-assignments) + - [Validator assignments](#validator-assignments) - [Lookahead](#lookahead) - [How to avoid slashing](#how-to-avoid-slashing) - [Proposer slashing](#proposer-slashing) @@ -371,7 +371,7 @@ def get_committee_assignment( if len(selected_committees) > 0: validators = selected_committees[0][0] shard = selected_committees[0][1] - is_proposer = validator_index == get_beacon_proposer_index(state, slot, registry_change) + is_proposer = validator_index == get_beacon_proposer_index(state, slot, registry_change=registry_change) assignment = (validators, shard, slot, is_proposer) return assignment From 7ab7abe2ca7f66ac75e854c20b4ff0ef26cab1f7 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 28 Feb 2019 11:32:54 +0100 Subject: [PATCH 031/161] Update simple-serialize.md --- specs/simple-serialize.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6f3186aee..8830ae7e4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -34,11 +34,11 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types * **container**: ordered heterogenous collection of values - * key-pair curly braket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` + * key-pair curly bracket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` * **tuple**: ordered fixed-length homogeneous collection of values - * angle braket notation `[N]`, e.g. `uint64[N]` + * angle bracket notation `[N]`, e.g. `uint64[N]` * **list**: ordered variable-length homogenous collection of values - * angle braket notation `[]`, e.g. `uint64[]` + * angle bracket notation `[]`, e.g. `uint64[]` ### Aliases @@ -50,7 +50,9 @@ For convenience we alias: ## Serialization -We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `bytes`. + +*Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type. ### `uintN` @@ -77,14 +79,14 @@ return serialized_length + serialized_bytes ## Deserialization -Given a type, serialization is an injective function from objects of that type to byte strings. That is, deserialization—the inverse function—is well-defined. +Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). ## Merkleization We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. +* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -96,7 +98,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi ## Self-signed containers -Let `container` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `container`. Further, the signed message for `container` is `signed_root(container) = hash_tree_root(truncate_last(container))` where `truncate_last` truncates the last element of `container`. +Let `value` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signed_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`. ## Implementations From 67793edc8d6c1b9c831c0b33246a442ceb69d3d3 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 28 Feb 2019 11:34:24 +0100 Subject: [PATCH 032/161] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 8830ae7e4..faba9e762 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -29,7 +29,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) -* `bool`: 1-bit unsigned integer +* `bool`: `True` or `False` ### Composite types From dbbfb938735a328d4ebad090d18eb7d8c3601f4b Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 28 Feb 2019 19:07:37 +0800 Subject: [PATCH 033/161] Order by `committees_per_epoch`, `seed`, `shuffling_epoch`, and `shuffling_start_shard` --- specs/core/0_beacon-chain.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 7d26c06f9..7ab7e42c5 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -55,6 +55,7 @@ - [Helper functions](#helper-functions) - [`hash`](#hash) - [`hash_tree_root`](#hash_tree_root) + - [`signed_root`](#signed_root) - [`slot_to_epoch`](#slot_to_epoch) - [`get_previous_epoch`](#get_previous_epoch) - [`get_current_epoch`](#get_current_epoch) @@ -879,19 +880,19 @@ def get_crosslink_committees_at_slot(state: BeaconState, epochs_since_last_registry_update = current_epoch - state.validator_registry_update_epoch if registry_change: committees_per_epoch = get_next_epoch_committee_count(state) - shuffling_epoch = next_epoch seed = generate_seed(state, next_epoch) + shuffling_epoch = next_epoch current_committees_per_epoch = get_current_epoch_committee_count(state) shuffling_start_shard = (state.current_shuffling_start_shard + current_committees_per_epoch) % SHARD_COUNT elif epochs_since_last_registry_update > 1 and is_power_of_two(epochs_since_last_registry_update): committees_per_epoch = get_next_epoch_committee_count(state) - shuffling_epoch = next_epoch seed = generate_seed(state, next_epoch) + shuffling_epoch = next_epoch shuffling_start_shard = state.current_shuffling_start_shard else: committees_per_epoch = get_current_epoch_committee_count(state) - shuffling_epoch = state.current_shuffling_epoch seed = state.current_shuffling_seed + shuffling_epoch = state.current_shuffling_epoch shuffling_start_shard = state.current_shuffling_start_shard shuffling = get_shuffling( From f38dc048e44068d3cae2a05b4d44d733168be721 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Thu, 28 Feb 2019 17:17:00 +0300 Subject: [PATCH 034/161] Make 'Any active validator' statement more strict. --- specs/core/0_beacon-chain.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index d73761af7..939c2d343 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1862,29 +1862,28 @@ Note: When applying penalties in the following balance recalculations implemente ##### Justification and finalization -Note: Rewards and penalties are for participation in the previous epoch, so the "active validator" set is drawn from `get_active_validator_indices(state.validator_registry, previous_epoch)`. - +* Let `previous_active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch)` * Let `epochs_since_finality = next_epoch - state.finalized_epoch`. Case 1: `epochs_since_finality <= 4`: * Expected FFG source: * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` gains `base_reward(state, index) * previous_epoch_attesting_balance // previous_total_balance`. - * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_attester_indices` loses `base_reward(state, index)`. + * Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_attester_indices` loses `base_reward(state, index)`. * Expected FFG target: * Any [validator](#dfn-validator) `index` in `previous_epoch_boundary_attester_indices` gains `base_reward(state, index) * previous_epoch_boundary_attesting_balance // previous_total_balance`. - * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_boundary_attester_indices` loses `base_reward(state, index)`. + * Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_boundary_attester_indices` loses `base_reward(state, index)`. * Expected beacon chain head: * Any [validator](#dfn-validator) `index` in `previous_epoch_head_attester_indices` gains `base_reward(state, index) * previous_epoch_head_attesting_balance // previous_total_balance)`. - * Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_head_attester_indices` loses `base_reward(state, index)`. + * Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_head_attester_indices` loses `base_reward(state, index)`. * Inclusion distance: * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` gains `base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index)` Case 2: `epochs_since_finality > 4`: -* Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. -* Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_boundary_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. -* Any [active validator](#dfn-active-validator) `index` not in `previous_epoch_head_attester_indices`, loses `base_reward(state, index)`. +* Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. +* Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_boundary_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. +* Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_head_attester_indices`, loses `base_reward(state, index)`. * Any [active validator](#dfn-active-validator) `index` with `validator.slashed is True`, loses `2 * inactivity_penalty(state, index, epochs_since_finality) + base_reward(state, index)`. * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` loses `base_reward(state, index) - base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index)` From b836b30bff9154a0dd809344c45033fc677f4b47 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 26 Feb 2019 15:10:34 +0800 Subject: [PATCH 035/161] Fix "is_ready_to_exit" condition --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index d73761af7..6d72c301c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1964,7 +1964,7 @@ def update_validator_registry(state: BeaconState) -> None: # Exit validators within the allowable balance churn balance_churn = 0 for index, validator in enumerate(state.validator_registry): - if validator.activation_epoch == FAR_FUTURE_EPOCH and validator.initiated_exit: + if validator.exit_epoch == FAR_FUTURE_EPOCH and validator.initiated_exit: # Check the balance churn would be within the allowance balance_churn += get_effective_balance(state, index) if balance_churn > max_balance_churn: From bdd68687361531c7a4680d24a9e38a88d09d6691 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 26 Feb 2019 17:07:10 +0800 Subject: [PATCH 036/161] Fix ToC --- specs/core/0_beacon-chain.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6d72c301c..1c71cf4cf 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -55,6 +55,7 @@ - [Helper functions](#helper-functions) - [`hash`](#hash) - [`hash_tree_root`](#hash_tree_root) + - [`signed_root`](#signed_root) - [`slot_to_epoch`](#slot_to_epoch) - [`get_previous_epoch`](#get_previous_epoch) - [`get_current_epoch`](#get_current_epoch) From bcb0b8bf874eb438327deda2894c0fb51da8f6b8 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 26 Feb 2019 17:07:59 +0800 Subject: [PATCH 037/161] Refactor `exit_validator` --- specs/core/0_beacon-chain.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 1c71cf4cf..e31ecf2ea 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1316,12 +1316,13 @@ def exit_validator(state: BeaconState, index: ValidatorIndex) -> None: Note that this function mutates ``state``. """ validator = state.validator_registry[index] + delayed_activation_exit_epoch = get_delayed_activation_exit_epoch(get_current_epoch(state)) # The following updates only occur if not previous exited - if validator.exit_epoch <= get_delayed_activation_exit_epoch(get_current_epoch(state)): + if validator.exit_epoch <= delayed_activation_exit_epoch: return - - validator.exit_epoch = get_delayed_activation_exit_epoch(get_current_epoch(state)) + else: + validator.exit_epoch = delayed_activation_exit_epoch ``` #### `slash_validator` From d5900836aadcd63fb1b02cbdc21a9fcc41653fdd Mon Sep 17 00:00:00 2001 From: vbuterin Date: Thu, 28 Feb 2019 20:54:59 -0600 Subject: [PATCH 038/161] Convert transaction descriptions from English to python code Only a change of presentation, does not make any substantive changes. Does NOT handle Deposit transactions because that is a special case for which I recommend a deeper change (will do this in another PR soon). --- specs/core/0_beacon-chain.md | 190 +++++++++++++++++++++++++---------- 1 file changed, 139 insertions(+), 51 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3ea4d4101..e179ec43a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1629,59 +1629,107 @@ Below are the processing steps that happen at every `block`. Verify that `len(block.body.proposer_slashings) <= MAX_PROPOSER_SLASHINGS`. -For each `proposer_slashing` in `block.body.proposer_slashings`: +For each `proposer_slashing` in `block.body.proposer_slashings`, run the following function: -* Let `proposer = state.validator_registry[proposer_slashing.proposer_index]`. -* Verify that `proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot`. -* Verify that `proposer_slashing.proposal_1.shard == proposer_slashing.proposal_2.shard`. -* Verify that `proposer_slashing.proposal_1.block_root != proposer_slashing.proposal_2.block_root`. -* Verify that `proposer.slashed is False`. -* Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposer_slashing.proposal_1, "signature"), signature=proposer_slashing.proposal_1.signature, domain=get_domain(state.fork, slot_to_epoch(proposer_slashing.proposal_1.slot), DOMAIN_PROPOSAL))`. -* Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposer_slashing.proposal_2, "signature"), signature=proposer_slashing.proposal_2.signature, domain=get_domain(state.fork, slot_to_epoch(proposer_slashing.proposal_2.slot), DOMAIN_PROPOSAL))`. -* Run `slash_validator(state, proposer_slashing.proposer_index)`. +```python +def process_proposer_slashing(proposer_slashing: ProposerSlashing, + state: BeaconState): + proposer = state.validator_registry[proposer_slashing.proposer_index] + # Verify that the slot is the same + assert proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot + # Verify that the shard is the same (or that both proposals are beacon chain proposals) + assert proposer_slashing.proposal_1.shard == proposer_slashing.proposal_2.shard + # But the roots are different! + assert proposer_slashing.proposal_1.block_root != proposer_slashing.proposal_2.block_root + # Proposer is not yet slashed + assert proposer.slashed is False + # Signatures are valid + for proposal in (proposer_slashing.proposal_1, proposer_slashing.proposal_2): + assert bls_verify( + pubkey=proposer.pubkey, + message_hash=signed_root(proposal, "signature"), + signature=proposal.signature, + domain=get_domain(state.fork, slot_to_epoch(proposal.slot), DOMAIN_PROPOSAL) + ) + slash_validator(state, proposer_slashing.proposer_index) +``` ##### Attester slashings Verify that `len(block.body.attester_slashings) <= MAX_ATTESTER_SLASHINGS`. -For each `attester_slashing` in `block.body.attester_slashings`: +For each `attester_slashing` in `block.body.attester_slashings`, run the following function: -* Let `slashable_attestation_1 = attester_slashing.slashable_attestation_1`. -* Let `slashable_attestation_2 = attester_slashing.slashable_attestation_2`. -* Verify that `slashable_attestation_1.data != slashable_attestation_2.data`. -* Verify that `is_double_vote(slashable_attestation_1.data, slashable_attestation_2.data)` or `is_surround_vote(slashable_attestation_1.data, slashable_attestation_2.data)`. -* Verify that `verify_slashable_attestation(state, slashable_attestation_1)`. -* Verify that `verify_slashable_attestation(state, slashable_attestation_2)`. -* Let `slashable_indices = [index for index in slashable_attestation_1.validator_indices if index in slashable_attestation_2.validator_indices and state.validator_registry[index].slashed is False]`. -* Verify that `len(slashable_indices) >= 1`. -* Run `slash_validator(state, index)` for each `index` in `slashable_indices`. +```python +def process_attester_slashing(attester_slashing: ProposerSlashing, + state: BeaconState): + attestation1 = attester_slashing.slashable_attestation_1 + attestation2 = attester_slashing.slashable_attestation_2 + # Check that the attestations are conflicting + assert attestation1.data != attestation2.data + assert ( + is_double_vote(attestation1.data, attestation2.data) or + is_surround_vote(attestation1.data, attestation2.data) + ) + assert verify_slashable_attestation(state, attestation1) + assert verify_slashable_attestation(state, attestation2) + slashable_indices = [ + index for index in attestation1.validator_indices if + index in attestation2.validator_indices and state.validator_registry[index].slashed is False + ] + assert len(slashable_indices) >= 1 + for index in slashable_indices: + slash_validator(state, index) +``` ##### Attestations Verify that `len(block.body.attestations) <= MAX_ATTESTATIONS`. -For each `attestation` in `block.body.attestations`: - -* Verify that `attestation.data.slot >= GENESIS_SLOT`. -* Verify that `attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot`. -* Verify that `state.slot < attestation.data.slot + SLOTS_PER_EPOCH. -* Verify that `attestation.data.justified_epoch` is equal to `state.justified_epoch if slot_to_epoch(attestation.data.slot + 1) >= get_current_epoch(state) else state.previous_justified_epoch`. -* Verify that `attestation.data.justified_block_root` is equal to `get_block_root(state, get_epoch_start_slot(attestation.data.justified_epoch))`. -* Verify that either (i) `state.latest_crosslinks[attestation.data.shard] == attestation.data.latest_crosslink` or (ii) `state.latest_crosslinks[attestation.data.shard] == Crosslink(crosslink_data_root=attestation.data.crosslink_data_root, epoch=slot_to_epoch(attestation.data.slot))`. -* Verify bitfields and aggregate signature: +For each `attestation` in `block.body.attestations`, , run the following function: ```python - assert attestation.custody_bitfield == b'\x00' * len(attestation.custody_bitfield) # [TO BE REMOVED IN PHASE 1] +def process_attestation(attestation: Attestation, state: BeaconState): + # Can't submit attestations that are too far in history (or in prehistory) + assert attestation.data.slot >= GENESIS_SLOT + state.slot < attestation.data.slot + SLOTS_PER_EPOCH + # Can't submit attestations too quickly + attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot + # Verify that the justified epoch is correct, case 1: current epoch attestations + if slot_to_epoch(attestation.data.slot + 1) >= get_current_epoch(state): + assert attestation.data.justified_epoch == state.justified_epoch + # Case 2: previous epoch attestations + else: + assert attestation.data.justified_epoch == state.previous_justified_epoch + # Check that the justified block root is correct + assert attestation.data.justified_block_root == get_block_root( + state, get_epoch_start_slot(attestation.data.justified_epoch) + ) + # Check that the crosslink data is valid + assert ( + # Case 1: Latest crosslink matches the one in the state + state.latest_crosslinks[attestation.data.shard] == attestation.data.latest_crosslink or + # Case 2: State has already been updated, state's latest crosslink matches the crosslink + # the attestation is trying to create + state.latest_crosslinks[attestation.data.shard] == Crosslink( + crosslink_data_root=attestation.data.crosslink_data_root, + epoch=slot_to_epoch(attestation.data.slot) + ) + ) + # Attestation must be nonempty! assert attestation.aggregation_bitfield != b'\x00' * len(attestation.aggregation_bitfield) - + # Custody must be empty (to be removed in phase 1) + assert attestation.custody_bitfield == b'\x00' * len(attestation.custody_bitfield) + # Get the committee for the specific shard that this attestation is for crosslink_committee = [ committee for committee, shard in get_crosslink_committees_at_slot(state, attestation.data.slot) if shard == attestation.data.shard ][0] + # Custody bitfield must be a subset of the attestation bitfield for i in range(len(crosslink_committee)): if get_bitfield_bit(attestation.aggregation_bitfield, i) == 0b0: assert get_bitfield_bit(attestation.custody_bitfield, i) == 0b0 - + # Verify aggregate signature participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) custody_bit_1_participants = get_attestation_participants(state, attestation.data, attestation.custody_bitfield) custody_bit_0_participants = [i in participants for i not in custody_bit_1_participants] @@ -1698,11 +1746,17 @@ For each `attestation` in `block.body.attestations`: signature=attestation.aggregate_signature, domain=get_domain(state.fork, slot_to_epoch(attestation.data.slot), DOMAIN_ATTESTATION), ) + # Crosslink data root is zero (to be removed in phase 1) + assert attestation.data.crosslink_data_root == ZERO_HASH + # Apply the attestation + state.latest_attestations.append(PendingAttestation( + data=attestation.data, + aggregation_bitfield=attestation.aggregation_bitfield, + custody_bitfield=attestation.custody_bitfield, + inclusion_slot=state.slot) + ) ``` -* [TO BE REMOVED IN PHASE 1] Verify that `attestation.data.crosslink_data_root == ZERO_HASH`. -* Append `PendingAttestation(data=attestation.data, aggregation_bitfield=attestation.aggregation_bitfield, custody_bitfield=attestation.custody_bitfield, inclusion_slot=state.slot)` to `state.latest_attestations`. - ##### Deposits Verify that `len(block.body.deposits) <= MAX_DEPOSITS`. @@ -1741,13 +1795,25 @@ process_deposit(state, deposit) Verify that `len(block.body.voluntary_exits) <= MAX_VOLUNTARY_EXITS`. -For each `exit` in `block.body.voluntary_exits`: +For each `exit` in `block.body.voluntary_exits`, run the following function: -* Let `validator = state.validator_registry[exit.validator_index]`. -* Verify that `validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state))`. -* Verify that `get_current_epoch(state) >= exit.epoch`. -* Verify that `bls_verify(pubkey=validator.pubkey, message_hash=signed_root(exit, "signature"), signature=exit.signature, domain=get_domain(state.fork, exit.epoch, DOMAIN_EXIT))`. -* Run `initiate_validator_exit(state, exit.validator_index)`. +```python +def process_exit(exit: VoluntaryExit, state: BeaconState): + validator = state.validator_registry[exit.validator_index] + # Verify the validator has not yet exited + assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) + # Exits must specify an epoch when they become valid; they are not valid before then + assert get_current_epoch(state) >= exit.epoch + # Verify signature + assert bls_verify( + pubkey=validator.pubkey, + message_hash=signed_root(exit, "signature"), + signature=exit.signature, + domain=get_domain(state.fork, exit.epoch, DOMAIN_EXIT) + ) + # Run the exit + initiate_validator_exit(state, exit.validator_index) +``` ##### Transfers @@ -1755,18 +1821,40 @@ Note: Transfers are a temporary functionality for phases 0 and 1, to be removed Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers are distinct. -For each `transfer` in `block.body.transfers`: +For each `transfer` in `block.body.transfers`, run the following function: -* Verify that `state.validator_balances[transfer.from] >= transfer.amount`. -* Verify that `state.validator_balances[transfer.from] >= transfer.fee`. -* Verify that `state.validator_balances[transfer.from] == transfer.amount + transfer.fee` or `state.validator_balances[transfer.from] >= transfer.amount + transfer.fee + MIN_DEPOSIT_AMOUNT`. -* Verify that `state.slot == transfer.slot`. -* Verify that `get_current_epoch(state) >= state.validator_registry[transfer.from].withdrawable_epoch` or `state.validator_registry[transfer.from].activation_epoch == FAR_FUTURE_EPOCH`. -* Verify that `state.validator_registry[transfer.from].withdrawal_credentials == BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:]`. -* Verify that `bls_verify(pubkey=transfer.pubkey, message_hash=signed_root(transfer, "signature"), signature=transfer.signature, domain=get_domain(state.fork, slot_to_epoch(transfer.slot), DOMAIN_TRANSFER))`. -* Set `state.validator_balances[transfer.from] -= transfer.amount + transfer.fee`. -* Set `state.validator_balances[transfer.to] += transfer.amount`. -* Set `state.validator_balances[get_beacon_proposer_index(state, state.slot)] += transfer.fee`. +```python +def process_exit(transfer: Transfer, state: BeaconState): + # Verify that we have enough ETH to send, and that after the transfer the balance will be either + # exactly zero or at least MIN_DEPOSIT_AMOUNT + assert ( + state.validator_balances[transfer.from] == transfer.amount + transfer.fee or + state.validator_balances[transfer.from] >= transfer.amount + transfer.fee + MIN_DEPOSIT_AMOUNT + ) + # A transfer is valid in only one slot + assert state.slot == transfer.slot + # Only withdrawn or not-yet-deposited accounts can transfer + assert ( + get_current_epoch(state) >= state.validator_registry[transfer.from].withdrawable_epoch or + state.validator_registry[transfer.from].activation_epoch == FAR_FUTURE_EPOCH + ) + # Verify that the pubkey is valid + assert ( + state.validator_registry[transfer.from].withdrawal_credentials == + BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:] + ) + # Verify that the signature is valid + assert bls_verify( + pubkey=transfer.pubkey, + message_hash=signed_root(transfer, "signature"), + signature=transfer.signature, + domain=get_domain(state.fork, slot_to_epoch(transfer.slot), DOMAIN_TRANSFER) + ) + # Process the transfer + state.validator_balances[transfer.from] -= transfer.amount + transfer.fee + state.validator_balances[transfer.to] += transfer.amount + state.validator_balances[get_beacon_proposer_index(state, state.slot)] += transfer.fee +``` ### Per-epoch processing From 80ac62606cdfe5ed00b001a48eaef00093dc4edb Mon Sep 17 00:00:00 2001 From: vbuterin Date: Thu, 28 Feb 2019 21:07:10 -0600 Subject: [PATCH 039/161] Unified deposit processing between genesis and transaction --- specs/core/0_beacon-chain.md | 77 +++++++++++++++++++++--------------- 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3ea4d4101..dd6f440d6 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -76,6 +76,7 @@ - [`generate_seed`](#generate_seed) - [`get_beacon_proposer_index`](#get_beacon_proposer_index) - [`merkle_root`](#merkle_root) + - [`verify_merkle_branch`](#verify_merkle_branch) - [`get_attestation_participants`](#get_attestation_participants) - [`is_power_of_two`](#is_power_of_two) - [`int_to_bytes1`, `int_to_bytes2`, ...](#int_to_bytes1-int_to_bytes2-) @@ -990,6 +991,22 @@ def merkle_root(values: List[Bytes32]) -> Bytes32: return o[1] ``` +### `verify_merkle_branch` + +```python +def verify_merkle_branch(leaf: Bytes32, branch: List[Bytes32], depth: int, index: int, root: Bytes32) -> bool: + """ + Verify that the given ``leaf`` is on the merkle branch ``branch``. + """ + value = leaf + for i in range(depth): + if index // (2**i) % 2: + value = hash(branch[i] + value) + else: + value = hash(value + branch[i]) + return value == root +``` + ### `get_attestation_participants` ```python @@ -1236,6 +1253,32 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: """ deposit_input = deposit.deposit_data.deposit_input + # Should equal 8 bytes for deposit_data.amount + + # 8 bytes for deposit_data.timestamp + + # 176 bytes for deposit_data.deposit_input + # It should match the deposit_data in the eth1.0 deposit contract + serialized_deposit_data = ssz_serialize(deposit.deposit_data) + # Deposits must be processed in order + assert deposit.index == state.deposit_index + + # Verify the Merkle branch + merkle_branch_is_valid = verify_merkle_branch( + leaf=hash(serialized_deposit_data), + branch=deposit.branch, + depth=DEPOSIT_CONTRACT_TREE_DEPTH, + index=deposit.index, + root=state.latest_eth1_data.deposit_root + ) + if not merkle_branch_is_valid: + return + + # Increment the next deposit index we are expecting. Note that this + # needs to be done here because while the deposit contract will never + # create an invalid Merkle branch, it may admit an invalid deposit + # object, and we need to be able to skip over it + state.deposit_index += 1 + + # Verify the proof of possession proof_is_valid = bls_verify( pubkey=deposit_input.pubkey, message_hash=signed_root(deposit_input, "proof_of_possession"), @@ -1247,7 +1290,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: ) ) - if not proof_is_valid: + if not not proof_is_valid: return validator_pubkeys = [v.pubkey for v in state.validator_registry] @@ -1476,7 +1519,7 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], # Ethereum 1.0 chain data latest_eth1_data=latest_eth1_data, eth1_data_votes=[], - deposit_index=len(genesis_validator_deposits) + deposit_index=0, ) # Process genesis deposits @@ -1707,35 +1750,7 @@ For each `attestation` in `block.body.attestations`: Verify that `len(block.body.deposits) <= MAX_DEPOSITS`. -[TODO: update the call to `verify_merkle_branch` below if it needs to change after we process deposits in order] - -For each `deposit` in `block.body.deposits`: - -* Let `serialized_deposit_data` be the serialized form of `deposit.deposit_data`. It should be 8 bytes for `deposit_data.amount` followed by 8 bytes for `deposit_data.timestamp` and then the `DepositInput` bytes. That is, it should match `deposit_data` in the [Ethereum 1.0 deposit contract](#ethereum-10-deposit-contract) of which the hash was placed into the Merkle tree. -* Verify that `deposit.index == state.deposit_index`. -* Verify that `verify_merkle_branch(hash(serialized_deposit_data), deposit.branch, DEPOSIT_CONTRACT_TREE_DEPTH, deposit.index, state.latest_eth1_data.deposit_root)` is `True`. - -```python -def verify_merkle_branch(leaf: Bytes32, branch: List[Bytes32], depth: int, index: int, root: Bytes32) -> bool: - """ - Verify that the given ``leaf`` is on the merkle branch ``branch``. - """ - value = leaf - for i in range(depth): - if index // (2**i) % 2: - value = hash(branch[i] + value) - else: - value = hash(value + branch[i]) - return value == root -``` - -* Run the following: - -```python -process_deposit(state, deposit) -``` - -* Set `state.deposit_index += 1`. +For each `deposit` in `block.body.deposits`, run `process_deposit(state, deposit)` ##### Voluntary exits From 087576a7bba6ebae69a0372adf7c38b1a6e97429 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 1 Mar 2019 00:12:45 -0600 Subject: [PATCH 040/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index dd6f440d6..87d48a225 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1257,7 +1257,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: # 8 bytes for deposit_data.timestamp + # 176 bytes for deposit_data.deposit_input # It should match the deposit_data in the eth1.0 deposit contract - serialized_deposit_data = ssz_serialize(deposit.deposit_data) + serialized_deposit_data = serialize(deposit.deposit_data) # Deposits must be processed in order assert deposit.index == state.deposit_index From 19c3189c1c4e81fd996b2acf8d99ec6c7df50031 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 1 Mar 2019 00:13:22 -0600 Subject: [PATCH 041/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 87d48a225..629fdc041 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1750,7 +1750,7 @@ For each `attestation` in `block.body.attestations`: Verify that `len(block.body.deposits) <= MAX_DEPOSITS`. -For each `deposit` in `block.body.deposits`, run `process_deposit(state, deposit)` +For each `deposit` in `block.body.deposits`, run `process_deposit(state, deposit)`. ##### Voluntary exits From a9be1018d3d83d73873ea9bea2ddd55e7def69d0 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 1 Mar 2019 00:13:32 -0600 Subject: [PATCH 042/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 629fdc041..3f8223520 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1290,7 +1290,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: ) ) - if not not proof_is_valid: + if not proof_is_valid: return validator_pubkeys = [v.pubkey for v in state.validator_registry] From 5ed0f276f6b183d30e8b34428cff4d59918e00a6 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 1 Mar 2019 00:13:59 -0600 Subject: [PATCH 043/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index e179ec43a..dfca7ffe8 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1686,7 +1686,7 @@ def process_attester_slashing(attester_slashing: ProposerSlashing, Verify that `len(block.body.attestations) <= MAX_ATTESTATIONS`. -For each `attestation` in `block.body.attestations`, , run the following function: +For each `attestation` in `block.body.attestations`, run the following function: ```python def process_attestation(attestation: Attestation, state: BeaconState): From 0fb4cae2d62a04920709a626a48369665594b120 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 1 Mar 2019 00:15:49 -0600 Subject: [PATCH 044/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index dfca7ffe8..719afd079 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1694,7 +1694,7 @@ def process_attestation(attestation: Attestation, state: BeaconState): assert attestation.data.slot >= GENESIS_SLOT state.slot < attestation.data.slot + SLOTS_PER_EPOCH # Can't submit attestations too quickly - attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot + assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot # Verify that the justified epoch is correct, case 1: current epoch attestations if slot_to_epoch(attestation.data.slot + 1) >= get_current_epoch(state): assert attestation.data.justified_epoch == state.justified_epoch From 19665e4ca534fe17ba1253dc74ea903639f8164e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 1 Mar 2019 00:16:05 -0600 Subject: [PATCH 045/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 719afd079..6c8acb7fb 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1692,7 +1692,7 @@ For each `attestation` in `block.body.attestations`, run the following function: def process_attestation(attestation: Attestation, state: BeaconState): # Can't submit attestations that are too far in history (or in prehistory) assert attestation.data.slot >= GENESIS_SLOT - state.slot < attestation.data.slot + SLOTS_PER_EPOCH + assert state.slot < attestation.data.slot + SLOTS_PER_EPOCH # Can't submit attestations too quickly assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot # Verify that the justified epoch is correct, case 1: current epoch attestations From 236298a8e4f783a7b3fdb3338b4e1c0123d5b575 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 03:58:18 -0600 Subject: [PATCH 046/161] Hard assert that the Merkle branch is valid --- specs/core/0_beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3f8223520..aea187730 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1269,8 +1269,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: index=deposit.index, root=state.latest_eth1_data.deposit_root ) - if not merkle_branch_is_valid: - return + assert merkle_branch_is_valid # Increment the next deposit index we are expecting. Note that this # needs to be done here because while the deposit contract will never From 25ef2553c790add9b66f0c5d5eb67bfe0ddc4839 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 1 Mar 2019 12:02:29 +0100 Subject: [PATCH 047/161] Update simple-serialize.md --- specs/simple-serialize.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index faba9e762..31839b27e 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,7 +12,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`uintN`](#uintn) - [`bool`](#bool) - - [Containers, tuples, lists](#containers-tuples-lists) + - [Tuples](#lists) + - [Containers, lists](#containers-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -68,7 +69,13 @@ assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -### Containers, tuples, lists +### Tuples + +```python +return ''.join([serialize(element) for element in value]) +``` + +### Containers, lists ```python serialized_bytes = ''.join([serialize(element) for element in value]) From 5ceec70dd32b568fc95ea804d00c8d9df056f052 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 07:59:52 -0600 Subject: [PATCH 048/161] Pythonize epoch transitions (in progress) --- specs/core/0_beacon-chain.md | 342 +++++++++++++++++++++++++---------- 1 file changed, 249 insertions(+), 93 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3ea4d4101..2b6c3bd07 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1772,135 +1772,291 @@ For each `transfer` in `block.body.transfers`: The steps below happen when `(state.slot + 1) % SLOTS_PER_EPOCH == 0`. -#### Helper variables +#### Helper functions + +We define some helper functions: -* Let `current_epoch = get_current_epoch(state)`. -* Let `previous_epoch = get_previous_epoch(state)`. -* Let `next_epoch = current_epoch + 1`. +```python +def get_current_attestations(state: BeaconState) -> List[PendingAttestation]: + return [a for a in state.latest_attestations if current_epoch == slot_to_epoch(a.data.slot)] +``` -[Validators](#dfn-Validator) attesting during the current epoch: +```python +def get_current_total_balance(state: BeaconState): + return get_total_balance(state, get_active_validator_indices(state.validator_registry, current_epoch)) +``` -* Let `current_total_balance = get_total_balance(state, get_active_validator_indices(state.validator_registry, current_epoch))`. -* Let `current_epoch_attestations = [a for a in state.latest_attestations if current_epoch == slot_to_epoch(a.data.slot)]`. (Note: Each of these attestations votes for the current justified epoch/block root because of the [attestation block validity rules](#attestations-1).) -* Validators justifying the epoch boundary block at the start of the current epoch: - * Let `current_epoch_boundary_attestations = [a for a in current_epoch_attestations if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(current_epoch))]`. - * Let `current_epoch_boundary_attester_indices` be the union of the [validator](#dfn-validator) index sets given by `[get_attestation_participants(state, a.data, a.aggregation_bitfield) for a in current_epoch_boundary_attestations]`. - * Let `current_epoch_boundary_attesting_balance = get_total_balance(state, current_epoch_boundary_attester_indices)`. +```python +def get_previous_total_balance(state: BeaconState): + return get_total_balance(state, get_active_validator_indices(state.validator_registry, previous_epoch)) +``` -[Validators](#dfn-Validator) attesting during the previous epoch: +```python +def get_previous_attestations(state: BeaconState) -> List[PendingAttestation]: + return [a for a in state.latest_attestations if previous_epoch == slot_to_epoch(a.data.slot)] +``` -* Let `previous_total_balance = get_total_balance(state, get_active_validator_indices(state.validator_registry, previous_epoch))`. -* Validators that made an attestation during the previous epoch, targeting the previous justified slot: - * Let `previous_epoch_attestations = [a for a in state.latest_attestations if previous_epoch == slot_to_epoch(a.data.slot)]`. (Note: Each of these attestations votes for the previous justified epoch/block root because of the [attestation block validity rules](#attestations-1).) - * Let `previous_epoch_attester_indices` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.aggregation_bitfield) for a in previous_epoch_attestations]`. - * Let `previous_epoch_attesting_balance = get_total_balance(state, previous_epoch_attester_indices)`. -* Validators justifying the epoch boundary block at the start of the previous epoch: - * Let `previous_epoch_boundary_attestations = [a for a in previous_epoch_attestations if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(previous_epoch))]`. - * Let `previous_epoch_boundary_attester_indices` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.aggregation_bitfield) for a in previous_epoch_boundary_attestations]`. - * Let `previous_epoch_boundary_attesting_balance = get_total_balance(state, previous_epoch_boundary_attester_indices)`. -* Validators attesting to the expected beacon chain head during the previous epoch: - * Let `previous_epoch_head_attestations = [a for a in previous_epoch_attestations if a.data.beacon_block_root == get_block_root(state, a.data.slot)]`. - * Let `previous_epoch_head_attester_indices` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.aggregation_bitfield) for a in previous_epoch_head_attestations]`. - * Let `previous_epoch_head_attesting_balance = get_total_balance(state, previous_epoch_head_attester_indices)`. +```python +def get_attesting_indices(attestations: List[PendingAttestation], state: BeaconState) -> List[int]: + output = set({}) + for a in attestations: + output = output.union([get_attestation_participants(state, a.data, a.aggregation_bitfield)]) + return sorted(list(output)) +``` -**Note**: `previous_total_balance` and `previous_epoch_boundary_attesting_balance` balance might be marginally different than the actual balances during previous epoch transition. Due to the tight bound on validator churn each epoch and small per-epoch rewards/penalties, the potential balance difference is very low and only marginally affects consensus safety. +```python +def get_attesting_balance(attestations: List[PendingAttestation], state: BeaconState) -> List[int]: + return get_total_balance(state, get_attesting_indices(attestations, state)) +``` -For every `slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch))`, let `crosslink_committees_at_slot = get_crosslink_committees_at_slot(state, slot)`. For every `(crosslink_committee, shard)` in `crosslink_committees_at_slot`, compute: +```python +def get_current_epoch_boundary_attestations(state: BeaconState): + return [ + a for a in get_current_attestations(state) if + a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) + ] +``` -* Let `crosslink_data_root` be `state.latest_crosslinks[shard].crosslink_data_root` -* Let `attesting_validator_indices(crosslink_committee, crosslink_data_root)` be the union of the [validator](#dfn-validator) index sets given by `[get_attestation_participants(state, a.data, a.aggregation_bitfield) for a in current_epoch_attestations + previous_epoch_attestations if a.data.shard == shard and a.data.crosslink_data_root == crosslink_data_root]`. -* Let `winning_root(crosslink_committee)` be equal to the value of `crosslink_data_root` such that `get_total_balance(state, attesting_validator_indices(crosslink_committee, crosslink_data_root))` is maximized (ties broken by favoring lexicographically smallest `crosslink_data_root`). -* Let `attesting_validators(crosslink_committee)` be equal to `attesting_validator_indices(crosslink_committee, winning_root(crosslink_committee))` for convenience. -* Let `total_attesting_balance(crosslink_committee) = get_total_balance(state, attesting_validators(crosslink_committee))`. +```python +def get_previous_epoch_boundary_attestations(state: BeaconState): + return [ + a for a in get_previous_attestations(state) if + a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) + ] +``` -Define the following helpers to process attestation inclusion rewards and inclusion distance reward/penalty. For every attestation `a` in `previous_epoch_attestations`: +**Note**: Total balances computed for the previous epoch might be marginally different than the actual total balances during the previous epoch transition. Due to the tight bound on validator churn each epoch and small per-epoch rewards/penalties, the potential balance difference is very low and only marginally affects consensus safety. -* Let `inclusion_slot(state, index) = a.inclusion_slot` for the attestation `a` where `index` is in `get_attestation_participants(state, a.data, a.aggregation_bitfield)`. If multiple attestations are applicable, the attestation with lowest `inclusion_slot` is considered. -* Let `inclusion_distance(state, index) = a.inclusion_slot - a.data.slot` where `a` is the above attestation. +```python +def get_winning_root_and_participants(state: BeaconState, shard: Shard): + all_attestations = get_current_attestations(state) + get_previous_attestations(state) + valid_attestations = [ + a for a in all_attestations if a.data.latest_crosslink == state.latest_crosslinks[shard] + ] + all_roots = [a.data.crosslink_data_root for a in valid_attestations] + + def get_attestations_for(root): + return [a for a in valid_attestations if a.data.crosslink_data_root == root] + + winning_root = max(all_roots, key=lambda r: get_attesting_balance(get_attestations_for(r))) + + return winning_root, get_attesting_indices(get_attestations_for(winning_root)) +``` -#### Eth1 data +```python +def earliest_attestation(state: BeaconState, validator_index: ValidatorIndex): + return min([ + a for a in state.previous_epoch_attestations if + validator_index in get_attestation_participants(state, a.data, a.aggregation_bitfield) + ], key = lambda a: a.inclusion_slot) +``` -If `next_epoch % EPOCHS_PER_ETH1_VOTING_PERIOD == 0`: +```python +def inclusion_slot(state: BeaconState, validator_index: ValidatorIndex): + return earliest_attestation(state, validator_index).inclusion_slot +``` -* If `eth1_data_vote.vote_count * 2 > EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH` for some `eth1_data_vote` in `state.eth1_data_votes` (ie. more than half the votes in this voting period were for that value), set `state.latest_eth1_data = eth1_data_vote.eth1_data`. -* Set `state.eth1_data_votes = []`. +```python +def inclusion_distance(state: BeaconState, validator_index: ValidatorIndex): + attestation = earliest_attestation(state, validator_index) + return attestation.inclusion_slot - attestation.data_slot +``` #### Justification -First, update the justification bitfield: +Run the following function: -* Let `new_justified_epoch = state.justified_epoch`. -* Set `state.justification_bitfield = state.justification_bitfield << 1`. -* Set `state.justification_bitfield |= 2` and `new_justified_epoch = previous_epoch` if `3 * previous_epoch_boundary_attesting_balance >= 2 * previous_total_balance`. -* Set `state.justification_bitfield |= 1` and `new_justified_epoch = current_epoch` if `3 * current_epoch_boundary_attesting_balance >= 2 * current_total_balance`. +```python +def update_justification_and_finalization(state): -Next, update last finalized epoch if possible: - -* Set `state.finalized_epoch = state.previous_justified_epoch` if `(state.justification_bitfield >> 1) % 8 == 0b111 and state.previous_justified_epoch == previous_epoch - 2`. -* Set `state.finalized_epoch = state.previous_justified_epoch` if `(state.justification_bitfield >> 1) % 4 == 0b11 and state.previous_justified_epoch == previous_epoch - 1`. -* Set `state.finalized_epoch = state.justified_epoch` if `(state.justification_bitfield >> 0) % 8 == 0b111 and state.justified_epoch == previous_epoch - 1`. -* Set `state.finalized_epoch = state.justified_epoch` if `(state.justification_bitfield >> 0) % 4 == 0b11 and state.justified_epoch == previous_epoch`. - -Finally, update the following: - -* Set `state.previous_justified_epoch = state.justified_epoch`. -* Set `state.justified_epoch = new_justified_epoch`. + new_justified_epoch = state.justified_epoch + # Rotate the justification bitfield up one epoch to make room for the current epoch + state.justification_bitfield <<= 1 + # If the previous epoch gets justified, fill the second last bit + previous_boundary_attesting_balance = get_attesting_balance(get_previous_epoch_boundary_attestations(state)) + if previous_boundary_attesting_balance * 3 >= get_previous_total_balance(state) * 2: + new_justified_epoch = get_current_epoch(state) - 1 + state.justification_bitfield |= 2 + # If the current epoch gets justified, fill the last bit + current_boundary_attesting_balance = get_attesting_balance(get_current_epoch_boundary_attestations(state)) + if current_boundary_attesting_balance * 3 >= get_current_total_balance(state) * 2: + new_justified_epoch = get_current_epoch(state) + state.justification_bitfield |= 1 + + # Process finalizations + bitfield = state.justification_bitfield + current_epoch = get_current_epoch(state) + # The 2nd/3rd/4th most recent epochs are all justified, the 2nd using the 4th as source + if (bitfield >> 1) % 8 == 0b111 and state.previous_justified_epoch == current_epoch - 3: + state.finalized_epoch = state.previous_justified_epoch + # The 2nd/3rd most recent epochs are both justified, the 2nd using the 3rd as source + if (bitfield >> 1) % 4 == 0b11 and state.previous_justified_epoch == current_epoch - 2: + state.finalized_epoch = state.previous_justified_epoch + # The 1st/2nd/3rd most recent epochs are all justified, the 1st using the 3rd as source + if (bitfield >> 0) % 8 == 0b111 and state.justified_epoch == current_epoch - 2: + state.finalized_epoch = state.justified_epoch + # The 1st/2nd most recent epochs are both justified, the 1st using the 2nd as source + if (bitfield >> 0) % 4 == 0b11 and state.justified_epoch == current_epoch - 1: + state.finalized_epoch = state.justified_epoch + + # Rotate justified epochs + state.previous_justified_epoch = state.justified_epoch + state.justified_epoch = new_justified_epoch +``` #### Crosslinks -For every `slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch))`, let `crosslink_committees_at_slot = get_crosslink_committees_at_slot(state, slot)`. For every `(crosslink_committee, shard)` in `crosslink_committees_at_slot`, compute: +Run the following function: -* Set `state.latest_crosslinks[shard] = Crosslink(epoch=slot_to_epoch(slot), crosslink_data_root=winning_root(crosslink_committee))` if `3 * total_attesting_balance(crosslink_committee) >= 2 * get_total_balance(crosslink_committee)`. +```python +for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): + for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): + winning_root, participants = get_winning_root_and_participants(state, shard) + if 3 * total_balance(participants) >= 2 * total_balance(crosslink_committee): + state.latest_crosslinks[shard] = Crosslink( + epoch=slot_to_epoch(slot), + crosslink_data_root=winning_root + ) +``` + +#### Eth1 data + +Run the following function: + +```python +def maybe_reset_eth1_period(state): + if (get_current_epoch(state) + 1) % EPOCHS_PER_ETH1_VOTING_PERIOD == 0: + for eth1_data_vote in state.eth1_data_votes: + # If a majority of all votes were for a particular eth1_data value, + # then set that as the new canonical value + if eth1_data_vote.vote_count * 2 > EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH: + state.latest_eth1_data = eth1_data_vote.eth1_data + state.eth1_data_votes = [] +``` #### Rewards and penalties First, we define some additional helpers: -* Let `base_reward_quotient = integer_squareroot(previous_total_balance) // BASE_REWARD_QUOTIENT`. -* Let `base_reward(state, index) = get_effective_balance(state, index) // base_reward_quotient // 5` for any validator with the given `index`. -* Let `inactivity_penalty(state, index, epochs_since_finality) = base_reward(state, index) + get_effective_balance(state, index) * epochs_since_finality // INACTIVITY_PENALTY_QUOTIENT // 2` for any validator with the given `index`. +```python +def get_base_reward(state: BeaconState, index: ValidatorIndex) -> int: + adjusted_quotient = integer_squareroot(get_previous_total_balance(state)) // BASE_REWARD_QUOTIENT + return get_effective_balance(state, index) // adjusted_quotient // 5 +``` + +```python +def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex) -> int: + return ( + base_reward(state, index) + + get_effective_balance(state, index) * epochs_since_finality // INACTIVITY_PENALTY_QUOTIENT // 2 + ) +``` Note: When applying penalties in the following balance recalculations implementers should make sure the `uint64` does not underflow. ##### Justification and finalization -* Let `previous_active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch)` -* Let `epochs_since_finality = next_epoch - state.finalized_epoch`. - -Case 1: `epochs_since_finality <= 4`: - -* Expected FFG source: - * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` gains `base_reward(state, index) * previous_epoch_attesting_balance // previous_total_balance`. - * Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_attester_indices` loses `base_reward(state, index)`. -* Expected FFG target: - * Any [validator](#dfn-validator) `index` in `previous_epoch_boundary_attester_indices` gains `base_reward(state, index) * previous_epoch_boundary_attesting_balance // previous_total_balance`. - * Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_boundary_attester_indices` loses `base_reward(state, index)`. -* Expected beacon chain head: - * Any [validator](#dfn-validator) `index` in `previous_epoch_head_attester_indices` gains `base_reward(state, index) * previous_epoch_head_attesting_balance // previous_total_balance)`. - * Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_head_attester_indices` loses `base_reward(state, index)`. -* Inclusion distance: - * Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` gains `base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index)` - -Case 2: `epochs_since_finality > 4`: - -* Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. -* Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_boundary_attester_indices`, loses `inactivity_penalty(state, index, epochs_since_finality)`. -* Any [active validator](#dfn-active-validator) `index` from `previous_active_validator_indices` not in `previous_epoch_head_attester_indices`, loses `base_reward(state, index)`. -* Any [active validator](#dfn-active-validator) `index` with `validator.slashed is True`, loses `2 * inactivity_penalty(state, index, epochs_since_finality) + base_reward(state, index)`. -* Any [validator](#dfn-validator) `index` in `previous_epoch_attester_indices` loses `base_reward(state, index) - base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index)` - -##### Attestation inclusion - -For each `index` in `previous_epoch_attester_indices`, we determine the proposer `proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index))` and set `state.validator_balances[proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT`. +```python +def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Int -> Int]: + previous_active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) + epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch + active_validator_indices = [ + i for i in range(len(state.validator_registry)) if + is_active_validator(state.validator_registry[i], get_current_epoch(state) - 1) + ] + # Initialize deltas that we are returning to zero + deltas = {index: 0 for index in range(len(state.validator_registry))} + # Some helper variables + boundary_attestations = get_previous_epoch_boundary_attestations(state) + boundary_attesting_balance = get_attesting_balance(boundary_attestations) + total_balance = get_previous_total_balance(state) + total_attesting_balance = get_attesting_balance(get_previous_epoch_attestations(state)) + matching_head_attestations = [ + a for a in get_previous_epoch_attestations(state) if + a.data.beacon_block_root == get_block_root(state, a.data.slot) + ] + matching_head_balance = get_attesting_balance(matching_head_attestations) + # Normal case + if epochs_since_finality <= 4: + for index in active_validator_indices: + # Expected FFG source + if index in get_attesting_indices(get_previous_epoch_attestations(state)): + deltas[index] += get_base_reward(state, index) * total_attesting_balance // total_balance + # Inclusion speed bonus + deltas[index] += ( + get_base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // + inclusion_distance(state, index) + ) + else: + deltas[index] -= get_base_reward(state, index) + # Expected FFG target + if index in get_attesting_indices(boundary_attestations): + deltas[index] += get_base_reward(state, index) * boundary_attesting_balance // total_balance + else: + deltas[index] -= get_base_reward(state, index) + # Expected head + if index in matching_head_attestations: + deltas[index] += get_base_reward(state, index) * matching_head_balance // total_balance + else: + deltas[index] -= get_base_reward(state, index) + # Proposer bonus + proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) + deltas[proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT + # Quadratic leak case + else: + for index in active_validator_indices: + if index not in get_attesting_indices(get_previous_epoch_attestations(state)): + deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) + else: + # If a validator did attest, apply a small penalty for getting attestations included late + deltas[index] += ( + base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // + inclusion_distance(state, index) + ) + deltas[index] -= base_reward(state, index) + + if index not in get_attesting_indices(boundary_attestations): + deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) + if index not in matching_head_attestations: + deltas[index] -= get_base_reward(state, index) + for index in range(len(state.validator_registry)): + if index not in active_validator_indices and state.validator_registry[index].slashed: + deltas[index] -= ( + 2 * get_inactivity_penalty(state, index, epochs_since_finality) + + get_base_reward(state, index) + ) + return deltas +``` ##### Crosslinks -For every `slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(current_epoch))`: +```python +def get_crosslink_reward_deltas(state: BeaconState) -> Dict[int, int]: + deltas = {index: 0 for index in range(len(state.validator_registry))} + for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): + for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): + winning_root, participants = get_winning_root_and_participants(state, shard) + participating_balance = get_total_balance(state, participants) + total_balance = get_total_balance(state, committee) + for index in crosslink_committee: + if index in participants: + deltas[index] += get_base_reward(state, index) * participating_balance // total_balance + else: + deltas[index] -= get_base_reward(state, index) + return deltas +``` -* Let `crosslink_committees_at_slot = get_crosslink_committees_at_slot(state, slot)`. -* For every `(crosslink_committee, shard)` in `crosslink_committees_at_slot` and every `index` in `crosslink_committee`: - * If `index in attesting_validators(crosslink_committee)`, `state.validator_balances[index] += base_reward(state, index) * total_attesting_balance(crosslink_committee) // get_total_balance(state, crosslink_committee))`. - * If `index not in attesting_validators(crosslink_committee)`, `state.validator_balances[index] -= base_reward(state, index)`. +#### Apply rewards + +Run the following: + +```python +def apply_rewards(state: BeaconState): + deltas1 = get_justification_and_finalization_deltas(state) + deltas2 = get_crosslink_deltas(state) + for i in range(len(state.validator_registry)): + state.validator_balances[i] += deltas1[i] + deltas2[i] +``` #### Ejections From 0c24ca9bb0ee023214a99d8f4f62a861f5680635 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 18:59:55 -0600 Subject: [PATCH 049/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index aea187730..f146fff5b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1267,7 +1267,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: branch=deposit.branch, depth=DEPOSIT_CONTRACT_TREE_DEPTH, index=deposit.index, - root=state.latest_eth1_data.deposit_root + root=state.latest_eth1_data.deposit_root, ) assert merkle_branch_is_valid From 26179ede5f873c0bde8df390a12246b33d0a4610 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 19:01:40 -0600 Subject: [PATCH 050/161] branch -> proof --- specs/core/0_beacon-chain.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index f146fff5b..8d0f55505 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -387,7 +387,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git ```python { # Branch in the deposit tree - 'branch': ['bytes32'], + 'proof': ['bytes32'], # Index in the deposit tree 'index': 'uint64', # Data @@ -994,16 +994,17 @@ def merkle_root(values: List[Bytes32]) -> Bytes32: ### `verify_merkle_branch` ```python -def verify_merkle_branch(leaf: Bytes32, branch: List[Bytes32], depth: int, index: int, root: Bytes32) -> bool: +def verify_merkle_branch(leaf: Bytes32, proof: List[Bytes32], depth: int, index: int, root: Bytes32) -> bool: """ - Verify that the given ``leaf`` is on the merkle branch ``branch``. + Verify that the given ``leaf`` is on the merkle branch ``proof`` + starting with the given ``root``. """ value = leaf for i in range(depth): if index // (2**i) % 2: - value = hash(branch[i] + value) + value = hash(proof[i] + value) else: - value = hash(value + branch[i]) + value = hash(value + proof[i]) return value == root ``` @@ -1264,7 +1265,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: # Verify the Merkle branch merkle_branch_is_valid = verify_merkle_branch( leaf=hash(serialized_deposit_data), - branch=deposit.branch, + proof=deposit.proof, depth=DEPOSIT_CONTRACT_TREE_DEPTH, index=deposit.index, root=state.latest_eth1_data.deposit_root, From ff88d22f9aa210bbf58c4c264b1bd8201c1861d0 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 19:04:49 -0600 Subject: [PATCH 051/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6c8acb7fb..b066bdcce 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1689,7 +1689,7 @@ Verify that `len(block.body.attestations) <= MAX_ATTESTATIONS`. For each `attestation` in `block.body.attestations`, run the following function: ```python -def process_attestation(attestation: Attestation, state: BeaconState): +def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Can't submit attestations that are too far in history (or in prehistory) assert attestation.data.slot >= GENESIS_SLOT assert state.slot < attestation.data.slot + SLOTS_PER_EPOCH From 3d8e0f19207463daeb4bf7b67704be43cea07f96 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 19:04:55 -0600 Subject: [PATCH 052/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b066bdcce..9c2c84c8c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1798,7 +1798,7 @@ Verify that `len(block.body.voluntary_exits) <= MAX_VOLUNTARY_EXITS`. For each `exit` in `block.body.voluntary_exits`, run the following function: ```python -def process_exit(exit: VoluntaryExit, state: BeaconState): +def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: validator = state.validator_registry[exit.validator_index] # Verify the validator has not yet exited assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) From ec20edb80efbee9102de5de22d7530b3e0b546f0 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 19:05:02 -0600 Subject: [PATCH 053/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 9c2c84c8c..0435e9ee1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1824,7 +1824,7 @@ Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers For each `transfer` in `block.body.transfers`, run the following function: ```python -def process_exit(transfer: Transfer, state: BeaconState): +def process_exit(state: BeaconState,transfer: Transfer) -> None: # Verify that we have enough ETH to send, and that after the transfer the balance will be either # exactly zero or at least MIN_DEPOSIT_AMOUNT assert ( From b8d1003c2cb9be86744f21776bc59ef0cd183536 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 19:11:13 -0600 Subject: [PATCH 054/161] Updates to make HW and Danny happy --- specs/core/0_beacon-chain.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0435e9ee1..250cad6c7 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1632,8 +1632,8 @@ Verify that `len(block.body.proposer_slashings) <= MAX_PROPOSER_SLASHINGS`. For each `proposer_slashing` in `block.body.proposer_slashings`, run the following function: ```python -def process_proposer_slashing(proposer_slashing: ProposerSlashing, - state: BeaconState): +def process_proposer_slashing(state: BeaconState, + proposer_slashing: ProposerSlashing): proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same assert proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot @@ -1661,8 +1661,8 @@ Verify that `len(block.body.attester_slashings) <= MAX_ATTESTER_SLASHINGS`. For each `attester_slashing` in `block.body.attester_slashings`, run the following function: ```python -def process_attester_slashing(attester_slashing: ProposerSlashing, - state: BeaconState): +def process_attester_slashing(state: BeaconState, + attester_slashing: AttesterSlashing): attestation1 = attester_slashing.slashable_attestation_1 attestation2 = attester_slashing.slashable_attestation_2 # Check that the attestations are conflicting @@ -1674,8 +1674,11 @@ def process_attester_slashing(attester_slashing: ProposerSlashing, assert verify_slashable_attestation(state, attestation1) assert verify_slashable_attestation(state, attestation2) slashable_indices = [ - index for index in attestation1.validator_indices if - index in attestation2.validator_indices and state.validator_registry[index].slashed is False + index for index in attestation1.validator_indices + if ( + index in attestation2.validator_indices and + state.validator_registry[index].slashed is False + ) ] assert len(slashable_indices) >= 1 for index in slashable_indices: @@ -1706,16 +1709,17 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: state, get_epoch_start_slot(attestation.data.justified_epoch) ) # Check that the crosslink data is valid - assert ( + acceptable_crosslink_data = { # Case 1: Latest crosslink matches the one in the state - state.latest_crosslinks[attestation.data.shard] == attestation.data.latest_crosslink or + attestation.data.latest_crosslink, # Case 2: State has already been updated, state's latest crosslink matches the crosslink # the attestation is trying to create - state.latest_crosslinks[attestation.data.shard] == Crosslink( + Crosslink( crosslink_data_root=attestation.data.crosslink_data_root, epoch=slot_to_epoch(attestation.data.slot) ) - ) + } + assert state.latest_crosslinks[attestation.data.shard] in acceptable_crosslink_data # Attestation must be nonempty! assert attestation.aggregation_bitfield != b'\x00' * len(attestation.aggregation_bitfield) # Custody must be empty (to be removed in phase 1) @@ -1825,6 +1829,8 @@ For each `transfer` in `block.body.transfers`, run the following function: ```python def process_exit(state: BeaconState,transfer: Transfer) -> None: + # Verify the amount and fee aren't individually too big (for anti-overflow purposes) + assert state.validator_balances[transfer.from] >= max(transfer.amount, transfer.fee) # Verify that we have enough ETH to send, and that after the transfer the balance will be either # exactly zero or at least MIN_DEPOSIT_AMOUNT assert ( From 6c1abbc2fc8cfa3a74f6b69ff5dbaa65cdbc878e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 19:12:11 -0600 Subject: [PATCH 055/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2b6c3bd07..bf65b66c1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1782,7 +1782,7 @@ def get_current_attestations(state: BeaconState) -> List[PendingAttestation]: ``` ```python -def get_current_total_balance(state: BeaconState): +def get_current_total_balance(state: BeaconState) -> Gwei: return get_total_balance(state, get_active_validator_indices(state.validator_registry, current_epoch)) ``` From 46269310ee228591708fa27fdbd14c3896d38756 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 19:12:21 -0600 Subject: [PATCH 056/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index bf65b66c1..09f8b7280 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1787,7 +1787,7 @@ def get_current_total_balance(state: BeaconState) -> Gwei: ``` ```python -def get_previous_total_balance(state: BeaconState): +def get_previous_total_balance(state: BeaconState) -> Gwei: return get_total_balance(state, get_active_validator_indices(state.validator_registry, previous_epoch)) ``` From e0b8eea7d6c27512dea0e67120f2badc8eec5257 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 1 Mar 2019 19:12:31 -0600 Subject: [PATCH 057/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 09f8b7280..ab4866bba 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1793,7 +1793,7 @@ def get_previous_total_balance(state: BeaconState) -> Gwei: ```python def get_previous_attestations(state: BeaconState) -> List[PendingAttestation]: - return [a for a in state.latest_attestations if previous_epoch == slot_to_epoch(a.data.slot)] + return [a for a in state.latest_attestations if previous_epoch == slot_to_epoch(a.data.slot)] ``` ```python From 4cc738dc539aa2f9498923562b9089cf029abf4b Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 19:17:07 -0600 Subject: [PATCH 058/161] Apply suggestions from code review Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 44 ++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ab4866bba..7a66ade84 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1797,7 +1797,7 @@ def get_previous_attestations(state: BeaconState) -> List[PendingAttestation]: ``` ```python -def get_attesting_indices(attestations: List[PendingAttestation], state: BeaconState) -> List[int]: +def get_attesting_indices(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: output = set({}) for a in attestations: output = output.union([get_attestation_participants(state, a.data, a.aggregation_bitfield)]) @@ -1805,12 +1805,12 @@ def get_attesting_indices(attestations: List[PendingAttestation], state: BeaconS ``` ```python -def get_attesting_balance(attestations: List[PendingAttestation], state: BeaconState) -> List[int]: +def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: return get_total_balance(state, get_attesting_indices(attestations, state)) ``` ```python -def get_current_epoch_boundary_attestations(state: BeaconState): +def get_current_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ a for a in get_current_attestations(state) if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) @@ -1818,7 +1818,7 @@ def get_current_epoch_boundary_attestations(state: BeaconState): ``` ```python -def get_previous_epoch_boundary_attestations(state: BeaconState): +def get_previous_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ a for a in get_previous_attestations(state) if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) @@ -1828,14 +1828,14 @@ def get_previous_epoch_boundary_attestations(state: BeaconState): **Note**: Total balances computed for the previous epoch might be marginally different than the actual total balances during the previous epoch transition. Due to the tight bound on validator churn each epoch and small per-epoch rewards/penalties, the potential balance difference is very low and only marginally affects consensus safety. ```python -def get_winning_root_and_participants(state: BeaconState, shard: Shard): +def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple[Bytes32, List[ValidatorIndex]]: all_attestations = get_current_attestations(state) + get_previous_attestations(state) valid_attestations = [ a for a in all_attestations if a.data.latest_crosslink == state.latest_crosslinks[shard] ] all_roots = [a.data.crosslink_data_root for a in valid_attestations] - def get_attestations_for(root): + def get_attestations_for(root) -> List[PendingAttestation]: return [a for a in valid_attestations if a.data.crosslink_data_root == root] winning_root = max(all_roots, key=lambda r: get_attesting_balance(get_attestations_for(r))) @@ -1844,22 +1844,22 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard): ``` ```python -def earliest_attestation(state: BeaconState, validator_index: ValidatorIndex): +def earliest_attestation(state: BeaconState, validator_index: ValidatorIndex) -> PendingAttestation: return min([ a for a in state.previous_epoch_attestations if validator_index in get_attestation_participants(state, a.data, a.aggregation_bitfield) - ], key = lambda a: a.inclusion_slot) + ], key=lambda a: a.inclusion_slot) ``` ```python -def inclusion_slot(state: BeaconState, validator_index: ValidatorIndex): +def inclusion_slot(state: BeaconState, validator_index: ValidatorIndex) -> Slot: return earliest_attestation(state, validator_index).inclusion_slot ``` ```python -def inclusion_distance(state: BeaconState, validator_index: ValidatorIndex): +def inclusion_distance(state: BeaconState, validator_index: ValidatorIndex) -> int: attestation = earliest_attestation(state, validator_index) - return attestation.inclusion_slot - attestation.data_slot + return attestation.inclusion_slot - attestation.data.slot ``` #### Justification @@ -1867,7 +1867,7 @@ def inclusion_distance(state: BeaconState, validator_index: ValidatorIndex): Run the following function: ```python -def update_justification_and_finalization(state): +def update_justification_and_finalization(state: BeaconState) -> None: new_justified_epoch = state.justified_epoch # Rotate the justification bitfield up one epoch to make room for the current epoch @@ -1924,7 +1924,7 @@ for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(nex Run the following function: ```python -def maybe_reset_eth1_period(state): +def maybe_reset_eth1_period(state: BeaconState) -> None: if (get_current_epoch(state) + 1) % EPOCHS_PER_ETH1_VOTING_PERIOD == 0: for eth1_data_vote in state.eth1_data_votes: # If a majority of all votes were for a particular eth1_data value, @@ -1939,15 +1939,15 @@ def maybe_reset_eth1_period(state): First, we define some additional helpers: ```python -def get_base_reward(state: BeaconState, index: ValidatorIndex) -> int: +def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: adjusted_quotient = integer_squareroot(get_previous_total_balance(state)) // BASE_REWARD_QUOTIENT return get_effective_balance(state, index) // adjusted_quotient // 5 ``` ```python -def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex) -> int: +def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex) -> Gwei: return ( - base_reward(state, index) + + get_base_reward(state, index) + get_effective_balance(state, index) * epochs_since_finality // INACTIVITY_PENALTY_QUOTIENT // 2 ) ``` @@ -1957,7 +1957,7 @@ Note: When applying penalties in the following balance recalculations implemente ##### Justification and finalization ```python -def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Int -> Int]: +def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[ValidatorIndex, Gwei]: previous_active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch active_validator_indices = [ @@ -1995,7 +1995,7 @@ def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Int -> else: deltas[index] -= get_base_reward(state, index) # Expected head - if index in matching_head_attestations: + if index in get_attesting_indices(matching_head_attestations): deltas[index] += get_base_reward(state, index) * matching_head_balance // total_balance else: deltas[index] -= get_base_reward(state, index) @@ -2017,7 +2017,7 @@ def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Int -> if index not in get_attesting_indices(boundary_attestations): deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) - if index not in matching_head_attestations: + if index not in get_attesting_indices(matching_head_attestations): deltas[index] -= get_base_reward(state, index) for index in range(len(state.validator_registry)): if index not in active_validator_indices and state.validator_registry[index].slashed: @@ -2031,9 +2031,9 @@ def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Int -> ##### Crosslinks ```python -def get_crosslink_reward_deltas(state: BeaconState) -> Dict[int, int]: +def get_crosslink_deltas(state: BeaconState) -> Dict[ValidatorIndex, Gwei]: deltas = {index: 0 for index in range(len(state.validator_registry))} - for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): + for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(current_epoch)): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) participating_balance = get_total_balance(state, participants) @@ -2051,7 +2051,7 @@ def get_crosslink_reward_deltas(state: BeaconState) -> Dict[int, int]: Run the following: ```python -def apply_rewards(state: BeaconState): +def apply_rewards(state: BeaconState) -> None: deltas1 = get_justification_and_finalization_deltas(state) deltas2 = get_crosslink_deltas(state) for i in range(len(state.validator_registry)): From 7c6232a602c8cea2eabcd359f46026b18d157962 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 19:22:24 -0600 Subject: [PATCH 059/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 7a66ade84..aa57d2d8e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1958,12 +1958,8 @@ Note: When applying penalties in the following balance recalculations implemente ```python def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[ValidatorIndex, Gwei]: - previous_active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) + active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch - active_validator_indices = [ - i for i in range(len(state.validator_registry)) if - is_active_validator(state.validator_registry[i], get_current_epoch(state) - 1) - ] # Initialize deltas that we are returning to zero deltas = {index: 0 for index in range(len(state.validator_registry))} # Some helper variables @@ -1980,7 +1976,7 @@ def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Valida if epochs_since_finality <= 4: for index in active_validator_indices: # Expected FFG source - if index in get_attesting_indices(get_previous_epoch_attestations(state)): + if index in get_attesting_indices(get_previous_attestations(state)): deltas[index] += get_base_reward(state, index) * total_attesting_balance // total_balance # Inclusion speed bonus deltas[index] += ( @@ -2019,6 +2015,7 @@ def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[Valida deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) if index not in get_attesting_indices(matching_head_attestations): deltas[index] -= get_base_reward(state, index) + # Penalize slashed-but-inactive validators as though they were active but offline for index in range(len(state.validator_registry)): if index not in active_validator_indices and state.validator_registry[index].slashed: deltas[index] -= ( @@ -2055,7 +2052,7 @@ def apply_rewards(state: BeaconState) -> None: deltas1 = get_justification_and_finalization_deltas(state) deltas2 = get_crosslink_deltas(state) for i in range(len(state.validator_registry)): - state.validator_balances[i] += deltas1[i] + deltas2[i] + state.validator_balances[i] = max(0, state.validator_balances[i] + deltas1[i] + deltas2[i]) ``` #### Ejections From 2d4c8af4893ce362ebff5d95c789016cf070a877 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 19:34:26 -0600 Subject: [PATCH 060/161] Split up big function into two smaller functions --- specs/core/0_beacon-chain.md | 139 ++++++++++++++++++++--------------- 1 file changed, 80 insertions(+), 59 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index aa57d2d8e..0e0662f39 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1825,6 +1825,14 @@ def get_previous_epoch_boundary_attestations(state: BeaconState) -> List[Pending ] ``` +```python +def get_previous_epoch_matching_head_attestations(state: BeaconState) -> List[PendingAttestation]: + return [ + a for a in get_previous_epoch_attestations(state) if + a.data.beacon_block_root == get_block_root(state, a.data.slot) + ] +``` + **Note**: Total balances computed for the previous epoch might be marginally different than the actual total balances during the previous epoch transition. Due to the tight bound on validator churn each epoch and small per-epoch rewards/penalties, the potential balance difference is very low and only marginally affects consensus safety. ```python @@ -1957,79 +1965,92 @@ Note: When applying penalties in the following balance recalculations implemente ##### Justification and finalization ```python -def get_justification_and_finalization_deltas(state: BeaconState) -> Dict[ValidatorIndex, Gwei]: - active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) +def get_justification_and_finalization_deltas(state: BeaconState) -> List[Gwei]: epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch - # Initialize deltas that we are returning to zero - deltas = {index: 0 for index in range(len(state.validator_registry))} + if epochs_since_finality <= 4: + return compute_normal_justification_and_finalization_deltas(state) + else: + return compute_inactivity_leak_deltas(state) +``` + +When blocks are finalizing normally... + +```python +def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> List[Gwei]: + deltas = [0 for index in range(len(state.validator_registry))] # Some helper variables boundary_attestations = get_previous_epoch_boundary_attestations(state) boundary_attesting_balance = get_attesting_balance(boundary_attestations) total_balance = get_previous_total_balance(state) total_attesting_balance = get_attesting_balance(get_previous_epoch_attestations(state)) - matching_head_attestations = [ - a for a in get_previous_epoch_attestations(state) if - a.data.beacon_block_root == get_block_root(state, a.data.slot) - ] + matching_head_attestations = get_previous_epoch_matching_head_attestations(state) matching_head_balance = get_attesting_balance(matching_head_attestations) - # Normal case - if epochs_since_finality <= 4: - for index in active_validator_indices: - # Expected FFG source - if index in get_attesting_indices(get_previous_attestations(state)): - deltas[index] += get_base_reward(state, index) * total_attesting_balance // total_balance - # Inclusion speed bonus - deltas[index] += ( - get_base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // - inclusion_distance(state, index) - ) - else: - deltas[index] -= get_base_reward(state, index) - # Expected FFG target - if index in get_attesting_indices(boundary_attestations): - deltas[index] += get_base_reward(state, index) * boundary_attesting_balance // total_balance - else: - deltas[index] -= get_base_reward(state, index) - # Expected head - if index in get_attesting_indices(matching_head_attestations): - deltas[index] += get_base_reward(state, index) * matching_head_balance // total_balance - else: - deltas[index] -= get_base_reward(state, index) - # Proposer bonus - proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) - deltas[proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT - # Quadratic leak case - else: - for index in active_validator_indices: - if index not in get_attesting_indices(get_previous_epoch_attestations(state)): - deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) - else: - # If a validator did attest, apply a small penalty for getting attestations included late - deltas[index] += ( - base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // - inclusion_distance(state, index) - ) - deltas[index] -= base_reward(state, index) + # Process rewards or penalties for all validators + for index in get_active_validator_indices(state.validator_registry, previous_epoch): + # Expected FFG source + if index in get_attesting_indices(get_previous_attestations(state)): + deltas[index] += get_base_reward(state, index) * total_attesting_balance // total_balance + # Inclusion speed bonus + deltas[index] += ( + get_base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // + inclusion_distance(state, index) + ) + else: + deltas[index] -= get_base_reward(state, index) + # Expected FFG target + if index in get_attesting_indices(boundary_attestations): + deltas[index] += get_base_reward(state, index) * boundary_attesting_balance // total_balance + else: + deltas[index] -= get_base_reward(state, index) + # Expected head + if index in get_attesting_indices(matching_head_attestations): + deltas[index] += get_base_reward(state, index) * matching_head_balance // total_balance + else: + deltas[index] -= get_base_reward(state, index) + # Proposer bonus + proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) + deltas[proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT +``` + +When blocks are not finalizing normally... + +```python +def compute_inactivity_leak_deltas(state: BeaconState) -> List[Gwei]: + deltas = [0 for index in range(len(state.validator_registry))] + boundary_attestations = get_previous_epoch_boundary_attestations(state) + matching_head_attestations = get_previous_epoch_matching_head_attestations(state) + active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) + epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch + for index in active_validator_indices: + if index not in get_attesting_indices(get_previous_epoch_attestations(state)): + deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) + else: + # If a validator did attest, apply a small penalty for getting attestations included late + deltas[index] += ( + base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // + inclusion_distance(state, index) + ) + deltas[index] -= base_reward(state, index) - if index not in get_attesting_indices(boundary_attestations): - deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) - if index not in get_attesting_indices(matching_head_attestations): - deltas[index] -= get_base_reward(state, index) - # Penalize slashed-but-inactive validators as though they were active but offline - for index in range(len(state.validator_registry)): - if index not in active_validator_indices and state.validator_registry[index].slashed: - deltas[index] -= ( - 2 * get_inactivity_penalty(state, index, epochs_since_finality) + - get_base_reward(state, index) - ) + if index not in get_attesting_indices(boundary_attestations): + deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) + if index not in get_attesting_indices(matching_head_attestations): + deltas[index] -= get_base_reward(state, index) + # Penalize slashed-but-inactive validators as though they were active but offline + for index in range(len(state.validator_registry)): + if index not in active_validator_indices and state.validator_registry[index].slashed: + deltas[index] -= ( + 2 * get_inactivity_penalty(state, index, epochs_since_finality) + + get_base_reward(state, index) + ) return deltas ``` ##### Crosslinks ```python -def get_crosslink_deltas(state: BeaconState) -> Dict[ValidatorIndex, Gwei]: - deltas = {index: 0 for index in range(len(state.validator_registry))} +def get_crosslink_deltas(state: BeaconState) -> List[Gwei]: + deltas = [0 for index in range(len(state.validator_registry))] for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(current_epoch)): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) From 4ad3529e4eb5ef100845b255aa42171900e17e3d Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 1 Mar 2019 19:49:28 -0600 Subject: [PATCH 061/161] Finished epoch transition --- specs/core/0_beacon-chain.md | 96 +++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 29 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0e0662f39..a75218329 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -137,6 +137,7 @@ - [Crosslinks](#crosslinks-1) - [Ejections](#ejections) - [Validator registry and shuffling seed data](#validator-registry-and-shuffling-seed-data) + - [Slashings and exit queue](#slashings-and-exit-queue) - [Final updates](#final-updates) - [State root verification](#state-root-verification) - [References](#references) @@ -2093,18 +2094,21 @@ def process_ejections(state: BeaconState) -> None: #### Validator registry and shuffling seed data -First, update the following: - -* Set `state.previous_shuffling_epoch = state.current_shuffling_epoch`. -* Set `state.previous_shuffling_start_shard = state.current_shuffling_start_shard`. -* Set `state.previous_shuffling_seed = state.current_shuffling_seed`. - -If the following are satisfied: - -* `state.finalized_epoch > state.validator_registry_update_epoch` -* `state.latest_crosslinks[shard].epoch > state.validator_registry_update_epoch` for every shard number `shard` in `[(state.current_shuffling_start_shard + i) % SHARD_COUNT for i in range(get_current_epoch_committee_count(state))]` (that is, for every shard in the current committees) - -update the validator registry and associated fields by running +```python +def should_update_validator_registry(state: BeaconState) -> bool: + # Must have finalized a new block + if state.finalized_epoch <= state.validator_registry_update_epoch: + return False + # Must have processed new crosslinks on all shards of the current epoch + shards_to_check [ + (state.current_shuffling_start_shard + i) % SHARD_COUNT + for i in range(get_current_epoch_committee_count(state)) + ] + for shard in shards_to_check: + if state.latest_crosslinks[shard].epoch <= state.validator_registry_update_epoch: + return False + return True +``` ```python def update_validator_registry(state: BeaconState) -> None: @@ -2151,23 +2155,38 @@ def update_validator_registry(state: BeaconState) -> None: state.validator_registry_update_epoch = current_epoch ``` -and perform the following updates: +Run the following function: -* Set `state.current_shuffling_epoch = next_epoch` -* Set `state.current_shuffling_start_shard = (state.current_shuffling_start_shard + get_current_epoch_committee_count(state)) % SHARD_COUNT` -* Set `state.current_shuffling_seed = generate_seed(state, state.current_shuffling_epoch)` - -If a validator registry update does _not_ happen do the following: - -* Let `epochs_since_last_registry_update = current_epoch - state.validator_registry_update_epoch`. -* If `epochs_since_last_registry_update > 1` and `is_power_of_two(epochs_since_last_registry_update)`: - * Set `state.current_shuffling_epoch = next_epoch`. - * Set `state.current_shuffling_seed = generate_seed(state, state.current_shuffling_epoch)` - * _Note_ that `state.current_shuffling_start_shard` is left unchanged. +```python +def update_registry_and_shuffling_data(state: BeaconState) -> None: + # First set previous shuffling data to current shuffling data + state.previous_shuffling_epoch = state.current_shuffling_epoch + state.previous_shuffling_start_shard = state.current_shuffling_start_shard + state.previous_shuffling_seed = state.current_shuffling_seed + # Check if we should update, and if so, update + if should_update_validator_registry(state): + update_validator_registry(state) + # If we update the registry, update the shuffling data and shards as well + state.current_shuffling_epoch = next_epoch + state.current_shuffling_start_shard = ( + state.current_shuffling_start_shard + + get_current_epoch_committee_count(state)) % SHARD_COUNT + ) + state.current_shuffling_seed = generate_seed(state, state.current_shuffling_epoch) + else: + # If processing at least one crosslink keeps failing, then reshuffle every power of two, + # but don't update the current_shuffling_start_shard + epochs_since_last_registry_update = current_epoch - state.validator_registry_update_epoch + if epochs_since_last_registry_update > 1 and is_power_of_two(epochs_since_last_registry_update): + state.current_shuffling_epoch = next_epoch + state.current_shuffling_seed = generate_seed(state, state.current_shuffling_epoch) +``` **Invariant**: the active index root that is hashed into the shuffling seed actually is the `hash_tree_root` of the validator set that is used for that epoch. -Regardless of whether or not a validator set change happens run `process_slashings(state)` and `process_exit_queue(state)`: +#### Slashings and exit queue + +Run `process_slashings(state)` and `process_exit_queue(state)`: ```python def process_slashings(state: BeaconState) -> None: @@ -2218,10 +2237,29 @@ def process_exit_queue(state: BeaconState) -> None: #### Final updates -* Set `state.latest_active_index_roots[(next_epoch + ACTIVATION_EXIT_DELAY) % LATEST_ACTIVE_INDEX_ROOTS_LENGTH] = hash_tree_root(get_active_validator_indices(state.validator_registry, next_epoch + ACTIVATION_EXIT_DELAY))`. -* Set `state.latest_slashed_balances[next_epoch % LATEST_SLASHED_EXIT_LENGTH] = state.latest_slashed_balances[current_epoch % LATEST_SLASHED_EXIT_LENGTH]`. -* Set `state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch)`. -* Remove any `attestation` in `state.latest_attestations` such that `slot_to_epoch(attestation.data.slot) < current_epoch`. +Run the following function: + +```python +def finish_epoch_update(state: BeaconState) -> None: + current_epoch = get_current_epoch(state) + next_epoch = current_epoch + 1 + # Set active index root + index_root_position = (next_epoch + ACTIVATION_EXIT_DELAY) % LATEST_ACTIVE_INDEX_ROOTS_LENGTH + state.latest_active_index_roots[index_root_position] = hash_tree_root( + get_active_validator_indices(state.validator_registry, next_epoch + ACTIVATION_EXIT_DELAY) + ) + # Set total slashed balances + state.latest_slashed_balances[next_epoch % LATEST_SLASHED_EXIT_LENGTH] = ( + state.latest_slashed_balances[current_epoch % LATEST_SLASHED_EXIT_LENGTH] + ) + # Set randao mix + state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch) + # Remove previous epoch attestations + state.latest_attestations = [ + a for a in state.latest_attestations if + slot_to_epoch(attestation.data.slot) == current_epoch + ] +``` ### State root verification From 00804da13e7473f2ad1ea9fcebb621e15f78d041 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 Mar 2019 13:48:46 +0800 Subject: [PATCH 062/161] Fix type hinting --- specs/core/0_beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 250cad6c7..adc7c6bcb 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1633,7 +1633,7 @@ For each `proposer_slashing` in `block.body.proposer_slashings`, run the followi ```python def process_proposer_slashing(state: BeaconState, - proposer_slashing: ProposerSlashing): + proposer_slashing: ProposerSlashing) -> None: proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same assert proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot @@ -1662,7 +1662,7 @@ For each `attester_slashing` in `block.body.attester_slashings`, run the followi ```python def process_attester_slashing(state: BeaconState, - attester_slashing: AttesterSlashing): + attester_slashing: AttesterSlashing) -> None: attestation1 = attester_slashing.slashable_attestation_1 attestation2 = attester_slashing.slashable_attestation_2 # Check that the attestations are conflicting @@ -1828,7 +1828,7 @@ Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers For each `transfer` in `block.body.transfers`, run the following function: ```python -def process_exit(state: BeaconState,transfer: Transfer) -> None: +def process_exit(state: BeaconState, transfer: Transfer) -> None: # Verify the amount and fee aren't individually too big (for anti-overflow purposes) assert state.validator_balances[transfer.from] >= max(transfer.amount, transfer.fee) # Verify that we have enough ETH to send, and that after the transfer the balance will be either From fa327a18ea7e2d1551e0df5b982a1e2b66ee8462 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 2 Mar 2019 13:51:48 +0800 Subject: [PATCH 063/161] Add docstring --- specs/core/0_beacon-chain.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index adc7c6bcb..79d3577ed 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1634,6 +1634,10 @@ For each `proposer_slashing` in `block.body.proposer_slashings`, run the followi ```python def process_proposer_slashing(state: BeaconState, proposer_slashing: ProposerSlashing) -> None: + """ + Process ``ProposerSlashing`` transaction. + Note that this function mutates ``state``. + """ proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same assert proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot @@ -1663,6 +1667,10 @@ For each `attester_slashing` in `block.body.attester_slashings`, run the followi ```python def process_attester_slashing(state: BeaconState, attester_slashing: AttesterSlashing) -> None: + """ + Process ``AttesterSlashing`` transaction. + Note that this function mutates ``state``. + """ attestation1 = attester_slashing.slashable_attestation_1 attestation2 = attester_slashing.slashable_attestation_2 # Check that the attestations are conflicting @@ -1693,6 +1701,10 @@ For each `attestation` in `block.body.attestations`, run the following function: ```python def process_attestation(state: BeaconState, attestation: Attestation) -> None: + """ + Process ``Attestation`` transaction. + Note that this function mutates ``state``. + """ # Can't submit attestations that are too far in history (or in prehistory) assert attestation.data.slot >= GENESIS_SLOT assert state.slot < attestation.data.slot + SLOTS_PER_EPOCH @@ -1803,6 +1815,10 @@ For each `exit` in `block.body.voluntary_exits`, run the following function: ```python def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: + """ + Process ``VoluntaryExit`` transaction. + Note that this function mutates ``state``. + """ validator = state.validator_registry[exit.validator_index] # Verify the validator has not yet exited assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) @@ -1828,7 +1844,11 @@ Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers For each `transfer` in `block.body.transfers`, run the following function: ```python -def process_exit(state: BeaconState, transfer: Transfer) -> None: +def process_transfer(state: BeaconState, transfer: Transfer) -> None: + """ + Process ``Transfer`` transaction. + Note that this function mutates ``state``. + """ # Verify the amount and fee aren't individually too big (for anti-overflow purposes) assert state.validator_balances[transfer.from] >= max(transfer.amount, transfer.fee) # Verify that we have enough ETH to send, and that after the transfer the balance will be either From 0134e560101d15284dd3403f242b20a9be1e58ee Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 2 Mar 2019 00:40:43 -0600 Subject: [PATCH 064/161] Fixed table of contents --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a75218329..fac0cb640 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -127,10 +127,10 @@ - [Voluntary exits](#voluntary-exits-1) - [Transfers](#transfers-1) - [Per-epoch processing](#per-epoch-processing) - - [Helper variables](#helper-variables) - - [Eth1 data](#eth1-data-1) + - [Helper functions](#helper-functions-1) - [Justification](#justification) - [Crosslinks](#crosslinks) + - [Eth1 data](#eth1-data-1) - [Rewards and penalties](#rewards-and-penalties) - [Justification and finalization](#justification-and-finalization) - [Attestation inclusion](#attestation-inclusion) From ce5599db2fb4a8f96325af9900c379efb7c0e60f Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 2 Mar 2019 00:42:20 -0600 Subject: [PATCH 065/161] Made crosslink processing a function --- specs/core/0_beacon-chain.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index fac0cb640..70ffcf858 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1918,14 +1918,15 @@ def update_justification_and_finalization(state: BeaconState) -> None: Run the following function: ```python -for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): - for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): - winning_root, participants = get_winning_root_and_participants(state, shard) - if 3 * total_balance(participants) >= 2 * total_balance(crosslink_committee): - state.latest_crosslinks[shard] = Crosslink( - epoch=slot_to_epoch(slot), - crosslink_data_root=winning_root - ) +def process_crosslinks(state: BeaconState) -> None: + for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): + for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): + winning_root, participants = get_winning_root_and_participants(state, shard) + if 3 * total_balance(participants) >= 2 * total_balance(crosslink_committee): + state.latest_crosslinks[shard] = Crosslink( + epoch=slot_to_epoch(slot), + crosslink_data_root=winning_root + ) ``` #### Eth1 data From f137f1ce907623b96da0304c4fcfc815f7679906 Mon Sep 17 00:00:00 2001 From: protolambda Date: Sat, 2 Mar 2019 18:01:51 +0100 Subject: [PATCH 066/161] fix randao mixes type, bytes32 like in function and state spec --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 79d3577ed..d18987141 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1451,7 +1451,7 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], validator_registry_update_epoch=GENESIS_EPOCH, # Randomness and committees - latest_randao_mixes=[EMPTY_SIGNATURE for _ in range(LATEST_RANDAO_MIXES_LENGTH)], + latest_randao_mixes=[ZERO_HASH for _ in range(LATEST_RANDAO_MIXES_LENGTH)], previous_shuffling_start_shard=GENESIS_START_SHARD, current_shuffling_start_shard=GENESIS_START_SHARD, previous_shuffling_epoch=GENESIS_EPOCH, From fde66a7105ab2b38c755cc564aac3d66c33069be Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 2 Mar 2019 20:11:33 -0600 Subject: [PATCH 067/161] Update to make CC happy --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 70ffcf858..023d64339 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1847,7 +1847,7 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple def get_attestations_for(root) -> List[PendingAttestation]: return [a for a in valid_attestations if a.data.crosslink_data_root == root] - winning_root = max(all_roots, key=lambda r: get_attesting_balance(get_attestations_for(r))) + winning_root = max(all_roots, key=lambda r: get_attesting_balance(state, get_attestations_for(r))) return winning_root, get_attesting_indices(get_attestations_for(winning_root)) ``` From cefe9e995bbd456a44e179a10627db2edf29a6e8 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 2 Mar 2019 20:36:04 -0600 Subject: [PATCH 068/161] Proof of custody game, take 2 (#682) * Proof of custody game, take 2 Unfortunately not simpler than before, but moves challenges outside of the validator records and so keeps validator records constant-size. * Removed unneeded challenge codes * Formatting fixes --- specs/core/1_shard-data-chains.md | 383 ++++++++++++++++++++++++++---- 1 file changed, 341 insertions(+), 42 deletions(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 1cd5d89de..db68591e7 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -37,8 +37,8 @@ At the current stage, Phase 1, while fundamentally feature-complete, is still su - [`BranchResponse`](#branchresponse) - [`BranchChallengeRecord`](#branchchallengerecord) - [`SubkeyReveal`](#subkeyreveal) - - [Helpers](#helpers) - - [`get_attestation_merkle_depth`](#get_attestation_merkle_depth) + - [Helpers](#helpers) + - [`get_attestation_data_merkle_depth`](#get_attestation_data_merkle_depth) - [`epoch_to_custody_period`](#epoch_to_custody_period) - [`slot_to_custody_period`](#slot_to_custody_period) - [`get_current_custody_period`](#get_current_custody_period) @@ -74,6 +74,9 @@ Phase 1 depends upon all of the constants defined in [Phase 0](0_beacon-chain.md | `SHARD_CHUNK_SIZE` | 2**5 (= 32) | bytes | | `SHARD_BLOCK_SIZE` | 2**14 (= 16,384) | bytes | | `MINOR_REWARD_QUOTIENT` | 2**8 (= 256) | | +| `MAX_POC_RESPONSE_DEPTH` | 5 | | +| `ZERO_PUBKEY` | int_to_bytes48(0)| | +| `VALIDATOR_NULL` | 2**64 - 1 | | #### Time parameters @@ -87,19 +90,23 @@ Phase 1 depends upon all of the constants defined in [Phase 0](0_beacon-chain.md #### Max operations per block -| Name | Value | -|-------------------------------|---------------| -| `MAX_BRANCH_CHALLENGES` | 2**2 (= 4) | -| `MAX_BRANCH_RESPONSES` | 2**4 (= 16) | -| `MAX_EARLY_SUBKEY_REVEALS` | 2**4 (= 16) | +| Name | Value | +|----------------------------------------------------|---------------| +| `MAX_BRANCH_CHALLENGES` | 2**2 (= 4) | +| `MAX_BRANCH_RESPONSES` | 2**4 (= 16) | +| `MAX_EARLY_SUBKEY_REVEALS` | 2**4 (= 16) | +| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_INITIATIONS` | 2 | +| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_RESPONSES` | 16 | +| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_CONTINUTATIONS` | 16 | #### Signature domains -| Name | Value | -|------------------------|-----------------| -| `DOMAIN_SHARD_PROPOSER`| 129 | -| `DOMAIN_SHARD_ATTESTER`| 130 | -| `DOMAIN_CUSTODY_SUBKEY`| 131 | +| Name | Value | +|------------------------------|-----------------| +| `DOMAIN_SHARD_PROPOSER` | 129 | +| `DOMAIN_SHARD_ATTESTER` | 130 | +| `DOMAIN_CUSTODY_SUBKEY` | 131 | +| `DOMAIN_CUSTODY_INTERACTIVE` | 132 | # Shard chains and crosslink data @@ -163,7 +170,6 @@ def get_persistent_committee(state: BeaconState, [i for i in later_committee if epoch % PERSISTENT_COMMITTEE_PERIOD >= get_switchover_epoch(i)] ))) ``` - #### `get_shard_proposer_index` ```python @@ -295,7 +301,7 @@ The `shard_chain_commitment` is only valid if it equals `compute_commitment(head ### Shard block fork choice rule -The fork choice rule for any shard is LMD GHOST using the shard chain attestations of the persistent committee and the beacon chain attestations of the crosslink committee currently assigned to that shard, but instead of being rooted in the genesis it is rooted in the latest block referenced in the most recent accepted crosslink (ie. `state.crosslinks[shard].crosslink_data_root`). Only blocks whose `beacon_chain_ref` is the block in the main beacon chain at the specified `slot` should be considered (if the beacon chain skips a slot, then the block at that slot is considered to be the block in the beacon chain at the highest slot lower than a slot). +The fork choice rule for any shard is LMD GHOST using the shard chain attestations of the persistent committee and the beacon chain attestations of the crosslink committee currently assigned to that shard, but instead of being rooted in the genesis it is rooted in the block referenced in the most recent accepted crosslink (ie. `state.crosslinks[shard].shard_block_root`). Only blocks whose `beacon_chain_ref` is the block in the main beacon chain at the specified `slot` should be considered (if the beacon chain skips a slot, then the block at that slot is considered to be the block in the beacon chain at the highest slot lower than a slot). # Updates to the beacon chain @@ -306,7 +312,6 @@ The fork choice rule for any shard is LMD GHOST using the shard chain attestatio Add member values to the end of the `Validator` object: ```python - 'open_branch_challenges': [BranchChallengeRecord], 'next_subkey_to_reveal': 'uint64', 'reveal_max_periods_late': 'uint64', ``` @@ -314,7 +319,6 @@ Add member values to the end of the `Validator` object: And the initializers: ```python - 'open_branch_challenges': [], 'next_subkey_to_reveal': get_current_custody_period(state), 'reveal_max_periods_late': 0, ``` @@ -327,6 +331,10 @@ Add member values to the `BeaconBlockBody` structure: 'branch_challenges': [BranchChallenge], 'branch_responses': [BranchResponse], 'subkey_reveals': [SubkeyReveal], + 'interactive_custody_challenge_initiations': [InteractiveCustodyChallengeInitiation], + 'interactive_custody_challenge_responses': [InteractiveCustodyChallengeResponse], + 'interactive_custody_challenge_continuations': [InteractiveCustodyChallengeContinuation], + ``` And initialize to the following: @@ -337,6 +345,17 @@ And initialize to the following: 'subkey_reveals': [], ``` +### `BeaconState` + +Add member values to the `BeaconState` structure: + +```python + 'branch_challenge_records': [BranchChallengeRecord], + 'next_branch_challenge_id': 'uint64', + 'custody_challenge_records': [InteractiveCustodyChallengeRecord], + 'next_custody_challenge_id': 'uint64', +``` + ### `BranchChallenge` Define a `BranchChallenge` as follows: @@ -355,11 +374,10 @@ Define a `BranchResponse` as follows: ```python { - 'responder_index': 'uint64', + 'challenge_id': 'uint64', + 'responding_to_custody_challenge': 'bool', 'data': 'bytes32', 'branch': ['bytes32'], - 'data_index': 'uint64', - 'root': 'bytes32', } ``` @@ -369,14 +387,75 @@ Define a `BranchChallengeRecord` as follows: ```python { + 'challenge_id': 'uint64', 'challenger_index': 'uint64', + 'responder_index': 'uint64', 'root': 'bytes32', 'depth': 'uint64', - 'inclusion_epoch': 'uint64', + 'deadline': 'uint64', 'data_index': 'uint64', } ``` +### `InteractiveCustodyChallengeRecord` + +```python +{ + 'challenge_id': 'uint64', + 'challenger_index': 'uint64', + 'responder_index': 'uint64', + # Initial data root + 'data_root': 'bytes32', + # Initial custody bit + 'custody_bit': 'bool', + # Responder subkey + 'responder_subkey': 'bytes96', + # The hash in the PoC tree in the position that we are currently at + 'current_custody_tree_node': 'bytes32', + # The position in the tree, in terms of depth and position offset + 'depth': 'uint64', + 'offset': 'uint64', + # Max depth of the branch + 'max_depth': 'uint64', + # Deadline to respond (as an epoch) + 'deadline': 'uint64', +} +``` + +### `InteractiveCustodyChallengeInitiation` + +```python +{ + 'attestation': SlashableAttestation, + 'responder_index': 'uint64', + 'challenger_index': 'uint64', + 'responder_subkey': 'bytes96', + 'signature': 'bytes96', +} +``` + +### `InteractiveCustodyChallengeResponse` + +```python +{ + 'challenge_id': 'uint64', + 'hashes': ['bytes32'], + 'signature': 'bytes96', +} +``` + +### `InteractiveCustodyChallengeContinuation` + +```python +{ + 'challenge_id': 'uint64', + 'sub_index': 'uint64', + 'new_custody_tree_node': 'bytes32', + 'proof': ['bytes32'], + 'signature': 'bytes96', +} +``` + ### `SubkeyReveal` Define a `SubkeyReveal` as follows: @@ -393,6 +472,20 @@ Define a `SubkeyReveal` as follows: ## Helpers +### `get_branch_challenge_record_by_id` + +```python +def get_branch_challenge_record_by_id(state: BeaconState, id: int) -> BranchChallengeRecord: + return [c for c in state.branch_challenges if c.challenge_id == id][0] +``` + +### `get_custody_challenge_record_by_id` + +```python +def get_custody_challenge_record_by_id(state: BeaconState, id: int) -> BranchChallengeRecord: + return [c for c in state.branch_challenges if c.challenge_id == id][0] +``` + ### `get_attestation_merkle_depth` ```python @@ -458,6 +551,19 @@ def verify_custody_subkey_reveal(pubkey: bytes48, ) ``` +### `verify_signed_challenge_message` + +```python +def verify_signed_challenge_message(message: Any, pubkey: bytes48) -> bool: + return bls_verify( + message_hash=signed_root(message, 'signature'), + pubkey=pubkey, + signature=message.signature, + domain=get_domain(state, get_current_epoch(state), DOMAIN_CUSTODY_INTERACTIVE) + ) + +``` + ### `penalize_validator` Change the definition of `penalize_validator` as follows: @@ -498,29 +604,88 @@ Add the following operations to the per-slot processing, in order the given belo Verify that `len(block.body.branch_challenges) <= MAX_BRANCH_CHALLENGES`. -For each `challenge` in `block.body.branch_challenges`: +For each `challenge` in `block.body.branch_challenges`, run: -* Verify that `slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY`. -* Verify that `state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY`. -* Verify that `verify_slashable_attestation(state, challenge.attestation)` returns `True`. -* Verify that `challenge.responder_index` is in `challenge.attestation.validator_indices`. -* Let `depth = get_attestation_merkle_depth(challenge.attestation)`. Verify that `challenge.data_index < 2**depth`. -* Verify that there does not exist a `BranchChallengeRecord` in `state.validator_registry[challenge.responder_index].open_branch_challenges` with `root == challenge.attestation.data.shard_chain_commitment` and `data_index == data_index`. -* Append to `state.validator_registry[challenge.responder_index].open_branch_challenges` the object `BranchChallengeRecord(challenger_index=get_beacon_proposer_index(state, state.slot), root=challenge.attestation.data.shard_chain_commitment, depth=depth, inclusion_epoch=get_current_epoch(state), data_index=data_index)`. - -**Invariant**: the `open_branch_challenges` array will always stay sorted in order of `inclusion_epoch`. +```python +def process_branch_challenge(challenge: BranchChallenge, + state: BeaconState): + # Check that it's not too late to challenge + assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY + assert state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY + # Check the attestation is valid + assert verify_slashable_attestation(state, challenge.attestation) + # Check that the responder participated + assert challenger.responder_index in challenge.attestation.validator_indices + # Check the challenge is not a duplicate + assert [ + c for c in state.branch_challenge_records if c.root == challenge.attestation.data.crosslink_data_root and + c.data_index == challenge.data_index + ] == [] + # Check validity of depth + depth = get_attestation_merkle_depth(challenge.attestation) + assert c.data_index < 2**depth + # Add new challenge + state.branch_challenge_records.append(BranchChallengeRecord( + challenge_id=state.next_branch_challenge_id, + challenger_index=get_beacon_proposer_index(state, state.slot), + root=challenge.attestation.data.shard_chain_commitment, + depth=depth, + deadline=get_current_epoch(state) + CHALLENGE_RESPONSE_DEADLINE, + data_index=challenge.data_index + )) + state.next_branch_challenge_id += 1 +``` #### Branch responses Verify that `len(block.body.branch_responses) <= MAX_BRANCH_RESPONSES`. -For each `response` in `block.body.branch_responses`: +For each `response` in `block.body.branch_responses`, if `response.responding_to_custody_challenge == False`, run: -* Find the `BranchChallengeRecord` in `state.validator_registry[response.responder_index].open_branch_challenges` whose (`root`, `data_index`) match the (`root`, `data_index`) of the `response`. Verify that one such record exists (it is not possible for there to be more than one), call it `record`. -* Verify that `verify_merkle_branch(leaf=response.data, branch=response.branch, depth=record.depth, index=record.data_index, root=record.root)` is True. -* Verify that `get_current_epoch(state) >= record.inclusion_epoch + ENTRY_EXIT_DELAY`. -* Remove the `record` from `state.validator_registry[response.responder_index].open_branch_challenges` -* Determine the proposer `proposer_index = get_beacon_proposer_index(state, state.slot)` and set `state.validator_balances[proposer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT`. +```python +def process_branch_exploration_response(response: BranchResponse, + state: BeaconState): + challenge = get_branch_challenge_record_by_id(response.challenge_id) + assert verify_merkle_branch( + leaf=response.data, + branch=response.branch, + depth=challenge.depth, + index=challenge.data_index, + root=challenge.root + ) + # Must wait at least ENTRY_EXIT_DELAY before responding to a branch challenge + assert get_current_epoch(state) >= challenge.inclusion_epoch + ENTRY_EXIT_DELAY + state.branch_challenge_records.pop(challenge) + # Reward the proposer + proposer_index = get_beacon_proposer_index(state, state.slot) + state.validator_balances[proposer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT +``` + +If `response.responding_to_custody_challenge == True`, run: + +```python +def process_branch_custody_response(response: BranchResponse, + state: BeaconState): + challenge = get_custody_challenge_record_by_id(response.challenge_id) + responder = state.validator_registry[challenge.responder_index] + # Verify we're not too late + assert get_current_epoch(state) < responder.withdrawable_epoch + # Verify the Merkle branch *of the data tree* + assert verify_merkle_branch( + leaf=response.data, + branch=response.branch, + depth=challenge.max_depth, + index=challenge.offset, + root=challenge.data_root + ) + # Responder wins + if hash(challenge.responder_subkey + response.data) == challenge.current_custody_tree_node: + penalize_validator(state, challenge.challenger_index, challenge.responder_index) + # Challenger wins + else: + penalize_validator(state, challenge.responder_index, challenge.challenger_index) + state.custody_challenge_records.pop(challenge) +``` #### Subkey reveals @@ -546,6 +711,126 @@ In case (ii): * Set `state.validator_registry[reveal.validator_index].next_subkey_to_reveal += 1` * Set `state.validator_registry[reveal.validator_index].reveal_max_periods_late = max(state.validator_registry[reveal.validator_index].reveal_max_periods_late, get_current_period(state) - reveal.period)`. +#### Interactive custody challenge initiations + +Verify that `len(block.body.interactive_custody_challenge_initiations) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_INITIATIONS`. + +For each `initiation` in `block.body.interactive_custody_challenge_initiations`, use the following function to process it: + +```python +def process_initiation(initiation: InteractiveCustodyChallengeInitiation, + state: BeaconState): + challenger = state.validator_registry[initiation.challenger_index] + responder = state.validator_registry[initiation.responder_index] + # Verify the signature + assert verify_signed_challenge_message(initiation, challenger.pubkey) + # Verify the attestation + assert verify_slashable_attestation(initiation.attestation, state) + # Check that the responder actually participated in the attestation + assert initiation.responder_index in attestation.validator_indices + # Any validator can be a challenger or responder of max 1 challenge at a time + for c in state.custody_challenge_records: + assert c.challenger_index != initiation.challenger_index + assert c.responder_index != initiation.responder_index + # Can't challenge if you've been penalized + assert challenger.penalized_epoch == FAR_FUTURE_EPOCH + # Make sure the revealed subkey is valid + assert verify_custody_subkey_reveal( + pubkey=state.validator_registry[responder_index].pubkey, + subkey=initiation.responder_subkey, + period=slot_to_custody_period(attestation.data.slot) + ) + # Verify that the attestation is still eligible for challenging + min_challengeable_epoch = responder.exit_epoch - CUSTODY_PERIOD_LENGTH * (1 + responder.reveal_max_periods_late) + assert min_challengeable_epoch <= slot_to_epoch(initiation.attestation.data.slot) + # Create a new challenge object + state.branch_challenge_records.append(InteractiveCustodyChallengeRecord( + challenge_id=state.next_branch_challenge_id, + challenger_index=initiation.challenger_index, + responder_index=initiation.responder_index, + data_root=attestation.custody_commitment, + custody_bit=get_bitfield_bit(attestation.custody_bitfield, attestation.validator_indices.index(responder_index)), + responder_subkey=responder_subkey, + current_custody_tree_node=ZERO_HASH, + depth=0, + offset=0, + max_depth=get_attestation_data_merkle_depth(initiation.attestation.data), + deadline=get_current_epoch(state) + CHALLENGE_RESPONSE_DEADLINE + )) + state.next_branch_challenge_id += 1 + # Responder can't withdraw yet! + state.validator_registry[responder_index].withdrawable_epoch = FAR_FUTURE_EPOCH +``` + +#### Interactive custody challenge responses + +A response provides 32 hashes that are under current known proof of custody tree node. Note that at the beginning the tree node is just one bit of the custody root, so we ask the responder to sign to commit to the top 5 levels of the tree and therefore the root hash; at all other stages in the game responses are self-verifying. + +Verify that `len(block.body.interactive_custody_challenge_responses) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_RESPONSES`. + +For each `response` in `block.body.interactive_custody_challenge_responses`, use the following function to process it: + +```python +def process_response(response: InteractiveCustodyChallengeResponse, + state: State): + challenge = get_custody_challenge_record_by_id(state, response.challenge_id) + responder = state.validator_registry[challenge.responder_index] + # Check that the right number of hashes was provided + expected_depth = min(challenge.max_depth - challenge.depth, MAX_POC_RESPONSE_DEPTH) + assert 2**expected_depth == len(response.hashes) + # Must make some progress! + assert expected_depth > 0 + # Check the hashes match the previously provided root + root = merkle_root(response.hashes) + # If this is the first response check the bit and the signature and set the root + if challenge.depth == 0: + assert get_bitfield_bit(root, 0) == challenge.custody_bit + assert verify_signed_challenge_message(response, responder.pubkey) + challenge.current_custody_tree_node = root + # Otherwise just check the response against the root + else: + assert root == challenge_data.current_custody_tree_node + # Update challenge data + challenge.deadline=FAR_FUTURE_EPOCH + responder.withdrawable_epoch = get_current_epoch(state) + MAX_POC_RESPONSE_DEPTH +``` + +#### Interactive custody challenge continuations + +Once a response provides 32 hashes, the challenger has the right to choose any one of them that they feel is constructed incorrectly to continue the game. Note that eventually, the game will get to the point where the `new_custody_tree_node` is a leaf node. + +Verify that `len(block.body.interactive_custody_challenge_continuations) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_CONTINUATIONS`. + +For each `continuation` in `block.body.interactive_custody_challenge_continuations`, use the following function to process it: + +```python +def process_continuation(continuation: InteractiveCustodyChallengeContinuation, + state: State): + challenge = get_custody_challenge_record_by_id(state, continuation.challenge_id) + challenger = state.validator_registry[challenge.challenger_index] + responder = state.validator_registry[challenge.responder_index] + expected_depth = min(challenge_data.max_depth - challenge_data.depth, MAX_POC_RESPONSE_DEPTH) + # Verify we're not too late + assert get_current_epoch(state) < responder.withdrawable_epoch + # Verify the Merkle branch (the previous custody response provided the next level of hashes so the + # challenger has the info to make any Merkle branch) + assert verify_merkle_branch( + leaf=new_custody_tree_node, + branch=continuation.proof, + depth=expected_depth, + index=sub_index, + root=challenge_data.current_custody_tree_node + ) + # Verify signature + assert verify_signed_challenge_message(continuation, challenger.pubkey) + # Update the challenge data + challenge.current_custody_tree_node = continuation.new_custody_tree_node + challenge.depth += expected_depth + challenge.deadline = get_current_epoch(state) + MAX_POC_RESPONSE_DEPTH + responder.withdrawable_epoch = FAR_FUTURE_EPOCH + challenge.offset = challenge_data.offset * 2**expected_depth + sub_index +``` + ## Per-epoch processing Add the following loop immediately below the `process_ejections` loop: @@ -553,12 +838,18 @@ Add the following loop immediately below the `process_ejections` loop: ```python def process_challenge_absences(state: BeaconState) -> None: """ - Iterate through the validator registry + Iterate through the challenge list and penalize validators with balance that did not answer challenges. """ - for index, validator in enumerate(state.validator_registry): - if len(validator.open_branch_challenges) > 0 and get_current_epoch(state) > validator.open_branch_challenges[0].inclusion_epoch + CHALLENGE_RESPONSE_DEADLINE: - penalize_validator(state, index, validator.open_branch_challenges[0].challenger_index) + for c in state.branch_challenge_records: + if get_current_epoch(state) > c.deadline: + penalize_validator(state, c.responder_index, c.challenger_index) + + for c in state.custody_challenge_records: + if get_current_epoch(state) > c.deadline: + penalize_validator(state, c.responder_index, c.challenger_index) + if get_current_epoch(state) > state.validator_registry[c.responder_index].withdrawable_epoch: + penalize_validator(state, c.challenger_index, c.responder_index) ``` In `process_penalties_and_exits`, change the definition of `eligible` to the following (note that it is not a pure function because `state` is declared in the surrounding scope): @@ -567,7 +858,7 @@ In `process_penalties_and_exits`, change the definition of `eligible` to the fol def eligible(index): validator = state.validator_registry[index] # Cannot exit if there are still open branch challenges - if len(validator.open_branch_challenges) > 0: + if [c for c in state.branch_challenge_records if c.responder_index == index] != []: return False # Cannot exit if you have not revealed all of your subkeys elif validator.next_subkey_to_reveal <= epoch_to_custody_period(validator.exit_epoch): @@ -587,7 +878,15 @@ Run the following on the fork block after per-slot processing and before per-blo For all `validator` in `ValidatorRegistry`, update it to the new format and fill the new member values with: ```python - 'open_branch_challenges': [], 'next_subkey_to_reveal': get_current_custody_period(state), 'reveal_max_periods_late': 0, ``` + +Update the `BeaconState` to the new format and fill the new member values with: + +```python + 'branch_challenge_records': [], + 'next_branch_challenge_id': 0, + 'custody_challenge_records': [], + 'next_custody_challenge_id': 0, +``` From c42cab87a790cc94422ddce90c364fd2b24f90b2 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:55:20 -0600 Subject: [PATCH 069/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 023d64339..801be22f0 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1789,7 +1789,7 @@ def get_current_total_balance(state: BeaconState) -> Gwei: ```python def get_previous_total_balance(state: BeaconState) -> Gwei: - return get_total_balance(state, get_active_validator_indices(state.validator_registry, previous_epoch)) + return get_total_balance(state, get_active_validator_indices(state.validator_registry, get_previous_epoch(state))) ``` ```python From 2d6697eb62372d74c87ba2aae36a52558053de81 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:56:12 -0600 Subject: [PATCH 070/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 801be22f0..da107dad5 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1794,7 +1794,7 @@ def get_previous_total_balance(state: BeaconState) -> Gwei: ```python def get_previous_attestations(state: BeaconState) -> List[PendingAttestation]: - return [a for a in state.latest_attestations if previous_epoch == slot_to_epoch(a.data.slot)] + return [a for a in state.latest_attestations if get_previous_epoch(state) == slot_to_epoch(a.data.slot)] ``` ```python From fa613b401f8b2f1df85fc96832c5cf5aaeadbc74 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:56:54 -0600 Subject: [PATCH 071/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index da107dad5..bd14fa079 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2164,6 +2164,8 @@ def update_registry_and_shuffling_data(state: BeaconState) -> None: state.previous_shuffling_epoch = state.current_shuffling_epoch state.previous_shuffling_start_shard = state.current_shuffling_start_shard state.previous_shuffling_seed = state.current_shuffling_seed + current_epoch = get_current_epoch(state) + next_epoch = current_epoch + 1 # Check if we should update, and if so, update if should_update_validator_registry(state): update_validator_registry(state) From 1103fd1ee86ddb2ef8b382454bc48cd4cdf07773 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:57:27 -0600 Subject: [PATCH 072/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index bd14fa079..a84d6fde8 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1779,7 +1779,7 @@ We define some helper functions: ```python def get_current_attestations(state: BeaconState) -> List[PendingAttestation]: - return [a for a in state.latest_attestations if current_epoch == slot_to_epoch(a.data.slot)] + return [a for a in state.latest_attestations if get_current_epoch(state) == slot_to_epoch(a.data.slot)] ``` ```python From 07599b3a4532bdb4b3d6712d1fad2926ab6b0f3a Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:57:48 -0600 Subject: [PATCH 073/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a84d6fde8..dadd9d3a5 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1784,7 +1784,7 @@ def get_current_attestations(state: BeaconState) -> List[PendingAttestation]: ```python def get_current_total_balance(state: BeaconState) -> Gwei: - return get_total_balance(state, get_active_validator_indices(state.validator_registry, current_epoch)) + return get_total_balance(state, get_active_validator_indices(state.validator_registry, get_current_epoch(state))) ``` ```python From 19924f314873d61e04f35160b02d04a10f5eccdf Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:58:18 -0600 Subject: [PATCH 074/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index dadd9d3a5..5b3af73bf 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2021,7 +2021,7 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> List[Gwei]: deltas = [0 for index in range(len(state.validator_registry))] boundary_attestations = get_previous_epoch_boundary_attestations(state) matching_head_attestations = get_previous_epoch_matching_head_attestations(state) - active_validator_indices = get_active_validator_indices(state.validator_registry, previous_epoch) + active_validator_indices = get_active_validator_indices(state.validator_registry, get_previous_epoch(state)) epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch for index in active_validator_indices: if index not in get_attesting_indices(get_previous_epoch_attestations(state)): From 72c28bc59a225bd82a46cd7790f6b5439eab9f5f Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 17:58:45 +0800 Subject: [PATCH 075/161] Refactor `process_slashings` --- specs/core/0_beacon-chain.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index d18987141..efbff604b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2119,12 +2119,14 @@ def process_slashings(state: BeaconState) -> None: active_validator_indices = get_active_validator_indices(state.validator_registry, current_epoch) total_balance = sum(get_effective_balance(state, i) for i in active_validator_indices) + # Compute `total_penalties` + epoch_index = current_epoch % LATEST_SLASHED_EXIT_LENGTH + total_at_start = state.latest_slashed_balances[(epoch_index + 1) % LATEST_SLASHED_EXIT_LENGTH] + total_at_end = state.latest_slashed_balances[epoch_index] + total_penalties = total_at_end - total_at_start + for index, validator in enumerate(state.validator_registry): if validator.slashed and current_epoch == validator.withdrawable_epoch - LATEST_SLASHED_EXIT_LENGTH // 2: - epoch_index = current_epoch % LATEST_SLASHED_EXIT_LENGTH - total_at_start = state.latest_slashed_balances[(epoch_index + 1) % LATEST_SLASHED_EXIT_LENGTH] - total_at_end = state.latest_slashed_balances[epoch_index] - total_penalties = total_at_end - total_at_start penalty = max( get_effective_balance(state, index) * min(total_penalties * 3, total_balance) // total_balance, get_effective_balance(state, index) // MIN_PENALTY_QUOTIENT From cd7c7228c908f6e271b7d937eb5bad5f3f165ffb Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 03:59:12 -0600 Subject: [PATCH 076/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 5b3af73bf..c10f1d6f5 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2053,7 +2053,9 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> List[Gwei]: ```python def get_crosslink_deltas(state: BeaconState) -> List[Gwei]: deltas = [0 for index in range(len(state.validator_registry))] - for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(current_epoch)): + previous_epoch_start_slot = get_epoch_start_slot(get_previous_epoch(state)) + current_epoch_start_slot = get_epoch_start_slot(get_current_epoch(state)) + for slot in range(previous_epoch_start_slot, current_epoch_start_slot): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) participating_balance = get_total_balance(state, participants) From 868359a6747d05c60774244d234caa764947069c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sun, 3 Mar 2019 04:00:09 -0600 Subject: [PATCH 077/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c10f1d6f5..23dccc4e3 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1877,7 +1877,6 @@ Run the following function: ```python def update_justification_and_finalization(state: BeaconState) -> None: - new_justified_epoch = state.justified_epoch # Rotate the justification bitfield up one epoch to make room for the current epoch state.justification_bitfield <<= 1 From 46b1d4eef5e5971e0e820348d28a7091db6e2a27 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 04:28:13 -0600 Subject: [PATCH 078/161] Assimilated #697 --- specs/core/0_beacon-chain.md | 38 ++++++++++++++---------------------- 1 file changed, 15 insertions(+), 23 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 23dccc4e3..20d8ffcec 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -533,6 +533,8 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git 'current_shuffling_seed': 'bytes32', # Finality + 'previous_epoch_attestations': [PendingAttestation], + 'current_epoch_attestations': [PendingAttestation], 'previous_justified_epoch': 'uint64', 'justified_epoch': 'uint64', 'justification_bitfield': 'uint64', @@ -543,7 +545,6 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git 'latest_block_roots': ['bytes32'], 'latest_active_index_roots': ['bytes32'], 'latest_slashed_balances': ['uint64'], # Balances slashed at every withdrawal period - 'latest_attestations': [PendingAttestation], 'batched_block_roots': ['bytes32'], # Ethereum 1.0 chain data @@ -1461,6 +1462,8 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], current_shuffling_seed=ZERO_HASH, # Finality + previous_epoch_attestations=[], + current_epoch_attestations=[], previous_justified_epoch=GENESIS_EPOCH, justified_epoch=GENESIS_EPOCH, justification_bitfield=0, @@ -1471,7 +1474,6 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], latest_block_roots=[ZERO_HASH for _ in range(LATEST_BLOCK_ROOTS_LENGTH)], latest_active_index_roots=[ZERO_HASH for _ in range(LATEST_ACTIVE_INDEX_ROOTS_LENGTH)], latest_slashed_balances=[0 for _ in range(LATEST_SLASHED_EXIT_LENGTH)], - latest_attestations=[], batched_block_roots=[], # Ethereum 1.0 chain data @@ -1702,7 +1704,9 @@ For each `attestation` in `block.body.attestations`: ``` * [TO BE REMOVED IN PHASE 1] Verify that `attestation.data.crosslink_data_root == ZERO_HASH`. -* Append `PendingAttestation(data=attestation.data, aggregation_bitfield=attestation.aggregation_bitfield, custody_bitfield=attestation.custody_bitfield, inclusion_slot=state.slot)` to `state.latest_attestations`. +* Let `pending_attestation = PendingAttestation(data=attestation.data, aggregation_bitfield=attestation.aggregation_bitfield, custody_bitfield=attestation.custody_bitfield, inclusion_slot=state.slot)`. +* Append `pending_attestation` to `state.previous_epoch_attestations` if `slot_to_epoch(attestation.data.slot) == get_previous_epoch(state)`. +* Append `pending_attestation` to `state.current_epoch_attestations` if `slot_to_epoch(attestation.data.slot) == get_current_epoch(state)`. ##### Deposits @@ -1777,11 +1781,6 @@ The steps below happen when `(state.slot + 1) % SLOTS_PER_EPOCH == 0`. We define some helper functions: -```python -def get_current_attestations(state: BeaconState) -> List[PendingAttestation]: - return [a for a in state.latest_attestations if get_current_epoch(state) == slot_to_epoch(a.data.slot)] -``` - ```python def get_current_total_balance(state: BeaconState) -> Gwei: return get_total_balance(state, get_active_validator_indices(state.validator_registry, get_current_epoch(state))) @@ -1792,11 +1791,6 @@ def get_previous_total_balance(state: BeaconState) -> Gwei: return get_total_balance(state, get_active_validator_indices(state.validator_registry, get_previous_epoch(state))) ``` -```python -def get_previous_attestations(state: BeaconState) -> List[PendingAttestation]: - return [a for a in state.latest_attestations if get_previous_epoch(state) == slot_to_epoch(a.data.slot)] -``` - ```python def get_attesting_indices(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: output = set({}) @@ -1813,7 +1807,7 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat ```python def get_current_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ - a for a in get_current_attestations(state) if + a for a in state.current_epoch_attestations if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) ] ``` @@ -1821,7 +1815,7 @@ def get_current_epoch_boundary_attestations(state: BeaconState) -> List[PendingA ```python def get_previous_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ - a for a in get_previous_attestations(state) if + a for a in state.previous_epoch_attestations if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) ] ``` @@ -1829,7 +1823,7 @@ def get_previous_epoch_boundary_attestations(state: BeaconState) -> List[Pending ```python def get_previous_epoch_matching_head_attestations(state: BeaconState) -> List[PendingAttestation]: return [ - a for a in get_previous_epoch_attestations(state) if + a for a in state.previous_epoch_attestations if a.data.beacon_block_root == get_block_root(state, a.data.slot) ] ``` @@ -1838,7 +1832,7 @@ def get_previous_epoch_matching_head_attestations(state: BeaconState) -> List[Pe ```python def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple[Bytes32, List[ValidatorIndex]]: - all_attestations = get_current_attestations(state) + get_previous_attestations(state) + all_attestations = state.current_epoch_attestations + state.previous_epoch_attestations valid_attestations = [ a for a in all_attestations if a.data.latest_crosslink == state.latest_crosslinks[shard] ] @@ -1989,7 +1983,7 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> # Process rewards or penalties for all validators for index in get_active_validator_indices(state.validator_registry, previous_epoch): # Expected FFG source - if index in get_attesting_indices(get_previous_attestations(state)): + if index in get_attesting_indices(state.previous_epoch_attestations): deltas[index] += get_base_reward(state, index) * total_attesting_balance // total_balance # Inclusion speed bonus deltas[index] += ( @@ -2258,11 +2252,9 @@ def finish_epoch_update(state: BeaconState) -> None: ) # Set randao mix state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch) - # Remove previous epoch attestations - state.latest_attestations = [ - a for a in state.latest_attestations if - slot_to_epoch(attestation.data.slot) == current_epoch - ] + # Rotate current/previous epoch attestations + state.previous_epoch_attestations = state.current_epoch_attestations + state.current_epoch_attestations = [] ``` ### State root verification From d0fc455a1f3953fedaab3ccacc8ecebd3586bf19 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 05:04:28 -0600 Subject: [PATCH 079/161] Assimilated #649 --- specs/core/0_beacon-chain.md | 253 +++++++++++++++++++++-------------- 1 file changed, 156 insertions(+), 97 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 569635879..3d473f669 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -41,8 +41,8 @@ - [`Transfer`](#transfer) - [Beacon chain blocks](#beacon-chain-blocks) - [`BeaconBlock`](#beaconblock) + - [`BeaconBlockHeader`](#beaconblockheader) - [`BeaconBlockBody`](#beaconblockbody) - - [`Proposal`](#proposal) - [Beacon chain state](#beacon-chain-state) - [`BeaconState`](#beaconstate) - [`Validator`](#validator) @@ -56,6 +56,7 @@ - [`hash`](#hash) - [`hash_tree_root`](#hash_tree_root) - [`signed_root`](#signed_root) + - [`get_temporary_block_header`](#get_temporary_block_header) - [`slot_to_epoch`](#slot_to_epoch) - [`get_previous_epoch`](#get_previous_epoch) - [`get_current_epoch`](#get_current_epoch) @@ -115,8 +116,7 @@ - [Slot](#slot) - [Block roots](#block-roots) - [Per-block processing](#per-block-processing) - - [Slot](#slot-1) - - [Block signature](#block-signature) + - [Block header](#block-header) - [RANDAO](#randao) - [Eth1 data](#eth1-data) - [Transactions](#transactions) @@ -185,7 +185,6 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | `SHARD_COUNT` | `2**10` (= 1,024) | | `TARGET_COMMITTEE_SIZE` | `2**7` (= 128) | | `MAX_BALANCE_CHURN_QUOTIENT` | `2**5` (= 32) | -| `BEACON_CHAIN_SHARD_NUMBER` | `2**64 - 1` | | `MAX_INDICES_PER_SLASHABLE_VOTE` | `2**12` (= 4,096) | | `MAX_EXIT_DEQUEUES_PER_EPOCH` | `2**2` (= 4) | | `SHUFFLE_ROUND_COUNT` | 90 | @@ -233,13 +232,13 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | `MIN_SEED_LOOKAHEAD` | `2**0` (= 1) | epochs | 6.4 minutes | | `ACTIVATION_EXIT_DELAY` | `2**2` (= 4) | epochs | 25.6 minutes | | `EPOCHS_PER_ETH1_VOTING_PERIOD` | `2**4` (= 16) | epochs | ~1.7 hours | +| `SLOTS_PER_HISTORICAL_ROOT` | `2**13` (= 8,192) | slots | ~13 hours | | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY` | `2**8` (= 256) | epochs | ~27 hours | ### State list lengths | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `LATEST_BLOCK_ROOTS_LENGTH` | `2**13` (= 8,192) | slots | ~13 hours | | `LATEST_RANDAO_MIXES_LENGTH` | `2**13` (= 8,192) | epochs | ~36 days | | `LATEST_ACTIVE_INDEX_ROOTS_LENGTH` | `2**13` (= 8,192) | epochs | ~36 days | | `LATEST_SLASHED_EXIT_LENGTH` | `2**13` (= 8,192) | epochs | ~36 days | @@ -273,12 +272,12 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | Name | Value | | - | - | -| `DOMAIN_DEPOSIT` | `0` | -| `DOMAIN_ATTESTATION` | `1` | -| `DOMAIN_PROPOSAL` | `2` | -| `DOMAIN_EXIT` | `3` | -| `DOMAIN_RANDAO` | `4` | -| `DOMAIN_TRANSFER` | `5` | +| `DOMAIN_BEACON_BLOCK` | `0` | +| `DOMAIN_RANDAO` | `1` | +| `DOMAIN_ATTESTATION` | `2` | +| `DOMAIN_DEPOSIT` | `3` | +| `DOMAIN_VOLUNTARY_EXIT` | `4` | +| `DOMAIN_TRANSFER` | `5` | ## Data structures @@ -294,10 +293,10 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git { # Proposer index 'proposer_index': 'uint64', - # First proposal - 'proposal_1': Proposal, - # Second proposal - 'proposal_2': Proposal, + # First block header + 'header_1': BeaconBlockHeader, + # Second block header + 'header_2': BeaconBlockHeader, } ``` @@ -467,14 +466,21 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git { # Header 'slot': 'uint64', - 'parent_root': 'bytes32', + 'previous_block_root': 'bytes32', 'state_root': 'bytes32', - 'randao_reveal': 'bytes96', - 'eth1_data': Eth1Data, - - # Body 'body': BeaconBlockBody, - # Signature + 'signature': 'bytes96', +} +``` + +#### `BeaconBlockHeader` + +```python +{ + 'slot': 'uint64', + 'previous_block_root': 'bytes32', + 'state_root': 'bytes32', + 'block_body_root': 'bytes32', 'signature': 'bytes96', } ``` @@ -483,6 +489,8 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git ```python { + 'randao_reveal': 'bytes96', + 'eth1_data': Eth1Data, 'proposer_slashings': [ProposerSlashing], 'attester_slashings': [AttesterSlashing], 'attestations': [Attestation], @@ -492,21 +500,6 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -#### `Proposal` - -```python -{ - # Slot number - 'slot': 'uint64', - # Shard number (`BEACON_CHAIN_SHARD_NUMBER` for beacon chain) - 'shard': 'uint64', - # Block root - 'block_root': 'bytes32', - # Signature - 'signature': 'bytes96', -} -``` - ### Beacon chain state #### `BeaconState` @@ -543,9 +536,11 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git # Recent state 'latest_crosslinks': [Crosslink], 'latest_block_roots': ['bytes32'], + 'latest_state_roots': ['bytes32'], 'latest_active_index_roots': ['bytes32'], 'latest_slashed_balances': ['uint64'], # Balances slashed at every withdrawal period - 'batched_block_roots': ['bytes32'], + 'latest_block_header': BeaconBlockHeader, # `latest_block_header.state_root == ZERO_HASH` temporarily + 'historical_roots': ['bytes32'], # Ethereum 1.0 chain data 'latest_eth1_data': Eth1Data, @@ -669,6 +664,22 @@ Note: We aim to migrate to a S[T/N]ARK-friendly hash function in a future Ethere `def signed_root(object: SSZContainer) -> Bytes32` is a function defined in the [SimpleSerialize spec](https://github.com/ethereum/eth2.0-specs/blob/master/specs/simple-serialize.md#signed-roots) to compute signed messages. +### `get_temporary_block_header` + +```python +def get_temporary_block_header(block: BeaconBlock) -> BeaconBlockHeader: + """ + Return the block header corresponding to a block with ``state_root`` set to ``ZERO_HASH``. + """ + return BeaconBlockHeader( + slot=block.slot, + previous_block_root=block.previous_block_root, + state_root=ZERO_HASH, + block_body_root=hash_tree_root(block.body), + signature=block.signature, + ) +``` + ### `slot_to_epoch` ```python @@ -920,9 +931,8 @@ def get_block_root(state: BeaconState, """ Return the block root at a recent ``slot``. """ - assert state.slot <= slot + LATEST_BLOCK_ROOTS_LENGTH - assert slot < state.slot - return state.latest_block_roots[slot % LATEST_BLOCK_ROOTS_LENGTH] + assert slot < state.slot <= slot + SLOTS_PER_HISTORICAL_ROOT + return state.latest_block_roots[slot % SLOTS_PER_HISTORICAL_ROOT] ``` `get_block_root(_, s)` should always return `hash_tree_root` of the block in the beacon chain at slot `s`, and `get_crosslink_committees_at_slot(_, s)` should not change unless the [validator](#dfn-validator) registry changes. @@ -1405,35 +1415,47 @@ For convenience, we provide the interface to the contract here: ## On genesis -A valid block with slot `GENESIS_SLOT` (a "genesis block") has the following values. Other validity rules (e.g. requiring a signature) do not apply. +When enough full deposits have been made to the deposit contract, an `Eth2Genesis` log is emitted. Construct a corresponding `genesis_state` and `genesis_block` as follows: + +* Let `genesis_validator_deposits` be the list of deposits, ordered chronologically, up to and including the deposit that triggered the `Eth2Genesis` log. +* Let `genesis_time` be the timestamp specified in the `Eth2Genesis` log. +* Let `genesis_eth1_data` be the `Eth1Data` object where: + * `genesis_eth1_data.deposit_root` is the `deposit_root` contained in the `Eth2Genesis` log. + * `genesis_eth1_data.block_hash` is the hash of the Ethereum 1.0 block that emitted the `Eth2Genesis` log. +* Let `genesis_state = get_genesis_beacon_state(genesis_validator_deposits, genesis_time, genesis_eth1_data)`. +* Let `genesis_block = get_empty_block()`. +* Set `genesis_block.state_root = hash_tree_root(genesis_state)`. ```python -{ - slot=GENESIS_SLOT, - parent_root=ZERO_HASH, - state_root=GENESIS_STATE_ROOT, - randao_reveal=EMPTY_SIGNATURE, - eth1_data=Eth1Data( - deposit_root=ZERO_HASH, - block_hash=ZERO_HASH - ), - signature=EMPTY_SIGNATURE, - body=BeaconBlockBody( - proposer_slashings=[], - attester_slashings=[], - attestations=[], - deposits=[], - exits=[], - ), -} +def get_empty_block() -> BeaconBlock: + """ + Get an empty ``BeaconBlock``. + """ + return BeaconBlock( + slot=GENESIS_SLOT, + previous_block_root=ZERO_HASH, + state_root=ZERO_HASH, + body=BeaconBlockBody( + randao_reveal=EMPTY_SIGNATURE, + eth1_data=Eth1Data( + deposit_root=ZERO_HASH, + block_hash=ZERO_HASH, + ), + proposer_slashings=[], + attester_slashings=[], + attestations=[], + deposits=[], + exits=[], + transfers=[], + ), + signature=EMPTY_SIGNATURE, + ) ``` -`GENESIS_STATE_ROOT` (in the above "genesis block") is generated from the `get_genesis_beacon_state` function below. When enough full deposits have been made to the deposit contract and the `Eth2Genesis` log has been emitted, `get_genesis_beacon_state` will execute to compute the `hash_tree_root` of `BeaconState`. - ```python def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], genesis_time: int, - latest_eth1_data: Eth1Data) -> BeaconState: + genesis_eth1_data: Eth1Data) -> BeaconState: """ Get the genesis ``BeaconState``. """ @@ -1471,13 +1493,15 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], # Recent state latest_crosslinks=[Crosslink(epoch=GENESIS_EPOCH, crosslink_data_root=ZERO_HASH) for _ in range(SHARD_COUNT)], - latest_block_roots=[ZERO_HASH for _ in range(LATEST_BLOCK_ROOTS_LENGTH)], + latest_block_roots=[ZERO_HASH for _ in range(SLOTS_PER_HISTORICAL_ROOT)], + latest_state_roots=[ZERO_HASH for _ in range(SLOTS_PER_HISTORICAL_ROOT)], latest_active_index_roots=[ZERO_HASH for _ in range(LATEST_ACTIVE_INDEX_ROOTS_LENGTH)], latest_slashed_balances=[0 for _ in range(LATEST_SLASHED_EXIT_LENGTH)], - batched_block_roots=[], + latest_block_header=get_temporary_block_header(get_empty_block()), + historical_roots=[], # Ethereum 1.0 chain data - latest_eth1_data=latest_eth1_data, + latest_eth1_data=genesis_eth1_data, eth1_data_votes=[], deposit_index=len(genesis_validator_deposits) ) @@ -1511,7 +1535,7 @@ Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Cli For a beacon chain block, `block`, to be processed by a node, the following conditions must be met: -* The parent block with root `block.parent_root` has been processed and accepted. +* The parent block with root `block.previous_block_root` has been processed and accepted. * An Ethereum 1.0 block pointed to by the `state.latest_eth1_data.block_hash` has been processed and accepted. * The node's Unix time is greater than or equal to `state.genesis_time + (block.slot - GENESIS_SLOT) * SECONDS_PER_SLOT`. (Note that leap seconds mean that slots will occasionally last `SECONDS_PER_SLOT + 1` or `SECONDS_PER_SLOT - 1` seconds, possibly several times a year.) @@ -1590,41 +1614,73 @@ _Note_: If there are skipped slots between a block and its parent block, run the ### Per-slot processing -Below are the processing steps that happen at every slot. +At every `slot > GENESIS_SLOT` run the following function: -#### Slot - -* Set `state.slot += 1`. - -#### Block roots - -* Let `previous_block_root` be the `hash_tree_root` of the previous beacon block processed in the chain. -* Set `state.latest_block_roots[(state.slot - 1) % LATEST_BLOCK_ROOTS_LENGTH] = previous_block_root`. -* If `state.slot % LATEST_BLOCK_ROOTS_LENGTH == 0` append `merkle_root(state.latest_block_roots)` to `state.batched_block_roots`. +```python +def advance_slot(state: BeaconState) -> None: + state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) + state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = get_block_root(state, state.slot - 1) + if state.latest_block_header.state_root == ZERO_HASH: + state.latest_block_header.state_root = get_state_root(state, state.slot) + state.slot += 1 +``` ### Per-block processing -Below are the processing steps that happen at every `block`. +For every `block` except the genesis block, run `process_block_header(state, block)`, `process_randao(state, block)` and `process_eth1_data(state, block)`. -#### Slot +#### Block header -* Verify that `block.slot == state.slot`. - -#### Block signature - -* Let `proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)]`. -* Let `proposal = Proposal(block.slot, BEACON_CHAIN_SHARD_NUMBER, signed_root(block, "signature"), block.signature)`. -* Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposal, "signature"), signature=proposal.signature, domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_PROPOSAL))`. +```python +def process_block_header(state: BeaconState, block: BeaconBlock) -> None: + # Verify that the slots match + assert block.slot == state.slot + # Verify that the parent matches + assert block.previous_block_root == hash_tree_root(state.latest_block_header) + # Save previous block root + state.latest_block_roots[(state.slot - 1) % SLOTS_PER_HISTORICAL_ROOT] = block.previous_block_root + # Save current block as the new latest block + state.latest_block_header = get_temporary_block_header(block) + # Verify proposer signature + proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)] + assert bls_verify( + pubkey=proposer.pubkey, + message_hash=signed_root(block, "signature"), + signature=block.signature, + domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_BEACON_BLOCK) + ) +``` #### RANDAO -* Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=hash_tree_root(get_current_epoch(state)), signature=block.randao_reveal, domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_RANDAO))`. -* Set `state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = xor(get_randao_mix(state, get_current_epoch(state)), hash(block.randao_reveal))`. +```python +def process_randao(state: BeaconState, block: BeaconBlock) -> None: + # Verify that the provided randao value is valid + assert bls_verify( + pubkey=proposer.pubkey, + message_hash=hash_tree_root(get_current_epoch(state)), + signature=block.body.randao_reveal, + domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_RANDAO) + ) + # Mix it in + state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = ( + xor(get_randao_mix(state, get_current_epoch(state)), + hash(block.body.randao_reveal)) + ) +``` #### Eth1 data -* If there exists an `eth1_data_vote` in `state.eth1_data_votes` for which `eth1_data_vote.eth1_data == block.eth1_data` (there will be at most one), set `eth1_data_vote.vote_count += 1`. -* Otherwise, append to `state.eth1_data_votes` a new `Eth1DataVote(eth1_data=block.eth1_data, vote_count=1)`. +```python +def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: + for eth1_data_vote in state.eth1_data_votes: + # If someone else has already voted for the same hash, add to its counter + if eth1_data_vote.eth1_data == block.body.eth1_data: + eth1_data_vote.vote_count += 1 + return + # If we're seeing this hash for the first time, make a new counter + state.eth1_data_votes.append(Eth1DataVote(eth1_data=block.body.eth1_data, vote_count=1)) +``` #### Transactions @@ -1643,20 +1699,20 @@ def process_proposer_slashing(state: BeaconState, """ proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same - assert proposer_slashing.proposal_1.slot == proposer_slashing.proposal_2.slot + assert proposer_slashing.header_1.slot == proposer_slashing.header_2.slot # Verify that the shard is the same (or that both proposals are beacon chain proposals) - assert proposer_slashing.proposal_1.shard == proposer_slashing.proposal_2.shard + assert proposer_slashing.header_1.shard == proposer_slashing.header_2.shard # But the roots are different! - assert proposer_slashing.proposal_1.block_root != proposer_slashing.proposal_2.block_root + assert proposer_slashing.header_1.block_root != proposer_slashing.header_2.block_root # Proposer is not yet slashed assert proposer.slashed is False # Signatures are valid - for proposal in (proposer_slashing.proposal_1, proposer_slashing.proposal_2): + for header in (proposer_slashing.header_1, proposer_slashing.header_2): assert bls_verify( pubkey=proposer.pubkey, - message_hash=signed_root(proposal, "signature"), - signature=proposal.signature, - domain=get_domain(state.fork, slot_to_epoch(proposal.slot), DOMAIN_PROPOSAL) + message_hash=signed_root(header, "signature"), + signature=header.signature, + domain=get_domain(state.fork, slot_to_epoch(header.slot), DOMAIN_PROPOSAL) ) slash_validator(state, proposer_slashing.proposer_index) ``` @@ -1836,7 +1892,7 @@ def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: pubkey=validator.pubkey, message_hash=signed_root(exit, "signature"), signature=exit.signature, - domain=get_domain(state.fork, exit.epoch, DOMAIN_EXIT) + domain=get_domain(state.fork, exit.epoch, DOMAIN_VOLUNTARY_EXIT) ) # Run the exit initiate_validator_exit(state, exit.validator_index) @@ -2368,6 +2424,9 @@ def finish_epoch_update(state: BeaconState) -> None: ) # Set randao mix state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch) + # Set historical root accumulator + if next_epoch % (SLOTS_PER_HISTORICAL_ROOT // SLOTS_PER_EPOCH) == 0: + state.historical_roots.append(merkle_root(state.latest_block_roots + state.latest_state_roots)) # Rotate current/previous epoch attestations state.previous_epoch_attestations = state.current_epoch_attestations state.current_epoch_attestations = [] From 73f9ad0512c8dfd06a11dcb3078be092f9fe8bb4 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 05:54:58 -0600 Subject: [PATCH 080/161] Added missing colon --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3d473f669..89e7584b6 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -800,7 +800,7 @@ def get_epoch_committee_count(active_validator_count: int) -> int: ```python def get_shuffling(seed: Bytes32, validators: List[Validator], - epoch: Epoch) -> List[List[ValidatorIndex]] + epoch: Epoch) -> List[List[ValidatorIndex]]: """ Shuffle active validators and split into crosslink committees. Return a list of committees (each a list of validator indices). From 68bc91bb3a7fab36b11ea8d56f356c8197687449 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 05:56:57 -0600 Subject: [PATCH 081/161] Fixed custody bit participants typo --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 89e7584b6..a4bb8908e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1807,7 +1807,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Verify aggregate signature participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) custody_bit_1_participants = get_attestation_participants(state, attestation.data, attestation.custody_bitfield) - custody_bit_0_participants = [i in participants for i not in custody_bit_1_participants] + custody_bit_0_participants = [i for i in participants if i not in custody_bit_1_participants] assert bls_verify_multiple( pubkeys=[ From 805ea5dad51e68f640a2cf7869c1d80b1f7ecacd Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 05:59:11 -0600 Subject: [PATCH 082/161] Transfer from -> sender Needed to avoid triggering errors in python --- specs/core/0_beacon-chain.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a4bb8908e..6e4e74b63 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -442,7 +442,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git ```python { # Sender index - 'from': 'uint64', + 'sender': 'uint64', # Recipient index 'to': 'uint64', # Amount in Gwei @@ -1913,23 +1913,23 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: Note that this function mutates ``state``. """ # Verify the amount and fee aren't individually too big (for anti-overflow purposes) - assert state.validator_balances[transfer.from] >= max(transfer.amount, transfer.fee) + assert state.validator_balances[transfer.sender] >= max(transfer.amount, transfer.fee) # Verify that we have enough ETH to send, and that after the transfer the balance will be either # exactly zero or at least MIN_DEPOSIT_AMOUNT assert ( - state.validator_balances[transfer.from] == transfer.amount + transfer.fee or - state.validator_balances[transfer.from] >= transfer.amount + transfer.fee + MIN_DEPOSIT_AMOUNT + state.validator_balances[transfer.sender] == transfer.amount + transfer.fee or + state.validator_balances[transfer.sender] >= transfer.amount + transfer.fee + MIN_DEPOSIT_AMOUNT ) # A transfer is valid in only one slot assert state.slot == transfer.slot # Only withdrawn or not-yet-deposited accounts can transfer assert ( - get_current_epoch(state) >= state.validator_registry[transfer.from].withdrawable_epoch or - state.validator_registry[transfer.from].activation_epoch == FAR_FUTURE_EPOCH + get_current_epoch(state) >= state.validator_registry[transfer.sender].withdrawable_epoch or + state.validator_registry[transfer.sender].activation_epoch == FAR_FUTURE_EPOCH ) # Verify that the pubkey is valid assert ( - state.validator_registry[transfer.from].withdrawal_credentials == + state.validator_registry[transfer.sender].withdrawal_credentials == BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:] ) # Verify that the signature is valid @@ -1940,7 +1940,7 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: domain=get_domain(state.fork, slot_to_epoch(transfer.slot), DOMAIN_TRANSFER) ) # Process the transfer - state.validator_balances[transfer.from] -= transfer.amount + transfer.fee + state.validator_balances[transfer.sender] -= transfer.amount + transfer.fee state.validator_balances[transfer.to] += transfer.amount state.validator_balances[get_beacon_proposer_index(state, state.slot)] += transfer.fee ``` From 38e166d4a3233449542bf421d40a67e3ef947355 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 06:01:03 -0600 Subject: [PATCH 083/161] More python typos! --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6e4e74b63..369c74a20 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2268,7 +2268,7 @@ def should_update_validator_registry(state: BeaconState) -> bool: if state.finalized_epoch <= state.validator_registry_update_epoch: return False # Must have processed new crosslinks on all shards of the current epoch - shards_to_check [ + shards_to_check = [ (state.current_shuffling_start_shard + i) % SHARD_COUNT for i in range(get_current_epoch_committee_count(state)) ] @@ -2340,7 +2340,7 @@ def update_registry_and_shuffling_data(state: BeaconState) -> None: state.current_shuffling_epoch = next_epoch state.current_shuffling_start_shard = ( state.current_shuffling_start_shard + - get_current_epoch_committee_count(state)) % SHARD_COUNT + get_current_epoch_committee_count(state) % SHARD_COUNT ) state.current_shuffling_seed = generate_seed(state, state.current_shuffling_epoch) else: From 1444687798bec9c01b1f88400479a501bacb8ef7 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Sun, 3 Mar 2019 20:14:03 -0600 Subject: [PATCH 084/161] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 88a8f12eb..6ef5e449f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -277,7 +277,7 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | `DOMAIN_ATTESTATION` | `2` | | `DOMAIN_DEPOSIT` | `3` | | `DOMAIN_VOLUNTARY_EXIT` | `4` | -| `DOMAIN_TRANSFER` | `5` | +| `DOMAIN_TRANSFER` | `5` | ## Data structures From 044c87fe3988daa7c7e2770903897720ad95cef8 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 3 Mar 2019 22:41:23 -0600 Subject: [PATCH 085/161] fix State -> BeaconState --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6ef5e449f..c82e2aefd 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1055,7 +1055,7 @@ def bytes_to_int(data: bytes) -> int: ### `get_effective_balance` ```python -def get_effective_balance(state: State, index: ValidatorIndex) -> Gwei: +def get_effective_balance(state: BeaconState, index: ValidatorIndex) -> Gwei: """ Return the effective balance (also known as "balance at stake") for a validator with the given ``index``. """ From 5282b289cf6eed3ac8cf89e50d600ef1d2bded84 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 4 Mar 2019 04:49:24 -0600 Subject: [PATCH 086/161] exits -> voluntary_exits --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c82e2aefd..e0b7ae61f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1448,7 +1448,7 @@ def get_empty_block() -> BeaconBlock: attester_slashings=[], attestations=[], deposits=[], - exits=[], + voluntary_exits=[], transfers=[], ), signature=EMPTY_SIGNATURE, From e2ce0d95831a92c91ea86cced387dc51815eec92 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 4 Mar 2019 06:45:55 -0800 Subject: [PATCH 087/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0fc48f090..c701f57d1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1063,7 +1063,7 @@ def get_effective_balance(state: State, index: ValidatorIndex) -> Gwei: ```python def get_total_balance(state: BeaconState, validators: List[ValidatorIndex]) -> Gwei: """ - Return the combined effective balance of an array of validators. + Return the combined effective balance of an array of ``validators``. """ return sum([get_effective_balance(state, i) for i in validators]) ``` From 908d7c42ad3ea9ce7c912fd7d0051bc2264e7987 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 09:17:43 -0700 Subject: [PATCH 088/161] pr feedback --- specs/core/0_beacon-chain.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index e0b7ae61f..19b729f30 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1658,6 +1658,7 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: ```python def process_randao(state: BeaconState, block: BeaconBlock) -> None: + proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)] # Verify that the provided randao value is valid assert bls_verify( pubkey=proposer.pubkey, @@ -1968,7 +1969,7 @@ def get_previous_total_balance(state: BeaconState) -> Gwei: ```python def get_attesting_indices(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: - output = set({}) + output = set() for a in attestations: output = output.union([get_attestation_participants(state, a.data, a.aggregation_bitfield)]) return sorted(list(output)) @@ -2087,6 +2088,9 @@ Run the following function: ```python def process_crosslinks(state: BeaconState) -> None: + current_epoch = get_current_epoch(state) + previous_epoch = current_epoch - 1 + next_epoch = current_epoch + 1 for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) @@ -2124,6 +2128,7 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: ```python def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex) -> Gwei: + epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch return ( get_base_reward(state, index) + get_effective_balance(state, index) * epochs_since_finality // INACTIVITY_PENALTY_QUOTIENT // 2 From 831f04e3ea03c619f790794b7f236aba4aea531c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 09:45:55 -0700 Subject: [PATCH 089/161] separate our rewards/penalty deltas --- specs/core/0_beacon-chain.md | 67 ++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 25 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 19b729f30..4e8f9a25e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2140,7 +2140,7 @@ Note: When applying penalties in the following balance recalculations implemente ##### Justification and finalization ```python -def get_justification_and_finalization_deltas(state: BeaconState) -> List[Gwei]: +def get_justification_and_finalization_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch if epochs_since_finality <= 4: return compute_normal_justification_and_finalization_deltas(state) @@ -2151,8 +2151,13 @@ def get_justification_and_finalization_deltas(state: BeaconState) -> List[Gwei]: When blocks are finalizing normally... ```python -def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> List[Gwei]: - deltas = [0 for index in range(len(state.validator_registry))] +def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: + # deltas[0] for rewards + # deltas[1] for penalties + deltas = [ + [0 for index in range(len(state.validator_registry))], + [0 for index in range(len(state.validator_registry))] + ] # Some helper variables boundary_attestations = get_previous_epoch_boundary_attestations(state) boundary_attesting_balance = get_attesting_balance(boundary_attestations) @@ -2164,57 +2169,61 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> for index in get_active_validator_indices(state.validator_registry, previous_epoch): # Expected FFG source if index in get_attesting_indices(state.previous_epoch_attestations): - deltas[index] += get_base_reward(state, index) * total_attesting_balance // total_balance + deltas[0][index] += get_base_reward(state, index) * total_attesting_balance // total_balance # Inclusion speed bonus - deltas[index] += ( + deltas[0][index] += ( get_base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index) ) else: - deltas[index] -= get_base_reward(state, index) + deltas[1][index] += get_base_reward(state, index) # Expected FFG target if index in get_attesting_indices(boundary_attestations): - deltas[index] += get_base_reward(state, index) * boundary_attesting_balance // total_balance + deltas[0][index] += get_base_reward(state, index) * boundary_attesting_balance // total_balance else: - deltas[index] -= get_base_reward(state, index) + deltas[1][index] += get_base_reward(state, index) # Expected head if index in get_attesting_indices(matching_head_attestations): - deltas[index] += get_base_reward(state, index) * matching_head_balance // total_balance + deltas[0][index] += get_base_reward(state, index) * matching_head_balance // total_balance else: - deltas[index] -= get_base_reward(state, index) + deltas[1][index] += get_base_reward(state, index) # Proposer bonus proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) - deltas[proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT + deltas[0][proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT ``` When blocks are not finalizing normally... ```python -def compute_inactivity_leak_deltas(state: BeaconState) -> List[Gwei]: - deltas = [0 for index in range(len(state.validator_registry))] +def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: + # deltas[0] for rewards + # deltas[1] for penalties + deltas = [ + [0 for index in range(len(state.validator_registry))], + [0 for index in range(len(state.validator_registry))] + ] boundary_attestations = get_previous_epoch_boundary_attestations(state) matching_head_attestations = get_previous_epoch_matching_head_attestations(state) active_validator_indices = get_active_validator_indices(state.validator_registry, get_previous_epoch(state)) epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch for index in active_validator_indices: if index not in get_attesting_indices(get_previous_epoch_attestations(state)): - deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) + deltas[1][index] += get_inactivity_penalty(state, index, epochs_since_finality) else: # If a validator did attest, apply a small penalty for getting attestations included late - deltas[index] += ( + deltas[0][index] += ( base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index) ) - deltas[index] -= base_reward(state, index) - + deltas[1][index] += base_reward(state, index) if index not in get_attesting_indices(boundary_attestations): - deltas[index] -= get_inactivity_penalty(state, index, epochs_since_finality) + deltas[1][index] += get_inactivity_penalty(state, index, epochs_since_finality) if index not in get_attesting_indices(matching_head_attestations): - deltas[index] -= get_base_reward(state, index) + deltas[1][index] += get_base_reward(state, index) # Penalize slashed-but-inactive validators as though they were active but offline for index in range(len(state.validator_registry)): if index not in active_validator_indices and state.validator_registry[index].slashed: - deltas[index] -= ( + deltas[1][index] += ( 2 * get_inactivity_penalty(state, index, epochs_since_finality) + get_base_reward(state, index) ) @@ -2224,8 +2233,13 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> List[Gwei]: ##### Crosslinks ```python -def get_crosslink_deltas(state: BeaconState) -> List[Gwei]: - deltas = [0 for index in range(len(state.validator_registry))] +def get_crosslink_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: + # deltas[0] for rewards + # deltas[1] for penalties + deltas = [ + [0 for index in range(len(state.validator_registry))], + [0 for index in range(len(state.validator_registry))] + ] previous_epoch_start_slot = get_epoch_start_slot(get_previous_epoch(state)) current_epoch_start_slot = get_epoch_start_slot(get_current_epoch(state)) for slot in range(previous_epoch_start_slot, current_epoch_start_slot): @@ -2235,9 +2249,9 @@ def get_crosslink_deltas(state: BeaconState) -> List[Gwei]: total_balance = get_total_balance(state, committee) for index in crosslink_committee: if index in participants: - deltas[index] += get_base_reward(state, index) * participating_balance // total_balance + deltas[0][index] += get_base_reward(state, index) * participating_balance // total_balance else: - deltas[index] -= get_base_reward(state, index) + deltas[1][index] += get_base_reward(state, index) return deltas ``` @@ -2250,7 +2264,10 @@ def apply_rewards(state: BeaconState) -> None: deltas1 = get_justification_and_finalization_deltas(state) deltas2 = get_crosslink_deltas(state) for i in range(len(state.validator_registry)): - state.validator_balances[i] = max(0, state.validator_balances[i] + deltas1[i] + deltas2[i]) + state.validator_balances[i] = max( + 0, + state.validator_balances[i] + deltas1[0][i] + deltas2[0][i] - deltas1[1][i] - deltas2[1][i] + ) ``` #### Ejections From a561656b68923cc1ae4d2e751eeeca119a13130d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 11:45:41 -0700 Subject: [PATCH 090/161] add get_state_root helper --- specs/core/0_beacon-chain.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0d330c32b..a0cbb063f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -72,6 +72,7 @@ - [`get_next_epoch_committee_count`](#get_next_epoch_committee_count) - [`get_crosslink_committees_at_slot`](#get_crosslink_committees_at_slot) - [`get_block_root`](#get_block_root) + - [`get_state_root`](#get_state_root) - [`get_randao_mix`](#get_randao_mix) - [`get_active_index_root`](#get_active_index_root) - [`generate_seed`](#generate_seed) @@ -941,6 +942,17 @@ def get_block_root(state: BeaconState, `get_block_root(_, s)` should always return `hash_tree_root` of the block in the beacon chain at slot `s`, and `get_crosslink_committees_at_slot(_, s)` should not change unless the [validator](#dfn-validator) registry changes. +### `get_state_root` + +```python +def get_state_root(state: BeaconState, + slot: Slot) -> Bytes32: + """ + Return the state root at a recent ``slot``. + """ + assert slot < state.slot <= slot + SLOTS_PER_HISTORICAL_ROOT + return state.latest_state_roots[slot % SLOTS_PER_HISTORICAL_ROOT] +``` ### `get_randao_mix` ```python @@ -1674,9 +1686,9 @@ At every `slot > GENESIS_SLOT` run the following function: def advance_slot(state: BeaconState) -> None: state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = get_block_root(state, state.slot - 1) - if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = get_state_root(state, state.slot) state.slot += 1 + if state.latest_block_header.state_root == ZERO_HASH: + state.latest_block_header.state_root = get_state_root(state, state.slot - 1) ``` ### Per-block processing From f7397a5970819b2aaaf122a61901da499c65e582 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 15:49:21 -0700 Subject: [PATCH 091/161] add xor def --- specs/core/0_beacon-chain.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a0cbb063f..5f50b67ca 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -53,6 +53,7 @@ - [`Eth1DataVote`](#eth1datavote) - [Custom Types](#custom-types) - [Helper functions](#helper-functions) + - [`xor`](#xor) - [`hash`](#hash) - [`hash_tree_root`](#hash_tree_root) - [`signed_root`](#signed_root) @@ -652,6 +653,13 @@ We define the following Python custom types for type hinting and readability: Note: The definitions below are for specification purposes and are not necessarily optimal implementations. +### `xor` + +```python +def xor(bytes1: Bytes32, bytes2: Bytes32) -> Bytes32: + return bytes(a ^ b for a, b in zip(bytes1, bytes2)) +``` + ### `hash` The hash function is denoted by `hash`. In Phase 0 the beacon chain is deployed with the same hash function as Ethereum 1.0, i.e. Keccak-256 (also incorrectly known as SHA3). From 8802f57790df8d2fe6e6a149d0cc75c763a76665 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 17:11:38 -0700 Subject: [PATCH 092/161] add missing var to get_attesting_balance calls --- specs/core/0_beacon-chain.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 5f50b67ca..d3ca1c96c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2094,12 +2094,12 @@ def update_justification_and_finalization(state: BeaconState) -> None: # Rotate the justification bitfield up one epoch to make room for the current epoch state.justification_bitfield <<= 1 # If the previous epoch gets justified, fill the second last bit - previous_boundary_attesting_balance = get_attesting_balance(get_previous_epoch_boundary_attestations(state)) + previous_boundary_attesting_balance = get_attesting_balance(state, get_previous_epoch_boundary_attestations(state)) if previous_boundary_attesting_balance * 3 >= get_previous_total_balance(state) * 2: new_justified_epoch = get_current_epoch(state) - 1 state.justification_bitfield |= 2 # If the current epoch gets justified, fill the last bit - current_boundary_attesting_balance = get_attesting_balance(get_current_epoch_boundary_attestations(state)) + current_boundary_attesting_balance = get_attesting_balance(state, get_current_epoch_boundary_attestations(state)) if current_boundary_attesting_balance * 3 >= get_current_total_balance(state) * 2: new_justified_epoch = get_current_epoch(state) state.justification_bitfield |= 1 @@ -2203,11 +2203,11 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> ] # Some helper variables boundary_attestations = get_previous_epoch_boundary_attestations(state) - boundary_attesting_balance = get_attesting_balance(boundary_attestations) + boundary_attesting_balance = get_attesting_balance(state, boundary_attestations) total_balance = get_previous_total_balance(state) - total_attesting_balance = get_attesting_balance(get_previous_epoch_attestations(state)) + total_attesting_balance = get_attesting_balance(state, get_previous_epoch_attestations(state)) matching_head_attestations = get_previous_epoch_matching_head_attestations(state) - matching_head_balance = get_attesting_balance(matching_head_attestations) + matching_head_balance = get_attesting_balance(state, matching_head_attestations) # Process rewards or penalties for all validators for index in get_active_validator_indices(state.validator_registry, previous_epoch): # Expected FFG source From d4a4c735996f2903fccb35ab9be0fbb2a28c216f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 17:17:31 -0700 Subject: [PATCH 093/161] fix calls to get_attesting_indices --- specs/core/0_beacon-chain.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index d3ca1c96c..8e9cfb6fb 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2020,7 +2020,7 @@ def get_attesting_indices(state: BeaconState, attestations: List[PendingAttestat ```python def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: - return get_total_balance(state, get_attesting_indices(attestations, state)) + return get_total_balance(state, get_attesting_indices(state, attestations)) ``` ```python @@ -2062,7 +2062,7 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple winning_root = max(all_roots, key=lambda r: get_attesting_balance(state, get_attestations_for(r))) - return winning_root, get_attesting_indices(get_attestations_for(winning_root)) + return winning_root, get_attesting_indices(state, get_attestations_for(winning_root)) ``` ```python @@ -2211,7 +2211,7 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> # Process rewards or penalties for all validators for index in get_active_validator_indices(state.validator_registry, previous_epoch): # Expected FFG source - if index in get_attesting_indices(state.previous_epoch_attestations): + if index in get_attesting_indices(state, state.previous_epoch_attestations): deltas[0][index] += get_base_reward(state, index) * total_attesting_balance // total_balance # Inclusion speed bonus deltas[0][index] += ( @@ -2221,12 +2221,12 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> else: deltas[1][index] += get_base_reward(state, index) # Expected FFG target - if index in get_attesting_indices(boundary_attestations): + if index in get_attesting_indices(state, boundary_attestations): deltas[0][index] += get_base_reward(state, index) * boundary_attesting_balance // total_balance else: deltas[1][index] += get_base_reward(state, index) # Expected head - if index in get_attesting_indices(matching_head_attestations): + if index in get_attesting_indices(state, matching_head_attestations): deltas[0][index] += get_base_reward(state, index) * matching_head_balance // total_balance else: deltas[1][index] += get_base_reward(state, index) @@ -2250,7 +2250,7 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List active_validator_indices = get_active_validator_indices(state.validator_registry, get_previous_epoch(state)) epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch for index in active_validator_indices: - if index not in get_attesting_indices(get_previous_epoch_attestations(state)): + if index not in get_attesting_indices(state, get_previous_epoch_attestations(state)): deltas[1][index] += get_inactivity_penalty(state, index, epochs_since_finality) else: # If a validator did attest, apply a small penalty for getting attestations included late @@ -2259,9 +2259,9 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List inclusion_distance(state, index) ) deltas[1][index] += base_reward(state, index) - if index not in get_attesting_indices(boundary_attestations): + if index not in get_attesting_indices(state, boundary_attestations): deltas[1][index] += get_inactivity_penalty(state, index, epochs_since_finality) - if index not in get_attesting_indices(matching_head_attestations): + if index not in get_attesting_indices(state, matching_head_attestations): deltas[1][index] += get_base_reward(state, index) # Penalize slashed-but-inactive validators as though they were active but offline for index in range(len(state.validator_registry)): From 66105b01e4169509af2688aab755190f05cd7208 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 17:27:53 -0700 Subject: [PATCH 094/161] previous epoch uses -1 in all cases --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 8e9cfb6fb..4525c609b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -707,7 +707,7 @@ def get_previous_epoch(state: BeaconState) -> Epoch: """` Return the previous epoch of the given ``state``. """ - return max(get_current_epoch(state) - 1, GENESIS_EPOCH) + return get_current_epoch(state) - 1 ``` ### `get_current_epoch` From 47cca60148dbf74af01ef491e7b58fbac0241b75 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 17:56:50 -0700 Subject: [PATCH 095/161] handle when no attestations availble for crosslinking (#717) --- specs/core/0_beacon-chain.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4525c609b..4c608c0b3 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2056,12 +2056,16 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple a for a in all_attestations if a.data.latest_crosslink == state.latest_crosslinks[shard] ] all_roots = [a.data.crosslink_data_root for a in valid_attestations] - + + # handle when no attestations for shard available + if len(all_roots) == 0: + return ZERO_HASH, [] + def get_attestations_for(root) -> List[PendingAttestation]: return [a for a in valid_attestations if a.data.crosslink_data_root == root] - + winning_root = max(all_roots, key=lambda r: get_attesting_balance(state, get_attestations_for(r))) - + return winning_root, get_attesting_indices(state, get_attestations_for(winning_root)) ``` From 738acf0e05cb85b2aa4f43c2d402d05ac0ef449f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 18:00:38 -0700 Subject: [PATCH 096/161] fix calls for get_total_balance --- specs/core/0_beacon-chain.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4525c609b..e69fcf458 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2137,7 +2137,9 @@ def process_crosslinks(state: BeaconState) -> None: for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) - if 3 * total_balance(participants) >= 2 * total_balance(crosslink_committee): + participating_balance = get_total_balance(state, participants) + total_balance = get_total_balance(state, crosslink_committee) + if 3 * participating_balance >= 2 * total_balance: state.latest_crosslinks[shard] = Crosslink( epoch=slot_to_epoch(slot), crosslink_data_root=winning_root @@ -2434,7 +2436,7 @@ def process_slashings(state: BeaconState) -> None: """ current_epoch = get_current_epoch(state) active_validator_indices = get_active_validator_indices(state.validator_registry, current_epoch) - total_balance = sum(get_effective_balance(state, i) for i in active_validator_indices) + total_balance = get_total_balance(state, active_validator_indices) # Compute `total_penalties` epoch_index = current_epoch % LATEST_SLASHED_EXIT_LENGTH From 0fdfbc473e15fbc318ce2e9222f2982ea6db9211 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 18:53:31 -0700 Subject: [PATCH 097/161] minor variable issues in helpers --- specs/core/0_beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 8c28adbf6..0e836df1f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2211,11 +2211,11 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> boundary_attestations = get_previous_epoch_boundary_attestations(state) boundary_attesting_balance = get_attesting_balance(state, boundary_attestations) total_balance = get_previous_total_balance(state) - total_attesting_balance = get_attesting_balance(state, get_previous_epoch_attestations(state)) + total_attesting_balance = get_attesting_balance(state, state.previous_epoch_attestations) matching_head_attestations = get_previous_epoch_matching_head_attestations(state) matching_head_balance = get_attesting_balance(state, matching_head_attestations) # Process rewards or penalties for all validators - for index in get_active_validator_indices(state.validator_registry, previous_epoch): + for index in get_active_validator_indices(state.validator_registry, get_previous_epoch(state)): # Expected FFG source if index in get_attesting_indices(state, state.previous_epoch_attestations): deltas[0][index] += get_base_reward(state, index) * total_attesting_balance // total_balance @@ -2256,7 +2256,7 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List active_validator_indices = get_active_validator_indices(state.validator_registry, get_previous_epoch(state)) epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch for index in active_validator_indices: - if index not in get_attesting_indices(state, get_previous_epoch_attestations(state)): + if index not in get_attesting_indices(state, state.previous_epoch_attestations): deltas[1][index] += get_inactivity_penalty(state, index, epochs_since_finality) else: # If a validator did attest, apply a small penalty for getting attestations included late @@ -2295,7 +2295,7 @@ def get_crosslink_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) participating_balance = get_total_balance(state, participants) - total_balance = get_total_balance(state, committee) + total_balance = get_total_balance(state, crossling_committee) for index in crosslink_committee: if index in participants: deltas[0][index] += get_base_reward(state, index) * participating_balance // total_balance From 02e8e8974049f1965df3f7b0b9b874a46a26c150 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 19:05:00 -0700 Subject: [PATCH 098/161] handle get_base_reward when no previous balance --- specs/core/0_beacon-chain.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0e836df1f..13eae0432 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2171,6 +2171,9 @@ First, we define some additional helpers: ```python def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: + if get_previous_total_balance(state) == 0: + return 0 + adjusted_quotient = integer_squareroot(get_previous_total_balance(state)) // BASE_REWARD_QUOTIENT return get_effective_balance(state, index) // adjusted_quotient // 5 ``` From bd936fa6abc11454b769e7053c9052477d7c1dc8 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 19:07:24 -0700 Subject: [PATCH 099/161] add missing return for justification/finalization reward deltas --- specs/core/0_beacon-chain.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0e836df1f..b44200226 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2239,6 +2239,7 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> # Proposer bonus proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) deltas[0][proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT + return deltas ``` When blocks are not finalizing normally... From d2f7fa9a7eec82b7210cf7b935f39aa003c78d81 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 4 Mar 2019 19:09:31 -0700 Subject: [PATCH 100/161] fix call to get_current_epoch --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b44200226..151fa6273 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2330,7 +2330,7 @@ def process_ejections(state: BeaconState) -> None: Iterate through the validator registry and eject active validators with balance below ``EJECTION_BALANCE``. """ - for index in get_active_validator_indices(state.validator_registry, current_epoch(state)): + for index in get_active_validator_indices(state.validator_registry, get_current_epoch(state)): if state.validator_balances[index] < EJECTION_BALANCE: exit_validator(state, index) ``` From ecb76420ecd124c39849aef478d71165633acc68 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 5 Mar 2019 17:30:42 +0800 Subject: [PATCH 101/161] `1e9` is float in Python, change it to `10**9`. --- specs/core/0_beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 151fa6273..81a757db2 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -205,10 +205,10 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | Name | Value | Unit | | - | - | :-: | -| `MIN_DEPOSIT_AMOUNT` | `2**0 * 1e9` (= 1,000,000,000) | Gwei | -| `MAX_DEPOSIT_AMOUNT` | `2**5 * 1e9` (= 32,000,000,000) | Gwei | -| `FORK_CHOICE_BALANCE_INCREMENT` | `2**0 * 1e9` (= 1,000,000,000) | Gwei | -| `EJECTION_BALANCE` | `2**4 * 1e9` (= 16,000,000,000) | Gwei | +| `MIN_DEPOSIT_AMOUNT` | `2**0 * 10**9` (= 1,000,000,000) | Gwei | +| `MAX_DEPOSIT_AMOUNT` | `2**5 * 10**9` (= 32,000,000,000) | Gwei | +| `FORK_CHOICE_BALANCE_INCREMENT` | `2**0 * 10**9` (= 1,000,000,000) | Gwei | +| `EJECTION_BALANCE` | `2**4 * 10**9` (= 16,000,000,000) | Gwei | ### Initial values From e2a045210827a80e57e1cc2815bb183c48d3950b Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 5 Mar 2019 06:07:00 -0700 Subject: [PATCH 102/161] fix previous root (#716) --- specs/core/0_beacon-chain.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 81a757db2..ff1ec1ea1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1693,10 +1693,10 @@ At every `slot > GENESIS_SLOT` run the following function: ```python def advance_slot(state: BeaconState) -> None: state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) - state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = get_block_root(state, state.slot - 1) - state.slot += 1 if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = get_state_root(state, state.slot - 1) + state.latest_block_header.state_root = get_state_root(state, state.slot) + state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) + state.slot += 1 ``` ### Per-block processing @@ -1711,8 +1711,6 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: assert block.slot == state.slot # Verify that the parent matches assert block.previous_block_root == hash_tree_root(state.latest_block_header) - # Save previous block root - state.latest_block_roots[(state.slot - 1) % SLOTS_PER_HISTORICAL_ROOT] = block.previous_block_root # Save current block as the new latest block state.latest_block_header = get_temporary_block_header(block) # Verify proposer signature From 57971aacb42a406c7c547a754e4682d727383550 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 5 Mar 2019 15:20:36 +0100 Subject: [PATCH 103/161] Update simple-serialize.md --- specs/simple-serialize.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 31839b27e..a452213b5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,8 +12,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`uintN`](#uintn) - [`bool`](#bool) - - [Tuples](#lists) - - [Containers, lists](#containers-lists) + - [Tuples, containers, lists](#tuples-containers-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -69,13 +68,15 @@ assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -### Tuples +### Tuples, containers, lists + +If `value` is fixed-length (i.e. does not embed a list): ```python return ''.join([serialize(element) for element in value]) ``` -### Containers, lists +If `value` is variable-length (i.e. embeds a list): ```python serialized_bytes = ''.join([serialize(element) for element in value]) From 99da6fe14148fb21ed55ef2eaa0ec33256ed5809 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 5 Mar 2019 08:50:51 -0700 Subject: [PATCH 104/161] fix advance_slot ordering issue --- specs/core/0_beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ff1ec1ea1..36fbad847 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1693,10 +1693,10 @@ At every `slot > GENESIS_SLOT` run the following function: ```python def advance_slot(state: BeaconState) -> None: state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) - if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = get_state_root(state, state.slot) - state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) state.slot += 1 + if state.latest_block_header.state_root == ZERO_HASH: + state.latest_block_header.state_root = get_state_root(state, state.slot - 1) + state.latest_block_roots[(state.slot - 1) % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) ``` ### Per-block processing From 0eacabc2732a6532f8b54a834ff63078484825b1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 5 Mar 2019 08:51:34 -0700 Subject: [PATCH 105/161] fix minor typo --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36fbad847..ca37fa457 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2294,7 +2294,7 @@ def get_crosslink_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): winning_root, participants = get_winning_root_and_participants(state, shard) participating_balance = get_total_balance(state, participants) - total_balance = get_total_balance(state, crossling_committee) + total_balance = get_total_balance(state, crosslink_committee) for index in crosslink_committee: if index in participants: deltas[0][index] += get_base_reward(state, index) * participating_balance // total_balance From b5f050959568b6cdedcf73ea17aceaf2168191e8 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 5 Mar 2019 11:29:40 -0700 Subject: [PATCH 106/161] fix proposal slashing minor bugs --- specs/core/0_beacon-chain.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 43a679833..6cd56231d 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1773,19 +1773,17 @@ def process_proposer_slashing(state: BeaconState, proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same assert proposer_slashing.header_1.slot == proposer_slashing.header_2.slot - # Verify that the shard is the same (or that both proposals are beacon chain proposals) - assert proposer_slashing.header_1.shard == proposer_slashing.header_2.shard # But the roots are different! - assert proposer_slashing.header_1.block_root != proposer_slashing.header_2.block_root + assert hash_tree_root(proposer_slashing.header_1) != hash_tree_root(proposer_slashing.header_2) # Proposer is not yet slashed assert proposer.slashed is False # Signatures are valid for header in (proposer_slashing.header_1, proposer_slashing.header_2): assert bls_verify( pubkey=proposer.pubkey, - message_hash=signed_root(header, "signature"), + message_hash=signed_root(header, "signature"), signature=header.signature, - domain=get_domain(state.fork, slot_to_epoch(header.slot), DOMAIN_PROPOSAL) + domain=get_domain(state.fork, slot_to_epoch(header.slot), DOMAIN_BEACON_BLOCK) ) slash_validator(state, proposer_slashing.proposer_index) ``` From b36e70040bef5b966f60b06d275e83014f046d63 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 5 Mar 2019 16:18:41 -0700 Subject: [PATCH 107/161] fix call to union --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6cd56231d..b94a74e84 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2010,7 +2010,7 @@ def get_previous_total_balance(state: BeaconState) -> Gwei: def get_attesting_indices(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: output = set() for a in attestations: - output = output.union([get_attestation_participants(state, a.data, a.aggregation_bitfield)]) + output = output.union(get_attestation_participants(state, a.data, a.aggregation_bitfield)) return sorted(list(output)) ``` From 776196e66d593cb988474549fc5b6c13fb393043 Mon Sep 17 00:00:00 2001 From: dankrad Date: Wed, 6 Mar 2019 14:46:52 +0100 Subject: [PATCH 108/161] Add tuple lengths Adding tuple lengths in BeaconState objects (this changes merkleization/serialization as no length mixin required) --- specs/core/0_beacon-chain.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6bff0f705..2771d19fa 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -522,7 +522,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git 'validator_registry_update_epoch': 'uint64', # Randomness and committees - 'latest_randao_mixes': ['bytes32'], + 'latest_randao_mixes': ['bytes32', LATEST_RANDAO_MIXES_LENGTH], 'previous_shuffling_start_shard': 'uint64', 'current_shuffling_start_shard': 'uint64', 'previous_shuffling_epoch': 'uint64', @@ -537,10 +537,10 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git 'finalized_epoch': 'uint64', # Recent state - 'latest_crosslinks': [Crosslink], - 'latest_block_roots': ['bytes32'], - 'latest_active_index_roots': ['bytes32'], - 'latest_slashed_balances': ['uint64'], # Balances slashed at every withdrawal period + 'latest_crosslinks': [Crosslink, SHARD_COUNT], + 'latest_block_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], + 'latest_active_index_roots': ['bytes32', LATEST_ACTIVE_INDEX_ROOTS_LENGTH], + 'latest_slashed_balances': ['uint64', LATEST_SLASHED_EXIT_LENGTH], # Balances slashed at every withdrawal period 'latest_attestations': [PendingAttestation], 'batched_block_roots': ['bytes32'], From c9e06d31f381640a826016340970de6b56a40115 Mon Sep 17 00:00:00 2001 From: Carl Beekhuizen Date: Wed, 6 Mar 2019 07:29:06 -0700 Subject: [PATCH 109/161] Apply suggestions from code review base_reward -> get_base_reward Co-Authored-By: djrtwo --- specs/core/0_beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b94a74e84..81af1a90e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2237,7 +2237,7 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> deltas[1][index] += get_base_reward(state, index) # Proposer bonus proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) - deltas[0][proposer_index] += base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT + deltas[0][proposer_index] += get_base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT return deltas ``` @@ -2261,10 +2261,10 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List else: # If a validator did attest, apply a small penalty for getting attestations included late deltas[0][index] += ( - base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // + get_base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // inclusion_distance(state, index) ) - deltas[1][index] += base_reward(state, index) + deltas[1][index] += get_base_reward(state, index) if index not in get_attesting_indices(state, boundary_attestations): deltas[1][index] += get_inactivity_penalty(state, index, epochs_since_finality) if index not in get_attesting_indices(state, matching_head_attestations): From b4779cd83dad62f8c2505c96e955fa9d8605bd61 Mon Sep 17 00:00:00 2001 From: Carl Beekhuizen Date: Wed, 6 Mar 2019 08:55:39 -0700 Subject: [PATCH 110/161] Apply suggestions from code review Co-Authored-By: djrtwo --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 81af1a90e..b0a7ea999 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -447,7 +447,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git # Sender index 'sender': 'uint64', # Recipient index - 'to': 'uint64', + 'recipient': 'uint64', # Amount in Gwei 'amount': 'uint64', # Fee in Gwei for block proposer @@ -1984,7 +1984,7 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: ) # Process the transfer state.validator_balances[transfer.sender] -= transfer.amount + transfer.fee - state.validator_balances[transfer.to] += transfer.amount + state.validator_balances[transfer.recipient] += transfer.amount state.validator_balances[get_beacon_proposer_index(state, state.slot)] += transfer.fee ``` From eece029cdfdf6672a929a9b9d8c960a53bd2fac1 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 6 Mar 2019 17:46:40 +0100 Subject: [PATCH 111/161] Array spec to [type] and tuple to [type, N]. Also make notation consistent: use "base" for base types --- specs/simple-serialize.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index a452213b5..bb15bb6da 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -10,8 +10,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Composite types](#composite-types) - [Aliases](#aliases) - [Serialization](#serialization) - - [`uintN`](#uintn) - - [`bool`](#bool) + - [`"uintN"`](#uintn) + - [`"bool"`](#bool) - [Tuples, containers, lists](#tuples-containers-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) @@ -28,29 +28,29 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Typing ### Basic types -* `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) -* `bool`: `True` or `False` +* `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) +* `"bool"`: `True` or `False` ### Composite types * **container**: ordered heterogenous collection of values * key-pair curly bracket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` * **tuple**: ordered fixed-length homogeneous collection of values - * angle bracket notation `[N]`, e.g. `uint64[N]` + * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values - * angle bracket notation `[]`, e.g. `uint64[]` + * angle bracket notation `[type]`, e.g. `["uint64"]` ### Aliases For convenience we alias: -* `byte` to `uint8` -* `bytes` to `byte[]` -* `bytesN` to `byte[N]` +* `"byte"` to `"uint8"` +* `"bytes"` to `["byte"]` +* `"bytesN"` to `["byte", N]` ## Serialization -We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `bytes`. +We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type. @@ -95,7 +95,7 @@ We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. * `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. -* `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. +* `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -106,7 +106,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi ## Self-signed containers -Let `value` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signed_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`. +Let `value` be a self-signed container object. The convention is that the signature (e.g. a `"bytes96"` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signed_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`. ## Implementations From f4e0d9804a18c8e09315fa92effe1be75668ca21 Mon Sep 17 00:00:00 2001 From: Akhila Raju Date: Tue, 5 Mar 2019 20:14:00 -0500 Subject: [PATCH 112/161] Minor spelling fixes. --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index e1083c5c5..0c95fb446 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -379,7 +379,7 @@ def get_committee_assignment( ### Lookahead -The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming assignemnts of proposing and attesting dictated by the shuffling and slot. +The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming assignments of proposing and attesting dictated by the shuffling and slot. There are three possibilities for the shuffling at the next epoch: 1. The shuffling changes due to a "validator registry change". From 81f48ea1c3bc16c5b48429da7eb1dce57463b095 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 6 Mar 2019 21:11:50 +0100 Subject: [PATCH 113/161] Also fix length of Deposit.proof --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2031587cf..b50f4c36a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -387,7 +387,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git ```python { # Branch in the deposit tree - 'proof': ['bytes32'], + 'proof': ['bytes32', DEPOSIT_CONTRACT_TREE_DEPTH], # Index in the deposit tree 'index': 'uint64', # Data From 37b77725d88da0bd17e4d66e7def30249fd4253c Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 6 Mar 2019 22:32:26 +0100 Subject: [PATCH 114/161] Make basic/non-basic for bytes explicit --- specs/simple-serialize.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index a452213b5..a169f8f93 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -44,9 +44,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio For convenience we alias: -* `byte` to `uint8` -* `bytes` to `byte[]` -* `bytesN` to `byte[N]` +* `byte` to `uint8` (this is a basic type) +* `bytes` to `byte[]` (this is *not* a basic type) +* `bytesN` to `byte[N]` (this is *not* a basic type) ## Serialization From ee2fbd0df7582790876d1051417932d6718ea793 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 6 Mar 2019 13:48:06 -0800 Subject: [PATCH 115/161] Fix calls of `signed_root` --- specs/core/0_beacon-chain.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index e09e7c261..239f262df 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1292,7 +1292,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: # Verify the proof of possession proof_is_valid = bls_verify( pubkey=deposit_input.pubkey, - message_hash=signed_root(deposit_input, "proof_of_possession"), + message_hash=signed_root(deposit_input), signature=deposit_input.proof_of_possession, domain=get_domain( state.fork, @@ -1664,8 +1664,8 @@ Below are the processing steps that happen at every `block`. #### Block signature * Let `proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)]`. -* Let `proposal = Proposal(block.slot, BEACON_CHAIN_SHARD_NUMBER, signed_root(block, "signature"), block.signature)`. -* Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposal, "signature"), signature=proposal.signature, domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_PROPOSAL))`. +* Let `proposal = Proposal(block.slot, BEACON_CHAIN_SHARD_NUMBER, signed_root(block), block.signature)`. +* Verify that `bls_verify(pubkey=proposer.pubkey, message_hash=signed_root(proposal), signature=proposal.signature, domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_PROPOSAL))`. #### RANDAO @@ -1705,7 +1705,7 @@ def process_proposer_slashing(state: BeaconState, for proposal in (proposer_slashing.proposal_1, proposer_slashing.proposal_2): assert bls_verify( pubkey=proposer.pubkey, - message_hash=signed_root(proposal, "signature"), + message_hash=signed_root(proposal), signature=proposal.signature, domain=get_domain(state.fork, slot_to_epoch(proposal.slot), DOMAIN_PROPOSAL) ) @@ -1853,7 +1853,7 @@ def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: # Verify signature assert bls_verify( pubkey=validator.pubkey, - message_hash=signed_root(exit, "signature"), + message_hash=signed_root(exit), signature=exit.signature, domain=get_domain(state.fork, exit.epoch, DOMAIN_EXIT) ) @@ -1898,7 +1898,7 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: # Verify that the signature is valid assert bls_verify( pubkey=transfer.pubkey, - message_hash=signed_root(transfer, "signature"), + message_hash=signed_root(transfer), signature=transfer.signature, domain=get_domain(state.fork, slot_to_epoch(transfer.slot), DOMAIN_TRANSFER) ) From 663bc489b60ea89708e6115763ca185628eb3edc Mon Sep 17 00:00:00 2001 From: vbuterin Date: Wed, 6 Mar 2019 22:54:52 -0600 Subject: [PATCH 116/161] Added lexicographic tiebreaking --- specs/core/0_beacon-chain.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 081e877c3..49f8ba3e9 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2060,7 +2060,9 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple def get_attestations_for(root) -> List[PendingAttestation]: return [a for a in valid_attestations if a.data.crosslink_data_root == root] - winning_root = max(all_roots, key=lambda r: get_attesting_balance(state, get_attestations_for(r))) + # Winning crosslink root is the root with the most votes for it, ties broken in favor of + # lexicographically higher hash + winning_root = max(all_roots, key=lambda r: (get_attesting_balance(state, get_attestations_for(r)), r)) return winning_root, get_attesting_indices(state, get_attestations_for(winning_root)) ``` From ec12460b8bae49c2272870b4490039a68f3c4761 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Thu, 7 Mar 2019 01:21:17 -0600 Subject: [PATCH 117/161] Make wthdrawn validators immune to inactivity leak --- specs/core/0_beacon-chain.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 49f8ba3e9..bd709218a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2273,7 +2273,12 @@ def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List deltas[1][index] += get_base_reward(state, index) # Penalize slashed-but-inactive validators as though they were active but offline for index in range(len(state.validator_registry)): - if index not in active_validator_indices and state.validator_registry[index].slashed: + eligible = ( + index not in active_validator_indices and + state.validator_registry[index].slashed and + get_current_epoch(state) < state.validator_registry[index].withdrawable_epoch + ) + if eligible: deltas[1][index] += ( 2 * get_inactivity_penalty(state, index, epochs_since_finality) + get_base_reward(state, index) From 987c741bea582fce0839bc2ad2e14abe50a15d66 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 7 Mar 2019 09:44:55 -0700 Subject: [PATCH 118/161] fix state types --- specs/core/0_beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index d4739173e..796305953 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -543,8 +543,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git 'latest_active_index_roots': ['bytes32', LATEST_ACTIVE_INDEX_ROOTS_LENGTH], 'latest_slashed_balances': ['uint64', LATEST_SLASHED_EXIT_LENGTH], # Balances slashed at every withdrawal period 'latest_block_header': BeaconBlockHeader, # `latest_block_header.state_root == ZERO_HASH` temporarily - 'latest_attestations': [PendingAttestation], - 'batched_block_roots': ['bytes32'], + 'historical_roots': ['bytes32'], # Ethereum 1.0 chain data 'latest_eth1_data': Eth1Data, From 250455a67e18e51abe5566a0ea129873922c7427 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 7 Mar 2019 10:03:25 -0700 Subject: [PATCH 119/161] Apply suggestions from code review Co-Authored-By: djrtwo --- specs/core/0_beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 796305953..002770913 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1899,7 +1899,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: data=attestation.data, aggregation_bitfield=attestation.aggregation_bitfield, custody_bitfield=attestation.custody_bitfield, - inclusion_slot=state.slot + inclusion_slot=state.slot, ) if slot_to_epoch(attestation.data.slot) == get_current_epoch(state): state.current_epoch_attestations.append(pending_attestation) @@ -2057,7 +2057,7 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple if len(all_roots) == 0: return ZERO_HASH, [] - def get_attestations_for(root) -> List[PendingAttestation]: + def get_attestations_for(root: Bytes32) -> List[PendingAttestation]: return [a for a in valid_attestations if a.data.crosslink_data_root == root] # Winning crosslink root is the root with the most votes for it, ties broken in favor of @@ -2134,7 +2134,7 @@ Run the following function: ```python def process_crosslinks(state: BeaconState) -> None: current_epoch = get_current_epoch(state) - previous_epoch = current_epoch - 1 + previous_epoch = get_previous_epoch(state) next_epoch = current_epoch + 1 for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): @@ -2144,7 +2144,7 @@ def process_crosslinks(state: BeaconState) -> None: if 3 * participating_balance >= 2 * total_balance: state.latest_crosslinks[shard] = Crosslink( epoch=slot_to_epoch(slot), - crosslink_data_root=winning_root + crosslink_data_root=winning_root, ) ``` From 8dec7d805cf7673f77b762522532fe15aa944d0c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 7 Mar 2019 11:02:13 -0700 Subject: [PATCH 120/161] reorder ssz types topologically --- specs/core/0_beacon-chain.md | 395 +++++++++++++++++------------------ 1 file changed, 190 insertions(+), 205 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 002770913..898fc3c86 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -21,36 +21,31 @@ - [Max transactions per block](#max-transactions-per-block) - [Signature domains](#signature-domains) - [Data structures](#data-structures) - - [Beacon chain transactions](#beacon-chain-transactions) - - [Proposer slashings](#proposer-slashings) - - [`ProposerSlashing`](#proposerslashing) - - [Attester slashings](#attester-slashings) - - [`AttesterSlashing`](#attesterslashing) - - [`SlashableAttestation`](#slashableattestation) - - [Attestations](#attestations) - - [`Attestation`](#attestation) - - [`AttestationData`](#attestationdata) - - [`AttestationDataAndCustodyBit`](#attestationdataandcustodybit) - - [Deposits](#deposits) - - [`Deposit`](#deposit) - - [`DepositData`](#depositdata) - - [`DepositInput`](#depositinput) - - [Voluntary exits](#voluntary-exits) - - [`VoluntaryExit`](#voluntaryexit) - - [Transfers](#transfers) - - [`Transfer`](#transfer) - - [Beacon chain blocks](#beacon-chain-blocks) - - [`BeaconBlock`](#beaconblock) - - [`BeaconBlockHeader`](#beaconblockheader) - - [`BeaconBlockBody`](#beaconblockbody) - - [Beacon chain state](#beacon-chain-state) - - [`BeaconState`](#beaconstate) - - [`Validator`](#validator) - - [`Crosslink`](#crosslink) - - [`PendingAttestation`](#pendingattestation) + - [Misc dependencies](#misc-dependencies) - [`Fork`](#fork) + - [`Crosslink`](#crosslink) - [`Eth1Data`](#eth1data) - [`Eth1DataVote`](#eth1datavote) + - [`AttestationData`](#attestationdata) + - [`AttestationDataAndCustodyBit`](#attestationdataandcustodybit) + - [`SlashableAttestation`](#slashableattestation) + - [`DepositInput`](#depositinput) + - [`DepositData`](#depositdata) + - [`BeaconBlockHeader`](#beaconblockheader) + - [`Validator`](#validator) + - [`PendingAttestation`](#pendingattestation) + - [Beacon transactions](#beacon-transactions) + - [`ProposerSlashing`](#proposerslashing) + - [`AttesterSlashing`](#attesterslashing) + - [`Attestation`](#attestation) + - [`Deposit`](#deposit) + - [`VoluntaryExit`](#voluntaryexit) + - [`Transfer`](#transfer) + - [Beacon blocks](#beacon-blocks) + - [`BeaconBlockBody`](#beaconblockbody) + - [`BeaconBlock`](#beaconblock) + - [Beacon state](#beacon-state) + - [`BeaconState`](#beaconstate) - [Custom Types](#custom-types) - [Helper functions](#helper-functions) - [`xor`](#xor) @@ -116,8 +111,6 @@ - [Beacon chain fork choice rule](#beacon-chain-fork-choice-rule) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - [Per-slot processing](#per-slot-processing) - - [Slot](#slot) - - [Block roots](#block-roots) - [Per-block processing](#per-block-processing) - [Block header](#block-header) - [RANDAO](#randao) @@ -136,8 +129,8 @@ - [Eth1 data](#eth1-data-1) - [Rewards and penalties](#rewards-and-penalties) - [Justification and finalization](#justification-and-finalization) - - [Attestation inclusion](#attestation-inclusion) - [Crosslinks](#crosslinks-1) + - [Apply rewards](#apply-rewards) - [Ejections](#ejections) - [Validator registry and shuffling seed data](#validator-registry-and-shuffling-seed-data) - [Slashings and exit queue](#slashings-and-exit-queue) @@ -286,69 +279,57 @@ Code snippets appearing in `this style` are to be interpreted as Python code. The following data structures are defined as [SimpleSerialize (SSZ)](https://github.com/ethereum/eth2.0-specs/blob/master/specs/simple-serialize.md) objects. -### Beacon chain transactions +The types are defined topologically to aid in facilitating an executable version of the spec. -#### Proposer slashings +### Misc dependencies -##### `ProposerSlashing` +#### `Fork` ```python { - # Proposer index - 'proposer_index': 'uint64', - # First block header - 'header_1': BeaconBlockHeader, - # Second block header - 'header_2': BeaconBlockHeader, + # Previous fork version + 'previous_version': 'uint64', + # Current fork version + 'current_version': 'uint64', + # Fork epoch number + 'epoch': 'uint64', } ``` -#### Attester slashings - -##### `AttesterSlashing` +#### `Crosslink` ```python { - # First slashable attestation - 'slashable_attestation_1': SlashableAttestation, - # Second slashable attestation - 'slashable_attestation_2': SlashableAttestation, + # Epoch number + 'epoch': 'uint64', + # Shard data since the previous crosslink + 'crosslink_data_root': 'bytes32', } ``` -##### `SlashableAttestation` +#### `Eth1Data` ```python { - # Validator indices - 'validator_indices': ['uint64'], - # Attestation data - 'data': AttestationData, - # Custody bitfield - 'custody_bitfield': 'bytes', - # Aggregate signature - 'aggregate_signature': 'bytes96', + # Root of the deposit tree + 'deposit_root': 'bytes32', + # Block hash + 'block_hash': 'bytes32', } ``` -#### Attestations - -##### `Attestation` +#### `Eth1DataVote` ```python { - # Attester aggregation bitfield - 'aggregation_bitfield': 'bytes', - # Attestation data - 'data': AttestationData, - # Custody bitfield - 'custody_bitfield': 'bytes', - # BLS aggregate signature - 'aggregate_signature': 'bytes96', + # Data being voted for + 'eth1_data': Eth1Data, + # Vote count + 'vote_count': 'uint64', } ``` -##### `AttestationData` +#### `AttestationData` ```python { @@ -371,7 +352,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -##### `AttestationDataAndCustodyBit` +#### `AttestationDataAndCustodyBit` ```python { @@ -382,35 +363,22 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -#### Deposits - -##### `Deposit` +#### `SlashableAttestation` ```python { - # Branch in the deposit tree - 'proof': ['bytes32', DEPOSIT_CONTRACT_TREE_DEPTH], - # Index in the deposit tree - 'index': 'uint64', - # Data - 'deposit_data': DepositData, + # Validator indices + 'validator_indices': ['uint64'], + # Attestation data + 'data': AttestationData, + # Custody bitfield + 'custody_bitfield': 'bytes', + # Aggregate signature + 'aggregate_signature': 'bytes96', } ``` -##### `DepositData` - -```python -{ - # Amount in Gwei - 'amount': 'uint64', - # Timestamp from deposit contract - 'timestamp': 'uint64', - # Deposit input - 'deposit_input': DepositInput, -} -``` - -##### `DepositInput` +#### `DepositInput` ```python { @@ -423,9 +391,122 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -#### Voluntary exits +#### `DepositData` -##### `VoluntaryExit` +```python +{ + # Amount in Gwei + 'amount': 'uint64', + # Timestamp from deposit contract + 'timestamp': 'uint64', + # Deposit input + 'deposit_input': DepositInput, +} +``` + +#### `BeaconBlockHeader` + +```python +{ + 'slot': 'uint64', + 'previous_block_root': 'bytes32', + 'state_root': 'bytes32', + 'block_body_root': 'bytes32', + 'signature': 'bytes96', +} +``` + +#### `Validator` + +```python +{ + # BLS public key + 'pubkey': 'bytes48', + # Withdrawal credentials + 'withdrawal_credentials': 'bytes32', + # Epoch when validator activated + 'activation_epoch': 'uint64', + # Epoch when validator exited + 'exit_epoch': 'uint64', + # Epoch when validator is eligible to withdraw + 'withdrawable_epoch': 'uint64', + # Did the validator initiate an exit + 'initiated_exit': 'bool', + # Was the validator slashed + 'slashed': 'bool', +} +``` + +#### `PendingAttestation` + +```python +{ + # Attester aggregation bitfield + 'aggregation_bitfield': 'bytes', + # Attestation data + 'data': AttestationData, + # Custody bitfield + 'custody_bitfield': 'bytes', + # Inclusion slot + 'inclusion_slot': 'uint64', +} +``` + +### Beacon transactions + +#### `ProposerSlashing` + +```python +{ + # Proposer index + 'proposer_index': 'uint64', + # First block header + 'header_1': BeaconBlockHeader, + # Second block header + 'header_2': BeaconBlockHeader, +} +``` + +#### `AttesterSlashing` + +```python +{ + # First slashable attestation + 'slashable_attestation_1': SlashableAttestation, + # Second slashable attestation + 'slashable_attestation_2': SlashableAttestation, +} +``` + +#### `Attestation` + +```python +{ + # Attester aggregation bitfield + 'aggregation_bitfield': 'bytes', + # Attestation data + 'data': AttestationData, + # Custody bitfield + 'custody_bitfield': 'bytes', + # BLS aggregate signature + 'aggregate_signature': 'bytes96', +} +``` + +#### `Deposit` + +```python +{ + # Branch in the deposit tree + 'proof': ['bytes32', DEPOSIT_CONTRACT_TREE_DEPTH], + # Index in the deposit tree + 'index': 'uint64', + # Data + 'deposit_data': DepositData, +} +``` + +#### `VoluntaryExit` ```python { @@ -438,9 +519,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -#### Transfers - -##### `Transfer` +#### `Transfer` ```python { @@ -461,32 +540,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -### Beacon chain blocks - -#### `BeaconBlock` - -```python -{ - # Header - 'slot': 'uint64', - 'previous_block_root': 'bytes32', - 'state_root': 'bytes32', - 'body': BeaconBlockBody, - 'signature': 'bytes96', -} -``` - -#### `BeaconBlockHeader` - -```python -{ - 'slot': 'uint64', - 'previous_block_root': 'bytes32', - 'state_root': 'bytes32', - 'block_body_root': 'bytes32', - 'signature': 'bytes96', -} -``` +### Beacon blocks #### `BeaconBlockBody` @@ -503,7 +557,20 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -### Beacon chain state +#### `BeaconBlock` + +```python +{ + # Header + 'slot': 'uint64', + 'previous_block_root': 'bytes32', + 'state_root': 'bytes32', + 'body': BeaconBlockBody, + 'signature': 'bytes96', +} +``` + +### Beacon state #### `BeaconState` @@ -552,88 +619,6 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git } ``` -#### `Validator` - -```python -{ - # BLS public key - 'pubkey': 'bytes48', - # Withdrawal credentials - 'withdrawal_credentials': 'bytes32', - # Epoch when validator activated - 'activation_epoch': 'uint64', - # Epoch when validator exited - 'exit_epoch': 'uint64', - # Epoch when validator is eligible to withdraw - 'withdrawable_epoch': 'uint64', - # Did the validator initiate an exit - 'initiated_exit': 'bool', - # Was the validator slashed - 'slashed': 'bool', -} -``` - -#### `Crosslink` - -```python -{ - # Epoch number - 'epoch': 'uint64', - # Shard data since the previous crosslink - 'crosslink_data_root': 'bytes32', -} -``` - -#### `PendingAttestation` - -```python -{ - # Attester aggregation bitfield - 'aggregation_bitfield': 'bytes', - # Attestation data - 'data': AttestationData, - # Custody bitfield - 'custody_bitfield': 'bytes', - # Inclusion slot - 'inclusion_slot': 'uint64', -} -``` - -#### `Fork` - -```python -{ - # Previous fork version - 'previous_version': 'uint64', - # Current fork version - 'current_version': 'uint64', - # Fork epoch number - 'epoch': 'uint64', -} -``` - -#### `Eth1Data` - -```python -{ - # Root of the deposit tree - 'deposit_root': 'bytes32', - # Block hash - 'block_hash': 'bytes32', -} -``` - -#### `Eth1DataVote` - -```python -{ - # Data being voted for - 'eth1_data': Eth1Data, - # Vote count - 'vote_count': 'uint64', -} -``` - ## Custom Types We define the following Python custom types for type hinting and readability: From 6cf14884a83474ef164bb482f4d976dca818c08c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 7 Mar 2019 12:05:34 -0700 Subject: [PATCH 121/161] epoch transition at start of epoch --- specs/core/0_beacon-chain.md | 687 ++++++++++++++++++----------------- 1 file changed, 351 insertions(+), 336 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 002770913..281acfa79 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -115,9 +115,20 @@ - [Beacon chain processing](#beacon-chain-processing) - [Beacon chain fork choice rule](#beacon-chain-fork-choice-rule) - [Beacon chain state transition function](#beacon-chain-state-transition-function) + - [Per-epoch processing](#per-epoch-processing) + - [Helper functions](#helper-functions-1) + - [Justification](#justification) + - [Crosslinks](#crosslinks) + - [Eth1 data](#eth1-data-1) + - [Rewards and penalties](#rewards-and-penalties) + - [Justification and finalization](#justification-and-finalization) + - [Crosslinks](#crosslinks-1) + - [Apply rewards](#apply-rewards) + - [Ejections](#ejections) + - [Validator registry and shuffling seed data](#validator-registry-and-shuffling-seed-data) + - [Slashings and exit queue](#slashings-and-exit-queue) + - [Final updates](#final-updates) - [Per-slot processing](#per-slot-processing) - - [Slot](#slot) - - [Block roots](#block-roots) - [Per-block processing](#per-block-processing) - [Block header](#block-header) - [RANDAO](#randao) @@ -129,20 +140,7 @@ - [Deposits](#deposits-1) - [Voluntary exits](#voluntary-exits-1) - [Transfers](#transfers-1) - - [Per-epoch processing](#per-epoch-processing) - - [Helper functions](#helper-functions-1) - - [Justification](#justification) - - [Crosslinks](#crosslinks) - - [Eth1 data](#eth1-data-1) - - [Rewards and penalties](#rewards-and-penalties) - - [Justification and finalization](#justification-and-finalization) - - [Attestation inclusion](#attestation-inclusion) - - [Crosslinks](#crosslinks-1) - - [Ejections](#ejections) - - [Validator registry and shuffling seed data](#validator-registry-and-shuffling-seed-data) - - [Slashings and exit queue](#slashings-and-exit-queue) - - [Final updates](#final-updates) - - [State root verification](#state-root-verification) + - [State root verification](#state-root-verification) - [References](#references) - [Normative](#normative) - [Informative](#informative) @@ -1674,327 +1672,38 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock) ## Beacon chain state transition function -We now define the state transition function. At a high level the state transition is made up of three parts: +We now define the state transition function. At a high level the state transition is made up of four parts: -1. The per-slot transitions, which happens at the start of every slot. -2. The per-block transitions, which happens at every block. -3. The per-epoch transitions, which happens at the end of the last slot of every epoch (i.e. `(state.slot + 1) % SLOTS_PER_EPOCH == 0`). +1. State-root caching, which happens at the start of every slot. +2. The per-epoch transitions, which happens at the start of the first slot of every epoch. +3. The per-slot transitions, which happens at every slot. +4. The per-block transitions, which happens at every block. -The per-slot transitions focus on the slot counter and block roots records updates; the per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`; the per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization. +The state-root caching, caches the state root of the previous slot; +The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization; +The per-slot transitions focus on the slot counter and block roots records updates; +The per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`. Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. -_Note_: If there are skipped slots between a block and its parent block, run the steps in the [per-slot](#per-slot-processing) and [per-epoch](#per-epoch-processing) sections once for each skipped slot and then once for the slot containing the new block. +_Note_: If there are skipped slots between a block and its parent block, run the steps in the [per-epoch](#per-epoch-processing) and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. -### Per-slot processing +### State-root caching At every `slot > GENESIS_SLOT` run the following function: ```python -def advance_slot(state: BeaconState) -> None: +def store_state_root(state: BeaconState) -> None: state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) - state.slot += 1 - if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = get_state_root(state, state.slot - 1) - state.latest_block_roots[(state.slot - 1) % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) -``` - -### Per-block processing - -For every `block` except the genesis block, run `process_block_header(state, block)`, `process_randao(state, block)` and `process_eth1_data(state, block)`. - -#### Block header - -```python -def process_block_header(state: BeaconState, block: BeaconBlock) -> None: - # Verify that the slots match - assert block.slot == state.slot - # Verify that the parent matches - assert block.previous_block_root == hash_tree_root(state.latest_block_header) - # Save current block as the new latest block - state.latest_block_header = get_temporary_block_header(block) - # Verify proposer signature - proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)] - assert bls_verify( - pubkey=proposer.pubkey, - message_hash=signed_root(block), - signature=block.signature, - domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_BEACON_BLOCK) - ) -``` - -#### RANDAO - -```python -def process_randao(state: BeaconState, block: BeaconBlock) -> None: - proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)] - # Verify that the provided randao value is valid - assert bls_verify( - pubkey=proposer.pubkey, - message_hash=hash_tree_root(get_current_epoch(state)), - signature=block.body.randao_reveal, - domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_RANDAO) - ) - # Mix it in - state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = ( - xor(get_randao_mix(state, get_current_epoch(state)), - hash(block.body.randao_reveal)) - ) -``` - -#### Eth1 data - -```python -def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: - for eth1_data_vote in state.eth1_data_votes: - # If someone else has already voted for the same hash, add to its counter - if eth1_data_vote.eth1_data == block.body.eth1_data: - eth1_data_vote.vote_count += 1 - return - # If we're seeing this hash for the first time, make a new counter - state.eth1_data_votes.append(Eth1DataVote(eth1_data=block.body.eth1_data, vote_count=1)) -``` - -#### Transactions - -##### Proposer slashings - -Verify that `len(block.body.proposer_slashings) <= MAX_PROPOSER_SLASHINGS`. - -For each `proposer_slashing` in `block.body.proposer_slashings`, run the following function: - -```python -def process_proposer_slashing(state: BeaconState, - proposer_slashing: ProposerSlashing) -> None: - """ - Process ``ProposerSlashing`` transaction. - Note that this function mutates ``state``. - """ - proposer = state.validator_registry[proposer_slashing.proposer_index] - # Verify that the slot is the same - assert proposer_slashing.header_1.slot == proposer_slashing.header_2.slot - # But the roots are different! - assert hash_tree_root(proposer_slashing.header_1) != hash_tree_root(proposer_slashing.header_2) - # Proposer is not yet slashed - assert proposer.slashed is False - # Signatures are valid - for header in (proposer_slashing.header_1, proposer_slashing.header_2): - assert bls_verify( - pubkey=proposer.pubkey, - message_hash=signed_root(header), - signature=header.signature, - domain=get_domain(state.fork, slot_to_epoch(header.slot), DOMAIN_BEACON_BLOCK) - ) - slash_validator(state, proposer_slashing.proposer_index) -``` - -##### Attester slashings - -Verify that `len(block.body.attester_slashings) <= MAX_ATTESTER_SLASHINGS`. - -For each `attester_slashing` in `block.body.attester_slashings`, run the following function: - -```python -def process_attester_slashing(state: BeaconState, - attester_slashing: AttesterSlashing) -> None: - """ - Process ``AttesterSlashing`` transaction. - Note that this function mutates ``state``. - """ - attestation1 = attester_slashing.slashable_attestation_1 - attestation2 = attester_slashing.slashable_attestation_2 - # Check that the attestations are conflicting - assert attestation1.data != attestation2.data - assert ( - is_double_vote(attestation1.data, attestation2.data) or - is_surround_vote(attestation1.data, attestation2.data) - ) - assert verify_slashable_attestation(state, attestation1) - assert verify_slashable_attestation(state, attestation2) - slashable_indices = [ - index for index in attestation1.validator_indices - if ( - index in attestation2.validator_indices and - state.validator_registry[index].slashed is False - ) - ] - assert len(slashable_indices) >= 1 - for index in slashable_indices: - slash_validator(state, index) -``` - -##### Attestations - -Verify that `len(block.body.attestations) <= MAX_ATTESTATIONS`. - -For each `attestation` in `block.body.attestations`, run the following function: - -```python -def process_attestation(state: BeaconState, attestation: Attestation) -> None: - """ - Process ``Attestation`` transaction. - Note that this function mutates ``state``. - """ - # Can't submit attestations that are too far in history (or in prehistory) - assert attestation.data.slot >= GENESIS_SLOT - assert state.slot < attestation.data.slot + SLOTS_PER_EPOCH - # Can't submit attestations too quickly - assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot - # Verify that the justified epoch is correct, case 1: current epoch attestations - if slot_to_epoch(attestation.data.slot + 1) >= get_current_epoch(state): - assert attestation.data.justified_epoch == state.justified_epoch - # Case 2: previous epoch attestations - else: - assert attestation.data.justified_epoch == state.previous_justified_epoch - # Check that the justified block root is correct - assert attestation.data.justified_block_root == get_block_root( - state, get_epoch_start_slot(attestation.data.justified_epoch) - ) - # Check that the crosslink data is valid - acceptable_crosslink_data = { - # Case 1: Latest crosslink matches the one in the state - attestation.data.latest_crosslink, - # Case 2: State has already been updated, state's latest crosslink matches the crosslink - # the attestation is trying to create - Crosslink( - crosslink_data_root=attestation.data.crosslink_data_root, - epoch=slot_to_epoch(attestation.data.slot) - ) - } - assert state.latest_crosslinks[attestation.data.shard] in acceptable_crosslink_data - # Attestation must be nonempty! - assert attestation.aggregation_bitfield != b'\x00' * len(attestation.aggregation_bitfield) - # Custody must be empty (to be removed in phase 1) - assert attestation.custody_bitfield == b'\x00' * len(attestation.custody_bitfield) - # Get the committee for the specific shard that this attestation is for - crosslink_committee = [ - committee for committee, shard in get_crosslink_committees_at_slot(state, attestation.data.slot) - if shard == attestation.data.shard - ][0] - # Custody bitfield must be a subset of the attestation bitfield - for i in range(len(crosslink_committee)): - if get_bitfield_bit(attestation.aggregation_bitfield, i) == 0b0: - assert get_bitfield_bit(attestation.custody_bitfield, i) == 0b0 - # Verify aggregate signature - participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) - custody_bit_1_participants = get_attestation_participants(state, attestation.data, attestation.custody_bitfield) - custody_bit_0_participants = [i for i in participants if i not in custody_bit_1_participants] - - assert bls_verify_multiple( - pubkeys=[ - bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_0_participants]), - bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_participants]), - ], - message_hashes=[ - hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b0)), - hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b1)), - ], - signature=attestation.aggregate_signature, - domain=get_domain(state.fork, slot_to_epoch(attestation.data.slot), DOMAIN_ATTESTATION), - ) - # Crosslink data root is zero (to be removed in phase 1) - assert attestation.data.crosslink_data_root == ZERO_HASH - # Apply the attestation - pending_attestation = PendingAttestation( - data=attestation.data, - aggregation_bitfield=attestation.aggregation_bitfield, - custody_bitfield=attestation.custody_bitfield, - inclusion_slot=state.slot, - ) - if slot_to_epoch(attestation.data.slot) == get_current_epoch(state): - state.current_epoch_attestations.append(pending_attestation) - elif slot_to_epoch(attestation.data.slot) == get_previous_epoch(state): - state.previous_epoch_attestations.append(pending_attestation) -``` - -##### Deposits - -Verify that `len(block.body.deposits) <= MAX_DEPOSITS`. - -For each `deposit` in `block.body.deposits`, run `process_deposit(state, deposit)`. - -##### Voluntary exits - -Verify that `len(block.body.voluntary_exits) <= MAX_VOLUNTARY_EXITS`. - -For each `exit` in `block.body.voluntary_exits`, run the following function: - -```python -def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: - """ - Process ``VoluntaryExit`` transaction. - Note that this function mutates ``state``. - """ - validator = state.validator_registry[exit.validator_index] - # Verify the validator has not yet exited - assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) - # Exits must specify an epoch when they become valid; they are not valid before then - assert get_current_epoch(state) >= exit.epoch - # Verify signature - assert bls_verify( - pubkey=validator.pubkey, - message_hash=signed_root(exit), - signature=exit.signature, - domain=get_domain(state.fork, exit.epoch, DOMAIN_VOLUNTARY_EXIT) - ) - # Run the exit - initiate_validator_exit(state, exit.validator_index) -``` - -##### Transfers - -Note: Transfers are a temporary functionality for phases 0 and 1, to be removed in phase 2. - -Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers are distinct. - -For each `transfer` in `block.body.transfers`, run the following function: - -```python -def process_transfer(state: BeaconState, transfer: Transfer) -> None: - """ - Process ``Transfer`` transaction. - Note that this function mutates ``state``. - """ - # Verify the amount and fee aren't individually too big (for anti-overflow purposes) - assert state.validator_balances[transfer.sender] >= max(transfer.amount, transfer.fee) - # Verify that we have enough ETH to send, and that after the transfer the balance will be either - # exactly zero or at least MIN_DEPOSIT_AMOUNT - assert ( - state.validator_balances[transfer.sender] == transfer.amount + transfer.fee or - state.validator_balances[transfer.sender] >= transfer.amount + transfer.fee + MIN_DEPOSIT_AMOUNT - ) - # A transfer is valid in only one slot - assert state.slot == transfer.slot - # Only withdrawn or not-yet-deposited accounts can transfer - assert ( - get_current_epoch(state) >= state.validator_registry[transfer.sender].withdrawable_epoch or - state.validator_registry[transfer.sender].activation_epoch == FAR_FUTURE_EPOCH - ) - # Verify that the pubkey is valid - assert ( - state.validator_registry[transfer.sender].withdrawal_credentials == - BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:] - ) - # Verify that the signature is valid - assert bls_verify( - pubkey=transfer.pubkey, - message_hash=signed_root(transfer), - signature=transfer.signature, - domain=get_domain(state.fork, slot_to_epoch(transfer.slot), DOMAIN_TRANSFER) - ) - # Process the transfer - state.validator_balances[transfer.sender] -= transfer.amount + transfer.fee - state.validator_balances[transfer.recipient] += transfer.amount - state.validator_balances[get_beacon_proposer_index(state, state.slot)] += transfer.fee ``` ### Per-epoch processing -The steps below happen when `(state.slot + 1) % SLOTS_PER_EPOCH == 0`. +The steps below happen when `state.slot > GENESIS_SLOT and (state.slot + 1) % SLOTS_PER_EPOCH == 0`. #### Helper functions - -We define some helper functions: + +We define some helper functions utilized when processing an epoch transition: ```python def get_current_total_balance(state: BeaconState) -> Gwei: @@ -2022,24 +1731,24 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat ```python def get_current_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ - a for a in state.current_epoch_attestations if - a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) + a for a in state.current_epoch_attestations + if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) ] ``` ```python def get_previous_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ - a for a in state.previous_epoch_attestations if - a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) + a for a in state.previous_epoch_attestations + if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) ] ``` ```python def get_previous_epoch_matching_head_attestations(state: BeaconState) -> List[PendingAttestation]: return [ - a for a in state.previous_epoch_attestations if - a.data.beacon_block_root == get_block_root(state, a.data.slot) + a for a in state.previous_epoch_attestations + if a.data.beacon_block_root == get_block_root(state, a.data.slot) ] ``` @@ -2057,7 +1766,7 @@ def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple if len(all_roots) == 0: return ZERO_HASH, [] - def get_attestations_for(root: Bytes32) -> List[PendingAttestation]: + def get_attestations_for(root) -> List[PendingAttestation]: return [a for a in valid_attestations if a.data.crosslink_data_root == root] # Winning crosslink root is the root with the most votes for it, ties broken in favor of @@ -2105,7 +1814,7 @@ def update_justification_and_finalization(state: BeaconState) -> None: if current_boundary_attesting_balance * 3 >= get_current_total_balance(state) * 2: new_justified_epoch = get_current_epoch(state) state.justification_bitfield |= 1 - + # Process finalizations bitfield = state.justification_bitfield current_epoch = get_current_epoch(state) @@ -2118,10 +1827,10 @@ def update_justification_and_finalization(state: BeaconState) -> None: # The 1st/2nd/3rd most recent epochs are all justified, the 1st using the 3rd as source if (bitfield >> 0) % 8 == 0b111 and state.justified_epoch == current_epoch - 2: state.finalized_epoch = state.justified_epoch - # The 1st/2nd most recent epochs are both justified, the 1st using the 2nd as source + # The 1st/2nd most recent epochs are both justified, the 1st using the 2nd as source if (bitfield >> 0) % 4 == 0b11 and state.justified_epoch == current_epoch - 1: state.finalized_epoch = state.justified_epoch - + # Rotate justified epochs state.previous_justified_epoch = state.justified_epoch state.justified_epoch = new_justified_epoch @@ -2134,7 +1843,7 @@ Run the following function: ```python def process_crosslinks(state: BeaconState) -> None: current_epoch = get_current_epoch(state) - previous_epoch = get_previous_epoch(state) + previous_epoch = current_epoch - 1 next_epoch = current_epoch + 1 for slot in range(get_epoch_start_slot(previous_epoch), get_epoch_start_slot(next_epoch)): for crosslink_committee, shard in get_crosslink_committees_at_slot(state, slot): @@ -2144,7 +1853,7 @@ def process_crosslinks(state: BeaconState) -> None: if 3 * participating_balance >= 2 * total_balance: state.latest_crosslinks[shard] = Crosslink( epoch=slot_to_epoch(slot), - crosslink_data_root=winning_root, + crosslink_data_root=winning_root ) ``` @@ -2244,7 +1953,7 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> ``` When blocks are not finalizing normally... - + ```python def compute_inactivity_leak_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: # deltas[0] for rewards @@ -2328,7 +2037,7 @@ def apply_rewards(state: BeaconState) -> None: #### Ejections -* Run `process_ejections(state)`. +Run `process_ejections(state)`. ```python def process_ejections(state: BeaconState) -> None: @@ -2515,9 +2224,315 @@ def finish_epoch_update(state: BeaconState) -> None: state.current_epoch_attestations = [] ``` -### State root verification +### Per-slot processing -Verify `block.state_root == hash_tree_root(state)` if there exists a `block` for the slot being processed. +At every `slot > GENESIS_SLOT` run the following function: + +```python +def advance_slot(state: BeaconState) -> None: + state.slot += 1 + if state.latest_block_header.state_root == ZERO_HASH: + state.latest_block_header.state_root = get_state_root(state, state.slot - 1) + state.latest_block_roots[(state.slot - 1) % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) +``` + +### Per-block processing + +For every `block` except the genesis block, run `process_block_header(state, block)`, `process_randao(state, block)` and `process_eth1_data(state, block)`. + +#### Block header + +```python +def process_block_header(state: BeaconState, block: BeaconBlock) -> None: + # Verify that the slots match + assert block.slot == state.slot + # Verify that the parent matches + assert block.previous_block_root == hash_tree_root(state.latest_block_header) + # Save current block as the new latest block + state.latest_block_header = get_temporary_block_header(block) + # Verify proposer signature + proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)] + assert bls_verify( + pubkey=proposer.pubkey, + message_hash=signed_root(block), + signature=block.signature, + domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_BEACON_BLOCK) + ) +``` + +#### RANDAO + +```python +def process_randao(state: BeaconState, block: BeaconBlock) -> None: + proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)] + # Verify that the provided randao value is valid + assert bls_verify( + pubkey=proposer.pubkey, + message_hash=hash_tree_root(get_current_epoch(state)), + signature=block.body.randao_reveal, + domain=get_domain(state.fork, get_current_epoch(state), DOMAIN_RANDAO) + ) + # Mix it in + state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = ( + xor(get_randao_mix(state, get_current_epoch(state)), + hash(block.body.randao_reveal)) + ) +``` + +#### Eth1 data + +```python +def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: + for eth1_data_vote in state.eth1_data_votes: + # If someone else has already voted for the same hash, add to its counter + if eth1_data_vote.eth1_data == block.body.eth1_data: + eth1_data_vote.vote_count += 1 + return + # If we're seeing this hash for the first time, make a new counter + state.eth1_data_votes.append(Eth1DataVote(eth1_data=block.body.eth1_data, vote_count=1)) +``` + +#### Transactions + +##### Proposer slashings + +Verify that `len(block.body.proposer_slashings) <= MAX_PROPOSER_SLASHINGS`. + +For each `proposer_slashing` in `block.body.proposer_slashings`, run the following function: + +```python +def process_proposer_slashing(state: BeaconState, + proposer_slashing: ProposerSlashing) -> None: + """ + Process ``ProposerSlashing`` transaction. + Note that this function mutates ``state``. + """ + proposer = state.validator_registry[proposer_slashing.proposer_index] + # Verify that the slot is the same + assert proposer_slashing.header_1.slot == proposer_slashing.header_2.slot + # But the roots are different! + assert hash_tree_root(proposer_slashing.header_1) != hash_tree_root(proposer_slashing.header_2) + # Proposer is not yet slashed + assert proposer.slashed is False + # Signatures are valid + for header in (proposer_slashing.header_1, proposer_slashing.header_2): + assert bls_verify( + pubkey=proposer.pubkey, + message_hash=signed_root(header), + signature=header.signature, + domain=get_domain(state.fork, slot_to_epoch(header.slot), DOMAIN_BEACON_BLOCK) + ) + slash_validator(state, proposer_slashing.proposer_index) +``` + +##### Attester slashings + +Verify that `len(block.body.attester_slashings) <= MAX_ATTESTER_SLASHINGS`. + +For each `attester_slashing` in `block.body.attester_slashings`, run the following function: + +```python +def process_attester_slashing(state: BeaconState, + attester_slashing: AttesterSlashing) -> None: + """ + Process ``AttesterSlashing`` transaction. + Note that this function mutates ``state``. + """ + attestation1 = attester_slashing.slashable_attestation_1 + attestation2 = attester_slashing.slashable_attestation_2 + # Check that the attestations are conflicting + assert attestation1.data != attestation2.data + assert ( + is_double_vote(attestation1.data, attestation2.data) or + is_surround_vote(attestation1.data, attestation2.data) + ) + assert verify_slashable_attestation(state, attestation1) + assert verify_slashable_attestation(state, attestation2) + slashable_indices = [ + index for index in attestation1.validator_indices + if ( + index in attestation2.validator_indices and + state.validator_registry[index].slashed is False + ) + ] + assert len(slashable_indices) >= 1 + for index in slashable_indices: + slash_validator(state, index) +``` + +##### Attestations + +Verify that `len(block.body.attestations) <= MAX_ATTESTATIONS`. + +For each `attestation` in `block.body.attestations`, run the following function: + +```python +def process_attestation(state: BeaconState, attestation: Attestation) -> None: + """ + Process ``Attestation`` transaction. + Note that this function mutates ``state``. + """ + # Can't submit attestations that are too far in history (or in prehistory) + assert attestation.data.slot >= GENESIS_SLOT + assert state.slot < attestation.data.slot + SLOTS_PER_EPOCH + # Can't submit attestations too quickly + assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot + # Verify that the justified epoch is correct, case 1: current epoch attestations + if slot_to_epoch(attestation.data.slot) >= get_current_epoch(state): + assert attestation.data.justified_epoch == state.justified_epoch + # Case 2: previous epoch attestations + else: + assert attestation.data.justified_epoch == state.previous_justified_epoch + # Check that the justified block root is correct + assert attestation.data.justified_block_root == get_block_root( + state, get_epoch_start_slot(attestation.data.justified_epoch) + ) + # Check that the crosslink data is valid + acceptable_crosslink_data = { + # Case 1: Latest crosslink matches the one in the state + attestation.data.latest_crosslink, + # Case 2: State has already been updated, state's latest crosslink matches the crosslink + # the attestation is trying to create + Crosslink( + crosslink_data_root=attestation.data.crosslink_data_root, + epoch=slot_to_epoch(attestation.data.slot) + ) + } + assert state.latest_crosslinks[attestation.data.shard] in acceptable_crosslink_data + # Attestation must be nonempty! + assert attestation.aggregation_bitfield != b'\x00' * len(attestation.aggregation_bitfield) + # Custody must be empty (to be removed in phase 1) + assert attestation.custody_bitfield == b'\x00' * len(attestation.custody_bitfield) + # Get the committee for the specific shard that this attestation is for + crosslink_committee = [ + committee for committee, shard in get_crosslink_committees_at_slot(state, attestation.data.slot) + if shard == attestation.data.shard + ][0] + # Custody bitfield must be a subset of the attestation bitfield + for i in range(len(crosslink_committee)): + if get_bitfield_bit(attestation.aggregation_bitfield, i) == 0b0: + assert get_bitfield_bit(attestation.custody_bitfield, i) == 0b0 + # Verify aggregate signature + participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) + custody_bit_1_participants = get_attestation_participants(state, attestation.data, attestation.custody_bitfield) + custody_bit_0_participants = [i for i in participants if i not in custody_bit_1_participants] + + assert bls_verify_multiple( + pubkeys=[ + bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_0_participants]), + bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_participants]), + ], + message_hashes=[ + hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b0)), + hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b1)), + ], + signature=attestation.aggregate_signature, + domain=get_domain(state.fork, slot_to_epoch(attestation.data.slot), DOMAIN_ATTESTATION), + ) + # Crosslink data root is zero (to be removed in phase 1) + assert attestation.data.crosslink_data_root == ZERO_HASH + # Apply the attestation + pending_attestation = PendingAttestation( + data=attestation.data, + aggregation_bitfield=attestation.aggregation_bitfield, + custody_bitfield=attestation.custody_bitfield, + inclusion_slot=state.slot + ) + if slot_to_epoch(attestation.data.slot) == get_current_epoch(state): + state.current_epoch_attestations.append(pending_attestation) + elif slot_to_epoch(attestation.data.slot) == get_previous_epoch(state): + state.previous_epoch_attestations.append(pending_attestation) +``` + +##### Deposits + +Verify that `len(block.body.deposits) <= MAX_DEPOSITS`. + +For each `deposit` in `block.body.deposits`, run `process_deposit(state, deposit)`. + +##### Voluntary exits + +Verify that `len(block.body.voluntary_exits) <= MAX_VOLUNTARY_EXITS`. + +For each `exit` in `block.body.voluntary_exits`, run the following function: + +```python +def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: + """ + Process ``VoluntaryExit`` transaction. + Note that this function mutates ``state``. + """ + validator = state.validator_registry[exit.validator_index] + # Verify the validator has not yet exited + assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) + # Exits must specify an epoch when they become valid; they are not valid before then + assert get_current_epoch(state) >= exit.epoch + # Verify signature + assert bls_verify( + pubkey=validator.pubkey, + message_hash=signed_root(exit), + signature=exit.signature, + domain=get_domain(state.fork, exit.epoch, DOMAIN_VOLUNTARY_EXIT) + ) + # Run the exit + initiate_validator_exit(state, exit.validator_index) +``` + +##### Transfers + +Note: Transfers are a temporary functionality for phases 0 and 1, to be removed in phase 2. + +Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers are distinct. + +For each `transfer` in `block.body.transfers`, run the following function: + +```python +def process_transfer(state: BeaconState, transfer: Transfer) -> None: + """ + Process ``Transfer`` transaction. + Note that this function mutates ``state``. + """ + # Verify the amount and fee aren't individually too big (for anti-overflow purposes) + assert state.validator_balances[transfer.sender] >= max(transfer.amount, transfer.fee) + # Verify that we have enough ETH to send, and that after the transfer the balance will be either + # exactly zero or at least MIN_DEPOSIT_AMOUNT + assert ( + state.validator_balances[transfer.sender] == transfer.amount + transfer.fee or + state.validator_balances[transfer.sender] >= transfer.amount + transfer.fee + MIN_DEPOSIT_AMOUNT + ) + # A transfer is valid in only one slot + assert state.slot == transfer.slot + # Only withdrawn or not-yet-deposited accounts can transfer + assert ( + get_current_epoch(state) >= state.validator_registry[transfer.sender].withdrawable_epoch or + state.validator_registry[transfer.sender].activation_epoch == FAR_FUTURE_EPOCH + ) + # Verify that the pubkey is valid + assert ( + state.validator_registry[transfer.sender].withdrawal_credentials == + BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:] + ) + # Verify that the signature is valid + assert bls_verify( + pubkey=transfer.pubkey, + message_hash=signed_root(transfer), + signature=transfer.signature, + domain=get_domain(state.fork, slot_to_epoch(transfer.slot), DOMAIN_TRANSFER) + ) + # Process the transfer + state.validator_balances[transfer.sender] -= transfer.amount + transfer.fee + state.validator_balances[transfer.recipient] += transfer.amount + state.validator_balances[get_beacon_proposer_index(state, state.slot)] += transfer.fee +``` + +#### State root verification + +Verify the block's `state_root` by running the following function: + +```python +def verify_block_state_root(state: BeaconState, block: BeaconBlock) -> None: + assert block.state_root == hash_tree_root(state) +``` # References From e57bfaab7c4096cf022d3948f78cf553f9e1a75d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 7 Mar 2019 13:36:22 -0700 Subject: [PATCH 122/161] clean up state transition notes --- specs/core/0_beacon-chain.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 740ace0df..144262d8b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1667,14 +1667,15 @@ We now define the state transition function. At a high level the state transitio 3. The per-slot transitions, which happens at every slot. 4. The per-block transitions, which happens at every block. -The state-root caching, caches the state root of the previous slot; -The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization; -The per-slot transitions focus on the slot counter and block roots records updates; -The per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`. +Transition section notes: +* The state-root caching, caches the state root of the previous slot; +* The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization; +* The per-slot transitions focus on the slot counter and block roots records updates; +* The per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`. Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. -_Note_: If there are skipped slots between a block and its parent block, run the steps in the [per-epoch](#per-epoch-processing) and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. +_Note_: If there are skipped slots between a block and its parent block, run the steps in the [state-root](#state-root-caching), [per-epoch](#per-epoch-processing), and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. ### State-root caching From 2d9724dbfc30315f947ca583956f3b5bad4dc764 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 7 Mar 2019 23:13:06 +0100 Subject: [PATCH 123/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 144262d8b..36cb0cc6f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2148,9 +2148,8 @@ def process_slashings(state: BeaconState) -> None: total_balance = get_total_balance(state, active_validator_indices) # Compute `total_penalties` - epoch_index = current_epoch % LATEST_SLASHED_EXIT_LENGTH - total_at_start = state.latest_slashed_balances[(epoch_index + 1) % LATEST_SLASHED_EXIT_LENGTH] - total_at_end = state.latest_slashed_balances[epoch_index] + total_at_start = state.latest_slashed_balances[(current_epoch + 1) % LATEST_SLASHED_EXIT_LENGTH] + total_at_end = state.latest_slashed_balances[current_epoch % LATEST_SLASHED_EXIT_LENGTH] total_penalties = total_at_end - total_at_start for index, validator in enumerate(state.validator_registry): From 339a7fb63b098b4b91cf31b390d98a4f76c02f76 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 7 Mar 2019 23:14:47 +0100 Subject: [PATCH 124/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36cb0cc6f..ed92cec23 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1682,7 +1682,7 @@ _Note_: If there are skipped slots between a block and its parent block, run the At every `slot > GENESIS_SLOT` run the following function: ```python -def store_state_root(state: BeaconState) -> None: +def cache_state_root(state: BeaconState) -> None: state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) ``` From 33a05109eaa79db5f8d3b6fd5c9ce42fa33422b1 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 7 Mar 2019 23:56:03 +0100 Subject: [PATCH 125/161] Fair proposer sampling I think we want `first_committee[epoch % len(first_committee)]` as opposed to `first_committee[slot % len(first_committee)]`. The reason is that if the shuffling happens infrequently and `len(first_committee)` is a multiple of `SLOTS_PER_EPOCH` then the proposers will not be sampled fairly. Taking this logic further, we may want to avoiding always picking the proposer from `first_committee`, e.g.: ``` validators_at_slot = [] for crosslink_committee, _ in get_crosslink_committees_at_slot(state, slot, registry_change): validators_at_slot.append(crosslink_committee) return validators_at_slot[epoch % len(validators_at_slot)] ``` --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 898fc3c86..dfce48f96 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1002,7 +1002,7 @@ def get_beacon_proposer_index(state: BeaconState, assert previous_epoch <= epoch <= next_epoch first_committee, _ = get_crosslink_committees_at_slot(state, slot, registry_change)[0] - return first_committee[slot % len(first_committee)] + return first_committee[epoch % len(first_committee)] ``` ### `merkle_root` From e74c79e353a63ac43ec3d3367f2de272169e01b6 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 09:08:30 +0100 Subject: [PATCH 126/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ed92cec23..c0b5bfd8b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2218,10 +2218,10 @@ At every `slot > GENESIS_SLOT` run the following function: ```python def advance_slot(state: BeaconState) -> None: - state.slot += 1 if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = get_state_root(state, state.slot - 1) - state.latest_block_roots[(state.slot - 1) % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) + state.latest_block_header.state_root = state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] + state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) + state.slot += 1 ``` ### Per-block processing From 1f147486fccf4557c5cfd74f4a4f47a244e6393c Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 09:57:09 +0100 Subject: [PATCH 127/161] Semantic fork versions and signature domains Fix #706 --- specs/core/0_beacon-chain.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 898fc3c86..cfa4782a5 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -268,12 +268,12 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | Name | Value | | - | - | -| `DOMAIN_BEACON_BLOCK` | `0` | -| `DOMAIN_RANDAO` | `1` | -| `DOMAIN_ATTESTATION` | `2` | -| `DOMAIN_DEPOSIT` | `3` | -| `DOMAIN_VOLUNTARY_EXIT` | `4` | -| `DOMAIN_TRANSFER` | `5` | +| `DOMAIN_BEACON_BLOCK` | `int_to_bytes4(0)` | +| `DOMAIN_RANDAO` | `int_to_bytes4(1)` | +| `DOMAIN_ATTESTATION` | `int_to_bytes4(2)` | +| `DOMAIN_DEPOSIT` | `int_to_bytes4(3)` | +| `DOMAIN_VOLUNTARY_EXIT` | `int_to_bytes4(4)` | +| `DOMAIN_TRANSFER` | `int_to_bytes4(5)` | ## Data structures @@ -288,9 +288,9 @@ The types are defined topologically to aid in facilitating an executable version ```python { # Previous fork version - 'previous_version': 'uint64', + 'previous_version': 'bytes4', # Current fork version - 'current_version': 'uint64', + 'current_version': 'bytes4', # Fork epoch number 'epoch': 'uint64', } @@ -1107,7 +1107,7 @@ def get_total_balance(state: BeaconState, validators: List[ValidatorIndex]) -> G ```python def get_fork_version(fork: Fork, - epoch: Epoch) -> int: + epoch: Epoch) -> bytes8: """ Return the fork version of the given ``epoch``. """ @@ -1122,12 +1122,11 @@ def get_fork_version(fork: Fork, ```python def get_domain(fork: Fork, epoch: Epoch, - domain_type: int) -> int: + domain_type: bytes4) -> bytes8: """ Get the domain number that represents the fork meta and signature domain. """ - fork_version = get_fork_version(fork, epoch) - return fork_version * 2**32 + domain_type + return get_fork_version(fork, epoch) + domain_type ``` ### `get_bitfield_bit` From 8dcc1ba930332980a88a1745539cea61e5c07d87 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 8 Mar 2019 02:58:38 -0600 Subject: [PATCH 128/161] Break LMD GHOST ties in favor of higher hash tree roots (#737) --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 898fc3c86..c51effc9a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1654,7 +1654,7 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock) children = get_children(store, head) if len(children) == 0: return head - head = max(children, key=get_vote_count) + head = max(children, key=lambda x: (get_vote_count(x), hash_tree_root(x))) ``` ## Beacon chain state transition function From 12695425c9ba8ef705665d36788cf7751be15b39 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 10:32:40 +0100 Subject: [PATCH 129/161] Use hash_tree_root everywhere And get rid of merkle_root. This is possible because of SSZ tuples. --- specs/core/0_beacon-chain.md | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c51effc9a..f79140379 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -73,7 +73,6 @@ - [`get_active_index_root`](#get_active_index_root) - [`generate_seed`](#generate_seed) - [`get_beacon_proposer_index`](#get_beacon_proposer_index) - - [`merkle_root`](#merkle_root) - [`verify_merkle_branch`](#verify_merkle_branch) - [`get_attestation_participants`](#get_attestation_participants) - [`is_power_of_two`](#is_power_of_two) @@ -1005,20 +1004,6 @@ def get_beacon_proposer_index(state: BeaconState, return first_committee[slot % len(first_committee)] ``` -### `merkle_root` - -```python -def merkle_root(values: List[Bytes32]) -> Bytes32: - """ - Merkleize ``values`` (where ``len(values)`` is a power of two) and return the Merkle root. - Note that the leaves are not hashed. - """ - o = [0] * len(values) + values - for i in range(len(values) - 1, 0, -1): - o[i] = hash(o[i * 2] + o[i * 2 + 1]) - return o[1] -``` - ### `verify_merkle_branch` ```python @@ -2494,7 +2479,7 @@ def finish_epoch_update(state: BeaconState) -> None: state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch) # Set historical root accumulator if next_epoch % (SLOTS_PER_HISTORICAL_ROOT // SLOTS_PER_EPOCH) == 0: - state.historical_roots.append(merkle_root(state.latest_block_roots + state.latest_state_roots)) + state.historical_roots.append(hash_tree_root(state.latest_block_roots + state.latest_state_roots)) # Rotate current/previous epoch attestations state.previous_epoch_attestations = state.current_epoch_attestations state.current_epoch_attestations = [] From a51d7d5db591ada9b7ae01ba37fe4fecf6ca0653 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 10:50:45 +0100 Subject: [PATCH 130/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index cfa4782a5..7b7c58ab0 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1107,7 +1107,7 @@ def get_total_balance(state: BeaconState, validators: List[ValidatorIndex]) -> G ```python def get_fork_version(fork: Fork, - epoch: Epoch) -> bytes8: + epoch: Epoch) -> bytes4: """ Return the fork version of the given ``epoch``. """ From 11414673495b0fdf5df7223f710ded05bfad9cd9 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 17:28:00 +0100 Subject: [PATCH 131/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 971a521dc..679b5a65b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2191,7 +2191,7 @@ def finish_epoch_update(state: BeaconState) -> None: state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch) # Set historical root accumulator if next_epoch % (SLOTS_PER_HISTORICAL_ROOT // SLOTS_PER_EPOCH) == 0: - state.historical_roots.append(merkle_root(state.latest_block_roots + state.latest_state_roots)) + state.historical_roots.append(hash_tree_root(state.latest_block_roots + state.latest_state_roots)) # Rotate current/previous epoch attestations state.previous_epoch_attestations = state.current_epoch_attestations state.current_epoch_attestations = [] From 3aedf1226a5913b3545f35cefcca058a9dfdf9a1 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 17:36:58 +0100 Subject: [PATCH 132/161] Update 0_beacon-chain.md Did I get the type casting OK @hwwhww? --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 7b7c58ab0..9efc73f1a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1122,11 +1122,11 @@ def get_fork_version(fork: Fork, ```python def get_domain(fork: Fork, epoch: Epoch, - domain_type: bytes4) -> bytes8: + domain_type: bytes4) -> uint64: """ Get the domain number that represents the fork meta and signature domain. """ - return get_fork_version(fork, epoch) + domain_type + return bytes_to_int(get_fork_version(fork, epoch) + domain_type)) ``` ### `get_bitfield_bit` From ecd93468a233ba54af5a7a1492a195c4b3a18db1 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 17:38:19 +0100 Subject: [PATCH 133/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 9efc73f1a..900708e7a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1122,7 +1122,7 @@ def get_fork_version(fork: Fork, ```python def get_domain(fork: Fork, epoch: Epoch, - domain_type: bytes4) -> uint64: + domain_type: bytes4) -> int: """ Get the domain number that represents the fork meta and signature domain. """ From f081f23c318e0dfc190b784988ff05ce38813eea Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 8 Mar 2019 09:48:46 -0700 Subject: [PATCH 134/161] cache more than just state root at start of state transition --- specs/core/0_beacon-chain.md | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a339d5058..286855aef 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -110,7 +110,7 @@ - [Beacon chain processing](#beacon-chain-processing) - [Beacon chain fork choice rule](#beacon-chain-fork-choice-rule) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - - [State-root caching](#state-root-caching) + - [State caching](#state-caching) - [Per-epoch processing](#per-epoch-processing) - [Helper functions](#helper-functions-1) - [Justification](#justification) @@ -1662,28 +1662,38 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock) We now define the state transition function. At a high level the state transition is made up of four parts: -1. State-root caching, which happens at the start of every slot. +1. State caching, which happens at the start of every slot. 2. The per-epoch transitions, which happens at the start of the first slot of every epoch. 3. The per-slot transitions, which happens at every slot. 4. The per-block transitions, which happens at every block. Transition section notes: -* The state-root caching, caches the state root of the previous slot; -* The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization; -* The per-slot transitions focus on the slot counter and block roots records updates; +* The state caching, caches the state root of the previous slot. +* The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization. +* The per-slot transitions focus on the slot counter and block roots records updates. * The per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`. Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. -_Note_: If there are skipped slots between a block and its parent block, run the steps in the [state-root](#state-root-caching), [per-epoch](#per-epoch-processing), and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. +_Note_: If there are skipped slots between a block and its parent block, run the steps in the [state-root](#state-caching), [per-epoch](#per-epoch-processing), and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. -### State-root caching +### State caching At every `slot > GENESIS_SLOT` run the following function: ```python -def cache_state_root(state: BeaconState) -> None: - state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state) +def cache_state(state: BeaconState) -> None: + previous_slot_state_root = hash_tree_root(state) + + # store the previous slot's post state transition root + state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_slot_state_root + + # cache state root in stored latest_block_header if empty + if state.latest_block_header.state_root == ZERO_HASH: + state.latest_block_header.state_root = previous_slot_state_root + + # store latest known block for previous slot + state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) ``` ### Per-epoch processing @@ -2218,9 +2228,6 @@ At every `slot > GENESIS_SLOT` run the following function: ```python def advance_slot(state: BeaconState) -> None: - if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] - state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = hash_tree_root(state.latest_block_header) state.slot += 1 ``` From de60533d7238c4010439152176e9a10828d50fa2 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 18:13:05 +0100 Subject: [PATCH 135/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 679b5a65b..cc7cc5b18 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -34,6 +34,7 @@ - [`BeaconBlockHeader`](#beaconblockheader) - [`Validator`](#validator) - [`PendingAttestation`](#pendingattestation) + - [`HistoricalBatch`](#historicalbatch) - [Beacon transactions](#beacon-transactions) - [`ProposerSlashing`](#proposerslashing) - [`AttesterSlashing`](#attesterslashing) @@ -452,6 +453,17 @@ The types are defined topologically to aid in facilitating an executable version } ``` +#### `HistoricalBatch` + +```python +{ + // Block roots + 'block_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], + // State roots + 'state_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], +} +``` + ### Beacon transactions #### `ProposerSlashing` @@ -2191,7 +2203,11 @@ def finish_epoch_update(state: BeaconState) -> None: state.latest_randao_mixes[next_epoch % LATEST_RANDAO_MIXES_LENGTH] = get_randao_mix(state, current_epoch) # Set historical root accumulator if next_epoch % (SLOTS_PER_HISTORICAL_ROOT // SLOTS_PER_EPOCH) == 0: - state.historical_roots.append(hash_tree_root(state.latest_block_roots + state.latest_state_roots)) + historical_batch = HistoricalBatch( + block_roots=state.latest_block_roots, + state_roots=state.latest_state_roots, + ) + state.historical_roots.append(hash_tree_root(historical_batch)) # Rotate current/previous epoch attestations state.previous_epoch_attestations = state.current_epoch_attestations state.current_epoch_attestations = [] From f180eb5e9e2bcf33325c8aece7655adb76bf0561 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 18:14:00 +0100 Subject: [PATCH 136/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index cc7cc5b18..8c9aafd38 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -457,9 +457,9 @@ The types are defined topologically to aid in facilitating an executable version ```python { - // Block roots + # Block roots 'block_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], - // State roots + # State roots 'state_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], } ``` From f253feeacf6639f62e1260eee8c2dab80628a4bc Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 8 Mar 2019 18:34:51 +0100 Subject: [PATCH 137/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a339d5058..f33a827b4 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -130,12 +130,12 @@ - [RANDAO](#randao) - [Eth1 data](#eth1-data) - [Transactions](#transactions) - - [Proposer slashings](#proposer-slashings-1) - - [Attester slashings](#attester-slashings-1) - - [Attestations](#attestations-1) - - [Deposits](#deposits-1) - - [Voluntary exits](#voluntary-exits-1) - - [Transfers](#transfers-1) + - [Proposer slashings](#proposer-slashings) + - [Attester slashings](#attester-slashings) + - [Attestations](#attestations) + - [Deposits](#deposits) + - [Voluntary exits](#voluntary-exits) + - [Transfers](#transfers) - [State root verification](#state-root-verification) - [References](#references) - [Normative](#normative) @@ -2298,8 +2298,8 @@ def process_proposer_slashing(state: BeaconState, proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same assert proposer_slashing.header_1.slot == proposer_slashing.header_2.slot - # But the roots are different! - assert hash_tree_root(proposer_slashing.header_1) != hash_tree_root(proposer_slashing.header_2) + # But the headers are different + assert proposer_slashing.header_1 != proposer_slashing.header_2 # Proposer is not yet slashed assert proposer.slashed is False # Signatures are valid From f88db44e81b7f9fa44c7bfdf4d146ded2fb90442 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 8 Mar 2019 12:07:57 -0700 Subject: [PATCH 138/161] skip proposer bonus if no attestation for v index --- specs/core/0_beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 286855aef..3065dbfd2 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1946,8 +1946,9 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> else: deltas[1][index] += get_base_reward(state, index) # Proposer bonus - proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) - deltas[0][proposer_index] += get_base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT + if index in get_attesting_indices(state, state.previous_epoch_attestations): + proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) + deltas[0][proposer_index] += get_base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT return deltas ``` From 30e64d7de6bd72d20d3cfe91a9434bc980e113a0 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 8 Mar 2019 12:14:21 -0700 Subject: [PATCH 139/161] fix get_inactivity_penalty function signature --- specs/core/0_beacon-chain.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3065dbfd2..44b96272a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1885,8 +1885,7 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: ``` ```python -def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex) -> Gwei: - epochs_since_finality = get_current_epoch(state) + 1 - state.finalized_epoch +def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex, epochs_since_finality: int) -> Gwei: return ( get_base_reward(state, index) + get_effective_balance(state, index) * epochs_since_finality // INACTIVITY_PENALTY_QUOTIENT // 2 From 902e65e072bbb4ca825f610962aafbc6c37acb8e Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 8 Mar 2019 15:16:06 -0700 Subject: [PATCH 140/161] add min persistent committee period resitriction on exits --- specs/core/0_beacon-chain.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c88155f77..43e159ad2 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -229,8 +229,9 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | `MIN_SEED_LOOKAHEAD` | `2**0` (= 1) | epochs | 6.4 minutes | | `ACTIVATION_EXIT_DELAY` | `2**2` (= 4) | epochs | 25.6 minutes | | `EPOCHS_PER_ETH1_VOTING_PERIOD` | `2**4` (= 16) | epochs | ~1.7 hours | -| `SLOTS_PER_HISTORICAL_ROOT` | `2**13` (= 8,192) | slots | ~13 hours | +| `SLOTS_PER_HISTORICAL_ROOT` | `2**13` (= 8,192) | slots | ~13 hours | | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY` | `2**8` (= 256) | epochs | ~27 hours | +| `PERSISTENT_COMMITTEE_PERIOD` | `2**11` (= 2,048) | epochs | 9 days | ### State list lengths @@ -2463,6 +2464,8 @@ def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) # Exits must specify an epoch when they become valid; they are not valid before then assert get_current_epoch(state) >= exit.epoch + # Must have been in the validator set long enough + assert get_current_epoch(state) - validator.activation_epoch >= PERSISTENT_COMMITTEE_PERIOD # Verify signature assert bls_verify( pubkey=validator.pubkey, From 02428ec2520febb41f5be19fb6434d270d49586a Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 10 Mar 2019 13:25:57 +0100 Subject: [PATCH 141/161] Do not check withdrawal credentials for existing validators We should not invalidate blocks that contain a deposit with an inconsistent withdrawal credential as that would stall the chain. --- specs/core/0_beacon-chain.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c88155f77..25e8085de 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1341,10 +1341,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: state.validator_balances.append(amount) else: # Increase balance by deposit amount - index = validator_pubkeys.index(pubkey) - assert state.validator_registry[index].withdrawal_credentials == withdrawal_credentials - - state.validator_balances[index] += amount + state.validator_balances[validator_pubkeys.index(pubkey)] += amount ``` ### Routines for updating validator status From b7376aea5ca8b3c772f4b69f193724ea2a881106 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 10 Mar 2019 13:49:37 +0100 Subject: [PATCH 142/161] Disallow duplicate voluntary exits Stricter processing of voluntary exits to remove an edge case --- specs/core/0_beacon-chain.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c88155f77..4979eb91a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2460,7 +2460,9 @@ def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: """ validator = state.validator_registry[exit.validator_index] # Verify the validator has not yet exited - assert validator.exit_epoch > get_delayed_activation_exit_epoch(get_current_epoch(state)) + assert validator.exit_epoch == FAR_FUTURE_EPOCH + # Verify the validator has not initiated an exit + assert validator.initiated_exit is False # Exits must specify an epoch when they become valid; they are not valid before then assert get_current_epoch(state) >= exit.epoch # Verify signature From d425ea26e2cb5cabc5e6288dc2ca2f8c77753923 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 10 Mar 2019 13:50:28 +0100 Subject: [PATCH 143/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4979eb91a..0de38f219 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2453,7 +2453,7 @@ Verify that `len(block.body.voluntary_exits) <= MAX_VOLUNTARY_EXITS`. For each `exit` in `block.body.voluntary_exits`, run the following function: ```python -def process_exit(state: BeaconState, exit: VoluntaryExit) -> None: +def process_voluntary_exit(state: BeaconState, exit: VoluntaryExit) -> None: """ Process ``VoluntaryExit`` transaction. Note that this function mutates ``state``. From 2d3d7e33b3fed03b44f6bd0950b62e273099d037 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 10 Mar 2019 21:58:32 +0100 Subject: [PATCH 144/161] Weaken criterion for attestation inclusion The invariant that `get_current_epoch(state) in [get_current_epoch(state), get_previous_epoch(state)]` is preserved, as well as symmetry/fairness across blocks. --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 317d929c9..8bfc52993 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2370,7 +2370,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: """ # Can't submit attestations that are too far in history (or in prehistory) assert attestation.data.slot >= GENESIS_SLOT - assert state.slot < attestation.data.slot + SLOTS_PER_EPOCH + assert state.slot <= attestation.data.slot + SLOTS_PER_EPOCH # Can't submit attestations too quickly assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot # Verify that the justified epoch is correct, case 1: current epoch attestations From 0704297480b5706927999e4c337071a0e8d4abe4 Mon Sep 17 00:00:00 2001 From: Justin Date: Mon, 11 Mar 2019 17:28:39 +0100 Subject: [PATCH 145/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 25e8085de..5d824f288 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1303,21 +1303,6 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: # create an invalid Merkle branch, it may admit an invalid deposit # object, and we need to be able to skip over it state.deposit_index += 1 - - # Verify the proof of possession - proof_is_valid = bls_verify( - pubkey=deposit_input.pubkey, - message_hash=signed_root(deposit_input), - signature=deposit_input.proof_of_possession, - domain=get_domain( - state.fork, - get_current_epoch(state), - DOMAIN_DEPOSIT, - ) - ) - - if not proof_is_valid: - return validator_pubkeys = [v.pubkey for v in state.validator_registry] pubkey = deposit_input.pubkey @@ -1325,6 +1310,19 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: withdrawal_credentials = deposit_input.withdrawal_credentials if pubkey not in validator_pubkeys: + # Verify the proof of possession + if not bls_verify( + pubkey=deposit_input.pubkey, + message_hash=signed_root(deposit_input), + signature=deposit_input.proof_of_possession, + domain=get_domain( + state.fork, + get_current_epoch(state), + DOMAIN_DEPOSIT, + ) + ): + return + # Add new validator validator = Validator( pubkey=pubkey, From f06a3b82e7f97f12f1f70e07f5a09b854755b8a1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 11 Mar 2019 14:38:10 -0600 Subject: [PATCH 146/161] cache current and previous justified root in state --- specs/core/0_beacon-chain.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ceca50962..0a86db3f9 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -613,6 +613,8 @@ The types are defined topologically to aid in facilitating an executable version 'current_epoch_attestations': [PendingAttestation], 'previous_justified_epoch': 'uint64', 'justified_epoch': 'uint64', + 'previous_justified_root': 'bytes32', + 'justified_root': 'bytes32', 'justification_bitfield': 'uint64', 'finalized_epoch': 'uint64', @@ -1546,6 +1548,8 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], current_epoch_attestations=[], previous_justified_epoch=GENESIS_EPOCH, justified_epoch=GENESIS_EPOCH, + previous_justified_root=ZERO_HASH, + justified_root=ZERO_HASH, justification_bitfield=0, finalized_epoch=GENESIS_EPOCH, @@ -1830,7 +1834,9 @@ def update_justification_and_finalization(state: BeaconState) -> None: # Rotate justified epochs state.previous_justified_epoch = state.justified_epoch + state.previous_justified_root = state.justified_root state.justified_epoch = new_justified_epoch + state.justified_root = get_block_root(state, get_epoch_start_slot(new_justified_epoch)) ``` #### Crosslinks @@ -2374,16 +2380,15 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: assert state.slot <= attestation.data.slot + SLOTS_PER_EPOCH # Can't submit attestations too quickly assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot - # Verify that the justified epoch is correct, case 1: current epoch attestations + # Verify that the justified epoch/root is correct if slot_to_epoch(attestation.data.slot) >= get_current_epoch(state): + # Case 1: current epoch attestations assert attestation.data.justified_epoch == state.justified_epoch - # Case 2: previous epoch attestations + assert attestation.data.justified_block_root == state.justified_root else: + # Case 2: previous epoch attestations assert attestation.data.justified_epoch == state.previous_justified_epoch - # Check that the justified block root is correct - assert attestation.data.justified_block_root == get_block_root( - state, get_epoch_start_slot(attestation.data.justified_epoch) - ) + assert attestation.data.justified_block_root == state.previous_justified_root # Check that the crosslink data is valid acceptable_crosslink_data = { # Case 1: Latest crosslink matches the one in the state From 3916643ef6dc663f4c0da91efd86e351721bdf7d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 11 Mar 2019 18:23:17 -0600 Subject: [PATCH 147/161] only update justified epoch/root if changed --- specs/core/0_beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0a86db3f9..47dd765af 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1835,8 +1835,9 @@ def update_justification_and_finalization(state: BeaconState) -> None: # Rotate justified epochs state.previous_justified_epoch = state.justified_epoch state.previous_justified_root = state.justified_root - state.justified_epoch = new_justified_epoch - state.justified_root = get_block_root(state, get_epoch_start_slot(new_justified_epoch)) + if new_justified_epoch != state.justified_epoch: + state.justified_epoch = new_justified_epoch + state.justified_root = get_block_root(state, get_epoch_start_slot(new_justified_epoch)) ``` #### Crosslinks From 578bf02b6f76339b8a4fb21249326b4c46f7b2f4 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 12 Mar 2019 10:17:34 +0000 Subject: [PATCH 148/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 73 +++++++++++++++++------------------- 1 file changed, 35 insertions(+), 38 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 47dd765af..7eb9752bb 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -335,22 +335,19 @@ The types are defined topologically to aid in facilitating an executable version ```python { - # Slot number + # LMD GHOST vote 'slot': 'uint64', - # Shard number - 'shard': 'uint64', - # Root of the signed beacon block 'beacon_block_root': 'bytes32', - # Root of the ancestor at the epoch boundary - 'epoch_boundary_root': 'bytes32', - # Data from the shard since the last attestation + + # FFG vote + 'source_epoch': 'uint64', + 'source_root': 'bytes32', + 'target_root': 'bytes32', + + # Crosslink vote + 'shard': 'uint64', + 'previous_crosslink': Crosslink, 'crosslink_data_root': 'bytes32', - # Last crosslink - 'latest_crosslink': Crosslink, - # Last justified epoch in the beacon state - 'justified_epoch': 'uint64', - # Hash of the last justified beacon block - 'justified_block_root': 'bytes32', } ``` @@ -612,9 +609,9 @@ The types are defined topologically to aid in facilitating an executable version 'previous_epoch_attestations': [PendingAttestation], 'current_epoch_attestations': [PendingAttestation], 'previous_justified_epoch': 'uint64', - 'justified_epoch': 'uint64', + 'current_justified_epoch': 'uint64', 'previous_justified_root': 'bytes32', - 'justified_root': 'bytes32', + 'current_justified_root': 'bytes32', 'justification_bitfield': 'uint64', 'finalized_epoch': 'uint64', @@ -1225,8 +1222,8 @@ def is_surround_vote(attestation_data_1: AttestationData, """ Check if ``attestation_data_1`` surrounds ``attestation_data_2``. """ - source_epoch_1 = attestation_data_1.justified_epoch - source_epoch_2 = attestation_data_2.justified_epoch + source_epoch_1 = attestation_data_1.source_epoch + source_epoch_2 = attestation_data_2.source_epoch target_epoch_1 = slot_to_epoch(attestation_data_1.slot) target_epoch_2 = slot_to_epoch(attestation_data_2.slot) @@ -1547,9 +1544,9 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], previous_epoch_attestations=[], current_epoch_attestations=[], previous_justified_epoch=GENESIS_EPOCH, - justified_epoch=GENESIS_EPOCH, + current_justified_epoch=GENESIS_EPOCH, previous_justified_root=ZERO_HASH, - justified_root=ZERO_HASH, + current_justified_root=ZERO_HASH, justification_bitfield=0, finalized_epoch=GENESIS_EPOCH, @@ -1733,7 +1730,7 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat def get_current_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ a for a in state.current_epoch_attestations - if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) + if a.data.target_root == get_block_root(state, get_epoch_start_slot(get_current_epoch(state))) ] ``` @@ -1741,7 +1738,7 @@ def get_current_epoch_boundary_attestations(state: BeaconState) -> List[PendingA def get_previous_epoch_boundary_attestations(state: BeaconState) -> List[PendingAttestation]: return [ a for a in state.previous_epoch_attestations - if a.data.epoch_boundary_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) + if a.data.target_root == get_block_root(state, get_epoch_start_slot(get_previous_epoch(state))) ] ``` @@ -1759,7 +1756,7 @@ def get_previous_epoch_matching_head_attestations(state: BeaconState) -> List[Pe def get_winning_root_and_participants(state: BeaconState, shard: Shard) -> Tuple[Bytes32, List[ValidatorIndex]]: all_attestations = state.current_epoch_attestations + state.previous_epoch_attestations valid_attestations = [ - a for a in all_attestations if a.data.latest_crosslink == state.latest_crosslinks[shard] + a for a in all_attestations if a.data.previous_crosslink == state.latest_crosslinks[shard] ] all_roots = [a.data.crosslink_data_root for a in valid_attestations] @@ -1802,7 +1799,7 @@ Run the following function: ```python def update_justification_and_finalization(state: BeaconState) -> None: - new_justified_epoch = state.justified_epoch + new_justified_epoch = state.current_justified_epoch # Rotate the justification bitfield up one epoch to make room for the current epoch state.justification_bitfield <<= 1 # If the previous epoch gets justified, fill the second last bit @@ -1826,18 +1823,18 @@ def update_justification_and_finalization(state: BeaconState) -> None: if (bitfield >> 1) % 4 == 0b11 and state.previous_justified_epoch == current_epoch - 2: state.finalized_epoch = state.previous_justified_epoch # The 1st/2nd/3rd most recent epochs are all justified, the 1st using the 3rd as source - if (bitfield >> 0) % 8 == 0b111 and state.justified_epoch == current_epoch - 2: - state.finalized_epoch = state.justified_epoch + if (bitfield >> 0) % 8 == 0b111 and state.current_justified_epoch == current_epoch - 2: + state.finalized_epoch = state.current_justified_epoch # The 1st/2nd most recent epochs are both justified, the 1st using the 2nd as source - if (bitfield >> 0) % 4 == 0b11 and state.justified_epoch == current_epoch - 1: - state.finalized_epoch = state.justified_epoch + if (bitfield >> 0) % 4 == 0b11 and state.current_justified_epoch == current_epoch - 1: + state.finalized_epoch = state.current_justified_epoch # Rotate justified epochs - state.previous_justified_epoch = state.justified_epoch - state.previous_justified_root = state.justified_root - if new_justified_epoch != state.justified_epoch: - state.justified_epoch = new_justified_epoch - state.justified_root = get_block_root(state, get_epoch_start_slot(new_justified_epoch)) + state.previous_justified_epoch = state.current_justified_epoch + state.previous_justified_root = state.current_justified_root + if new_justified_epoch != state.current_justified_epoch: + state.current_justified_epoch = new_justified_epoch + state.current_justified_root = get_block_root(state, get_epoch_start_slot(new_justified_epoch)) ``` #### Crosslinks @@ -2381,19 +2378,19 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: assert state.slot <= attestation.data.slot + SLOTS_PER_EPOCH # Can't submit attestations too quickly assert attestation.data.slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot - # Verify that the justified epoch/root is correct + # Verify that the justified epoch and root is correct if slot_to_epoch(attestation.data.slot) >= get_current_epoch(state): # Case 1: current epoch attestations - assert attestation.data.justified_epoch == state.justified_epoch - assert attestation.data.justified_block_root == state.justified_root + assert attestation.data.source_epoch == state.current_justified_epoch + assert attestation.data.source_root == state.current_justified_root else: # Case 2: previous epoch attestations - assert attestation.data.justified_epoch == state.previous_justified_epoch - assert attestation.data.justified_block_root == state.previous_justified_root + assert attestation.data.source_epoch == state.previous_justified_epoch + assert attestation.data.source_root == state.previous_justified_root # Check that the crosslink data is valid acceptable_crosslink_data = { # Case 1: Latest crosslink matches the one in the state - attestation.data.latest_crosslink, + attestation.data.previous_crosslink, # Case 2: State has already been updated, state's latest crosslink matches the crosslink # the attestation is trying to create Crosslink( From addf7b77ab13cfae82ae72d5a20986fb865ffd27 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 12 Mar 2019 10:26:34 +0000 Subject: [PATCH 149/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 7eb9752bb..319f826af 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -614,6 +614,7 @@ The types are defined topologically to aid in facilitating an executable version 'current_justified_root': 'bytes32', 'justification_bitfield': 'uint64', 'finalized_epoch': 'uint64', + 'finalized_root': 'bytes32', # Recent state 'latest_crosslinks': [Crosslink, SHARD_COUNT], @@ -1828,6 +1829,7 @@ def update_justification_and_finalization(state: BeaconState) -> None: # The 1st/2nd most recent epochs are both justified, the 1st using the 2nd as source if (bitfield >> 0) % 4 == 0b11 and state.current_justified_epoch == current_epoch - 1: state.finalized_epoch = state.current_justified_epoch + state.finalized_root = get_block_root(state, get_epoch_start_slot(finalized_epoch)) # Rotate justified epochs state.previous_justified_epoch = state.current_justified_epoch From 2e6c5171175bbb9f4d01ab229afda3246d7b6be4 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 12 Mar 2019 11:59:08 +0000 Subject: [PATCH 150/161] CC0 1.0 Universal for repo --- LICENSE | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 000000000..670154e35 --- /dev/null +++ b/LICENSE @@ -0,0 +1,116 @@ +CC0 1.0 Universal + +Statement of Purpose + +The laws of most jurisdictions throughout the world automatically confer +exclusive Copyright and Related Rights (defined below) upon the creator and +subsequent owner(s) (each and all, an "owner") of an original work of +authorship and/or a database (each, a "Work"). + +Certain owners wish to permanently relinquish those rights to a Work for the +purpose of contributing to a commons of creative, cultural and scientific +works ("Commons") that the public can reliably and without fear of later +claims of infringement build upon, modify, incorporate in other works, reuse +and redistribute as freely as possible in any form whatsoever and for any +purposes, including without limitation commercial purposes. These owners may +contribute to the Commons to promote the ideal of a free culture and the +further production of creative, cultural and scientific works, or to gain +reputation or greater distribution for their Work in part through the use and +efforts of others. + +For these and/or other purposes and motivations, and without any expectation +of additional consideration or compensation, the person associating CC0 with a +Work (the "Affirmer"), to the extent that he or she is an owner of Copyright +and Related Rights in the Work, voluntarily elects to apply CC0 to the Work +and publicly distribute the Work under its terms, with knowledge of his or her +Copyright and Related Rights in the Work and the meaning and intended legal +effect of CC0 on those rights. + +1. Copyright and Related Rights. A Work made available under CC0 may be +protected by copyright and related or neighboring rights ("Copyright and +Related Rights"). Copyright and Related Rights include, but are not limited +to, the following: + + i. the right to reproduce, adapt, distribute, perform, display, communicate, + and translate a Work; + + ii. moral rights retained by the original author(s) and/or performer(s); + + iii. publicity and privacy rights pertaining to a person's image or likeness + depicted in a Work; + + iv. rights protecting against unfair competition in regards to a Work, + subject to the limitations in paragraph 4(a), below; + + v. rights protecting the extraction, dissemination, use and reuse of data in + a Work; + + vi. database rights (such as those arising under Directive 96/9/EC of the + European Parliament and of the Council of 11 March 1996 on the legal + protection of databases, and under any national implementation thereof, + including any amended or successor version of such directive); and + + vii. other similar, equivalent or corresponding rights throughout the world + based on applicable law or treaty, and any national implementations thereof. + +2. Waiver. To the greatest extent permitted by, but not in contravention of, +applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and +unconditionally waives, abandons, and surrenders all of Affirmer's Copyright +and Related Rights and associated claims and causes of action, whether now +known or unknown (including existing as well as future claims and causes of +action), in the Work (i) in all territories worldwide, (ii) for the maximum +duration provided by applicable law or treaty (including future time +extensions), (iii) in any current or future medium and for any number of +copies, and (iv) for any purpose whatsoever, including without limitation +commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes +the Waiver for the benefit of each member of the public at large and to the +detriment of Affirmer's heirs and successors, fully intending that such Waiver +shall not be subject to revocation, rescission, cancellation, termination, or +any other legal or equitable action to disrupt the quiet enjoyment of the Work +by the public as contemplated by Affirmer's express Statement of Purpose. + +3. Public License Fallback. Should any part of the Waiver for any reason be +judged legally invalid or ineffective under applicable law, then the Waiver +shall be preserved to the maximum extent permitted taking into account +Affirmer's express Statement of Purpose. In addition, to the extent the Waiver +is so judged Affirmer hereby grants to each affected person a royalty-free, +non transferable, non sublicensable, non exclusive, irrevocable and +unconditional license to exercise Affirmer's Copyright and Related Rights in +the Work (i) in all territories worldwide, (ii) for the maximum duration +provided by applicable law or treaty (including future time extensions), (iii) +in any current or future medium and for any number of copies, and (iv) for any +purpose whatsoever, including without limitation commercial, advertising or +promotional purposes (the "License"). The License shall be deemed effective as +of the date CC0 was applied by Affirmer to the Work. Should any part of the +License for any reason be judged legally invalid or ineffective under +applicable law, such partial invalidity or ineffectiveness shall not +invalidate the remainder of the License, and in such case Affirmer hereby +affirms that he or she will not (i) exercise any of his or her remaining +Copyright and Related Rights in the Work or (ii) assert any associated claims +and causes of action with respect to the Work, in either case contrary to +Affirmer's express Statement of Purpose. + +4. Limitations and Disclaimers. + + a. No trademark or patent rights held by Affirmer are waived, abandoned, + surrendered, licensed or otherwise affected by this document. + + b. Affirmer offers the Work as-is and makes no representations or warranties + of any kind concerning the Work, express, implied, statutory or otherwise, + including without limitation warranties of title, merchantability, fitness + for a particular purpose, non infringement, or the absence of latent or + other defects, accuracy, or the present or absence of errors, whether or not + discoverable, all to the greatest extent permissible under applicable law. + + c. Affirmer disclaims responsibility for clearing rights of other persons + that may apply to the Work or any use thereof, including without limitation + any person's Copyright and Related Rights in the Work. Further, Affirmer + disclaims responsibility for obtaining any necessary consents, permissions + or other rights required for any use of the Work. + + d. Affirmer understands and acknowledges that Creative Commons is not a + party to this document and has no duty or obligation with respect to this + CC0 or use of the Work. + +For more information, please see + From 64ba3a31073346d278b66f42bc656ce214580043 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 12 Mar 2019 12:24:37 +0000 Subject: [PATCH 151/161] Epoch-based proposer slashing See #675 item 25. --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ceca50962..cfb87e91e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2306,7 +2306,7 @@ def process_proposer_slashing(state: BeaconState, """ proposer = state.validator_registry[proposer_slashing.proposer_index] # Verify that the slot is the same - assert proposer_slashing.header_1.slot == proposer_slashing.header_2.slot + assert slot_to_epoch(proposer_slashing.header_1.slot) == slot_to_epoch(proposer_slashing.header_2.slot) # But the headers are different assert proposer_slashing.header_1 != proposer_slashing.header_2 # Proposer is not yet slashed From d8a3048f2c119687e92072fe797fa1d377b3948f Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 12 Mar 2019 12:36:09 +0000 Subject: [PATCH 152/161] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 900708e7a..b2696892f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -268,12 +268,12 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | Name | Value | | - | - | -| `DOMAIN_BEACON_BLOCK` | `int_to_bytes4(0)` | -| `DOMAIN_RANDAO` | `int_to_bytes4(1)` | -| `DOMAIN_ATTESTATION` | `int_to_bytes4(2)` | -| `DOMAIN_DEPOSIT` | `int_to_bytes4(3)` | -| `DOMAIN_VOLUNTARY_EXIT` | `int_to_bytes4(4)` | -| `DOMAIN_TRANSFER` | `int_to_bytes4(5)` | +| `DOMAIN_BEACON_BLOCK` | `0` | +| `DOMAIN_RANDAO` | `1` | +| `DOMAIN_ATTESTATION` | `2` | +| `DOMAIN_DEPOSIT` | `3` | +| `DOMAIN_VOLUNTARY_EXIT` | `4` | +| `DOMAIN_TRANSFER` | `5` | ## Data structures @@ -1107,7 +1107,7 @@ def get_total_balance(state: BeaconState, validators: List[ValidatorIndex]) -> G ```python def get_fork_version(fork: Fork, - epoch: Epoch) -> bytes4: + epoch: Epoch) -> bytes: """ Return the fork version of the given ``epoch``. """ @@ -1122,11 +1122,11 @@ def get_fork_version(fork: Fork, ```python def get_domain(fork: Fork, epoch: Epoch, - domain_type: bytes4) -> int: + domain_type: int) -> int: """ Get the domain number that represents the fork meta and signature domain. """ - return bytes_to_int(get_fork_version(fork, epoch) + domain_type)) + return bytes_to_int(get_fork_version(fork, epoch) + int_to_bytes4(domain_type)) ``` ### `get_bitfield_bit` From 0f120415b5f4d7da9dbd88ecb38bc80952f8b30e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 12 Mar 2019 16:49:04 +0100 Subject: [PATCH 153/161] Update specs/core/0_beacon-chain.md Co-Authored-By: JustinDrake --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index cfb87e91e..9571c23db 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2305,7 +2305,7 @@ def process_proposer_slashing(state: BeaconState, Note that this function mutates ``state``. """ proposer = state.validator_registry[proposer_slashing.proposer_index] - # Verify that the slot is the same + # Verify that the epoch is the same assert slot_to_epoch(proposer_slashing.header_1.slot) == slot_to_epoch(proposer_slashing.header_2.slot) # But the headers are different assert proposer_slashing.header_1 != proposer_slashing.header_2 From 25f6647ef2a5a01a9000e658b276b609fb03479d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 12 Mar 2019 11:07:20 -0600 Subject: [PATCH 154/161] minor formatting --- specs/core/0_beacon-chain.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a76da5ec3..b4554e0fd 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1288,7 +1288,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: serialized_deposit_data = serialize(deposit.deposit_data) # Deposits must be processed in order assert deposit.index == state.deposit_index - + # Verify the Merkle branch merkle_branch_is_valid = verify_merkle_branch( leaf=hash(serialized_deposit_data), @@ -1298,7 +1298,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: root=state.latest_eth1_data.deposit_root, ) assert merkle_branch_is_valid - + # Increment the next deposit index we are expecting. Note that this # needs to be done here because while the deposit contract will never # create an invalid Merkle branch, it may admit an invalid deposit @@ -1312,7 +1312,7 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: if pubkey not in validator_pubkeys: # Verify the proof of possession - if not bls_verify( + proof_is_valid = bls_verify( pubkey=deposit_input.pubkey, message_hash=signed_root(deposit_input), signature=deposit_input.proof_of_possession, @@ -1321,7 +1321,8 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: get_current_epoch(state), DOMAIN_DEPOSIT, ) - ): + ) + if not proof_is_valid: return # Add new validator From a68b050053bcb08b5c0faa14348ecc88773dd137 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 12 Mar 2019 11:35:59 -0600 Subject: [PATCH 155/161] set fork values in genesis as bytes4 --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 21bf3e7a6..bb3d5269c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1517,8 +1517,8 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], slot=GENESIS_SLOT, genesis_time=genesis_time, fork=Fork( - previous_version=GENESIS_FORK_VERSION, - current_version=GENESIS_FORK_VERSION, + previous_version=int_to_bytes4(GENESIS_FORK_VERSION), + current_version=int_to_bytes4(GENESIS_FORK_VERSION), epoch=GENESIS_EPOCH, ), From e8a5cd074766c3dfe6d424e8a08572bdae9a7445 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 12 Mar 2019 12:32:11 -0600 Subject: [PATCH 156/161] fix up for tests --- specs/core/0_beacon-chain.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 82505a220..bd0187a9f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1545,6 +1545,7 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], current_justified_root=ZERO_HASH, justification_bitfield=0, finalized_epoch=GENESIS_EPOCH, + finalized_root=ZERO_HASH, # Recent state latest_crosslinks=[Crosslink(epoch=GENESIS_EPOCH, crosslink_data_root=ZERO_HASH) for _ in range(SHARD_COUNT)], @@ -1796,6 +1797,8 @@ Run the following function: ```python def update_justification_and_finalization(state: BeaconState) -> None: new_justified_epoch = state.current_justified_epoch + new_finalized_epoch = state.finalized_epoch + # Rotate the justification bitfield up one epoch to make room for the current epoch state.justification_bitfield <<= 1 # If the previous epoch gets justified, fill the second last bit @@ -1814,24 +1817,26 @@ def update_justification_and_finalization(state: BeaconState) -> None: current_epoch = get_current_epoch(state) # The 2nd/3rd/4th most recent epochs are all justified, the 2nd using the 4th as source if (bitfield >> 1) % 8 == 0b111 and state.previous_justified_epoch == current_epoch - 3: - state.finalized_epoch = state.previous_justified_epoch + new_finalized_epoch = state.previous_justified_epoch # The 2nd/3rd most recent epochs are both justified, the 2nd using the 3rd as source if (bitfield >> 1) % 4 == 0b11 and state.previous_justified_epoch == current_epoch - 2: - state.finalized_epoch = state.previous_justified_epoch + new_finalized_epoch = state.previous_justified_epoch # The 1st/2nd/3rd most recent epochs are all justified, the 1st using the 3rd as source if (bitfield >> 0) % 8 == 0b111 and state.current_justified_epoch == current_epoch - 2: - state.finalized_epoch = state.current_justified_epoch + new_finalized_epoch = state.current_justified_epoch # The 1st/2nd most recent epochs are both justified, the 1st using the 2nd as source if (bitfield >> 0) % 4 == 0b11 and state.current_justified_epoch == current_epoch - 1: - state.finalized_epoch = state.current_justified_epoch - state.finalized_root = get_block_root(state, get_epoch_start_slot(finalized_epoch)) + new_finalized_epoch = state.current_justified_epoch - # Rotate justified epochs + # Update state jusification/finality fields state.previous_justified_epoch = state.current_justified_epoch state.previous_justified_root = state.current_justified_root if new_justified_epoch != state.current_justified_epoch: state.current_justified_epoch = new_justified_epoch state.current_justified_root = get_block_root(state, get_epoch_start_slot(new_justified_epoch)) + if new_finalized_epoch != state.finalized_epoch: + state.finalized_epoch = new_finalized_epoch + state.finalized_root = get_block_root(state, get_epoch_start_slot(new_finalized_epoch)) ``` #### Crosslinks From b40236685c1fd8bf016761dfa861e05686dc6d1f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 13 Mar 2019 09:04:12 -0600 Subject: [PATCH 157/161] phase 1 nitpicks --- specs/core/1_shard-data-chains.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index db68591e7..d8efe0c85 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -556,7 +556,7 @@ def verify_custody_subkey_reveal(pubkey: bytes48, ```python def verify_signed_challenge_message(message: Any, pubkey: bytes48) -> bool: return bls_verify( - message_hash=signed_root(message, 'signature'), + message_hash=signed_root(message), pubkey=pubkey, signature=message.signature, domain=get_domain(state, get_current_epoch(state), DOMAIN_CUSTODY_INTERACTIVE) @@ -607,8 +607,8 @@ Verify that `len(block.body.branch_challenges) <= MAX_BRANCH_CHALLENGES`. For each `challenge` in `block.body.branch_challenges`, run: ```python -def process_branch_challenge(challenge: BranchChallenge, - state: BeaconState): +def process_branch_challenge(state: BeaconState, + challenge: BranchChallenge): # Check that it's not too late to challenge assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY assert state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY @@ -643,8 +643,8 @@ Verify that `len(block.body.branch_responses) <= MAX_BRANCH_RESPONSES`. For each `response` in `block.body.branch_responses`, if `response.responding_to_custody_challenge == False`, run: ```python -def process_branch_exploration_response(response: BranchResponse, - state: BeaconState): +def process_branch_exploration_response(state: BeaconState, + response: BranchResponse): challenge = get_branch_challenge_record_by_id(response.challenge_id) assert verify_merkle_branch( leaf=response.data, @@ -664,8 +664,8 @@ def process_branch_exploration_response(response: BranchResponse, If `response.responding_to_custody_challenge == True`, run: ```python -def process_branch_custody_response(response: BranchResponse, - state: BeaconState): +def process_branch_custody_response(state: BeaconState, + response: BranchResponse): challenge = get_custody_challenge_record_by_id(response.challenge_id) responder = state.validator_registry[challenge.responder_index] # Verify we're not too late @@ -718,8 +718,8 @@ Verify that `len(block.body.interactive_custody_challenge_initiations) <= MAX_IN For each `initiation` in `block.body.interactive_custody_challenge_initiations`, use the following function to process it: ```python -def process_initiation(initiation: InteractiveCustodyChallengeInitiation, - state: BeaconState): +def process_initiation(state: BeaconState, + initiation: InteractiveCustodyChallengeInitiation): challenger = state.validator_registry[initiation.challenger_index] responder = state.validator_registry[initiation.responder_index] # Verify the signature @@ -771,8 +771,8 @@ Verify that `len(block.body.interactive_custody_challenge_responses) <= MAX_INTE For each `response` in `block.body.interactive_custody_challenge_responses`, use the following function to process it: ```python -def process_response(response: InteractiveCustodyChallengeResponse, - state: State): +def process_response(state: BeaconState, + response: InteractiveCustodyChallengeResponse): challenge = get_custody_challenge_record_by_id(state, response.challenge_id) responder = state.validator_registry[challenge.responder_index] # Check that the right number of hashes was provided @@ -804,8 +804,8 @@ Verify that `len(block.body.interactive_custody_challenge_continuations) <= MAX_ For each `continuation` in `block.body.interactive_custody_challenge_continuations`, use the following function to process it: ```python -def process_continuation(continuation: InteractiveCustodyChallengeContinuation, - state: State): +def process_continuation(state: BeaconState, + continuation: InteractiveCustodyChallengeContinuation): challenge = get_custody_challenge_record_by_id(state, continuation.challenge_id) challenger = state.validator_registry[challenge.challenger_index] responder = state.validator_registry[challenge.responder_index] From cdd59ae2309dddbe211418e03e9ccc459b4a7ddb Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 13 Mar 2019 09:11:35 -0600 Subject: [PATCH 158/161] add return types to phase 1 functions Co-Authored-By: djrtwo --- specs/core/1_shard-data-chains.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index d8efe0c85..1713c6cbf 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -608,7 +608,7 @@ For each `challenge` in `block.body.branch_challenges`, run: ```python def process_branch_challenge(state: BeaconState, - challenge: BranchChallenge): + challenge: BranchChallenge) -> None: # Check that it's not too late to challenge assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY assert state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY @@ -644,7 +644,7 @@ For each `response` in `block.body.branch_responses`, if `response.responding_to ```python def process_branch_exploration_response(state: BeaconState, - response: BranchResponse): + response: BranchResponse) -> None: challenge = get_branch_challenge_record_by_id(response.challenge_id) assert verify_merkle_branch( leaf=response.data, @@ -665,7 +665,7 @@ If `response.responding_to_custody_challenge == True`, run: ```python def process_branch_custody_response(state: BeaconState, - response: BranchResponse): + response: BranchResponse) -> None: challenge = get_custody_challenge_record_by_id(response.challenge_id) responder = state.validator_registry[challenge.responder_index] # Verify we're not too late @@ -719,7 +719,7 @@ For each `initiation` in `block.body.interactive_custody_challenge_initiations`, ```python def process_initiation(state: BeaconState, - initiation: InteractiveCustodyChallengeInitiation): + initiation: InteractiveCustodyChallengeInitiation) -> None: challenger = state.validator_registry[initiation.challenger_index] responder = state.validator_registry[initiation.responder_index] # Verify the signature @@ -772,7 +772,7 @@ For each `response` in `block.body.interactive_custody_challenge_responses`, use ```python def process_response(state: BeaconState, - response: InteractiveCustodyChallengeResponse): + response: InteractiveCustodyChallengeResponse) -> None: challenge = get_custody_challenge_record_by_id(state, response.challenge_id) responder = state.validator_registry[challenge.responder_index] # Check that the right number of hashes was provided @@ -805,7 +805,7 @@ For each `continuation` in `block.body.interactive_custody_challenge_continuatio ```python def process_continuation(state: BeaconState, - continuation: InteractiveCustodyChallengeContinuation): + continuation: InteractiveCustodyChallengeContinuation) -> None: challenge = get_custody_challenge_record_by_id(state, continuation.challenge_id) challenger = state.validator_registry[challenge.challenger_index] responder = state.validator_registry[challenge.responder_index] From 0e837c3386bbb919610665fe70ca747fb4fb9afe Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 13 Mar 2019 12:17:21 -0600 Subject: [PATCH 159/161] update v-guide to v0.5.0 --- specs/validator/0_beacon-chain-validator.md | 78 ++++++++++----------- 1 file changed, 37 insertions(+), 41 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 0c95fb446..7293675f1 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -101,8 +101,7 @@ In phase 0, all incoming validator deposits originate from the Ethereum 1.0 PoW To submit a deposit: * Pack the validator's [initialization parameters](#initialization) into `deposit_input`, a [`DepositInput`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#depositinput) SSZ object. -* Set `deposit_input.proof_of_possession = EMPTY_SIGNATURE`. -* Let `proof_of_possession` be the result of `bls_sign` of the `hash_tree_root(deposit_input)` with `domain=DOMAIN_DEPOSIT`. +* Let `proof_of_possession` be the result of `bls_sign` of the `signed_root(deposit_input)` with `domain=DOMAIN_DEPOSIT`. * Set `deposit_input.proof_of_possession = proof_of_possession`. * Let `amount` be the amount in Gwei to be deposited by the validator where `MIN_DEPOSIT_AMOUNT <= amount <= MAX_DEPOSIT_AMOUNT`. * Send a transaction on the Ethereum 1.0 chain to `DEPOSIT_CONTRACT_ADDRESS` executing `deposit` along with `serialize(deposit_input)` as the singular `bytes` input along with a deposit `amount` in Gwei. @@ -121,11 +120,12 @@ Once a validator has been processed and added to the beacon state's `validator_r In normal operation, the validator is quickly activated at which point the validator is added to the shuffling and begins validation after an additional `ACTIVATION_EXIT_DELAY` epochs (25.6 minutes). -The function [`is_active_validator`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_active_validator) can be used to check if a validator is active during a given epoch. Usage is as follows: +The function [`is_active_validator`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_active_validator) can be used to check if a validator is active during a given shuffling epoch. Note that the `BeaconState` contains a field `current_shuffling_epoch` which dictates from which epoch the current active validators are taken. Usage is as follows: ```python +shuffling_epoch = state.current_shuffling_epoch validator = state.validator_registry[validator_index] -is_active = is_active_validator(validator, epoch) +is_active = is_active_validator(validator, shuffling_epoch) ``` Once a validator is activated, the validator is assigned [responsibilities](#beacon-chain-responsibilities) until exited. @@ -138,7 +138,7 @@ A validator has two primary responsibilities to the beacon chain -- [proposing b ### Block proposal -A validator is expected to propose a [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `get_beacon_proposer_index(state, slot)` returns the validator's `validator_index`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beacon-chain-state-transition-function). +A validator is expected to propose a [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `get_beacon_proposer_index(state, slot)` returns the validator's `validator_index`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot - 1`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beacon-chain-state-transition-function). There is one proposer per slot, so if there are N active validators any individual validator will on average be assigned to propose once per N slots (eg. at 312500 validators = 10 million ETH, that's once per ~3 weeks). @@ -152,13 +152,13 @@ _Note:_ there might be "skipped" slots between the `parent` and `block`. These s ##### Parent root -Set `block.parent_root = hash_tree_root(parent)`. +Set `block.previous_block_root = hash_tree_root(parent)`. ##### State root Set `block.state_root = hash_tree_root(state)` of the resulting `state` of the `parent -> block` state transition. -_Note_: To calculate `state_root`, the validator should first run the state transition function on an unsigned `block` containing a stub for the `state_root`. It is useful to be able to run a state transition function that does _not_ validate signatures for this purpose. +_Note_: To calculate `state_root`, the validator should first run the state transition function on an unsigned `block` containing a stub for the `state_root`. It is useful to be able to run a state transition function that does _not_ validate signatures or state root for this purpose. ##### Randao reveal @@ -166,8 +166,8 @@ Set `block.randao_reveal = epoch_signature` where `epoch_signature` is defined a ```python epoch_signature = bls_sign( - privkey=validator.privkey, # privkey store locally, not in state - message_hash=int_to_bytes32(slot_to_epoch(block.slot)), + privkey=validator.privkey, # privkey stored locally, not in state + message_hash=hash_tree_root(slot_to_epoch(block.slot)), domain=get_domain( fork=fork, # `fork` is the fork object at the slot `block.slot` epoch=slot_to_epoch(block.slot), @@ -194,23 +194,16 @@ epoch_signature = bls_sign( ##### Signature -Set `block.signature = signed_proposal_data` where `signed_proposal_data` is defined as: +Set `block.signature = block_signature` where `block_signature` is defined as: ```python -proposal_data = ProposalSignedData( - slot=slot, - shard=BEACON_CHAIN_SHARD_NUMBER, - block_root=hash_tree_root(block), # where `block.sigature == EMPTY_SIGNATURE -) -proposal_root = hash_tree_root(proposal_data) - -signed_proposal_data = bls_sign( +block_signature = bls_sign( privkey=validator.privkey, # privkey store locally, not in state - message_hash=proposal_root, + message_hash=signed_root(block), domain=get_domain( fork=fork, # `fork` is the fork object at the slot `block.slot` epoch=slot_to_epoch(block.slot), - domain_type=DOMAIN_PROPOSAL, + domain_type=DOMAIN_BEACON_BLOCK, ) ) ``` @@ -227,12 +220,14 @@ Up to `MAX_ATTESTER_SLASHINGS` [`AttesterSlashing`](https://github.com/ethereum/ ##### Attestations -Up to `MAX_ATTESTATIONS` aggregate attestations can be included in the `block`. The attestations added must satisfy the verification conditions found in [attestation processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestations-1). To maximize profit, the validator should attempt to create aggregate attestations that include singular attestations from the largest number of validators whose signatures from the same epoch have not previously been added on chain. +Up to `MAX_ATTESTATIONS` aggregate attestations can be included in the `block`. The attestations added must satisfy the verification conditions found in [attestation processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestations-1). To maximize profit, the validator should attempt to gather aggregate attestations that include singular attestations from the largest number of validators whose signatures from the same epoch have not previously been added on chain. ##### Deposits Up to `MAX_DEPOSITS` [`Deposit`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#deposit) objects can be included in the `block`. These deposits are constructed from the `Deposit` logs from the [Eth1.0 deposit contract](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#ethereum-10-deposit-contract) and must be processed in sequential order. The deposits included in the `block` must satisfy the verification conditions found in [deposits processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#deposits-1). +The `proof` for each deposit must be constructed against the deposit root contained in `state.latest_eth1_data` rather than the deposit root at the time the deposit was initially logged from the 1.0 chain. This entails storing a full deposit merkle tree locally and computing updated proofs against the `latest_eth1_data.deposit_root` as needed. See [`minimal_merkle.py`](https://github.com/ethereum/research/blob/master/spec_pythonizer/utils/merkle_minimal.py) for a sample implementation. + ##### Voluntary exits Up to `MAX_VOLUNTARY_EXITS` [`VoluntaryExit`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#voluntaryexit) objects can be included in the `block`. The exits must satisfy the verification conditions found in [exits processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#exits-1). @@ -247,9 +242,12 @@ A validator should create and broadcast the attestation halfway through the `slo First the validator should construct `attestation_data`, an [`AttestationData`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestationdata) object based upon the state at the assigned slot. +* Let `head_block` be the result of running the fork choice during the assigned slot. +* Let `head_state` be the state of `head_block` processed through any empty slots up to the assigned slot. + ##### Slot -Set `attestation_data.slot = slot` where `slot` is the current slot of which the validator is a member of a committee. +Set `attestation_data.slot = head_state.slot`. ##### Shard @@ -257,15 +255,15 @@ Set `attestation_data.shard = shard` where `shard` is the shard associated with ##### Beacon block root -Set `attestation_data.beacon_block_root = hash_tree_root(head)` where `head` is the validator's view of the `head` block of the beacon chain during `slot`. +Set `attestation_data.beacon_block_root = hash_tree_root(head_block)`. -##### Epoch boundary root +##### Target root -Set `attestation_data.epoch_boundary_root = hash_tree_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary in the chain defined by `head` -- i.e. the `BeaconBlock` where `block.slot == get_epoch_start_slot(slot_to_epoch(head.slot))`. +Set `attestation_data.target_root = hash_tree_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary. _Note:_ This can be looked up in the state using: -* Let `epoch_start_slot = get_epoch_start_slot(slot_to_epoch(head.slot))`. -* Set `epoch_boundary_root = hash_tree_root(head) if epoch_start_slot == head.slot else get_block_root(state, epoch_start_slot)`. +* Let `epoch_start_slot = get_epoch_start_slot(get_current_epoch(head_state))`. +* Set `epoch_boundary = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`. ##### Crosslink data root @@ -275,17 +273,15 @@ _Note:_ This is a stub for phase 0. ##### Latest crosslink -Set `attestation_data.latest_crosslink = state.latest_crosslinks[shard]` where `state` is the beacon state at `head` and `shard` is the validator's assigned shard. +Set `attestation_data.previous_crosslink = head_state.latest_crosslinks[shard]`. -##### Justified epoch +##### Source epoch -Set `attestation_data.justified_epoch = state.justified_epoch` where `state` is the beacon state at `head`. +Set `attestation_data.source_epoch = head_state.justified_epoch`. -##### Justified block root +##### Source root -Set `attestation_data.justified_block_root = hash_tree_root(justified_block)` where `justified_block` is the block at the slot `get_epoch_start_slot(state.justified_epoch)` in the chain defined by `head`. - -_Note:_ This can be looked up in the state using `get_block_root(state, get_epoch_start_slot(state.justified_epoch))`. +Set `attestation_data.source_root = head_state.current_justified_root`. #### Construct attestation @@ -320,11 +316,11 @@ attestation_data_and_custody_bit = AttestationDataAndCustodyBit( data=attestation.data, custody_bit=0b0, ) -attestation_message_to_sign = hash_tree_root(attestation_data_and_custody_bit) +attestation_message = hash_tree_root(attestation_data_and_custody_bit) signed_attestation_data = bls_sign( - privkey=validator.privkey, # privkey store locally, not in state - message_hash=attestation_message_to_sign, + privkey=validator.privkey, # privkey stored locally, not in state + message_hash=attestation_message, domain=get_domain( fork=fork, # `fork` is the fork object at the slot, `attestation_data.slot` epoch=slot_to_epoch(attestation_data.slot), @@ -402,12 +398,12 @@ _Note_: Signed data must be within a sequential `Fork` context to conflict. Mess ### Proposer slashing -To avoid "proposer slashings", a validator must not sign two conflicting [`ProposalSignedData`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#proposalsigneddata) where conflicting is defined as having the same `slot` and `shard` but a different `block_root`. In phase 0, proposals are only made for the beacon chain (`shard == BEACON_CHAIN_SHARD_NUMBER`). +To avoid "proposer slashings", a validator must not sign two conflicting [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#proposalsigneddata) where conflicting is defined as two distinct blocks within the same epoch. -_In phase 0, as long as the validator does not sign two different beacon chain proposals for the same slot, the validator is safe against proposer slashings._ +_In phase 0, as long as the validator does not sign two different beacon blocks for the same epoch, the validator is safe against proposer slashings._ Specifically, when signing an `BeaconBlock`, a validator should perform the following steps in the following order: -1. Save a record to hard disk that an beacon block has been signed for the `slot=slot` and `shard=BEACON_CHAIN_SHARD_NUMBER`. +1. Save a record to hard disk that an beacon block has been signed for the `epoch=slot_to_epoch(block.slot)`. 2. Generate and broadcast the block. If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the _potentially_ signed/broadcast block and can effectively avoid slashing. @@ -417,7 +413,7 @@ If the software crashes at some point within this routine, then when the validat To avoid "attester slashings", a validator must not sign two conflicting [`AttestationData`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestationdata) objects where conflicting is defined as a set of two attestations that satisfy either [`is_double_vote`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_double_vote) or [`is_surround_vote`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_surround_vote). Specifically, when signing an `Attestation`, a validator should perform the following steps in the following order: -1. Save a record to hard disk that an attestation has been signed for source -- `attestation_data.justified_epoch` -- and target -- `slot_to_epoch(attestation_data.slot)`. +1. Save a record to hard disk that an attestation has been signed for source -- `attestation_data.source_epoch` -- and target -- `slot_to_epoch(attestation_data.slot)`. 2. Generate and broadcast attestation. If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the _potentially_ signed/broadcast attestation and can effectively avoid slashing. From 9774a3d5811396c609ffa1b74d0f6e4f3a642d02 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 13 Mar 2019 17:01:47 -0700 Subject: [PATCH 160/161] Helper function returns correct type of `Gwei` instead of indices --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index bd0187a9f..daa1bc108 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1719,7 +1719,7 @@ def get_attesting_indices(state: BeaconState, attestations: List[PendingAttestat ``` ```python -def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: +def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestation]) -> Gwei: return get_total_balance(state, get_attesting_indices(state, attestations)) ``` From c30018a71657b62f6c088f19eb85a721f0641305 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Wed, 13 Mar 2019 18:45:52 -0700 Subject: [PATCH 161/161] Update 0_beacon-chain-validator.md --- specs/validator/0_beacon-chain-validator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 7293675f1..be3008227 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -40,11 +40,11 @@ __NOTICE__: This document is a work-in-progress for researchers and implementers - [Slot](#slot-1) - [Shard](#shard) - [Beacon block root](#beacon-block-root) - - [Epoch boundary root](#epoch-boundary-root) + - [Target root](#target-root) - [Crosslink data root](#crosslink-data-root) - [Latest crosslink](#latest-crosslink) - - [Justified epoch](#justified-epoch) - - [Justified block root](#justified-block-root) + - [Source epoch](#source-epoch) + - [Source root](#source-root) - [Construct attestation](#construct-attestation) - [Data](#data) - [Aggregation bitfield](#aggregation-bitfield)