From cd855469543648f887f0cfb98fa505971d1ef25f Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 10 Apr 2019 15:09:53 +0100 Subject: [PATCH 01/91] Add option type and null --- specs/simple-serialize.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b78eff93e..1f2c01949 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,6 +30,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` +* '"null"': `Null` ### Composite types @@ -39,6 +40,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` +* **option**: option type containing one of the given subtypes + * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". @@ -70,7 +73,13 @@ assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### Vectors, containers, lists +### `"null"` + +```python +return b"" +``` + +### Vectors, containers, lists, options If `value` is fixed-size: @@ -87,6 +96,17 @@ serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "lit return serialized_length + serialized_bytes ``` +If `value` is an option type: + +Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. + +```python +serialized_bytes = serialize(value.value) +serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") +return serialized_length + serialized_bytes +``` + + ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). @@ -98,6 +118,7 @@ We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. * `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. +* `mix_in_type`: Given a Merkle root `root` and a type_index `type_index` (`"uint256"` little-endian serialization) return `hash(root + type_index)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -105,6 +126,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects +* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of option type ## Self-signed containers From 283ba8f7611e3112f1466eac63c0fc2966f94c79 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Thu, 11 Apr 2019 16:00:53 +0100 Subject: [PATCH 02/91] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 1f2c01949..5b302e2f5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -103,7 +103,7 @@ Define value as an object that has properties `value.value` with the contained v ```python serialized_bytes = serialize(value.value) serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") -return serialized_length + serialized_bytes +return serialized_type_index + serialized_bytes ``` From 2017ce96149306721bfd8d18553f531594b937b5 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 11 Apr 2019 16:05:16 +0100 Subject: [PATCH 03/91] Rename "option" -> "union"; "null" only in unions --- specs/simple-serialize.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 5b302e2f5..8a8c78446 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,7 +30,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` -* '"null"': `Null` +* `"null"`: `None` + +The type `"null"` is only legal as one of several type in a `union` type. ### Composite types @@ -40,7 +42,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` -* **option**: option type containing one of the given subtypes +* **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". @@ -79,7 +81,7 @@ return b"\x01" if value is True else b"\x00" return b"" ``` -### Vectors, containers, lists, options +### Vectors, containers, lists, unions If `value` is fixed-size: @@ -96,7 +98,7 @@ serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "lit return serialized_length + serialized_bytes ``` -If `value` is an option type: +If `value` is an union type: Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. @@ -126,7 +128,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects -* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of option type +* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of union type ## Self-signed containers From 63bdf95e79e2dcced7e2f5ba3945700972a5ef9d Mon Sep 17 00:00:00 2001 From: dankrad Date: Sat, 20 Apr 2019 11:46:31 +0100 Subject: [PATCH 04/91] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 8a8c78446..c9f845873 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -45,7 +45,7 @@ The type `"null"` is only legal as one of several type in a `union` type. * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` -We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". +We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". ### Aliases From a481a4e96cf29d13dd19fef33172d0cefa390e04 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 20 Apr 2019 20:57:50 +1000 Subject: [PATCH 05/91] Update simple-serialize.md --- specs/simple-serialize.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index c9f845873..2d9b715b3 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -32,7 +32,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"bool"`: `True` or `False` * `"null"`: `None` -The type `"null"` is only legal as one of several type in a `union` type. +The `"null"` type is only legal as a union sub-type. ### Composite types @@ -108,7 +108,6 @@ serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "litt return serialized_type_index + serialized_bytes ``` - ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). From f7ef9a1ba5e7b863538ab48097060f98ec536412 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Sat, 20 Apr 2019 17:26:07 -0700 Subject: [PATCH 06/91] Don't use SSZ in RPC request/response wrappers --- specs/networking/rpc-interface.md | 42 +++++++++++++++++-------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index 5d408b5a0..ca6008a40 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -39,32 +39,36 @@ To facilitate RPC-over-`libp2p`, a single protocol name is used: `/eth/serenity/ Remote method calls are wrapped in a "request" structure: ``` -( - id: uint64 - method_id: uint16 - body: Request -) + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| | ++ id (uint64) + +| | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| method_id (uint16) | body_len (uint32) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| | body (body_len bytes) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ``` and their corresponding responses are wrapped in a "response" structure: ``` -( - id: uint64 - response_code: uint16 - result: bytes -) + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| | ++ id (uint64) + +| | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| response_code (uint16) | result_len (uint32) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| | result (result_len bytes) | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ``` -If an error occurs, a variant of the response structure is returned: - -``` -( - id: uint64 - response_code: uint16 - result: bytes -) -``` +Note that the above structures are NOT encoded as SSZ but rather as sequences of bytes according to the packet diagrams above. This is because SSZ does not support structures without an explicit schema. Since the `body` and `result` fields depend on the value of `method_id` and `response_code`, a schema for the above structure cannot be known beforehand. The details of the RPC-Over-`libp2p` protocol are similar to [JSON-RPC 2.0](https://www.jsonrpc.org/specification). Specifically: From 101449e71a3492ff97c649c706866a0da2ec6962 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Sat, 27 Apr 2019 21:00:50 +0100 Subject: [PATCH 07/91] Define null as alias of {} --- specs/simple-serialize.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index abef5c669..6858561e8 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -8,12 +8,13 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) + - [Illegal empty composites](#illegal-empty-composites) - [Aliases](#aliases) - [Default values](#default-values) - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) - - [Containers, vectors, lists](#containers-vectors-lists) + - [Vectors, containers, lists, unions](#vectors-containers-lists-unions) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -32,9 +33,6 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` -* `"null"`: `None` - -The `"null"` type is only legal as a union sub-type. ### Composite types @@ -49,6 +47,10 @@ The `"null"` type is only legal as a union sub-type. We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". +#### Illegal empty composites + +The empty container `{}` (except as the `"null"` type inside a union, see below) and the empty fixed length list `[type, 0]` are **not** legal types. + ### Aliases For convenience we alias: @@ -56,6 +58,9 @@ For convenience we alias: * `"byte"` to `"uint8"` (this is a basic type) * `"bytes"` to `["byte"]` (this is *not* a basic type) * `"bytesN"` to `["byte", N]` (this is *not* a basic type) +* `"null"`: `{}`, i.e. the empty container + +The `"null"` type is only legal as a union sub-type. ### Default values From 4a483309a544dcd24095e5bb9d5a05b832e97b3b Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:07 +0200 Subject: [PATCH 08/91] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6858561e8..79ed0ef7b 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -113,7 +113,7 @@ fixed_parts = [part if part != None else variable_offsets[i] for i, part in enum return b"".join(fixed_parts + variable_parts) ``` -If `value` is an union type: +If `value` is a union type: Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. From d0447022cb4982e544abdea22e616e2cc7c44444 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:24 +0200 Subject: [PATCH 09/91] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 79ed0ef7b..0e0468eb7 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -119,7 +119,7 @@ Define value as an object that has properties `value.value` with the contained v ```python serialized_bytes = serialize(value.value) -serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") +serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_OFFSET, "little") return serialized_type_index + serialized_bytes ``` From cc22432bb91626d48ba62e786da72891f60424de Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:44 +0200 Subject: [PATCH 10/91] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 0e0468eb7..f6b5cb232 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -45,6 +45,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` +#### Variable-size and fixed-size + We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". #### Illegal empty composites From 8b316c6db4bcbd96416a1a12e76f57a626f03df6 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 12:04:27 +0100 Subject: [PATCH 11/91] Start moving state_transition.py to state transitition spec The state transition spec should be reasonably self-contained, limiting the amount of "magic" outside of it. This PR is a first step in this direction, specifically for operation processing. --- specs/core/0_beacon-chain.md | 44 ++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 55791e25f..bfb51e99a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1691,12 +1691,28 @@ def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: #### Operations +The sub-sections below define helper functions, one per operation type. The full processing of operations is done by running `process_operations(state, block.body)`. + +```python +def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: + assert len(body.deposits) == min(MAX_DEPOSITS, state.latest_eth1_data.deposit_count - state.deposit_index) + assert len(body.transfers) == len(set(body.transfers)) + + for operations, max_operations, function in { + (body.proposer_slashings, MAX_PROPOSER_SLASHINGS, process_proposer_slashing), + (body.attester_slashings, MAX_ATTESTER_SLASHINGS, attester_slashings), + (body.attestations, MAX_ATTESTATIONS, process_attestation), + (body.deposits, MAX_DEPOSITS, process_deposit), + (body.voluntary_exits, MAX_VOLUNTARY_EXITS, process_voluntary_exit), + (body.transfers, MAX_TRANSFERS, process_transfer), + }: + assert len(operations) <= max_operations + for operation in operations: + function(state, operation) +``` + ##### Proposer slashings -Verify that `len(block.body.proposer_slashings) <= MAX_PROPOSER_SLASHINGS`. - -For each `proposer_slashing` in `block.body.proposer_slashings`, run the following function: - ```python def process_proposer_slashing(state: BeaconState, proposer_slashing: ProposerSlashing) -> None: @@ -1721,10 +1737,6 @@ def process_proposer_slashing(state: BeaconState, ##### Attester slashings -Verify that `len(block.body.attester_slashings) <= MAX_ATTESTER_SLASHINGS`. - -For each `attester_slashing` in `block.body.attester_slashings`, run the following function: - ```python def process_attester_slashing(state: BeaconState, attester_slashing: AttesterSlashing) -> None: @@ -1759,10 +1771,6 @@ def process_attester_slashing(state: BeaconState, ##### Attestations -Verify that `len(block.body.attestations) <= MAX_ATTESTATIONS`. - -For each `attestation` in `block.body.attestations`, run the following function: - ```python def process_attestation(state: BeaconState, attestation: Attestation) -> None: """ @@ -1801,10 +1809,6 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: ##### Deposits -Verify that `len(block.body.deposits) == min(MAX_DEPOSITS, state.latest_eth1_data.deposit_count - state.deposit_index)`. - -For each `deposit` in `block.body.deposits`, run the following function: - ```python def process_deposit(state: BeaconState, deposit: Deposit) -> None: """ @@ -1851,10 +1855,6 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: ##### Voluntary exits -Verify that `len(block.body.voluntary_exits) <= MAX_VOLUNTARY_EXITS`. - -For each `exit` in `block.body.voluntary_exits`, run the following function: - ```python def process_voluntary_exit(state: BeaconState, exit: VoluntaryExit) -> None: """ @@ -1879,10 +1879,6 @@ def process_voluntary_exit(state: BeaconState, exit: VoluntaryExit) -> None: ##### Transfers -Verify that `len(block.body.transfers) <= MAX_TRANSFERS` and that all transfers are distinct. - -For each `transfer` in `block.body.transfers`, run the following function: - ```python def process_transfer(state: BeaconState, transfer: Transfer) -> None: """ From 591a2b47c8af28a43f5349435749d7f60f9894d4 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Wed, 1 May 2019 12:08:15 +0100 Subject: [PATCH 12/91] Simplify state_transition.py --- specs/core/0_beacon-chain.md | 6 +- .../eth2spec/phase0/state_transition.py | 69 +------------------ 2 files changed, 5 insertions(+), 70 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index bfb51e99a..e57fce037 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1698,14 +1698,14 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: assert len(body.deposits) == min(MAX_DEPOSITS, state.latest_eth1_data.deposit_count - state.deposit_index) assert len(body.transfers) == len(set(body.transfers)) - for operations, max_operations, function in { + for operations, max_operations, function in ( (body.proposer_slashings, MAX_PROPOSER_SLASHINGS, process_proposer_slashing), - (body.attester_slashings, MAX_ATTESTER_SLASHINGS, attester_slashings), + (body.attester_slashings, MAX_ATTESTER_SLASHINGS, process_attester_slashing), (body.attestations, MAX_ATTESTATIONS, process_attestation), (body.deposits, MAX_DEPOSITS, process_deposit), (body.voluntary_exits, MAX_VOLUNTARY_EXITS, process_voluntary_exit), (body.transfers, MAX_TRANSFERS, process_transfer), - }: + ): assert len(operations) <= max_operations for operation in operations: function(state, operation) diff --git a/test_libs/pyspec/eth2spec/phase0/state_transition.py b/test_libs/pyspec/eth2spec/phase0/state_transition.py index 1bef358d4..2aa0a38a6 100644 --- a/test_libs/pyspec/eth2spec/phase0/state_transition.py +++ b/test_libs/pyspec/eth2spec/phase0/state_transition.py @@ -14,76 +14,11 @@ from .spec import ( ) -def expected_deposit_count(state: BeaconState) -> int: - return min( - spec.MAX_DEPOSITS, - state.latest_eth1_data.deposit_count - state.deposit_index - ) - - -def process_operation_type(state: BeaconState, - operations: List[Any], - max_operations: int, - tx_fn: Callable[[BeaconState, Any], None]) -> None: - assert len(operations) <= max_operations - for operation in operations: - tx_fn(state, operation) - - -def process_operations(state: BeaconState, block: BeaconBlock) -> None: - process_operation_type( - state, - block.body.proposer_slashings, - spec.MAX_PROPOSER_SLASHINGS, - spec.process_proposer_slashing, - ) - - process_operation_type( - state, - block.body.attester_slashings, - spec.MAX_ATTESTER_SLASHINGS, - spec.process_attester_slashing, - ) - - process_operation_type( - state, - block.body.attestations, - spec.MAX_ATTESTATIONS, - spec.process_attestation, - ) - - assert len(block.body.deposits) == expected_deposit_count(state) - process_operation_type( - state, - block.body.deposits, - spec.MAX_DEPOSITS, - spec.process_deposit, - ) - - process_operation_type( - state, - block.body.voluntary_exits, - spec.MAX_VOLUNTARY_EXITS, - spec.process_voluntary_exit, - ) - - assert len(block.body.transfers) == len(set(block.body.transfers)) - process_operation_type( - state, - block.body.transfers, - spec.MAX_TRANSFERS, - spec.process_transfer, - ) - - -def process_block(state: BeaconState, - block: BeaconBlock, - verify_state_root: bool=False) -> None: +def process_block(state: BeaconState, block: BeaconBlock, verify_state_root) -> None: spec.process_block_header(state, block) spec.process_randao(state, block) spec.process_eth1_data(state, block) - - process_operations(state, block) + spec.process_operations(state, block.body) if verify_state_root: spec.verify_block_state_root(state, block) From 5df79d7565fd4fc49fe4367cb40afa7b121d5718 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Wed, 1 May 2019 13:14:10 +0100 Subject: [PATCH 13/91] Get rid of state_transition.py --- specs/core/0_beacon-chain.md | 81 ++++++++----------- .../eth2spec/phase0/state_transition.py | 47 ----------- .../test_process_attestation.py | 4 +- .../test_process_block_header.py | 2 +- .../test_process_crosslinks.py | 4 +- test_libs/pyspec/tests/helpers.py | 8 +- test_libs/pyspec/tests/test_finality.py | 5 +- test_libs/pyspec/tests/test_sanity.py | 6 +- 8 files changed, 44 insertions(+), 113 deletions(-) delete mode 100644 test_libs/pyspec/eth2spec/phase0/state_transition.py diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index e57fce037..50e390326 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -105,7 +105,6 @@ - [Registry updates](#registry-updates) - [Slashings](#slashings) - [Final updates](#final-updates) - - [Per-slot processing](#per-slot-processing) - [Per-block processing](#per-block-processing) - [Block header](#block-header) - [RANDAO](#randao) @@ -1264,27 +1263,27 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], ## Beacon chain state transition function -We now define the state transition function. At a high level, the state transition is made up of four parts: +The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. -1. State caching, which happens at the start of every slot. -2. The per-epoch transitions, which happens at the start of the first slot of every epoch. -3. The per-slot transitions, which happens at every slot. -4. The per-block transitions, which happens at every block. +```python +def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: + assert state.slot < block.slot + while state.slot < block.slot: + # State caching at the start of every slot + cache_state(state) + # Per-epoch processing at the start of the first slot of every epoch + if (state.slot + 1) % SLOTS_PER_EPOCH == 0: + process_epoch_transition(state) + # Slot incrementing + state.slot += 1 + # Block processing at every block + process_block(state, block) +``` -Transition section notes: -* The state caching caches the state root of the previous slot and updates block and state roots records. -* The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization. -* The per-slot transitions focus on the slot counter. -* The per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`. - -Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. - -Note: If there are skipped slots between a block and its parent block, run the steps in the [state-root](#state-caching), [per-epoch](#per-epoch-processing), and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. +Note: Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. ### State caching -At every `slot > GENESIS_SLOT` run the following function: - ```python def cache_state(state: BeaconState) -> None: # Cache latest known state root (for previous slot) @@ -1302,12 +1301,18 @@ def cache_state(state: BeaconState) -> None: ### Per-epoch processing -The steps below happen when `state.slot > GENESIS_SLOT and (state.slot + 1) % SLOTS_PER_EPOCH == 0`. +```python +def process_epoch_transition(state: BeaconState) -> None: + process_justification_and_finalization(state) + process_crosslinks(state) + process_rewards_and_penalties(state) + process_registry_updates(state) + process_slashings(state) + process_final_updates(state) +``` #### Helper functions -We define epoch transition helper functions: - ```python def get_total_active_balance(state: BeaconState) -> Gwei: return get_total_balance(state, get_active_validator_indices(state, get_current_epoch(state))) @@ -1387,8 +1392,6 @@ def get_earliest_attestation(state: BeaconState, attestations: List[PendingAttes #### Justification and finalization -Run the following function: - ```python def process_justification_and_finalization(state: BeaconState) -> None: if get_current_epoch(state) <= GENESIS_EPOCH + 1: @@ -1436,8 +1439,6 @@ def process_justification_and_finalization(state: BeaconState) -> None: #### Crosslinks -Run the following function: - ```python def process_crosslinks(state: BeaconState) -> None: state.previous_crosslinks = [c for c in state.current_crosslinks] @@ -1453,8 +1454,6 @@ def process_crosslinks(state: BeaconState) -> None: #### Rewards and penalties -First, we define additional helpers: - ```python def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: adjusted_quotient = integer_squareroot(get_total_active_balance(state)) // BASE_REWARD_QUOTIENT @@ -1526,8 +1525,6 @@ def get_crosslink_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: return [rewards, penalties] ``` -Run the following function: - ```python def process_rewards_and_penalties(state: BeaconState) -> None: if get_current_epoch(state) == GENESIS_EPOCH: @@ -1542,8 +1539,6 @@ def process_rewards_and_penalties(state: BeaconState) -> None: #### Registry updates -Run the following function: - ```python def process_registry_updates(state: BeaconState) -> None: # Process activation eligibility and ejections @@ -1568,8 +1563,6 @@ def process_registry_updates(state: BeaconState) -> None: #### Slashings -Run the following function: - ```python def process_slashings(state: BeaconState) -> None: current_epoch = get_current_epoch(state) @@ -1592,8 +1585,6 @@ def process_slashings(state: BeaconState) -> None: #### Final updates -Run the following function: - ```python def process_final_updates(state: BeaconState) -> None: current_epoch = get_current_epoch(state) @@ -1632,18 +1623,16 @@ def process_final_updates(state: BeaconState) -> None: state.current_epoch_attestations = [] ``` -### Per-slot processing - -At every `slot > GENESIS_SLOT` run the following function: - -```python -def advance_slot(state: BeaconState) -> None: - state.slot += 1 -``` - ### Per-block processing -For every `block` except the genesis block, run `process_block_header(state, block)`, `process_randao(state, block)` and `process_eth1_data(state, block)`. +```python +def process_block(state: BeaconState, block: BeaconBlock) -> None: + process_block_header(state, block) + process_randao(state, block) + process_eth1_data(state, block) + process_operations(state, block.body) + # verify_block_state_root(state, block) +``` #### Block header @@ -1691,8 +1680,6 @@ def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: #### Operations -The sub-sections below define helper functions, one per operation type. The full processing of operations is done by running `process_operations(state, block.body)`. - ```python def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: assert len(body.deposits) == min(MAX_DEPOSITS, state.latest_eth1_data.deposit_count - state.deposit_index) @@ -1913,8 +1900,6 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: #### State root verification -Verify the block's `state_root` by running the following function: - ```python def verify_block_state_root(state: BeaconState, block: BeaconBlock) -> None: assert block.state_root == hash_tree_root(state) diff --git a/test_libs/pyspec/eth2spec/phase0/state_transition.py b/test_libs/pyspec/eth2spec/phase0/state_transition.py deleted file mode 100644 index 2aa0a38a6..000000000 --- a/test_libs/pyspec/eth2spec/phase0/state_transition.py +++ /dev/null @@ -1,47 +0,0 @@ -from . import spec - - -from typing import ( - Any, - Callable, - List -) - -from .spec import ( - BeaconState, - BeaconBlock, - Slot, -) - - -def process_block(state: BeaconState, block: BeaconBlock, verify_state_root) -> None: - spec.process_block_header(state, block) - spec.process_randao(state, block) - spec.process_eth1_data(state, block) - spec.process_operations(state, block.body) - if verify_state_root: - spec.verify_block_state_root(state, block) - - -def process_epoch_transition(state: BeaconState) -> None: - spec.process_justification_and_finalization(state) - spec.process_crosslinks(state) - spec.process_rewards_and_penalties(state) - spec.process_registry_updates(state) - spec.process_slashings(state) - spec.process_final_updates(state) - - -def state_transition_to(state: BeaconState, up_to: Slot) -> BeaconState: - while state.slot < up_to: - spec.cache_state(state) - if (state.slot + 1) % spec.SLOTS_PER_EPOCH == 0: - process_epoch_transition(state) - spec.advance_slot(state) - - -def state_transition(state: BeaconState, - block: BeaconBlock, - verify_state_root: bool=False) -> BeaconState: - state_transition_to(state, block.slot) - process_block(state, block, verify_state_root) diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 1be60c860..105d1e0a4 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -3,13 +3,11 @@ import pytest import eth2spec.phase0.spec as spec -from eth2spec.phase0.state_transition import ( - state_transition, -) from eth2spec.phase0.spec import ( get_current_epoch, process_attestation, slot_to_epoch, + state_transition, ) from tests.helpers import ( build_empty_block_for_next_slot, diff --git a/test_libs/pyspec/tests/block_processing/test_process_block_header.py b/test_libs/pyspec/tests/block_processing/test_process_block_header.py index b35b0a9c1..6fd6e674e 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_block_header.py +++ b/test_libs/pyspec/tests/block_processing/test_process_block_header.py @@ -5,10 +5,10 @@ import pytest from eth2spec.phase0.spec import ( get_beacon_proposer_index, cache_state, - advance_slot, process_block_header, ) from tests.helpers import ( + advance_slot, build_empty_block_for_next_slot, next_slot, ) diff --git a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py index fe694724a..60e0dec53 100644 --- a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py +++ b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py @@ -3,13 +3,11 @@ import pytest import eth2spec.phase0.spec as spec -from eth2spec.phase0.state_transition import ( - state_transition, -) from eth2spec.phase0.spec import ( cache_state, get_crosslink_deltas, process_crosslinks, + state_transition, ) from tests.helpers import ( add_attestation_to_state, diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index 63e4cd710..2cb272ff3 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -2,9 +2,6 @@ from copy import deepcopy from py_ecc import bls -from eth2spec.phase0.state_transition import ( - state_transition, -) import eth2spec.phase0.spec as spec from eth2spec.utils.minimal_ssz import signing_root from eth2spec.phase0.spec import ( @@ -38,6 +35,7 @@ from eth2spec.phase0.spec import ( get_shard_delta, hash_tree_root, slot_to_epoch, + state_transition, verify_merkle_branch, hash, ) @@ -53,6 +51,10 @@ pubkeys = [bls.privtopub(privkey) for privkey in privkeys] pubkey_to_privkey = {pubkey: privkey for privkey, pubkey in zip(privkeys, pubkeys)} +def advance_slot(state) -> None: + state.slot += 1 + + def get_balance(state, index): return state.balances[index] diff --git a/test_libs/pyspec/tests/test_finality.py b/test_libs/pyspec/tests/test_finality.py index ca048c2b2..816dfd6bd 100644 --- a/test_libs/pyspec/tests/test_finality.py +++ b/test_libs/pyspec/tests/test_finality.py @@ -4,9 +4,6 @@ import pytest import eth2spec.phase0.spec as spec -from eth2spec.phase0.state_transition import ( - state_transition, -) from .helpers import ( build_empty_block_for_next_slot, fill_aggregate_attestation, @@ -67,7 +64,7 @@ def next_epoch_with_attestations(state, fill_aggregate_attestation(post_state, prev_attestation) block.body.attestations.append(prev_attestation) - state_transition(post_state, block) + spec.state_transition(post_state, block) blocks.append(block) return state, blocks, post_state diff --git a/test_libs/pyspec/tests/test_sanity.py b/test_libs/pyspec/tests/test_sanity.py index b7d31f122..4d826fc13 100644 --- a/test_libs/pyspec/tests/test_sanity.py +++ b/test_libs/pyspec/tests/test_sanity.py @@ -20,13 +20,10 @@ from eth2spec.phase0.spec import ( get_state_root, get_current_epoch, get_domain, - advance_slot, cache_state, verify_merkle_branch, - hash, -) -from eth2spec.phase0.state_transition import ( state_transition, + hash, ) from eth2spec.utils.merkle_minimal import ( calc_merkle_tree_from_leaves, @@ -34,6 +31,7 @@ from eth2spec.utils.merkle_minimal import ( get_merkle_root, ) from .helpers import ( + advance_slot, get_balance, build_deposit_data, build_empty_block_for_next_slot, From 2e63a9b5a0322e0e87f72baee3a279999dbdddc7 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Wed, 1 May 2019 13:29:03 +0100 Subject: [PATCH 14/91] clean up --- specs/core/0_beacon-chain.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 50e390326..8f69213bf 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -97,7 +97,7 @@ - [On genesis](#on-genesis) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - [State caching](#state-caching) - - [Per-epoch processing](#per-epoch-processing) + - [Epoch processing](#epoch-processing) - [Helper functions](#helper-functions-1) - [Justification and finalization](#justification-and-finalization) - [Crosslinks](#crosslinks) @@ -105,7 +105,7 @@ - [Registry updates](#registry-updates) - [Slashings](#slashings) - [Final updates](#final-updates) - - [Per-block processing](#per-block-processing) + - [Block processing](#block-processing) - [Block header](#block-header) - [RANDAO](#randao) - [Eth1 data](#eth1-data) @@ -1263,7 +1263,7 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], ## Beacon chain state transition function -The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. +The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled exception (e.g. failed `assert`s and out-of-range list accesses) are considered invalid. ```python def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: @@ -1271,17 +1271,15 @@ def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: while state.slot < block.slot: # State caching at the start of every slot cache_state(state) - # Per-epoch processing at the start of the first slot of every epoch + # Epoch processing at the start of the first slot of every epoch if (state.slot + 1) % SLOTS_PER_EPOCH == 0: - process_epoch_transition(state) + process_epoch(state) # Slot incrementing state.slot += 1 # Block processing at every block process_block(state, block) ``` -Note: Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. - ### State caching ```python @@ -1299,10 +1297,10 @@ def cache_state(state: BeaconState) -> None: state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = latest_block_root ``` -### Per-epoch processing +### Epoch processing ```python -def process_epoch_transition(state: BeaconState) -> None: +def process_epoch(state: BeaconState) -> None: process_justification_and_finalization(state) process_crosslinks(state) process_rewards_and_penalties(state) @@ -1623,7 +1621,7 @@ def process_final_updates(state: BeaconState) -> None: state.current_epoch_attestations = [] ``` -### Per-block processing +### Block processing ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: From a33ee00239f0ecd1f8bb557372844250431d0f68 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 13:52:37 +0100 Subject: [PATCH 15/91] Update simple-serialize.md --- specs/simple-serialize.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f6b5cb232..b7a1375e4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -8,9 +8,10 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) - - [Illegal empty composites](#illegal-empty-composites) + - [Variable-size and fixed-size](#variable-size-and-fixed-size) - [Aliases](#aliases) - [Default values](#default-values) + - [Illegal types](#illegal-types) - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) @@ -45,14 +46,10 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` -#### Variable-size and fixed-size +### Variable-size and fixed-size We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". -#### Illegal empty composites - -The empty container `{}` (except as the `"null"` type inside a union, see below) and the empty fixed length list `[type, 0]` are **not** legal types. - ### Aliases For convenience we alias: @@ -62,19 +59,20 @@ For convenience we alias: * `"bytesN"` to `["byte", N]` (this is *not* a basic type) * `"null"`: `{}`, i.e. the empty container -The `"null"` type is only legal as a union sub-type. - ### Default values The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. +### Illegal types + +Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as a union subtype. + ## Serialization We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. > *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signing_root`, `is_variable_size`, etc.) objects implicitly carry their type. - ### `"uintN"` ```python From 563df146b9073da8f91b6ad200cd19273ea8f2ac Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 13:55:02 +0100 Subject: [PATCH 16/91] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b7a1375e4..21a87a6f9 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -44,7 +44,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **list**: ordered variable-length homogeneous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` * **union**: union type containing one of the given subtypes - * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` + * round bracket notation `(type_1, type_2, ...)`, e.g. `("uint64", "null")` ### Variable-size and fixed-size From 9b24d06b2cbfa494a8169eebc3f97f32b9ab3399 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Wed, 1 May 2019 14:16:55 +0100 Subject: [PATCH 17/91] Cleanup --- specs/core/0_beacon-chain.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 8f69213bf..153e42cf6 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1263,18 +1263,19 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], ## Beacon chain state transition function -The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled exception (e.g. failed `assert`s and out-of-range list accesses) are considered invalid. +The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. ```python def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: assert state.slot < block.slot + # Slot processing (including slots with no blocks) while state.slot < block.slot: # State caching at the start of every slot cache_state(state) # Epoch processing at the start of the first slot of every epoch if (state.slot + 1) % SLOTS_PER_EPOCH == 0: process_epoch(state) - # Slot incrementing + # Increment slot number state.slot += 1 # Block processing at every block process_block(state, block) @@ -1284,17 +1285,15 @@ def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: ```python def cache_state(state: BeaconState) -> None: - # Cache latest known state root (for previous slot) - latest_state_root = hash_tree_root(state) - state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = latest_state_root + # Cache state and block roots of previous slot + previous_state_root = hash_tree_root(state) + state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_state_root + previous_block_root = signing_root(state.latest_block_header) + state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_block_root - # Store latest known state root (for previous slot) in latest_block_header if it is empty + # Cache previous state root in latest_block_header, if empty if state.latest_block_header.state_root == ZERO_HASH: - state.latest_block_header.state_root = latest_state_root - - # Cache latest known block root (for previous slot) - latest_block_root = signing_root(state.latest_block_header) - state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = latest_block_root + state.latest_block_header.state_root = previous_state_root ``` ### Epoch processing From 7980cf3ef79ee4f7ee418de5c8169907280169a2 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Wed, 1 May 2019 15:07:55 +0100 Subject: [PATCH 18/91] Fix --- specs/core/0_beacon-chain.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 153e42cf6..4c1498ccf 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1285,15 +1285,18 @@ def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: ```python def cache_state(state: BeaconState) -> None: - # Cache state and block roots of previous slot + # Cache state root of previous slot previous_state_root = hash_tree_root(state) state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_state_root - previous_block_root = signing_root(state.latest_block_header) - state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_block_root # Cache previous state root in latest_block_header, if empty if state.latest_block_header.state_root == ZERO_HASH: state.latest_block_header.state_root = previous_state_root + + # Cache block root of previous slot + previous_block_root = signing_root(state.latest_block_header) + state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_block_root + ``` ### Epoch processing From b1930d22394911c9549f479adb92f62ec41b0958 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 1 May 2019 15:12:49 +0100 Subject: [PATCH 19/91] Union default values --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 21a87a6f9..6adc9c4b8 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -61,11 +61,11 @@ For convenience we alias: ### Default values -The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. +The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. Unions default to the first type in the union (with type index zero), which is `"null"` if present in the union. ### Illegal types -Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as a union subtype. +Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as the first type in a union subtype (i.e., with type index zero). ## Serialization From c761fbc3181fcf94ef9201a8174806bacd859caa Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 2 May 2019 09:24:24 +0100 Subject: [PATCH 20/91] Clean up verify_indexed_attestation Cosmetic changes: * Add 4 lines of comments (now every statement has a comment) * Avoid unnecessary `assert` (the end goal for me is for `assert`s to be exclusive to the operation processing helpers). * Merge `return`s into one (increase readability, reduce verbosity) * Use shorter-named `bit_0_indices` and `bit_1_indices` helper variables Substantive change: * Remove the condition that `len(0_indices) + len(1_indices) > 0`. This condition is redundant in the context of `process_attester_slashing` because of `slashed_any`. It is largely artificial in `process_attestation` where validators are incentivised to maximise new attestations. --- specs/core/0_beacon-chain.md | 54 +++++++++++++++++------------------- 1 file changed, 25 insertions(+), 29 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2a0b0c11d..963c2ac44 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1018,37 +1018,33 @@ def convert_to_indexed(state: BeaconState, attestation: Attestation) -> IndexedA ```python def verify_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool: """ - Verify validity of ``indexed_attestation`` fields. + Verify validity of ``indexed_attestation``. """ - custody_bit_0_indices = indexed_attestation.custody_bit_0_indices - custody_bit_1_indices = indexed_attestation.custody_bit_1_indices + bit_0_indices = indexed_attestation.custody_bit_0_indices + bit_1_indices = indexed_attestation.custody_bit_1_indices - # Ensure no duplicate indices across custody bits - assert len(set(custody_bit_0_indices).intersection(set(custody_bit_1_indices))) == 0 - - if len(custody_bit_1_indices) > 0: # [TO BE REMOVED IN PHASE 1] - return False - - if not (1 <= len(custody_bit_0_indices) + len(custody_bit_1_indices) <= MAX_INDICES_PER_ATTESTATION): - return False - - if custody_bit_0_indices != sorted(custody_bit_0_indices): - return False - - if custody_bit_1_indices != sorted(custody_bit_1_indices): - return False - - return bls_verify_multiple( - pubkeys=[ - bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_0_indices]), - bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_indices]), - ], - message_hashes=[ - hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b0)), - hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b1)), - ], - signature=indexed_attestation.signature, - domain=get_domain(state, DOMAIN_ATTESTATION, indexed_attestation.data.target_epoch), + return ( + # Verify no index has custody bit equal to 1 [to be removed in phase 1] + len(bit_1_indices) == 0 and + # Verify max number of indices + len(bit_0_indices) + len(bit_1_indices) <= MAX_INDICES_PER_ATTESTATION and + # Verify index sets are disjoint + len(set(bit_0_indices).intersection(bit_1_indices)) == 0 and + # Verify indices are sorted + bit_0_indices == sorted(bit_0_indices) and bit_1_indices == sorted(bit_1_indices) and + # Verify aggregate signature + bls_verify_multiple( + pubkeys=[ + bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in bit_0_indices]), + bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in bit_1_indices]), + ], + message_hashes=[ + hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b0)), + hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b1)), + ], + signature=indexed_attestation.signature, + domain=get_domain(state, DOMAIN_ATTESTATION, indexed_attestation.data.target_epoch), + ) ) ``` From 973f07223537e33dd05587b895143eb45ee6e3d0 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Thu, 2 May 2019 09:25:29 +0100 Subject: [PATCH 21/91] Remove unnecessary test --- .../block_processing/test_process_attestation.py | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index bcf71376c..165f0c84a 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -142,14 +142,3 @@ def test_non_empty_custody_bitfield(state): pre_state, post_state = run_attestation_processing(state, attestation, False) return pre_state, attestation, post_state - - -def test_empty_aggregation_bitfield(state): - attestation = get_valid_attestation(state) - state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY - - attestation.aggregation_bitfield = b'\x00' * len(attestation.aggregation_bitfield) - - pre_state, post_state = run_attestation_processing(state, attestation, False) - - return pre_state, attestation, post_state From 9f2cdd9c7807265edcb259dd91d37a9ca51988ca Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 2 May 2019 11:07:25 +0100 Subject: [PATCH 22/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4c1498ccf..86caf7402 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1266,19 +1266,22 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. ```python -def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: +def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconBlock: + # Block must post-date the state assert state.slot < block.slot - # Slot processing (including slots with no blocks) + # Process slots (including those with no blocks) since block while state.slot < block.slot: - # State caching at the start of every slot + # Cache state at the start of every slot cache_state(state) - # Epoch processing at the start of the first slot of every epoch + # Process epoch at the start of the first slot of every epoch if (state.slot + 1) % SLOTS_PER_EPOCH == 0: process_epoch(state) # Increment slot number state.slot += 1 - # Block processing at every block + # Process block process_block(state, block) + # Return post-state + return state ``` ### State caching From 66b152f79ecad957b38ecccc16ce2f131afd10cc Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 3 May 2019 05:07:11 -0500 Subject: [PATCH 23/91] Allow multiple bit challenges, and recover withdrawability Resolves #864 items 4, 7, 14 --- specs/core/1_custody-game.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index d56526611..0de84ff06 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -146,7 +146,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenge_index': 'uint64', 'challenger_index': ValidatorIndex, 'responder_index': ValidatorIndex, - 'deadline': Epoch, + 'inclusion_epoch': Epoch, 'crosslink_data_root': Hash, 'depth': 'uint64', 'chunk_index': 'uint64', @@ -160,7 +160,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenge_index': 'uint64', 'challenger_index': ValidatorIndex, 'responder_index': ValidatorIndex, - 'deadline': Epoch, + 'inclusion_epoch': Epoch, 'crosslink_data_root': Hash, 'chunk_count': 'uint64', 'chunk_bits_merkle_root': Hash, @@ -485,7 +485,7 @@ def process_chunk_challenge(state: BeaconState, challenge_index=state.custody_challenge_index, challenger_index=get_beacon_proposer_index(state), responder_index=challenge.responder_index - deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, + inclusion_epoch=get_current_epoch(state), crosslink_data_root=challenge.attestation.data.crosslink_data_root, depth=depth, chunk_index=challenge.chunk_index, @@ -528,10 +528,9 @@ def process_bit_challenge(state: BeaconState, attesters = get_attesting_indices(state, attestation.data, attestation.aggregation_bitfield) assert challenge.responder_index in attesters - # A validator can be the challenger or responder for at most one challenge at a time + # A validator can be the challenger for at most one challenge at a time for record in state.custody_bit_challenge_records: assert record.challenger_index != challenge.challenger_index - assert record.responder_index != challenge.responder_index # Verify the responder is a valid custody key epoch_to_sign = get_randao_epoch_for_custody_period( @@ -563,7 +562,7 @@ def process_bit_challenge(state: BeaconState, challenge_index=state.custody_challenge_index, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, - deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, + inclusion_epoch=get_current_epoch(state), crosslink_data_root=challenge.attestation.data.crosslink_data_root, chunk_count=chunk_count, chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), @@ -604,6 +603,8 @@ def process_chunk_challenge_response(state: BeaconState, assert response.chunk_index == challenge.chunk_index # Verify bit challenge data is null assert response.chunk_bits_branch == [] and response.chunk_bits_leaf == ZERO_HASH + # Verify minimum delay + assert get_current_epoch(state) >= challenge.inclusion_epoch + ACTIVATION_EXIT_DELAY # Verify the chunk matches the crosslink data root assert verify_merkle_branch( leaf=hash_tree_root(response.chunk), @@ -626,6 +627,9 @@ def process_bit_challenge_response(state: BeaconState, challenge: CustodyBitChallengeRecord) -> None: # Verify chunk index assert response.chunk_index < challenge.chunk_count + # Verify responder has not been slashed + responder = state.validator_registry[record.responder_index] + assert not responder.slashed # Verify the chunk matches the crosslink data root assert verify_merkle_branch( leaf=hash_tree_root(response.chunk), @@ -671,13 +675,13 @@ Run `process_challenge_deadlines(state)` immediately after `process_reveal_deadl ```python def process_challenge_deadlines(state: BeaconState) -> None: for challenge in state.custody_chunk_challenge_records: - if get_current_epoch(state) > challenge.deadline: + if get_current_epoch(state) > challenge.inclusion_epoch + CUSTODY_RESPONSE_DEADLINE: slash_validator(state, challenge.responder_index, challenge.challenger_index) records = state.custody_chunk_challenge_records records[records.index(challenge)] = CustodyChunkChallengeRecord() for challenge in state.custody_bit_challenge_records: - if get_current_epoch(state) > challenge.deadline: + if get_current_epoch(state) > challenge.inclusion_epoch + CUSTODY_RESPONSE_DEADLINE: slash_validator(state, challenge.responder_index, challenge.challenger_index) records = state.custody_bit_challenge_records records[records.index(challenge)] = CustodyBitChallengeRecord() @@ -688,6 +692,18 @@ Append this to `process_final_updates(state)`: ```python # Clean up exposed RANDAO key reveals state.exposed_derived_secrets[current_epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS] = [] + # Reset withdrawable epochs if challenge records are empty + for index, validator in enumerate(state.validator_registry): + eligible = True + for records in (state.custody_chunk_challenge_records, state.bit_challenge_records): + for filter_func in (lambda rec: rec.challenger_index == index, lambda rec: rec.responder_index == index): + if len(list(filter(filter_func, records))) > 0: + eligible = False + if eligible: + if validator.exit_epoch == FAR_FUTURE_EPOCH: + validator.withdrawable_epoch = FAR_FUTURE_EPOCH + else: + validator.withdrawable_epoch = validator.exit_epoch + MIN_VALIDATOR_WITHDRAWABILITY_DELAY ``` In `process_penalties_and_exits`, change the definition of `eligible` to the following (note that it is not a pure function because `state` is declared in the surrounding scope): From 2ccd357f0eab881eaf7f6bd39e09987467731c91 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 3 May 2019 21:05:54 +0800 Subject: [PATCH 24/91] Update specs/core/1_custody-game.md Co-Authored-By: vbuterin --- specs/core/1_custody-game.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index 0de84ff06..307e18573 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -628,7 +628,7 @@ def process_bit_challenge_response(state: BeaconState, # Verify chunk index assert response.chunk_index < challenge.chunk_count # Verify responder has not been slashed - responder = state.validator_registry[record.responder_index] + responder = state.validator_registry[challenge.responder_index] assert not responder.slashed # Verify the chunk matches the crosslink data root assert verify_merkle_branch( From 197a7200efaf1bf9024910e3c41563e3b0c96f77 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Fri, 3 May 2019 21:21:42 +0100 Subject: [PATCH 25/91] Add HW and Danny comments --- specs/core/0_beacon-chain.md | 61 +++++++++---------- .../test_process_block_header.py | 4 +- .../test_process_crosslinks.py | 4 +- test_libs/pyspec/tests/test_sanity.py | 4 +- 4 files changed, 36 insertions(+), 37 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 86caf7402..b4b44e628 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -96,7 +96,6 @@ - [`slash_validator`](#slash_validator) - [On genesis](#on-genesis) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - - [State caching](#state-caching) - [Epoch processing](#epoch-processing) - [Helper functions](#helper-functions-1) - [Justification and finalization](#justification-and-finalization) @@ -1267,36 +1266,36 @@ The post-state corresponding to a pre-state `state` and a block `block` is defin ```python def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconBlock: - # Block must post-date the state - assert state.slot < block.slot # Process slots (including those with no blocks) since block - while state.slot < block.slot: - # Cache state at the start of every slot - cache_state(state) - # Process epoch at the start of the first slot of every epoch - if (state.slot + 1) % SLOTS_PER_EPOCH == 0: - process_epoch(state) - # Increment slot number - state.slot += 1 + process_slots(state, block.slot) # Process block process_block(state, block) # Return post-state return state ``` -### State caching +```python +def process_slots(state: BeaconState, slot: Slot) -> None: + assert state.slot < slot + while state.slot < slot: + process_slot(state) + # Process epoch on the last slot of every epoch + if (state.slot + 1) % SLOTS_PER_EPOCH == 0: + process_epoch(state) + state.slot += 1 +``` ```python -def cache_state(state: BeaconState) -> None: - # Cache state root of previous slot +def process_slot(state: BeaconState) -> None: + # Cache state root previous_state_root = hash_tree_root(state) state.latest_state_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_state_root - # Cache previous state root in latest_block_header, if empty + # Cache latest block header state root if state.latest_block_header.state_root == ZERO_HASH: state.latest_block_header.state_root = previous_state_root - # Cache block root of previous slot + # Cache block root previous_block_root = signing_root(state.latest_block_header) state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_block_root @@ -1631,10 +1630,10 @@ def process_final_updates(state: BeaconState) -> None: ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) - process_randao(state, block) - process_eth1_data(state, block) + process_randao(state, block.body) + process_eth1_data(state, block.body) process_operations(state, block.body) - # verify_block_state_root(state, block) + # process_state_root(state, block) ``` #### Block header @@ -1661,31 +1660,33 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: #### RANDAO ```python -def process_randao(state: BeaconState, block: BeaconBlock) -> None: +def process_randao(state: BeaconState, body: BeaconBlockBody) -> None: proposer = state.validator_registry[get_beacon_proposer_index(state)] # Verify that the provided randao value is valid - assert bls_verify(proposer.pubkey, hash_tree_root(get_current_epoch(state)), block.body.randao_reveal, get_domain(state, DOMAIN_RANDAO)) + assert bls_verify(proposer.pubkey, hash_tree_root(get_current_epoch(state)), body.randao_reveal, get_domain(state, DOMAIN_RANDAO)) # Mix it in state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = ( xor(get_randao_mix(state, get_current_epoch(state)), - hash(block.body.randao_reveal)) + hash(body.randao_reveal)) ) ``` #### Eth1 data ```python -def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: - state.eth1_data_votes.append(block.body.eth1_data) - if state.eth1_data_votes.count(block.body.eth1_data) * 2 > SLOTS_PER_ETH1_VOTING_PERIOD: - state.latest_eth1_data = block.body.eth1_data +def process_eth1_data(state: BeaconState, body: BeaconBlockBody) -> None: + state.eth1_data_votes.append(body.eth1_data) + if state.eth1_data_votes.count(body.eth1_data) * 2 > SLOTS_PER_ETH1_VOTING_PERIOD: + state.latest_eth1_data = body.eth1_data ``` #### Operations ```python def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: + # Verify that outstanding deposits are processed up to the maximum number of deposits assert len(body.deposits) == min(MAX_DEPOSITS, state.latest_eth1_data.deposit_count - state.deposit_index) + # Verify that there are no duplicate transfers assert len(body.transfers) == len(set(body.transfers)) for operations, max_operations, function in ( @@ -1704,8 +1705,7 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: ##### Proposer slashings ```python -def process_proposer_slashing(state: BeaconState, - proposer_slashing: ProposerSlashing) -> None: +def process_proposer_slashing(state: BeaconState, proposer_slashing: ProposerSlashing) -> None: """ Process ``ProposerSlashing`` operation. Note that this function mutates ``state``. @@ -1728,8 +1728,7 @@ def process_proposer_slashing(state: BeaconState, ##### Attester slashings ```python -def process_attester_slashing(state: BeaconState, - attester_slashing: AttesterSlashing) -> None: +def process_attester_slashing(state: BeaconState, attester_slashing: AttesterSlashing) -> None: """ Process ``AttesterSlashing`` operation. Note that this function mutates ``state``. @@ -1904,6 +1903,6 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: #### State root verification ```python -def verify_block_state_root(state: BeaconState, block: BeaconBlock) -> None: +def process_state_root(state: BeaconState, block: BeaconBlock) -> None: assert block.state_root == hash_tree_root(state) ``` diff --git a/test_libs/pyspec/tests/block_processing/test_process_block_header.py b/test_libs/pyspec/tests/block_processing/test_process_block_header.py index 6fd6e674e..149bab514 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_block_header.py +++ b/test_libs/pyspec/tests/block_processing/test_process_block_header.py @@ -4,7 +4,7 @@ import pytest from eth2spec.phase0.spec import ( get_beacon_proposer_index, - cache_state, + process_slot, process_block_header, ) from tests.helpers import ( @@ -18,7 +18,7 @@ pytestmark = pytest.mark.header def prepare_state_for_header_processing(state): - cache_state(state) + process_slot(state) advance_slot(state) diff --git a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py index 60e0dec53..4e3d4b5d8 100644 --- a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py +++ b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py @@ -4,7 +4,7 @@ import pytest import eth2spec.phase0.spec as spec from eth2spec.phase0.spec import ( - cache_state, + process_slot, get_crosslink_deltas, process_crosslinks, state_transition, @@ -33,7 +33,7 @@ def run_process_crosslinks(state, valid=True): state_transition(state, block) # cache state before epoch transition - cache_state(state) + process_slot(state) post_state = deepcopy(state) process_crosslinks(post_state) diff --git a/test_libs/pyspec/tests/test_sanity.py b/test_libs/pyspec/tests/test_sanity.py index 4d826fc13..7fff6fb55 100644 --- a/test_libs/pyspec/tests/test_sanity.py +++ b/test_libs/pyspec/tests/test_sanity.py @@ -20,7 +20,7 @@ from eth2spec.phase0.spec import ( get_state_root, get_current_epoch, get_domain, - cache_state, + process_slot, verify_merkle_branch, state_transition, hash, @@ -51,7 +51,7 @@ pytestmark = pytest.mark.sanity def test_slot_transition(state): test_state = deepcopy(state) - cache_state(test_state) + process_slot(test_state) advance_slot(test_state) assert test_state.slot == state.slot + 1 assert get_state_root(test_state, state.slot) == state.hash_tree_root() From 4c1073fa2f898de7ffe26fd8aa197d38c25a8f35 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 3 May 2019 22:22:19 +0100 Subject: [PATCH 26/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b4b44e628..8d999c65c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1279,7 +1279,7 @@ def process_slots(state: BeaconState, slot: Slot) -> None: assert state.slot < slot while state.slot < slot: process_slot(state) - # Process epoch on the last slot of every epoch + # Process epoch on the first slot of the next epoch if (state.slot + 1) % SLOTS_PER_EPOCH == 0: process_epoch(state) state.slot += 1 From a6e825d46056a4a92473e52f1ea0bcb19210e67b Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 5 May 2019 12:04:34 +0100 Subject: [PATCH 27/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 58 +++++++++++++++--------------------- 1 file changed, 24 insertions(+), 34 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36ac92796..addaa865e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -279,6 +279,8 @@ The types are defined topologically to aid in facilitating an executable version ```python { + # Shard number + 'shard': 'uint64', # Epoch number 'epoch': 'uint64', # Root of the previous crosslink @@ -315,9 +317,7 @@ The types are defined topologically to aid in facilitating an executable version 'target_root': 'bytes32', # Crosslink vote - 'shard': 'uint64', - 'previous_crosslink_root': 'bytes32', - 'crosslink_data_root': 'bytes32', + 'crosslink': Crosslink, } ``` @@ -765,7 +765,7 @@ def get_epoch_start_shard(state: BeaconState, epoch: Epoch) -> Shard: def get_attestation_slot(state: BeaconState, attestation: Attestation) -> Slot: epoch = attestation.data.target_epoch committee_count = get_epoch_committee_count(state, epoch) - offset = (attestation.data.shard + SHARD_COUNT - get_epoch_start_shard(state, epoch)) % SHARD_COUNT + offset = (attestation.data.crosslink.shard + SHARD_COUNT - get_epoch_start_shard(state, epoch)) % SHARD_COUNT return get_epoch_start_slot(epoch) + offset // (committee_count // SLOTS_PER_EPOCH) ``` @@ -927,7 +927,7 @@ def get_attesting_indices(state: BeaconState, """ Return the sorted attesting indices corresponding to ``attestation_data`` and ``bitfield``. """ - committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.shard) + committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.crosslink.shard) assert verify_bitfield(bitfield, len(committee)) return sorted([index for i, index in enumerate(committee) if get_bitfield_bit(bitfield, i) == 0b1]) ``` @@ -1296,28 +1296,18 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat return get_total_balance(state, get_unslashed_attesting_indices(state, attestations)) ``` -```python -def get_crosslink_from_attestation_data(state: BeaconState, data: AttestationData) -> Crosslink: - return Crosslink( - epoch=min(data.target_epoch, state.current_crosslinks[data.shard].epoch + MAX_CROSSLINK_EPOCHS), - previous_crosslink_root=data.previous_crosslink_root, - crosslink_data_root=data.crosslink_data_root, - ) -``` - ```python def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: - shard_attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.shard == shard] - shard_crosslinks = [get_crosslink_from_attestation_data(state, a.data) for a in shard_attestations] + shard_attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] candidate_crosslinks = [ - c for c in shard_crosslinks + c for c in [a.data.crosslink for a in shard_attestations] if hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)) ] if len(candidate_crosslinks) == 0: return Crosslink(), [] def get_attestations_for(crosslink: Crosslink) -> List[PendingAttestation]: - return [a for a in shard_attestations if get_crosslink_from_attestation_data(state, a.data) == crosslink] + return [a for a in shard_attestations if a.data.crosslink == crosslink] # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) winning_crosslink = max(candidate_crosslinks, key=lambda crosslink: ( get_attesting_balance(state, get_attestations_for(crosslink)), crosslink.crosslink_data_root @@ -1705,30 +1695,30 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: attestation_slot = get_attestation_slot(state, attestation) assert attestation_slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot <= attestation_slot + SLOTS_PER_EPOCH - # Check target epoch, source epoch, source root, and source crosslink data = attestation.data - assert (data.target_epoch, data.source_epoch, data.source_root, data.previous_crosslink_root) in { - (get_current_epoch(state), state.current_justified_epoch, state.current_justified_root, hash_tree_root(state.current_crosslinks[data.shard])), - (get_previous_epoch(state), state.previous_justified_epoch, state.previous_justified_root, hash_tree_root(state.previous_crosslinks[data.shard])), - } - - # Check crosslink data root - assert data.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] - - # Check signature and bitfields - assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) - - # Cache pending attestation pending_attestation = PendingAttestation( data=data, aggregation_bitfield=attestation.aggregation_bitfield, inclusion_delay=state.slot - attestation_slot, proposer_index=get_beacon_proposer_index(state), ) - if data.target_epoch == get_current_epoch(state): - state.current_epoch_attestations.append(pending_attestation) - else: + + assert data.target_epoch in (get_previous_epoch(state), get_current_epoch(state)) + if data.target_epoch == get_previous_epoch(state): + ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) + previous_crosslink = state.previous_crosslinks[data.crosslink.shard] state.previous_epoch_attestations.append(pending_attestation) + if data.target_epoch == get_current_epoch(state): + ffg_data = (state.current_justified_epoch, state.current_justified_root, get_current_epoch(state)) + previous_crosslink = state.current_crosslinks[data.crosslink.shard] + state.current_epoch_attestations.append(pending_attestation) + + # Check FFG data, crosslink data, and signature + assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) + assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_CROSSLINK_EPOCHS) + assert data.crosslink.previous_crosslink_root == hash_tree_root(previous_crosslink) + assert data.crosslink.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] + assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) ``` ##### Deposits From 5fb32fd19b780d172e8738d887fd0675df7ec3ef Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Sun, 5 May 2019 12:10:39 +0100 Subject: [PATCH 28/91] Fix tests --- configs/constant_presets/mainnet.yaml | 4 ++-- configs/constant_presets/minimal.yaml | 4 ++-- specs/core/0_beacon-chain.md | 6 +++--- specs/core/1_custody-game.md | 16 ++++++++-------- specs/core/1_shard-data-chains.md | 16 ++++++++-------- specs/validator/0_beacon-chain-validator.md | 4 ++-- .../block_processing/test_process_attestation.py | 4 ++-- .../epoch_processing/test_process_crosslinks.py | 12 ++++++------ test_libs/pyspec/tests/helpers.py | 15 ++++++++++----- 9 files changed, 43 insertions(+), 38 deletions(-) diff --git a/configs/constant_presets/mainnet.yaml b/configs/constant_presets/mainnet.yaml index 72d0fdc8f..6ac3f422f 100644 --- a/configs/constant_presets/mainnet.yaml +++ b/configs/constant_presets/mainnet.yaml @@ -72,7 +72,7 @@ MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256 # 2**11 (= 2,048) epochs 9 days PERSISTENT_COMMITTEE_PERIOD: 2048 # 2**6 (= 64) epochs ~7 hours -MAX_CROSSLINK_EPOCHS: 64 +MAX_EPOCHS_PER_CROSSLINK: 64 # 2**2 (= 4) epochs 25.6 minutes MIN_EPOCHS_TO_INACTIVITY_PENALTY: 4 @@ -124,4 +124,4 @@ DOMAIN_RANDAO: 1 DOMAIN_ATTESTATION: 2 DOMAIN_DEPOSIT: 3 DOMAIN_VOLUNTARY_EXIT: 4 -DOMAIN_TRANSFER: 5 \ No newline at end of file +DOMAIN_TRANSFER: 5 diff --git a/configs/constant_presets/minimal.yaml b/configs/constant_presets/minimal.yaml index 0a6cab687..caae4623b 100644 --- a/configs/constant_presets/minimal.yaml +++ b/configs/constant_presets/minimal.yaml @@ -71,7 +71,7 @@ MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256 # 2**11 (= 2,048) epochs 9 days PERSISTENT_COMMITTEE_PERIOD: 2048 # 2**6 (= 64) epochs ~7 hours -MAX_CROSSLINK_EPOCHS: 64 +MAX_EPOCHS_PER_CROSSLINK: 64 # 2**2 (= 4) epochs 25.6 minutes MIN_EPOCHS_TO_INACTIVITY_PENALTY: 4 @@ -123,4 +123,4 @@ DOMAIN_RANDAO: 1 DOMAIN_ATTESTATION: 2 DOMAIN_DEPOSIT: 3 DOMAIN_VOLUNTARY_EXIT: 4 -DOMAIN_TRANSFER: 5 \ No newline at end of file +DOMAIN_TRANSFER: 5 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index addaa865e..156a2523d 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -206,10 +206,10 @@ These configurations are updated for releases, but may be out of sync during `de | `SLOTS_PER_HISTORICAL_ROOT` | `2**13` (= 8,192) | slots | ~13 hours | | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY` | `2**8` (= 256) | epochs | ~27 hours | | `PERSISTENT_COMMITTEE_PERIOD` | `2**11` (= 2,048) | epochs | 9 days | -| `MAX_CROSSLINK_EPOCHS` | `2**6` (= 64) | epochs | ~7 hours | +| `MAX_EPOCHS_PER_CROSSLINK` | `2**6` (= 64) | epochs | ~7 hours | | `MIN_EPOCHS_TO_INACTIVITY_PENALTY` | `2**2` (= 4) | epochs | 25.6 minutes | -* `MAX_CROSSLINK_EPOCHS` should be a small constant times `SHARD_COUNT // SLOTS_PER_EPOCH` +* `MAX_EPOCHS_PER_CROSSLINK` should be a small constant times `SHARD_COUNT // SLOTS_PER_EPOCH` ### State list lengths @@ -1715,7 +1715,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) - assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_CROSSLINK_EPOCHS) + assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) assert data.crosslink.previous_crosslink_root == hash_tree_root(previous_crosslink) assert data.crosslink.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index d56526611..f386ee76e 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -265,7 +265,7 @@ The `empty` function accepts and SSZ type as input and returns an object of that def get_custody_chunk_count(attestation: Attestation) -> int: crosslink_start_epoch = attestation.data.latest_crosslink.epoch crosslink_end_epoch = slot_to_epoch(attestation.data.slot) - crosslink_crosslink_length = min(MAX_CROSSLINK_EPOCHS, end_epoch - start_epoch) + crosslink_crosslink_length = min(MAX_EPOCHS_PER_CROSSLINK, end_epoch - start_epoch) chunks_per_epoch = 2 * BYTES_PER_SHARD_BLOCK * SLOTS_PER_EPOCH // BYTES_PER_CUSTODY_CHUNK return crosslink_crosslink_length * chunks_per_epoch ``` @@ -426,10 +426,10 @@ def process_early_derived_secret_reveal(state: BeaconState, # round key slash_validator(state, reveal.revealed_index, reveal.masker_index) else: - # Only a small penalty proportional to proposer slot reward for RANDAO reveal + # Only a small penalty proportional to proposer slot reward for RANDAO reveal # that does not interfere with the custody period - # The penalty is proportional to the max proposer reward - + # The penalty is proportional to the max proposer reward + # Calculate penalty max_proposer_slot_reward = ( get_base_reward(state, reveal.revealed_index) * @@ -448,7 +448,7 @@ def process_early_derived_secret_reveal(state: BeaconState, increase_balance(state, whistleblower_index, whistleblowing_reward - proposer_reward) decrease_balance(state, reveal.revealed_index, penalty) - # Mark this derived secret as exposed so validator cannot be punished repeatedly + # Mark this derived secret as exposed so validator cannot be punished repeatedly state.exposed_derived_secrets[reveal.epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS].append(reveal.revealed_index) ``` @@ -474,7 +474,7 @@ def process_chunk_challenge(state: BeaconState, # Verify the challenge is not a duplicate for record in state.custody_chunk_challenge_records: assert ( - record.crosslink_data_root != challenge.attestation.data.crosslink_data_root or + record.crosslink_data_root != challenge.attestation.data.crosslink.crosslink_data_root or record.chunk_index != challenge.chunk_index ) # Verify depth @@ -486,7 +486,7 @@ def process_chunk_challenge(state: BeaconState, challenger_index=get_beacon_proposer_index(state), responder_index=challenge.responder_index deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink_data_root, + crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, depth=depth, chunk_index=challenge.chunk_index, ) @@ -564,7 +564,7 @@ def process_bit_challenge(state: BeaconState, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink_data_root, + crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, chunk_count=chunk_count, chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), responder_key=challenge.responder_key, diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 33ef8632b..673a64a4a 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -215,7 +215,7 @@ def get_shard_header(block: ShardBlock) -> ShardBlockHeader: def verify_shard_attestation_signature(state: BeaconState, attestation: ShardAttestation) -> None: data = attestation.data - persistent_committee = get_persistent_committee(state, data.shard, data.slot) + persistent_committee = get_persistent_committee(state, data.crosslink.shard, data.slot) assert verify_bitfield(attestation.aggregation_bitfield, len(persistent_committee)) pubkeys = [] for i, index in enumerate(persistent_committee): @@ -225,7 +225,7 @@ def verify_shard_attestation_signature(state: BeaconState, pubkeys.append(validator.pubkey) assert bls_verify( pubkey=bls_aggregate_pubkeys(pubkeys), - message_hash=data.shard_block_root, + message_hash=data.crosslink.shard_block_root, signature=attestation.aggregate_signature, domain=get_domain(state, slot_to_epoch(data.slot), DOMAIN_SHARD_ATTESTER) ) @@ -312,7 +312,7 @@ def is_valid_shard_block(beacon_blocks: List[BeaconBlock], for _, attestation in enumerate(block.attestations): assert max(GENESIS_SHARD_SLOT, block.slot - SLOTS_PER_EPOCH) <= attestation.data.slot assert attestation.data.slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY - assert attestation.data.shard == block.shard + assert attestation.data.crosslink.shard == block.shard verify_shard_attestation_signature(beacon_state, attestation) # Check signature @@ -343,11 +343,11 @@ def is_valid_shard_attestation(valid_shard_blocks: List[ShardBlock], # Check shard block shard_block = next( block for block in valid_shard_blocks if - signing_root(block) == candidate.attestation.data.shard_block_root + signing_root(block) == candidate.attestation.data.crosslink.shard_block_root , None) assert shard_block != None assert shard_block.slot == attestation.data.slot - assert shard_block.shard == attestation.data.shard + assert shard_block.shard == attestation.data.crosslink.shard # Check signature verify_shard_attestation_signature(beacon_state, attestation) @@ -382,18 +382,18 @@ def is_valid_beacon_attestation(shard: Shard, else: previous_attestation = next( attestation for attestation in valid_attestations if - attestation.data.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root + attestation.data.crosslink.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root , None) assert previous_attestation != None assert candidate.data.previous_attestation.epoch < slot_to_epoch(candidate.data.slot) # Check crosslink data root start_epoch = state.latest_crosslinks[shard].epoch - end_epoch = min(slot_to_epoch(candidate.data.slot) - CROSSLINK_LOOKBACK, start_epoch + MAX_CROSSLINK_EPOCHS) + end_epoch = min(slot_to_epoch(candidate.data.slot) - CROSSLINK_LOOKBACK, start_epoch + MAX_EPOCHS_PER_CROSSLINK) blocks = [] for slot in range(start_epoch * SLOTS_PER_EPOCH, end_epoch * SLOTS_PER_EPOCH): blocks.append(shard_blocks[slot]) - assert candidate.data.crosslink_data_root == compute_crosslink_data_root(blocks) + assert candidate.data.crosslink.crosslink_data_root == compute_crosslink_data_root(blocks) return True ``` diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index ca7f0eb3b..61f80f68e 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -271,7 +271,7 @@ _Note:_ This can be looked up in the state using: ##### Shard -Set `attestation_data.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. ##### Previous crosslink root @@ -279,7 +279,7 @@ Set `attestation_data.previous_crosslink_root = hash_tree_root(head_state.curren ##### Crosslink data root -Set `attestation_data.crosslink_data_root = ZERO_HASH`. +Set `attestation_data.crosslink.crosslink_data_root = ZERO_HASH`. _Note:_ This is a stub for phase 0. diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index bcf71376c..24cafc275 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -113,7 +113,7 @@ def test_non_zero_crosslink_data_root(state): attestation = get_valid_attestation(state) state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY - attestation.data.crosslink_data_root = b'\x42' * 32 + attestation.data.crosslink.crosslink_data_root = b'\x42' * 32 pre_state, post_state = run_attestation_processing(state, attestation, False) @@ -126,7 +126,7 @@ def test_bad_previous_crosslink(state): for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): next_slot(state) - state.current_crosslinks[attestation.data.shard].epoch += 10 + state.current_crosslinks[attestation.data.crosslink.shard].epoch += 10 pre_state, post_state = run_attestation_processing(state, attestation, False) diff --git a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py index d6765e3a7..29e7347b1 100644 --- a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py +++ b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py @@ -64,7 +64,7 @@ def test_single_crosslink_update_from_current_epoch(state): pre_state, post_state = run_process_crosslinks(state) - shard = attestation.data.shard + shard = attestation.data.crosslink.shard assert post_state.previous_crosslinks[shard] != post_state.current_crosslinks[shard] assert pre_state.current_crosslinks[shard] != post_state.current_crosslinks[shard] @@ -84,11 +84,11 @@ def test_single_crosslink_update_from_previous_epoch(state): pre_state, post_state = run_process_crosslinks(state) crosslink_deltas = get_crosslink_deltas(state) - shard = attestation.data.shard + shard = attestation.data.crosslink.shard assert post_state.previous_crosslinks[shard] != post_state.current_crosslinks[shard] assert pre_state.current_crosslinks[shard] != post_state.current_crosslinks[shard] # ensure rewarded - for index in get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.shard): + for index in get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.crosslink.shard): assert crosslink_deltas[0][index] > 0 assert crosslink_deltas[1][index] == 0 @@ -108,7 +108,7 @@ def test_double_late_crosslink(state): for slot in range(spec.SLOTS_PER_EPOCH): attestation_2 = get_valid_attestation(state) - if attestation_2.data.shard == attestation_1.data.shard: + if attestation_2.data.crosslink.shard == attestation_1.data.crosslink.shard: break next_slot(state) fill_aggregate_attestation(state, attestation_2) @@ -124,12 +124,12 @@ def test_double_late_crosslink(state): pre_state, post_state = run_process_crosslinks(state) crosslink_deltas = get_crosslink_deltas(state) - shard = attestation_2.data.shard + shard = attestation_2.data.crosslink.shard # ensure that the current crosslinks were not updated by the second attestation assert post_state.previous_crosslinks[shard] == post_state.current_crosslinks[shard] # ensure no reward, only penalties for the failed crosslink - for index in get_crosslink_committee(state, attestation_2.data.target_epoch, attestation_2.data.shard): + for index in get_crosslink_committee(state, attestation_2.data.target_epoch, attestation_2.data.crosslink.shard): assert crosslink_deltas[0][index] == 0 assert crosslink_deltas[1][index] > 0 diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index 3b9b6904d..1ea299453 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -10,6 +10,7 @@ from eth2spec.utils.minimal_ssz import signing_root from eth2spec.phase0.spec import ( # constants ZERO_HASH, + MAX_EPOCHS_PER_CROSSLINK, # SSZ Attestation, AttestationData, @@ -17,6 +18,7 @@ from eth2spec.phase0.spec import ( AttesterSlashing, BeaconBlock, BeaconBlockHeader, + Crosslink, Deposit, DepositData, Eth1Data, @@ -174,14 +176,17 @@ def build_attestation_data(state, slot, shard): crosslinks = state.current_crosslinks if slot_to_epoch(slot) == get_current_epoch(state) else state.previous_crosslinks return AttestationData( - shard=shard, beacon_block_root=block_root, source_epoch=justified_epoch, source_root=justified_block_root, target_epoch=slot_to_epoch(slot), target_root=epoch_boundary_root, - crosslink_data_root=spec.ZERO_HASH, - previous_crosslink_root=hash_tree_root(crosslinks[shard]), + crosslink=Crosslink( + shard=shard, + epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), + crosslink_data_root=spec.ZERO_HASH, + previous_crosslink_root=hash_tree_root(crosslinks[shard]), + ), ) @@ -288,7 +293,7 @@ def get_valid_attestation(state, slot=None): attestation_data = build_attestation_data(state, slot, shard) - crosslink_committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.shard) + crosslink_committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.crosslink.shard) committee_size = len(crosslink_committee) bitfield_length = (committee_size + 7) // 8 @@ -381,7 +386,7 @@ def get_attestation_signature(state, attestation_data, privkey, custody_bit=0b0) def fill_aggregate_attestation(state, attestation): - crosslink_committee = get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.shard) + crosslink_committee = get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.crosslink.shard) for i in range(len(crosslink_committee)): attestation.aggregation_bitfield = set_bitfield_bit(attestation.aggregation_bitfield, i) From b15105e1cbb4a597fea233b16eac23ae911570a6 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 17:34:03 +0100 Subject: [PATCH 29/91] Address Danny's comment --- specs/core/0_beacon-chain.md | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 156a2523d..0bd85478e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1298,22 +1298,17 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat ```python def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: - shard_attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] - candidate_crosslinks = [ - c for c in [a.data.crosslink for a in shard_attestations] - if hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)) - ] - if len(candidate_crosslinks) == 0: - return Crosslink(), [] - - def get_attestations_for(crosslink: Crosslink) -> List[PendingAttestation]: - return [a for a in shard_attestations if a.data.crosslink == crosslink] - # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) - winning_crosslink = max(candidate_crosslinks, key=lambda crosslink: ( - get_attesting_balance(state, get_attestations_for(crosslink)), crosslink.crosslink_data_root + attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] + crosslinks = list(filter( + lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)), + [a.data.crosslink for a in attestations] )) - - return winning_crosslink, get_unslashed_attesting_indices(state, get_attestations_for(winning_crosslink)) + # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) + winning_crosslink = max(crosslinks, key=lambda c: ( + get_attesting_balance(state, [a for a in attestations if a.data.crosslink == c]), c.crosslink_data_root + ), default=Crosslink()) + winning_attestations = [a for a in attestations if a.data.crosslink == winning_crosslink] + return winning_crosslink, get_unslashed_attesting_indices(state, winning_attestations) ``` #### Justification and finalization From 8b1a2edb7c650e0e0205e11c44f44a2a578ce392 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 17:53:49 +0100 Subject: [PATCH 30/91] Fix genesis bug --- specs/core/0_beacon-chain.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 9aed2001a..04ab38e18 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -767,7 +767,7 @@ def get_epoch_start_shard(state: BeaconState, epoch: Epoch) -> Shard: ```python def get_attestation_data_slot(state: BeaconState, data: AttestationData) -> Slot: committee_count = get_epoch_committee_count(state, data.target_epoch) - offset = (data.shard + SHARD_COUNT - get_epoch_start_shard(state, data.target_epoch)) % SHARD_COUNT + offset = (data.crosslink.shard + SHARD_COUNT - get_epoch_start_shard(state, data.target_epoch)) % SHARD_COUNT return get_epoch_start_slot(data.target_epoch) + offset // (committee_count // SLOTS_PER_EPOCH) ``` @@ -1705,14 +1705,14 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: ) assert data.target_epoch in (get_previous_epoch(state), get_current_epoch(state)) - if data.target_epoch == get_previous_epoch(state): - ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) - previous_crosslink = state.previous_crosslinks[data.crosslink.shard] - state.previous_epoch_attestations.append(pending_attestation) if data.target_epoch == get_current_epoch(state): ffg_data = (state.current_justified_epoch, state.current_justified_root, get_current_epoch(state)) previous_crosslink = state.current_crosslinks[data.crosslink.shard] state.current_epoch_attestations.append(pending_attestation) + else: + ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) + previous_crosslink = state.previous_crosslinks[data.crosslink.shard] + state.previous_epoch_attestations.append(pending_attestation) # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) From f4db9ebae00338ba336943a08271012099bcef1d Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 18:26:14 +0100 Subject: [PATCH 31/91] Renamings triggered by HW comment --- specs/core/0_beacon-chain.md | 26 +++++++++---------- specs/core/0_fork-choice.md | 2 +- specs/core/1_custody-game.md | 14 +++++----- specs/core/1_shard-data-chains.md | 16 ++++++------ specs/validator/0_beacon-chain-validator.md | 6 ++--- .../test_process_attestation.py | 2 +- .../test_process_block_header.py | 4 +-- test_libs/pyspec/tests/helpers.py | 12 ++++----- test_libs/pyspec/tests/test_sanity.py | 6 ++--- 9 files changed, 44 insertions(+), 44 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 04ab38e18..afca9eb1b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -287,9 +287,9 @@ The types are defined topologically to aid in facilitating an executable version # Epoch number 'epoch': 'uint64', # Root of the previous crosslink - 'previous_crosslink_root': 'bytes32', + 'parent_crosslink_root': 'bytes32', # Root of the crosslinked shard data since the previous crosslink - 'crosslink_data_root': 'bytes32', + 'data_root': 'bytes32', } ``` @@ -369,7 +369,7 @@ The types are defined topologically to aid in facilitating an executable version ```python { 'slot': 'uint64', - 'previous_block_root': 'bytes32', + 'parent_block_root': 'bytes32', 'state_root': 'bytes32', 'block_body_root': 'bytes32', 'signature': 'bytes96', @@ -536,7 +536,7 @@ The types are defined topologically to aid in facilitating an executable version { # Header 'slot': 'uint64', - 'previous_block_root': 'bytes32', + 'parent_block_root': 'bytes32', 'state_root': 'bytes32', 'body': BeaconBlockBody, 'signature': 'bytes96', @@ -1306,12 +1306,12 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] crosslinks = list(filter( - lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)), + lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_crosslink_root, hash_tree_root(c)), [a.data.crosslink for a in attestations] )) # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) winning_crosslink = max(crosslinks, key=lambda c: ( - get_attesting_balance(state, [a for a in attestations if a.data.crosslink == c]), c.crosslink_data_root + get_attesting_balance(state, [a for a in attestations if a.data.crosslink == c]), c.data_root ), default=Crosslink()) winning_attestations = [a for a in attestations if a.data.crosslink == winning_crosslink] return winning_crosslink, get_unslashed_attesting_indices(state, winning_attestations) @@ -1586,11 +1586,11 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: # Verify that the slots match assert block.slot == state.slot # Verify that the parent matches - assert block.previous_block_root == signing_root(state.latest_block_header) + assert block.parent_block_root == signing_root(state.latest_block_header) # Save current block as the new latest block state.latest_block_header = BeaconBlockHeader( slot=block.slot, - previous_block_root=block.previous_block_root, + parent_block_root=block.parent_block_root, block_body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed @@ -1707,18 +1707,18 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: assert data.target_epoch in (get_previous_epoch(state), get_current_epoch(state)) if data.target_epoch == get_current_epoch(state): ffg_data = (state.current_justified_epoch, state.current_justified_root, get_current_epoch(state)) - previous_crosslink = state.current_crosslinks[data.crosslink.shard] + parent_crosslink = state.current_crosslinks[data.crosslink.shard] state.current_epoch_attestations.append(pending_attestation) else: ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) - previous_crosslink = state.previous_crosslinks[data.crosslink.shard] + parent_crosslink = state.previous_crosslinks[data.crosslink.shard] state.previous_epoch_attestations.append(pending_attestation) # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) - assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) - assert data.crosslink.previous_crosslink_root == hash_tree_root(previous_crosslink) - assert data.crosslink.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] + assert data.crosslink.epoch == min(data.target_epoch, parent_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) + assert data.crosslink.parent_crosslink_root == hash_tree_root(parent_crosslink) + assert data.crosslink.data_root == ZERO_HASH # [to be removed in phase 1] assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) ``` diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index 549e9e207..891120de8 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -38,7 +38,7 @@ All terminology, constants, functions, and protocol mechanics defined in the [Ph Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Clients download and process blocks and maintain a view of what is the current "canonical chain", terminating at the current "head". For a beacon block, `block`, to be processed by a node, the following conditions must be met: -* The parent block with root `block.previous_block_root` has been processed and accepted. +* The parent block with root `block.parent_block_root` has been processed and accepted. * An Ethereum 1.0 block pointed to by the `state.latest_eth1_data.block_hash` has been processed and accepted. * The node's Unix time is greater than or equal to `state.genesis_time + block.slot * SECONDS_PER_SLOT`. diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index e259b82a5..a030c577b 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -147,7 +147,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenger_index': ValidatorIndex, 'responder_index': ValidatorIndex, 'deadline': Epoch, - 'crosslink_data_root': Hash, + 'data_root': Hash, 'depth': 'uint64', 'chunk_index': 'uint64', } @@ -161,7 +161,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenger_index': ValidatorIndex, 'responder_index': ValidatorIndex, 'deadline': Epoch, - 'crosslink_data_root': Hash, + 'data_root': Hash, 'chunk_count': 'uint64', 'chunk_bits_merkle_root': Hash, 'responder_key': BLSSignature, @@ -474,7 +474,7 @@ def process_chunk_challenge(state: BeaconState, # Verify the challenge is not a duplicate for record in state.custody_chunk_challenge_records: assert ( - record.crosslink_data_root != challenge.attestation.data.crosslink.crosslink_data_root or + record.data_root != challenge.attestation.data.crosslink.data_root or record.chunk_index != challenge.chunk_index ) # Verify depth @@ -486,7 +486,7 @@ def process_chunk_challenge(state: BeaconState, challenger_index=get_beacon_proposer_index(state), responder_index=challenge.responder_index deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, + data_root=challenge.attestation.data.crosslink.data_root, depth=depth, chunk_index=challenge.chunk_index, ) @@ -564,7 +564,7 @@ def process_bit_challenge(state: BeaconState, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, + data_root=challenge.attestation.data.crosslink.data_root, chunk_count=chunk_count, chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), responder_key=challenge.responder_key, @@ -610,7 +610,7 @@ def process_chunk_challenge_response(state: BeaconState, branch=response.data_branch, depth=challenge.depth, index=response.chunk_index, - root=challenge.crosslink_data_root, + root=challenge.data_root, ) # Clear the challenge records = state.custody_chunk_challenge_records @@ -632,7 +632,7 @@ def process_bit_challenge_response(state: BeaconState, branch=response.data_branch, depth=math.log2(next_power_of_two(challenge.chunk_count)), index=response.chunk_index, - root=challenge.crosslink_data_root, + root=challenge.data_root, ) # Verify the chunk bit leaf matches the challenge data assert verify_merkle_branch( diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 9fbc69692..316bd0068 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -78,7 +78,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'previous_block_root': Hash, + 'parent_block_root': Hash, 'data': ShardBlockBody, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -93,7 +93,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'previous_block_root': Hash, + 'parent_block_root': Hash, 'body_root': Hash, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -201,7 +201,7 @@ def get_shard_header(block: ShardBlock) -> ShardBlockHeader: slot: block.slot, shard: block.shard, beacon_chain_root: block.beacon_chain_root, - previous_block_root: block.previous_block_root, + parent_block_root: block.parent_block_root, body_root: hash_tree_root(block.body), state_root: block.state_root, attestations: block.attestations, @@ -296,11 +296,11 @@ def is_valid_shard_block(beacon_blocks: List[BeaconBlock], # Check parent block if block.slot == PHASE_1_GENESIS_SLOT: - assert candidate.previous_block_root == ZERO_HASH + assert candidate.parent_block_root == ZERO_HASH else: parent_block = next( block for block in valid_shard_blocks if - signing_root(block) == candidate.previous_block_root + signing_root(block) == candidate.parent_block_root , None) assert parent_block != None assert parent_block.shard == block.shard @@ -378,11 +378,11 @@ def is_valid_beacon_attestation(shard: Shard, # Check previous attestation if candidate.data.previous_crosslink.epoch <= PHASE_1_GENESIS_EPOCH: - assert candidate.data.previous_crosslink.crosslink_data_root == ZERO_HASH + assert candidate.data.previous_crosslink.data_root == ZERO_HASH else: previous_attestation = next( attestation for attestation in valid_attestations if - attestation.data.crosslink.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root + attestation.data.crosslink.data_root == candidate.data.previous_crosslink.data_root , None) assert previous_attestation != None assert candidate.data.previous_attestation.epoch < slot_to_epoch(candidate.data.slot) @@ -393,7 +393,7 @@ def is_valid_beacon_attestation(shard: Shard, blocks = [] for slot in range(start_epoch * SLOTS_PER_EPOCH, end_epoch * SLOTS_PER_EPOCH): blocks.append(shard_blocks[slot]) - assert candidate.data.crosslink.crosslink_data_root == compute_crosslink_data_root(blocks) + assert candidate.data.crosslink.data_root == compute_crosslink_data_root(blocks) return True ``` diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index cfe6a8f1f..dae9ac85f 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -152,7 +152,7 @@ Set `block.slot = slot` where `slot` is the current slot at which the validator ##### Parent root -Set `block.previous_block_root = signing_root(parent)`. +Set `block.parent_block_root = signing_root(parent)`. ##### State root @@ -275,11 +275,11 @@ Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associ ##### Previous crosslink root -Set `attestation_data.previous_crosslink_root = hash_tree_root(head_state.current_crosslinks[shard])`. +Set `attestation_data.parent_crosslink_root = hash_tree_root(head_state.current_crosslinks[shard])`. ##### Crosslink data root -Set `attestation_data.crosslink.crosslink_data_root = ZERO_HASH`. +Set `attestation_data.crosslink.data_root = ZERO_HASH`. *Note*: This is a stub for Phase 0. diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 24cafc275..fcbc68fe9 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -113,7 +113,7 @@ def test_non_zero_crosslink_data_root(state): attestation = get_valid_attestation(state) state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY - attestation.data.crosslink.crosslink_data_root = b'\x42' * 32 + attestation.data.crosslink.data_root = b'\x42' * 32 pre_state, post_state = run_attestation_processing(state, attestation, False) diff --git a/test_libs/pyspec/tests/block_processing/test_process_block_header.py b/test_libs/pyspec/tests/block_processing/test_process_block_header.py index b35b0a9c1..e4fa516b1 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_block_header.py +++ b/test_libs/pyspec/tests/block_processing/test_process_block_header.py @@ -53,9 +53,9 @@ def test_invalid_slot(state): return pre_state, block, None -def test_invalid_previous_block_root(state): +def test_invalid_parent_block_root(state): block = build_empty_block_for_next_slot(state) - block.previous_block_root = b'\12' * 32 # invalid prev root + block.parent_block_root = b'\12' * 32 # invalid prev root pre_state, post_state = run_block_header_processing(state, block, valid=False) return pre_state, block, None diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index 1ea299453..e7ee826db 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -128,7 +128,7 @@ def build_empty_block_for_next_slot(state): previous_block_header = deepcopy(state.latest_block_header) if previous_block_header.state_root == spec.ZERO_HASH: previous_block_header.state_root = state.hash_tree_root() - empty_block.previous_block_root = signing_root(previous_block_header) + empty_block.parent_block_root = signing_root(previous_block_header) return empty_block @@ -155,7 +155,7 @@ def build_attestation_data(state, slot, shard): assert state.slot >= slot if slot == state.slot: - block_root = build_empty_block_for_next_slot(state).previous_block_root + block_root = build_empty_block_for_next_slot(state).parent_block_root else: block_root = get_block_root_at_slot(state, slot) @@ -184,8 +184,8 @@ def build_attestation_data(state, slot, shard): crosslink=Crosslink( shard=shard, epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), - crosslink_data_root=spec.ZERO_HASH, - previous_crosslink_root=hash_tree_root(crosslinks[shard]), + data_root=spec.ZERO_HASH, + parent_crosslink_root=hash_tree_root(crosslinks[shard]), ), ) @@ -240,12 +240,12 @@ def get_valid_proposer_slashing(state): header_1 = BeaconBlockHeader( slot=slot, - previous_block_root=ZERO_HASH, + parent_block_root=ZERO_HASH, state_root=ZERO_HASH, block_body_root=ZERO_HASH, ) header_2 = deepcopy(header_1) - header_2.previous_block_root = b'\x02' * 32 + header_2.parent_block_root = b'\x02' * 32 header_2.slot = slot + 1 domain = get_domain( diff --git a/test_libs/pyspec/tests/test_sanity.py b/test_libs/pyspec/tests/test_sanity.py index 1b4d20f4c..83ba9cc11 100644 --- a/test_libs/pyspec/tests/test_sanity.py +++ b/test_libs/pyspec/tests/test_sanity.py @@ -68,7 +68,7 @@ def test_empty_block_transition(state): state_transition(test_state, block) assert len(test_state.eth1_data_votes) == len(state.eth1_data_votes) + 1 - assert get_block_root_at_slot(test_state, state.slot) == block.previous_block_root + assert get_block_root_at_slot(test_state, state.slot) == block.parent_block_root return state, [block], test_state @@ -82,7 +82,7 @@ def test_skipped_slots(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.previous_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_block_root return state, [block], test_state @@ -96,7 +96,7 @@ def test_empty_epoch_transition(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.previous_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_block_root return state, [block], test_state From ea60fb632c3a07d2dae29b6ca8d37cea204bc6e7 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 20:49:46 +0100 Subject: [PATCH 32/91] More renaming --- specs/core/0_beacon-chain.md | 14 +++++++------- specs/core/0_fork-choice.md | 2 +- specs/core/1_shard-data-chains.md | 10 +++++----- specs/validator/0_beacon-chain-validator.md | 4 ++-- .../block_processing/test_process_block_header.py | 2 +- test_libs/pyspec/tests/helpers.py | 12 ++++++------ test_libs/pyspec/tests/test_sanity.py | 6 +++--- 7 files changed, 25 insertions(+), 25 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index afca9eb1b..ccba4c817 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -287,7 +287,7 @@ The types are defined topologically to aid in facilitating an executable version # Epoch number 'epoch': 'uint64', # Root of the previous crosslink - 'parent_crosslink_root': 'bytes32', + 'parent_root': 'bytes32', # Root of the crosslinked shard data since the previous crosslink 'data_root': 'bytes32', } @@ -369,7 +369,7 @@ The types are defined topologically to aid in facilitating an executable version ```python { 'slot': 'uint64', - 'parent_block_root': 'bytes32', + 'parent_root': 'bytes32', 'state_root': 'bytes32', 'block_body_root': 'bytes32', 'signature': 'bytes96', @@ -536,7 +536,7 @@ The types are defined topologically to aid in facilitating an executable version { # Header 'slot': 'uint64', - 'parent_block_root': 'bytes32', + 'parent_root': 'bytes32', 'state_root': 'bytes32', 'body': BeaconBlockBody, 'signature': 'bytes96', @@ -1306,7 +1306,7 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] crosslinks = list(filter( - lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_crosslink_root, hash_tree_root(c)), + lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_root, hash_tree_root(c)), [a.data.crosslink for a in attestations] )) # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) @@ -1586,11 +1586,11 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: # Verify that the slots match assert block.slot == state.slot # Verify that the parent matches - assert block.parent_block_root == signing_root(state.latest_block_header) + assert block.parent_root == signing_root(state.latest_block_header) # Save current block as the new latest block state.latest_block_header = BeaconBlockHeader( slot=block.slot, - parent_block_root=block.parent_block_root, + parent_root=block.parent_root, block_body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed @@ -1717,7 +1717,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) assert data.crosslink.epoch == min(data.target_epoch, parent_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) - assert data.crosslink.parent_crosslink_root == hash_tree_root(parent_crosslink) + assert data.crosslink.parent_root == hash_tree_root(parent_crosslink) assert data.crosslink.data_root == ZERO_HASH # [to be removed in phase 1] assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) ``` diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index 891120de8..91c3e27ee 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -38,7 +38,7 @@ All terminology, constants, functions, and protocol mechanics defined in the [Ph Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Clients download and process blocks and maintain a view of what is the current "canonical chain", terminating at the current "head". For a beacon block, `block`, to be processed by a node, the following conditions must be met: -* The parent block with root `block.parent_block_root` has been processed and accepted. +* The parent block with root `block.parent_root` has been processed and accepted. * An Ethereum 1.0 block pointed to by the `state.latest_eth1_data.block_hash` has been processed and accepted. * The node's Unix time is greater than or equal to `state.genesis_time + block.slot * SECONDS_PER_SLOT`. diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 316bd0068..ca59c3ebc 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -78,7 +78,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'parent_block_root': Hash, + 'parent_root': Hash, 'data': ShardBlockBody, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -93,7 +93,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'parent_block_root': Hash, + 'parent_root': Hash, 'body_root': Hash, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -201,7 +201,7 @@ def get_shard_header(block: ShardBlock) -> ShardBlockHeader: slot: block.slot, shard: block.shard, beacon_chain_root: block.beacon_chain_root, - parent_block_root: block.parent_block_root, + parent_root: block.parent_root, body_root: hash_tree_root(block.body), state_root: block.state_root, attestations: block.attestations, @@ -296,11 +296,11 @@ def is_valid_shard_block(beacon_blocks: List[BeaconBlock], # Check parent block if block.slot == PHASE_1_GENESIS_SLOT: - assert candidate.parent_block_root == ZERO_HASH + assert candidate.parent_root == ZERO_HASH else: parent_block = next( block for block in valid_shard_blocks if - signing_root(block) == candidate.parent_block_root + signing_root(block) == candidate.parent_root , None) assert parent_block != None assert parent_block.shard == block.shard diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index dae9ac85f..a0173ebe9 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -152,7 +152,7 @@ Set `block.slot = slot` where `slot` is the current slot at which the validator ##### Parent root -Set `block.parent_block_root = signing_root(parent)`. +Set `block.parent_root = signing_root(parent)`. ##### State root @@ -275,7 +275,7 @@ Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associ ##### Previous crosslink root -Set `attestation_data.parent_crosslink_root = hash_tree_root(head_state.current_crosslinks[shard])`. +Set `attestation_data.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. ##### Crosslink data root diff --git a/test_libs/pyspec/tests/block_processing/test_process_block_header.py b/test_libs/pyspec/tests/block_processing/test_process_block_header.py index e4fa516b1..36c729821 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_block_header.py +++ b/test_libs/pyspec/tests/block_processing/test_process_block_header.py @@ -55,7 +55,7 @@ def test_invalid_slot(state): def test_invalid_parent_block_root(state): block = build_empty_block_for_next_slot(state) - block.parent_block_root = b'\12' * 32 # invalid prev root + block.parent_root = b'\12' * 32 # invalid prev root pre_state, post_state = run_block_header_processing(state, block, valid=False) return pre_state, block, None diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index e7ee826db..202488576 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -128,7 +128,7 @@ def build_empty_block_for_next_slot(state): previous_block_header = deepcopy(state.latest_block_header) if previous_block_header.state_root == spec.ZERO_HASH: previous_block_header.state_root = state.hash_tree_root() - empty_block.parent_block_root = signing_root(previous_block_header) + empty_block.parent_root = signing_root(previous_block_header) return empty_block @@ -155,7 +155,7 @@ def build_attestation_data(state, slot, shard): assert state.slot >= slot if slot == state.slot: - block_root = build_empty_block_for_next_slot(state).parent_block_root + block_root = build_empty_block_for_next_slot(state).parent_root else: block_root = get_block_root_at_slot(state, slot) @@ -185,7 +185,7 @@ def build_attestation_data(state, slot, shard): shard=shard, epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), data_root=spec.ZERO_HASH, - parent_crosslink_root=hash_tree_root(crosslinks[shard]), + parent_root=hash_tree_root(crosslinks[shard]), ), ) @@ -240,12 +240,12 @@ def get_valid_proposer_slashing(state): header_1 = BeaconBlockHeader( slot=slot, - parent_block_root=ZERO_HASH, + parent_root=ZERO_HASH, state_root=ZERO_HASH, - block_body_root=ZERO_HASH, + body_root=ZERO_HASH, ) header_2 = deepcopy(header_1) - header_2.parent_block_root = b'\x02' * 32 + header_2.parent_root = b'\x02' * 32 header_2.slot = slot + 1 domain = get_domain( diff --git a/test_libs/pyspec/tests/test_sanity.py b/test_libs/pyspec/tests/test_sanity.py index 83ba9cc11..0cb1b9be3 100644 --- a/test_libs/pyspec/tests/test_sanity.py +++ b/test_libs/pyspec/tests/test_sanity.py @@ -68,7 +68,7 @@ def test_empty_block_transition(state): state_transition(test_state, block) assert len(test_state.eth1_data_votes) == len(state.eth1_data_votes) + 1 - assert get_block_root_at_slot(test_state, state.slot) == block.parent_block_root + assert get_block_root_at_slot(test_state, state.slot) == block.parent_root return state, [block], test_state @@ -82,7 +82,7 @@ def test_skipped_slots(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.parent_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_root return state, [block], test_state @@ -96,7 +96,7 @@ def test_empty_epoch_transition(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.parent_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_root return state, [block], test_state From 54fd822b55aa6fd9426253716c0e6da70d43b367 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 6 May 2019 17:36:55 -0700 Subject: [PATCH 33/91] Update simple-serialize.md typo --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 343c967d4..834f56645 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -85,7 +85,7 @@ return b"\x01" if value is True else b"\x00" ### Containers, vectors, lists ```python -# Reccursively serialize +# Recursively serialize fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in value] variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in value] From 24966d71be58066dc9586d355ec219220e5bd94a Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 6 May 2019 17:54:40 -0700 Subject: [PATCH 34/91] Update 0_beacon-chain-validator.md --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 600ed0839..0a1997788 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -134,7 +134,7 @@ Once a validator is activated, the validator is assigned [responsibilities](#bea ## Beacon chain responsibilities -A validator has two primary responsibilities to the beacon chain: [proposing blocks](block-proposal) and [creating attestations](attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. +A validator has two primary responsibilities to the beacon chain: [proposing blocks](#block-proposal) and [creating attestations](#attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. ### Block proposal From fc1239c0ff8fda299d4a5e5a532bb2203e130446 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 6 May 2019 18:50:20 -0700 Subject: [PATCH 35/91] Add clarity around merkleize on a single chunk --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 834f56645..c02d7fbed 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -111,7 +111,7 @@ Because serialization is an injective function (i.e. two distinct objects of the We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. +* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. Note that `merkleize` on a single chunk is simply that chunk, i.e. the identity when the number of chunks is one. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 964e55cd4a8b8bfecbd70e279c13e69c6c977e22 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 08:52:56 +0100 Subject: [PATCH 36/91] block_body_root => body_root --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ccba4c817..9f80fe173 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -371,7 +371,7 @@ The types are defined topologically to aid in facilitating an executable version 'slot': 'uint64', 'parent_root': 'bytes32', 'state_root': 'bytes32', - 'block_body_root': 'bytes32', + 'body_root': 'bytes32', 'signature': 'bytes96', } ``` @@ -1591,7 +1591,7 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: state.latest_block_header = BeaconBlockHeader( slot=block.slot, parent_root=block.parent_root, - block_body_root=hash_tree_root(block.body), + body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed proposer = state.validator_registry[get_beacon_proposer_index(state)] From f371daeb204c1ede12fcb1bf6e5b15263f1bc504 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 7 May 2019 09:01:07 +0100 Subject: [PATCH 37/91] Update specs/core/0_beacon-chain.md Co-Authored-By: JustinDrake --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 8d999c65c..aaa85e425 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1265,7 +1265,7 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. ```python -def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconBlock: +def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: # Process slots (including those with no blocks) since block process_slots(state, block.slot) # Process block From 50009ea85bd1fd96b81c3f3b6fb3f39b58662050 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 10:12:33 +0100 Subject: [PATCH 38/91] Implement HW's exception-handling suggestion --- scripts/phase0/build_spec.py | 5 +++-- specs/core/0_beacon-chain.md | 17 +++++++++++------ 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index 54adfdde7..a1f0209bb 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -5,7 +5,8 @@ import function_puller def build_phase0_spec(sourcefile, outfile): code_lines = [] code_lines.append(""" - + +import copy from typing import ( Any, Callable, @@ -88,7 +89,7 @@ def apply_constants_preset(preset: Dict[str, Any]): # Deal with derived constants global_vars['GENESIS_EPOCH'] = slot_to_epoch(GENESIS_SLOT) - + # Initialize SSZ types again, to account for changed lengths init_SSZ_types() """) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index aaa85e425..84d780ad6 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1262,16 +1262,21 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], ## Beacon chain state transition function -The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. +The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. ```python def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: - # Process slots (including those with no blocks) since block - process_slots(state, block.slot) - # Process block - process_block(state, block) - # Return post-state + pre_state = copy.deepcopy(state) + try: + # Process slots (including those with no blocks) since block + process_slots(state, block.slot) + # Process block + process_block(state, block) + except Exception: + # State is not advanced on exceptions (e.g. a failed `assert` or an out-of-range list access) + return pre_state return state + ``` ```python From b1520ea96766e34397c161cea3518e7e7f0104be Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 7 May 2019 10:33:51 +0100 Subject: [PATCH 39/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index dd2d3d1a6..f0798f587 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -75,7 +75,7 @@ - [`compute_committee`](#compute_committee) - [`get_crosslink_committee`](#get_crosslink_committee) - [`get_attesting_indices`](#get_attesting_indices) - - [`int_to_bytes1`, `int_to_bytes2`, ...](#int_to_bytes1-int_to_bytes2-) + - [`int_to_bytes`](#int_to_bytes) - [`bytes_to_int`](#bytes_to_int) - [`get_total_balance`](#get_total_balance) - [`get_domain`](#get_domain) @@ -194,8 +194,8 @@ These configurations are updated for releases, but may be out of sync during `de | `GENESIS_SLOT` | `0` | | `GENESIS_EPOCH` | `0` | | `FAR_FUTURE_EPOCH` | `2**64 - 1` | -| `ZERO_HASH` | `int_to_bytes32(0)` | -| `BLS_WITHDRAWAL_PREFIX_BYTE` | `int_to_bytes1(0)` | +| `ZERO_HASH` | `int_to_bytes(0, length=32)` | +| `BLS_WITHDRAWAL_PREFIX_BYTE` | `int_to_bytes(0, length=1)` | ### Time parameters @@ -830,7 +830,7 @@ def generate_seed(state: BeaconState, return hash( get_randao_mix(state, epoch + LATEST_RANDAO_MIXES_LENGTH - MIN_SEED_LOOKAHEAD) + get_active_index_root(state, epoch) + - int_to_bytes32(epoch) + int_to_bytes(epoch, length=32) ) ``` @@ -851,7 +851,7 @@ def get_beacon_proposer_index(state: BeaconState) -> ValidatorIndex: i = 0 while True: candidate_index = first_committee[(epoch + i) % len(first_committee)] - random_byte = hash(seed + int_to_bytes8(i // 32))[i % 32] + random_byte = hash(seed + int_to_bytes(i // 32, length=8))[i % 32] effective_balance = state.validator_registry[candidate_index].effective_balance if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: return candidate_index @@ -888,10 +888,10 @@ def get_shuffled_index(index: ValidatorIndex, index_count: int, seed: Bytes32) - # Swap or not (https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf) # See the 'generalized domain' algorithm on page 3 for round in range(SHUFFLE_ROUND_COUNT): - pivot = bytes_to_int(hash(seed + int_to_bytes1(round))[0:8]) % index_count + pivot = bytes_to_int(hash(seed + int_to_bytes(round, length=1))[0:8]) % index_count flip = (pivot - index) % index_count position = max(index, flip) - source = hash(seed + int_to_bytes1(round) + int_to_bytes4(position // 256)) + source = hash(seed + int_to_bytes(round, length=1) + int_to_bytes(position // 256, length=4)) byte = source[(position % 256) // 8] bit = (byte >> (position % 8)) % 2 index = flip if bit else index @@ -934,9 +934,12 @@ def get_attesting_indices(state: BeaconState, return sorted([index for i, index in enumerate(committee) if get_bitfield_bit(bitfield, i) == 0b1]) ``` -### `int_to_bytes1`, `int_to_bytes2`, ... +### `int_to_bytes` -`int_to_bytes1(x): return x.to_bytes(1, 'little')`, `int_to_bytes2(x): return x.to_bytes(2, 'little')`, and so on for all integers, particularly 1, 2, 3, 4, 8, 32, 48, 96. +```python +def int_to_bytes(integer: int, length: int) -> bytes: + return integer.to_bytes(length, 'little') +``` ### `bytes_to_int` @@ -966,7 +969,7 @@ def get_domain(state: BeaconState, """ epoch = get_current_epoch(state) if message_epoch is None else message_epoch fork_version = state.fork.previous_version if epoch < state.fork.epoch else state.fork.current_version - return bytes_to_int(fork_version + int_to_bytes4(domain_type)) + return bytes_to_int(fork_version + int_to_bytes(domain_type, length=4)) ``` ### `get_bitfield_bit` From 22b06d581dcdc84ca3cfedd9e21af529557d14d6 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 10:57:41 +0100 Subject: [PATCH 40/91] Update instances of int_to_bytes --- scripts/phase0/build_spec.py | 8 ++------ specs/core/0_beacon-chain.md | 4 ++-- specs/core/1_shard-data-chains.md | 2 +- specs/light_client/sync_protocol.md | 2 +- test_generators/shuffling/main.py | 2 +- 5 files changed, 7 insertions(+), 11 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index da5845951..28e5049ae 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -15,15 +15,11 @@ from typing import ( from eth2spec.utils.minimal_ssz import * from eth2spec.utils.bls_stub import * -""") - for i in (1, 2, 3, 4, 8, 32, 48, 96): - code_lines.append("def int_to_bytes%d(x): return x.to_bytes(%d, 'little')" % (i, i)) - - code_lines.append(""" - # stub, will get overwritten by real var SLOTS_PER_EPOCH = 64 +def int_to_bytes(integer: int, length: int) -> bytes: + return integer.to_bytes(length, 'little') Slot = NewType('Slot', int) # uint64 Epoch = NewType('Epoch', int) # uint64 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index f0798f587..1e8170036 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -195,7 +195,7 @@ These configurations are updated for releases, but may be out of sync during `de | `GENESIS_EPOCH` | `0` | | `FAR_FUTURE_EPOCH` | `2**64 - 1` | | `ZERO_HASH` | `int_to_bytes(0, length=32)` | -| `BLS_WITHDRAWAL_PREFIX_BYTE` | `int_to_bytes(0, length=1)` | +| `BLS_WITHDRAWAL_PREFIX` | `0` | ### Time parameters @@ -1840,7 +1840,7 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: # Verify that the pubkey is valid assert ( state.validator_registry[transfer.sender].withdrawal_credentials == - BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:] + int_to_bytes(BLS_WITHDRAWAL_PREFIX, length=1) + hash(transfer.pubkey)[1:] ) # Verify that the signature is valid assert bls_verify(transfer.pubkey, signing_root(transfer), transfer.signature, get_domain(state, DOMAIN_TRANSFER)) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 14eb51193..90ea30841 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -180,7 +180,7 @@ def get_shard_proposer_index(state: BeaconState, slot: Slot) -> ValidatorIndex: # Randomly shift persistent committee persistent_committee = get_persistent_committee(state, shard, slot) - seed = hash(state.current_shuffling_seed + int_to_bytes8(shard) + int_to_bytes8(slot)) + seed = hash(state.current_shuffling_seed + int_to_bytes(shard, length=8) + int_to_bytes(slot, length=8)) random_index = bytes_to_int(seed[0:8]) % len(persistent_committee) persistent_committee = persistent_committee[random_index:] + persistent_committee[:random_index] diff --git a/specs/light_client/sync_protocol.md b/specs/light_client/sync_protocol.md index f6e3d2265..8501c5869 100644 --- a/specs/light_client/sync_protocol.md +++ b/specs/light_client/sync_protocol.md @@ -146,7 +146,7 @@ def compute_committee(header: BeaconBlockHeader, ] def get_switchover_epoch(index): return ( - bytes_to_int(hash(validator_memory.earlier_period_data.seed + int_to_bytes3(index))[0:8]) % + bytes_to_int(hash(validator_memory.earlier_period_data.seed + int_to_bytes(index, length=3))[0:8]) % PERSISTENT_COMMITTEE_PERIOD ) diff --git a/test_generators/shuffling/main.py b/test_generators/shuffling/main.py index 2c4faeb8f..9ca3e2d36 100644 --- a/test_generators/shuffling/main.py +++ b/test_generators/shuffling/main.py @@ -15,7 +15,7 @@ def shuffling_case(seed: spec.Bytes32, count: int): @to_tuple def shuffling_test_cases(): - for seed in [spec.hash(spec.int_to_bytes4(seed_init_value)) for seed_init_value in range(30)]: + for seed in [spec.hash(spec.int_to_bytes(seed_init_value, length=4)) for seed_init_value in range(30)]: for count in [0, 1, 2, 3, 5, 10, 33, 100, 1000]: yield shuffling_case(seed, count) From ecc6429b9eaef80b13ee8f9f72daa76930f4ad25 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 15:01:23 +0100 Subject: [PATCH 41/91] Address Danny's comment --- scripts/phase0/build_spec.py | 3 --- specs/core/0_beacon-chain.md | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index 28e5049ae..e3c431a4a 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -18,9 +18,6 @@ from eth2spec.utils.bls_stub import * # stub, will get overwritten by real var SLOTS_PER_EPOCH = 64 -def int_to_bytes(integer: int, length: int) -> bytes: - return integer.to_bytes(length, 'little') - Slot = NewType('Slot', int) # uint64 Epoch = NewType('Epoch', int) # uint64 Shard = NewType('Shard', int) # uint64 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 1e8170036..a26a4e979 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -194,7 +194,7 @@ These configurations are updated for releases, but may be out of sync during `de | `GENESIS_SLOT` | `0` | | `GENESIS_EPOCH` | `0` | | `FAR_FUTURE_EPOCH` | `2**64 - 1` | -| `ZERO_HASH` | `int_to_bytes(0, length=32)` | +| `ZERO_HASH` | `b'\x00' * 32` | | `BLS_WITHDRAWAL_PREFIX` | `0` | ### Time parameters From 62c44ffce3068b0e9d9146c3e4ca30deb68583eb Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 17:34:19 +0100 Subject: [PATCH 42/91] Refactor to validate_indexed_attestation --- specs/core/0_beacon-chain.md | 54 +++++++++++++++++------------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 963c2ac44..27c7fd461 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -82,7 +82,7 @@ - [`get_bitfield_bit`](#get_bitfield_bit) - [`verify_bitfield`](#verify_bitfield) - [`convert_to_indexed`](#convert_to_indexed) - - [`verify_indexed_attestation`](#verify_indexed_attestation) + - [`validate_indexed_attestation`](#validate_indexed_attestation) - [`is_slashable_attestation_data`](#is_slashable_attestation_data) - [`integer_squareroot`](#integer_squareroot) - [`get_delayed_activation_exit_epoch`](#get_delayed_activation_exit_epoch) @@ -1013,38 +1013,36 @@ def convert_to_indexed(state: BeaconState, attestation: Attestation) -> IndexedA ) ``` -### `verify_indexed_attestation` +### `validate_indexed_attestation` ```python -def verify_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool: +def validate_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> None: """ Verify validity of ``indexed_attestation``. """ bit_0_indices = indexed_attestation.custody_bit_0_indices bit_1_indices = indexed_attestation.custody_bit_1_indices - return ( - # Verify no index has custody bit equal to 1 [to be removed in phase 1] - len(bit_1_indices) == 0 and - # Verify max number of indices - len(bit_0_indices) + len(bit_1_indices) <= MAX_INDICES_PER_ATTESTATION and - # Verify index sets are disjoint - len(set(bit_0_indices).intersection(bit_1_indices)) == 0 and - # Verify indices are sorted - bit_0_indices == sorted(bit_0_indices) and bit_1_indices == sorted(bit_1_indices) and - # Verify aggregate signature - bls_verify_multiple( - pubkeys=[ - bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in bit_0_indices]), - bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in bit_1_indices]), - ], - message_hashes=[ - hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b0)), - hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b1)), - ], - signature=indexed_attestation.signature, - domain=get_domain(state, DOMAIN_ATTESTATION, indexed_attestation.data.target_epoch), - ) + # Verify no index has custody bit equal to 1 [to be removed in phase 1] + assert len(bit_1_indices) == 0 + # Verify max number of indices + assert len(bit_0_indices) + len(bit_1_indices) <= MAX_INDICES_PER_ATTESTATION + # Verify index sets are disjoint + assert len(set(bit_0_indices).intersection(bit_1_indices)) == 0 + # Verify indices are sorted + assert bit_0_indices == sorted(bit_0_indices) and bit_1_indices == sorted(bit_1_indices) + # Verify aggregate signature + assert bls_verify_multiple( + pubkeys=[ + bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in bit_0_indices]), + bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in bit_1_indices]), + ], + message_hashes=[ + hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b0)), + hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b1)), + ], + signature=indexed_attestation.signature, + domain=get_domain(state, DOMAIN_ATTESTATION, indexed_attestation.data.target_epoch), ) ``` @@ -1669,8 +1667,8 @@ def process_attester_slashing(state: BeaconState, attestation_1 = attester_slashing.attestation_1 attestation_2 = attester_slashing.attestation_2 assert is_slashable_attestation_data(attestation_1.data, attestation_2.data) - assert verify_indexed_attestation(state, attestation_1) - assert verify_indexed_attestation(state, attestation_2) + validate_indexed_attestation(state, attestation_1) + validate_indexed_attestation(state, attestation_2) slashed_any = False attesting_indices_1 = attestation_1.custody_bit_0_indices + attestation_1.custody_bit_1_indices @@ -1707,7 +1705,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: assert data.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] # Check signature and bitfields - assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) + validate_indexed_attestation(state, convert_to_indexed(state, attestation)) # Cache pending attestation pending_attestation = PendingAttestation( From 13d2ee696939e192534c0278eb271fe53510caf7 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 7 May 2019 11:49:45 -0600 Subject: [PATCH 43/91] cleanup validator guide for crosslinks --- specs/validator/0_beacon-chain-validator.md | 53 +++++++-------------- 1 file changed, 17 insertions(+), 36 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index a0173ebe9..a95052b86 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -37,14 +37,9 @@ - [Voluntary exits](#voluntary-exits) - [Attestations](#attestations-1) - [Attestation data](#attestation-data) - - [Slot](#slot-1) - - [Beacon block root](#beacon-block-root) - - [Source epoch](#source-epoch) - - [Source root](#source-root) - - [Target root](#target-root) - - [Shard](#shard) - - [Previous crosslink root](#previous-crosslink-root) - - [Crosslink data root](#crosslink-data-root) + - [LMD GHOST vote](#lmd-ghost-vote) + - [FFG vote](#ffg-vote) + - [Crosslink vote](#crosslink-vote) - [Construct attestation](#construct-attestation) - [Data](#data) - [Aggregation bitfield](#aggregation-bitfield) @@ -245,43 +240,29 @@ First the validator should construct `attestation_data`, an [`AttestationData`]( * Let `head_block` be the result of running the fork choice during the assigned slot. * Let `head_state` be the state of `head_block` processed through any empty slots up to the assigned slot. -##### Slot - -Set `attestation_data.slot = head_state.slot`. - -##### Beacon block root +##### LMD GHOST vote Set `attestation_data.beacon_block_root = signing_root(head_block)`. -##### Source epoch +##### FFG vote -Set `attestation_data.source_epoch = head_state.justified_epoch`. +* Set `attestation_data.source_epoch = head_state.justified_epoch`. +* Set `attestation_data.source_root = head_state.current_justified_root`. +* Set `attestation_data.target_epoch = get_current_epoch(head_state)` +* Set `attestation_data.target_root = signing_root(epoch_boundary_block)` where `epoch_boundary_block` is the block at the most recent epoch boundary. -##### Source root - -Set `attestation_data.source_root = head_state.current_justified_root`. - -##### Target root - -Set `attestation_data.target_root = signing_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary. - -*Note*: This can be looked up in the state using: +*Note*: `epoch_boundary_block` can be looked up in the state using: * Let `epoch_start_slot = get_epoch_start_slot(get_current_epoch(head_state))`. -* Set `epoch_boundary = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`. +* Let `epoch_boundary_block = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`. -##### Shard +##### Crosslink vote -Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +Construct `attestation_data.crosslink` via the following -##### Previous crosslink root - -Set `attestation_data.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. - -##### Crosslink data root - -Set `attestation_data.crosslink.data_root = ZERO_HASH`. - -*Note*: This is a stub for Phase 0. +* Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +* Set `attestation_data.crosslink.epoch = min(attestation_data.target_epoch, head_state.current_crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK)`. +* Set `attestation_data.crosslink.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. +* Set `attestation_data.crosslink.data_root = ZERO_HASH`. *Note*: This is a stub for Phase 0. #### Construct attestation From 513c44bd3d19c70da61359376dc440fc86bddc23 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 8 May 2019 08:38:14 -0600 Subject: [PATCH 44/91] add back in empty attestation test --- .../block_processing/test_process_attestation.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 165f0c84a..c986cc4c8 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -142,3 +142,14 @@ def test_non_empty_custody_bitfield(state): pre_state, post_state = run_attestation_processing(state, attestation, False) return pre_state, attestation, post_state + + +def test_empty_aggregation_bitfield(state): + attestation = get_valid_attestation(state) + state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY + + attestation.aggregation_bitfield = b'\x00' * len(attestation.aggregation_bitfield) + + pre_state, post_state = run_attestation_processing(state, attestation) + + return pre_state, attestation, post_state From c37157ead1a97fd3fb2fccdda4f0b67ea258a1ea Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Wed, 8 May 2019 19:15:23 +0100 Subject: [PATCH 45/91] Revert exception handling --- specs/core/0_beacon-chain.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3470459e1..bf210c60c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1216,19 +1216,15 @@ Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. ## Beacon chain state transition function -The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. +The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled excpetion (e.g. a failed `assert` or an out-of-range list access) are considered invalid. ```python def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: - pre_state = copy.deepcopy(state) - try: - # Process slots (including those with no blocks) since block - process_slots(state, block.slot) - # Process block - process_block(state, block) - except Exception: - # State is not advanced on exceptions (e.g. a failed `assert` or an out-of-range list access) - return pre_state + # Process slots (including those with no blocks) since block + process_slots(state, block.slot) + # Process block + process_block(state, block) + # Return post-state return state ``` From 8da4b8173e38bec3f46820beab731e173384c689 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 8 May 2019 14:49:53 -0600 Subject: [PATCH 46/91] remove unnecessary import of copy --- scripts/phase0/build_spec.py | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index ef7c055d2..f7587bad5 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -6,7 +6,6 @@ def build_phase0_spec(sourcefile, outfile): code_lines = [] code_lines.append(""" -import copy from typing import ( Any, Dict, From d7c1305ce4056e8778266fd83ced806aa5ef9ad4 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 9 May 2019 13:11:07 +0800 Subject: [PATCH 47/91] Add flake8 check max-line-length=120 --- .circleci/config.yml | 14 ++++++++++++++ Makefile | 7 +++++++ 2 files changed, 21 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 4f806b00f..fd7708f8d 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -84,11 +84,25 @@ jobs: command: make citest - store_test_results: path: test_libs/pyspec/test-reports + lint: + docker: + - image: circleci/python:3.6 + working_directory: ~/specs-repo + steps: + - restore_cache: + key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} + - restore_cached_venv: + venv_name: v1-pyspec + reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' + - run: + name: Run linter + command: make install_lint && make pyspec && make lint workflows: version: 2.1 test_spec: jobs: - checkout_specs + - lint - install_test: requires: - checkout_specs diff --git a/Makefile b/Makefile index 73d8adea8..8cc889f21 100644 --- a/Makefile +++ b/Makefile @@ -39,6 +39,13 @@ test: $(PY_SPEC_ALL_TARGETS) citest: $(PY_SPEC_ALL_TARGETS) cd $(PY_SPEC_DIR); mkdir -p test-reports/eth2spec; . venv/bin/activate; python -m pytest --junitxml=test-reports/eth2spec/test_results.xml . +install_lint: + cd $(PY_SPEC_DIR); python3 -m venv venv; . venv/bin/activate; pip3 install flake8==3.5.0 + +lint: + cd $(PY_SPEC_DIR); . venv/bin/activate; \ + flake8 --max-line-length=120 ./eth2spec; + # "make pyspec" to create the pyspec for all phases. pyspec: $(PY_SPEC_ALL_TARGETS) From ef41d7a0649de997939c65618cde5ad0401b015e Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 9 May 2019 13:45:00 +0800 Subject: [PATCH 48/91] No "import *" --- scripts/phase0/build_spec.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index e3c431a4a..7ec84bed0 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -12,8 +12,16 @@ from typing import ( NewType, Tuple, ) -from eth2spec.utils.minimal_ssz import * -from eth2spec.utils.bls_stub import * +from eth2spec.utils.minimal_ssz import ( + SSZType, + hash_tree_root, + signing_root, +) +from eth2spec.utils.bls_stub import ( + bls_aggregate_pubkeys, + bls_verify, + bls_verify_multiple, +) # stub, will get overwritten by real var SLOTS_PER_EPOCH = 64 From a4ba283d6770630a51b1524f8ba5bc696619e553 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 9 May 2019 13:54:38 +0800 Subject: [PATCH 49/91] Limit to 120 characters per line Limit to 120 characters per line --- specs/core/0_beacon-chain.md | 45 +++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a0c26bf59..1b3c1e9d1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -579,8 +579,10 @@ The types are defined topologically to aid in facilitating an executable version 'latest_block_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], 'latest_state_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], 'latest_active_index_roots': ['bytes32', LATEST_ACTIVE_INDEX_ROOTS_LENGTH], - 'latest_slashed_balances': ['uint64', LATEST_SLASHED_EXIT_LENGTH], # Balances slashed at every withdrawal period - 'latest_block_header': BeaconBlockHeader, # `latest_block_header.state_root == ZERO_HASH` temporarily + # Balances slashed at every withdrawal period + 'latest_slashed_balances': ['uint64', LATEST_SLASHED_EXIT_LENGTH], + # `latest_block_header.state_root == ZERO_HASH` temporarily + 'latest_block_header': BeaconBlockHeader, 'historical_roots': ['bytes32'], # Ethereum 1.0 chain data @@ -1149,7 +1151,9 @@ def initiate_validator_exit(state: BeaconState, index: ValidatorIndex) -> None: #### `slash_validator` ```python -def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whistleblower_index: ValidatorIndex=None) -> None: +def slash_validator(state: BeaconState, + slashed_index: ValidatorIndex, + whistleblower_index: ValidatorIndex=None) -> None: """ Slash the validator with index ``slashed_index``. """ @@ -1287,7 +1291,8 @@ def get_matching_head_attestations(state: BeaconState, epoch: Epoch) -> List[Pen ``` ```python -def get_unslashed_attesting_indices(state: BeaconState, attestations: List[PendingAttestation]) -> List[ValidatorIndex]: +def get_unslashed_attesting_indices(state: BeaconState, + attestations: List[PendingAttestation]) -> List[ValidatorIndex]: output = set() for a in attestations: output = output.union(get_attesting_indices(state, a.data, a.aggregation_bitfield)) @@ -1300,7 +1305,9 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat ``` ```python -def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: +def get_winning_crosslink_and_attesting_indices(state: BeaconState, + epoch: Epoch, + shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] crosslinks = list(filter( lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_root, hash_tree_root(c)), @@ -1332,12 +1339,16 @@ def process_justification_and_finalization(state: BeaconState) -> None: state.previous_justified_epoch = state.current_justified_epoch state.previous_justified_root = state.current_justified_root state.justification_bitfield = (state.justification_bitfield << 1) % 2**64 - previous_epoch_matching_target_balance = get_attesting_balance(state, get_matching_target_attestations(state, previous_epoch)) + previous_epoch_matching_target_balance = get_attesting_balance( + state, get_matching_target_attestations(state, previous_epoch) + ) if previous_epoch_matching_target_balance * 3 >= get_total_active_balance(state) * 2: state.current_justified_epoch = previous_epoch state.current_justified_root = get_block_root(state, state.current_justified_epoch) state.justification_bitfield |= (1 << 1) - current_epoch_matching_target_balance = get_attesting_balance(state, get_matching_target_attestations(state, current_epoch)) + current_epoch_matching_target_balance = get_attesting_balance( + state, get_matching_target_attestations(state, current_epoch) + ) if current_epoch_matching_target_balance * 3 >= get_total_active_balance(state) * 2: state.current_justified_epoch = current_epoch state.current_justified_root = get_block_root(state, state.current_justified_epoch) @@ -1431,7 +1442,9 @@ def get_attestation_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: for index in eligible_validator_indices: penalties[index] += BASE_REWARDS_PER_EPOCH * get_base_reward(state, index) if index not in matching_target_attesting_indices: - penalties[index] += state.validator_registry[index].effective_balance * finality_delay // INACTIVITY_PENALTY_QUOTIENT + penalties[index] += ( + state.validator_registry[index].effective_balance * finality_delay // INACTIVITY_PENALTY_QUOTIENT + ) return rewards, penalties ``` @@ -1478,7 +1491,10 @@ Run the following function: def process_registry_updates(state: BeaconState) -> None: # Process activation eligibility and ejections for index, validator in enumerate(state.validator_registry): - if validator.activation_eligibility_epoch == FAR_FUTURE_EPOCH and validator.effective_balance >= MAX_EFFECTIVE_BALANCE: + if ( + validator.activation_eligibility_epoch == FAR_FUTURE_EPOCH and + validator.effective_balance >= MAX_EFFECTIVE_BALANCE + ): validator.activation_eligibility_epoch = get_current_epoch(state) if is_active_validator(validator, get_current_epoch(state)) and validator.effective_balance <= EJECTION_BALANCE: @@ -1603,7 +1619,12 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: def process_randao(state: BeaconState, block: BeaconBlock) -> None: proposer = state.validator_registry[get_beacon_proposer_index(state)] # Verify that the provided randao value is valid - assert bls_verify(proposer.pubkey, hash_tree_root(get_current_epoch(state)), block.body.randao_reveal, get_domain(state, DOMAIN_RANDAO)) + assert bls_verify( + proposer.pubkey, + hash_tree_root(get_current_epoch(state)), + block.body.randao_reveal, + get_domain(state, DOMAIN_RANDAO), + ) # Mix it in state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = ( xor(get_randao_mix(state, get_current_epoch(state)), @@ -1748,7 +1769,9 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: validator_pubkeys = [v.pubkey for v in state.validator_registry] if pubkey not in validator_pubkeys: # Verify the deposit signature (proof of possession) - if not bls_verify(pubkey, signing_root(deposit.data), deposit.data.signature, get_domain(state, DOMAIN_DEPOSIT)): + if not bls_verify( + pubkey, signing_root(deposit.data), deposit.data.signature, get_domain(state, DOMAIN_DEPOSIT) + ): return # Add validator and balance entries From 84472a5a6e06324e457f3776e43f1192f005a418 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 9 May 2019 14:25:08 +0800 Subject: [PATCH 50/91] fix pyspec lint --- scripts/phase0/build_spec.py | 3 +++ scripts/phase0/function_puller.py | 10 +++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index 7ec84bed0..c8cd7348b 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -22,6 +22,8 @@ from eth2spec.utils.bls_stub import ( bls_verify, bls_verify_multiple, ) +from eth2spec.utils.hash_function import hash + # stub, will get overwritten by real var SLOTS_PER_EPOCH = 64 @@ -69,6 +71,7 @@ def hash(x): hash_cache[x] = ret return ret + # Access to overwrite spec constants based on configuration def apply_constants_preset(preset: Dict[str, Any]): global_vars = globals() diff --git a/scripts/phase0/function_puller.py b/scripts/phase0/function_puller.py index 1fad41fa9..750f19590 100644 --- a/scripts/phase0/function_puller.py +++ b/scripts/phase0/function_puller.py @@ -55,15 +55,19 @@ def get_spec(file_name: str) -> List[str]: if eligible: code_lines.append(row[0] + ' = ' + (row[1].replace('**TBD**', '0x1234567890123456789012345678901234567890'))) # Build type-def re-initialization - code_lines.append('') + code_lines.append('\n') code_lines.append('def init_SSZ_types():') code_lines.append(' global_vars = globals()') for ssz_type_name, ssz_type in type_defs: code_lines.append('') for type_line in ssz_type: - code_lines.append(' ' + type_line) + if len(type_line) > 0: + code_lines.append(' ' + type_line) code_lines.append('\n') - code_lines.append('ssz_types = [' + ', '.join([f'\'{ssz_type_name}\'' for (ssz_type_name, _) in type_defs]) + ']') + code_lines.append('ssz_types = [\n') + for (ssz_type_name, _) in type_defs: + code_lines.append(f' {ssz_type_name},\n') + code_lines.append(']') code_lines.append('\n') code_lines.append('def get_ssz_type_by_name(name: str) -> SSZType:') code_lines.append(' return globals()[name]') From e1343f99bbff2054a89e06a3b6d904ec7c3519d7 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 9 May 2019 14:29:21 +0800 Subject: [PATCH 51/91] Fix pyspec/eth2spec/debug lint --- test_libs/pyspec/eth2spec/debug/decode.py | 4 ++-- test_libs/pyspec/eth2spec/debug/encode.py | 1 - .../pyspec/eth2spec/debug/random_value.py | 23 +++++++++++++++---- 3 files changed, 21 insertions(+), 7 deletions(-) diff --git a/test_libs/pyspec/eth2spec/debug/decode.py b/test_libs/pyspec/eth2spec/debug/decode.py index aeac3924d..e9aa8bc2b 100644 --- a/test_libs/pyspec/eth2spec/debug/decode.py +++ b/test_libs/pyspec/eth2spec/debug/decode.py @@ -16,11 +16,11 @@ def decode(json, typ): for field, subtype in typ.fields.items(): temp[field] = decode(json[field], subtype) if field + "_hash_tree_root" in json: - assert(json[field + "_hash_tree_root"][2:] == + assert(json[field + "_hash_tree_root"][2:] == hash_tree_root(temp[field], subtype).hex()) ret = typ(**temp) if "hash_tree_root" in json: - assert(json["hash_tree_root"][2:] == + assert(json["hash_tree_root"][2:] == hash_tree_root(ret, typ).hex()) return ret else: diff --git a/test_libs/pyspec/eth2spec/debug/encode.py b/test_libs/pyspec/eth2spec/debug/encode.py index d3513e638..b38e5fe98 100644 --- a/test_libs/pyspec/eth2spec/debug/encode.py +++ b/test_libs/pyspec/eth2spec/debug/encode.py @@ -25,4 +25,3 @@ def encode(value, typ, include_hash_tree_roots=False): else: print(value, typ) raise Exception("Type not recognized") - diff --git a/test_libs/pyspec/eth2spec/debug/random_value.py b/test_libs/pyspec/eth2spec/debug/random_value.py index a853d2328..f28181943 100644 --- a/test_libs/pyspec/eth2spec/debug/random_value.py +++ b/test_libs/pyspec/eth2spec/debug/random_value.py @@ -31,7 +31,12 @@ class RandomizationMode(Enum): return self.value in [0, 4, 5] -def get_random_ssz_object(rng: Random, typ: Any, max_bytes_length: int, max_list_length: int, mode: RandomizationMode, chaos: bool) -> Any: +def get_random_ssz_object(rng: Random, + typ: Any, + max_bytes_length: int, + max_list_length: int, + mode: RandomizationMode, + chaos: bool) -> Any: """ Create an object for a given type, filled with random data. :param rng: The random number generator to use. @@ -77,7 +82,10 @@ def get_random_ssz_object(rng: Random, typ: Any, max_bytes_length: int, max_list return get_random_basic_value(rng, typ) # Vector: elif isinstance(typ, list) and len(typ) == 2: - return [get_random_ssz_object(rng, typ[0], max_bytes_length, max_list_length, mode, chaos) for _ in range(typ[1])] + return [ + get_random_ssz_object(rng, typ[0], max_bytes_length, max_list_length, mode, chaos) + for _ in range(typ[1]) + ] # List: elif isinstance(typ, list) and len(typ) == 1: length = rng.randint(0, max_list_length) @@ -85,10 +93,17 @@ def get_random_ssz_object(rng: Random, typ: Any, max_bytes_length: int, max_list length = 1 if mode == RandomizationMode.mode_max_count: length = max_list_length - return [get_random_ssz_object(rng, typ[0], max_bytes_length, max_list_length, mode, chaos) for _ in range(length)] + return [ + get_random_ssz_object(rng, typ[0], max_bytes_length, max_list_length, mode, chaos) + for _ in range(length) + ] # Container: elif hasattr(typ, 'fields'): - return typ(**{field: get_random_ssz_object(rng, subtype, max_bytes_length, max_list_length, mode, chaos) for field, subtype in typ.fields.items()}) + return typ(**{ + field: + get_random_ssz_object(rng, subtype, max_bytes_length, max_list_length, mode, chaos) + for field, subtype in typ.fields.items() + }) else: print(typ) raise Exception("Type not recognized") From 4db4d879301b130eba3667d8f455de8eb07e7c80 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 9 May 2019 14:57:36 +0800 Subject: [PATCH 52/91] Refactor `process_final_updates` --- specs/core/1_custody-game.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index 1b6b1d2e4..e03e54ed0 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -683,13 +683,12 @@ Append this to `process_final_updates(state)`: # Clean up exposed RANDAO key reveals state.exposed_derived_secrets[current_epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS] = [] # Reset withdrawable epochs if challenge records are empty + records = state.custody_chunk_challenge_records + state.bit_challenge_records + validator_indices_in_records = set( + [record.challenger_index for record in records] + [record.responder_index for record in records] + ) for index, validator in enumerate(state.validator_registry): - eligible = True - for records in (state.custody_chunk_challenge_records, state.bit_challenge_records): - for filter_func in (lambda rec: rec.challenger_index == index, lambda rec: rec.responder_index == index): - if len(list(filter(filter_func, records))) > 0: - eligible = False - if eligible: + if index not in validator_indices_in_records: if validator.exit_epoch == FAR_FUTURE_EPOCH: validator.withdrawable_epoch = FAR_FUTURE_EPOCH else: From a552b94d46d744e5ec005d09da1f0b4e9b49c81e Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 10 May 2019 11:02:39 -0700 Subject: [PATCH 53/91] Update 0_beacon-chain-validator.md typo fix --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 35de666d7..49290b432 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -246,7 +246,7 @@ Set `attestation_data.beacon_block_root = signing_root(head_block)`. ##### FFG vote -* Set `attestation_data.source_epoch = head_state.justified_epoch`. +* Set `attestation_data.source_epoch = head_state.current_justified_epoch`. * Set `attestation_data.source_root = head_state.current_justified_root`. * Set `attestation_data.target_epoch = get_current_epoch(head_state)` * Set `attestation_data.target_root = signing_root(epoch_boundary_block)` where `epoch_boundary_block` is the block at the most recent epoch boundary. From c0d4334ced5216b002b65bd682a1f5ab8f65d43b Mon Sep 17 00:00:00 2001 From: NIC619 Date: Sat, 11 May 2019 15:12:11 +0800 Subject: [PATCH 54/91] Fix SSZ union type example --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b4ea8135a..8633c7ed1 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -47,7 +47,7 @@ * **list**: ordered variable-length homogeneous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` * **union**: union type containing one of the given subtypes - * round bracket notation `(type_1, type_2, ...)`, e.g. `("uint64", "null")` + * round bracket notation `(type_1, type_2, ...)`, e.g. `("null", "uint64")` ### Variable-size and fixed-size From ae19521ea8e0e66ff9f0d1592d28bec1799d9d61 Mon Sep 17 00:00:00 2001 From: Ryuya Nakamura Date: Sat, 11 May 2019 20:41:05 +0900 Subject: [PATCH 55/91] Update the docstring of initiate_validator_exit (#1072) --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a0c26bf59..103870ce2 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1127,7 +1127,7 @@ def get_churn_limit(state: BeaconState) -> int: ```python def initiate_validator_exit(state: BeaconState, index: ValidatorIndex) -> None: """ - Initiate the validator of the given ``index``. + Initiate the exit of the validator of the given ``index``. """ # Return if validator already initiated exit validator = state.validator_registry[index] From 36514d1c384290870e726c306e57d298be22a40b Mon Sep 17 00:00:00 2001 From: Jonny Rhea Date: Sat, 11 May 2019 16:49:46 -0500 Subject: [PATCH 56/91] Update 0_deposit-contract.md get_genesis_beacon_state is not defined here. i modified the spec to link to the function. --- specs/core/0_deposit-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_deposit-contract.md b/specs/core/0_deposit-contract.md index 6843e407e..68ba02756 100644 --- a/specs/core/0_deposit-contract.md +++ b/specs/core/0_deposit-contract.md @@ -72,7 +72,7 @@ Every Ethereum 1.0 deposit, of size at least `MIN_DEPOSIT_AMOUNT`, emits a `Depo ### `Eth2Genesis` log -When `CHAIN_START_FULL_DEPOSIT_THRESHOLD` of full deposits have been made, the deposit contract emits the `Eth2Genesis` log. The beacon chain state may then be initialized by calling the `get_genesis_beacon_state` function (defined below) where: +When `CHAIN_START_FULL_DEPOSIT_THRESHOLD` of full deposits have been made, the deposit contract emits the `Eth2Genesis` log. The beacon chain state may then be initialized by calling the `get_genesis_beacon_state` function (defined [here](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/core/0_beacon-chain.md#genesis-state)) where: * `genesis_time` equals `time` in the `Eth2Genesis` log * `latest_eth1_data.deposit_root` equals `deposit_root` in the `Eth2Genesis` log From e85847719f9b4b9d4ca7ef4a719cd823b09e51e2 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Sun, 12 May 2019 08:50:46 -0600 Subject: [PATCH 57/91] use relative path for beacon chain link --- specs/core/0_deposit-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_deposit-contract.md b/specs/core/0_deposit-contract.md index 68ba02756..f0259a179 100644 --- a/specs/core/0_deposit-contract.md +++ b/specs/core/0_deposit-contract.md @@ -72,7 +72,7 @@ Every Ethereum 1.0 deposit, of size at least `MIN_DEPOSIT_AMOUNT`, emits a `Depo ### `Eth2Genesis` log -When `CHAIN_START_FULL_DEPOSIT_THRESHOLD` of full deposits have been made, the deposit contract emits the `Eth2Genesis` log. The beacon chain state may then be initialized by calling the `get_genesis_beacon_state` function (defined [here](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/core/0_beacon-chain.md#genesis-state)) where: +When `CHAIN_START_FULL_DEPOSIT_THRESHOLD` of full deposits have been made, the deposit contract emits the `Eth2Genesis` log. The beacon chain state may then be initialized by calling the `get_genesis_beacon_state` function (defined [here](./0_beacon-chain.md#genesis-state)) where: * `genesis_time` equals `time` in the `Eth2Genesis` log * `latest_eth1_data.deposit_root` equals `deposit_root` in the `Eth2Genesis` log From ba1949b2bc59be426bca8db1f0cd5a9eb0c2c376 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sun, 12 May 2019 13:46:17 -0700 Subject: [PATCH 58/91] Update 1_custody-game.md --- specs/core/1_custody-game.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index a030c577b..429e8e035 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -104,6 +104,8 @@ This document details the beacon chain additions and changes in Phase 1 of Ether ### Reward and penalty quotients +| Name | Value | +| - | - | | `EARLY_DERIVED_SECRET_REVEAL_SLOT_REWARD_MULTIPLE` | `2**1` (= 2) | ### Signature domains From 7bb85a69ed5759bbf64a73bac111865fc53b2412 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 13 May 2019 16:34:30 -0400 Subject: [PATCH 59/91] add process_slots usage to validator guide --- specs/validator/0_beacon-chain-validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 49290b432..ef6e8bf25 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -238,7 +238,7 @@ A validator should create and broadcast the attestation halfway through the `slo First the validator should construct `attestation_data`, an [`AttestationData`](../core/0_beacon-chain.md#attestationdata) object based upon the state at the assigned slot. * Let `head_block` be the result of running the fork choice during the assigned slot. -* Let `head_state` be the state of `head_block` processed through any empty slots up to the assigned slot. +* Let `head_state` be the state of `head_block` processed through any empty slots up to the assigned slot using `process_slots(state, slot)`. ##### LMD GHOST vote @@ -360,7 +360,7 @@ def is_proposer_at_slot(state: BeaconState, return get_beacon_proposer_index(state) == validator_index ``` -*Note*: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot. +*Note*: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot using `process_slots(state, current_slot)`. ### Lookahead From a0a2aa90de166d840ba69dcc487079c7b26571f7 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 13 May 2019 16:40:45 -0400 Subject: [PATCH 60/91] lint --- specs/core/0_beacon-chain.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0d0e37679..524a40b17 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1253,7 +1253,6 @@ def process_slot(state: BeaconState) -> None: # Cache block root previous_block_root = signing_root(state.latest_block_header) state.latest_block_roots[state.slot % SLOTS_PER_HISTORICAL_ROOT] = previous_block_root - ``` ### Epoch processing @@ -1610,7 +1609,12 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: def process_randao(state: BeaconState, body: BeaconBlockBody) -> None: proposer = state.validator_registry[get_beacon_proposer_index(state)] # Verify that the provided randao value is valid - assert bls_verify(proposer.pubkey, hash_tree_root(get_current_epoch(state)), body.randao_reveal, get_domain(state, DOMAIN_RANDAO)) + assert bls_verify( + proposer.pubkey, + hash_tree_root(get_current_epoch(state)), + body.randao_reveal, + get_domain(state, DOMAIN_RANDAO), + ) # Mix it in state.latest_randao_mixes[get_current_epoch(state) % LATEST_RANDAO_MIXES_LENGTH] = ( xor(get_randao_mix(state, get_current_epoch(state)), From 5ba90d68e194310816f22f36caa8a7cc3d95cf52 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 13 May 2019 16:53:28 -0400 Subject: [PATCH 61/91] add flag for validate state root --- specs/core/0_beacon-chain.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 524a40b17..a9c4f8811 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1220,11 +1220,14 @@ Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. The post-state corresponding to a pre-state `state` and a block `block` is defined as `state_transition(state, block)`. State transitions that trigger an unhandled excpetion (e.g. a failed `assert` or an out-of-range list access) are considered invalid. ```python -def state_transition(state: BeaconState, block: BeaconBlock) -> BeaconState: +def state_transition(state: BeaconState, block: BeaconBlock, validate_state_root: bool=False) -> BeaconState: # Process slots (including those with no blocks) since block process_slots(state, block.slot) # Process block process_block(state, block) + # Validate state root if received from external source + if validate_state_root: + process_state_root(state, block) # Return post-state return state ``` @@ -1579,7 +1582,6 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) - # process_state_root(state, block) ``` #### Block header From c2942c00c6555cd03aa0595b9ee947e5eb06e6f1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 13 May 2019 17:02:20 -0400 Subject: [PATCH 62/91] lint requires install_test in ci --- .circleci/config.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index fd7708f8d..2bfda20ff 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -102,10 +102,12 @@ workflows: test_spec: jobs: - checkout_specs - - lint - install_test: requires: - checkout_specs - test: requires: - install_test + - lint: + requires: + - install_test From f830f83a1d6ae1b6728bfd357da6cbef0d4a5107 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 14 May 2019 10:32:20 +0800 Subject: [PATCH 63/91] Update config.yml --- .circleci/config.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 2bfda20ff..439afc9a9 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -102,12 +102,12 @@ workflows: test_spec: jobs: - checkout_specs + - lint: + requires: + - checkout_specs - install_test: requires: - checkout_specs - test: requires: - install_test - - lint: - requires: - - install_test From c60635d2c9c829865189b4856ef4748cff7a3d9d Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 14 May 2019 06:15:03 +0100 Subject: [PATCH 64/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a9c4f8811..ee37d9217 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -117,7 +117,6 @@ - [Deposits](#deposits) - [Voluntary exits](#voluntary-exits) - [Transfers](#transfers) - - [State root verification](#state-root-verification) @@ -373,6 +372,7 @@ The types are defined topologically to aid in facilitating an executable version 'signature': 'bytes96', } ``` + #### `Validator` ```python @@ -577,9 +577,7 @@ The types are defined topologically to aid in facilitating an executable version 'latest_block_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], 'latest_state_roots': ['bytes32', SLOTS_PER_HISTORICAL_ROOT], 'latest_active_index_roots': ['bytes32', LATEST_ACTIVE_INDEX_ROOTS_LENGTH], - # Balances slashed at every withdrawal period 'latest_slashed_balances': ['uint64', LATEST_SLASHED_EXIT_LENGTH], - # `latest_block_header.state_root == ZERO_HASH` temporarily 'latest_block_header': BeaconBlockHeader, 'historical_roots': ['bytes32'], @@ -1225,9 +1223,9 @@ def state_transition(state: BeaconState, block: BeaconBlock, validate_state_root process_slots(state, block.slot) # Process block process_block(state, block) - # Validate state root if received from external source + # Validate state root (`validate_state_root == True` in production) if validate_state_root: - process_state_root(state, block) + assert block.state_root == hash_tree_root(state) # Return post-state return state ``` @@ -1838,10 +1836,3 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: assert not (0 < state.balances[transfer.sender] < MIN_DEPOSIT_AMOUNT) assert not (0 < state.balances[transfer.recipient] < MIN_DEPOSIT_AMOUNT) ``` - -#### State root verification - -```python -def process_state_root(state: BeaconState, block: BeaconBlock) -> None: - assert block.state_root == hash_tree_root(state) -``` From 5e7b173b22e0d550092ac967f1dc84d2c794e935 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 14 May 2019 17:29:11 -0400 Subject: [PATCH 65/91] fix up validator guide crosslink committee instructions --- specs/validator/0_beacon-chain-validator.md | 119 +++++++++----------- 1 file changed, 53 insertions(+), 66 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index ef6e8bf25..67a09c797 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -20,6 +20,8 @@ - [Process deposit](#process-deposit) - [Validator index](#validator-index) - [Activation](#activation) + - [Validator assignments](#validator-assignments) + - [Lookahead](#lookahead) - [Beacon chain responsibilities](#beacon-chain-responsibilities) - [Block proposal](#block-proposal) - [Block header](#block-header) @@ -45,8 +47,6 @@ - [Aggregation bitfield](#aggregation-bitfield) - [Custody bitfield](#custody-bitfield) - [Aggregate signature](#aggregate-signature) - - [Validator assignments](#validator-assignments) - - [Lookahead](#lookahead) - [How to avoid slashing](#how-to-avoid-slashing) - [Proposer slashing](#proposer-slashing) - [Attester slashing](#attester-slashing) @@ -127,13 +127,61 @@ Once a validator is activated, the validator is assigned [responsibilities](#bea *Note*: There is a maximum validator churn per finalized epoch so the delay until activation is variable depending upon finality, total active validator balance, and the number of validators in the queue to be activated. +## Validator assignments + +A validator can get committee assignments for a given epoch using the following helper via `get_committee_assignment(state, epoch, validator_index)` where `epoch <= next_epoch`. + +```python +def get_committee_assignment( + state: BeaconState, + epoch: Epoch, + validator_index: ValidatorIndex) -> Tuple[List[ValidatorIndex], Shard, Slot]: + """ + Return the committee assignment in the ``epoch`` for ``validator_index``. + ``assignment`` returned is a tuple of the following form: + * ``assignment[0]`` is the list of validators in the committee + * ``assignment[1]`` is the shard to which the committee is assigned + * ``assignment[2]`` is the slot at which the committee is assigned + """ + next_epoch = get_current_epoch(state) + 1 + assert epoch <= next_epoch + + committees_per_slot = get_epoch_committee_count(state, epoch) // SLOTS_PER_EPOCH + epoch_start_slot = get_epoch_start_slot(epoch) + for slot in range(epoch_start_slot, epoch_start_slot + SLOTS_PER_EPOCH) + offset = committees_per_slot * (slot % SLOTS_PER_EPOCH) + for i in range(committees_per_slot): + shard = (get_epoch_start_shard(state, epoch) + i) % SHARD_COUNT + committee = get_crosslink_committee(state, epoch, shard) + if validator_index in committee: + return committee, shard, slot +``` + +A validator can use the following function to see if they are supposed to propose during their assigned committee slot. This function can only be run during the slot in question. Proposer selection is only stable within the context of the current epoch. + +```python +def is_proposer(state: BeaconState, + validator_index: ValidatorIndex) -> bool: + return get_beacon_proposer_index(state) == validator_index +``` + +*Note*: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot using `process_slots(state, current_slot)`. + +### Lookahead + +The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must checked during the slot in question. + +`get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in Phase 1+). + +Specifically, a validator should call `get_committee_assignment(state, next_epoch, validator_index)` when checking for next epoch assignments. + ## Beacon chain responsibilities A validator has two primary responsibilities to the beacon chain: [proposing blocks](#block-proposal) and [creating attestations](#attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. ### Block proposal -A validator is expected to propose a [`BeaconBlock`](../core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `get_beacon_proposer_index(state)` returns the validator's `validator_index`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot - 1`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](../core/0_beacon-chain.md#beacon-chain-state-transition-function). +A validator is expected to propose a [`BeaconBlock`](../core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `is_proposer(state, validator_index)` returns `True`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot - 1`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](../core/0_beacon-chain.md#beacon-chain-state-transition-function). There is one proposer per slot, so if there are N active validators any individual validator will on average be assigned to propose once per N slots (e.g. at 312500 validators = 10 million ETH, that's once per ~3 weeks). @@ -229,7 +277,7 @@ Up to `MAX_VOLUNTARY_EXITS` [`VoluntaryExit`](../core/0_beacon-chain.md#voluntar ### Attestations -A validator is expected to create, sign, and broadcast an attestation during each epoch. The slot during which the validator performs this role is any slot at which `get_crosslink_committees_at_slot(state, slot)` contains a committee that contains `validator_index`. +A validator is expected to create, sign, and broadcast an attestation during each epoch. The committee, assigned shard, and assigned slot for which the validator performs this role during an epoch is defined by `get_committee_assignment(state, epoch, validator_index)`. A validator should create and broadcast the attestation halfway through the `slot` during which the validator is assigned ― that is, `SECONDS_PER_SLOT * 0.5` seconds after the start of `slot`. @@ -259,7 +307,7 @@ Set `attestation_data.beacon_block_root = signing_root(head_block)`. Construct `attestation_data.crosslink` via the following -* Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +* Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee. * Set `attestation_data.crosslink.epoch = min(attestation_data.target_epoch, head_state.current_crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK)`. * Set `attestation_data.crosslink.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. * Set `attestation_data.crosslink.data_root = ZERO_HASH`. *Note*: This is a stub for Phase 0. @@ -310,67 +358,6 @@ signed_attestation_data = bls_sign( ) ``` -## Validator assignments - -A validator can get the current, previous, and next epoch committee assignments using the following helper via `get_committee_assignment(state, epoch, validator_index)` where `previous_epoch <= epoch <= next_epoch`. - -```python -def get_committee_assignment( - state: BeaconState, - epoch: Epoch, - validator_index: ValidatorIndex) -> Tuple[List[ValidatorIndex], Shard, Slot]: - """ - Return the committee assignment in the ``epoch`` for ``validator_index``. - ``assignment`` returned is a tuple of the following form: - * ``assignment[0]`` is the list of validators in the committee - * ``assignment[1]`` is the shard to which the committee is assigned - * ``assignment[2]`` is the slot at which the committee is assigned - """ - previous_epoch = get_previous_epoch(state) - next_epoch = get_current_epoch(state) + 1 - assert previous_epoch <= epoch <= next_epoch - - epoch_start_slot = get_epoch_start_slot(epoch) - for slot in range(epoch_start_slot, epoch_start_slot + SLOTS_PER_EPOCH): - crosslink_committees = get_crosslink_committees_at_slot( - state, - slot, - ) - selected_committees = [ - committee # Tuple[List[ValidatorIndex], Shard] - for committee in crosslink_committees - if validator_index in committee[0] - ] - if len(selected_committees) > 0: - validators = selected_committees[0][0] - shard = selected_committees[0][1] - - assignment = (validators, shard, slot) - return assignment -``` - -A validator can use the following function to see if they are supposed to propose during their assigned committee slot. This function can only be run during the slot in question. Proposer selection is only stable within the context of the current epoch. - -```python -def is_proposer_at_slot(state: BeaconState, - slot: Slot, - validator_index: ValidatorIndex) -> bool: - assert state.slot == slot - - return get_beacon_proposer_index(state) == validator_index -``` - -*Note*: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot using `process_slots(state, current_slot)`. - - -### Lookahead - -The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must checked during the slot in question. - -`get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in Phase 1+). - -Specifically, a validator should call `get_committee_assignment(state, next_epoch, validator_index)` when checking for next epoch assignments. - ## How to avoid slashing "Slashing" is the burning of some amount of validator funds and immediate ejection from the active validator set. In Phase 0, there are two ways in which funds can be slashed -- [proposer slashing](#proposer-slashing) and [attester slashing](#attester-slashing). Although being slashed has serious repercussions, it is simple enough to avoid being slashed all together by remaining _consistent_ with respect to the messages a validator has previously signed. From 01efe52eb0d7523994b96ccb6e44b85929b0ad42 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 14 May 2019 17:32:44 -0400 Subject: [PATCH 66/91] fix start shard --- specs/validator/0_beacon-chain-validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 67a09c797..3b498bfd4 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -149,9 +149,9 @@ def get_committee_assignment( committees_per_slot = get_epoch_committee_count(state, epoch) // SLOTS_PER_EPOCH epoch_start_slot = get_epoch_start_slot(epoch) for slot in range(epoch_start_slot, epoch_start_slot + SLOTS_PER_EPOCH) - offset = committees_per_slot * (slot % SLOTS_PER_EPOCH) + slot_start_shard = get_epoch_start_shard(state, epoch) + committees_per_slot * (slot % SLOTS_PER_EPOCH) for i in range(committees_per_slot): - shard = (get_epoch_start_shard(state, epoch) + i) % SHARD_COUNT + shard = (slot_start_shard + i) % SHARD_COUNT committee = get_crosslink_committee(state, epoch, shard) if validator_index in committee: return committee, shard, slot From bc95906e4f706460f9e431178da45ae8d1244b41 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bruno=20=C5=A0kvorc?= Date: Wed, 15 May 2019 10:40:42 +0200 Subject: [PATCH 67/91] Typo fix --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index ef6e8bf25..ae0e02a92 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -365,7 +365,7 @@ def is_proposer_at_slot(state: BeaconState, ### Lookahead -The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must checked during the slot in question. +The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must be checked during the slot in question. `get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in Phase 1+). From 2885f853c28d2442b4507b80d24471292a41ddb2 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 May 2019 16:34:07 +0800 Subject: [PATCH 68/91] clean up lint --- .circleci/config.yml | 2 +- Makefile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 439afc9a9..3fe6643b7 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -96,7 +96,7 @@ jobs: reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Run linter - command: make install_lint && make pyspec && make lint + command: make install_lint && make lint workflows: version: 2.1 test_spec: diff --git a/Makefile b/Makefile index 8cc889f21..a6b379b71 100644 --- a/Makefile +++ b/Makefile @@ -42,7 +42,7 @@ citest: $(PY_SPEC_ALL_TARGETS) install_lint: cd $(PY_SPEC_DIR); python3 -m venv venv; . venv/bin/activate; pip3 install flake8==3.5.0 -lint: +lint: $(PY_SPEC_ALL_TARGETS) cd $(PY_SPEC_DIR); . venv/bin/activate; \ flake8 --max-line-length=120 ./eth2spec; From b0747703a77c36d821b6bf049aa8abc8e1603603 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 May 2019 17:23:26 +0800 Subject: [PATCH 69/91] Make CI job `lint` require `test` --- .circleci/config.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 3fe6643b7..d2b284e25 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -52,7 +52,7 @@ jobs: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} paths: - ~/specs-repo - install_test: + install_env: docker: - image: circleci/python:3.6 working_directory: ~/specs-repo @@ -64,7 +64,7 @@ jobs: reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Install pyspec requirements - command: make install_test + command: make install_test && make install_lint - save_cached_venv: venv_name: v1-pyspec reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' @@ -96,18 +96,18 @@ jobs: reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Run linter - command: make install_lint && make lint + command: make lint workflows: version: 2.1 test_spec: jobs: - checkout_specs - - lint: - requires: - - checkout_specs - - install_test: + - install_env: requires: - checkout_specs - test: requires: - - install_test + - install_env + - lint: + requires: + - test From 1b3dfa67813e435761a71a3cc1c5e552efc043dd Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 May 2019 17:34:49 +0800 Subject: [PATCH 70/91] kick the cache --- .circleci/config.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index d2b284e25..b9f09d9cc 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -60,13 +60,13 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec + venv_name: v1-pyspec-02 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Install pyspec requirements command: make install_test && make install_lint - save_cached_venv: - venv_name: v1-pyspec + venv_name: v1-pyspec-02 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' venv_path: ./test_libs/pyspec/venv test: @@ -77,7 +77,7 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec + venv_name: v1-pyspec-02 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Run py-tests @@ -92,7 +92,7 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec + venv_name: v1-pyspec-02 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Run linter From 32f76641e34afb703355ab8b8ac93fafa0d59478 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 May 2019 17:41:48 +0800 Subject: [PATCH 71/91] Add build_pyspec job --- .circleci/config.yml | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index b9f09d9cc..a0700a004 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -69,6 +69,19 @@ jobs: venv_name: v1-pyspec-02 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' venv_path: ./test_libs/pyspec/venv + build_pyspec: + docker: + - image: circleci/python:3.6 + working_directory: ~/specs-repo + steps: + - restore_cache: + key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} + - restore_cached_venv: + venv_name: v1-pyspec-02 + reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' + - run: + name: Build pyspec + command: make pyspec test: docker: - image: circleci/python:3.6 @@ -105,9 +118,13 @@ workflows: - install_env: requires: - checkout_specs + - build_pyspec: + requires: + - checkout_specs - test: requires: - install_env + - build_pyspec - lint: requires: - - test + - build_pyspec From ee4fdf5d06c909b4d7bc6f1dff3bce5a253b54e7 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 May 2019 17:45:09 +0800 Subject: [PATCH 72/91] Kick the cache again --- .circleci/config.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index a0700a004..972f0e351 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -60,13 +60,13 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec-02 + venv_name: v1-pyspec-03 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Install pyspec requirements command: make install_test && make install_lint - save_cached_venv: - venv_name: v1-pyspec-02 + venv_name: v1-pyspec-03 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' venv_path: ./test_libs/pyspec/venv build_pyspec: @@ -77,7 +77,7 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec-02 + venv_name: v1-pyspec-03 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Build pyspec @@ -90,7 +90,7 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec-02 + venv_name: v1-pyspec-03 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Run py-tests @@ -105,7 +105,7 @@ jobs: - restore_cache: key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_cached_venv: - venv_name: v1-pyspec-02 + venv_name: v1-pyspec-03 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - run: name: Run linter From be4c792fc301b7ee254b61466720e40e155b1601 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 16 May 2019 18:20:20 +0800 Subject: [PATCH 73/91] Change it back to serial workflow --- .circleci/config.yml | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 972f0e351..f3c5f6a81 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -69,19 +69,6 @@ jobs: venv_name: v1-pyspec-03 reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' venv_path: ./test_libs/pyspec/venv - build_pyspec: - docker: - - image: circleci/python:3.6 - working_directory: ~/specs-repo - steps: - - restore_cache: - key: v1-specs-repo-{{ .Branch }}-{{ .Revision }} - - restore_cached_venv: - venv_name: v1-pyspec-03 - reqs_checksum: '{{ checksum "test_libs/pyspec/requirements.txt" }}-{{ checksum "test_libs/pyspec/requirements-testing.txt" }}' - - run: - name: Build pyspec - command: make pyspec test: docker: - image: circleci/python:3.6 @@ -118,13 +105,9 @@ workflows: - install_env: requires: - checkout_specs - - build_pyspec: - requires: - - checkout_specs - test: requires: - install_env - - build_pyspec - lint: requires: - - build_pyspec + - test From 7f6896cca3931c0e4a6b352bfd0ba3fa3dca3212 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Thu, 16 May 2019 16:43:10 -0700 Subject: [PATCH 74/91] Update to use union type --- specs/networking/rpc-interface.md | 34 ++++++++++--------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index ca6008a40..48ee9333b 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -39,36 +39,24 @@ To facilitate RPC-over-`libp2p`, a single protocol name is used: `/eth/serenity/ Remote method calls are wrapped in a "request" structure: ``` - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| | -+ id (uint64) + -| | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| method_id (uint16) | body_len (uint32) | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| | body (body_len bytes) | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +( + id: uint64 + method_id: uint16 + body: (message_body...) +) ``` and their corresponding responses are wrapped in a "response" structure: ``` - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| | -+ id (uint64) + -| | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| response_code (uint16) | result_len (uint32) | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| | result (result_len bytes) | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +( + id: uint64 + response_code: uint16 + result: bytes +) ``` -Note that the above structures are NOT encoded as SSZ but rather as sequences of bytes according to the packet diagrams above. This is because SSZ does not support structures without an explicit schema. Since the `body` and `result` fields depend on the value of `method_id` and `response_code`, a schema for the above structure cannot be known beforehand. +A union type is used to determine the contents of the `body` field in the request structure. Each "body" entry in the RPC calls below corresponds to one subtype in the `body` type union. The details of the RPC-Over-`libp2p` protocol are similar to [JSON-RPC 2.0](https://www.jsonrpc.org/specification). Specifically: From 05f9dc7baa356402a5a52b2bc0f910340ce75c52 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 17 May 2019 05:59:01 -0400 Subject: [PATCH 75/91] Fix #1090 Avoid signed integer --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ee37d9217..aa8cf57ca 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -887,7 +887,7 @@ def get_shuffled_index(index: ValidatorIndex, index_count: int, seed: Bytes32) - # See the 'generalized domain' algorithm on page 3 for round in range(SHUFFLE_ROUND_COUNT): pivot = bytes_to_int(hash(seed + int_to_bytes(round, length=1))[0:8]) % index_count - flip = (pivot - index) % index_count + flip = (pivot + index_count - index) % index_count position = max(index, flip) source = hash(seed + int_to_bytes(round, length=1) + int_to_bytes(position // 256, length=4)) byte = source[(position % 256) // 8] From f19188816b2bac23b41222181be1bde31fdc79d1 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 17 May 2019 06:07:38 -0400 Subject: [PATCH 76/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index aa8cf57ca..622eabaf8 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1362,19 +1362,19 @@ def process_justification_and_finalization(state: BeaconState) -> None: # Process finalizations bitfield = state.justification_bitfield # The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th as source - if (bitfield >> 1) % 8 == 0b111 and old_previous_justified_epoch == current_epoch - 3: + if (bitfield >> 1) % 8 == 0b111 and old_previous_justified_epoch + 3 == current_epoch: state.finalized_epoch = old_previous_justified_epoch state.finalized_root = get_block_root(state, state.finalized_epoch) # The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as source - if (bitfield >> 1) % 4 == 0b11 and old_previous_justified_epoch == current_epoch - 2: + if (bitfield >> 1) % 4 == 0b11 and old_previous_justified_epoch + 2 == current_epoch: state.finalized_epoch = old_previous_justified_epoch state.finalized_root = get_block_root(state, state.finalized_epoch) # The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as source - if (bitfield >> 0) % 8 == 0b111 and old_current_justified_epoch == current_epoch - 2: + if (bitfield >> 0) % 8 == 0b111 and old_current_justified_epoch + 2 == current_epoch: state.finalized_epoch = old_current_justified_epoch state.finalized_root = get_block_root(state, state.finalized_epoch) # The 1st/2nd most recent epochs are justified, the 1st using the 2nd as source - if (bitfield >> 0) % 4 == 0b11 and old_current_justified_epoch == current_epoch - 1: + if (bitfield >> 0) % 4 == 0b11 and old_current_justified_epoch + 1 == current_epoch: state.finalized_epoch = old_current_justified_epoch state.finalized_root = get_block_root(state, state.finalized_epoch) ``` From 694b31b934d13ffbddf17574fa7f04b824d67c98 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 17 May 2019 06:11:39 -0400 Subject: [PATCH 77/91] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 622eabaf8..b75f14153 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -647,7 +647,7 @@ def get_previous_epoch(state: BeaconState) -> Epoch: Return the current epoch if it's genesis epoch. """ current_epoch = get_current_epoch(state) - return (current_epoch - 1) if current_epoch > GENESIS_EPOCH else current_epoch + return GENESIS_EPOCH if current_epoch == GENESIS_EPOCH else current_epoch - 1 ``` ### `get_current_epoch` From 37aca60fae778268841a3e64253b85bb616b61ed Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 17 May 2019 10:55:07 -0400 Subject: [PATCH 78/91] pr feedback --- specs/validator/0_beacon-chain-validator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 5c92c9953..46eb55173 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -165,11 +165,11 @@ def is_proposer(state: BeaconState, return get_beacon_proposer_index(state) == validator_index ``` -*Note*: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot using `process_slots(state, current_slot)`. +*Note*: To see if a validator is assigned to propose during the slot, the beacon state must be in the epoch in question. At the epoch boundaries, the validator must run an epoch transition into the epoch to successfully check the proposal assignment of the first slot. ### Lookahead -The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must be checked during the slot in question. +The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must be checked during the epoch in question. `get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in Phase 1+). @@ -181,7 +181,7 @@ A validator has two primary responsibilities to the beacon chain: [proposing blo ### Block proposal -A validator is expected to propose a [`BeaconBlock`](../core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `is_proposer(state, validator_index)` returns `True`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot - 1`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](../core/0_beacon-chain.md#beacon-chain-state-transition-function). +A validator is expected to propose a [`BeaconBlock`](../core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `is_proposer(state, validator_index)` returns `True`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot - 1`. The validator creates, signs, and broadcasts a `block` that is a child of `parent` that satisfies a valid [beacon chain state transition](../core/0_beacon-chain.md#beacon-chain-state-transition-function). There is one proposer per slot, so if there are N active validators any individual validator will on average be assigned to propose once per N slots (e.g. at 312500 validators = 10 million ETH, that's once per ~3 weeks). From acb7444184b533de6c72fc65be1b0f7c506414c3 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 17 May 2019 10:58:02 -0400 Subject: [PATCH 79/91] pr feedback --- specs/validator/0_beacon-chain-validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 46eb55173..302deeae1 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -157,7 +157,7 @@ def get_committee_assignment( return committee, shard, slot ``` -A validator can use the following function to see if they are supposed to propose during their assigned committee slot. This function can only be run during the slot in question. Proposer selection is only stable within the context of the current epoch. +A validator can use the following function to see if they are supposed to propose during their assigned committee slot. This function can only be run with a `state` of the slot in question. Proposer selection is only stable within the context of the current epoch. ```python def is_proposer(state: BeaconState, @@ -171,7 +171,7 @@ def is_proposer(state: BeaconState, The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must be checked during the epoch in question. -`get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in Phase 1+). +`get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments by noting at which future slot they will have to attest and also which shard they should begin syncing (in Phase 1+). Specifically, a validator should call `get_committee_assignment(state, next_epoch, validator_index)` when checking for next epoch assignments. From e1d973d5467e25fe496cc67ba3800aab096d1c54 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 17 May 2019 11:01:18 -0400 Subject: [PATCH 80/91] Update specs/validator/0_beacon-chain-validator.md Co-Authored-By: Hsiao-Wei Wang --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 302deeae1..9e4fa7960 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -148,7 +148,7 @@ def get_committee_assignment( committees_per_slot = get_epoch_committee_count(state, epoch) // SLOTS_PER_EPOCH epoch_start_slot = get_epoch_start_slot(epoch) - for slot in range(epoch_start_slot, epoch_start_slot + SLOTS_PER_EPOCH) + for slot in range(epoch_start_slot, epoch_start_slot + SLOTS_PER_EPOCH): slot_start_shard = get_epoch_start_shard(state, epoch) + committees_per_slot * (slot % SLOTS_PER_EPOCH) for i in range(committees_per_slot): shard = (slot_start_shard + i) % SHARD_COUNT From 174e1e4dbef7605f1e4131f66d3fd0f52276b6a4 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 17 May 2019 11:04:05 -0400 Subject: [PATCH 81/91] pr feedback --- specs/validator/0_beacon-chain-validator.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 302deeae1..0940d592a 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -149,7 +149,8 @@ def get_committee_assignment( committees_per_slot = get_epoch_committee_count(state, epoch) // SLOTS_PER_EPOCH epoch_start_slot = get_epoch_start_slot(epoch) for slot in range(epoch_start_slot, epoch_start_slot + SLOTS_PER_EPOCH) - slot_start_shard = get_epoch_start_shard(state, epoch) + committees_per_slot * (slot % SLOTS_PER_EPOCH) + offset = committees_per_slot * (slot % SLOTS_PER_EPOCH) + slot_start_shard = (get_epoch_start_shard(state, epoch) + offset) % SHARD_COUNT for i in range(committees_per_slot): shard = (slot_start_shard + i) % SHARD_COUNT committee = get_crosslink_committee(state, epoch, shard) From 24edca3456166c11ebce912d3397c1618b79a3dc Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 17 May 2019 13:52:23 -0400 Subject: [PATCH 82/91] Fix to make Danny and hww happy --- specs/core/1_custody-game.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index e03e54ed0..e28b30794 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -689,9 +689,7 @@ Append this to `process_final_updates(state)`: ) for index, validator in enumerate(state.validator_registry): if index not in validator_indices_in_records: - if validator.exit_epoch == FAR_FUTURE_EPOCH: - validator.withdrawable_epoch = FAR_FUTURE_EPOCH - else: + if validator.exit_epoch != FAR_FUTURE_EPOCH and validator.withdrawable_epoch == FAR_FUTURE_EPOCH: validator.withdrawable_epoch = validator.exit_epoch + MIN_VALIDATOR_WITHDRAWABILITY_DELAY ``` From 85c16544566f82be57c5905ccf3b7b48e2888d1d Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 19 May 2019 09:33:01 -0400 Subject: [PATCH 83/91] Crosslinks store start and end epoch Solves #1034 --- specs/core/0_beacon-chain.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b75f14153..643985e85 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -281,8 +281,9 @@ The types are defined topologically to aid in facilitating an executable version { # Shard number 'shard': 'uint64', - # Epoch number - 'epoch': 'uint64', + # Crosslinking data from epochs [start....end-1] + 'start_epoch': 'uint64', + 'end_epoch': 'uint64', # Root of the previous crosslink 'parent_root': 'bytes32', # Root of the crosslinked shard data since the previous crosslink @@ -1728,7 +1729,8 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) - assert data.crosslink.epoch == min(data.target_epoch, parent_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) + assert data.crosslink.start_epoch == parent_crosslink.end_epoch + assert data.crosslink.end_epoch == min(data.target_epoch, parent_crosslink.end_epoch + MAX_EPOCHS_PER_CROSSLINK) assert data.crosslink.parent_root == hash_tree_root(parent_crosslink) assert data.crosslink.data_root == ZERO_HASH # [to be removed in phase 1] validate_indexed_attestation(state, convert_to_indexed(state, attestation)) From a2108741e804046b1ba721676482091a27f0e0fb Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Sun, 19 May 2019 15:47:59 -0600 Subject: [PATCH 84/91] fix tests with new starT_epoch and end_epoch in Crosslink --- test_libs/pyspec/eth2spec/utils/bls_stub.py | 2 +- .../tests/block_processing/test_process_attestation.py | 2 +- test_libs/pyspec/tests/helpers.py | 6 ++++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/test_libs/pyspec/eth2spec/utils/bls_stub.py b/test_libs/pyspec/eth2spec/utils/bls_stub.py index 108c4ef71..ae97de175 100644 --- a/test_libs/pyspec/eth2spec/utils/bls_stub.py +++ b/test_libs/pyspec/eth2spec/utils/bls_stub.py @@ -9,4 +9,4 @@ def bls_verify_multiple(pubkeys, message_hashes, signature, domain): def bls_aggregate_pubkeys(pubkeys): - return b'\x42' * 96 + return b'\x42' * 48 diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 708d68dca..6851561e9 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -124,7 +124,7 @@ def test_bad_previous_crosslink(state): for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): next_slot(state) - state.current_crosslinks[attestation.data.crosslink.shard].epoch += 10 + state.current_crosslinks[attestation.data.crosslink.shard].end_epoch += 10 pre_state, post_state = run_attestation_processing(state, attestation, False) diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index 5ddb2dc15..7af210f85 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -177,6 +177,7 @@ def build_attestation_data(state, slot, shard): justified_block_root = state.current_justified_root crosslinks = state.current_crosslinks if slot_to_epoch(slot) == get_current_epoch(state) else state.previous_crosslinks + parent_crosslink = crosslinks[shard] return AttestationData( beacon_block_root=block_root, source_epoch=justified_epoch, @@ -185,9 +186,10 @@ def build_attestation_data(state, slot, shard): target_root=epoch_boundary_root, crosslink=Crosslink( shard=shard, - epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), + start_epoch=parent_crosslink.end_epoch, + end_epoch=min(slot_to_epoch(slot), parent_crosslink.end_epoch + MAX_EPOCHS_PER_CROSSLINK), data_root=spec.ZERO_HASH, - parent_root=hash_tree_root(crosslinks[shard]), + parent_root=hash_tree_root(parent_crosslink), ), ) From 62f8d19ffc2f80763e7dfef5a66a19782080d5c4 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Sun, 19 May 2019 16:06:10 -0600 Subject: [PATCH 85/91] add some attestation crosslink tests --- .../test_process_attestation.py | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 6851561e9..97eddb902 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -64,6 +64,22 @@ def test_success_prevous_epoch(state): return pre_state, attestation, post_state +def test_success_since_max_epochs_per_crosslink(state): + for _ in range(spec.MAX_EPOCHS_PER_CROSSLINK + 2): + next_epoch(state) + + attestation = get_valid_attestation(state) + data = attestation.data + assert data.crosslink.end_epoch - data.crosslink.start_epoch == spec.MAX_EPOCHS_PER_CROSSLINK + + for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): + next_slot(state) + + pre_state, post_state = run_attestation_processing(state, attestation) + + return pre_state, attestation, post_state + + def test_before_inclusion_delay(state): attestation = get_valid_attestation(state) # do not increment slot to allow for inclusion delay @@ -131,6 +147,32 @@ def test_bad_previous_crosslink(state): return pre_state, attestation, post_state +def test_bad_crosslink_start_epoch(state): + next_epoch(state) + attestation = get_valid_attestation(state) + for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): + next_slot(state) + + attestation.data.crosslink.start_epoch += 1 + + pre_state, post_state = run_attestation_processing(state, attestation, False) + + return pre_state, attestation, post_state + + +def test_bad_crosslink_end_epoch(state): + next_epoch(state) + attestation = get_valid_attestation(state) + for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): + next_slot(state) + + attestation.data.crosslink.end_epoch += 1 + + pre_state, post_state = run_attestation_processing(state, attestation, False) + + return pre_state, attestation, post_state + + def test_non_empty_custody_bitfield(state): attestation = get_valid_attestation(state) state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY From 4c5e0548833bd3b4d92c22806afce48a1b4c1ab3 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Sun, 19 May 2019 16:11:39 -0600 Subject: [PATCH 86/91] fix previous crosslink root test --- .../pyspec/tests/block_processing/test_process_attestation.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 97eddb902..763178717 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -140,7 +140,7 @@ def test_bad_previous_crosslink(state): for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): next_slot(state) - state.current_crosslinks[attestation.data.crosslink.shard].end_epoch += 10 + attestation.data.crosslink.parent_root = b'\x27' * 32 pre_state, post_state = run_attestation_processing(state, attestation, False) From c14452bcf445761926da99a1259f0a792268bdff Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 19 May 2019 19:44:12 -0400 Subject: [PATCH 87/91] Updated get_custody_chunk_count Co-requisite with #1097 --- specs/core/1_custody-game.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index 0e8fcf6e9..7965e25f9 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -255,8 +255,8 @@ The `empty` function accepts and SSZ type as input and returns an object of that ```python def get_custody_chunk_count(attestation: Attestation) -> int: - crosslink_start_epoch = attestation.data.latest_crosslink.epoch - crosslink_end_epoch = slot_to_epoch(attestation.data.slot) + crosslink_start_epoch = attestation.data.latest_crosslink.start_epoch + crosslink_end_epoch = attestation.data.latest_crosslink.end_epoch crosslink_crosslink_length = min(MAX_EPOCHS_PER_CROSSLINK, end_epoch - start_epoch) chunks_per_epoch = 2 * BYTES_PER_SHARD_BLOCK * SLOTS_PER_EPOCH // BYTES_PER_CUSTODY_CHUNK return crosslink_crosslink_length * chunks_per_epoch From a68aa82b894153eca18f290d306623bf4118357b Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 20 May 2019 11:39:13 +0800 Subject: [PATCH 88/91] Update validator guide --- specs/validator/0_beacon-chain-validator.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 0940d592a..d05f25ef2 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -306,10 +306,12 @@ Set `attestation_data.beacon_block_root = signing_root(head_block)`. ##### Crosslink vote -Construct `attestation_data.crosslink` via the following +Construct `attestation_data.crosslink` via the following. * Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee. -* Set `attestation_data.crosslink.epoch = min(attestation_data.target_epoch, head_state.current_crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK)`. +* Let `parent_crosslink = head_state.current_crosslinks[shard]`. +* Set `attestation_data.crosslink.start_epoch = parent_crosslink.end_epoch`. +* Set `attestation_data.crosslink.end_epoch = min(attestation_data.target_epoch, parent_crosslink.end_epoch + MAX_EPOCHS_PER_CROSSLINK)`. * Set `attestation_data.crosslink.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. * Set `attestation_data.crosslink.data_root = ZERO_HASH`. *Note*: This is a stub for Phase 0. From 2018dd83f5a324472623db15a159be2882ffe8fa Mon Sep 17 00:00:00 2001 From: Justin Date: Mon, 20 May 2019 09:29:09 +0100 Subject: [PATCH 89/91] Update 1_custody-game.md --- specs/core/1_custody-game.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index 7965e25f9..f91fe81de 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -254,12 +254,10 @@ The `empty` function accepts and SSZ type as input and returns an object of that ### `get_crosslink_chunk_count` ```python -def get_custody_chunk_count(attestation: Attestation) -> int: - crosslink_start_epoch = attestation.data.latest_crosslink.start_epoch - crosslink_end_epoch = attestation.data.latest_crosslink.end_epoch - crosslink_crosslink_length = min(MAX_EPOCHS_PER_CROSSLINK, end_epoch - start_epoch) +def get_custody_chunk_count(crosslink: Crosslink) -> int: + crosslink_length = min(MAX_EPOCHS_PER_CROSSLINK, crosslink.end_epoch - crosslink.start_epoch) chunks_per_epoch = 2 * BYTES_PER_SHARD_BLOCK * SLOTS_PER_EPOCH // BYTES_PER_CUSTODY_CHUNK - return crosslink_crosslink_length * chunks_per_epoch + return crosslink_length * chunks_per_epoch ``` ### `get_custody_chunk_bit` @@ -470,7 +468,7 @@ def process_chunk_challenge(state: BeaconState, record.chunk_index != challenge.chunk_index ) # Verify depth - depth = math.log2(next_power_of_two(get_custody_chunk_count(challenge.attestation))) + depth = math.log2(next_power_of_two(get_custody_chunk_count(challenge.attestation.data.crosslink))) assert challenge.chunk_index < 2**depth # Add new chunk challenge record new_record = CustodyChunkChallengeRecord( @@ -544,7 +542,7 @@ def process_bit_challenge(state: BeaconState, ) # Verify the chunk count - chunk_count = get_custody_chunk_count(challenge.attestation) + chunk_count = get_custody_chunk_count(challenge.attestation.data.crosslink) assert verify_bitfield(challenge.chunk_bits, chunk_count) # Verify the first bit of the hash of the chunk bits does not equal the custody bit custody_bit = get_bitfield_bit(attestation.custody_bitfield, attesters.index(responder_index)) From 83123a33da792397101bdee263d03a25ee281e0b Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 20 May 2019 17:16:20 +0800 Subject: [PATCH 90/91] Set genesis_state.latest_block_header with `body_root` of empty BeaconBlockBody (#1098) --- specs/core/0_beacon-chain.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b75f14153..b34bbe7d1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1189,7 +1189,11 @@ Let `genesis_state = get_genesis_beacon_state(genesis_deposits, eth2genesis.gene ```python def get_genesis_beacon_state(deposits: List[Deposit], genesis_time: int, genesis_eth1_data: Eth1Data) -> BeaconState: - state = BeaconState(genesis_time=genesis_time, latest_eth1_data=genesis_eth1_data) + state = BeaconState( + genesis_time=genesis_time, + latest_eth1_data=genesis_eth1_data, + latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), + ) # Process genesis deposits for deposit in deposits: From 90a3f56e0feedd491e0c835ab33114e4d8a6d660 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 21 May 2019 09:13:57 +0100 Subject: [PATCH 91/91] Edit BLS spec warning Fix #898. --- specs/bls_signature.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/bls_signature.md b/specs/bls_signature.md index d119c4499..3fe1bcc0e 100644 --- a/specs/bls_signature.md +++ b/specs/bls_signature.md @@ -1,6 +1,8 @@ # BLS signature verification -**Warning: This document is pending academic review and should not yet be considered secure.** +**Notice**: This document is a placeholder to facilitate the emergence of cross-client testnets. Substantive changes are postponed until [BLS standardisation](https://github.com/pairingwg/bls_standard) is finalized. + +**Warning**: The constructions in this document should not be considered secure. In particular, the `hash_to_G2` function is known to be unsecure. ## Table of contents