From cd855469543648f887f0cfb98fa505971d1ef25f Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 10 Apr 2019 15:09:53 +0100 Subject: [PATCH 01/45] Add option type and null --- specs/simple-serialize.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b78eff93e..1f2c01949 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,6 +30,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` +* '"null"': `Null` ### Composite types @@ -39,6 +40,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` +* **option**: option type containing one of the given subtypes + * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". @@ -70,7 +73,13 @@ assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### Vectors, containers, lists +### `"null"` + +```python +return b"" +``` + +### Vectors, containers, lists, options If `value` is fixed-size: @@ -87,6 +96,17 @@ serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "lit return serialized_length + serialized_bytes ``` +If `value` is an option type: + +Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. + +```python +serialized_bytes = serialize(value.value) +serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") +return serialized_length + serialized_bytes +``` + + ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). @@ -98,6 +118,7 @@ We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. * `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. +* `mix_in_type`: Given a Merkle root `root` and a type_index `type_index` (`"uint256"` little-endian serialization) return `hash(root + type_index)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -105,6 +126,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects +* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of option type ## Self-signed containers From 283ba8f7611e3112f1466eac63c0fc2966f94c79 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Thu, 11 Apr 2019 16:00:53 +0100 Subject: [PATCH 02/45] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 1f2c01949..5b302e2f5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -103,7 +103,7 @@ Define value as an object that has properties `value.value` with the contained v ```python serialized_bytes = serialize(value.value) serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") -return serialized_length + serialized_bytes +return serialized_type_index + serialized_bytes ``` From 2017ce96149306721bfd8d18553f531594b937b5 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Thu, 11 Apr 2019 16:05:16 +0100 Subject: [PATCH 03/45] Rename "option" -> "union"; "null" only in unions --- specs/simple-serialize.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 5b302e2f5..8a8c78446 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,7 +30,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` -* '"null"': `Null` +* `"null"`: `None` + +The type `"null"` is only legal as one of several type in a `union` type. ### Composite types @@ -40,7 +42,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * angle bracket notation `[type, N]`, e.g. `["uint64", N]` * **list**: ordered variable-length homogenous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` -* **option**: option type containing one of the given subtypes +* **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". @@ -79,7 +81,7 @@ return b"\x01" if value is True else b"\x00" return b"" ``` -### Vectors, containers, lists, options +### Vectors, containers, lists, unions If `value` is fixed-size: @@ -96,7 +98,7 @@ serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "lit return serialized_length + serialized_bytes ``` -If `value` is an option type: +If `value` is an union type: Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. @@ -126,7 +128,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects -* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of option type +* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of union type ## Self-signed containers From 63bdf95e79e2dcced7e2f5ba3945700972a5ef9d Mon Sep 17 00:00:00 2001 From: dankrad Date: Sat, 20 Apr 2019 11:46:31 +0100 Subject: [PATCH 04/45] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 8a8c78446..c9f845873 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -45,7 +45,7 @@ The type `"null"` is only legal as one of several type in a `union` type. * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` -We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size". +We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". ### Aliases From a481a4e96cf29d13dd19fef33172d0cefa390e04 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 20 Apr 2019 20:57:50 +1000 Subject: [PATCH 05/45] Update simple-serialize.md --- specs/simple-serialize.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index c9f845873..2d9b715b3 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -32,7 +32,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"bool"`: `True` or `False` * `"null"`: `None` -The type `"null"` is only legal as one of several type in a `union` type. +The `"null"` type is only legal as a union sub-type. ### Composite types @@ -108,7 +108,6 @@ serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "litt return serialized_type_index + serialized_bytes ``` - ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). From 101449e71a3492ff97c649c706866a0da2ec6962 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Sat, 27 Apr 2019 21:00:50 +0100 Subject: [PATCH 06/45] Define null as alias of {} --- specs/simple-serialize.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index abef5c669..6858561e8 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -8,12 +8,13 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) + - [Illegal empty composites](#illegal-empty-composites) - [Aliases](#aliases) - [Default values](#default-values) - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) - - [Containers, vectors, lists](#containers-vectors-lists) + - [Vectors, containers, lists, unions](#vectors-containers-lists-unions) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -32,9 +33,6 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `"bool"`: `True` or `False` -* `"null"`: `None` - -The `"null"` type is only legal as a union sub-type. ### Composite types @@ -49,6 +47,10 @@ The `"null"` type is only legal as a union sub-type. We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". +#### Illegal empty composites + +The empty container `{}` (except as the `"null"` type inside a union, see below) and the empty fixed length list `[type, 0]` are **not** legal types. + ### Aliases For convenience we alias: @@ -56,6 +58,9 @@ For convenience we alias: * `"byte"` to `"uint8"` (this is a basic type) * `"bytes"` to `["byte"]` (this is *not* a basic type) * `"bytesN"` to `["byte", N]` (this is *not* a basic type) +* `"null"`: `{}`, i.e. the empty container + +The `"null"` type is only legal as a union sub-type. ### Default values From 4a483309a544dcd24095e5bb9d5a05b832e97b3b Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:07 +0200 Subject: [PATCH 07/45] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6858561e8..79ed0ef7b 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -113,7 +113,7 @@ fixed_parts = [part if part != None else variable_offsets[i] for i, part in enum return b"".join(fixed_parts + variable_parts) ``` -If `value` is an union type: +If `value` is a union type: Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. From d0447022cb4982e544abdea22e616e2cc7c44444 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:24 +0200 Subject: [PATCH 08/45] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 79ed0ef7b..0e0468eb7 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -119,7 +119,7 @@ Define value as an object that has properties `value.value` with the contained v ```python serialized_bytes = serialize(value.value) -serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_PREFIX, "little") +serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_OFFSET, "little") return serialized_type_index + serialized_bytes ``` From cc22432bb91626d48ba62e786da72891f60424de Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 1 May 2019 12:39:44 +0200 Subject: [PATCH 09/45] Update specs/simple-serialize.md Co-Authored-By: dankrad --- specs/simple-serialize.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 0e0468eb7..f6b5cb232 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -45,6 +45,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` +#### Variable-size and fixed-size + We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". #### Illegal empty composites From a33ee00239f0ecd1f8bb557372844250431d0f68 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 13:52:37 +0100 Subject: [PATCH 10/45] Update simple-serialize.md --- specs/simple-serialize.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f6b5cb232..b7a1375e4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -8,9 +8,10 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) - - [Illegal empty composites](#illegal-empty-composites) + - [Variable-size and fixed-size](#variable-size-and-fixed-size) - [Aliases](#aliases) - [Default values](#default-values) + - [Illegal types](#illegal-types) - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) @@ -45,14 +46,10 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **union**: union type containing one of the given subtypes * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` -#### Variable-size and fixed-size +### Variable-size and fixed-size We recursively define "variable-size" types to be lists and unions and all types that contain a variable-size type. All other types are said to be "fixed-size". -#### Illegal empty composites - -The empty container `{}` (except as the `"null"` type inside a union, see below) and the empty fixed length list `[type, 0]` are **not** legal types. - ### Aliases For convenience we alias: @@ -62,19 +59,20 @@ For convenience we alias: * `"bytesN"` to `["byte", N]` (this is *not* a basic type) * `"null"`: `{}`, i.e. the empty container -The `"null"` type is only legal as a union sub-type. - ### Default values The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. +### Illegal types + +Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as a union subtype. + ## Serialization We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. > *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signing_root`, `is_variable_size`, etc.) objects implicitly carry their type. - ### `"uintN"` ```python From 563df146b9073da8f91b6ad200cd19273ea8f2ac Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 1 May 2019 13:55:02 +0100 Subject: [PATCH 11/45] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b7a1375e4..21a87a6f9 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -44,7 +44,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **list**: ordered variable-length homogeneous collection of values * angle bracket notation `[type]`, e.g. `["uint64"]` * **union**: union type containing one of the given subtypes - * round bracket notation `(type1, type2, ...)`, e.g. `("uint64", "null")` + * round bracket notation `(type_1, type_2, ...)`, e.g. `("uint64", "null")` ### Variable-size and fixed-size From b1930d22394911c9549f479adb92f62ec41b0958 Mon Sep 17 00:00:00 2001 From: Dankrad Feist Date: Wed, 1 May 2019 15:12:49 +0100 Subject: [PATCH 12/45] Union default values --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 21a87a6f9..6adc9c4b8 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -61,11 +61,11 @@ For convenience we alias: ### Default values -The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. +The default value of a type upon initialization is recursively defined using `0` for `"uintN"`, `False` for `"bool"`, and `[]` for lists. Unions default to the first type in the union (with type index zero), which is `"null"` if present in the union. ### Illegal types -Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as a union subtype. +Empty vector types (i.e. `[subtype, 0]` for some `subtype`) are not legal. The `"null"` type is only legal as the first type in a union subtype (i.e., with type index zero). ## Serialization From 524d7b09ba246e33a1007d4c4f72d99d59158260 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 2 May 2019 00:34:17 +0200 Subject: [PATCH 13/45] implement SOS serialization in minimal_ssz.py --- .../pyspec/eth2spec/utils/minimal_ssz.py | 55 +++++++++++-------- 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/test_libs/pyspec/eth2spec/utils/minimal_ssz.py b/test_libs/pyspec/eth2spec/utils/minimal_ssz.py index dbe9d1359..35960a7ce 100644 --- a/test_libs/pyspec/eth2spec/utils/minimal_ssz.py +++ b/test_libs/pyspec/eth2spec/utils/minimal_ssz.py @@ -3,7 +3,7 @@ from typing import Any from .hash_function import hash BYTES_PER_CHUNK = 32 -BYTES_PER_LENGTH_PREFIX = 4 +BYTES_PER_LENGTH_OFFSET = 4 ZERO_CHUNK = b'\x00' * BYTES_PER_CHUNK @@ -111,19 +111,34 @@ def coerce_to_bytes(x): raise Exception("Expecting bytes") -def encode_bytes(value): - serialized_bytes = coerce_to_bytes(value) - assert len(serialized_bytes) < 2 ** (8 * BYTES_PER_LENGTH_PREFIX) - serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, 'little') - return serialized_length + serialized_bytes +def encode_series(values, types): + # Recursively serialize + parts = [(is_constant_sized(types[i]), serialize_value(values[i], types[i])) for i in range(len(values))] + # Compute and check lengths + fixed_lengths = [len(serialized) if constant_size else BYTES_PER_LENGTH_OFFSET + for (constant_size, serialized) in parts] + variable_lengths = [len(serialized) if not constant_size else 0 + for (constant_size, serialized) in parts] -def encode_variable_size_container(values, types): - return encode_bytes(encode_fixed_size_container(values, types)) + # Check if integer is not out of bounds (Python) + assert sum(fixed_lengths + variable_lengths) < 2 ** (BYTES_PER_LENGTH_OFFSET * 8) + # Interleave offsets of variable-size parts with fixed-size parts. + # Avoid quadratic complexity in calculation of offsets. + offset = sum(fixed_lengths) + variable_parts = [] + fixed_parts = [] + for (constant_size, serialized) in parts: + if constant_size: + fixed_parts.append(serialized) + else: + fixed_parts.append(offset.to_bytes(BYTES_PER_LENGTH_OFFSET, 'little')) + variable_parts.append(serialized) + offset += len(serialized) -def encode_fixed_size_container(values, types): - return b''.join([serialize_value(v, typ) for (v, typ) in zip(values, types)]) + # Return the concatenation of the fixed-size parts (offsets interleaved) with the variable-size parts + return b"".join(fixed_parts + variable_parts) def serialize_value(value, typ=None): @@ -142,18 +157,13 @@ def serialize_value(value, typ=None): elif isinstance(typ, list) and len(typ) == 2: # (regardless of element type, sanity-check if the length reported in the vector type matches the value length) assert len(value) == typ[1] - # If value is fixed-size (i.e. element type is fixed-size): - if is_constant_sized(typ): - return encode_fixed_size_container(value, [typ[0]] * len(value)) - # If value is variable-size (i.e. element type is variable-size) - else: - return encode_variable_size_container(value, [typ[0]] * len(value)) - # "bytes" (variable size) - elif isinstance(typ, str) and typ == 'bytes': - return encode_bytes(value) + return encode_series(value, [typ[0]] * len(value)) # List elif isinstance(typ, list) and len(typ) == 1: - return encode_variable_size_container(value, [typ[0]] * len(value)) + return encode_series(value, [typ[0]] * len(value)) + # "bytes" (variable size) + elif isinstance(typ, str) and typ == 'bytes': + return (b"\x00" * BYTES_PER_LENGTH_OFFSET) + coerce_to_bytes(value) # "bytesN" (fixed size) elif isinstance(typ, str) and len(typ) > 5 and typ[:5] == 'bytes': assert len(value) == int(typ[5:]), (value, int(typ[5:])) @@ -162,10 +172,7 @@ def serialize_value(value, typ=None): elif hasattr(typ, 'fields'): values = [getattr(value, field) for field in typ.fields.keys()] types = list(typ.fields.values()) - if is_constant_sized(typ): - return encode_fixed_size_container(values, types) - else: - return encode_variable_size_container(values, types) + return encode_series(values, types) else: print(value, typ) raise Exception("Type not recognized") From faf36e056f223927564d2fbf327926ddc455ce59 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Wed, 1 May 2019 16:10:01 -0700 Subject: [PATCH 14/45] Add notes to compute fork choice at genesis --- specs/core/0_fork-choice.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index 0b19f1559..cb11ea860 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -13,6 +13,8 @@ - [Time parameters](#time-parameters) - [Beacon chain processing](#beacon-chain-processing) - [Beacon chain fork choice rule](#beacon-chain-fork-choice-rule) + - [Implementation notes](#implementation-notes) + - [Justification and finality at genesis](#justification-and-finality-at-genesis) @@ -99,3 +101,9 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock) # Ties broken by favoring block with lexicographically higher root head = max(children, key=lambda x: (get_vote_count(x), hash_tree_root(x))) ``` + +## Implementation notes + +### Justification and finality at genesis + +Clients may choose to refer to the justification and finality data in a given `BeaconState` to determine the finalized/justified head. During the early epochs right after genesis, the justification and finality data are not stored in the `BeaconState`. In their place are "empty" values; for example, the 32-byte zero value as the `finalized_root`. Clients wishing to compute the fork choice in these early epochs should work around this fact of the `BeaconState` to recognize that the genesis epoch and root of the genesis block are _both_ the finalized and justified heads until updated via the state transition function defined in [Phase 0 -- The Beacon Chain](./0_beacon-chain.md). Solutions will be language-specific but one possibility is to treat the zero-value hash as an exceptional case that references the genesis block/epoch. From d28e5e4843ae39fba177e8c8744bd248cd938c15 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Wed, 1 May 2019 16:26:18 -0700 Subject: [PATCH 15/45] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 55791e25f..68e7ff484 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1151,6 +1151,9 @@ def get_delayed_activation_exit_epoch(epoch: Epoch) -> Epoch: ```python def get_churn_limit(state: BeaconState) -> int: + """ + Return the churn limit based on the active validator count. + """ return max( MIN_PER_EPOCH_CHURN_LIMIT, len(get_active_validator_indices(state, get_current_epoch(state))) // CHURN_LIMIT_QUOTIENT From 68488a34d0bb5d47f3b4324ee3858d852f4505dd Mon Sep 17 00:00:00 2001 From: terence tsao Date: Wed, 1 May 2019 16:33:21 -0700 Subject: [PATCH 16/45] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ad7204f21..27ff7fcb1 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Core specifications for eth2.0 client validation can be found in [specs/core](sp * [The Beacon Chain](specs/core/0_beacon-chain.md) * [Fork Choice](specs/core/0_fork-choice.md) * [Deposit Contract](specs/core/0_deposit-contract.md) -* [Honest validator implementation doc](specs/validator/0_beacon-chain-validator.md) +* [Honest Validator](specs/validator/0_beacon-chain-validator.md) ### Phase 1 * [Custody Game](specs/core/1_custody-game.md) From b169c42339205b483eac2fc98b9132c001f4ded4 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 1 May 2019 17:44:34 -0600 Subject: [PATCH 17/45] fix underflows in generate_seed, get_randao_mix, and get_active_index_roots --- scripts/phase0/build_spec.py | 4 ++-- specs/core/0_beacon-chain.md | 8 +++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index 9d6b5180b..011fc6e25 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -48,10 +48,10 @@ def compute_committee(indices: List[ValidatorIndex], seed: Bytes32, index: int, param_hash = (hash_tree_root(indices), seed, index, count) if param_hash in committee_cache: - print("Cache hit, param_hash: ", param_hash) + # print("Cache hit, param_hash: ", param_hash) return committee_cache[param_hash] else: - print("Cache miss, param_hash: ", param_hash) + # print("Cache miss, param_hash: ", param_hash) ret = _compute_committee(indices, seed, index, count) committee_cache[param_hash] = ret return ret diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2a0b0c11d..b30b5ca70 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -800,7 +800,8 @@ def get_randao_mix(state: BeaconState, """ Return the randao mix at a recent ``epoch``. """ - assert get_current_epoch(state) - LATEST_RANDAO_MIXES_LENGTH < epoch <= get_current_epoch(state) + min_epoch = epoch - LATEST_RANDAO_MIXES_LENGTH + 1 if epoch > LATEST_RANDAO_MIXES_LENGTH else GENESIS_EPOCH + assert min_epoch <= epoch <= get_current_epoch(state) return state.latest_randao_mixes[epoch % LATEST_RANDAO_MIXES_LENGTH] ``` @@ -812,7 +813,8 @@ def get_active_index_root(state: BeaconState, """ Return the index root at a recent ``epoch``. """ - assert get_current_epoch(state) - LATEST_ACTIVE_INDEX_ROOTS_LENGTH + ACTIVATION_EXIT_DELAY < epoch <= get_current_epoch(state) + ACTIVATION_EXIT_DELAY + min_epoch = get_current_epoch(state) + ACTIVATION_EXIT_DELAY - LATEST_ACTIVE_INDEX_ROOTS_LENGTH + 1 if epoch > LATEST_ACTIVE_INDEX_ROOTS_LENGTH - ACTIVATION_EXIT_DELAY else GENESIS_EPOCH + assert min_epoch <= epoch <= get_current_epoch(state) + ACTIVATION_EXIT_DELAY return state.latest_active_index_roots[epoch % LATEST_ACTIVE_INDEX_ROOTS_LENGTH] ``` @@ -825,7 +827,7 @@ def generate_seed(state: BeaconState, Generate a seed for the given ``epoch``. """ return hash( - get_randao_mix(state, epoch - MIN_SEED_LOOKAHEAD) + + get_randao_mix(state, epoch - MIN_SEED_LOOKAHEAD) if epoch >= MIN_SEED_LOOKAHEAD else ZERO_HASH + get_active_index_root(state, epoch) + int_to_bytes32(epoch) ) From bbaa1417a8fe52553a8a272ae50f7ab6331d1d1b Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 2 May 2019 09:16:01 +0200 Subject: [PATCH 18/45] remove incorrect unnecessary offset --- test_libs/pyspec/eth2spec/utils/minimal_ssz.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test_libs/pyspec/eth2spec/utils/minimal_ssz.py b/test_libs/pyspec/eth2spec/utils/minimal_ssz.py index 35960a7ce..9cc2baebb 100644 --- a/test_libs/pyspec/eth2spec/utils/minimal_ssz.py +++ b/test_libs/pyspec/eth2spec/utils/minimal_ssz.py @@ -163,7 +163,7 @@ def serialize_value(value, typ=None): return encode_series(value, [typ[0]] * len(value)) # "bytes" (variable size) elif isinstance(typ, str) and typ == 'bytes': - return (b"\x00" * BYTES_PER_LENGTH_OFFSET) + coerce_to_bytes(value) + return coerce_to_bytes(value) # "bytesN" (fixed size) elif isinstance(typ, str) and len(typ) > 5 and typ[:5] == 'bytes': assert len(value) == int(typ[5:]), (value, int(typ[5:])) From d15ca4c50cf5aecde76dfcc2687a0b52174dfdab Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 2 May 2019 18:11:11 -0600 Subject: [PATCH 19/45] replace asserts in get_active_index_root and get_randao_mix with comment --- scripts/phase0/build_spec.py | 2 -- specs/core/0_beacon-chain.md | 9 ++++----- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index 011fc6e25..da5845951 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -48,10 +48,8 @@ def compute_committee(indices: List[ValidatorIndex], seed: Bytes32, index: int, param_hash = (hash_tree_root(indices), seed, index, count) if param_hash in committee_cache: - # print("Cache hit, param_hash: ", param_hash) return committee_cache[param_hash] else: - # print("Cache miss, param_hash: ", param_hash) ret = _compute_committee(indices, seed, index, count) committee_cache[param_hash] = ret return ret diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index b30b5ca70..56f11bc7f 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -799,9 +799,8 @@ def get_randao_mix(state: BeaconState, epoch: Epoch) -> Bytes32: """ Return the randao mix at a recent ``epoch``. + ``epoch`` expected to be between (current_epoch - LATEST_RANDAO_MIXES_LENGTH, current_epoch]. """ - min_epoch = epoch - LATEST_RANDAO_MIXES_LENGTH + 1 if epoch > LATEST_RANDAO_MIXES_LENGTH else GENESIS_EPOCH - assert min_epoch <= epoch <= get_current_epoch(state) return state.latest_randao_mixes[epoch % LATEST_RANDAO_MIXES_LENGTH] ``` @@ -812,9 +811,9 @@ def get_active_index_root(state: BeaconState, epoch: Epoch) -> Bytes32: """ Return the index root at a recent ``epoch``. + ``epoch`` expected to be between + (current_epoch - LATEST_ACTIVE_INDEX_ROOTS_LENGTH + ACTIVATION_EXIT_DELAY, current_epoch + ACTIVATION_EXIT_DELAY]. """ - min_epoch = get_current_epoch(state) + ACTIVATION_EXIT_DELAY - LATEST_ACTIVE_INDEX_ROOTS_LENGTH + 1 if epoch > LATEST_ACTIVE_INDEX_ROOTS_LENGTH - ACTIVATION_EXIT_DELAY else GENESIS_EPOCH - assert min_epoch <= epoch <= get_current_epoch(state) + ACTIVATION_EXIT_DELAY return state.latest_active_index_roots[epoch % LATEST_ACTIVE_INDEX_ROOTS_LENGTH] ``` @@ -827,7 +826,7 @@ def generate_seed(state: BeaconState, Generate a seed for the given ``epoch``. """ return hash( - get_randao_mix(state, epoch - MIN_SEED_LOOKAHEAD) if epoch >= MIN_SEED_LOOKAHEAD else ZERO_HASH + + get_randao_mix(state, epoch + LATEST_RANDAO_MIXES_LENGTH - MIN_SEED_LOOKAHEAD) + get_active_index_root(state, epoch) + int_to_bytes32(epoch) ) From 5d688ddb9d5c156762ca2482c201374d3607e58e Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 2 May 2019 20:45:39 -0700 Subject: [PATCH 20/45] Update property name --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 82f740f7d..10a228e50 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -181,7 +181,7 @@ epoch_signature = bls_sign( `block.eth1_data` is a mechanism used by block proposers vote on a recent Ethereum 1.0 block hash and an associated deposit root found in the Ethereum 1.0 deposit contract. When consensus is formed, `state.latest_eth1_data` is updated, and validator deposits up to this root can be processed. The deposit root can be calculated by calling the `get_deposit_root()` function of the deposit contract using the post-state of the block hash. * Let `D` be the set of `Eth1DataVote` objects `vote` in `state.eth1_data_votes` where: - * `vote.eth1_data.block_hash` is the hash of an eth1.0 block that is (i) part of the canonical chain, (ii) >= `ETH1_FOLLOW_DISTANCE` blocks behind the head, and (iii) newer than `state.latest_eth1_data.block_data`. + * `vote.eth1_data.block_hash` is the hash of an eth1.0 block that is (i) part of the canonical chain, (ii) >= `ETH1_FOLLOW_DISTANCE` blocks behind the head, and (iii) newer than `state.latest_eth1_data.block_hash`. * `vote.eth1_data.deposit_count` is the deposit count of the eth1.0 deposit contract at the block defined by `vote.eth1_data.block_hash`. * `vote.eth1_data.deposit_root` is the deposit root of the eth1.0 deposit contract at the block defined by `vote.eth1_data.block_hash`. * If `D` is empty: From 43e85f76d99b3d6c3e0becd7364c14e4ecb4f80c Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Thu, 2 May 2019 21:05:51 -0700 Subject: [PATCH 21/45] Update name of constant max deposit amount is now in spirit named max effective balance --- specs/validator/0_beacon-chain-validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 82f740f7d..0b23d9f46 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -101,12 +101,12 @@ In phase 0, all incoming validator deposits originate from the Ethereum 1.0 PoW To submit a deposit: * Pack the validator's [initialization parameters](#initialization) into `deposit_data`, a [`DepositData`](../core/0_beacon-chain.md#depositdata) SSZ object. -* Let `amount` be the amount in Gwei to be deposited by the validator where `MIN_DEPOSIT_AMOUNT <= amount <= MAX_DEPOSIT_AMOUNT`. +* Let `amount` be the amount in Gwei to be deposited by the validator where `MIN_DEPOSIT_AMOUNT <= amount <= MAX_EFFECTIVE_BALANCE`. * Set `deposit_data.amount = amount`. * Let `signature` be the result of `bls_sign` of the `signing_root(deposit_data)` with `domain=DOMAIN_DEPOSIT`. * Send a transaction on the Ethereum 1.0 chain to `DEPOSIT_CONTRACT_ADDRESS` executing `def deposit(pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96])` along with a deposit of `amount` Gwei. -_Note_: Deposits made for the same `pubkey` are treated as for the same validator. A singular `Validator` will be added to `state.validator_registry` with each additional deposit amount added to the validator's balance. A validator can only be activated when total deposits for the validator pubkey meet or exceed `MAX_DEPOSIT_AMOUNT`. +_Note_: Deposits made for the same `pubkey` are treated as for the same validator. A singular `Validator` will be added to `state.validator_registry` with each additional deposit amount added to the validator's balance. A validator can only be activated when total deposits for the validator pubkey meet or exceed `MAX_EFFECTIVE_BALANCE`. ### Process deposit From 4ca2f1182732d2dc88e3edf8fe60383e3dbc681d Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 3 May 2019 17:20:54 +0800 Subject: [PATCH 22/45] Custody game changes (#866) * Custody game changes 1. Don't store the full chunk bits, instead only store a Merkle root. Increased history size complexity from `N` to `N + log(N)` but with the benefit of decreasing storage requirements from `N` to a single 32 byte hash. 2. `custody_bit` is computed as the first bit of the hash of the custody bits, not the xor. This allows us to more safely use functions with more risky security assumptions for computing the chunk mix. * Update specs/core/1_custody-game.md * Update specs/core/1_custody-game.md * Update specs/core/1_custody-game.md * Update specs/core/1_custody-game.md * XOR aggregation before SHA256 to reduce number of hashes * Simplifed get_chunk_bits_root * standalone -> indexed * Fix missing "data" and ToC --- specs/core/1_custody-game.md | 55 +++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 16 deletions(-) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index 9f1c0e66a..0f1555e62 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -32,6 +32,7 @@ - [`empty`](#empty) - [`get_crosslink_chunk_count`](#get_crosslink_chunk_count) - [`get_custody_chunk_bit`](#get_custody_chunk_bit) + - [`get_chunk_bits_root`](#get_chunk_bits_root) - [`epoch_to_custody_period`](#epoch_to_custody_period) - [`replace_empty_or_append`](#replace_empty_or_append) - [`verify_custody_key`](#verify_custody_key) @@ -148,7 +149,8 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'responder_index': ValidatorIndex, 'deadline': Epoch, 'crosslink_data_root': Hash, - 'chunk_bits': Bitfield, + 'chunk_count': 'uint64', + 'chunk_bits_merkle_root': Hash, 'responder_key': BLSSignature, } ``` @@ -160,7 +162,9 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenge_index': 'uint64', 'chunk_index': 'uint64', 'chunk': ['byte', BYTES_PER_CUSTODY_CHUNK], - 'branch': [Hash], + 'data_branch': [Hash], + 'chunk_bits_branch': [Hash], + 'chunk_bits_leaf': Hash, } ``` @@ -233,6 +237,17 @@ def get_custody_chunk_bit(key: BLSSignature, chunk: bytes) -> bool: return get_bitfield_bit(hash(challenge.responder_key + chunk), 0) ``` +### `get_chunk_bits_root` + +```python +def get_chunk_bits_root(chunk_bitfield: Bitfield) -> Bytes32: + aggregated_bits = bytearray([0] * 32) + for i in range(0, len(chunk_bitfield), 32): + for j in range(32): + aggregated_bits[j] ^= chunk_bitfield[i+j] + return hash(aggregated_bits) +``` + ### `epoch_to_custody_period` ```python @@ -326,7 +341,7 @@ For each `challenge` in `block.body.custody_chunk_challenges`, run the following def process_chunk_challenge(state: BeaconState, challenge: CustodyChunkChallenge) -> None: # Verify the attestation - assert verify_standalone_attestation(state, convert_to_standalone(state, challenge.attestation)) + assert verify_indexed_attestation(state, convert_to_indexed(state, challenge.attestation)) # Verify it is not too late to challenge assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_CHUNK_CHALLENGE_DELAY responder = state.validator_registry[challenge.responder_index] @@ -380,7 +395,7 @@ def process_bit_challenge(state: BeaconState, # Verify the challenger is not slashed assert challenger.slashed is False # Verify the attestation - assert verify_standalone_attestation(state, convert_to_standalone(state, challenge.attestation)) + assert verify_indexed_attestation(state, convert_to_indexed(state, challenge.attestation)) # Verify the attestation is eligible for challenging responder = state.validator_registry[challenge.responder_index] min_challengeable_epoch = responder.exit_epoch - EPOCHS_PER_CUSTODY_PERIOD * (1 + responder.max_reveal_lateness) @@ -403,20 +418,18 @@ def process_bit_challenge(state: BeaconState, # Verify the chunk count chunk_count = get_custody_chunk_count(challenge.attestation) assert verify_bitfield(challenge.chunk_bits, chunk_count) - # Verify the xor of the chunk bits does not equal the custody bit - chunk_bits_xor = 0b0 - for i in range(chunk_count): - chunk_bits_xor ^ get_bitfield_bit(challenge.chunk_bits, i) + # Verify the first bit of the hash of the chunk bits does not equal the custody bit custody_bit = get_bitfield_bit(attestation.custody_bitfield, attesters.index(responder_index)) - assert custody_bit != chunk_bits_xor + assert custody_bit != get_bitfield_bit(get_chunk_bits_root(challenge.chunk_bits), 0) # Add new bit challenge record new_record = CustodyBitChallengeRecord( challenge_index=state.custody_challenge_index, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE - crosslink_data_root=challenge.attestation.crosslink_data_root, - chunk_bits=challenge.chunk_bits, + crosslink_data_root=challenge.attestation.data.crosslink_data_root, + chunk_count=chunk_count, + chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), responder_key=challenge.responder_key, ) replace_empty_or_append(state.custody_bit_challenge_records, new_record) @@ -451,10 +464,12 @@ def process_chunk_challenge_response(state: BeaconState, challenge: CustodyChunkChallengeRecord) -> None: # Verify chunk index assert response.chunk_index == challenge.chunk_index + # Verify bit challenge data is null + assert response.chunk_bits_branch == [] and response.chunk_bits_leaf == ZERO_HASH # Verify the chunk matches the crosslink data root assert verify_merkle_branch( leaf=hash_tree_root(response.chunk), - branch=response.branch, + branch=response.data_branch, depth=challenge.depth, index=response.chunk_index, root=challenge.crosslink_data_root, @@ -472,17 +487,25 @@ def process_bit_challenge_response(state: BeaconState, response: CustodyResponse, challenge: CustodyBitChallengeRecord) -> None: # Verify chunk index - assert response.chunk_index < len(challenge.chunk_bits) + assert response.chunk_index < challenge.chunk_count # Verify the chunk matches the crosslink data root assert verify_merkle_branch( leaf=hash_tree_root(response.chunk), - branch=response.branch, - depth=math.log2(next_power_of_two(len(challenge.chunk_bits))), + branch=response.data_branch, + depth=math.log2(next_power_of_two(challenge.chunk_count)), index=response.chunk_index, root=challenge.crosslink_data_root, ) + # Verify the chunk bit leaf matches the challenge data + assert verify_merkle_branch( + leaf=response.chunk_bits_leaf, + branch=response.chunk_bits_branch, + depth=math.log2(next_power_of_two(challenge.chunk_count) // 256), + index=response.chunk_index // 256, + root=challenge.chunk_bits_merkle_root + ) # Verify the chunk bit does not match the challenge chunk bit - assert get_custody_chunk_bit(challenge.responder_key, response.chunk) != get_bitfield_bit(challenge.chunk_bits, response.chunk_index) + assert get_custody_chunk_bit(challenge.responder_key, response.chunk) != get_bitfield_bit(challenge.chunk_bits_leaf, response.chunk_index % 256) # Clear the challenge records = state.custody_bit_challenge_records records[records.index(challenge)] = CustodyBitChallengeRecord() From c0f345309393eb8842fbd96fce65a71d5245ce62 Mon Sep 17 00:00:00 2001 From: dankrad Date: Fri, 3 May 2019 11:30:55 +0200 Subject: [PATCH 23/45] RANDAO reveal slashing, custody period staggering and integration of custody and RANDAO reveals (#880) * Add RandaoRevealSlashing for early Randao reveals * add some randao reveal slashing tests * Unifying RANDAO reveal slashing and Custody reveal slashing; implemented more lenient penalty for not-to-early reveals * Fix custody reveal domain * Also test exposed_randao_reveals * Re-add exposed_randao_reveals to validator initialization * Fix tests * Unified Randao Reveal slashing mechanism and Custody Reveal, CUSTODY_PERIOD_TO_RANDAO_PADDING = EPOCHS_PER_CUSTODY_PERIOD * Move exposed_randao_reveals into separate data structure to keep validator record clean * new shiny staggered custody periods * Fixes style and type hinting * removes whitespace * Cleans up multi-line conditionals * Make RANDAO key reveal penalties proportional to block rewards * Minor typos * Minor typos * Fixes off-by one error * Removes unnecicary whitepsace * Clean up comments; add test for key reveal too far in the future * Reduce the CUSTODY_PERIOD_TO_RANDAO_PADDING again * Fix max_proposer_slot_reward * Fix types * Move test * Fix RandaoKeyReveal tests * Move all RANDAO key reveal to phase 1 * Factor out signature checking * Some fixes * Update specs/core/1_custody-game.md Co-Authored-By: dankrad * Addressing Vitalik's suggestions: Separate RANDAO and Custody key reveals; append the cleanup of RANDAO reveals instead of adding a new function * Remove remnants of verify_custody_key * RandaoKeyReveal -> EarlyDerivedSecretReveal * Make penalty proportional to number of secrets already exposed * Update specs/core/1_custody-game.md Co-Authored-By: dankrad * Update specs/core/1_custody-game.md Co-Authored-By: dankrad * Update specs/core/1_custody-game.md Co-Authored-By: dankrad --- specs/core/1_custody-game.md | 315 +++++++++++++----- .../phase1_test_process_randao_key_reveal.py | 109 ++++++ test_libs/pyspec/tests/helpers_phase1.py | 50 +++ 3 files changed, 397 insertions(+), 77 deletions(-) create mode 100644 test_libs/pyspec/tests/block_processing_phase1/phase1_test_process_randao_key_reveal.py create mode 100644 test_libs/pyspec/tests/helpers_phase1.py diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index 0f1555e62..d56526611 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -14,6 +14,7 @@ - [Misc](#misc) - [Time parameters](#time-parameters) - [Max operations per block](#max-operations-per-block) + - [Reward and penalty quotients](#reward-and-penalty-quotients) - [Signature domains](#signature-domains) - [Data structures](#data-structures) - [Custody objects](#custody-objects) @@ -22,7 +23,9 @@ - [`CustodyChunkChallengeRecord`](#custodychunkchallengerecord) - [`CustodyBitChallengeRecord`](#custodybitchallengerecord) - [`CustodyResponse`](#custodyresponse) + - [New Beacon operations](#new-beacon-operations) - [`CustodyKeyReveal`](#custodykeyreveal) + - [`EarlyDerivedSecretReveal`](#earlyderivedsecretreveal) - [Phase 0 container updates](#phase-0-container-updates) - [`Validator`](#validator) - [`BeaconState`](#beaconstate) @@ -32,17 +35,19 @@ - [`empty`](#empty) - [`get_crosslink_chunk_count`](#get_crosslink_chunk_count) - [`get_custody_chunk_bit`](#get_custody_chunk_bit) + - [`get_randao_epoch_for_custody_period`](#get_randao_epoch_for_custody_period) + - [`get_validators_custody_reveal_period`](#get_validators_custody_reveal_period) - [`get_chunk_bits_root`](#get_chunk_bits_root) - - [`epoch_to_custody_period`](#epoch_to_custody_period) - [`replace_empty_or_append`](#replace_empty_or_append) - - [`verify_custody_key`](#verify_custody_key) - [Per-block processing](#per-block-processing) - [Operations](#operations) - - [Custody reveals](#custody-reveals) + - [Custody key reveals](#custody-key-reveals) + - [Early derived secret reveals](#early-derived-secret-reveals) - [Chunk challenges](#chunk-challenges) - [Bit challenges](#bit-challenges) - [Custody responses](#custody-responses) - [Per-epoch processing](#per-epoch-processing) + - [Handling of custody-related deadlines](#handling-of-custody-related-deadlines) @@ -80,24 +85,32 @@ This document details the beacon chain additions and changes in Phase 1 of Ether | Name | Value | Unit | Duration | | - | - | :-: | :-: | | `MAX_CHUNK_CHALLENGE_DELAY` | `2**11` (= 2,048) | epochs | ~9 days | -| `EPOCHS_PER_CUSTODY_PERIOD` | `2**11` (= 2,048) | epochs | ~9 days | | `CUSTODY_RESPONSE_DEADLINE` | `2**14` (= 16,384) | epochs | ~73 days | +| `RANDAO_PENALTY_EPOCHS` | `2**1` (= 2) | epochs | 12.8 minutes | +| `EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS` | `2**14` | epochs | ~73 days | +| `EPOCHS_PER_CUSTODY_PERIOD` | `2**11` (= 2,048) | epochs | ~9 days | +| `CUSTODY_PERIOD_TO_RANDAO_PADDING` | `2**11` (= 2,048) | epochs | ~9 days | +| `MAX_REVEAL_LATENESS_DECREMENT` | `2**7` (= 128) | epochs | ~14 hours | ### Max operations per block | Name | Value | | - | - | | `MAX_CUSTODY_KEY_REVEALS` | `2**4` (= 16) | +| `MAX_EARLY_DERIVED_SECRET_REVEALS` | `1` | | `MAX_CUSTODY_CHUNK_CHALLENGES` | `2**2` (= 4) | | `MAX_CUSTODY_BIT_CHALLENGES` | `2**2` (= 4) | | `MAX_CUSTODY_RESPONSES` | `2**5` (= 32) | +### Reward and penalty quotients + +| `EARLY_DERIVED_SECRET_REVEAL_SLOT_REWARD_MULTIPLE` | `2**1` (= 2) | + ### Signature domains | Name | Value | | - | - | -| `DOMAIN_CUSTODY_KEY_REVEAL` | `6` | -| `DOMAIN_CUSTODY_BIT_CHALLENGE` | `7` | +| `DOMAIN_CUSTODY_BIT_CHALLENGE` | `6` | ## Data structures @@ -168,15 +181,35 @@ This document details the beacon chain additions and changes in Phase 1 of Ether } ``` +### New Beacon operations + #### `CustodyKeyReveal` ```python { - 'revealer_index': ValidatorIndex, - 'period': 'uint64', - 'key': BLSSignature, - 'masker_index': ValidatorIndex, - 'mask': Hash, + # Index of the validator whose key is being revealed + 'revealer_index': 'uint64', + # Reveal (masked signature) + 'reveal': 'bytes96', +} +``` + +#### `EarlyDerivedSecretReveal` + +Represents an early (punishable) reveal of one of the derived secrets, where derived secrets are RANDAO reveals and custody reveals (both are part of the same domain). + +```python +{ + # Index of the validator whose key is being revealed + 'revealed_index': 'uint64', + # RANDAO epoch of the key that is being revealed + 'epoch': 'uint64', + # Reveal (masked signature) + 'reveal': 'bytes96', + # Index of the validator who revealed (whistleblower) + 'masker_index': 'uint64', + # Mask used to hide the actual reveal signature (prevent reveal from being stolen) + 'mask': 'bytes32', } ``` @@ -187,7 +220,10 @@ Add the following fields to the end of the specified container objects. Fields w #### `Validator` ```python - 'custody_reveal_index': 'uint64', + # next_custody_reveal_period is initialised to the custody period + # (of the particular validator) in which the validator is activated + # = get_validators_custody_reveal_period(...) + 'next_custody_reveal_period': 'uint64', 'max_reveal_lateness': 'uint64', ``` @@ -197,15 +233,20 @@ Add the following fields to the end of the specified container objects. Fields w 'custody_chunk_challenge_records': [CustodyChunkChallengeRecord], 'custody_bit_challenge_records': [CustodyBitChallengeRecord], 'custody_challenge_index': 'uint64', + + # Future derived secrets already exposed; contains the indices of the exposed validator + # at RANDAO reveal period % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS + 'exposed_derived_secrets': [['uint64'], EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS], ``` #### `BeaconBlockBody` ```python - 'custody_key_reveals': [CustodyKeyReveal], 'custody_chunk_challenges': [CustodyChunkChallenge], 'custody_bit_challenges': [CustodyBitChallenge], 'custody_responses': [CustodyResponse], + 'custody_key_reveals': [CustodyKeyReveal], + 'early_derived_secret_reveals': [EarlyDerivedSecretReveal], ``` ## Helpers @@ -248,15 +289,33 @@ def get_chunk_bits_root(chunk_bitfield: Bitfield) -> Bytes32: return hash(aggregated_bits) ``` -### `epoch_to_custody_period` +### `get_randao_epoch_for_custody_period` ```python -def epoch_to_custody_period(epoch: Epoch) -> int: - return epoch // EPOCHS_PER_CUSTODY_PERIOD +def get_randao_epoch_for_custody_period(period: int, validator_index: ValidatorIndex) -> Epoch: + next_period_start = (period + 1) * EPOCHS_PER_CUSTODY_PERIOD - validator_index % EPOCHS_PER_CUSTODY_PERIOD + return next_period_start + CUSTODY_PERIOD_TO_RANDAO_PADDING +``` + +### `get_validators_custody_reveal_period` + + ```python +def get_validators_custody_reveal_period(state: BeaconState, + validator_index: ValidatorIndex, + epoch: Epoch=None) -> int: + ''' + This function returns the reveal period for a given validator. + If no epoch is supplied, the current epoch is assumed. + Note: This function implicitly requires that validators are not removed from the + validator set in fewer than EPOCHS_PER_CUSTODY_PERIOD epochs + ''' + epoch = get_current_epoch(state) if epoch is None else epoch + return (epoch + validator_index % EPOCHS_PER_CUSTODY_PERIOD) // EPOCHS_PER_CUSTODY_PERIOD ``` ### `replace_empty_or_append` + ```python def replace_empty_or_append(list: List[Any], new_element: Any) -> int: for i in range(len(list)): @@ -267,68 +326,131 @@ def replace_empty_or_append(list: List[Any], new_element: Any) -> int: return len(list) - 1 ``` -### `verify_custody_key` - -```python -def verify_custody_key(state: BeaconState, reveal: CustodyKeyReveal) -> bool: - # Case 1: non-masked non-punitive non-early reveal - pubkeys = [state.validator_registry[reveal.revealer_index].pubkey] - message_hashes = [hash_tree_root(reveal.period)] - - # Case 2: masked punitive early reveal - # Masking prevents proposer stealing the whistleblower reward - # Secure under the aggregate extraction infeasibility assumption - # See pages 11-12 of https://crypto.stanford.edu/~dabo/pubs/papers/aggreg.pdf - if reveal.mask != ZERO_HASH: - pubkeys.append(state.validator_registry[reveal.masker_index].pubkey) - message_hashes.append(reveal.mask) - - return bls_verify_multiple( - pubkeys=pubkeys, - message_hashes=message_hashes, - signature=reveal.key, - domain=get_domain( - fork=state.fork, - epoch=reveal.period * EPOCHS_PER_CUSTODY_PERIOD, - domain_type=DOMAIN_CUSTODY_KEY_REVEAL, - ), - ) -``` - ## Per-block processing ### Operations Add the following operations to the per-block processing, in order the given below and after all other operations in phase 0. -#### Custody reveals +#### Custody key reveals Verify that `len(block.body.custody_key_reveals) <= MAX_CUSTODY_KEY_REVEALS`. For each `reveal` in `block.body.custody_key_reveals`, run the following function: ```python -def process_custody_reveal(state: BeaconState, +def process_custody_key_reveal(state: BeaconState, reveal: CustodyKeyReveal) -> None: - assert verify_custody_key(state, reveal) + + """ + Process ``CustodyKeyReveal`` operation. + Note that this function mutates ``state``. + """ + revealer = state.validator_registry[reveal.revealer_index] - current_custody_period = epoch_to_custody_period(get_current_epoch(state)) + epoch_to_sign = get_randao_epoch_for_custody_period(revealer.next_custody_reveal_period, reveal.revealed_index) - # Case 1: non-masked non-punitive non-early reveal - if reveal.mask == ZERO_HASH: - assert reveal.period == epoch_to_custody_period(revealer.activation_epoch) + revealer.custody_reveal_index - # Revealer is active or exited - assert is_active_validator(revealer, get_current_epoch(state)) or revealer.exit_epoch > get_current_epoch(state) - revealer.custody_reveal_index += 1 - revealer.max_reveal_lateness = max(revealer.max_reveal_lateness, current_custody_period - reveal.period) - proposer_index = get_beacon_proposer_index(state) - increase_balance(state, proposer_index, base_reward(state, index) // MINOR_REWARD_QUOTIENT) + assert revealer.next_custody_reveal_period < get_validators_custody_reveal_period(state, reveal.revealed_index) - # Case 2: masked punitive early reveal + # Revealed validator is active or exited, but not withdrawn + assert is_slashable_validator(revealer, get_current_epoch(state)) + + # Verify signature + assert bls_verify( + pubkey=revealer.pubkey, + message_hash=hash_tree_root(epoch_to_sign), + signature=reveal.reveal, + domain=get_domain( + state=state, + domain_type=DOMAIN_RANDAO, + message_epoch=epoch_to_sign, + ), + ) + + # Decrement max reveal lateness if response is timely + if revealer.next_custody_reveal_period == get_validators_custody_reveal_period(state, reveal.revealer_index) - 2: + revealer.max_reveal_lateness -= MAX_REVEAL_LATENESS_DECREMENT + revealer.max_reveal_lateness = max(revealed_validator.max_reveal_lateness, get_validators_custody_reveal_period(state, reveal.revealed_index) - revealer.next_custody_reveal_period) + + # Process reveal + revealer.next_custody_reveal_period += 1 + + # Reward Block Preposer + proposer_index = get_beacon_proposer_index(state) + increase_balance(state, proposer_index, base_reward(state, index) // MINOR_REWARD_QUOTIENT) +``` + +##### Early derived secret reveals + +Verify that `len(block.body.early_derived_secret_reveals) <= MAX_EARLY_DERIVED_SECRET_REVEALS`. + +For each `reveal` in `block.body.early_derived_secret_reveals`, run the following function: + +```python +def process_early_derived_secret_reveal(state: BeaconState, + reveal: EarlyDerivedSecretReveal) -> None: + """ + Process ``EarlyDerivedSecretReveal`` operation. + Note that this function mutates ``state``. + """ + + revealed_validator = state.validator_registry[reveal.revealed_index] + masker = state.validator_registry[reveal.masker_index] + + assert reveal.epoch >= get_current_epoch(state) + RANDAO_PENALTY_EPOCHS + assert reveal.epoch < get_current_epoch(state) + EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS + assert revealed_validator.slashed is False + assert reveal.revealed_index not in state.exposed_derived_secrets[reveal.epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS] + + # Verify signature correctness + masker = state.validator_registry[reveal.masker_index] + pubkeys = [revealed_validator.pubkey, masker.pubkey] + message_hashes = [ + hash_tree_root(reveal.epoch), + reveal.mask, + ] + + assert bls_verify_multiple( + pubkeys=pubkeys, + message_hashes=message_hashes, + signature=reveal.reveal, + domain=get_domain( + state=state, + domain_type=DOMAIN_RANDAO, + message_epoch=reveal.epoch, + ), + ) + + if reveal.epoch >= get_current_epoch(state) + CUSTODY_PERIOD_TO_RANDAO_PADDING: + # Full slashing when the secret was revealed so early it may be a valid custody + # round key + slash_validator(state, reveal.revealed_index, reveal.masker_index) else: - assert reveal.period > current_custody_period - assert revealer.slashed is False - slash_validator(state, reveal.revealer_index, reveal.masker_index) + # Only a small penalty proportional to proposer slot reward for RANDAO reveal + # that does not interfere with the custody period + # The penalty is proportional to the max proposer reward + + # Calculate penalty + max_proposer_slot_reward = ( + get_base_reward(state, reveal.revealed_index) * + SLOTS_PER_EPOCH // + len(get_active_validator_indices(state, get_current_epoch(state))) // + PROPOSER_REWARD_QUOTIENT + ) + penalty = max_proposer_slot_reward * EARLY_DERIVED_SECRET_REVEAL_SLOT_REWARD_MULTIPLE * (len(state.exposed_derived_secrets[reveal.epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS]) + 1) + + # Apply penalty + proposer_index = get_beacon_proposer_index(state) + whistleblower_index = reveal.masker_index + whistleblowing_reward = penalty // WHISTLEBLOWING_REWARD_QUOTIENT + proposer_reward = whistleblowing_reward // PROPOSER_REWARD_QUOTIENT + increase_balance(state, proposer_index, proposer_reward) + increase_balance(state, whistleblower_index, whistleblowing_reward - proposer_reward) + decrease_balance(state, reveal.revealed_index, penalty) + + # Mark this derived secret as exposed so validator cannot be punished repeatedly + state.exposed_derived_secrets[reveal.epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS].append(reveal.revealed_index) + ``` #### Chunk challenges @@ -384,6 +506,7 @@ For each `challenge` in `block.body.custody_bit_challenges`, run the following f ```python def process_bit_challenge(state: BeaconState, challenge: CustodyBitChallenge) -> None: + # Verify challenge signature challenger = state.validator_registry[challenge.challenger_index] assert bls_verify( @@ -392,29 +515,43 @@ def process_bit_challenge(state: BeaconState, signature=challenge.signature, domain=get_domain(state, get_current_epoch(state), DOMAIN_CUSTODY_BIT_CHALLENGE), ) - # Verify the challenger is not slashed - assert challenger.slashed is False + assert is_slashable_validator(challenger, get_current_epoch(state)) + # Verify the attestation assert verify_indexed_attestation(state, convert_to_indexed(state, challenge.attestation)) # Verify the attestation is eligible for challenging responder = state.validator_registry[challenge.responder_index] - min_challengeable_epoch = responder.exit_epoch - EPOCHS_PER_CUSTODY_PERIOD * (1 + responder.max_reveal_lateness) - assert min_challengeable_epoch <= slot_to_epoch(challenge.attestation.data.slot) + assert (slot_to_epoch(challenge.attestation.data.slot) + responder.max_reveal_lateness <= + get_validators_custody_reveal_period(state, challenge.responder_index)) + # Verify the responder participated in the attestation attesters = get_attesting_indices(state, attestation.data, attestation.aggregation_bitfield) assert challenge.responder_index in attesters + # A validator can be the challenger or responder for at most one challenge at a time for record in state.custody_bit_challenge_records: assert record.challenger_index != challenge.challenger_index assert record.responder_index != challenge.responder_index - # Verify the responder key - assert verify_custody_key(state, CustodyKeyReveal( - revealer_index=challenge.responder_index, - period=epoch_to_custody_period(slot_to_epoch(attestation.data.slot)), - key=challenge.responder_key, - masker_index=0, - mask=ZERO_HASH, - )) + + # Verify the responder is a valid custody key + epoch_to_sign = get_randao_epoch_for_custody_period( + get_validators_custody_reveal_period( + state=state, + index=challenge.responder_index, + epoch=slot_to_epoch(attestation.data.slot), + challenge.responder_index + ) + assert bls_verify( + pubkey=responder.pubkey, + message_hash=hash_tree_root(epoch_to_sign), + signature=challenge.responder_key, + domain=get_domain( + state=state, + domain_type=DOMAIN_RANDAO, + message_epoch=epoch_to_sign, + ), + ) + # Verify the chunk count chunk_count = get_custody_chunk_count(challenge.attestation) assert verify_bitfield(challenge.chunk_bits, chunk_count) @@ -426,7 +563,7 @@ def process_bit_challenge(state: BeaconState, challenge_index=state.custody_challenge_index, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, - deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE + deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, crosslink_data_root=challenge.attestation.data.crosslink_data_root, chunk_count=chunk_count, chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), @@ -434,6 +571,7 @@ def process_bit_challenge(state: BeaconState, ) replace_empty_or_append(state.custody_bit_challenge_records, new_record) state.custody_challenge_index += 1 + # Postpone responder withdrawability responder.withdrawable_epoch = FAR_FUTURE_EPOCH ``` @@ -515,7 +653,20 @@ def process_bit_challenge_response(state: BeaconState, ## Per-epoch processing -Run `process_challenge_deadlines(state)` immediately after `process_ejections(state)`: +### Handling of custody-related deadlines + + Run `process_reveal_deadlines(state)` immediately after `process_ejections(state)`: + + ```python +def process_reveal_deadlines(state: BeaconState) -> None: + for index, validator in enumerate(state.validator_registry): + if (validator.latest_custody_reveal_period + + (CUSTODY_RESPONSE_DEADLINE // EPOCHS_PER_CUSTODY_PERIOD) < + get_validators_custody_reveal_period(state, index)): + slash_validator(state, index) +``` + +Run `process_challenge_deadlines(state)` immediately after `process_reveal_deadlines(state)`: ```python def process_challenge_deadlines(state: BeaconState) -> None: @@ -532,16 +683,26 @@ def process_challenge_deadlines(state: BeaconState) -> None: records[records.index(challenge)] = CustodyBitChallengeRecord() ``` +Append this to `process_final_updates(state)`: + +```python + # Clean up exposed RANDAO key reveals + state.exposed_derived_secrets[current_epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS] = [] +``` + In `process_penalties_and_exits`, change the definition of `eligible` to the following (note that it is not a pure function because `state` is declared in the surrounding scope): ```python -def eligible(index): +def eligible(state: BeaconState, index: ValidatorIndex) -> bool: validator = state.validator_registry[index] # Cannot exit if there are still open chunk challenges if len([record for record in state.custody_chunk_challenge_records if record.responder_index == index]) > 0: return False + # Cannot exit if there are still open bit challenges + if len([record for record in state.custody_bit_challenge_records if record.responder_index == index]) > 0: + return False # Cannot exit if you have not revealed all of your custody keys - elif epoch_to_custody_period(revealer.activation_epoch) + validator.custody_reveal_index <= epoch_to_custody_period(validator.exit_epoch): + elif validator.next_custody_reveal_period <= get_validators_custody_reveal_period(state, index, validator.exit_epoch): return False # Cannot exit if you already have elif validator.withdrawable_epoch < FAR_FUTURE_EPOCH: diff --git a/test_libs/pyspec/tests/block_processing_phase1/phase1_test_process_randao_key_reveal.py b/test_libs/pyspec/tests/block_processing_phase1/phase1_test_process_randao_key_reveal.py new file mode 100644 index 000000000..0be8ab4a9 --- /dev/null +++ b/test_libs/pyspec/tests/block_processing_phase1/phase1_test_process_randao_key_reveal.py @@ -0,0 +1,109 @@ +from copy import deepcopy +import pytest + +import eth2spec.phase1.spec as spec +from eth2spec.phase1.spec import ( + get_current_epoch, + process_randao_key_reveal, + RANDAO_PENALTY_EPOCHS, + CUSTODY_PERIOD_TO_RANDAO_PADDING, + RANDAO_PENALTY_MAX_FUTURE_EPOCHS, +) +from tests.helpers_phase1 import ( + get_valid_randao_key_reveal, +) + +mark entire file as 'randao_key_reveals' +pytestmark = pytest.mark.randao_key_reveals + + +def run_randao_key_reveal_processing(state, randao_key_reveal, valid=True): + """ + Run ``process_randao_key_reveal`` returning the pre and post state. + If ``valid == False``, run expecting ``AssertionError`` + """ + post_state = deepcopy(state) + + if not valid: + with pytest.raises(AssertionError): + process_randao_key_reveal(post_state, randao_key_reveal) + return state, None + + process_randao_key_reveal(post_state, randao_key_reveal) + + slashed_validator = post_state.validator_registry[randao_key_reveal.revealed_index] + + if randao_key_reveal.epoch >= get_current_epoch(state) + CUSTODY_PERIOD_TO_RANDAO_PADDING: + assert slashed_validator.slashed + assert slashed_validator.exit_epoch < spec.FAR_FUTURE_EPOCH + assert slashed_validator.withdrawable_epoch < spec.FAR_FUTURE_EPOCH + # lost whistleblower reward + # FIXME: Currently broken because get_base_reward in genesis epoch is 0 + assert ( + post_state.balances[randao_key_reveal.revealed_index] < + state.balances[randao_key_reveal.revealed_index] + ) + + return state, post_state + + +def test_success(state): + randao_key_reveal = get_valid_randao_key_reveal(state) + + pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal) + + return pre_state, randao_key_reveal, post_state + + +def test_reveal_from_current_epoch(state): + randao_key_reveal = get_valid_randao_key_reveal(state, get_current_epoch(state)) + + pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal, False) + + return pre_state, randao_key_reveal, post_state + +# Not currently possible as we are testing at epoch 0 +# +#def test_reveal_from_past_epoch(state): +# randao_key_reveal = get_valid_randao_key_reveal(state, get_current_epoch(state) - 1) +# +# pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal, False) +# +# return pre_state, randao_key_reveal, post_state + +def test_reveal_with_custody_padding(state): + randao_key_reveal = get_valid_randao_key_reveal(state, get_current_epoch(state) + CUSTODY_PERIOD_TO_RANDAO_PADDING) + pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal, True) + + return pre_state, randao_key_reveal, post_state + +def test_reveal_with_custody_padding_minus_one(state): + randao_key_reveal = get_valid_randao_key_reveal(state, get_current_epoch(state) + CUSTODY_PERIOD_TO_RANDAO_PADDING - 1) + pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal, True) + + return pre_state, randao_key_reveal, post_state + +def test_double_reveal(state): + + randao_key_reveal1 = get_valid_randao_key_reveal(state, get_current_epoch(state) + RANDAO_PENALTY_EPOCHS + 1) + pre_state, intermediate_state = run_randao_key_reveal_processing(state, randao_key_reveal1) + + randao_key_reveal2 = get_valid_randao_key_reveal(intermediate_state, get_current_epoch(pre_state) + RANDAO_PENALTY_EPOCHS + 1) + intermediate_state_, post_state = run_randao_key_reveal_processing(intermediate_state, randao_key_reveal2, False) + + return pre_state, [randao_key_reveal1, randao_key_reveal2], post_state + +def test_revealer_is_slashed(state): + randao_key_reveal = get_valid_randao_key_reveal(state, get_current_epoch(state)) + state.validator_registry[randao_key_reveal.revealed_index].slashed = True + + pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal, False) + + return pre_state, randao_key_reveal, post_state + +def test_far_future_epoch(state): + randao_key_reveal = get_valid_randao_key_reveal(state, get_current_epoch(state) + RANDAO_PENALTY_MAX_FUTURE_EPOCHS) + + pre_state, post_state = run_randao_key_reveal_processing(state, randao_key_reveal, False) + + return pre_state, randao_key_reveal, post_state diff --git a/test_libs/pyspec/tests/helpers_phase1.py b/test_libs/pyspec/tests/helpers_phase1.py new file mode 100644 index 000000000..aba93e159 --- /dev/null +++ b/test_libs/pyspec/tests/helpers_phase1.py @@ -0,0 +1,50 @@ +from py_ecc import bls + +import eth2spec.phase1.spec as spec +from eth2spec.phase0.spec import ( + # constants + ZERO_HASH, + CUSTODY_PERIOD_TO_RANDAO_PADDING, + # SSZ + RandaoKeyReveal, + # functions + get_active_validator_indices, + get_current_epoch, + get_domain, + hash_tree_root, +) + +def get_valid_randao_key_reveal(state, epoch=None): + current_epoch = get_current_epoch(state) + revealed_index = get_active_validator_indices(state, current_epoch)[-1] + masker_index = get_active_validator_indices(state, current_epoch)[0] + + if epoch is None: + epoch = current_epoch + CUSTODY_PERIOD_TO_RANDAO_PADDING + + reveal = bls.sign( + message_hash=hash_tree_root(epoch), + privkey=pubkey_to_privkey[state.validator_registry[revealed_index].pubkey], + domain=get_domain( + state=state, + domain_type=spec.DOMAIN_RANDAO, + message_epoch=epoch, + ), + ) + mask = bls.sign( + message_hash=hash_tree_root(epoch), + privkey=pubkey_to_privkey[state.validator_registry[masker_index].pubkey], + domain=get_domain( + state=state, + domain_type=spec.DOMAIN_RANDAO, + message_epoch=epoch, + ), + ) + + return RandaoKeyReveal( + revealed_index=revealed_index, + epoch=epoch, + reveal=reveal, + masker_index=masker_index, + mask=mask, + ) From 6318a21ca6d721ed3bc58d6903a34a908e4a5b80 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Fri, 3 May 2019 10:34:16 -0600 Subject: [PATCH 24/45] fix minor typo in attestation rewards --- specs/core/0_beacon-chain.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 56f11bc7f..e56fd976c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1427,7 +1427,8 @@ def get_attestation_deltas(state: BeaconState) -> Tuple[List[Gwei], List[Gwei]]: # Proposer and inclusion delay micro-rewards for index in get_unslashed_attesting_indices(state, matching_source_attestations): attestation = min([ - a for a in attestations if index in get_attesting_indices(state, a.data, a.aggregation_bitfield) + a for a in matching_source_attestations + if index in get_attesting_indices(state, a.data, a.aggregation_bitfield) ], key=lambda a: a.inclusion_delay) rewards[attestation.proposer_index] += get_base_reward(state, index) // PROPOSER_REWARD_QUOTIENT rewards[index] += get_base_reward(state, index) * MIN_ATTESTATION_INCLUSION_DELAY // attestation.inclusion_delay From d5d2f7835aed7d4e2400ebce09af54fa70613173 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 3 May 2019 22:51:59 -0500 Subject: [PATCH 25/45] Rearranging one >= statement to make it more non-underflow-friendly --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 420ea3ef9..36ac92796 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1799,7 +1799,7 @@ def process_voluntary_exit(state: BeaconState, exit: VoluntaryExit) -> None: # Exits must specify an epoch when they become valid; they are not valid before then assert get_current_epoch(state) >= exit.epoch # Verify the validator has been active long enough - assert get_current_epoch(state) - validator.activation_epoch >= PERSISTENT_COMMITTEE_PERIOD + assert get_current_epoch(state) >= validator.activation_epoch + PERSISTENT_COMMITTEE_PERIOD # Verify signature domain = get_domain(state, DOMAIN_VOLUNTARY_EXIT, exit.epoch) assert bls_verify(validator.pubkey, signing_root(exit), exit.signature, domain) From a6e825d46056a4a92473e52f1ea0bcb19210e67b Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 5 May 2019 12:04:34 +0100 Subject: [PATCH 26/45] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 58 +++++++++++++++--------------------- 1 file changed, 24 insertions(+), 34 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36ac92796..addaa865e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -279,6 +279,8 @@ The types are defined topologically to aid in facilitating an executable version ```python { + # Shard number + 'shard': 'uint64', # Epoch number 'epoch': 'uint64', # Root of the previous crosslink @@ -315,9 +317,7 @@ The types are defined topologically to aid in facilitating an executable version 'target_root': 'bytes32', # Crosslink vote - 'shard': 'uint64', - 'previous_crosslink_root': 'bytes32', - 'crosslink_data_root': 'bytes32', + 'crosslink': Crosslink, } ``` @@ -765,7 +765,7 @@ def get_epoch_start_shard(state: BeaconState, epoch: Epoch) -> Shard: def get_attestation_slot(state: BeaconState, attestation: Attestation) -> Slot: epoch = attestation.data.target_epoch committee_count = get_epoch_committee_count(state, epoch) - offset = (attestation.data.shard + SHARD_COUNT - get_epoch_start_shard(state, epoch)) % SHARD_COUNT + offset = (attestation.data.crosslink.shard + SHARD_COUNT - get_epoch_start_shard(state, epoch)) % SHARD_COUNT return get_epoch_start_slot(epoch) + offset // (committee_count // SLOTS_PER_EPOCH) ``` @@ -927,7 +927,7 @@ def get_attesting_indices(state: BeaconState, """ Return the sorted attesting indices corresponding to ``attestation_data`` and ``bitfield``. """ - committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.shard) + committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.crosslink.shard) assert verify_bitfield(bitfield, len(committee)) return sorted([index for i, index in enumerate(committee) if get_bitfield_bit(bitfield, i) == 0b1]) ``` @@ -1296,28 +1296,18 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat return get_total_balance(state, get_unslashed_attesting_indices(state, attestations)) ``` -```python -def get_crosslink_from_attestation_data(state: BeaconState, data: AttestationData) -> Crosslink: - return Crosslink( - epoch=min(data.target_epoch, state.current_crosslinks[data.shard].epoch + MAX_CROSSLINK_EPOCHS), - previous_crosslink_root=data.previous_crosslink_root, - crosslink_data_root=data.crosslink_data_root, - ) -``` - ```python def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: - shard_attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.shard == shard] - shard_crosslinks = [get_crosslink_from_attestation_data(state, a.data) for a in shard_attestations] + shard_attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] candidate_crosslinks = [ - c for c in shard_crosslinks + c for c in [a.data.crosslink for a in shard_attestations] if hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)) ] if len(candidate_crosslinks) == 0: return Crosslink(), [] def get_attestations_for(crosslink: Crosslink) -> List[PendingAttestation]: - return [a for a in shard_attestations if get_crosslink_from_attestation_data(state, a.data) == crosslink] + return [a for a in shard_attestations if a.data.crosslink == crosslink] # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) winning_crosslink = max(candidate_crosslinks, key=lambda crosslink: ( get_attesting_balance(state, get_attestations_for(crosslink)), crosslink.crosslink_data_root @@ -1705,30 +1695,30 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: attestation_slot = get_attestation_slot(state, attestation) assert attestation_slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot <= attestation_slot + SLOTS_PER_EPOCH - # Check target epoch, source epoch, source root, and source crosslink data = attestation.data - assert (data.target_epoch, data.source_epoch, data.source_root, data.previous_crosslink_root) in { - (get_current_epoch(state), state.current_justified_epoch, state.current_justified_root, hash_tree_root(state.current_crosslinks[data.shard])), - (get_previous_epoch(state), state.previous_justified_epoch, state.previous_justified_root, hash_tree_root(state.previous_crosslinks[data.shard])), - } - - # Check crosslink data root - assert data.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] - - # Check signature and bitfields - assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) - - # Cache pending attestation pending_attestation = PendingAttestation( data=data, aggregation_bitfield=attestation.aggregation_bitfield, inclusion_delay=state.slot - attestation_slot, proposer_index=get_beacon_proposer_index(state), ) - if data.target_epoch == get_current_epoch(state): - state.current_epoch_attestations.append(pending_attestation) - else: + + assert data.target_epoch in (get_previous_epoch(state), get_current_epoch(state)) + if data.target_epoch == get_previous_epoch(state): + ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) + previous_crosslink = state.previous_crosslinks[data.crosslink.shard] state.previous_epoch_attestations.append(pending_attestation) + if data.target_epoch == get_current_epoch(state): + ffg_data = (state.current_justified_epoch, state.current_justified_root, get_current_epoch(state)) + previous_crosslink = state.current_crosslinks[data.crosslink.shard] + state.current_epoch_attestations.append(pending_attestation) + + # Check FFG data, crosslink data, and signature + assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) + assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_CROSSLINK_EPOCHS) + assert data.crosslink.previous_crosslink_root == hash_tree_root(previous_crosslink) + assert data.crosslink.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] + assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) ``` ##### Deposits From 5fb32fd19b780d172e8738d887fd0675df7ec3ef Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Sun, 5 May 2019 12:10:39 +0100 Subject: [PATCH 27/45] Fix tests --- configs/constant_presets/mainnet.yaml | 4 ++-- configs/constant_presets/minimal.yaml | 4 ++-- specs/core/0_beacon-chain.md | 6 +++--- specs/core/1_custody-game.md | 16 ++++++++-------- specs/core/1_shard-data-chains.md | 16 ++++++++-------- specs/validator/0_beacon-chain-validator.md | 4 ++-- .../block_processing/test_process_attestation.py | 4 ++-- .../epoch_processing/test_process_crosslinks.py | 12 ++++++------ test_libs/pyspec/tests/helpers.py | 15 ++++++++++----- 9 files changed, 43 insertions(+), 38 deletions(-) diff --git a/configs/constant_presets/mainnet.yaml b/configs/constant_presets/mainnet.yaml index 72d0fdc8f..6ac3f422f 100644 --- a/configs/constant_presets/mainnet.yaml +++ b/configs/constant_presets/mainnet.yaml @@ -72,7 +72,7 @@ MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256 # 2**11 (= 2,048) epochs 9 days PERSISTENT_COMMITTEE_PERIOD: 2048 # 2**6 (= 64) epochs ~7 hours -MAX_CROSSLINK_EPOCHS: 64 +MAX_EPOCHS_PER_CROSSLINK: 64 # 2**2 (= 4) epochs 25.6 minutes MIN_EPOCHS_TO_INACTIVITY_PENALTY: 4 @@ -124,4 +124,4 @@ DOMAIN_RANDAO: 1 DOMAIN_ATTESTATION: 2 DOMAIN_DEPOSIT: 3 DOMAIN_VOLUNTARY_EXIT: 4 -DOMAIN_TRANSFER: 5 \ No newline at end of file +DOMAIN_TRANSFER: 5 diff --git a/configs/constant_presets/minimal.yaml b/configs/constant_presets/minimal.yaml index 0a6cab687..caae4623b 100644 --- a/configs/constant_presets/minimal.yaml +++ b/configs/constant_presets/minimal.yaml @@ -71,7 +71,7 @@ MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256 # 2**11 (= 2,048) epochs 9 days PERSISTENT_COMMITTEE_PERIOD: 2048 # 2**6 (= 64) epochs ~7 hours -MAX_CROSSLINK_EPOCHS: 64 +MAX_EPOCHS_PER_CROSSLINK: 64 # 2**2 (= 4) epochs 25.6 minutes MIN_EPOCHS_TO_INACTIVITY_PENALTY: 4 @@ -123,4 +123,4 @@ DOMAIN_RANDAO: 1 DOMAIN_ATTESTATION: 2 DOMAIN_DEPOSIT: 3 DOMAIN_VOLUNTARY_EXIT: 4 -DOMAIN_TRANSFER: 5 \ No newline at end of file +DOMAIN_TRANSFER: 5 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index addaa865e..156a2523d 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -206,10 +206,10 @@ These configurations are updated for releases, but may be out of sync during `de | `SLOTS_PER_HISTORICAL_ROOT` | `2**13` (= 8,192) | slots | ~13 hours | | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY` | `2**8` (= 256) | epochs | ~27 hours | | `PERSISTENT_COMMITTEE_PERIOD` | `2**11` (= 2,048) | epochs | 9 days | -| `MAX_CROSSLINK_EPOCHS` | `2**6` (= 64) | epochs | ~7 hours | +| `MAX_EPOCHS_PER_CROSSLINK` | `2**6` (= 64) | epochs | ~7 hours | | `MIN_EPOCHS_TO_INACTIVITY_PENALTY` | `2**2` (= 4) | epochs | 25.6 minutes | -* `MAX_CROSSLINK_EPOCHS` should be a small constant times `SHARD_COUNT // SLOTS_PER_EPOCH` +* `MAX_EPOCHS_PER_CROSSLINK` should be a small constant times `SHARD_COUNT // SLOTS_PER_EPOCH` ### State list lengths @@ -1715,7 +1715,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) - assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_CROSSLINK_EPOCHS) + assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) assert data.crosslink.previous_crosslink_root == hash_tree_root(previous_crosslink) assert data.crosslink.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index d56526611..f386ee76e 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -265,7 +265,7 @@ The `empty` function accepts and SSZ type as input and returns an object of that def get_custody_chunk_count(attestation: Attestation) -> int: crosslink_start_epoch = attestation.data.latest_crosslink.epoch crosslink_end_epoch = slot_to_epoch(attestation.data.slot) - crosslink_crosslink_length = min(MAX_CROSSLINK_EPOCHS, end_epoch - start_epoch) + crosslink_crosslink_length = min(MAX_EPOCHS_PER_CROSSLINK, end_epoch - start_epoch) chunks_per_epoch = 2 * BYTES_PER_SHARD_BLOCK * SLOTS_PER_EPOCH // BYTES_PER_CUSTODY_CHUNK return crosslink_crosslink_length * chunks_per_epoch ``` @@ -426,10 +426,10 @@ def process_early_derived_secret_reveal(state: BeaconState, # round key slash_validator(state, reveal.revealed_index, reveal.masker_index) else: - # Only a small penalty proportional to proposer slot reward for RANDAO reveal + # Only a small penalty proportional to proposer slot reward for RANDAO reveal # that does not interfere with the custody period - # The penalty is proportional to the max proposer reward - + # The penalty is proportional to the max proposer reward + # Calculate penalty max_proposer_slot_reward = ( get_base_reward(state, reveal.revealed_index) * @@ -448,7 +448,7 @@ def process_early_derived_secret_reveal(state: BeaconState, increase_balance(state, whistleblower_index, whistleblowing_reward - proposer_reward) decrease_balance(state, reveal.revealed_index, penalty) - # Mark this derived secret as exposed so validator cannot be punished repeatedly + # Mark this derived secret as exposed so validator cannot be punished repeatedly state.exposed_derived_secrets[reveal.epoch % EARLY_DERIVED_SECRET_PENALTY_MAX_FUTURE_EPOCHS].append(reveal.revealed_index) ``` @@ -474,7 +474,7 @@ def process_chunk_challenge(state: BeaconState, # Verify the challenge is not a duplicate for record in state.custody_chunk_challenge_records: assert ( - record.crosslink_data_root != challenge.attestation.data.crosslink_data_root or + record.crosslink_data_root != challenge.attestation.data.crosslink.crosslink_data_root or record.chunk_index != challenge.chunk_index ) # Verify depth @@ -486,7 +486,7 @@ def process_chunk_challenge(state: BeaconState, challenger_index=get_beacon_proposer_index(state), responder_index=challenge.responder_index deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink_data_root, + crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, depth=depth, chunk_index=challenge.chunk_index, ) @@ -564,7 +564,7 @@ def process_bit_challenge(state: BeaconState, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink_data_root, + crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, chunk_count=chunk_count, chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), responder_key=challenge.responder_key, diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 33ef8632b..673a64a4a 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -215,7 +215,7 @@ def get_shard_header(block: ShardBlock) -> ShardBlockHeader: def verify_shard_attestation_signature(state: BeaconState, attestation: ShardAttestation) -> None: data = attestation.data - persistent_committee = get_persistent_committee(state, data.shard, data.slot) + persistent_committee = get_persistent_committee(state, data.crosslink.shard, data.slot) assert verify_bitfield(attestation.aggregation_bitfield, len(persistent_committee)) pubkeys = [] for i, index in enumerate(persistent_committee): @@ -225,7 +225,7 @@ def verify_shard_attestation_signature(state: BeaconState, pubkeys.append(validator.pubkey) assert bls_verify( pubkey=bls_aggregate_pubkeys(pubkeys), - message_hash=data.shard_block_root, + message_hash=data.crosslink.shard_block_root, signature=attestation.aggregate_signature, domain=get_domain(state, slot_to_epoch(data.slot), DOMAIN_SHARD_ATTESTER) ) @@ -312,7 +312,7 @@ def is_valid_shard_block(beacon_blocks: List[BeaconBlock], for _, attestation in enumerate(block.attestations): assert max(GENESIS_SHARD_SLOT, block.slot - SLOTS_PER_EPOCH) <= attestation.data.slot assert attestation.data.slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY - assert attestation.data.shard == block.shard + assert attestation.data.crosslink.shard == block.shard verify_shard_attestation_signature(beacon_state, attestation) # Check signature @@ -343,11 +343,11 @@ def is_valid_shard_attestation(valid_shard_blocks: List[ShardBlock], # Check shard block shard_block = next( block for block in valid_shard_blocks if - signing_root(block) == candidate.attestation.data.shard_block_root + signing_root(block) == candidate.attestation.data.crosslink.shard_block_root , None) assert shard_block != None assert shard_block.slot == attestation.data.slot - assert shard_block.shard == attestation.data.shard + assert shard_block.shard == attestation.data.crosslink.shard # Check signature verify_shard_attestation_signature(beacon_state, attestation) @@ -382,18 +382,18 @@ def is_valid_beacon_attestation(shard: Shard, else: previous_attestation = next( attestation for attestation in valid_attestations if - attestation.data.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root + attestation.data.crosslink.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root , None) assert previous_attestation != None assert candidate.data.previous_attestation.epoch < slot_to_epoch(candidate.data.slot) # Check crosslink data root start_epoch = state.latest_crosslinks[shard].epoch - end_epoch = min(slot_to_epoch(candidate.data.slot) - CROSSLINK_LOOKBACK, start_epoch + MAX_CROSSLINK_EPOCHS) + end_epoch = min(slot_to_epoch(candidate.data.slot) - CROSSLINK_LOOKBACK, start_epoch + MAX_EPOCHS_PER_CROSSLINK) blocks = [] for slot in range(start_epoch * SLOTS_PER_EPOCH, end_epoch * SLOTS_PER_EPOCH): blocks.append(shard_blocks[slot]) - assert candidate.data.crosslink_data_root == compute_crosslink_data_root(blocks) + assert candidate.data.crosslink.crosslink_data_root == compute_crosslink_data_root(blocks) return True ``` diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index ca7f0eb3b..61f80f68e 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -271,7 +271,7 @@ _Note:_ This can be looked up in the state using: ##### Shard -Set `attestation_data.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. ##### Previous crosslink root @@ -279,7 +279,7 @@ Set `attestation_data.previous_crosslink_root = hash_tree_root(head_state.curren ##### Crosslink data root -Set `attestation_data.crosslink_data_root = ZERO_HASH`. +Set `attestation_data.crosslink.crosslink_data_root = ZERO_HASH`. _Note:_ This is a stub for phase 0. diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index bcf71376c..24cafc275 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -113,7 +113,7 @@ def test_non_zero_crosslink_data_root(state): attestation = get_valid_attestation(state) state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY - attestation.data.crosslink_data_root = b'\x42' * 32 + attestation.data.crosslink.crosslink_data_root = b'\x42' * 32 pre_state, post_state = run_attestation_processing(state, attestation, False) @@ -126,7 +126,7 @@ def test_bad_previous_crosslink(state): for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY): next_slot(state) - state.current_crosslinks[attestation.data.shard].epoch += 10 + state.current_crosslinks[attestation.data.crosslink.shard].epoch += 10 pre_state, post_state = run_attestation_processing(state, attestation, False) diff --git a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py index d6765e3a7..29e7347b1 100644 --- a/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py +++ b/test_libs/pyspec/tests/epoch_processing/test_process_crosslinks.py @@ -64,7 +64,7 @@ def test_single_crosslink_update_from_current_epoch(state): pre_state, post_state = run_process_crosslinks(state) - shard = attestation.data.shard + shard = attestation.data.crosslink.shard assert post_state.previous_crosslinks[shard] != post_state.current_crosslinks[shard] assert pre_state.current_crosslinks[shard] != post_state.current_crosslinks[shard] @@ -84,11 +84,11 @@ def test_single_crosslink_update_from_previous_epoch(state): pre_state, post_state = run_process_crosslinks(state) crosslink_deltas = get_crosslink_deltas(state) - shard = attestation.data.shard + shard = attestation.data.crosslink.shard assert post_state.previous_crosslinks[shard] != post_state.current_crosslinks[shard] assert pre_state.current_crosslinks[shard] != post_state.current_crosslinks[shard] # ensure rewarded - for index in get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.shard): + for index in get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.crosslink.shard): assert crosslink_deltas[0][index] > 0 assert crosslink_deltas[1][index] == 0 @@ -108,7 +108,7 @@ def test_double_late_crosslink(state): for slot in range(spec.SLOTS_PER_EPOCH): attestation_2 = get_valid_attestation(state) - if attestation_2.data.shard == attestation_1.data.shard: + if attestation_2.data.crosslink.shard == attestation_1.data.crosslink.shard: break next_slot(state) fill_aggregate_attestation(state, attestation_2) @@ -124,12 +124,12 @@ def test_double_late_crosslink(state): pre_state, post_state = run_process_crosslinks(state) crosslink_deltas = get_crosslink_deltas(state) - shard = attestation_2.data.shard + shard = attestation_2.data.crosslink.shard # ensure that the current crosslinks were not updated by the second attestation assert post_state.previous_crosslinks[shard] == post_state.current_crosslinks[shard] # ensure no reward, only penalties for the failed crosslink - for index in get_crosslink_committee(state, attestation_2.data.target_epoch, attestation_2.data.shard): + for index in get_crosslink_committee(state, attestation_2.data.target_epoch, attestation_2.data.crosslink.shard): assert crosslink_deltas[0][index] == 0 assert crosslink_deltas[1][index] > 0 diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index 3b9b6904d..1ea299453 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -10,6 +10,7 @@ from eth2spec.utils.minimal_ssz import signing_root from eth2spec.phase0.spec import ( # constants ZERO_HASH, + MAX_EPOCHS_PER_CROSSLINK, # SSZ Attestation, AttestationData, @@ -17,6 +18,7 @@ from eth2spec.phase0.spec import ( AttesterSlashing, BeaconBlock, BeaconBlockHeader, + Crosslink, Deposit, DepositData, Eth1Data, @@ -174,14 +176,17 @@ def build_attestation_data(state, slot, shard): crosslinks = state.current_crosslinks if slot_to_epoch(slot) == get_current_epoch(state) else state.previous_crosslinks return AttestationData( - shard=shard, beacon_block_root=block_root, source_epoch=justified_epoch, source_root=justified_block_root, target_epoch=slot_to_epoch(slot), target_root=epoch_boundary_root, - crosslink_data_root=spec.ZERO_HASH, - previous_crosslink_root=hash_tree_root(crosslinks[shard]), + crosslink=Crosslink( + shard=shard, + epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), + crosslink_data_root=spec.ZERO_HASH, + previous_crosslink_root=hash_tree_root(crosslinks[shard]), + ), ) @@ -288,7 +293,7 @@ def get_valid_attestation(state, slot=None): attestation_data = build_attestation_data(state, slot, shard) - crosslink_committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.shard) + crosslink_committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.crosslink.shard) committee_size = len(crosslink_committee) bitfield_length = (committee_size + 7) // 8 @@ -381,7 +386,7 @@ def get_attestation_signature(state, attestation_data, privkey, custody_bit=0b0) def fill_aggregate_attestation(state, attestation): - crosslink_committee = get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.shard) + crosslink_committee = get_crosslink_committee(state, attestation.data.target_epoch, attestation.data.crosslink.shard) for i in range(len(crosslink_committee)): attestation.aggregation_bitfield = set_bitfield_bit(attestation.aggregation_bitfield, i) From bf20031755804aa98ea7f0b597f698efac3e01a2 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 5 May 2019 17:15:05 +0100 Subject: [PATCH 28/45] Cosmetic genesis cleanups --- specs/core/0_beacon-chain.md | 43 +++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36ac92796..df871dc60 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -93,7 +93,10 @@ - [Routines for updating validator status](#routines-for-updating-validator-status) - [`initiate_validator_exit`](#initiate_validator_exit) - [`slash_validator`](#slash_validator) - - [On genesis](#on-genesis) + - [Genesis](#genesis) + - [`Eth2Genesis`](#eth2genesis) + - [Genesis state](#genesis-state) + - [Genesis block](#genesis-block) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - [State caching](#state-caching) - [Per-epoch processing](#per-epoch-processing) @@ -1171,30 +1174,29 @@ def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whistlebl decrease_balance(state, slashed_index, whistleblowing_reward) ``` -## On genesis +## Genesis -When enough full deposits have been made to the deposit contract, an `Eth2Genesis` log is emitted. Construct a corresponding `genesis_state` and `genesis_block` as follows: +### `Eth2Genesis` -* Let `genesis_validator_deposits` be the list of deposits, ordered chronologically, up to and including the deposit that triggered the `Eth2Genesis` log. -* Let `genesis_time` be the timestamp specified in the `Eth2Genesis` log. -* Let `genesis_eth1_data` be the `Eth1Data` object where: - * `genesis_eth1_data.deposit_root` is the `deposit_root` contained in the `Eth2Genesis` log. - * `genesis_eth1_data.deposit_count` is the `deposit_count` contained in the `Eth2Genesis` log. - * `genesis_eth1_data.block_hash` is the hash of the Ethereum 1.0 block that emitted the `Eth2Genesis` log. -* Let `genesis_state = get_genesis_beacon_state(genesis_validator_deposits, genesis_time, genesis_eth1_data)`. -* Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. +When enough deposits of size `MAX_EFFECTIVE_BALANCE` have been made to the deposit contract an `Eth2Genesis` log is emitted triggering the genesis of the beacon chain. Let: + +* `eth2genesis` be the object corresponding to `Eth2Genesis` +* `genesis_eth1_data` be object of type `Eth1Data` where + * `genesis_eth1_data.deposit_root = eth2genesis.deposit_root` + * `genesis_eth1_data.deposit_count = eth2genesis.deposit_count` + * `genesis_eth1_data.block_hash` is the hash of the Ethereum 1.0 block that emitted the `Eth2Genesis` log +* `genesis_deposits` be the object of type `List[Deposit]` with deposits ordered chronologically up to and including the deposit that triggered the `Eth2Genesis` log + +### Genesis state + +Let `genesis_state = get_genesis_beacon_state(eth2genesis.genesis_time, genesis_eth1_data, genesis_deposits)`. ```python -def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], - genesis_time: int, - genesis_eth1_data: Eth1Data) -> BeaconState: - """ - Get the genesis ``BeaconState``. - """ +def get_genesis_beacon_state(genesis_time: int, eth1_data: Eth1Data, deposits: List[Deposit]) -> BeaconState: state = BeaconState(genesis_time=genesis_time, latest_eth1_data=genesis_eth1_data) # Process genesis deposits - for deposit in genesis_validator_deposits: + for deposit in deposits: process_deposit(state, deposit) # Process genesis activations @@ -1203,6 +1205,7 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], validator.activation_eligibility_epoch = GENESIS_EPOCH validator.activation_epoch = GENESIS_EPOCH + # Populate latest_active_index_roots genesis_active_index_root = hash_tree_root(get_active_validator_indices(state, GENESIS_EPOCH)) for index in range(LATEST_ACTIVE_INDEX_ROOTS_LENGTH): state.latest_active_index_roots[index] = genesis_active_index_root @@ -1210,6 +1213,10 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], return state ``` +### Genesis block + +Let `genesis_block = BeaconBlock(state_root=hash_tree_root(genesis_state))`. + ## Beacon chain state transition function We now define the state transition function. At a high level, the state transition is made up of four parts: From a3bbf20bf732b77eaddabad5a82f89b0c9f612c2 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 5 May 2019 17:17:18 +0100 Subject: [PATCH 29/45] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index df871dc60..4d3263a40 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1189,10 +1189,10 @@ When enough deposits of size `MAX_EFFECTIVE_BALANCE` have been made to the depos ### Genesis state -Let `genesis_state = get_genesis_beacon_state(eth2genesis.genesis_time, genesis_eth1_data, genesis_deposits)`. +Let `genesis_state = get_genesis_beacon_state(genesis_deposits, eth2genesis.genesis_time, genesis_eth1_data)`. ```python -def get_genesis_beacon_state(genesis_time: int, eth1_data: Eth1Data, deposits: List[Deposit]) -> BeaconState: +def get_genesis_beacon_state(deposits: List[Deposit], genesis_time: int, eth1_data: Eth1Data) -> BeaconState: state = BeaconState(genesis_time=genesis_time, latest_eth1_data=genesis_eth1_data) # Process genesis deposits From 6aff277b1a344a2ecf024ef0fa4e6a126d0cdecf Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 5 May 2019 17:20:25 +0100 Subject: [PATCH 30/45] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4d3263a40..381ef867d 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1192,7 +1192,7 @@ When enough deposits of size `MAX_EFFECTIVE_BALANCE` have been made to the depos Let `genesis_state = get_genesis_beacon_state(genesis_deposits, eth2genesis.genesis_time, genesis_eth1_data)`. ```python -def get_genesis_beacon_state(deposits: List[Deposit], genesis_time: int, eth1_data: Eth1Data) -> BeaconState: +def get_genesis_beacon_state(deposits: List[Deposit], genesis_time: int, genesis_eth1_data: Eth1Data) -> BeaconState: state = BeaconState(genesis_time=genesis_time, latest_eth1_data=genesis_eth1_data) # Process genesis deposits From 2c3bbac8034d9ad7eeff35d95618e032d7a04056 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 5 May 2019 19:30:55 +0100 Subject: [PATCH 31/45] Fix #1050 --- specs/core/0_beacon-chain.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36ac92796..a60245732 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -63,7 +63,7 @@ - [`get_epoch_committee_count`](#get_epoch_committee_count) - [`get_shard_delta`](#get_shard_delta) - [`get_epoch_start_shard`](#get_epoch_start_shard) - - [`get_attestation_slot`](#get_attestation_slot) + - [`get_attestation_data_slot`](#get_attestation_data_slot) - [`get_block_root_at_slot`](#get_block_root_at_slot) - [`get_block_root`](#get_block_root) - [`get_randao_mix`](#get_randao_mix) @@ -759,14 +759,13 @@ def get_epoch_start_shard(state: BeaconState, epoch: Epoch) -> Shard: return shard ``` -### `get_attestation_slot` +### `get_attestation_data_slot` ```python -def get_attestation_slot(state: BeaconState, attestation: Attestation) -> Slot: - epoch = attestation.data.target_epoch - committee_count = get_epoch_committee_count(state, epoch) - offset = (attestation.data.shard + SHARD_COUNT - get_epoch_start_shard(state, epoch)) % SHARD_COUNT - return get_epoch_start_slot(epoch) + offset // (committee_count // SLOTS_PER_EPOCH) +def get_attestation_data_slot(state: BeaconState, data: AttestationData) -> Slot: + committee_count = get_epoch_committee_count(state, data.target_epoch) + offset = (data.shard + SHARD_COUNT - get_epoch_start_shard(state, data.target_epoch)) % SHARD_COUNT + return get_epoch_start_slot(data.target_epoch) + offset // (committee_count // SLOTS_PER_EPOCH) ``` ### `get_block_root_at_slot` @@ -1279,7 +1278,7 @@ def get_matching_target_attestations(state: BeaconState, epoch: Epoch) -> List[P def get_matching_head_attestations(state: BeaconState, epoch: Epoch) -> List[PendingAttestation]: return [ a for a in get_matching_source_attestations(state, epoch) - if a.data.beacon_block_root == get_block_root_at_slot(state, get_attestation_slot(state, a)) + if a.data.beacon_block_root == get_block_root_at_slot(state, get_attestation_data_slot(state, a.data)) ] ``` @@ -1702,11 +1701,11 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: """ Process ``Attestation`` operation. """ - attestation_slot = get_attestation_slot(state, attestation) + data = attestation.data + attestation_slot = get_attestation_data_slot(state, data) assert attestation_slot + MIN_ATTESTATION_INCLUSION_DELAY <= state.slot <= attestation_slot + SLOTS_PER_EPOCH # Check target epoch, source epoch, source root, and source crosslink - data = attestation.data assert (data.target_epoch, data.source_epoch, data.source_root, data.previous_crosslink_root) in { (get_current_epoch(state), state.current_justified_epoch, state.current_justified_root, hash_tree_root(state.current_crosslinks[data.shard])), (get_previous_epoch(state), state.previous_justified_epoch, state.previous_justified_root, hash_tree_root(state.previous_crosslinks[data.shard])), From 7d0a6191edade473053cec49ce560ea253a220fd Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 6 May 2019 09:09:37 -0600 Subject: [PATCH 32/45] simplify justificatio/finality notes at genesis --- specs/core/0_fork-choice.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index cb11ea860..8ec0a604c 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -106,4 +106,4 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock) ### Justification and finality at genesis -Clients may choose to refer to the justification and finality data in a given `BeaconState` to determine the finalized/justified head. During the early epochs right after genesis, the justification and finality data are not stored in the `BeaconState`. In their place are "empty" values; for example, the 32-byte zero value as the `finalized_root`. Clients wishing to compute the fork choice in these early epochs should work around this fact of the `BeaconState` to recognize that the genesis epoch and root of the genesis block are _both_ the finalized and justified heads until updated via the state transition function defined in [Phase 0 -- The Beacon Chain](./0_beacon-chain.md). Solutions will be language-specific but one possibility is to treat the zero-value hash as an exceptional case that references the genesis block/epoch. +During genesis, justification and finality root fields within the `BeaconState` reference `ZERO_HASH` rather than a known block. `ZERO_HASH` in `previous_justified_root`, `current_justified_root`, and `finalized_root` should be considered as an alias to the root of the genesis block. From d1c96c1e0d3b97ac6b436cbaa070e4a39f6b5876 Mon Sep 17 00:00:00 2001 From: JSON <49416440+JSON@users.noreply.github.com> Date: Mon, 6 May 2019 10:30:32 -0500 Subject: [PATCH 33/45] Doc standardization (#1039) --- README.md | 8 +-- configs/constant_presets/README.md | 6 +-- configs/fork_timelines/README.md | 7 +-- specs/bls_signature.md | 2 +- specs/core/0_beacon-chain.md | 52 +++++++++--------- specs/core/0_deposit-contract.md | 14 ++--- specs/core/0_fork-choice.md | 18 +++---- specs/core/1_custody-game.md | 36 ++++++------- specs/core/1_shard-data-chains.md | 14 ++--- specs/light_client/merkle_proofs.md | 23 ++++---- specs/light_client/sync_protocol.md | 6 +-- specs/networking/messaging.md | 23 ++++---- specs/networking/node-identification.md | 13 +++-- specs/networking/rpc-interface.md | 59 ++++++++++---------- specs/simple-serialize.md | 37 +++++++------ specs/test_formats/README.md | 60 ++++++++++++--------- specs/test_formats/bls/README.md | 4 +- specs/test_formats/shuffling/README.md | 12 ++--- specs/test_formats/ssz_generic/README.md | 8 +-- specs/test_formats/ssz_static/README.md | 2 +- specs/test_formats/ssz_static/core.md | 8 +-- specs/validator/0_beacon-chain-validator.md | 60 ++++++++++----------- test_generators/README.md | 14 ++--- test_libs/pyspec/README.md | 16 +++--- 24 files changed, 259 insertions(+), 243 deletions(-) diff --git a/README.md b/README.md index 27ff7fcb1..4c4808059 100644 --- a/README.md +++ b/README.md @@ -2,14 +2,14 @@ [![Join the chat at https://gitter.im/ethereum/sharding](https://badges.gitter.im/ethereum/sharding.svg)](https://gitter.im/ethereum/sharding?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -To learn more about sharding and eth2.0/Serenity, see the [sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ) and the [research compendium](https://notes.ethereum.org/s/H1PGqDhpm). +To learn more about sharding and Ethereum 2.0 (Serenity), see the [sharding FAQ](https://github.com/ethereum/wiki/wiki/Sharding-FAQ) and the [research compendium](https://notes.ethereum.org/s/H1PGqDhpm). -This repo hosts the current eth2.0 specifications. Discussions about design rationale and proposed changes can be brought up and discussed as issues. Solidified, agreed upon changes to spec can be made through pull requests. +This repository hosts the current Eth 2.0 specifications. Discussions about design rationale and proposed changes can be brought up and discussed as issues. Solidified, agreed-upon changes to the spec can be made through pull requests. ## Specs -Core specifications for eth2.0 client validation can be found in [specs/core](specs/core). These are divided into phases. Each subsequent phase depends upon the prior. The current phases specified are: +Core specifications for Eth 2.0 client validation can be found in [specs/core](specs/core). These are divided into phases. Each subsequent phase depends upon the prior. The current phases specified are: ### Phase 0 * [The Beacon Chain](specs/core/0_beacon-chain.md) @@ -30,7 +30,7 @@ Core specifications for eth2.0 client validation can be found in [specs/core](sp * [Light client syncing protocol](specs/light_client/sync_protocol.md) -### Design goals +## Design goals The following are the broad design goals for Ethereum 2.0: * to minimize complexity, even at the cost of some losses in efficiency diff --git a/configs/constant_presets/README.md b/configs/constant_presets/README.md index 45148862e..61c9a3a63 100644 --- a/configs/constant_presets/README.md +++ b/configs/constant_presets/README.md @@ -10,11 +10,11 @@ Later-fork constants can be ignored, e.g. ignore phase1 constants as a client th Each preset is a key-value mapping. **Key**: an `UPPER_SNAKE_CASE` (a.k.a. "macro case") formatted string, name of the constant. -**Value**: can be any of: + +**Value** can be either: - an unsigned integer number, can be up to 64 bits (incl.) - a hexadecimal string, prefixed with `0x` Presets may contain comments to describe the values. -See `mainnet.yaml` for a complete example. - +See [`mainnet.yaml`](./mainnet.yaml) for a complete example. diff --git a/configs/fork_timelines/README.md b/configs/fork_timelines/README.md index c93b415f5..da7445767 100644 --- a/configs/fork_timelines/README.md +++ b/configs/fork_timelines/README.md @@ -3,16 +3,17 @@ This directory contains a set of fork timelines used for testing, testnets, and mainnet. A timeline file contains all the forks known for its target. -Later forks can be ignored, e.g. ignore fork `phase1` as a client that only supports phase 0 currently. +Later forks can be ignored, e.g. ignore fork `phase1` as a client that only supports Phase 0 currently. ## Format Each preset is a key-value mapping. **Key**: an `lower_snake_case` (a.k.a. "python case") formatted string, name of the fork. -**Value**: an unsigned integer number, epoch number of activation of the fork + +**Value**: an unsigned integer number, epoch number of activation of the fork. Timelines may contain comments to describe the values. -See `mainnet.yaml` for a complete example. +See [`mainnet.yaml`](./mainnet.yaml) for a complete example. diff --git a/specs/bls_signature.md b/specs/bls_signature.md index 18e2d8c9a..d119c4499 100644 --- a/specs/bls_signature.md +++ b/specs/bls_signature.md @@ -118,7 +118,7 @@ Let `bls_aggregate_signatures(signatures: List[Bytes96]) -> Bytes96` return `sig ## Signature verification -In the following `e` is the pairing function and `g` is the G1 generator with the following coordinates (see [here](https://github.com/zkcrypto/pairing/tree/master/src/bls12_381#g1)): +In the following, `e` is the pairing function and `g` is the G1 generator with the following coordinates (see [here](https://github.com/zkcrypto/pairing/tree/master/src/bls12_381#g1)): ```python g_x = 3685416753713387016781088315183077757961620795782546409894578378688607592378376318836054947676345821548104185464507 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 36ac92796..7e135d0d0 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1,6 +1,6 @@ # Ethereum 2.0 Phase 0 -- The Beacon Chain -**NOTICE**: This document is a work in progress for researchers and implementers. +**Notice**: This document is a work-in-progress for researchers and implementers. ## Table of contents @@ -45,7 +45,7 @@ - [`BeaconBlock`](#beaconblock) - [Beacon state](#beacon-state) - [`BeaconState`](#beaconstate) - - [Custom Types](#custom-types) + - [Custom types](#custom-types) - [Helper functions](#helper-functions) - [`xor`](#xor) - [`hash`](#hash) @@ -134,25 +134,25 @@ Code snippets appearing in `this style` are to be interpreted as Python code. ## Terminology -* **Validator** - a registered participant in the beacon chain. You can become one by sending ether into the Ethereum 1.0 deposit contract. -* **Active validator** - an active participant in the Ethereum 2.0 consensus invited to, among other things, propose and attest to blocks and vote for crosslinks. -* **Committee** - a (pseudo-) randomly sampled subset of [active validators](#dfn-active-validator). When a committee is referred to collectively, as in "this committee attests to X", this is assumed to mean "some subset of that committee that contains enough [validators](#dfn-validator) that the protocol recognizes it as representing the committee". -* **Proposer** - the [validator](#dfn-validator) that creates a beacon chain block. -* **Attester** - a [validator](#dfn-validator) that is part of a committee that needs to sign off on a beacon chain block while simultaneously creating a link (crosslink) to a recent shard block on a particular shard chain. -* **Beacon chain** - the central PoS chain that is the base of the sharding system. -* **Shard chain** - one of the chains on which user transactions take place and account data is stored. -* **Block root** - a 32-byte Merkle root of a beacon chain block or shard chain block. Previously called "block hash". -* **Crosslink** - a set of signatures from a committee attesting to a block in a shard chain that can be included into the beacon chain. Crosslinks are the main means by which the beacon chain "learns about" the updated state of shard chains. -* **Slot** - a period during which one proposer has the ability to create a beacon chain block and some attesters have the ability to make attestations. -* **Epoch** - an aligned span of slots during which all [validators](#dfn-validator) get exactly one chance to make an attestation. -* **Finalized**, **justified** - see the [Casper FFG paper](https://arxiv.org/abs/1710.09437). -* **Withdrawal period** - the number of slots between a [validator](#dfn-validator) exit and the [validator](#dfn-validator) balance being withdrawable. -* **Genesis time** - the Unix time of the genesis beacon chain block at slot 0. +* **Validator**—a registered participant in the beacon chain. You can become one by sending ether into the Ethereum 1.0 deposit contract. +* **Active validator**—an active participant in the Ethereum 2.0 consensus invited to, among other things, propose and attest to blocks and vote for crosslinks. +* **Committee**—a (pseudo-) randomly sampled subset of [active validators](#dfn-active-validator). When a committee is referred to collectively, as in "this committee attests to X", this is assumed to mean "some subset of that committee that contains enough [validators](#dfn-validator) that the protocol recognizes it as representing the committee". +* **Proposer**—the [validator](#dfn-validator) that creates a beacon chain block. +* **Attester**—a [validator](#dfn-validator) that is part of a committee that needs to sign off on a beacon chain block while simultaneously creating a link (crosslink) to a recent shard block on a particular shard chain. +* **Beacon chain**—the central PoS chain that is the base of the sharding system. +* **Shard chain**—one of the chains on which user transactions take place and account data is stored. +* **Block root**—a 32-byte Merkle root of a beacon chain block or shard chain block. Previously called "block hash". +* **Crosslink**—a set of signatures from a committee attesting to a block in a shard chain that can be included into the beacon chain. Crosslinks are the main means by which the beacon chain "learns about" the updated state of shard chains. +* **Slot**—a period during which one proposer has the ability to create a beacon chain block and some attesters have the ability to make attestations. +* **Epoch**—an aligned span of slots during which all [validators](#dfn-validator) get exactly one chance to make an attestation. +* **Finalized**, **justified**—see the [Casper FFG paper](https://arxiv.org/abs/1710.09437). +* **Withdrawal period**—the number of slots between a [validator](#dfn-validator) exit and the [validator](#dfn-validator) balance being withdrawable. +* **Genesis time**—the Unix time of the genesis beacon chain block at slot 0. ## Constants -Note: the default mainnet values for the constants are included here for spec-design purposes. -The different configurations for mainnet, testnets, and yaml-based testing can be found in the `configs/constant_presets/` directory. +*Note*: The default mainnet values for the constants are included here for spec-design purposes. +The different configurations for mainnet, testnets, and YAML-based testing can be found in the `configs/constant_presets/` directory. These configurations are updated for releases, but may be out of sync during `dev` changes. ### Misc @@ -165,7 +165,7 @@ These configurations are updated for releases, but may be out of sync during `de | `MIN_PER_EPOCH_CHURN_LIMIT` | `2**2` (= 4) | | `CHURN_LIMIT_QUOTIENT` | `2**16` (= 65,536) | | `BASE_REWARDS_PER_EPOCH` | `5` | -| `SHUFFLE_ROUND_COUNT` | 90 | +| `SHUFFLE_ROUND_COUNT` | `90` | * For the safety of crosslinks `TARGET_COMMITTEE_SIZE` exceeds [the recommended minimum committee size of 111](https://vitalik.ca/files/Ithaca201807_Sharding.pdf); with sufficient active validators (at least `SLOTS_PER_EPOCH * TARGET_COMMITTEE_SIZE`), the shuffling algorithm ensures committee sizes of at least `TARGET_COMMITTEE_SIZE`. (Unbiasable randomness with a Verifiable Delay Function (VDF) will improve committee robustness and lower the safe minimum committee size.) @@ -229,7 +229,7 @@ These configurations are updated for releases, but may be out of sync during `de | `INACTIVITY_PENALTY_QUOTIENT` | `2**25` (= 33,554,432) | | `MIN_SLASHING_PENALTY_QUOTIENT` | `2**5` (= 32) | -* **The `BASE_REWARD_QUOTIENT` is NOT final. Once all other protocol details are finalized it will be adjusted, to target a theoretical maximum total issuance of `2**21` ETH per year if `2**27` ETH is validating (and therefore `2**20` per year if `2**25` ETH is validating, etc etc)** +* **The `BASE_REWARD_QUOTIENT` is NOT final. Once all other protocol details are finalized, it will be adjusted to target a theoretical maximum total issuance of `2**21` ETH per year if `2**27` ETH is validating (and therefore `2**20` per year if `2**25` ETH is validating, etc.)** * The `INACTIVITY_PENALTY_QUOTIENT` equals `INVERSE_SQRT_E_DROP_TIME**2` where `INVERSE_SQRT_E_DROP_TIME := 2**12 epochs` (~18 days) is the time it takes the inactivity penalty to reduce the balance of non-participating [validators](#dfn-validator) to about `1/sqrt(e) ~= 60.6%`. Indeed, the balance retained by offline [validators](#dfn-validator) after `n` epochs is about `(1 - 1/INACTIVITY_PENALTY_QUOTIENT)**(n**2/2)` so after `INVERSE_SQRT_E_DROP_TIME` epochs it is roughly `(1 - 1/INACTIVITY_PENALTY_QUOTIENT)**(INACTIVITY_PENALTY_QUOTIENT/2) ~= 1/sqrt(e)`. ### Max operations per block @@ -587,7 +587,7 @@ The types are defined topologically to aid in facilitating an executable version } ``` -## Custom Types +## Custom types We define the following Python custom types for type hinting and readability: @@ -604,7 +604,7 @@ We define the following Python custom types for type hinting and readability: ## Helper functions -Note: The definitions below are for specification purposes and are not necessarily optimal implementations. +*Note*: The definitions below are for specification purposes and are not necessarily optimal implementations. ### `xor` @@ -617,7 +617,7 @@ def xor(bytes1: Bytes32, bytes2: Bytes32) -> Bytes32: The `hash` function is SHA256. -Note: We aim to migrate to a S[T/N]ARK-friendly hash function in a future Ethereum 2.0 deployment phase. +*Note*: We aim to migrate to a S[T/N]ARK-friendly hash function in a future Ethereum 2.0 deployment phase. ### `hash_tree_root` @@ -1121,7 +1121,7 @@ def get_churn_limit(state: BeaconState) -> int: ### Routines for updating validator status -Note: All functions in this section mutate `state`. +*Note*: All functions in this section mutate `state`. #### `initiate_validator_exit` @@ -1227,7 +1227,7 @@ Transition section notes: Beacon blocks that trigger unhandled Python exceptions (e.g. out-of-range list accesses) and failed `assert`s during the state transition are considered invalid. -Note: If there are skipped slots between a block and its parent block, run the steps in the [state-root](#state-caching), [per-epoch](#per-epoch-processing), and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. +*Note*: If there are skipped slots between a block and its parent block, run the steps in the [state-root](#state-caching), [per-epoch](#per-epoch-processing), and [per-slot](#per-slot-processing) sections once for each skipped slot and then once for the slot containing the new block. ### State caching @@ -1634,7 +1634,7 @@ def process_eth1_data(state: BeaconState, block: BeaconBlock) -> None: #### Operations -Note: All functions in this section mutate `state`. +*Note*: All functions in this section mutate `state`. ##### Proposer slashings diff --git a/specs/core/0_deposit-contract.md b/specs/core/0_deposit-contract.md index 6843e407e..917adaecb 100644 --- a/specs/core/0_deposit-contract.md +++ b/specs/core/0_deposit-contract.md @@ -1,6 +1,6 @@ # Ethereum 2.0 Phase 0 -- Deposit Contract -**NOTICE**: This document is a work in progress for researchers and implementers. +**Notice**: This document is a work-in-progress for researchers and implementers. ## Table of contents @@ -24,7 +24,7 @@ ## Introduction -This document represents is the specification for the beacon chain deposit contract, part of Ethereum 2.0 phase 0. +This document represents the specification for the beacon chain deposit contract, part of Ethereum 2.0 Phase 0. ## Constants @@ -40,11 +40,11 @@ This document represents is the specification for the beacon chain deposit contr | - | - | | `DEPOSIT_CONTRACT_ADDRESS` | **TBD** | | `DEPOSIT_CONTRACT_TREE_DEPTH` | `2**5` (= 32) | -| `CHAIN_START_FULL_DEPOSIT_THRESHOLD` | `2**16` (=65,536) | +| `CHAIN_START_FULL_DEPOSIT_THRESHOLD` | `2**16` (= 65,536) | ## Ethereum 1.0 deposit contract -The initial deployment phases of Ethereum 2.0 are implemented without consensus changes to Ethereum 1.0. A deposit contract at address `DEPOSIT_CONTRACT_ADDRESS` is added to Ethereum 1.0 for deposits of ETH to the beacon chain. Validator balances will be withdrawable to the shards in phase 2, i.e. when the EVM2.0 is deployed and the shards have state. +The initial deployment phases of Ethereum 2.0 are implemented without consensus changes to Ethereum 1.0. A deposit contract at address `DEPOSIT_CONTRACT_ADDRESS` is added to Ethereum 1.0 for deposits of ETH to the beacon chain. Validator balances will be withdrawable to the shards in Phase 2 (i.e. when the EVM 2.0 is deployed and the shards have state). ### Arguments @@ -52,7 +52,7 @@ The deposit contract has a `deposit` function which takes the amount in Ethereum #### Withdrawal credentials -One of the `DepositData` fields is `withdrawal_credentials`. It is a commitment to credentials for withdrawals to shards. The first byte of `withdrawal_credentials` is a version number. As of now the only expected format is as follows: +One of the `DepositData` fields is `withdrawal_credentials`. It is a commitment to credentials for withdrawals to shards. The first byte of `withdrawal_credentials` is a version number. As of now, the only expected format is as follows: * `withdrawal_credentials[:1] == BLS_WITHDRAWAL_PREFIX_BYTE` * `withdrawal_credentials[1:] == hash(withdrawal_pubkey)[1:]` where `withdrawal_pubkey` is a BLS pubkey @@ -84,10 +84,10 @@ When `CHAIN_START_FULL_DEPOSIT_THRESHOLD` of full deposits have been made, the d The source for the Vyper contract lives in a [separate repository](https://github.com/ethereum/deposit_contract) at [https://github.com/ethereum/deposit_contract/blob/master/deposit_contract/contracts/validator_registration.v.py](https://github.com/ethereum/deposit_contract/blob/master/deposit_contract/contracts/validator_registration.v.py). -Note: to save ~10x on gas this contract uses a somewhat unintuitive progressive Merkle root calculation algo that requires only O(log(n)) storage. See https://github.com/ethereum/research/blob/master/beacon_chain_impl/progressive_merkle_tree.py for an implementation of the same algo in python tested for correctness. +*Note*: To save ~10x on gas, this contract uses a somewhat unintuitive progressive Merkle root calculation algo that requires only O(log(n)) storage. See https://github.com/ethereum/research/blob/master/beacon_chain_impl/progressive_merkle_tree.py for an implementation of the same algo in Python tested for correctness. For convenience, we provide the interface to the contract here: * `__init__()`: initializes the contract * `get_deposit_root() -> bytes32`: returns the current root of the deposit tree -* `deposit(pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96])`: adds a deposit instance to the deposit tree, incorporating the input arguments and the value transferred in the given call. Note: the amount of value transferred *must* be at least `MIN_DEPOSIT_AMOUNT`. Each of these constants are specified in units of Gwei. +* `deposit(pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96])`: adds a deposit instance to the deposit tree, incorporating the input arguments and the value transferred in the given call. *Note*: The amount of value transferred *must* be at least `MIN_DEPOSIT_AMOUNT`. Each of these constants are specified in units of Gwei. diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index 23534ec4e..549e9e207 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -1,6 +1,6 @@ # Ethereum 2.0 Phase 0 -- Beacon Chain Fork Choice -**NOTICE**: This document is a work in progress for researchers and implementers. +**Notice**: This document is a work-in-progress for researchers and implementers. ## Table of contents @@ -20,7 +20,7 @@ ## Introduction -This document represents is the specification for the beacon chain fork choice rule, part of Ethereum 2.0 phase 0. +This document represents the specification for the beacon chain fork choice rule, part of Ethereum 2.0 Phase 0. ## Prerequisites @@ -42,17 +42,17 @@ Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Cli * An Ethereum 1.0 block pointed to by the `state.latest_eth1_data.block_hash` has been processed and accepted. * The node's Unix time is greater than or equal to `state.genesis_time + block.slot * SECONDS_PER_SLOT`. -Note: Leap seconds mean that slots will occasionally last `SECONDS_PER_SLOT + 1` or `SECONDS_PER_SLOT - 1` seconds, possibly several times a year. +*Note*: Leap seconds mean that slots will occasionally last `SECONDS_PER_SLOT + 1` or `SECONDS_PER_SLOT - 1` seconds, possibly several times a year. -Note: Nodes needs to have a clock that is roughly (i.e. within `SECONDS_PER_SLOT` seconds) synchronized with the other nodes. +*Note*: Nodes needs to have a clock that is roughly (i.e. within `SECONDS_PER_SLOT` seconds) synchronized with the other nodes. ### Beacon chain fork choice rule -The beacon chain fork choice rule is a hybrid that combines justification and finality with Latest Message Driven (LMD) Greediest Heaviest Observed SubTree (GHOST). At any point in time a validator `v` subjectively calculates the beacon chain head as follows. +The beacon chain fork choice rule is a hybrid that combines justification and finality with Latest Message Driven (LMD) Greediest Heaviest Observed SubTree (GHOST). At any point in time, a validator `v` subjectively calculates the beacon chain head as follows. -* Abstractly define `Store` as the type of storage object for the chain data and `store` be the set of attestations and blocks that the validator `v` has observed and verified (in particular, block ancestors must be recursively verified). Attestations not yet included in any chain are still included in `store`. -* Let `finalized_head` be the finalized block with the highest epoch. (A block `B` is finalized if there is a descendant of `B` in `store` the processing of which sets `B` as finalized.) -* Let `justified_head` be the descendant of `finalized_head` with the highest epoch that has been justified for at least 1 epoch. (A block `B` is justified if there is a descendant of `B` in `store` the processing of which sets `B` as justified.) If no such descendant exists set `justified_head` to `finalized_head`. +* Abstractly define `Store` as the type of storage object for the chain data, and let `store` be the set of attestations and blocks that the validator `v` has observed and verified (in particular, block ancestors must be recursively verified). Attestations not yet included in any chain are still included in `store`. +* Let `finalized_head` be the finalized block with the highest epoch. (A block `B` is finalized if there is a descendant of `B` in `store`, the processing of which sets `B` as finalized.) +* Let `justified_head` be the descendant of `finalized_head` with the highest epoch that has been justified for at least 1 epoch. (A block `B` is justified if there is a descendant of `B` in `store` the processing of which sets `B` as justified.) If no such descendant exists, set `justified_head` to `finalized_head`. * Let `get_ancestor(store: Store, block: BeaconBlock, slot: Slot) -> BeaconBlock` be the ancestor of `block` with slot number `slot`. The `get_ancestor` function can be defined recursively as: ```python @@ -70,7 +70,7 @@ def get_ancestor(store: Store, block: BeaconBlock, slot: Slot) -> BeaconBlock: * Let `get_latest_attestation(store: Store, index: ValidatorIndex) -> Attestation` be the attestation with the highest slot number in `store` from the validator with the given `index`. If several such attestations exist, use the one the validator `v` observed first. * Let `get_latest_attestation_target(store: Store, index: ValidatorIndex) -> BeaconBlock` be the target block in the attestation `get_latest_attestation(store, index)`. -* Let `get_children(store: Store, block: BeaconBlock) -> List[BeaconBlock]` returns the child blocks of the given `block`. +* Let `get_children(store: Store, block: BeaconBlock) -> List[BeaconBlock]` return the child blocks of the given `block`. * Let `justified_head_state` be the resulting `BeaconState` object from processing the chain up to the `justified_head`. * The `head` is `lmd_ghost(store, justified_head_state, justified_head)` where the function `lmd_ghost` is defined below. Note that the implementation below is suboptimal; there are implementations that compute the head in time logarithmic in slot count. diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index d56526611..b3ab519fa 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -1,6 +1,6 @@ # Ethereum 2.0 Phase 1 -- Custody Game -**NOTICE**: This spec is a work-in-progress for researchers and implementers. +**Notice**: This document is a work-in-progress for researchers and implementers. ## Table of contents @@ -23,7 +23,7 @@ - [`CustodyChunkChallengeRecord`](#custodychunkchallengerecord) - [`CustodyBitChallengeRecord`](#custodybitchallengerecord) - [`CustodyResponse`](#custodyresponse) - - [New Beacon operations](#new-beacon-operations) + - [New beacon operations](#new-beacon-operations) - [`CustodyKeyReveal`](#custodykeyreveal) - [`EarlyDerivedSecretReveal`](#earlyderivedsecretreveal) - [Phase 0 container updates](#phase-0-container-updates) @@ -53,22 +53,22 @@ ## Introduction -This document details the beacon chain additions and changes in Phase 1 of Ethereum 2.0 to support the shard data custody game, building upon the [phase 0](0_beacon-chain.md) specification. +This document details the beacon chain additions and changes in Phase 1 of Ethereum 2.0 to support the shard data custody game, building upon the [Phase 0](0_beacon-chain.md) specification. ## Terminology -* **Custody game**: -* **Custody period**: -* **Custody chunk**: -* **Custody chunk bit**: -* **Custody chunk challenge**: -* **Custody bit**: -* **Custody bit challenge**: -* **Custody key**: -* **Custody key reveal**: -* **Custody key mask**: -* **Custody response**: -* **Custody response deadline**: +* **Custody game**— +* **Custody period**— +* **Custody chunk**— +* **Custody chunk bit**— +* **Custody chunk challenge**— +* **Custody bit**— +* **Custody bit challenge**— +* **Custody key**— +* **Custody key reveal**— +* **Custody key mask**— +* **Custody response**— +* **Custody response deadline**— ## Constants @@ -181,7 +181,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether } ``` -### New Beacon operations +### New beacon operations #### `CustodyKeyReveal` @@ -220,7 +220,7 @@ Add the following fields to the end of the specified container objects. Fields w #### `Validator` ```python - # next_custody_reveal_period is initialised to the custody period + # next_custody_reveal_period is initialized to the custody period # (of the particular validator) in which the validator is activated # = get_validators_custody_reveal_period(...) 'next_custody_reveal_period': 'uint64', @@ -330,7 +330,7 @@ def replace_empty_or_append(list: List[Any], new_element: Any) -> int: ### Operations -Add the following operations to the per-block processing, in order the given below and after all other operations in phase 0. +Add the following operations to the per-block processing, in the order given below and after all other operations in Phase 0. #### Custody key reveals diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 33ef8632b..14eb51193 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -1,13 +1,13 @@ # Ethereum 2.0 Phase 1 -- Shard Data Chains -**NOTICE**: This document is a work-in-progress for researchers and implementers. +**Notice**: This document is a work-in-progress for researchers and implementers. -## Table of Contents +## Table of contents -- [Ethereum 2.0 Phase 1 -- Shards Data Chains](#ethereum-20-phase-1----shard-data-chains) - - [Table of Contents](#table-of-contents) +- [Ethereum 2.0 Phase 1 -- Shard Data Chains](#ethereum-20-phase-1----shard-data-chains) + - [Table of contents](#table-of-contents) - [Introduction](#introduction) - [Constants](#constants) - [Misc](#misc) @@ -53,8 +53,8 @@ This document describes the shard data layer and the shard fork choice rule in P | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `CROSSLINK_LOOKBACK` | 2**0 (= 1) | epochs | 6.2 minutes | -| `PERSISTENT_COMMITTEE_PERIOD` | 2**11 (= 2,048) | epochs | ~9 days | +| `CROSSLINK_LOOKBACK` | `2**0` (= 1) | epochs | 6.2 minutes | +| `PERSISTENT_COMMITTEE_PERIOD` | `2**11` (= 2,048) | epochs | ~9 days | ### Signature domains @@ -363,7 +363,7 @@ Let: * `shard_blocks` be the `ShardBlock` list such that `shard_blocks[slot]` is the canonical `ShardBlock` for shard `shard` at slot `slot` * `beacon_state` be the canonical `BeaconState` * `valid_attestations` be the list of valid `Attestation`, recursively defined -* `candidate` be a candidate `Attestation` which is valid under phase 0 rules, and for which validity is to be determined under phase 1 rules by running `is_valid_beacon_attestation` +* `candidate` be a candidate `Attestation` which is valid under Phase 0 rules, and for which validity is to be determined under Phase 1 rules by running `is_valid_beacon_attestation` ```python def is_valid_beacon_attestation(shard: Shard, diff --git a/specs/light_client/merkle_proofs.md b/specs/light_client/merkle_proofs.md index b38167bb5..f009d9737 100644 --- a/specs/light_client/merkle_proofs.md +++ b/specs/light_client/merkle_proofs.md @@ -1,16 +1,19 @@ -**NOTICE**: This document is a work-in-progress for researchers and implementers. +# Merkle proof formats -## Table of Contents +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents -- [Table of Contents](#table-of-contents) -- [Constants](#constants) -- [Generalized Merkle tree index](#generalized-merkle-tree-index) -- [SSZ object to index](#ssz-object-to-index) -- [Merkle multiproofs](#merkle-multiproofs) -- [MerklePartial](#merklepartial) - - [`SSZMerklePartial`](#sszmerklepartial) - - [Proofs for execution](#proofs-for-execution) +- [Merkle proof formats](#merkle-proof-formats) + - [Table of contents](#table-of-contents) + - [Constants](#constants) + - [Generalized Merkle tree index](#generalized-merkle-tree-index) + - [SSZ object to index](#ssz-object-to-index) + - [Merkle multiproofs](#merkle-multiproofs) + - [MerklePartial](#merklepartial) + - [`SSZMerklePartial`](#sszmerklepartial) + - [Proofs for execution](#proofs-for-execution) diff --git a/specs/light_client/sync_protocol.md b/specs/light_client/sync_protocol.md index 7cb1f6928..f6e3d2265 100644 --- a/specs/light_client/sync_protocol.md +++ b/specs/light_client/sync_protocol.md @@ -1,13 +1,13 @@ # Beacon Chain Light Client Syncing -__NOTICE__: This document is a work-in-progress for researchers and implementers. One of the design goals of the eth2 beacon chain is light-client friendliness, both to allow low-resource clients (mobile phones, IoT, etc) to maintain access to the blockchain in a reasonably safe way, but also to facilitate the development of "bridges" between the eth2 beacon chain and other chains. +**Notice**: This document is a work-in-progress for researchers and implementers. One of the design goals of the Eth 2.0 beacon chain is light-client friendliness, not only to allow low-resource clients (mobile phones, IoT, etc.) to maintain access to the blockchain in a reasonably safe way, but also to facilitate the development of "bridges" between the Eth 2.0 beacon chain and other chains. -## Table of Contents +## Table of contents - [Beacon Chain Light Client Syncing](#beacon-chain-light-client-syncing) - - [Table of Contents](#table-of-contents) + - [Table of contents](#table-of-contents) - [Preliminaries](#preliminaries) - [Expansions](#expansions) - [`get_active_validator_indices`](#get_active_validator_indices) diff --git a/specs/networking/messaging.md b/specs/networking/messaging.md index b64e1d5d8..d7cb5bb5b 100644 --- a/specs/networking/messaging.md +++ b/specs/networking/messaging.md @@ -1,23 +1,22 @@ -ETH 2.0 Networking Spec - Messaging -=== +# Eth 2.0 Networking Spec - Messaging -# Abstract +## Abstract This specification describes how individual Ethereum 2.0 messages are represented on the wire. -The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL”, NOT", “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119. +The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL”, NOT", “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). -# Motivation +## Motivation -This specification seeks to define a messaging protocol that is flexible enough to be changed easily as the ETH 2.0 specification evolves. +This specification seeks to define a messaging protocol that is flexible enough to be changed easily as the Eth 2.0 specification evolves. Note that while `libp2p` is the chosen networking stack for Ethereum 2.0, as of this writing some clients do not have workable `libp2p` implementations. To allow those clients to communicate, we define a message envelope that includes the body's compression, encoding, and body length. Once `libp2p` is available across all implementations, this message envelope will be removed because `libp2p` will negotiate the values defined in the envelope upfront. -# Specification +## Specification -## Message Structure +### Message structure -An ETH 2.0 message consists of an envelope that defines the message's compression, encoding, and length followed by the body itself. +An Eth 2.0 message consists of an envelope that defines the message's compression, encoding, and length followed by the body itself. Visually, a message looks like this: @@ -35,12 +34,12 @@ Visually, a message looks like this: +--------------------------+ ``` -Clients MUST ignore messages with mal-formed bodies. The compression/encoding nibbles MUST be one of the following values: +Clients MUST ignore messages with malformed bodies. The compression/encoding nibbles MUST be one of the following values: -## Compression Nibble Values +### Compression nibble values - `0x0`: no compression -## Encoding Nibble Values +### Encoding nibble values - `0x1`: SSZ diff --git a/specs/networking/node-identification.md b/specs/networking/node-identification.md index 0f1f9832b..32ec4dfad 100644 --- a/specs/networking/node-identification.md +++ b/specs/networking/node-identification.md @@ -1,13 +1,12 @@ -ETH 2.0 Networking Spec - Node Identification -=== +# Eth 2.0 Networking Spec - Node Identification -# Abstract +## Abstract This specification describes how Ethereum 2.0 nodes identify and address each other on the network. -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). -# Specification +## Specification Clients use Ethereum Node Records (as described in [EIP-778](http://eips.ethereum.org/EIPS/eip-778)) to discover one another. Each ENR includes, among other things, the following keys: @@ -21,11 +20,11 @@ The keys above are enough to construct a [multiaddr](https://github.com/multifor It is RECOMMENDED that clients set their TCP port to the default of `9000`. -## Peer ID Generation +### Peer ID generation The `libp2p` networking stack identifies peers via a "peer ID." Simply put, a node's Peer ID is the SHA2-256 `multihash` of the node's public key struct (serialized in protobuf, refer to the [Peer ID spec](https://github.com/libp2p/specs/pull/100)). `go-libp2p-crypto` contains the canonical implementation of how to hash `secp256k1` keys for use as a peer ID. -# See Also +## See also - [multiaddr](https://github.com/multiformats/multiaddr) - [multihash](https://multiformats.io/multihash/) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index f1da8f7e3..ab7a08392 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -1,19 +1,18 @@ -ETH 2.0 Networking Spec - RPC Interface -=== +# Eth 2.0 Networking Spec - RPC Interface -# Abstract +## Abstract The Ethereum 2.0 networking stack uses two modes of communication: a broadcast protocol that gossips information to interested parties via GossipSub, and an RPC protocol that retrieves information from specific clients. This specification defines the RPC protocol. -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). -# Dependencies +## Dependencies This specification assumes familiarity with the [Messaging](./messaging.md), [Node Identification](./node-identification.md), and [Beacon Chain](../core/0_beacon-chain.md) specifications. # Specification -## Message Schemas +## Message schemas Message body schemas are notated like this: @@ -26,13 +25,13 @@ Message body schemas are notated like this: Embedded types are serialized as SSZ Containers unless otherwise noted. -All referenced data structures can be found in the [0-beacon-chain](../core/0_beacon-chain.md#data-structures) specification. +All referenced data structures can be found in the [Beacon Chain](../core/0_beacon-chain.md#data-structures) specification. -## `libp2p` Protocol Names +## `libp2p` protocol names -A "Protocol ID" in `libp2p` parlance refers to a human-readable identifier `libp2p` uses in order to identify sub-protocols and stream messages of different types over the same connection. Peers exchange supported protocol IDs via the `Identify` protocol upon connection. When opening a new stream, peers pin a particular protocol ID to it, and the stream remains contextualised thereafter. Since messages are sent inside a stream, they do not need to bear the protocol ID. +A "Protocol ID" in `libp2p` parlance refers to a human-readable identifier `libp2p` uses in order to identify sub-protocols and stream messages of different types over the same connection. Peers exchange supported protocol IDs via the `Identify` protocol upon connection. When opening a new stream, peers pin a particular protocol ID to it, and the stream remains contextualized thereafter. Since messages are sent inside a stream, they do not need to bear the protocol ID. -## RPC-Over-`libp2p` +## RPC-over-`libp2p` To facilitate RPC-over-`libp2p`, a single protocol name is used: `/eth/serenity/beacon/rpc/1`. The version number in the protocol name is neither backwards or forwards compatible, and will be incremented whenever changes to the below structures are required. @@ -88,7 +87,7 @@ The first 1,000 values in `response_code` are reserved for system use. The follo 3. `30`: Method not found. 4. `40`: Server error. -### Alternative for Non-`libp2p` Clients +### Alternative for non-`libp2p` clients Since some clients are waiting for `libp2p` implementations in their respective languages. As such, they MAY listen for raw TCP messages on port `9000`. To distinguish RPC messages from other messages on that port, a byte prefix of `ETH` (`0x455448`) MUST be prepended to all messages. This option will be removed once `libp2p` is ready in all supported languages. @@ -145,7 +144,7 @@ Root B ^ +---+ ``` -Once the handshake completes, the client with the higher `latest_finalized_epoch` or `best_slot` (if the clients have equal `latest_finalized_epoch`s) SHOULD request beacon block roots from its counterparty via `beacon_block_roots` (i.e., RPC method `10`). +Once the handshake completes, the client with the higher `latest_finalized_epoch` or `best_slot` (if the clients have equal `latest_finalized_epoch`s) SHOULD request beacon block roots from its counterparty via `beacon_block_roots` (i.e. RPC method `10`). ### Goodbye @@ -167,11 +166,11 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MAY be o Clients MAY define custom goodbye reasons as long as the value is larger than `1000`. -### Get Status +### Get status **Method ID:** `2` -**Request Body:** +**Request body:** ``` ( @@ -181,7 +180,7 @@ Clients MAY define custom goodbye reasons as long as the value is larger than `1 ) ``` -**Response Body:** +**Response body:** ``` ( @@ -193,11 +192,11 @@ Clients MAY define custom goodbye reasons as long as the value is larger than `1 Returns metadata about the remote node. -### Request Beacon Block Roots +### Request beacon block roots **Method ID:** `10` -**Request Body** +**Request body** ``` ( @@ -206,7 +205,7 @@ Returns metadata about the remote node. ) ``` -**Response Body:** +**Response body:** ``` # BlockRootSlot @@ -222,11 +221,11 @@ Returns metadata about the remote node. Requests a list of block roots and slots from the peer. The `count` parameter MUST be less than or equal to `32768`. The slots MUST be returned in ascending slot order. -### Beacon Block Headers +### Beacon block headers **Method ID:** `11` -**Request Body** +**Request body** ``` ( @@ -237,7 +236,7 @@ Requests a list of block roots and slots from the peer. The `count` parameter MU ) ``` -**Response Body:** +**Response body:** ``` ( @@ -245,15 +244,15 @@ Requests a list of block roots and slots from the peer. The `count` parameter MU ) ``` -Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `1` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is empty for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were empty in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further empty, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. A `skip_slots` value of `0` returns all blocks. +Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `1` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is empty for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were empty in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further empty, the array would contain `[2, 6, 8, 10]`—i.e. duplicate blocks MUST be collapsed. A `skip_slots` value of `0` returns all blocks. The function of the `skip_slots` parameter helps facilitate light client sync - for example, in [#459](https://github.com/ethereum/eth2.0-specs/issues/459) - and allows clients to balance the peers from whom they request headers. Clients could, for instance, request every 10th block from a set of peers where each peer has a different starting block in order to populate block data. -### Beacon Block Bodies +### Beacon block bodies **Method ID:** `12` -**Request Body:** +**Request body:** ``` ( @@ -261,7 +260,7 @@ The function of the `skip_slots` parameter helps facilitate light client sync - ) ``` -**Response Body:** +**Response body:** ``` ( @@ -269,15 +268,15 @@ The function of the `skip_slots` parameter helps facilitate light client sync - ) ``` -Requests the `block_bodies` associated with the provided `block_roots` from the peer. Responses MUST return `block_roots` in the order provided in the request. If the receiver does not have a particular `block_root`, it must return a zero-value `block_body` (i.e., a `block_body` container with all zero fields). +Requests the `block_bodies` associated with the provided `block_roots` from the peer. Responses MUST return `block_roots` in the order provided in the request. If the receiver does not have a particular `block_root`, it must return a zero-value `block_body` (i.e. a `block_body` container with all zero fields). -### Beacon Chain State +### Beacon chain state -**Note:** This section is preliminary, pending the definition of the data structures to be transferred over the wire during fast sync operations. +*Note*: This section is preliminary, pending the definition of the data structures to be transferred over the wire during fast sync operations. **Method ID:** `13` -**Request Body:** +**Request body:** ``` ( @@ -285,7 +284,7 @@ Requests the `block_bodies` associated with the provided `block_roots` from the ) ``` -**Response Body:** TBD +**Response body:** TBD Requests contain the hashes of Merkle tree nodes that when merkleized yield the block's `state_root`. diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 0080f2447..343c967d4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,23 +1,28 @@ # SimpleSerialize (SSZ) -This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects. +**Notice**: This document is a work-in-progress describing typing, serialization, and Merkleization of Eth 2.0 objects. ## Table of contents + -- [Constants](#constants) -- [Typing](#typing) - - [Basic types](#basic-types) - - [Composite types](#composite-types) - - [Aliases](#aliases) - - [Default values](#default-values) -- [Serialization](#serialization) - - [`"uintN"`](#uintn) - - [`"bool"`](#bool) - - [Containers, vectors, lists](#containers-vectors-lists) -- [Deserialization](#deserialization) -- [Merkleization](#merkleization) -- [Self-signed containers](#self-signed-containers) -- [Implementations](#implementations) +- [SimpleSerialize (SSZ)](#simpleserialize-ssz) + - [Table of contents](#table-of-contents) + - [Constants](#constants) + - [Typing](#typing) + - [Basic types](#basic-types) + - [Composite types](#composite-types) + - [Aliases](#aliases) + - [Default values](#default-values) + - [Serialization](#serialization) + - [`"uintN"`](#uintn) + - [`"bool"`](#bool) + - [Containers, vectors, lists](#containers-vectors-lists) + - [Deserialization](#deserialization) + - [Merkleization](#merkleization) + - [Self-signed containers](#self-signed-containers) + - [Implementations](#implementations) + + ## Constants @@ -133,4 +138,4 @@ Let `value` be a self-signed container object. The convention is that the signat | Go | Prysm | Prysmatic Labs | [https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | | Swift | Yeeth | Dean Eigenmann | [https://github.com/yeeth/SimpleSerialize.swift](https://github.com/yeeth/SimpleSerialize.swift) | | C# | | Jordan Andrews | [https://github.com/codingupastorm/csharp-ssz](https://github.com/codingupastorm/csharp-ssz) | -| C++ | | | [https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | +| C++ | | Jiyun Kim | [https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | diff --git a/specs/test_formats/README.md b/specs/test_formats/README.md index 273659ce9..4783dc475 100644 --- a/specs/test_formats/README.md +++ b/specs/test_formats/README.md @@ -1,17 +1,27 @@ # General test format -This document defines the YAML format and structure used for ETH 2.0 testing. +This document defines the YAML format and structure used for Eth 2.0 testing. -## ToC +## Table of contents + -* [About](#about) -* [Glossary](#glossary) -* [Test format philosophy](#test-format-philosophy) -* [Test Suite](#test-suite) -* [Config](#config) -* [Fork-timeline](#fork-timeline) -* [Config sourcing](#config-sourcing) -* [Test structure](#test-structure) +- [General test format](#general-test-format) + - [Table of contents](#table-of-contents) + - [About](#about) + - [Test-case formats](#test-case-formats) + - [Glossary](#glossary) + - [Test format philosophy](#test-format-philosophy) + - [Config design](#config-design) + - [Fork config design](#fork-config-design) + - [Test completeness](#test-completeness) + - [Test suite](#test-suite) + - [Config](#config) + - [Fork-timeline](#fork-timeline) + - [Config sourcing](#config-sourcing) + - [Test structure](#test-structure) + - [Note for implementers](#note-for-implementers) + + ## About @@ -52,28 +62,28 @@ Test formats: ### Config design After long discussion, the following types of configured constants were identified: -- Never changing: genesis data +- Never changing: genesis data. - Changing, but reliant on old value: e.g. an epoch time may change, but if you want to do the conversion - `(genesis data, timestamp) -> epoch number` you end up needing both constants. + `(genesis data, timestamp) -> epoch number`, you end up needing both constants. - Changing, but kept around during fork transition: finalization may take a while, e.g. an executable has to deal with new deposits and old deposits at the same time. Another example may be economic constants. -- Additional, back-wards compatible: new constants are introduced for later phases +- Additional, backwards compatible: new constants are introduced for later phases. - Changing: there is a very small chance some constant may really be *replaced*. In this off-chance, it is likely better to include it as an additional variable, - and some clients may simply stop supporting the old one, if they do not want to sync from genesis. + and some clients may simply stop supporting the old one if they do not want to sync from genesis. Based on these types of changes, we model the config as a list of key value pairs, - that only grows with every fork (they may change in development versions of forks however, git manages this). -With this approach, configurations are backwards compatible (older clients ignore unknown variables), and easy to maintain. + that only grows with every fork (they may change in development versions of forks, however; git manages this). +With this approach, configurations are backwards compatible (older clients ignore unknown variables) and easy to maintain. ### Fork config design There are two types of fork-data: -1) timeline: when does a fork take place? -2) coverage: what forks are covered by a test? +1) Timeline: When does a fork take place? +2) Coverage: What forks are covered by a test? The first is neat to have as a separate form: we prevent duplication, and can run with different presets - (e.g. fork timeline for a minimal local test, for a public testnet, or for mainnet) + (e.g. fork timeline for a minimal local test, for a public testnet, or for mainnet). The second does not affect the result of the tests, it just states what is covered by the tests, so that the right suites can be executed to see coverage for a certain fork. @@ -90,7 +100,7 @@ The aim is to provide clients with a well-defined scope of work to run a particu - Clients that are not complete in functionality can choose to ignore suites that use certain test-runners, or specific handlers of these test-runners. - Clients that are on older versions can test their work based on older releases of the generated tests, and catch up with newer releases when possible. -## Test Suite +## Test suite ``` title: -- Display name for the test suite @@ -113,9 +123,9 @@ Separation of configuration and tests aims to: - Prevent duplication of configuration - Make all tests easy to upgrade (e.g. when a new config constant is introduced) - Clearly define which constants to use -- Shareable between clients, for cross-client short or long lived testnets +- Shareable between clients, for cross-client short- or long-lived testnets - Minimize the amounts of different constants permutations to compile as a client. - Note: Some clients prefer compile-time constants and optimizations. + *Note*: Some clients prefer compile-time constants and optimizations. They should compile for each configuration once, and run the corresponding tests per build target. The format is described in [`configs/constant_presets`](../../configs/constant_presets/README.md#format). @@ -124,9 +134,9 @@ The format is described in [`configs/constant_presets`](../../configs/constant_p ## Fork-timeline A fork timeline is (preferably) loaded in as a configuration object into a client, as opposed to the constants configuration: - - we do not allocate or optimize any code based on epoch numbers - - when we transition from one fork to the other, it is preferred to stay online. - - we may decide on an epoch number for a fork based on external events (e.g. Eth1 log event), + - We do not allocate or optimize any code based on epoch numbers. + - When we transition from one fork to the other, it is preferred to stay online. + - We may decide on an epoch number for a fork based on external events (e.g. Eth1 log event); a client should be able to activate a fork dynamically. The format is described in [`configs/fork_timelines`](../../configs/fork_timelines/README.md#format). diff --git a/specs/test_formats/bls/README.md b/specs/test_formats/bls/README.md index db63bba1d..4d95bdfd7 100644 --- a/specs/test_formats/bls/README.md +++ b/specs/test_formats/bls/README.md @@ -1,7 +1,7 @@ # BLS tests A test type for BLS. Primarily geared towards verifying the *integration* of any BLS library. -We do not recommend to roll your own crypto, or use an untested BLS library. +We do not recommend rolling your own crypto or using an untested BLS library. The BLS test suite runner has the following handlers: @@ -12,4 +12,4 @@ The BLS test suite runner has the following handlers: - [`priv_to_pub`](./priv_to_pub.md) - [`sign_msg`](./sign_msg.md) -Note: signature-verification and aggregate-verify test cases are not yet supported. +*Note*: Signature-verification and aggregate-verify test cases are not yet supported. diff --git a/specs/test_formats/shuffling/README.md b/specs/test_formats/shuffling/README.md index 57be96565..25074742d 100644 --- a/specs/test_formats/shuffling/README.md +++ b/specs/test_formats/shuffling/README.md @@ -1,16 +1,16 @@ # Test format: shuffling -The runner of the Shuffling test type has only one handler: `core` +The runner of the Shuffling test type has only one handler: `core`. -This does not mean however that testing is limited. +However, this does not mean that testing is limited. Clients may take different approaches to shuffling, for optimizing, and supporting advanced lookup behavior back in older history. For implementers, possible test runners implementing testing can include: -1) just test permute-index, run it for each index `i` in `range(count)`, and check against expected `output[i]` (default spec implementation) -2) test un-permute-index (the reverse lookup. Implemented by running the shuffling rounds in reverse: from `round_count-1` to `0`) -3) test the optimized complete shuffle, where all indices are shuffled at once, test output in one go. -4) test complete shuffle in reverse (reverse rounds, same as 2) +1) Just test permute-index, run it for each index `i` in `range(count)`, and check against expected `output[i]` (default spec implementation). +2) Test un-permute-index (the reverse lookup; implemented by running the shuffling rounds in reverse, from `round_count-1` to `0`). +3) Test the optimized complete shuffle, where all indices are shuffled at once; test output in one go. +4) Test complete shuffle in reverse (reverse rounds, same as #2). ## Test case format diff --git a/specs/test_formats/ssz_generic/README.md b/specs/test_formats/ssz_generic/README.md index 9fda0c368..da0898087 100644 --- a/specs/test_formats/ssz_generic/README.md +++ b/specs/test_formats/ssz_generic/README.md @@ -3,7 +3,7 @@ This set of test-suites provides general testing for SSZ: to instantiate any container/list/vector/other type from binary data. -Since SSZ is in a development-phase, not the full suite of features is covered yet. +Since SSZ is in a development-phase, the full suite of features is not covered yet. Note that these tests are based on the older SSZ package. The tests are still relevant, but limited in scope: more complex object encodings have changed since the original SSZ testing. @@ -11,10 +11,10 @@ The tests are still relevant, but limited in scope: A minimal but useful series of tests covering `uint` encoding and decoding is provided. This is a direct port of the older SSZ `uint` tests (minus outdated test cases). -[uint test format](./uint.md). +Test format documentation can be found here: [uint test format](./uint.md). -Note: the current phase-0 spec does not use larger uints, and uses byte vectors (fixed length) instead to represent roots etc. +*Note*: The current Phase 0 spec does not use larger uints, and uses byte vectors (fixed length) instead to represent roots etc. The exact uint lengths to support may be redefined in the future. -Extension of the SSZ tests collection is planned, with an update to the new spec-maintained `minimal_ssz.py`, +Extension of the SSZ tests collection is planned, with an update to the new spec-maintained `minimal_ssz.py`; see CI/testing issues for progress tracking. diff --git a/specs/test_formats/ssz_static/README.md b/specs/test_formats/ssz_static/README.md index 413b00c75..1df2cb5f6 100644 --- a/specs/test_formats/ssz_static/README.md +++ b/specs/test_formats/ssz_static/README.md @@ -1,7 +1,7 @@ # SSZ, static tests This set of test-suites provides static testing for SSZ: - to instantiate just the known ETH-2.0 SSZ types from binary data. + to instantiate just the known Eth 2.0 SSZ types from binary data. This series of tests is based on the spec-maintained `minimal_ssz.py`, i.e. fully consistent with the SSZ spec. diff --git a/specs/test_formats/ssz_static/core.md b/specs/test_formats/ssz_static/core.md index 1d470c338..64b09a329 100644 --- a/specs/test_formats/ssz_static/core.md +++ b/specs/test_formats/ssz_static/core.md @@ -1,7 +1,7 @@ # Test format: SSZ static types The goal of this type is to provide clients with a solid reference for how the known SSZ objects should be encoded. -Each object described in the Phase-0 spec is covered. +Each object described in the Phase 0 spec is covered. This is important, as many of the clients aiming to serialize/deserialize objects directly into structs/classes do not support (or have alternatives for) generic SSZ encoding/decoding. This test-format ensures these direct serializations are covered. @@ -27,6 +27,6 @@ A test-runner can implement the following assertions: ## References -**`serialized`**: [SSZ serialization](../../simple-serialize.md#serialization) -**`root`** - [hash_tree_root](../../simple-serialize.md#merkleization) function -**`signing_root`** - [signing_root](../../simple-serialize.md#self-signed-containers) function +**`serialized`**—[SSZ serialization](../../simple-serialize.md#serialization) +**`root`**—[hash_tree_root](../../simple-serialize.md#merkleization) function +**`signing_root`**—[signing_root](../../simple-serialize.md#self-signed-containers) function diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index ca7f0eb3b..600ed0839 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -1,13 +1,13 @@ # Ethereum 2.0 Phase 0 -- Honest Validator -__NOTICE__: This document is a work-in-progress for researchers and implementers. This is an accompanying document to [Ethereum 2.0 Phase 0 -- The Beacon Chain](../core/0_beacon-chain.md) that describes the expected actions of a "validator" participating in the Ethereum 2.0 protocol. +**Notice**: This document is a work-in-progress for researchers and implementers. This is an accompanying document to [Ethereum 2.0 Phase 0 -- The Beacon Chain](../core/0_beacon-chain.md), which describes the expected actions of a "validator" participating in the Ethereum 2.0 protocol. -## Table of Contents +## Table of contents - [Ethereum 2.0 Phase 0 -- Honest Validator](#ethereum-20-phase-0----honest-validator) - - [Table of Contents](#table-of-contents) + - [Table of contents](#table-of-contents) - [Introduction](#introduction) - [Prerequisites](#prerequisites) - [Constants](#constants) @@ -96,7 +96,7 @@ The validator constructs their `withdrawal_credentials` via the following: ### Submit deposit -In phase 0, all incoming validator deposits originate from the Ethereum 1.0 PoW chain. Deposits are made to the [deposit contract](../core/0_deposit-contract.md) located at `DEPOSIT_CONTRACT_ADDRESS`. +In Phase 0, all incoming validator deposits originate from the Ethereum 1.0 PoW chain. Deposits are made to the [deposit contract](../core/0_deposit-contract.md) located at `DEPOSIT_CONTRACT_ADDRESS`. To submit a deposit: @@ -106,11 +106,11 @@ To submit a deposit: * Let `signature` be the result of `bls_sign` of the `signing_root(deposit_data)` with `domain=DOMAIN_DEPOSIT`. * Send a transaction on the Ethereum 1.0 chain to `DEPOSIT_CONTRACT_ADDRESS` executing `def deposit(pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96])` along with a deposit of `amount` Gwei. -_Note_: Deposits made for the same `pubkey` are treated as for the same validator. A singular `Validator` will be added to `state.validator_registry` with each additional deposit amount added to the validator's balance. A validator can only be activated when total deposits for the validator pubkey meet or exceed `MAX_EFFECTIVE_BALANCE`. +*Note*: Deposits made for the same `pubkey` are treated as for the same validator. A singular `Validator` will be added to `state.validator_registry` with each additional deposit amount added to the validator's balance. A validator can only be activated when total deposits for the validator pubkey meet or exceed `MAX_EFFECTIVE_BALANCE`. ### Process deposit -Deposits cannot be processed into the beacon chain until the eth1.0 block in which they were deposited or any of its descendants is added to the beacon chain `state.eth1_data`. This takes _a minimum_ of `ETH1_FOLLOW_DISTANCE` eth1.0 blocks (~4 hours) plus `ETH1_DATA_VOTING_PERIOD` epochs (~1.7 hours). Once the requisite eth1.0 data is added, the deposit will normally be added to a beacon chain block and processed into the `state.validator_registry` within an epoch or two. The validator is then in a queue to be activated. +Deposits cannot be processed into the beacon chain until the Eth 1.0 block in which they were deposited or any of its descendants is added to the beacon chain `state.eth1_data`. This takes _a minimum_ of `ETH1_FOLLOW_DISTANCE` Eth 1.0 blocks (~4 hours) plus `ETH1_DATA_VOTING_PERIOD` epochs (~1.7 hours). Once the requisite Eth 1.0 data is added, the deposit will normally be added to a beacon chain block and processed into the `state.validator_registry` within an epoch or two. The validator is then in a queue to be activated. ### Validator index @@ -130,11 +130,11 @@ is_active = is_active_validator(validator, shuffling_epoch) Once a validator is activated, the validator is assigned [responsibilities](#beacon-chain-responsibilities) until exited. -_Note_: There is a maximum validator churn per finalized epoch so the delay until activation is variable depending upon finality, total active validator balance, and the number of validators in the queue to be activated. +*Note*: There is a maximum validator churn per finalized epoch so the delay until activation is variable depending upon finality, total active validator balance, and the number of validators in the queue to be activated. ## Beacon chain responsibilities -A validator has two primary responsibilities to the beacon chain -- [proposing blocks](block-proposal) and [creating attestations](attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. +A validator has two primary responsibilities to the beacon chain: [proposing blocks](block-proposal) and [creating attestations](attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. ### Block proposal @@ -148,7 +148,7 @@ There is one proposer per slot, so if there are N active validators any individu Set `block.slot = slot` where `slot` is the current slot at which the validator has been selected to propose. The `parent` selected must satisfy that `parent.slot < block.slot`. -_Note:_ there might be "skipped" slots between the `parent` and `block`. These skipped slots are processed in the state transition function without per-block processing. +*Note*: There might be "skipped" slots between the `parent` and `block`. These skipped slots are processed in the state transition function without per-block processing. ##### Parent root @@ -158,7 +158,7 @@ Set `block.previous_block_root = signing_root(parent)`. Set `block.state_root = hash_tree_root(state)` of the resulting `state` of the `parent -> block` state transition. -_Note_: To calculate `state_root`, the validator should first run the state transition function on an unsigned `block` containing a stub for the `state_root`. It is useful to be able to run a state transition function that does _not_ validate signatures or state root for this purpose. +*Note*: To calculate `state_root`, the validator should first run the state transition function on an unsigned `block` containing a stub for the `state_root`. It is useful to be able to run a state transition function that does _not_ validate signatures or state root for this purpose. ##### Randao reveal @@ -181,12 +181,12 @@ epoch_signature = bls_sign( `block.eth1_data` is a mechanism used by block proposers vote on a recent Ethereum 1.0 block hash and an associated deposit root found in the Ethereum 1.0 deposit contract. When consensus is formed, `state.latest_eth1_data` is updated, and validator deposits up to this root can be processed. The deposit root can be calculated by calling the `get_deposit_root()` function of the deposit contract using the post-state of the block hash. * Let `D` be the set of `Eth1DataVote` objects `vote` in `state.eth1_data_votes` where: - * `vote.eth1_data.block_hash` is the hash of an eth1.0 block that is (i) part of the canonical chain, (ii) >= `ETH1_FOLLOW_DISTANCE` blocks behind the head, and (iii) newer than `state.latest_eth1_data.block_hash`. - * `vote.eth1_data.deposit_count` is the deposit count of the eth1.0 deposit contract at the block defined by `vote.eth1_data.block_hash`. - * `vote.eth1_data.deposit_root` is the deposit root of the eth1.0 deposit contract at the block defined by `vote.eth1_data.block_hash`. + * `vote.eth1_data.block_hash` is the hash of an Eth 1.0 block that is (i) part of the canonical chain, (ii) >= `ETH1_FOLLOW_DISTANCE` blocks behind the head, and (iii) newer than `state.latest_eth1_data.block_hash`. + * `vote.eth1_data.deposit_count` is the deposit count of the Eth 1.0 deposit contract at the block defined by `vote.eth1_data.block_hash`. + * `vote.eth1_data.deposit_root` is the deposit root of the Eth 1.0 deposit contract at the block defined by `vote.eth1_data.block_hash`. * If `D` is empty: - * Let `block_hash` be the block hash of the `ETH1_FOLLOW_DISTANCE`'th ancestor of the head of the canonical eth1.0 chain. - * Let `deposit_root` and `deposit_count` be the deposit root and deposit count of the eth1.0 deposit contract in the post-state of the block referenced by `block_hash` + * Let `block_hash` be the block hash of the `ETH1_FOLLOW_DISTANCE`'th ancestor of the head of the canonical Eth 1.0 chain. + * Let `deposit_root` and `deposit_count` be the deposit root and deposit count of the Eth 1.0 deposit contract in the post-state of the block referenced by `block_hash` * Let `best_vote_data = Eth1Data(block_hash=block_hash, deposit_root=deposit_root, deposit_count=deposit_count)`. * If `D` is nonempty: * Let `best_vote_data` be the `eth1_data` of the member of `D` that has the highest `vote.vote_count`, breaking ties by favoring block hashes with higher associated block height. @@ -224,7 +224,7 @@ Up to `MAX_ATTESTATIONS` aggregate attestations can be included in the `block`. ##### Deposits -If there are any unprocessed deposits for the existing `state.latest_eth1_data` (i.e. `state.latest_eth1_data.deposit_count > state.deposit_index`), then pending deposits _must_ be added to the block. The expected number of deposits is exactly `min(MAX_DEPOSITS, latest_eth1_data.deposit_count - state.deposit_index)`. These [`deposits`](../core/0_beacon-chain.md#deposit) are constructed from the `Deposit` logs from the [Eth1.0 deposit contract](../core/0_deposit-contract) and must be processed in sequential order. The deposits included in the `block` must satisfy the verification conditions found in [deposits processing](../core/0_beacon-chain.md#deposits). +If there are any unprocessed deposits for the existing `state.latest_eth1_data` (i.e. `state.latest_eth1_data.deposit_count > state.deposit_index`), then pending deposits _must_ be added to the block. The expected number of deposits is exactly `min(MAX_DEPOSITS, latest_eth1_data.deposit_count - state.deposit_index)`. These [`deposits`](../core/0_beacon-chain.md#deposit) are constructed from the `Deposit` logs from the [Eth 1.0 deposit contract](../core/0_deposit-contract) and must be processed in sequential order. The deposits included in the `block` must satisfy the verification conditions found in [deposits processing](../core/0_beacon-chain.md#deposits). The `proof` for each deposit must be constructed against the deposit root contained in `state.latest_eth1_data` rather than the deposit root at the time the deposit was initially logged from the 1.0 chain. This entails storing a full deposit merkle tree locally and computing updated proofs against the `latest_eth1_data.deposit_root` as needed. See [`minimal_merkle.py`](https://github.com/ethereum/research/blob/master/spec_pythonizer/utils/merkle_minimal.py) for a sample implementation. @@ -236,7 +236,7 @@ Up to `MAX_VOLUNTARY_EXITS` [`VoluntaryExit`](../core/0_beacon-chain.md#voluntar A validator is expected to create, sign, and broadcast an attestation during each epoch. The slot during which the validator performs this role is any slot at which `get_crosslink_committees_at_slot(state, slot)` contains a committee that contains `validator_index`. -A validator should create and broadcast the attestation halfway through the `slot` during which the validator is assigned -- that is `SECONDS_PER_SLOT * 0.5` seconds after the start of `slot`. +A validator should create and broadcast the attestation halfway through the `slot` during which the validator is assigned ― that is, `SECONDS_PER_SLOT * 0.5` seconds after the start of `slot`. #### Attestation data @@ -265,7 +265,7 @@ Set `attestation_data.source_root = head_state.current_justified_root`. Set `attestation_data.target_root = signing_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary. -_Note:_ This can be looked up in the state using: +*Note*: This can be looked up in the state using: * Let `epoch_start_slot = get_epoch_start_slot(get_current_epoch(head_state))`. * Set `epoch_boundary = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`. @@ -281,7 +281,7 @@ Set `attestation_data.previous_crosslink_root = hash_tree_root(head_state.curren Set `attestation_data.crosslink_data_root = ZERO_HASH`. -_Note:_ This is a stub for phase 0. +*Note*: This is a stub for Phase 0. #### Construct attestation @@ -298,14 +298,14 @@ Set `attestation.data = attestation_data` where `attestation_data` is the `Attes * Set `aggregation_bitfield[index_into_committee // 8] |= 2 ** (index_into_committee % 8)`. * Set `attestation.aggregation_bitfield = aggregation_bitfield`. -_Note_: Calling `get_attesting_indices(state, attestation.data, attestation.aggregation_bitfield)` should return a list of length equal to 1, containing `validator_index`. +*Note*: Calling `get_attesting_indices(state, attestation.data, attestation.aggregation_bitfield)` should return a list of length equal to 1, containing `validator_index`. ##### Custody bitfield * Let `custody_bitfield` be a byte array filled with zeros of length `(len(committee) + 7) // 8`. * Set `attestation.custody_bitfield = custody_bitfield`. -_Note:_ This is a stub for phase 0. +*Note*: This is a stub for Phase 0. ##### Aggregate signature @@ -379,14 +379,14 @@ def is_proposer_at_slot(state: BeaconState, return get_beacon_proposer_index(state) == validator_index ``` -_Note_: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot. +*Note*: To see if a validator is assigned to proposer during the slot, the validator must run an empty slot transition from the previous state to the current slot. ### Lookahead -The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing which must checked during the slot in question. +The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming committee assignments for attesting dictated by the shuffling and slot. Note that this lookahead does not apply to proposing, which must checked during the slot in question. -`get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in phase 1+). +`get_committee_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments which involves noting at which future slot one will have to attest and also which shard one should begin syncing (in Phase 1+). Specifically, a validator should call `get_committee_assignment(state, next_epoch, validator_index)` when checking for next epoch assignments. @@ -394,19 +394,19 @@ Specifically, a validator should call `get_committee_assignment(state, next_epoc "Slashing" is the burning of some amount of validator funds and immediate ejection from the active validator set. In Phase 0, there are two ways in which funds can be slashed -- [proposer slashing](#proposer-slashing) and [attester slashing](#attester-slashing). Although being slashed has serious repercussions, it is simple enough to avoid being slashed all together by remaining _consistent_ with respect to the messages a validator has previously signed. -_Note_: Signed data must be within a sequential `Fork` context to conflict. Messages cannot be slashed across diverging forks. If the previous fork version is 1 and the chain splits into fork 2 and 102, messages from 1 can slashable against messages in forks 1, 2, and 102. Messages in 2 cannot be slashable against messages in 102 and vice versa. +*Note*: Signed data must be within a sequential `Fork` context to conflict. Messages cannot be slashed across diverging forks. If the previous fork version is 1 and the chain splits into fork 2 and 102, messages from 1 can slashable against messages in forks 1, 2, and 102. Messages in 2 cannot be slashable against messages in 102 and vice versa. ### Proposer slashing To avoid "proposer slashings", a validator must not sign two conflicting [`BeaconBlock`](../core/0_beacon-chain.md#beaconblock) where conflicting is defined as two distinct blocks within the same epoch. -_In phase 0, as long as the validator does not sign two different beacon blocks for the same epoch, the validator is safe against proposer slashings._ +*In Phase 0, as long as the validator does not sign two different beacon blocks for the same epoch, the validator is safe against proposer slashings.* -Specifically, when signing an `BeaconBlock`, a validator should perform the following steps in the following order: -1. Save a record to hard disk that an beacon block has been signed for the `epoch=slot_to_epoch(block.slot)`. +Specifically, when signing a `BeaconBlock`, a validator should perform the following steps in the following order: +1. Save a record to hard disk that a beacon block has been signed for the `epoch=slot_to_epoch(block.slot)`. 2. Generate and broadcast the block. -If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the _potentially_ signed/broadcast block and can effectively avoid slashing. +If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the *potentially* signed/broadcast block and can effectively avoid slashing. ### Attester slashing @@ -416,4 +416,4 @@ Specifically, when signing an `Attestation`, a validator should perform the foll 1. Save a record to hard disk that an attestation has been signed for source -- `attestation_data.source_epoch` -- and target -- `slot_to_epoch(attestation_data.slot)`. 2. Generate and broadcast attestation. -If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the _potentially_ signed/broadcast attestation and can effectively avoid slashing. +If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the *potentially* signed/broadcast attestation and can effectively avoid slashing. diff --git a/test_generators/README.md b/test_generators/README.md index 43bf7af03..f8124f9a7 100644 --- a/test_generators/README.md +++ b/test_generators/README.md @@ -2,7 +2,7 @@ This directory contains all the generators for YAML tests, consumed by Eth 2.0 client implementations. -Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests). +Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-spec-tests](https://github.com/ethereum/eth2.0-spec-tests). Whenever a release is made, the new tests are automatically built, and [eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository. @@ -12,7 +12,7 @@ Whenever a release is made, the new tests are automatically built, and Prerequisites: - Python 3 installed - PIP 3 -- GNU make +- GNU Make ### Cleaning @@ -66,7 +66,7 @@ eth-utils==1.4.1 The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself in order to prevent code duplication and outdated tests. Applying configurations to the spec is simple and enables you to create test suites with different contexts. -Note: make sure to run `make pyspec` from the root of the specs repository in order to build the pyspec requirement. +*Note*: Make sure to run `make pyspec` from the root of the specs repository in order to build the pyspec requirement. Install all the necessary requirements (re-run when you add more): ```bash @@ -134,7 +134,7 @@ if __name__ == "__main__": Recommendations: - You can have more than just one suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])`. - You can concatenate lists of test cases if you don't want to split it up in suites, however, make sure they can be run with one handler. -- You can split your suite creators into different python files/packages; this is good for code organization. +- You can split your suite creators into different Python files/packages; this is good for code organization. - Use config "minimal" for performance, but also implement a suite with the default config where necessary. - You may be able to write your test suite creator in a way where it does not make assumptions on constants. If so, you can generate test suites with different configurations for the same scenario (see example). @@ -157,8 +157,8 @@ To add a new test generator that builds `New Tests`: [circleci config file](https://github.com/ethereum/eth2.0-test-generators/blob/master/.circleci/config.yml) if desired to increase code quality. -Note: you do not have to change the makefile. -However, if necessary (e.g. not using python, or mixing in other languages), submit an issue, and it can be a special case. +*Note*: You do not have to change the makefile. +However, if necessary (e.g. not using Python, or mixing in other languages), submit an issue, and it can be a special case. Do note that generators should be easy to maintain, lean, and based on the spec. @@ -167,5 +167,5 @@ Do note that generators should be easy to maintain, lean, and based on the spec. If a test generator is not needed anymore, undo the steps described above and make a new release: 1. Remove the generator directory. -2. Remove the generated tests in the [`eth2.0-tests`](https://github.com/ethereum/eth2.0-tests) repository by opening a PR there. +2. Remove the generated tests in the [`eth2.0-spec-tests`](https://github.com/ethereum/eth2.0-spec-tests) repository by opening a pull request there. 3. Make a new release. diff --git a/test_libs/pyspec/README.md b/test_libs/pyspec/README.md index df1834210..bb6991a93 100644 --- a/test_libs/pyspec/README.md +++ b/test_libs/pyspec/README.md @@ -1,11 +1,11 @@ -# ETH 2.0 PySpec +# Eth 2.0 Executable Python Spec (PySpec) -The Python executable spec is built from the ETH 2.0 specification, +The executable Python spec is built from the Eth 2.0 specification, complemented with the necessary helper functions for hashing, BLS, and more. With this executable spec, test-generators can easily create test-vectors for client implementations, - and the spec itself can be verified to be consistent and coherent, through sanity tests implemented with pytest. + and the spec itself can be verified to be consistent and coherent through sanity tests implemented with pytest. ## Building @@ -14,12 +14,12 @@ All the dynamic parts of the spec can be build at once with `make pyspec`. Alternatively, you can build a sub-set of the pyspec: `make phase0`. -Or, to build a single file, specify the path, e.g. `make test_libs/pyspec/eth2spec/phase0/spec.py` +Or, to build a single file, specify the path, e.g. `make test_libs/pyspec/eth2spec/phase0/spec.py`. ## Py-tests -After building, you can install the dependencies for running the `pyspec` tests with `make install_test` +After building, you can install the dependencies for running the `pyspec` tests with `make install_test`. These tests are not intended for client-consumption. These tests are sanity tests, to verify if the spec itself is consistent. @@ -28,7 +28,7 @@ These tests are sanity tests, to verify if the spec itself is consistent. #### Automated -Run `make test` from the root of the spec repository. +Run `make test` from the root of the specs repository. #### Manual @@ -40,7 +40,7 @@ python3 -m venv venv . venv/bin/activate pip3 install -r requirements-testing.txt ``` -Note: make sure to run `make -B pyspec` from the root of the specs repository, +*Note*: Make sure to run `make -B pyspec` from the root of the specs repository, to build the parts of the pyspec module derived from the markdown specs. The `-B` flag may be helpful to force-overwrite the `pyspec` output after you made a change to the markdown source files. @@ -58,4 +58,4 @@ The pyspec is not a replacement. ## License -Same as the spec itself, see [LICENSE](../../LICENSE) file in spec repository root. +Same as the spec itself; see [LICENSE](../../LICENSE) file in the specs repository root. From b15105e1cbb4a597fea233b16eac23ae911570a6 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 17:34:03 +0100 Subject: [PATCH 34/45] Address Danny's comment --- specs/core/0_beacon-chain.md | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 156a2523d..0bd85478e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1298,22 +1298,17 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat ```python def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: - shard_attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] - candidate_crosslinks = [ - c for c in [a.data.crosslink for a in shard_attestations] - if hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)) - ] - if len(candidate_crosslinks) == 0: - return Crosslink(), [] - - def get_attestations_for(crosslink: Crosslink) -> List[PendingAttestation]: - return [a for a in shard_attestations if a.data.crosslink == crosslink] - # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) - winning_crosslink = max(candidate_crosslinks, key=lambda crosslink: ( - get_attesting_balance(state, get_attestations_for(crosslink)), crosslink.crosslink_data_root + attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] + crosslinks = list(filter( + lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)), + [a.data.crosslink for a in attestations] )) - - return winning_crosslink, get_unslashed_attesting_indices(state, get_attestations_for(winning_crosslink)) + # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) + winning_crosslink = max(crosslinks, key=lambda c: ( + get_attesting_balance(state, [a for a in attestations if a.data.crosslink == c]), c.crosslink_data_root + ), default=Crosslink()) + winning_attestations = [a for a in attestations if a.data.crosslink == winning_crosslink] + return winning_crosslink, get_unslashed_attesting_indices(state, winning_attestations) ``` #### Justification and finalization From 8b1a2edb7c650e0e0205e11c44f44a2a578ce392 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 17:53:49 +0100 Subject: [PATCH 35/45] Fix genesis bug --- specs/core/0_beacon-chain.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 9aed2001a..04ab38e18 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -767,7 +767,7 @@ def get_epoch_start_shard(state: BeaconState, epoch: Epoch) -> Shard: ```python def get_attestation_data_slot(state: BeaconState, data: AttestationData) -> Slot: committee_count = get_epoch_committee_count(state, data.target_epoch) - offset = (data.shard + SHARD_COUNT - get_epoch_start_shard(state, data.target_epoch)) % SHARD_COUNT + offset = (data.crosslink.shard + SHARD_COUNT - get_epoch_start_shard(state, data.target_epoch)) % SHARD_COUNT return get_epoch_start_slot(data.target_epoch) + offset // (committee_count // SLOTS_PER_EPOCH) ``` @@ -1705,14 +1705,14 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: ) assert data.target_epoch in (get_previous_epoch(state), get_current_epoch(state)) - if data.target_epoch == get_previous_epoch(state): - ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) - previous_crosslink = state.previous_crosslinks[data.crosslink.shard] - state.previous_epoch_attestations.append(pending_attestation) if data.target_epoch == get_current_epoch(state): ffg_data = (state.current_justified_epoch, state.current_justified_root, get_current_epoch(state)) previous_crosslink = state.current_crosslinks[data.crosslink.shard] state.current_epoch_attestations.append(pending_attestation) + else: + ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) + previous_crosslink = state.previous_crosslinks[data.crosslink.shard] + state.previous_epoch_attestations.append(pending_attestation) # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) From f4db9ebae00338ba336943a08271012099bcef1d Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 18:26:14 +0100 Subject: [PATCH 36/45] Renamings triggered by HW comment --- specs/core/0_beacon-chain.md | 26 +++++++++---------- specs/core/0_fork-choice.md | 2 +- specs/core/1_custody-game.md | 14 +++++----- specs/core/1_shard-data-chains.md | 16 ++++++------ specs/validator/0_beacon-chain-validator.md | 6 ++--- .../test_process_attestation.py | 2 +- .../test_process_block_header.py | 4 +-- test_libs/pyspec/tests/helpers.py | 12 ++++----- test_libs/pyspec/tests/test_sanity.py | 6 ++--- 9 files changed, 44 insertions(+), 44 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 04ab38e18..afca9eb1b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -287,9 +287,9 @@ The types are defined topologically to aid in facilitating an executable version # Epoch number 'epoch': 'uint64', # Root of the previous crosslink - 'previous_crosslink_root': 'bytes32', + 'parent_crosslink_root': 'bytes32', # Root of the crosslinked shard data since the previous crosslink - 'crosslink_data_root': 'bytes32', + 'data_root': 'bytes32', } ``` @@ -369,7 +369,7 @@ The types are defined topologically to aid in facilitating an executable version ```python { 'slot': 'uint64', - 'previous_block_root': 'bytes32', + 'parent_block_root': 'bytes32', 'state_root': 'bytes32', 'block_body_root': 'bytes32', 'signature': 'bytes96', @@ -536,7 +536,7 @@ The types are defined topologically to aid in facilitating an executable version { # Header 'slot': 'uint64', - 'previous_block_root': 'bytes32', + 'parent_block_root': 'bytes32', 'state_root': 'bytes32', 'body': BeaconBlockBody, 'signature': 'bytes96', @@ -1306,12 +1306,12 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] crosslinks = list(filter( - lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.previous_crosslink_root, hash_tree_root(c)), + lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_crosslink_root, hash_tree_root(c)), [a.data.crosslink for a in attestations] )) # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) winning_crosslink = max(crosslinks, key=lambda c: ( - get_attesting_balance(state, [a for a in attestations if a.data.crosslink == c]), c.crosslink_data_root + get_attesting_balance(state, [a for a in attestations if a.data.crosslink == c]), c.data_root ), default=Crosslink()) winning_attestations = [a for a in attestations if a.data.crosslink == winning_crosslink] return winning_crosslink, get_unslashed_attesting_indices(state, winning_attestations) @@ -1586,11 +1586,11 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: # Verify that the slots match assert block.slot == state.slot # Verify that the parent matches - assert block.previous_block_root == signing_root(state.latest_block_header) + assert block.parent_block_root == signing_root(state.latest_block_header) # Save current block as the new latest block state.latest_block_header = BeaconBlockHeader( slot=block.slot, - previous_block_root=block.previous_block_root, + parent_block_root=block.parent_block_root, block_body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed @@ -1707,18 +1707,18 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: assert data.target_epoch in (get_previous_epoch(state), get_current_epoch(state)) if data.target_epoch == get_current_epoch(state): ffg_data = (state.current_justified_epoch, state.current_justified_root, get_current_epoch(state)) - previous_crosslink = state.current_crosslinks[data.crosslink.shard] + parent_crosslink = state.current_crosslinks[data.crosslink.shard] state.current_epoch_attestations.append(pending_attestation) else: ffg_data = (state.previous_justified_epoch, state.previous_justified_root, get_previous_epoch(state)) - previous_crosslink = state.previous_crosslinks[data.crosslink.shard] + parent_crosslink = state.previous_crosslinks[data.crosslink.shard] state.previous_epoch_attestations.append(pending_attestation) # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) - assert data.crosslink.epoch == min(data.target_epoch, previous_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) - assert data.crosslink.previous_crosslink_root == hash_tree_root(previous_crosslink) - assert data.crosslink.crosslink_data_root == ZERO_HASH # [to be removed in phase 1] + assert data.crosslink.epoch == min(data.target_epoch, parent_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) + assert data.crosslink.parent_crosslink_root == hash_tree_root(parent_crosslink) + assert data.crosslink.data_root == ZERO_HASH # [to be removed in phase 1] assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) ``` diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index 549e9e207..891120de8 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -38,7 +38,7 @@ All terminology, constants, functions, and protocol mechanics defined in the [Ph Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Clients download and process blocks and maintain a view of what is the current "canonical chain", terminating at the current "head". For a beacon block, `block`, to be processed by a node, the following conditions must be met: -* The parent block with root `block.previous_block_root` has been processed and accepted. +* The parent block with root `block.parent_block_root` has been processed and accepted. * An Ethereum 1.0 block pointed to by the `state.latest_eth1_data.block_hash` has been processed and accepted. * The node's Unix time is greater than or equal to `state.genesis_time + block.slot * SECONDS_PER_SLOT`. diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md index e259b82a5..a030c577b 100644 --- a/specs/core/1_custody-game.md +++ b/specs/core/1_custody-game.md @@ -147,7 +147,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenger_index': ValidatorIndex, 'responder_index': ValidatorIndex, 'deadline': Epoch, - 'crosslink_data_root': Hash, + 'data_root': Hash, 'depth': 'uint64', 'chunk_index': 'uint64', } @@ -161,7 +161,7 @@ This document details the beacon chain additions and changes in Phase 1 of Ether 'challenger_index': ValidatorIndex, 'responder_index': ValidatorIndex, 'deadline': Epoch, - 'crosslink_data_root': Hash, + 'data_root': Hash, 'chunk_count': 'uint64', 'chunk_bits_merkle_root': Hash, 'responder_key': BLSSignature, @@ -474,7 +474,7 @@ def process_chunk_challenge(state: BeaconState, # Verify the challenge is not a duplicate for record in state.custody_chunk_challenge_records: assert ( - record.crosslink_data_root != challenge.attestation.data.crosslink.crosslink_data_root or + record.data_root != challenge.attestation.data.crosslink.data_root or record.chunk_index != challenge.chunk_index ) # Verify depth @@ -486,7 +486,7 @@ def process_chunk_challenge(state: BeaconState, challenger_index=get_beacon_proposer_index(state), responder_index=challenge.responder_index deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, + data_root=challenge.attestation.data.crosslink.data_root, depth=depth, chunk_index=challenge.chunk_index, ) @@ -564,7 +564,7 @@ def process_bit_challenge(state: BeaconState, challenger_index=challenge.challenger_index, responder_index=challenge.responder_index, deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, - crosslink_data_root=challenge.attestation.data.crosslink.crosslink_data_root, + data_root=challenge.attestation.data.crosslink.data_root, chunk_count=chunk_count, chunk_bits_merkle_root=merkle_root(pad_to_power_of_2((challenge.chunk_bits))), responder_key=challenge.responder_key, @@ -610,7 +610,7 @@ def process_chunk_challenge_response(state: BeaconState, branch=response.data_branch, depth=challenge.depth, index=response.chunk_index, - root=challenge.crosslink_data_root, + root=challenge.data_root, ) # Clear the challenge records = state.custody_chunk_challenge_records @@ -632,7 +632,7 @@ def process_bit_challenge_response(state: BeaconState, branch=response.data_branch, depth=math.log2(next_power_of_two(challenge.chunk_count)), index=response.chunk_index, - root=challenge.crosslink_data_root, + root=challenge.data_root, ) # Verify the chunk bit leaf matches the challenge data assert verify_merkle_branch( diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 9fbc69692..316bd0068 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -78,7 +78,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'previous_block_root': Hash, + 'parent_block_root': Hash, 'data': ShardBlockBody, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -93,7 +93,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'previous_block_root': Hash, + 'parent_block_root': Hash, 'body_root': Hash, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -201,7 +201,7 @@ def get_shard_header(block: ShardBlock) -> ShardBlockHeader: slot: block.slot, shard: block.shard, beacon_chain_root: block.beacon_chain_root, - previous_block_root: block.previous_block_root, + parent_block_root: block.parent_block_root, body_root: hash_tree_root(block.body), state_root: block.state_root, attestations: block.attestations, @@ -296,11 +296,11 @@ def is_valid_shard_block(beacon_blocks: List[BeaconBlock], # Check parent block if block.slot == PHASE_1_GENESIS_SLOT: - assert candidate.previous_block_root == ZERO_HASH + assert candidate.parent_block_root == ZERO_HASH else: parent_block = next( block for block in valid_shard_blocks if - signing_root(block) == candidate.previous_block_root + signing_root(block) == candidate.parent_block_root , None) assert parent_block != None assert parent_block.shard == block.shard @@ -378,11 +378,11 @@ def is_valid_beacon_attestation(shard: Shard, # Check previous attestation if candidate.data.previous_crosslink.epoch <= PHASE_1_GENESIS_EPOCH: - assert candidate.data.previous_crosslink.crosslink_data_root == ZERO_HASH + assert candidate.data.previous_crosslink.data_root == ZERO_HASH else: previous_attestation = next( attestation for attestation in valid_attestations if - attestation.data.crosslink.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root + attestation.data.crosslink.data_root == candidate.data.previous_crosslink.data_root , None) assert previous_attestation != None assert candidate.data.previous_attestation.epoch < slot_to_epoch(candidate.data.slot) @@ -393,7 +393,7 @@ def is_valid_beacon_attestation(shard: Shard, blocks = [] for slot in range(start_epoch * SLOTS_PER_EPOCH, end_epoch * SLOTS_PER_EPOCH): blocks.append(shard_blocks[slot]) - assert candidate.data.crosslink.crosslink_data_root == compute_crosslink_data_root(blocks) + assert candidate.data.crosslink.data_root == compute_crosslink_data_root(blocks) return True ``` diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index cfe6a8f1f..dae9ac85f 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -152,7 +152,7 @@ Set `block.slot = slot` where `slot` is the current slot at which the validator ##### Parent root -Set `block.previous_block_root = signing_root(parent)`. +Set `block.parent_block_root = signing_root(parent)`. ##### State root @@ -275,11 +275,11 @@ Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associ ##### Previous crosslink root -Set `attestation_data.previous_crosslink_root = hash_tree_root(head_state.current_crosslinks[shard])`. +Set `attestation_data.parent_crosslink_root = hash_tree_root(head_state.current_crosslinks[shard])`. ##### Crosslink data root -Set `attestation_data.crosslink.crosslink_data_root = ZERO_HASH`. +Set `attestation_data.crosslink.data_root = ZERO_HASH`. *Note*: This is a stub for Phase 0. diff --git a/test_libs/pyspec/tests/block_processing/test_process_attestation.py b/test_libs/pyspec/tests/block_processing/test_process_attestation.py index 24cafc275..fcbc68fe9 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_attestation.py +++ b/test_libs/pyspec/tests/block_processing/test_process_attestation.py @@ -113,7 +113,7 @@ def test_non_zero_crosslink_data_root(state): attestation = get_valid_attestation(state) state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY - attestation.data.crosslink.crosslink_data_root = b'\x42' * 32 + attestation.data.crosslink.data_root = b'\x42' * 32 pre_state, post_state = run_attestation_processing(state, attestation, False) diff --git a/test_libs/pyspec/tests/block_processing/test_process_block_header.py b/test_libs/pyspec/tests/block_processing/test_process_block_header.py index b35b0a9c1..e4fa516b1 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_block_header.py +++ b/test_libs/pyspec/tests/block_processing/test_process_block_header.py @@ -53,9 +53,9 @@ def test_invalid_slot(state): return pre_state, block, None -def test_invalid_previous_block_root(state): +def test_invalid_parent_block_root(state): block = build_empty_block_for_next_slot(state) - block.previous_block_root = b'\12' * 32 # invalid prev root + block.parent_block_root = b'\12' * 32 # invalid prev root pre_state, post_state = run_block_header_processing(state, block, valid=False) return pre_state, block, None diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index 1ea299453..e7ee826db 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -128,7 +128,7 @@ def build_empty_block_for_next_slot(state): previous_block_header = deepcopy(state.latest_block_header) if previous_block_header.state_root == spec.ZERO_HASH: previous_block_header.state_root = state.hash_tree_root() - empty_block.previous_block_root = signing_root(previous_block_header) + empty_block.parent_block_root = signing_root(previous_block_header) return empty_block @@ -155,7 +155,7 @@ def build_attestation_data(state, slot, shard): assert state.slot >= slot if slot == state.slot: - block_root = build_empty_block_for_next_slot(state).previous_block_root + block_root = build_empty_block_for_next_slot(state).parent_block_root else: block_root = get_block_root_at_slot(state, slot) @@ -184,8 +184,8 @@ def build_attestation_data(state, slot, shard): crosslink=Crosslink( shard=shard, epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), - crosslink_data_root=spec.ZERO_HASH, - previous_crosslink_root=hash_tree_root(crosslinks[shard]), + data_root=spec.ZERO_HASH, + parent_crosslink_root=hash_tree_root(crosslinks[shard]), ), ) @@ -240,12 +240,12 @@ def get_valid_proposer_slashing(state): header_1 = BeaconBlockHeader( slot=slot, - previous_block_root=ZERO_HASH, + parent_block_root=ZERO_HASH, state_root=ZERO_HASH, block_body_root=ZERO_HASH, ) header_2 = deepcopy(header_1) - header_2.previous_block_root = b'\x02' * 32 + header_2.parent_block_root = b'\x02' * 32 header_2.slot = slot + 1 domain = get_domain( diff --git a/test_libs/pyspec/tests/test_sanity.py b/test_libs/pyspec/tests/test_sanity.py index 1b4d20f4c..83ba9cc11 100644 --- a/test_libs/pyspec/tests/test_sanity.py +++ b/test_libs/pyspec/tests/test_sanity.py @@ -68,7 +68,7 @@ def test_empty_block_transition(state): state_transition(test_state, block) assert len(test_state.eth1_data_votes) == len(state.eth1_data_votes) + 1 - assert get_block_root_at_slot(test_state, state.slot) == block.previous_block_root + assert get_block_root_at_slot(test_state, state.slot) == block.parent_block_root return state, [block], test_state @@ -82,7 +82,7 @@ def test_skipped_slots(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.previous_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_block_root return state, [block], test_state @@ -96,7 +96,7 @@ def test_empty_epoch_transition(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.previous_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_block_root return state, [block], test_state From ea60fb632c3a07d2dae29b6ca8d37cea204bc6e7 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Mon, 6 May 2019 20:49:46 +0100 Subject: [PATCH 37/45] More renaming --- specs/core/0_beacon-chain.md | 14 +++++++------- specs/core/0_fork-choice.md | 2 +- specs/core/1_shard-data-chains.md | 10 +++++----- specs/validator/0_beacon-chain-validator.md | 4 ++-- .../block_processing/test_process_block_header.py | 2 +- test_libs/pyspec/tests/helpers.py | 12 ++++++------ test_libs/pyspec/tests/test_sanity.py | 6 +++--- 7 files changed, 25 insertions(+), 25 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index afca9eb1b..ccba4c817 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -287,7 +287,7 @@ The types are defined topologically to aid in facilitating an executable version # Epoch number 'epoch': 'uint64', # Root of the previous crosslink - 'parent_crosslink_root': 'bytes32', + 'parent_root': 'bytes32', # Root of the crosslinked shard data since the previous crosslink 'data_root': 'bytes32', } @@ -369,7 +369,7 @@ The types are defined topologically to aid in facilitating an executable version ```python { 'slot': 'uint64', - 'parent_block_root': 'bytes32', + 'parent_root': 'bytes32', 'state_root': 'bytes32', 'block_body_root': 'bytes32', 'signature': 'bytes96', @@ -536,7 +536,7 @@ The types are defined topologically to aid in facilitating an executable version { # Header 'slot': 'uint64', - 'parent_block_root': 'bytes32', + 'parent_root': 'bytes32', 'state_root': 'bytes32', 'body': BeaconBlockBody, 'signature': 'bytes96', @@ -1306,7 +1306,7 @@ def get_attesting_balance(state: BeaconState, attestations: List[PendingAttestat def get_winning_crosslink_and_attesting_indices(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[Crosslink, List[ValidatorIndex]]: attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard] crosslinks = list(filter( - lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_crosslink_root, hash_tree_root(c)), + lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_root, hash_tree_root(c)), [a.data.crosslink for a in attestations] )) # Winning crosslink has the crosslink data root with the most balance voting for it (ties broken lexicographically) @@ -1586,11 +1586,11 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: # Verify that the slots match assert block.slot == state.slot # Verify that the parent matches - assert block.parent_block_root == signing_root(state.latest_block_header) + assert block.parent_root == signing_root(state.latest_block_header) # Save current block as the new latest block state.latest_block_header = BeaconBlockHeader( slot=block.slot, - parent_block_root=block.parent_block_root, + parent_root=block.parent_root, block_body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed @@ -1717,7 +1717,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: # Check FFG data, crosslink data, and signature assert ffg_data == (data.source_epoch, data.source_root, data.target_epoch) assert data.crosslink.epoch == min(data.target_epoch, parent_crosslink.epoch + MAX_EPOCHS_PER_CROSSLINK) - assert data.crosslink.parent_crosslink_root == hash_tree_root(parent_crosslink) + assert data.crosslink.parent_root == hash_tree_root(parent_crosslink) assert data.crosslink.data_root == ZERO_HASH # [to be removed in phase 1] assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) ``` diff --git a/specs/core/0_fork-choice.md b/specs/core/0_fork-choice.md index 891120de8..91c3e27ee 100644 --- a/specs/core/0_fork-choice.md +++ b/specs/core/0_fork-choice.md @@ -38,7 +38,7 @@ All terminology, constants, functions, and protocol mechanics defined in the [Ph Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Clients download and process blocks and maintain a view of what is the current "canonical chain", terminating at the current "head". For a beacon block, `block`, to be processed by a node, the following conditions must be met: -* The parent block with root `block.parent_block_root` has been processed and accepted. +* The parent block with root `block.parent_root` has been processed and accepted. * An Ethereum 1.0 block pointed to by the `state.latest_eth1_data.block_hash` has been processed and accepted. * The node's Unix time is greater than or equal to `state.genesis_time + block.slot * SECONDS_PER_SLOT`. diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 316bd0068..ca59c3ebc 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -78,7 +78,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'parent_block_root': Hash, + 'parent_root': Hash, 'data': ShardBlockBody, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -93,7 +93,7 @@ This document describes the shard data layer and the shard fork choice rule in P 'slot': Slot, 'shard': Shard, 'beacon_chain_root': Hash, - 'parent_block_root': Hash, + 'parent_root': Hash, 'body_root': Hash, 'state_root': Hash, 'attestations': [ShardAttestation], @@ -201,7 +201,7 @@ def get_shard_header(block: ShardBlock) -> ShardBlockHeader: slot: block.slot, shard: block.shard, beacon_chain_root: block.beacon_chain_root, - parent_block_root: block.parent_block_root, + parent_root: block.parent_root, body_root: hash_tree_root(block.body), state_root: block.state_root, attestations: block.attestations, @@ -296,11 +296,11 @@ def is_valid_shard_block(beacon_blocks: List[BeaconBlock], # Check parent block if block.slot == PHASE_1_GENESIS_SLOT: - assert candidate.parent_block_root == ZERO_HASH + assert candidate.parent_root == ZERO_HASH else: parent_block = next( block for block in valid_shard_blocks if - signing_root(block) == candidate.parent_block_root + signing_root(block) == candidate.parent_root , None) assert parent_block != None assert parent_block.shard == block.shard diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index dae9ac85f..a0173ebe9 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -152,7 +152,7 @@ Set `block.slot = slot` where `slot` is the current slot at which the validator ##### Parent root -Set `block.parent_block_root = signing_root(parent)`. +Set `block.parent_root = signing_root(parent)`. ##### State root @@ -275,7 +275,7 @@ Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associ ##### Previous crosslink root -Set `attestation_data.parent_crosslink_root = hash_tree_root(head_state.current_crosslinks[shard])`. +Set `attestation_data.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. ##### Crosslink data root diff --git a/test_libs/pyspec/tests/block_processing/test_process_block_header.py b/test_libs/pyspec/tests/block_processing/test_process_block_header.py index e4fa516b1..36c729821 100644 --- a/test_libs/pyspec/tests/block_processing/test_process_block_header.py +++ b/test_libs/pyspec/tests/block_processing/test_process_block_header.py @@ -55,7 +55,7 @@ def test_invalid_slot(state): def test_invalid_parent_block_root(state): block = build_empty_block_for_next_slot(state) - block.parent_block_root = b'\12' * 32 # invalid prev root + block.parent_root = b'\12' * 32 # invalid prev root pre_state, post_state = run_block_header_processing(state, block, valid=False) return pre_state, block, None diff --git a/test_libs/pyspec/tests/helpers.py b/test_libs/pyspec/tests/helpers.py index e7ee826db..202488576 100644 --- a/test_libs/pyspec/tests/helpers.py +++ b/test_libs/pyspec/tests/helpers.py @@ -128,7 +128,7 @@ def build_empty_block_for_next_slot(state): previous_block_header = deepcopy(state.latest_block_header) if previous_block_header.state_root == spec.ZERO_HASH: previous_block_header.state_root = state.hash_tree_root() - empty_block.parent_block_root = signing_root(previous_block_header) + empty_block.parent_root = signing_root(previous_block_header) return empty_block @@ -155,7 +155,7 @@ def build_attestation_data(state, slot, shard): assert state.slot >= slot if slot == state.slot: - block_root = build_empty_block_for_next_slot(state).parent_block_root + block_root = build_empty_block_for_next_slot(state).parent_root else: block_root = get_block_root_at_slot(state, slot) @@ -185,7 +185,7 @@ def build_attestation_data(state, slot, shard): shard=shard, epoch=min(slot_to_epoch(slot), crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK), data_root=spec.ZERO_HASH, - parent_crosslink_root=hash_tree_root(crosslinks[shard]), + parent_root=hash_tree_root(crosslinks[shard]), ), ) @@ -240,12 +240,12 @@ def get_valid_proposer_slashing(state): header_1 = BeaconBlockHeader( slot=slot, - parent_block_root=ZERO_HASH, + parent_root=ZERO_HASH, state_root=ZERO_HASH, - block_body_root=ZERO_HASH, + body_root=ZERO_HASH, ) header_2 = deepcopy(header_1) - header_2.parent_block_root = b'\x02' * 32 + header_2.parent_root = b'\x02' * 32 header_2.slot = slot + 1 domain = get_domain( diff --git a/test_libs/pyspec/tests/test_sanity.py b/test_libs/pyspec/tests/test_sanity.py index 83ba9cc11..0cb1b9be3 100644 --- a/test_libs/pyspec/tests/test_sanity.py +++ b/test_libs/pyspec/tests/test_sanity.py @@ -68,7 +68,7 @@ def test_empty_block_transition(state): state_transition(test_state, block) assert len(test_state.eth1_data_votes) == len(state.eth1_data_votes) + 1 - assert get_block_root_at_slot(test_state, state.slot) == block.parent_block_root + assert get_block_root_at_slot(test_state, state.slot) == block.parent_root return state, [block], test_state @@ -82,7 +82,7 @@ def test_skipped_slots(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.parent_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_root return state, [block], test_state @@ -96,7 +96,7 @@ def test_empty_epoch_transition(state): assert test_state.slot == block.slot for slot in range(state.slot, test_state.slot): - assert get_block_root_at_slot(test_state, slot) == block.parent_block_root + assert get_block_root_at_slot(test_state, slot) == block.parent_root return state, [block], test_state From 54fd822b55aa6fd9426253716c0e6da70d43b367 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 6 May 2019 17:36:55 -0700 Subject: [PATCH 38/45] Update simple-serialize.md typo --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 343c967d4..834f56645 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -85,7 +85,7 @@ return b"\x01" if value is True else b"\x00" ### Containers, vectors, lists ```python -# Reccursively serialize +# Recursively serialize fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in value] variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in value] From 24966d71be58066dc9586d355ec219220e5bd94a Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 6 May 2019 17:54:40 -0700 Subject: [PATCH 39/45] Update 0_beacon-chain-validator.md --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 600ed0839..0a1997788 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -134,7 +134,7 @@ Once a validator is activated, the validator is assigned [responsibilities](#bea ## Beacon chain responsibilities -A validator has two primary responsibilities to the beacon chain: [proposing blocks](block-proposal) and [creating attestations](attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. +A validator has two primary responsibilities to the beacon chain: [proposing blocks](#block-proposal) and [creating attestations](#attestations-1). Proposals happen infrequently, whereas attestations should be created once per epoch. ### Block proposal From fc1239c0ff8fda299d4a5e5a532bb2203e130446 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 6 May 2019 18:50:20 -0700 Subject: [PATCH 40/45] Add clarity around merkleize on a single chunk --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 834f56645..c02d7fbed 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -111,7 +111,7 @@ Because serialization is an injective function (i.e. two distinct objects of the We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. +* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. Note that `merkleize` on a single chunk is simply that chunk, i.e. the identity when the number of chunks is one. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 964e55cd4a8b8bfecbd70e279c13e69c6c977e22 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 08:52:56 +0100 Subject: [PATCH 41/45] block_body_root => body_root --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ccba4c817..9f80fe173 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -371,7 +371,7 @@ The types are defined topologically to aid in facilitating an executable version 'slot': 'uint64', 'parent_root': 'bytes32', 'state_root': 'bytes32', - 'block_body_root': 'bytes32', + 'body_root': 'bytes32', 'signature': 'bytes96', } ``` @@ -1591,7 +1591,7 @@ def process_block_header(state: BeaconState, block: BeaconBlock) -> None: state.latest_block_header = BeaconBlockHeader( slot=block.slot, parent_root=block.parent_root, - block_body_root=hash_tree_root(block.body), + body_root=hash_tree_root(block.body), ) # Verify proposer is not slashed proposer = state.validator_registry[get_beacon_proposer_index(state)] From b1520ea96766e34397c161cea3518e7e7f0104be Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 7 May 2019 10:33:51 +0100 Subject: [PATCH 42/45] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index dd2d3d1a6..f0798f587 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -75,7 +75,7 @@ - [`compute_committee`](#compute_committee) - [`get_crosslink_committee`](#get_crosslink_committee) - [`get_attesting_indices`](#get_attesting_indices) - - [`int_to_bytes1`, `int_to_bytes2`, ...](#int_to_bytes1-int_to_bytes2-) + - [`int_to_bytes`](#int_to_bytes) - [`bytes_to_int`](#bytes_to_int) - [`get_total_balance`](#get_total_balance) - [`get_domain`](#get_domain) @@ -194,8 +194,8 @@ These configurations are updated for releases, but may be out of sync during `de | `GENESIS_SLOT` | `0` | | `GENESIS_EPOCH` | `0` | | `FAR_FUTURE_EPOCH` | `2**64 - 1` | -| `ZERO_HASH` | `int_to_bytes32(0)` | -| `BLS_WITHDRAWAL_PREFIX_BYTE` | `int_to_bytes1(0)` | +| `ZERO_HASH` | `int_to_bytes(0, length=32)` | +| `BLS_WITHDRAWAL_PREFIX_BYTE` | `int_to_bytes(0, length=1)` | ### Time parameters @@ -830,7 +830,7 @@ def generate_seed(state: BeaconState, return hash( get_randao_mix(state, epoch + LATEST_RANDAO_MIXES_LENGTH - MIN_SEED_LOOKAHEAD) + get_active_index_root(state, epoch) + - int_to_bytes32(epoch) + int_to_bytes(epoch, length=32) ) ``` @@ -851,7 +851,7 @@ def get_beacon_proposer_index(state: BeaconState) -> ValidatorIndex: i = 0 while True: candidate_index = first_committee[(epoch + i) % len(first_committee)] - random_byte = hash(seed + int_to_bytes8(i // 32))[i % 32] + random_byte = hash(seed + int_to_bytes(i // 32, length=8))[i % 32] effective_balance = state.validator_registry[candidate_index].effective_balance if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte: return candidate_index @@ -888,10 +888,10 @@ def get_shuffled_index(index: ValidatorIndex, index_count: int, seed: Bytes32) - # Swap or not (https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf) # See the 'generalized domain' algorithm on page 3 for round in range(SHUFFLE_ROUND_COUNT): - pivot = bytes_to_int(hash(seed + int_to_bytes1(round))[0:8]) % index_count + pivot = bytes_to_int(hash(seed + int_to_bytes(round, length=1))[0:8]) % index_count flip = (pivot - index) % index_count position = max(index, flip) - source = hash(seed + int_to_bytes1(round) + int_to_bytes4(position // 256)) + source = hash(seed + int_to_bytes(round, length=1) + int_to_bytes(position // 256, length=4)) byte = source[(position % 256) // 8] bit = (byte >> (position % 8)) % 2 index = flip if bit else index @@ -934,9 +934,12 @@ def get_attesting_indices(state: BeaconState, return sorted([index for i, index in enumerate(committee) if get_bitfield_bit(bitfield, i) == 0b1]) ``` -### `int_to_bytes1`, `int_to_bytes2`, ... +### `int_to_bytes` -`int_to_bytes1(x): return x.to_bytes(1, 'little')`, `int_to_bytes2(x): return x.to_bytes(2, 'little')`, and so on for all integers, particularly 1, 2, 3, 4, 8, 32, 48, 96. +```python +def int_to_bytes(integer: int, length: int) -> bytes: + return integer.to_bytes(length, 'little') +``` ### `bytes_to_int` @@ -966,7 +969,7 @@ def get_domain(state: BeaconState, """ epoch = get_current_epoch(state) if message_epoch is None else message_epoch fork_version = state.fork.previous_version if epoch < state.fork.epoch else state.fork.current_version - return bytes_to_int(fork_version + int_to_bytes4(domain_type)) + return bytes_to_int(fork_version + int_to_bytes(domain_type, length=4)) ``` ### `get_bitfield_bit` From 22b06d581dcdc84ca3cfedd9e21af529557d14d6 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 10:57:41 +0100 Subject: [PATCH 43/45] Update instances of int_to_bytes --- scripts/phase0/build_spec.py | 8 ++------ specs/core/0_beacon-chain.md | 4 ++-- specs/core/1_shard-data-chains.md | 2 +- specs/light_client/sync_protocol.md | 2 +- test_generators/shuffling/main.py | 2 +- 5 files changed, 7 insertions(+), 11 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index da5845951..28e5049ae 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -15,15 +15,11 @@ from typing import ( from eth2spec.utils.minimal_ssz import * from eth2spec.utils.bls_stub import * -""") - for i in (1, 2, 3, 4, 8, 32, 48, 96): - code_lines.append("def int_to_bytes%d(x): return x.to_bytes(%d, 'little')" % (i, i)) - - code_lines.append(""" - # stub, will get overwritten by real var SLOTS_PER_EPOCH = 64 +def int_to_bytes(integer: int, length: int) -> bytes: + return integer.to_bytes(length, 'little') Slot = NewType('Slot', int) # uint64 Epoch = NewType('Epoch', int) # uint64 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index f0798f587..1e8170036 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -195,7 +195,7 @@ These configurations are updated for releases, but may be out of sync during `de | `GENESIS_EPOCH` | `0` | | `FAR_FUTURE_EPOCH` | `2**64 - 1` | | `ZERO_HASH` | `int_to_bytes(0, length=32)` | -| `BLS_WITHDRAWAL_PREFIX_BYTE` | `int_to_bytes(0, length=1)` | +| `BLS_WITHDRAWAL_PREFIX` | `0` | ### Time parameters @@ -1840,7 +1840,7 @@ def process_transfer(state: BeaconState, transfer: Transfer) -> None: # Verify that the pubkey is valid assert ( state.validator_registry[transfer.sender].withdrawal_credentials == - BLS_WITHDRAWAL_PREFIX_BYTE + hash(transfer.pubkey)[1:] + int_to_bytes(BLS_WITHDRAWAL_PREFIX, length=1) + hash(transfer.pubkey)[1:] ) # Verify that the signature is valid assert bls_verify(transfer.pubkey, signing_root(transfer), transfer.signature, get_domain(state, DOMAIN_TRANSFER)) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 14eb51193..90ea30841 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -180,7 +180,7 @@ def get_shard_proposer_index(state: BeaconState, slot: Slot) -> ValidatorIndex: # Randomly shift persistent committee persistent_committee = get_persistent_committee(state, shard, slot) - seed = hash(state.current_shuffling_seed + int_to_bytes8(shard) + int_to_bytes8(slot)) + seed = hash(state.current_shuffling_seed + int_to_bytes(shard, length=8) + int_to_bytes(slot, length=8)) random_index = bytes_to_int(seed[0:8]) % len(persistent_committee) persistent_committee = persistent_committee[random_index:] + persistent_committee[:random_index] diff --git a/specs/light_client/sync_protocol.md b/specs/light_client/sync_protocol.md index f6e3d2265..8501c5869 100644 --- a/specs/light_client/sync_protocol.md +++ b/specs/light_client/sync_protocol.md @@ -146,7 +146,7 @@ def compute_committee(header: BeaconBlockHeader, ] def get_switchover_epoch(index): return ( - bytes_to_int(hash(validator_memory.earlier_period_data.seed + int_to_bytes3(index))[0:8]) % + bytes_to_int(hash(validator_memory.earlier_period_data.seed + int_to_bytes(index, length=3))[0:8]) % PERSISTENT_COMMITTEE_PERIOD ) diff --git a/test_generators/shuffling/main.py b/test_generators/shuffling/main.py index 2c4faeb8f..9ca3e2d36 100644 --- a/test_generators/shuffling/main.py +++ b/test_generators/shuffling/main.py @@ -15,7 +15,7 @@ def shuffling_case(seed: spec.Bytes32, count: int): @to_tuple def shuffling_test_cases(): - for seed in [spec.hash(spec.int_to_bytes4(seed_init_value)) for seed_init_value in range(30)]: + for seed in [spec.hash(spec.int_to_bytes(seed_init_value, length=4)) for seed_init_value in range(30)]: for count in [0, 1, 2, 3, 5, 10, 33, 100, 1000]: yield shuffling_case(seed, count) From ecc6429b9eaef80b13ee8f9f72daa76930f4ad25 Mon Sep 17 00:00:00 2001 From: Justin Drake Date: Tue, 7 May 2019 15:01:23 +0100 Subject: [PATCH 44/45] Address Danny's comment --- scripts/phase0/build_spec.py | 3 --- specs/core/0_beacon-chain.md | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/scripts/phase0/build_spec.py b/scripts/phase0/build_spec.py index 28e5049ae..e3c431a4a 100644 --- a/scripts/phase0/build_spec.py +++ b/scripts/phase0/build_spec.py @@ -18,9 +18,6 @@ from eth2spec.utils.bls_stub import * # stub, will get overwritten by real var SLOTS_PER_EPOCH = 64 -def int_to_bytes(integer: int, length: int) -> bytes: - return integer.to_bytes(length, 'little') - Slot = NewType('Slot', int) # uint64 Epoch = NewType('Epoch', int) # uint64 Shard = NewType('Shard', int) # uint64 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 1e8170036..a26a4e979 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -194,7 +194,7 @@ These configurations are updated for releases, but may be out of sync during `de | `GENESIS_SLOT` | `0` | | `GENESIS_EPOCH` | `0` | | `FAR_FUTURE_EPOCH` | `2**64 - 1` | -| `ZERO_HASH` | `int_to_bytes(0, length=32)` | +| `ZERO_HASH` | `b'\x00' * 32` | | `BLS_WITHDRAWAL_PREFIX` | `0` | ### Time parameters From 13d2ee696939e192534c0278eb271fe53510caf7 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 7 May 2019 11:49:45 -0600 Subject: [PATCH 45/45] cleanup validator guide for crosslinks --- specs/validator/0_beacon-chain-validator.md | 53 +++++++-------------- 1 file changed, 17 insertions(+), 36 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index a0173ebe9..a95052b86 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -37,14 +37,9 @@ - [Voluntary exits](#voluntary-exits) - [Attestations](#attestations-1) - [Attestation data](#attestation-data) - - [Slot](#slot-1) - - [Beacon block root](#beacon-block-root) - - [Source epoch](#source-epoch) - - [Source root](#source-root) - - [Target root](#target-root) - - [Shard](#shard) - - [Previous crosslink root](#previous-crosslink-root) - - [Crosslink data root](#crosslink-data-root) + - [LMD GHOST vote](#lmd-ghost-vote) + - [FFG vote](#ffg-vote) + - [Crosslink vote](#crosslink-vote) - [Construct attestation](#construct-attestation) - [Data](#data) - [Aggregation bitfield](#aggregation-bitfield) @@ -245,43 +240,29 @@ First the validator should construct `attestation_data`, an [`AttestationData`]( * Let `head_block` be the result of running the fork choice during the assigned slot. * Let `head_state` be the state of `head_block` processed through any empty slots up to the assigned slot. -##### Slot - -Set `attestation_data.slot = head_state.slot`. - -##### Beacon block root +##### LMD GHOST vote Set `attestation_data.beacon_block_root = signing_root(head_block)`. -##### Source epoch +##### FFG vote -Set `attestation_data.source_epoch = head_state.justified_epoch`. +* Set `attestation_data.source_epoch = head_state.justified_epoch`. +* Set `attestation_data.source_root = head_state.current_justified_root`. +* Set `attestation_data.target_epoch = get_current_epoch(head_state)` +* Set `attestation_data.target_root = signing_root(epoch_boundary_block)` where `epoch_boundary_block` is the block at the most recent epoch boundary. -##### Source root - -Set `attestation_data.source_root = head_state.current_justified_root`. - -##### Target root - -Set `attestation_data.target_root = signing_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary. - -*Note*: This can be looked up in the state using: +*Note*: `epoch_boundary_block` can be looked up in the state using: * Let `epoch_start_slot = get_epoch_start_slot(get_current_epoch(head_state))`. -* Set `epoch_boundary = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`. +* Let `epoch_boundary_block = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`. -##### Shard +##### Crosslink vote -Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +Construct `attestation_data.crosslink` via the following -##### Previous crosslink root - -Set `attestation_data.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. - -##### Crosslink data root - -Set `attestation_data.crosslink.data_root = ZERO_HASH`. - -*Note*: This is a stub for Phase 0. +* Set `attestation_data.crosslink.shard = shard` where `shard` is the shard associated with the validator's committee defined by `get_crosslink_committees_at_slot`. +* Set `attestation_data.crosslink.epoch = min(attestation_data.target_epoch, head_state.current_crosslinks[shard].epoch + MAX_EPOCHS_PER_CROSSLINK)`. +* Set `attestation_data.crosslink.parent_root = hash_tree_root(head_state.current_crosslinks[shard])`. +* Set `attestation_data.crosslink.data_root = ZERO_HASH`. *Note*: This is a stub for Phase 0. #### Construct attestation