From e4a1ef16e6a42424d7f617d342183c2d29ba9b56 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Tue, 12 Mar 2019 13:46:58 -0700 Subject: [PATCH 01/30] Add networking specs --- specs/networking/messaging.md | 41 ++++ specs/networking/node-identification.md | 32 +++ specs/networking/rpc-interface.md | 246 ++++++++++++++++++++++++ 3 files changed, 319 insertions(+) create mode 100644 specs/networking/messaging.md create mode 100644 specs/networking/node-identification.md create mode 100644 specs/networking/rpc-interface.md diff --git a/specs/networking/messaging.md b/specs/networking/messaging.md new file mode 100644 index 000000000..e88116f46 --- /dev/null +++ b/specs/networking/messaging.md @@ -0,0 +1,41 @@ +ETH 2.0 Networking Spec - Messaging +=== + +# Abstract + +This specification describes how individual Ethereum 2.0 messages are represented on the wire. + +The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL”, NOT", “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119. + +# Motivation + +This specification seeks to define a messaging protocol that is flexible enough to be changed easily as the ETH 2.0 specification evolves. + +# Specification + +## Message Structure + +An ETH 2.0 message consists of a single byte representing the message version followed by the encoded, potentially compressed body. We separate the message's version from the version included in the `libp2p` protocol path in order to allow encoding and compression schemes to be updated independently of the `libp2p` protocols themselves. + +It is unlikely that more than 255 message versions will need to be supported, so a single byte should suffice. + +Visually, a message looks like this: + +``` ++--------------------------+ +| version byte | ++--------------------------+ +| | +| body | +| | ++--------------------------+ +``` + +Clients MUST ignore messages with mal-formed bodies. The `version` byte MUST be one of the below values: + +## Version Byte Values + +### `0x01` + +- **Encoding Scheme:** SSZ +- **Compression Scheme:** Snappy diff --git a/specs/networking/node-identification.md b/specs/networking/node-identification.md new file mode 100644 index 000000000..27c1ebf9d --- /dev/null +++ b/specs/networking/node-identification.md @@ -0,0 +1,32 @@ +ETH 2.0 Networking Spec - Node Identification +=== + +# Abstract + +This specification describes how Ethereum 2.0 nodes identify and address each other on the network. + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. + +# Specification + +Clients use Ethereum Node Records (as described in [EIP-778](http://eips.ethereum.org/EIPS/eip-778)) to discover one another. Each ENR includes, among other things, the following keys: + +- The node's IP. +- The node's TCP port. +- The node's public key. + +For clients to be addressable, their ENR responses MUST contain all of the above keys. Client MUST verify the signature of any received ENRs, and disconnect from peers whose ENR signatures are invalid. Each node's public key MUST be unique. + +The keys above are enough to construct a [multiaddr](https://github.com/multiformats/multiaddr) for use with the rest of the `libp2p` stack. + +It is RECOMMENDED that clients set their TCP port to the default of `9000`. + +## Peer ID Generation + +The `libp2p` networking stack identifies peers via a "peer ID." Simply put, a node's Peer ID is the SHA2-256 `multihash` of the node's public key. `go-libp2p-crypto` contains the canonical implementation of how to hash `secp256k1` keys for use as a peer ID. + +# See Also + +- [multiaddr](https://github.com/multiformats/multiaddr) +- [multihash](https://multiformats.io/multihash/) +- [go-libp2p-crypto](https://github.com/libp2p/go-libp2p-crypto) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md new file mode 100644 index 000000000..fdc9a11b3 --- /dev/null +++ b/specs/networking/rpc-interface.md @@ -0,0 +1,246 @@ +ETH 2.0 Networking Spec - RPC Interface +=== + +# Abstract + +The Ethereum 2.0 networking stack uses two modes of communication: a broadcast protocol that gossips information to interested parties via GossipSub, and an RPC protocol that retrieves information from specific clients. This specification defines the RPC protocol. + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL", NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. + +# Dependencies + +This specification assumes familiarity with the [Messaging](./messaging.md), [Node Identification](./node-identification), and [Beacon Chain](../core/0_beacon-chain.md) specifications. + +# Specification + +## Message Schemas + +Message body schemas are notated like this: + +``` +( + field_name_1: type + field_name_2: type +) +``` + +SSZ serialization is field-order dependent. Therefore, fields MUST be encoded and decoded according to the order described in this document. The encoded values of each field are concatenated to form the final encoded message body. Embedded structs are serialized as Containers unless otherwise noted. + +All referenced data structures can be found in the [0-beacon-chain](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/core/0_beacon-chain.md#data-structures) specification. + +## `libp2p` Protocol Names + +A "Protocol Name" in `libp2p` parlance refers to a human-readable identifier `libp2p` uses in order to identify sub-protocols and stream messages of different types over the same connection. A client's supported protocol paths are negotiated by the `libp2p` stack at connection time; as such they are not part of individual message bodies. + +## RPC-Over-`libp2p` + +To facilitate RPC-over-`libp2p`, a single protocol path is used: `/eth/serenity/rpc/1.0.0`. Remote method calls are wrapped in a "request" structure: + +``` +( + id: uint64 + method_id: uint16 + body: Request +) +``` + +and their corresponding responses are wrapped in a "response" structure: + +``` +( + id: uint64 + result: Response +) +``` + +If an error occurs, a variant of the response structure is returned: + +``` +( + id: uint64 + error: ( + code: uint16 + data: bytes + ) +) +``` + +The details of the RPC-Over-`libp2p` protocol are similar to [JSON-RPC 2.0](https://www.jsonrpc.org/specification). Specifically: + +1. The `id` member is REQUIRED. +2. The `id` member in the response MUST be the same as the value of the `id` in the request. +3. The `method_id` member is REQUIRED. +4. The `result` member is required on success, and MUST NOT exist if there was an error. +5. The `error` member is REQUIRED on errors, and MUST NOT exist if there wasn't an error. + +Structuring RPC requests in this manner allows multiple calls and responses to be multiplexed over the same stream without switching. + +The "method ID" fields in the below messages refer to the `method` field in the request structure above. + +The first 1,000 values in `error.code` are reserved for system use. The following error codes are predefined: + +1. `0`: Parse error. +2. `10`: Invalid request. +3. `20`: Method not found. +4. `30`: Server error. + +## Messages + +### Hello + +**Method ID:** `0` + +**Body**: + +``` +( + network_id: uint8 + latest_finalized_root: bytes32 + latest_finalized_epoch: uint64 + best_root: bytes32 + best_slot: uint64 +) +``` + +Clients exchange `hello` messages upon connection, forming a two-phase handshake. The first message the initiating client sends MUST be the `hello` message. In response, the receiving client MUST respond with its own `hello` message. + +Clients SHOULD immediately disconnect from one another following the handshake above under the following conditions: + +1. If `network_id` belongs to a different chain, since the client definitionally cannot sync with this client. +2. If the `latest_finalized_root` shared by the peer is not in the client's chain at the expected epoch. For example, if Peer 1 in the diagram below has `(root, epoch)` of `(A, 5)` and Peer 2 has `(B, 3)`, Peer 1 would disconnect because it knows that `B` is not the root in their chain at epoch 3: + +``` + Root A + + +---+ + |xxx| +----+ Epoch 5 + +-+-+ + ^ + | + +-+-+ + | | +----+ Epoch 4 + +-+-+ +Root B ^ + | ++---+ +-+-+ +|xxx+<---+--->+ | +----+ Epoch 3 ++---+ | +---+ + | + +-+-+ + | | +-----------+ Epoch 2 + +-+-+ + ^ + | + +-+-+ + | | +-----------+ Epoch 1 + +---+ +``` + +Once the handshake completes, the client with the higher `latest_finalized_epoch` or `best_slot` (if the clients have equal `latest_finalized_epoch`s) SHOULD send beacon block roots to its counterparty via `beacon_block_roots` (i.e., RPC method `10`). + +### Goodbye + +**Method ID:** `1` + +**Body:** + +``` +( + reason: uint64 +) +``` + +Client MAY send `goodbye` messages upon disconnection. The reason field MUST be one of the following values: + +- `1`: Client shut down. +- `2`: Irrelevant network. +- `3`: Irrelevant shard. + +### Provide Beacon Block Roots + +**Method ID:** `10` + +**Body:** + +``` +# BlockRootSlot +( + block_root: HashTreeRoot + slot: uint64 +) + +( + roots: []BlockRootSlot +) +``` + +Send a list of block roots and slots to the peer. + +### Beacon Block Headers + +**Method ID:** `11` + +**Request Body** + +``` +( + start_root: HashTreeRoot + start_slot: uint64 + max_headers: uint64 + skip_slots: uint64 +) +``` + +**Response Body:** + +``` +( + headers: []BlockHeader +) +``` + +Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `2` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is undefined for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were undefined in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further undefined, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. + +The function of the `skip_slots` parameter helps facilitate light client sync - for example, in [#459](https://github.com/ethereum/eth2.0-specs/issues/459) - and allows clients to balance the peers from whom they request headers. Client could, for instance, request every 10th block from a set of peers where each per has a different starting block in order to populate block data. + +### Beacon Block Bodies + +**Method ID:** `12` + +**Request Body:** + +``` +( + block_roots: []HashTreeRoot +) +``` + +**Response Body:** + +``` +( + block_bodies: []BeaconBlockBody +) +``` + +Requests the `block_bodies` associated with the provided `block_roots` from the peer. Responses MUST return `block_roots` in the order provided in the request. If the receiver does not have a particular `block_root`, it must return a zero-value `block_body` (i.e., a `block_body` container with all zero fields). + +### Beacon Chain State + +**Note:** This section is preliminary, pending the definition of the data structures to be transferred over the wire during fast sync operations. + +**Method ID:** `13` + +**Request Body:** + +``` +( + hashes: []HashTreeRoot +) +``` + +**Response Body:** TBD + +Requests contain the hashes of Merkle tree nodes that when merkelized yield the block's `state_root`. + +The response will contain the values that, when hashed, yield the hashes inside the request body. From 29caafc7567096325c14e7961550c4ba6f7c046b Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 13 Mar 2019 21:52:25 -0700 Subject: [PATCH 02/30] Update specs/networking/rpc-interface.md Co-Authored-By: mslipper --- specs/networking/rpc-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index fdc9a11b3..e59f6a6b1 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -199,7 +199,7 @@ Send a list of block roots and slots to the peer. ) ``` -Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `2` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is undefined for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were undefined in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further undefined, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. +Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `2` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is empty for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were empty in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further empty, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. The function of the `skip_slots` parameter helps facilitate light client sync - for example, in [#459](https://github.com/ethereum/eth2.0-specs/issues/459) - and allows clients to balance the peers from whom they request headers. Client could, for instance, request every 10th block from a set of peers where each per has a different starting block in order to populate block data. From f3bddee7a5dcc8df1dfe0deeea9c875df0911415 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 13 Mar 2019 21:55:48 -0700 Subject: [PATCH 03/30] Update specs/networking/rpc-interface.md Co-Authored-By: mslipper --- specs/networking/rpc-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index e59f6a6b1..e087abe96 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -165,7 +165,7 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MUST be ``` # BlockRootSlot ( - block_root: HashTreeRoot + block_root: bytes32 slot: uint64 ) From 5a9ef0fd982f7c23c55afcfd43e07a022a2878b9 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 13 Mar 2019 21:55:59 -0700 Subject: [PATCH 04/30] Update specs/networking/rpc-interface.md Co-Authored-By: mslipper --- specs/networking/rpc-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index e087abe96..e69f60801 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -201,7 +201,7 @@ Send a list of block roots and slots to the peer. Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `2` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is empty for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were empty in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further empty, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. -The function of the `skip_slots` parameter helps facilitate light client sync - for example, in [#459](https://github.com/ethereum/eth2.0-specs/issues/459) - and allows clients to balance the peers from whom they request headers. Client could, for instance, request every 10th block from a set of peers where each per has a different starting block in order to populate block data. +The function of the `skip_slots` parameter helps facilitate light client sync - for example, in [#459](https://github.com/ethereum/eth2.0-specs/issues/459) - and allows clients to balance the peers from whom they request headers. Clients could, for instance, request every 10th block from a set of peers where each per has a different starting block in order to populate block data. ### Beacon Block Bodies From 22e6212e6f08581aeca48dd6efee5e3c81c78f9a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 13 Mar 2019 21:56:47 -0700 Subject: [PATCH 05/30] Update specs/networking/node-identification.md Co-Authored-By: mslipper --- specs/networking/node-identification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/networking/node-identification.md b/specs/networking/node-identification.md index 27c1ebf9d..0f1f9832b 100644 --- a/specs/networking/node-identification.md +++ b/specs/networking/node-identification.md @@ -23,7 +23,7 @@ It is RECOMMENDED that clients set their TCP port to the default of `9000`. ## Peer ID Generation -The `libp2p` networking stack identifies peers via a "peer ID." Simply put, a node's Peer ID is the SHA2-256 `multihash` of the node's public key. `go-libp2p-crypto` contains the canonical implementation of how to hash `secp256k1` keys for use as a peer ID. +The `libp2p` networking stack identifies peers via a "peer ID." Simply put, a node's Peer ID is the SHA2-256 `multihash` of the node's public key struct (serialized in protobuf, refer to the [Peer ID spec](https://github.com/libp2p/specs/pull/100)). `go-libp2p-crypto` contains the canonical implementation of how to hash `secp256k1` keys for use as a peer ID. # See Also From 863f85c45ab2e3327c8c2e5f620af040b239fb40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Kripalani?= Date: Wed, 13 Mar 2019 21:57:29 -0700 Subject: [PATCH 06/30] Update specs/networking/rpc-interface.md Co-Authored-By: mslipper --- specs/networking/rpc-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index e69f60801..d07e728c9 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -30,7 +30,7 @@ All referenced data structures can be found in the [0-beacon-chain](https://gith ## `libp2p` Protocol Names -A "Protocol Name" in `libp2p` parlance refers to a human-readable identifier `libp2p` uses in order to identify sub-protocols and stream messages of different types over the same connection. A client's supported protocol paths are negotiated by the `libp2p` stack at connection time; as such they are not part of individual message bodies. +A "Protocol ID" in `libp2p` parlance refers to a human-readable identifier `libp2p` uses in order to identify sub-protocols and stream messages of different types over the same connection. Peers exchange supported protocol IDs via the `Identify` protocol upon connection. When opening a new stream, peers pin a particular protocol ID to it, and the stream remains contextualised thereafter. Since messages are sent inside a stream, they do not need to bear the protocol ID. ## RPC-Over-`libp2p` From fba333c79185f8eaa84cad816f82dc124c581988 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Sun, 17 Mar 2019 21:19:12 -0700 Subject: [PATCH 07/30] Updates from review --- specs/networking/rpc-interface.md | 35 ++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index d07e728c9..f505a4663 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -24,7 +24,7 @@ Message body schemas are notated like this: ) ``` -SSZ serialization is field-order dependent. Therefore, fields MUST be encoded and decoded according to the order described in this document. The encoded values of each field are concatenated to form the final encoded message body. Embedded structs are serialized as Containers unless otherwise noted. +Embedded types are serialized as SSZ Containers unless otherwise noted. All referenced data structures can be found in the [0-beacon-chain](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/core/0_beacon-chain.md#data-structures) specification. @@ -34,7 +34,7 @@ A "Protocol ID" in `libp2p` parlance refers to a human-readable identifier `libp ## RPC-Over-`libp2p` -To facilitate RPC-over-`libp2p`, a single protocol path is used: `/eth/serenity/rpc/1.0.0`. Remote method calls are wrapped in a "request" structure: +To facilitate RPC-over-`libp2p`, a single protocol path is used: `/eth/serenity/beacon/rpc/1.0.0`. Remote method calls are wrapped in a "request" structure: ``` ( @@ -49,6 +49,7 @@ and their corresponding responses are wrapped in a "response" structure: ``` ( id: uint64 + is_error: boolean result: Response ) ``` @@ -58,7 +59,8 @@ If an error occurs, a variant of the response structure is returned: ``` ( id: uint64 - error: ( + is_error: boolean + result: ( code: uint16 data: bytes ) @@ -69,11 +71,13 @@ The details of the RPC-Over-`libp2p` protocol are similar to [JSON-RPC 2.0](http 1. The `id` member is REQUIRED. 2. The `id` member in the response MUST be the same as the value of the `id` in the request. -3. The `method_id` member is REQUIRED. -4. The `result` member is required on success, and MUST NOT exist if there was an error. -5. The `error` member is REQUIRED on errors, and MUST NOT exist if there wasn't an error. +3. The `id` member MUST be unique within the context of a single connection. Monotonically increasing `id`s are RECOMMENDED. +4. The `method_id` member is REQUIRED. +5. The `result` member is required on success, and MUST NOT exist if there was an error. +6. The `error` member is REQUIRED on errors, and MUST NOT exist if there wasn't an error. +7. `is_error` MUST be `true` on errors, or `false` otherwise. -Structuring RPC requests in this manner allows multiple calls and responses to be multiplexed over the same stream without switching. +Structuring RPC requests in this manner allows multiple calls and responses to be multiplexed over the same stream without switching. Note that this implies that responses MAY arrive in a different order than requests. The "method ID" fields in the below messages refer to the `method` field in the request structure above. @@ -136,7 +140,7 @@ Root B ^ +---+ ``` -Once the handshake completes, the client with the higher `latest_finalized_epoch` or `best_slot` (if the clients have equal `latest_finalized_epoch`s) SHOULD send beacon block roots to its counterparty via `beacon_block_roots` (i.e., RPC method `10`). +Once the handshake completes, the client with the higher `latest_finalized_epoch` or `best_slot` (if the clients have equal `latest_finalized_epoch`s) SHOULD request beacon block roots from its counterparty via `beacon_block_roots` (i.e., RPC method `10`). ### Goodbye @@ -154,13 +158,20 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MUST be - `1`: Client shut down. - `2`: Irrelevant network. -- `3`: Irrelevant shard. +- `3`: Too many peers. +- `4`: Fault/error. -### Provide Beacon Block Roots +### Request Beacon Block Roots **Method ID:** `10` -**Body:** +**Request Body** + +``` +() +``` + +**Response Body:** ``` # BlockRootSlot @@ -174,7 +185,7 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MUST be ) ``` -Send a list of block roots and slots to the peer. +Send a list of block roots and slots to the requesting peer. ### Beacon Block Headers From 2dce326310cc99adccf083c4a06b7cc09b68d244 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Mon, 18 Mar 2019 16:02:31 -0700 Subject: [PATCH 08/30] Bring back envelope --- specs/networking/messaging.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/specs/networking/messaging.md b/specs/networking/messaging.md index e88116f46..de92fe6d4 100644 --- a/specs/networking/messaging.md +++ b/specs/networking/messaging.md @@ -15,15 +15,17 @@ This specification seeks to define a messaging protocol that is flexible enough ## Message Structure -An ETH 2.0 message consists of a single byte representing the message version followed by the encoded, potentially compressed body. We separate the message's version from the version included in the `libp2p` protocol path in order to allow encoding and compression schemes to be updated independently of the `libp2p` protocols themselves. - -It is unlikely that more than 255 message versions will need to be supported, so a single byte should suffice. +An ETH 2.0 message consists of an envelope that defines the message's compression, encoding, and length followed by the body itself. Visually, a message looks like this: ``` +--------------------------+ -| version byte | +| compression nibble | ++--------------------------+ +| encoding nibble | ++--------------------------+ +| body length (uint64) | +--------------------------+ | | | body | @@ -31,11 +33,12 @@ Visually, a message looks like this: +--------------------------+ ``` -Clients MUST ignore messages with mal-formed bodies. The `version` byte MUST be one of the below values: +Clients MUST ignore messages with mal-formed bodies. The compression/encoding nibbles MUST be one of the following values: -## Version Byte Values +## Compression Nibble Values -### `0x01` +- `0x0`: no compression -- **Encoding Scheme:** SSZ -- **Compression Scheme:** Snappy +## Encoding Nibble Values + +- `0x1`: SSZ From 472d9c5c20a93c0b1608013c03f5ca92a0a9a1d8 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Tue, 19 Mar 2019 15:32:38 -0700 Subject: [PATCH 09/30] Updates from review --- specs/networking/messaging.md | 2 ++ specs/networking/rpc-interface.md | 24 +++++++++++++++++------- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/specs/networking/messaging.md b/specs/networking/messaging.md index de92fe6d4..b64e1d5d8 100644 --- a/specs/networking/messaging.md +++ b/specs/networking/messaging.md @@ -11,6 +11,8 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL This specification seeks to define a messaging protocol that is flexible enough to be changed easily as the ETH 2.0 specification evolves. +Note that while `libp2p` is the chosen networking stack for Ethereum 2.0, as of this writing some clients do not have workable `libp2p` implementations. To allow those clients to communicate, we define a message envelope that includes the body's compression, encoding, and body length. Once `libp2p` is available across all implementations, this message envelope will be removed because `libp2p` will negotiate the values defined in the envelope upfront. + # Specification ## Message Structure diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index f505a4663..ef85f32d5 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -34,7 +34,9 @@ A "Protocol ID" in `libp2p` parlance refers to a human-readable identifier `libp ## RPC-Over-`libp2p` -To facilitate RPC-over-`libp2p`, a single protocol path is used: `/eth/serenity/beacon/rpc/1.0.0`. Remote method calls are wrapped in a "request" structure: +To facilitate RPC-over-`libp2p`, a single protocol name is used: `/eth/serenity/beacon/rpc/1`. The version number in the protocol name is neither backwards or forwards compatible, and will be incremented whenever changes to the below structures are required. + +Remote method calls are wrapped in a "request" structure: ``` ( @@ -88,6 +90,10 @@ The first 1,000 values in `error.code` are reserved for system use. The followin 3. `20`: Method not found. 4. `30`: Server error. +### Alternative for Non-`libp2p` Clients + +Since some clients are waiting for `libp2p` implementations in their respective languages. As such, they MAY listen for raw TCP messages on port `9000`. To distinguish RPC messages from other messages on that port, a byte prefix of `ETH` (`0x455448`) MUST be prepended to all messages. This option will be removed once `libp2p` is ready in all supported languages. + ## Messages ### Hello @@ -154,12 +160,13 @@ Once the handshake completes, the client with the higher `latest_finalized_epoch ) ``` -Client MAY send `goodbye` messages upon disconnection. The reason field MUST be one of the following values: +Client MAY send `goodbye` messages upon disconnection. The reason field MAY be one of the following values: - `1`: Client shut down. - `2`: Irrelevant network. -- `3`: Too many peers. -- `4`: Fault/error. +- `3`: Fault/error. + +Clients MAY define custom goodbye reasons as long as the value is larger than `1000`. ### Request Beacon Block Roots @@ -168,7 +175,10 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MUST be **Request Body** ``` -() +( + start_slot: uint64 + count: uint64 +) ``` **Response Body:** @@ -185,7 +195,7 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MUST be ) ``` -Send a list of block roots and slots to the requesting peer. +Requests a list of block roots and slots from the peer. The `count` parameter MUST be less than or equal to `32768`. ### Beacon Block Headers @@ -210,7 +220,7 @@ Send a list of block roots and slots to the requesting peer. ) ``` -Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `2` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is empty for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were empty in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further empty, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. +Requests beacon block headers from the peer starting from `(start_root, start_slot)`. The response MUST contain no more than `max_headers` headers. `skip_slots` defines the maximum number of slots to skip between blocks. For example, requesting blocks starting at slots `2` a `skip_slots` value of `1` would return the blocks at `[2, 4, 6, 8, 10]`. In cases where a slot is empty for a given slot number, the closest previous block MUST be returned. For example, if slot `4` were empty in the previous example, the returned array would contain `[2, 3, 6, 8, 10]`. If slot three were further empty, the array would contain `[2, 6, 8, 10]` - i.e., duplicate blocks MUST be collapsed. A `skip_slots` value of `0` returns all blocks. The function of the `skip_slots` parameter helps facilitate light client sync - for example, in [#459](https://github.com/ethereum/eth2.0-specs/issues/459) - and allows clients to balance the peers from whom they request headers. Clients could, for instance, request every 10th block from a set of peers where each per has a different starting block in order to populate block data. From 8794d03517ea2b6160f032d6619fe01594f2a645 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Wed, 20 Mar 2019 19:04:04 -0700 Subject: [PATCH 10/30] Updates with Whiteblock --- specs/networking/rpc-interface.md | 59 ++++++++++++++++++++++--------- 1 file changed, 42 insertions(+), 17 deletions(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index ef85f32d5..51dc3a900 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -51,8 +51,8 @@ and their corresponding responses are wrapped in a "response" structure: ``` ( id: uint64 - is_error: boolean - result: Response + response_code: uint16 + result: bytes ) ``` @@ -61,11 +61,8 @@ If an error occurs, a variant of the response structure is returned: ``` ( id: uint64 - is_error: boolean - result: ( - code: uint16 - data: bytes - ) + response_code: uint16 + result: bytes ) ``` @@ -75,20 +72,21 @@ The details of the RPC-Over-`libp2p` protocol are similar to [JSON-RPC 2.0](http 2. The `id` member in the response MUST be the same as the value of the `id` in the request. 3. The `id` member MUST be unique within the context of a single connection. Monotonically increasing `id`s are RECOMMENDED. 4. The `method_id` member is REQUIRED. -5. The `result` member is required on success, and MUST NOT exist if there was an error. -6. The `error` member is REQUIRED on errors, and MUST NOT exist if there wasn't an error. -7. `is_error` MUST be `true` on errors, or `false` otherwise. +5. The `result` member is REQUIRED on success. +6. The `result` member is OPTIONAL on errors, and MAY contain additional information about the error. +7. `response_code` MUST be `0` on success. Structuring RPC requests in this manner allows multiple calls and responses to be multiplexed over the same stream without switching. Note that this implies that responses MAY arrive in a different order than requests. The "method ID" fields in the below messages refer to the `method` field in the request structure above. -The first 1,000 values in `error.code` are reserved for system use. The following error codes are predefined: +The first 1,000 values in `response_code` are reserved for system use. The following response codes are predefined: -1. `0`: Parse error. -2. `10`: Invalid request. -3. `20`: Method not found. -4. `30`: Server error. +1. `0`: No error. +2. `10`: Parse error. +2. `20`: Invalid request. +3. `30`: Method not found. +4. `40`: Server error. ### Alternative for Non-`libp2p` Clients @@ -105,6 +103,7 @@ Since some clients are waiting for `libp2p` implementations in their respective ``` ( network_id: uint8 + chain_id: uint8 latest_finalized_root: bytes32 latest_finalized_epoch: uint64 best_root: bytes32 @@ -168,6 +167,32 @@ Client MAY send `goodbye` messages upon disconnection. The reason field MAY be o Clients MAY define custom goodbye reasons as long as the value is larger than `1000`. +### Get Status + +**Method ID:** `2` + +**Request Body:** + +``` +( + sha: bytes32 + user_agent: bytes + timestamp: uint64 +) +``` + +**Response Body:** + +``` +( + sha: bytes32 + user_agent: bytes + timestamp: uint64 +) +``` + +Returns metadata about the remote node. + ### Request Beacon Block Roots **Method ID:** `10` @@ -195,7 +220,7 @@ Clients MAY define custom goodbye reasons as long as the value is larger than `1 ) ``` -Requests a list of block roots and slots from the peer. The `count` parameter MUST be less than or equal to `32768`. +Requests a list of block roots and slots from the peer. The `count` parameter MUST be less than or equal to `32768`. The slots MUST be returned in ascending slot order. ### Beacon Block Headers @@ -216,7 +241,7 @@ Requests a list of block roots and slots from the peer. The `count` parameter MU ``` ( - headers: []BlockHeader + headers: []BeaconBlockHeader ) ``` From 3ee9fc0cc775a05042f7acbfc46e03ec24d14104 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 22 Mar 2019 06:10:44 -0500 Subject: [PATCH 11/30] Merge attestation verification logic Also rename slashable attestation to standalone attestation to reflect its broader functionality in phase 1. --- specs/core/0_beacon-chain.md | 84 +++++++++++++++++++----------------- 1 file changed, 45 insertions(+), 39 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c29aa113d..a4d5f5ec6 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -28,7 +28,7 @@ - [`Eth1DataVote`](#eth1datavote) - [`AttestationData`](#attestationdata) - [`AttestationDataAndCustodyBit`](#attestationdataandcustodybit) - - [`SlashableAttestation`](#slashableattestation) + - [`StandaloneAttestation`](#standaloneattestation) - [`DepositInput`](#depositinput) - [`DepositData`](#depositdata) - [`BeaconBlockHeader`](#beaconblockheader) @@ -90,7 +90,8 @@ - [`get_domain`](#get_domain) - [`get_bitfield_bit`](#get_bitfield_bit) - [`verify_bitfield`](#verify_bitfield) - - [`verify_slashable_attestation`](#verify_slashable_attestation) + - [`convert_to_standalone`](#convert_to_standalone) + - [`verify_standalone_attestation`](#verify_standalone_attestation) - [`is_double_vote`](#is_double_vote) - [`is_surround_vote`](#is_surround_vote) - [`integer_squareroot`](#integer_squareroot) @@ -187,7 +188,7 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | `SHARD_COUNT` | `2**10` (= 1,024) | | `TARGET_COMMITTEE_SIZE` | `2**7` (= 128) | | `MAX_BALANCE_CHURN_QUOTIENT` | `2**5` (= 32) | -| `MAX_SLASHABLE_ATTESTATION_PARTICIPANTS` | `2**12` (= 4,096) | +| `MAX_ATTESTATION_PARTICIPANTS` | `2**12` (= 4,096) | | `MAX_EXIT_DEQUEUES_PER_EPOCH` | `2**2` (= 4) | | `SHUFFLE_ROUND_COUNT` | 90 | @@ -369,7 +370,7 @@ The types are defined topologically to aid in facilitating an executable version } ``` -#### `SlashableAttestation` +#### `StandaloneAttestation` ```python { @@ -489,10 +490,10 @@ The types are defined topologically to aid in facilitating an executable version ```python { - # First slashable attestation - 'slashable_attestation_1': SlashableAttestation, - # Second slashable attestation - 'slashable_attestation_2': SlashableAttestation, + # First attestation + 'attestation_1': StandaloneAttestation, + # Second attestation + 'attestation_2': StandaloneAttestation, } ``` @@ -1116,7 +1117,7 @@ def get_attestation_participants(state: BeaconState, aggregation_bit = get_bitfield_bit(bitfield, i) if aggregation_bit == 0b1: participants.append(validator_index) - return participants + return sorted(participants) ``` ### `is_power_of_two` @@ -1214,30 +1215,45 @@ def verify_bitfield(bitfield: bytes, committee_size: int) -> bool: return True ``` -### `verify_slashable_attestation` +### `convert_to_standalone` ```python -def verify_slashable_attestation(state: BeaconState, slashable_attestation: SlashableAttestation) -> bool: +def convert_to_standalone(state: BeaconState, attestation: Attestation): """ - Verify validity of ``slashable_attestation`` fields. + Converts an attestation to (almost) standalone-verifiable form """ - if slashable_attestation.custody_bitfield != b'\x00' * len(slashable_attestation.custody_bitfield): # [TO BE REMOVED IN PHASE 1] + return StandaloneAttestation( + validator_indices=get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield), + data=attestation.data, + custody_bitfield=attestation.custody_bitfield, + aggregate_signature=attestation.aggregate_signature + ) +``` + +### `verify_standalone_attestation` + +```python +def verify_standalone_attestation(state: BeaconState, standalone_attestation: StandaloneAttestation) -> bool: + """ + Verify validity of ``standalone_attestation`` fields. + """ + if standalone_attestation.custody_bitfield != b'\x00' * len(standalone_attestation.custody_bitfield): # [TO BE REMOVED IN PHASE 1] return False - if not (1 <= len(slashable_attestation.validator_indices) <= MAX_SLASHABLE_ATTESTATION_PARTICIPANTS): + if not (1 <= len(standalone_attestation.validator_indices) <= MAX_ATTESTATION_PARTICIPANTS): return False - for i in range(len(slashable_attestation.validator_indices) - 1): - if slashable_attestation.validator_indices[i] >= slashable_attestation.validator_indices[i + 1]: + for i in range(len(standalone_attestation.validator_indices) - 1): + if standalone_attestation.validator_indices[i] >= standalone_attestation.validator_indices[i + 1]: return False - if not verify_bitfield(slashable_attestation.custody_bitfield, len(slashable_attestation.validator_indices)): + if not verify_bitfield(standalone_attestation.custody_bitfield, len(standalone_attestation.validator_indices)): return False custody_bit_0_indices = [] custody_bit_1_indices = [] - for i, validator_index in enumerate(slashable_attestation.validator_indices): - if get_bitfield_bit(slashable_attestation.custody_bitfield, i) == 0b0: + for i, validator_index in enumerate(standalone_attestation.validator_indices): + if get_bitfield_bit(standalone_attestation.custody_bitfield, i) == 0b0: custody_bit_0_indices.append(validator_index) else: custody_bit_1_indices.append(validator_index) @@ -1248,11 +1264,11 @@ def verify_slashable_attestation(state: BeaconState, slashable_attestation: Slas bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_indices]), ], message_hashes=[ - hash_tree_root(AttestationDataAndCustodyBit(data=slashable_attestation.data, custody_bit=0b0)), - hash_tree_root(AttestationDataAndCustodyBit(data=slashable_attestation.data, custody_bit=0b1)), + hash_tree_root(AttestationDataAndCustodyBit(data=standalone_attestation.data, custody_bit=0b0)), + hash_tree_root(AttestationDataAndCustodyBit(data=standalone_attestation.data, custody_bit=0b1)), ], - signature=slashable_attestation.aggregate_signature, - domain=get_domain(state.fork, slot_to_epoch(slashable_attestation.data.slot), DOMAIN_ATTESTATION), + signature=standalone_attestation.aggregate_signature, + domain=get_domain(state.fork, slot_to_epoch(standalone_attestation.data.slot), DOMAIN_ATTESTATION), ) ``` @@ -2408,16 +2424,16 @@ def process_attester_slashing(state: BeaconState, Process ``AttesterSlashing`` transaction. Note that this function mutates ``state``. """ - attestation1 = attester_slashing.slashable_attestation_1 - attestation2 = attester_slashing.slashable_attestation_2 + attestation1 = attester_slashing.attestation_1 + attestation2 = attester_slashing.attestation_2 # Check that the attestations are conflicting assert attestation1.data != attestation2.data assert ( is_double_vote(attestation1.data, attestation2.data) or is_surround_vote(attestation1.data, attestation2.data) ) - assert verify_slashable_attestation(state, attestation1) - assert verify_slashable_attestation(state, attestation2) + assert verify_standalone_attestation(state, attestation1) + assert verify_standalone_attestation(state, attestation2) slashable_indices = [ index for index in attestation1.validator_indices if ( @@ -2462,18 +2478,8 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: ), } - # Check custody bits [to be generalised in phase 1] - assert attestation.custody_bitfield == b'\x00' * len(attestation.custody_bitfield) - - # Check aggregate signature [to be generalised in phase 1] - participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) - assert len(participants) != 0 - assert bls_verify( - pubkey=bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in participants]), - message_hash=hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b0)), - signature=attestation.aggregate_signature, - domain=get_domain(state.fork, target_epoch, DOMAIN_ATTESTATION), - ) + # Check signature and bitfields + assert verify_standalone_attestation(state, convert_to_standalone(state, attestation)) # Cache pending attestation pending_attestation = PendingAttestation( From ce18bde5c9cb81a85105bbd6f93980f29dbe714b Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 22 Mar 2019 06:20:38 -0500 Subject: [PATCH 12/30] Simplified sorted index check --- specs/core/0_beacon-chain.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a4d5f5ec6..94784e625 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1243,10 +1243,9 @@ def verify_standalone_attestation(state: BeaconState, standalone_attestation: St if not (1 <= len(standalone_attestation.validator_indices) <= MAX_ATTESTATION_PARTICIPANTS): return False - for i in range(len(standalone_attestation.validator_indices) - 1): - if standalone_attestation.validator_indices[i] >= standalone_attestation.validator_indices[i + 1]: - return False - + if standalone_attestation.validator_indices != sorted(standalone_attestation.validator_indices): + return False + if not verify_bitfield(standalone_attestation.custody_bitfield, len(standalone_attestation.validator_indices)): return False From 80e2553afd675f508a42b42a44a224b97fe2b6f1 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Fri, 22 Mar 2019 09:32:21 -0400 Subject: [PATCH 13/30] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 94784e625..3ae2c7e13 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1220,7 +1220,7 @@ def verify_bitfield(bitfield: bytes, committee_size: int) -> bool: ```python def convert_to_standalone(state: BeaconState, attestation: Attestation): """ - Converts an attestation to (almost) standalone-verifiable form + Convert an attestation to (almost) standalone-verifiable form """ return StandaloneAttestation( validator_indices=get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield), From 5b40baa69eaac7151a6c90b9ce292cef827339b5 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 23 Mar 2019 11:58:20 +0800 Subject: [PATCH 14/30] Adjust the sanity test for attestation verification integration --- tests/phase0/test_sanity.py | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/tests/phase0/test_sanity.py b/tests/phase0/test_sanity.py index 444075a13..f7670c126 100644 --- a/tests/phase0/test_sanity.py +++ b/tests/phase0/test_sanity.py @@ -227,22 +227,18 @@ def test_attestation(state, pubkeys, privkeys): crosslink_committees = get_crosslink_committees_at_slot(state, slot) crosslink_committee = [committee for committee, _shard in crosslink_committees if _shard == attestation_data.shard][0] - committee_size = len(crosslink_committee) - bitfield_length = (committee_size + 7) // 8 - aggregation_bitfield = b'\x01' + b'\x00' * (bitfield_length - 1) - custody_bitfield = b'\x00' * bitfield_length + # Select the first validator to be the attester + participants = [crosslink_committee[0]] + aggregation_bitfield_length = (len(crosslink_committee) + 7) // 8 + custody_bitfield_length = (len(participants) + 7) // 8 + aggregation_bitfield = b'\x01' + b'\x00' * (aggregation_bitfield_length - 1) + custody_bitfield = b'\x00' * custody_bitfield_length attestation = Attestation( aggregation_bitfield=aggregation_bitfield, data=attestation_data, custody_bitfield=custody_bitfield, aggregate_signature=EMPTY_SIGNATURE, ) - participants = get_attestation_participants( - test_state, - attestation.data, - attestation.aggregation_bitfield, - ) - assert len(participants) == 1 validator_index = participants[0] privkey = privkeys[validator_index] From b7441e8ab78560b3b48b3e1bd10de0aef6172080 Mon Sep 17 00:00:00 2001 From: Justin Date: Mon, 25 Mar 2019 14:30:59 +0000 Subject: [PATCH 15/30] Generalise `slash_validator` for phase 1 Make `slash_validator` friendly to phase 1. This is a cosmetic change in the context of phase 0. --- specs/core/0_beacon-chain.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index c29aa113d..2eeee7802 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -251,7 +251,8 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | Name | Value | | - | - | | `BASE_REWARD_QUOTIENT` | `2**5` (= 32) | -| `WHISTLEBLOWER_REWARD_QUOTIENT` | `2**9` (= 512) | +| `WHISTLEBLOWING_REWARD_QUOTIENT` | `2**9` (= 512) | +| `PROPOSER_REWARD_QUOTIENT` | `2**4` (= 16) | | `ATTESTATION_INCLUSION_REWARD_QUOTIENT` | `2**3` (= 8) | | `INACTIVITY_PENALTY_QUOTIENT` | `2**24` (= 16,777,216) | | `MIN_PENALTY_QUOTIENT` | `2**5` (= 32) | @@ -1448,21 +1449,25 @@ def exit_validator(state: BeaconState, index: ValidatorIndex) -> None: #### `slash_validator` ```python -def slash_validator(state: BeaconState, index: ValidatorIndex) -> None: +def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whitleblower_index: ValidatorIndex=None) -> None: """ - Slash the validator with index ``index``. + Slash the validator with index ``slashed_index``. Note that this function mutates ``state``. """ - validator = state.validator_registry[index] - exit_validator(state, index) - state.latest_slashed_balances[get_current_epoch(state) % LATEST_SLASHED_EXIT_LENGTH] += get_effective_balance(state, index) + exit_validator(state, slashed_index) + state.validator_registry[slashed_index].slashed = True + state.validator_registry[slashed_index].withdrawable_epoch = get_current_epoch(state) + LATEST_SLASHED_EXIT_LENGTH + slashed_balance = get_effective_balance(state, slashed_index) + state.latest_slashed_balances[get_current_epoch(state) % LATEST_SLASHED_EXIT_LENGTH] += slashed_balance - whistleblower_index = get_beacon_proposer_index(state, state.slot) - whistleblower_reward = get_effective_balance(state, index) // WHISTLEBLOWER_REWARD_QUOTIENT - increase_balance(state, whistleblower_index, whistleblower_reward) - decrease_balance(state, index, whistleblower_reward) - validator.slashed = True - validator.withdrawable_epoch = get_current_epoch(state) + LATEST_SLASHED_EXIT_LENGTH + proposer_index = get_beacon_proposer_index(state, state.slot) + if whileblower_index is None: + whileblower_index = proposer_index + whistleblowing_reward = slashed_balance // WHISTLEBLOWING_REWARD_QUOTIENT + proposer_reward = whistleblowing_reward // PROPOSER_REWARD_QUOTIENT + increase_balance(state, proposer_index, proposer_reward) + increase_balance(state, whitleblower_index, whistleblowing_reward - proposer_reward) + decrease_balance(state, slashed_index, whistleblower_reward) ``` #### `prepare_validator_for_withdrawal` From fb837400b2b2f5c14fd25d7e875fb1a236b83f64 Mon Sep 17 00:00:00 2001 From: Justin Date: Mon, 25 Mar 2019 14:49:35 +0000 Subject: [PATCH 16/30] Can't spell (thanks continuous integration!) --- specs/core/0_beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2eeee7802..ff142e048 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1449,7 +1449,7 @@ def exit_validator(state: BeaconState, index: ValidatorIndex) -> None: #### `slash_validator` ```python -def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whitleblower_index: ValidatorIndex=None) -> None: +def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whistleblower_index: ValidatorIndex=None) -> None: """ Slash the validator with index ``slashed_index``. Note that this function mutates ``state``. @@ -1461,12 +1461,12 @@ def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whitleblo state.latest_slashed_balances[get_current_epoch(state) % LATEST_SLASHED_EXIT_LENGTH] += slashed_balance proposer_index = get_beacon_proposer_index(state, state.slot) - if whileblower_index is None: - whileblower_index = proposer_index + if whistleblower_index is None: + whistleblower_index = proposer_index whistleblowing_reward = slashed_balance // WHISTLEBLOWING_REWARD_QUOTIENT proposer_reward = whistleblowing_reward // PROPOSER_REWARD_QUOTIENT increase_balance(state, proposer_index, proposer_reward) - increase_balance(state, whitleblower_index, whistleblowing_reward - proposer_reward) + increase_balance(state, whistleblower_index, whistleblowing_reward - proposer_reward) decrease_balance(state, slashed_index, whistleblower_reward) ``` From acc5f314ac601b3887722a9e6a9783dfd075ebfb Mon Sep 17 00:00:00 2001 From: Justin Date: Mon, 25 Mar 2019 14:54:43 +0000 Subject: [PATCH 17/30] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ff142e048..00c229036 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1467,7 +1467,7 @@ def slash_validator(state: BeaconState, slashed_index: ValidatorIndex, whistlebl proposer_reward = whistleblowing_reward // PROPOSER_REWARD_QUOTIENT increase_balance(state, proposer_index, proposer_reward) increase_balance(state, whistleblower_index, whistleblowing_reward - proposer_reward) - decrease_balance(state, slashed_index, whistleblower_reward) + decrease_balance(state, slashed_index, whistleblowing_reward) ``` #### `prepare_validator_for_withdrawal` From 6cc82278b4a1208bc2da94a37f398eb12c96e4e1 Mon Sep 17 00:00:00 2001 From: Matthew Slipper Date: Mon, 25 Mar 2019 13:27:18 -0700 Subject: [PATCH 18/30] Update rpc-interface.md --- specs/networking/rpc-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/networking/rpc-interface.md b/specs/networking/rpc-interface.md index 51dc3a900..fa49bcd75 100644 --- a/specs/networking/rpc-interface.md +++ b/specs/networking/rpc-interface.md @@ -103,7 +103,7 @@ Since some clients are waiting for `libp2p` implementations in their respective ``` ( network_id: uint8 - chain_id: uint8 + chain_id: uint64 latest_finalized_root: bytes32 latest_finalized_epoch: uint64 best_root: bytes32 From 87d2618a495ad382d6810a4fe4b96d4d91f9355f Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 26 Mar 2019 13:21:49 +0000 Subject: [PATCH 19/30] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 00c229036..61ebe5e83 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -252,8 +252,7 @@ Code snippets appearing in `this style` are to be interpreted as Python code. | - | - | | `BASE_REWARD_QUOTIENT` | `2**5` (= 32) | | `WHISTLEBLOWING_REWARD_QUOTIENT` | `2**9` (= 512) | -| `PROPOSER_REWARD_QUOTIENT` | `2**4` (= 16) | -| `ATTESTATION_INCLUSION_REWARD_QUOTIENT` | `2**3` (= 8) | +| `PROPOSER_REWARD_QUOTIENT` | `2**3` (= 8) | | `INACTIVITY_PENALTY_QUOTIENT` | `2**24` (= 16,777,216) | | `MIN_PENALTY_QUOTIENT` | `2**5` (= 32) | @@ -2016,7 +2015,7 @@ def compute_normal_justification_and_finalization_deltas(state: BeaconState) -> # Proposer bonus if index in get_attesting_indices(state, state.previous_epoch_attestations): proposer_index = get_beacon_proposer_index(state, inclusion_slot(state, index)) - deltas[0][proposer_index] += get_base_reward(state, index) // ATTESTATION_INCLUSION_REWARD_QUOTIENT + deltas[0][proposer_index] += get_base_reward(state, index) // PROPOSER_REWARD_QUOTIENT return deltas ``` From 63e7346cfbc4b000c28b981710f43b9ec48a284a Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 26 Mar 2019 13:40:19 -0600 Subject: [PATCH 20/30] standaline -> indexed --- specs/core/0_beacon-chain.md | 52 ++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2acf7ddbe..2e2c3ad59 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -28,7 +28,7 @@ - [`Eth1DataVote`](#eth1datavote) - [`AttestationData`](#attestationdata) - [`AttestationDataAndCustodyBit`](#attestationdataandcustodybit) - - [`StandaloneAttestation`](#standaloneattestation) + - [`IndexedAttestation`](#indexedattestation) - [`DepositData`](#depositdata) - [`BeaconBlockHeader`](#beaconblockheader) - [`Validator`](#validator) @@ -86,8 +86,8 @@ - [`get_domain`](#get_domain) - [`get_bitfield_bit`](#get_bitfield_bit) - [`verify_bitfield`](#verify_bitfield) - - [`convert_to_standalone`](#convert_to_standalone) - - [`verify_standalone_attestation`](#verify_standalone_attestation) + - [`convert_to_indexed`](#convert_to_indexed) + - [`verify_indexed_attestation`](#verify_indexed_attestation) - [`is_double_vote`](#is_double_vote) - [`is_surround_vote`](#is_surround_vote) - [`integer_squareroot`](#integer_squareroot) @@ -370,7 +370,7 @@ The types are defined topologically to aid in facilitating an executable version } ``` -#### `StandaloneAttestation` +#### `IndexedAttestation` ```python { @@ -480,9 +480,9 @@ The types are defined topologically to aid in facilitating an executable version ```python { # First attestation - 'attestation_1': StandaloneAttestation, + 'attestation_1': IndexedAttestation, # Second attestation - 'attestation_2': StandaloneAttestation, + 'attestation_2': IndexedAttestation, } ``` @@ -1148,14 +1148,14 @@ def verify_bitfield(bitfield: bytes, committee_size: int) -> bool: return True ``` -### `convert_to_standalone` +### `convert_to_indexed` ```python -def convert_to_standalone(state: BeaconState, attestation: Attestation): +def convert_to_indexed(state: BeaconState, attestation: Attestation): """ - Convert an attestation to (almost) standalone-verifiable form + Convert an attestation to (almost) indexed-verifiable form """ - return StandaloneAttestation( + return IndexedAttestation( validator_indices=get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield), data=attestation.data, custody_bitfield=attestation.custody_bitfield, @@ -1163,29 +1163,29 @@ def convert_to_standalone(state: BeaconState, attestation: Attestation): ) ``` -### `verify_standalone_attestation` +### `verify_indexed_attestation` ```python -def verify_standalone_attestation(state: BeaconState, standalone_attestation: StandaloneAttestation) -> bool: +def verify_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool: """ - Verify validity of ``standalone_attestation`` fields. + Verify validity of ``indexed_attestation`` fields. """ - if standalone_attestation.custody_bitfield != b'\x00' * len(standalone_attestation.custody_bitfield): # [TO BE REMOVED IN PHASE 1] + if indexed_attestation.custody_bitfield != b'\x00' * len(indexed_attestation.custody_bitfield): # [TO BE REMOVED IN PHASE 1] return False - if not (1 <= len(standalone_attestation.validator_indices) <= MAX_ATTESTATION_PARTICIPANTS): + if not (1 <= len(indexed_attestation.validator_indices) <= MAX_ATTESTATION_PARTICIPANTS): return False - if standalone_attestation.validator_indices != sorted(standalone_attestation.validator_indices): + if indexed_attestation.validator_indices != sorted(indexed_attestation.validator_indices): return False - if not verify_bitfield(standalone_attestation.custody_bitfield, len(standalone_attestation.validator_indices)): + if not verify_bitfield(indexed_attestation.custody_bitfield, len(indexed_attestation.validator_indices)): return False custody_bit_0_indices = [] custody_bit_1_indices = [] - for i, validator_index in enumerate(standalone_attestation.validator_indices): - if get_bitfield_bit(standalone_attestation.custody_bitfield, i) == 0b0: + for i, validator_index in enumerate(indexed_attestation.validator_indices): + if get_bitfield_bit(indexed_attestation.custody_bitfield, i) == 0b0: custody_bit_0_indices.append(validator_index) else: custody_bit_1_indices.append(validator_index) @@ -1196,11 +1196,11 @@ def verify_standalone_attestation(state: BeaconState, standalone_attestation: St bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_indices]), ], message_hashes=[ - hash_tree_root(AttestationDataAndCustodyBit(data=standalone_attestation.data, custody_bit=0b0)), - hash_tree_root(AttestationDataAndCustodyBit(data=standalone_attestation.data, custody_bit=0b1)), + hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b0)), + hash_tree_root(AttestationDataAndCustodyBit(data=indexed_attestation.data, custody_bit=0b1)), ], - signature=standalone_attestation.aggregate_signature, - domain=get_domain(state.fork, slot_to_epoch(standalone_attestation.data.slot), DOMAIN_ATTESTATION), + signature=indexed_attestation.aggregate_signature, + domain=get_domain(state.fork, slot_to_epoch(indexed_attestation.data.slot), DOMAIN_ATTESTATION), ) ``` @@ -2318,8 +2318,8 @@ def process_attester_slashing(state: BeaconState, is_double_vote(attestation1.data, attestation2.data) or is_surround_vote(attestation1.data, attestation2.data) ) - assert verify_standalone_attestation(state, attestation1) - assert verify_standalone_attestation(state, attestation2) + assert verify_indexed_attestation(state, attestation1) + assert verify_indexed_attestation(state, attestation2) slashable_indices = [ index for index in attestation1.validator_indices if ( @@ -2366,7 +2366,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: } # Check signature and bitfields - assert verify_standalone_attestation(state, convert_to_standalone(state, attestation)) + assert verify_indexed_attestation(state, convert_to_indexed(state, attestation)) # Cache pending attestation pending_attestation = PendingAttestation( From 1b975d2ceb669f860b7d7c73f71ad68f939618dc Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 27 Mar 2019 19:23:23 +0600 Subject: [PATCH 21/30] Use signed_root as block id in Honest V guide --- specs/validator/0_beacon-chain-validator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 4a4c63836..0d6033acd 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -152,7 +152,7 @@ _Note:_ there might be "skipped" slots between the `parent` and `block`. These s ##### Parent root -Set `block.previous_block_root = hash_tree_root(parent)`. +Set `block.previous_block_root = signed_root(parent)`. ##### State root @@ -255,11 +255,11 @@ Set `attestation_data.shard = shard` where `shard` is the shard associated with ##### Beacon block root -Set `attestation_data.beacon_block_root = hash_tree_root(head_block)`. +Set `attestation_data.beacon_block_root = signed_root(head_block)`. ##### Target root -Set `attestation_data.target_root = hash_tree_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary. +Set `attestation_data.target_root = signed_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary. _Note:_ This can be looked up in the state using: * Let `epoch_start_slot = get_epoch_start_slot(get_current_epoch(head_state))`. From fbb09795ed3dca6e98eb9ef97c572f4e590293cf Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 27 Mar 2019 08:31:56 -0600 Subject: [PATCH 22/30] fix convert_to_indexed custody bitfield bug --- specs/core/0_beacon-chain.md | 63 +++++++++++++++---- .../test_process_attestation.py | 2 +- tests/phase0/helpers.py | 36 +++++++---- 3 files changed, 78 insertions(+), 23 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2e2c3ad59..0bdfafb79 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -77,6 +77,7 @@ - [`generate_seed`](#generate_seed) - [`get_beacon_proposer_index`](#get_beacon_proposer_index) - [`verify_merkle_branch`](#verify_merkle_branch) + - [`get_crosslink_committee_for_attestation`](#get_crosslink_committee_for_attestation) - [`get_attestation_participants`](#get_attestation_participants) - [`int_to_bytes1`, `int_to_bytes2`, ...](#int_to_bytes1-int_to_bytes2-) - [`bytes_to_int`](#bytes_to_int) @@ -85,6 +86,7 @@ - [`get_fork_version`](#get_fork_version) - [`get_domain`](#get_domain) - [`get_bitfield_bit`](#get_bitfield_bit) + - [`set_bitfield_bit`](#set_bitfield_bit) - [`verify_bitfield`](#verify_bitfield) - [`convert_to_indexed`](#convert_to_indexed) - [`verify_indexed_attestation`](#verify_indexed_attestation) @@ -1037,6 +1039,20 @@ def verify_merkle_branch(leaf: Bytes32, proof: List[Bytes32], depth: int, index: return value == root ``` +### `get_crosslink_committee_for_attestation` + +```python +def get_crosslink_committee_for_attestation(state: BeaconState, + attestation_data: AttestationData) -> List[ValidatorIndex]: + # Find the committee in the list with the desired shard + crosslink_committees = get_crosslink_committees_at_slot(state, attestation_data.slot) + + assert attestation_data.shard in [shard for _, shard in crosslink_committees] + crosslink_committee = [committee for committee, shard in crosslink_committees if shard == attestation_data.shard][0] + + return crosslink_committee +``` + ### `get_attestation_participants` ```python @@ -1046,11 +1062,7 @@ def get_attestation_participants(state: BeaconState, """ Return the participant indices corresponding to ``attestation_data`` and ``bitfield``. """ - # Find the committee in the list with the desired shard - crosslink_committees = get_crosslink_committees_at_slot(state, attestation_data.slot) - - assert attestation_data.shard in [shard for _, shard in crosslink_committees] - crosslink_committee = [committee for committee, shard in crosslink_committees if shard == attestation_data.shard][0] + crosslink_committee = get_crosslink_committee_for_attestation(state, attestation_data) assert verify_bitfield(bitfield, len(crosslink_committee)) @@ -1060,7 +1072,7 @@ def get_attestation_participants(state: BeaconState, aggregation_bit = get_bitfield_bit(bitfield, i) if aggregation_bit == 0b1: participants.append(validator_index) - return sorted(participants) + return participants ``` ### `int_to_bytes1`, `int_to_bytes2`, ... @@ -1130,6 +1142,22 @@ def get_bitfield_bit(bitfield: bytes, i: int) -> int: return (bitfield[i // 8] >> (i % 8)) % 2 ``` +### `set_bitfield_bit` + +```python +def set_bitfield_bit(bitfield: bytes, i: int) -> int: + """ + Set the bit in ``bitfield`` at position ``i`` to ``1``. + """ + byte_index = i // 8 + bit_index = i % 8 + return ( + bitfield[:byte_index] + + bytes([bitfield[byte_index] | (1 << bit_index)]) + + bitfield[byte_index+1:] + ) +``` + ### `verify_bitfield` ```python @@ -1155,10 +1183,21 @@ def convert_to_indexed(state: BeaconState, attestation: Attestation): """ Convert an attestation to (almost) indexed-verifiable form """ + attesting_indices = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) + + # reconstruct custody bitfield for the truncated attesting_indices + custody_bit_1_indices = get_attestation_participants(state, attestation.data, attestation.custody_bitfield) + custody_bitfield = b'\x00' * ((len(attesting_indices) + 7) // 8) + + crosslink_committee = get_crosslink_committee_for_attestation(state, attestation.data) + for i, validator_index in enumerate(crosslink_committee): + if get_bitfield_bit(attestation.custody_bitfield, i): + custody_bitfield = set_bitfield_bit(custody_bitfield, attesting_indices.index(validator_index)) + return IndexedAttestation( - validator_indices=get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield), + validator_indices=attesting_indices, data=attestation.data, - custody_bitfield=attestation.custody_bitfield, + custody_bitfield=custody_bitfield, aggregate_signature=attestation.aggregate_signature ) ``` @@ -1176,9 +1215,6 @@ def verify_indexed_attestation(state: BeaconState, indexed_attestation: IndexedA if not (1 <= len(indexed_attestation.validator_indices) <= MAX_ATTESTATION_PARTICIPANTS): return False - if indexed_attestation.validator_indices != sorted(indexed_attestation.validator_indices): - return False - if not verify_bitfield(indexed_attestation.custody_bitfield, len(indexed_attestation.validator_indices)): return False @@ -2318,6 +2354,11 @@ def process_attester_slashing(state: BeaconState, is_double_vote(attestation1.data, attestation2.data) or is_surround_vote(attestation1.data, attestation2.data) ) + + # check that indices are sorted + assert attestation1.validator_indices == sorted(attestation1.validator_indices) + assert attestation2.validator_indices == sorted(attestation2.validator_indices) + assert verify_indexed_attestation(state, attestation1) assert verify_indexed_attestation(state, attestation2) slashable_indices = [ diff --git a/tests/phase0/block_processing/test_process_attestation.py b/tests/phase0/block_processing/test_process_attestation.py index 08cab11ff..ca6933ce7 100644 --- a/tests/phase0/block_processing/test_process_attestation.py +++ b/tests/phase0/block_processing/test_process_attestation.py @@ -135,7 +135,7 @@ def test_non_empty_custody_bitfield(state): attestation = get_valid_attestation(state) state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY - attestation.custody_bitfield = b'\x01' + attestation.custody_bitfield[1:] + attestation.custody_bitfield = deepcopy(attestation.aggregation_bitfield) pre_state, post_state = run_attestation_processing(state, attestation, False) diff --git a/tests/phase0/helpers.py b/tests/phase0/helpers.py index d7f4ae6e8..08ea6ca04 100644 --- a/tests/phase0/helpers.py +++ b/tests/phase0/helpers.py @@ -22,12 +22,14 @@ from build.phase0.spec import ( get_active_validator_indices, get_attestation_participants, get_block_root, + get_crosslink_committee_for_attestation, get_crosslink_committees_at_slot, get_current_epoch, get_domain, get_empty_block, get_epoch_start_slot, get_genesis_beacon_state, + slot_to_epoch, verify_merkle_branch, hash, ) @@ -248,12 +250,11 @@ def get_valid_attestation(state, slot=None): shard = state.latest_start_shard attestation_data = build_attestation_data(state, slot, shard) - crosslink_committees = get_crosslink_committees_at_slot(state, slot) - crosslink_committee = [committee for committee, _shard in crosslink_committees if _shard == attestation_data.shard][0] + crosslink_committee = get_crosslink_committee_for_attestation(state, attestation_data) committee_size = len(crosslink_committee) bitfield_length = (committee_size + 7) // 8 - aggregation_bitfield = b'\x01' + b'\x00' * (bitfield_length - 1) + aggregation_bitfield = b'\xC0' + b'\x00' * (bitfield_length - 1) custody_bitfield = b'\x00' * bitfield_length attestation = Attestation( aggregation_bitfield=aggregation_bitfield, @@ -266,23 +267,36 @@ def get_valid_attestation(state, slot=None): attestation.data, attestation.aggregation_bitfield, ) - assert len(participants) == 1 + assert len(participants) == 2 - validator_index = participants[0] - privkey = privkeys[validator_index] + signatures = [] + for validator_index in participants: + privkey = privkeys[validator_index] + signatures.append( + get_attestation_signature( + state, + attestation.data, + privkey + ) + ) + + attestation.aggregation_signature = bls.aggregate_signatures(signatures) + return attestation + + +def get_attestation_signature(state, attestation_data, privkey, custody_bit=0b0): message_hash = AttestationDataAndCustodyBit( - data=attestation.data, - custody_bit=0b0, + data=attestation_data, + custody_bit=custody_bit, ).hash_tree_root() - attestation.aggregation_signature = bls.sign( + return bls.sign( message_hash=message_hash, privkey=privkey, domain=get_domain( fork=state.fork, - epoch=get_current_epoch(state), + epoch=slot_to_epoch(attestation_data.slot), domain_type=spec.DOMAIN_ATTESTATION, ) ) - return attestation From b5bf56376bdd2e5e0034965b45c192169575440f Mon Sep 17 00:00:00 2001 From: William M Peaster Date: Wed, 27 Mar 2019 14:00:28 -0500 Subject: [PATCH 23/30] Minor copyediting corrections to 0_beacon-chain.md A handful of minor editing changes made to non-code text for the purposes of improved clarity, consistency, and accuracy. --- specs/core/0_beacon-chain.md | 40 ++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2c0bc2554..92acdd70d 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1,6 +1,6 @@ # Ethereum 2.0 Phase 0 -- The Beacon Chain -**NOTICE**: This document is a work-in-progress for researchers and implementers. It reflects recent spec changes and takes precedence over the Python proof-of-concept implementation [[python-poc]](#ref-python-poc). +**NOTICE**: This document is a work in progress for researchers and implementers. It reflects recent spec changes and takes precedence over the Python proof-of-concept implementation [[python-poc]](#ref-python-poc). ## Table of contents @@ -149,9 +149,9 @@ This document represents the specification for Phase 0 of Ethereum 2.0 -- The Beacon Chain. -At the core of Ethereum 2.0 is a system chain called the "beacon chain". The beacon chain stores and manages the registry of [validators](#dfn-validator). In the initial deployment phases of Ethereum 2.0 the only mechanism to become a [validator](#dfn-validator) is to make a one-way ETH transaction to a deposit contract on Ethereum 1.0. Activation as a [validator](#dfn-validator) happens when Ethereum 1.0 deposit receipts are processed by the beacon chain, the activation balance is reached, and after a queuing process. Exit is either voluntary or done forcibly as a penalty for misbehavior. +At the core of Ethereum 2.0 is a system chain called the "beacon chain". The beacon chain stores and manages the registry of [validators](#dfn-validator). In the initial deployment phases of Ethereum 2.0, the only mechanism to become a [validator](#dfn-validator) is to make a one-way ETH transaction to a deposit contract on Ethereum 1.0. Activation as a [validator](#dfn-validator) happens when Ethereum 1.0 deposit receipts are processed by the beacon chain, the activation balance is reached, and a queuing process is completed. Exit is either voluntary or done forcibly as a penalty for misbehavior. -The primary source of load on the beacon chain is "attestations". Attestations are availability votes for a shard block, and simultaneously proof of stake votes for a beacon block. A sufficient number of attestations for the same shard block create a "crosslink", confirming the shard segment up to that shard block into the beacon chain. Crosslinks also serve as infrastructure for asynchronous cross-shard communication. +The primary source of load on the beacon chain is "attestations". Attestations are simultaneously availability votes for a shard block and proof-of-stake votes for a beacon block. A sufficient number of attestations for the same shard block create a "crosslink", confirming the shard segment up to that shard block into the beacon chain. Crosslinks also serve as infrastructure for asynchronous cross-shard communication. ## Notation @@ -159,20 +159,20 @@ Code snippets appearing in `this style` are to be interpreted as Python code. ## Terminology -* **Validator** - a registered participant in the beacon chain. You can become one by sending Ether into the Ethereum 1.0 deposit contract. +* **Validator** - a registered participant in the beacon chain. You can become one by sending ether into the Ethereum 1.0 deposit contract. * **Active validator** - an active participant in the Ethereum 2.0 consensus invited to, among other things, propose and attest to blocks and vote for crosslinks. * **Committee** - a (pseudo-) randomly sampled subset of [active validators](#dfn-active-validator). When a committee is referred to collectively, as in "this committee attests to X", this is assumed to mean "some subset of that committee that contains enough [validators](#dfn-validator) that the protocol recognizes it as representing the committee". -* **Proposer** - the [validator](#dfn-validator) that creates a beacon chain block +* **Proposer** - the [validator](#dfn-validator) that creates a beacon chain block. * **Attester** - a [validator](#dfn-validator) that is part of a committee that needs to sign off on a beacon chain block while simultaneously creating a link (crosslink) to a recent shard block on a particular shard chain. * **Beacon chain** - the central PoS chain that is the base of the sharding system. * **Shard chain** - one of the chains on which user transactions take place and account data is stored. * **Block root** - a 32-byte Merkle root of a beacon chain block or shard chain block. Previously called "block hash". -* **Crosslink** - a set of signatures from a committee attesting to a block in a shard chain, which can be included into the beacon chain. Crosslinks are the main means by which the beacon chain "learns about" the updated state of shard chains. -* **Slot** - a period during which one proposer has the ability to create a beacon chain block and some attesters have the ability to make attestations -* **Epoch** - an aligned span of slots during which all [validators](#dfn-validator) get exactly one chance to make an attestation -* **Finalized**, **justified** - see Casper FFG finalization [[casper-ffg]](#ref-casper-ffg) -* **Withdrawal period** - the number of slots between a [validator](#dfn-validator) exit and the [validator](#dfn-validator) balance being withdrawable -* **Genesis time** - the Unix time of the genesis beacon chain block at slot 0 +* **Crosslink** - a set of signatures from a committee attesting to a block in a shard chain that can be included into the beacon chain. Crosslinks are the main means by which the beacon chain "learns about" the updated state of shard chains. +* **Slot** - a period during which one proposer has the ability to create a beacon chain block and some attesters have the ability to make attestations. +* **Epoch** - an aligned span of slots during which all [validators](#dfn-validator) get exactly one chance to make an attestation. +* **Finalized**, **justified** - see Casper FFG finalization [[casper-ffg]](#ref-casper-ffg). +* **Withdrawal period** - the number of slots between a [validator](#dfn-validator) exit and the [validator](#dfn-validator) balance being withdrawable. +* **Genesis time** - the Unix time of the genesis beacon chain block at slot 0. ## Constants @@ -871,7 +871,7 @@ def compute_committee(validator_indices: List[ValidatorIndex], ] ``` -**Note**: this definition and the next few definitions are highly inefficient as algorithms as they re-calculate many sub-expressions. Production implementations are expected to appropriately use caching/memoization to avoid redoing work. +**Note**: this definition and the next few definitions are highly inefficient as algorithms, as they re-calculate many sub-expressions. Production implementations are expected to appropriately use caching/memoization to avoid redoing work. ### `get_current_epoch_committee_count` @@ -1426,7 +1426,7 @@ Every Ethereum 1.0 deposit, of size between `MIN_DEPOSIT_AMOUNT` and `MAX_DEPOSI ### `Eth2Genesis` log -When sufficiently many full deposits have been made the deposit contract emits the `Eth2Genesis` log. The beacon chain state may then be initialized by calling the `get_genesis_beacon_state` function (defined below) where: +When a sufficient amount of full deposits have been made, the deposit contract emits the `Eth2Genesis` log. The beacon chain state may then be initialized by calling the `get_genesis_beacon_state` function (defined below) where: * `genesis_time` equals `time` in the `Eth2Genesis` log * `latest_eth1_data.deposit_root` equals `deposit_root` in the `Eth2Genesis` log @@ -1557,13 +1557,13 @@ def get_genesis_beacon_state(genesis_validator_deposits: List[Deposit], ## Beacon chain processing -The beacon chain is the system chain for Ethereum 2.0. The main responsibilities of the beacon chain are: +The beacon chain is the system chain for Ethereum 2.0. The main responsibilities of the beacon chain are as follows: * Store and maintain the registry of [validators](#dfn-validator) * Process crosslinks (see above) * Process its per-block consensus, as well as the finality gadget -Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Clients download and process blocks, and maintain a view of what is the current "canonical chain", terminating at the current "head". However, because of the beacon chain's relationship with Ethereum 1.0, and because it is a proof-of-stake chain, there are differences. +Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Clients download and process blocks and maintain a view of what is the current "canonical chain", terminating at the current "head". However, because of the beacon chain's relationship with Ethereum 1.0, and because it is a proof-of-stake chain, there are differences. For a beacon chain block, `block`, to be processed by a node, the following conditions must be met: @@ -1573,7 +1573,7 @@ For a beacon chain block, `block`, to be processed by a node, the following cond If these conditions are not met, the client should delay processing the beacon block until the conditions are all satisfied. -Beacon block production is significantly different because of the proof of stake mechanism. A client simply checks what it thinks is the canonical chain when it should create a block, and looks up what its slot number is; when the slot arrives, it either proposes or attests to a block as required. Note that this requires each node to have a clock that is roughly (i.e. within `SECONDS_PER_SLOT` seconds) synchronized with the other nodes. +Beacon block production is significantly different because of the proof-of-stake mechanism. A client simply checks what it thinks is the canonical chain when it should create a block and looks up what its slot number is; when the slot arrives, it either proposes or attests to a block as required. Note that this dynamic requires each node to have a clock that is roughly (i.e. within `SECONDS_PER_SLOT` seconds) synchronized with the other nodes. ### Beacon chain fork choice rule @@ -1635,7 +1635,7 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock) ## Beacon chain state transition function -We now define the state transition function. At a high level the state transition is made up of four parts: +We now define the state transition function. At a high level, the state transition is made up of four parts: 1. State caching, which happens at the start of every slot. 2. The per-epoch transitions, which happens at the start of the first slot of every epoch. @@ -1643,7 +1643,7 @@ We now define the state transition function. At a high level the state transitio 4. The per-block transitions, which happens at every block. Transition section notes: -* The state caching, caches the state root of the previous slot. +* The state caching caches the state root of the previous slot. * The per-epoch transitions focus on the [validator](#dfn-validator) registry, including adjusting balances and activating and exiting [validators](#dfn-validator), as well as processing crosslinks and managing block justification/finalization. * The per-slot transitions focus on the slot counter and block roots records updates. * The per-block transitions generally focus on verifying aggregate signatures and saving temporary records relating to the per-block activity in the `BeaconState`. @@ -1876,7 +1876,7 @@ def get_inactivity_penalty(state: BeaconState, index: ValidatorIndex, epochs_sin return get_base_reward(state, index) + extra_penalty ``` -Note: When applying penalties in the following balance recalculations implementers should make sure the `uint64` does not underflow. +Note: When applying penalties in the following balance recalculations, implementers should make sure the `uint64` does not underflow. ##### Justification and finalization @@ -2430,7 +2430,7 @@ def verify_block_state_root(state: BeaconState, block: BeaconBlock) -> None: # References -This section is divided into Normative and Informative references. Normative references are those that must be read in order to implement this specification, while Informative references are merely that, information. An example of the former might be the details of a required consensus algorithm, and an example of the latter might be a pointer to research that demonstrates why a particular consensus algorithm might be better suited for inclusion in the standard than another. +This section is divided into Normative and Informative references. Normative references are those that must be read in order to implement this specification, while Informative references are merely that, informative. An example of the former might be the details of a required consensus algorithm, and an example of the latter might be a pointer to research that demonstrates why a particular consensus algorithm might be better suited for inclusion in the standard than another. ## Normative From 458eb9913364792fef95d78609ffb83fe01cd83e Mon Sep 17 00:00:00 2001 From: William M Peaster Date: Wed, 27 Mar 2019 14:15:50 -0500 Subject: [PATCH 24/30] Minor copyedits to 0_beacon-chain.md Approximately a dozen minor copyediting fixes of non-code text for improved clarity, consistency, and accuracy. --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 92acdd70d..cf6527ad1 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -2430,7 +2430,7 @@ def verify_block_state_root(state: BeaconState, block: BeaconBlock) -> None: # References -This section is divided into Normative and Informative references. Normative references are those that must be read in order to implement this specification, while Informative references are merely that, informative. An example of the former might be the details of a required consensus algorithm, and an example of the latter might be a pointer to research that demonstrates why a particular consensus algorithm might be better suited for inclusion in the standard than another. +This section is divided into Normative and Informative references. Normative references are those that must be read in order to implement this specification, while Informative references are merely helpful information. An example of the former might be the details of a required consensus algorithm, and an example of the latter might be a pointer to research that demonstrates why a particular consensus algorithm might be better suited for inclusion in the standard than another. ## Normative From 1f657cfec50b1c41e53a9183193047fc420d3d8d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 28 Mar 2019 11:26:04 -0600 Subject: [PATCH 25/30] remove custody_bitfield from indexedattestation. add two separate arrays for 0 and 1 bit --- specs/core/0_beacon-chain.md | 45 +++++++++++++----------------------- 1 file changed, 16 insertions(+), 29 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0bdfafb79..057772293 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -377,11 +377,10 @@ The types are defined topologically to aid in facilitating an executable version ```python { # Validator indices - 'validator_indices': ['uint64'], + 'custody_bit_0_indices': ['uint64'], + 'custody_bit_1_indices': ['uint64'], # Attestation data 'data': AttestationData, - # Custody bitfield - 'custody_bitfield': 'bytes', # Aggregate signature 'aggregate_signature': 'bytes96', } @@ -1060,7 +1059,7 @@ def get_attestation_participants(state: BeaconState, attestation_data: AttestationData, bitfield: bytes) -> List[ValidatorIndex]: """ - Return the participant indices corresponding to ``attestation_data`` and ``bitfield``. + Return the sorted participant indices corresponding to ``attestation_data`` and ``bitfield``. """ crosslink_committee = get_crosslink_committee_for_attestation(state, attestation_data) @@ -1072,7 +1071,7 @@ def get_attestation_participants(state: BeaconState, aggregation_bit = get_bitfield_bit(bitfield, i) if aggregation_bit == 0b1: participants.append(validator_index) - return participants + return sorted(participants) ``` ### `int_to_bytes1`, `int_to_bytes2`, ... @@ -1184,20 +1183,13 @@ def convert_to_indexed(state: BeaconState, attestation: Attestation): Convert an attestation to (almost) indexed-verifiable form """ attesting_indices = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) - - # reconstruct custody bitfield for the truncated attesting_indices custody_bit_1_indices = get_attestation_participants(state, attestation.data, attestation.custody_bitfield) - custody_bitfield = b'\x00' * ((len(attesting_indices) + 7) // 8) - - crosslink_committee = get_crosslink_committee_for_attestation(state, attestation.data) - for i, validator_index in enumerate(crosslink_committee): - if get_bitfield_bit(attestation.custody_bitfield, i): - custody_bitfield = set_bitfield_bit(custody_bitfield, attesting_indices.index(validator_index)) + custody_bit_0_indices = [index for index in attesting_indices if index not in custody_bit_1_indices] return IndexedAttestation( - validator_indices=attesting_indices, + custody_bit_0_indices=custody_bit_0_indices, + custody_bit_1_indices=custody_bit_1_indices, data=attestation.data, - custody_bitfield=custody_bitfield, aggregate_signature=attestation.aggregate_signature ) ``` @@ -1209,22 +1201,21 @@ def verify_indexed_attestation(state: BeaconState, indexed_attestation: IndexedA """ Verify validity of ``indexed_attestation`` fields. """ - if indexed_attestation.custody_bitfield != b'\x00' * len(indexed_attestation.custody_bitfield): # [TO BE REMOVED IN PHASE 1] + custody_bit_0_indices = indexed_attestation.custody_bit_0_indices + custody_bit_1_indices = indexed_attestation.custody_bit_1_indices + + if len(custody_bit_1_indices) > 0: # [TO BE REMOVED IN PHASE 1] return False - if not (1 <= len(indexed_attestation.validator_indices) <= MAX_ATTESTATION_PARTICIPANTS): + total_attesting_indices = len(custody_bit_0_indices + custody_bit_1_indices) + if not (1 <= total_attesting_indices <= MAX_ATTESTATION_PARTICIPANTS): return False - if not verify_bitfield(indexed_attestation.custody_bitfield, len(indexed_attestation.validator_indices)): + if custody_bit_0_indices != sorted(custody_bit_0_indices): return False - custody_bit_0_indices = [] - custody_bit_1_indices = [] - for i, validator_index in enumerate(indexed_attestation.validator_indices): - if get_bitfield_bit(indexed_attestation.custody_bitfield, i) == 0b0: - custody_bit_0_indices.append(validator_index) - else: - custody_bit_1_indices.append(validator_index) + if custody_bit_1_indices != sorted(custody_bit_1_indices): + return False return bls_verify_multiple( pubkeys=[ @@ -2355,10 +2346,6 @@ def process_attester_slashing(state: BeaconState, is_surround_vote(attestation1.data, attestation2.data) ) - # check that indices are sorted - assert attestation1.validator_indices == sorted(attestation1.validator_indices) - assert attestation2.validator_indices == sorted(attestation2.validator_indices) - assert verify_indexed_attestation(state, attestation1) assert verify_indexed_attestation(state, attestation2) slashable_indices = [ From ba47a8f4c44adebf613f5507ca48d022141a389c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 28 Mar 2019 11:28:38 -0600 Subject: [PATCH 26/30] remove unused set_bitfield_bit hlper --- specs/core/0_beacon-chain.md | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 057772293..8363d9b22 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -86,7 +86,6 @@ - [`get_fork_version`](#get_fork_version) - [`get_domain`](#get_domain) - [`get_bitfield_bit`](#get_bitfield_bit) - - [`set_bitfield_bit`](#set_bitfield_bit) - [`verify_bitfield`](#verify_bitfield) - [`convert_to_indexed`](#convert_to_indexed) - [`verify_indexed_attestation`](#verify_indexed_attestation) @@ -1141,22 +1140,6 @@ def get_bitfield_bit(bitfield: bytes, i: int) -> int: return (bitfield[i // 8] >> (i % 8)) % 2 ``` -### `set_bitfield_bit` - -```python -def set_bitfield_bit(bitfield: bytes, i: int) -> int: - """ - Set the bit in ``bitfield`` at position ``i`` to ``1``. - """ - byte_index = i // 8 - bit_index = i % 8 - return ( - bitfield[:byte_index] + - bytes([bitfield[byte_index] | (1 << bit_index)]) + - bitfield[byte_index+1:] - ) -``` - ### `verify_bitfield` ```python From eb229089c842cac0445ab5393fe04b28c552b0ce Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 28 Mar 2019 11:31:12 -0600 Subject: [PATCH 27/30] lint --- tests/phase0/helpers.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/phase0/helpers.py b/tests/phase0/helpers.py index 08ea6ca04..e5e335d80 100644 --- a/tests/phase0/helpers.py +++ b/tests/phase0/helpers.py @@ -280,7 +280,6 @@ def get_valid_attestation(state, slot=None): ) ) - attestation.aggregation_signature = bls.aggregate_signatures(signatures) return attestation From 66d5026ffe53e3473ee5f1bd3b3d81a5a8f316e8 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Thu, 28 Mar 2019 13:15:38 -0600 Subject: [PATCH 28/30] minor copy edit --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index cf6527ad1..3a4de1973 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1573,7 +1573,7 @@ For a beacon chain block, `block`, to be processed by a node, the following cond If these conditions are not met, the client should delay processing the beacon block until the conditions are all satisfied. -Beacon block production is significantly different because of the proof-of-stake mechanism. A client simply checks what it thinks is the canonical chain when it should create a block and looks up what its slot number is; when the slot arrives, it either proposes or attests to a block as required. Note that this dynamic requires each node to have a clock that is roughly (i.e. within `SECONDS_PER_SLOT` seconds) synchronized with the other nodes. +Beacon block production is significantly different because of the proof-of-stake mechanism. A client simply checks what it thinks is the canonical chain when it should create a block and looks up what its slot number is; when the slot arrives, it either proposes or attests to a block as required. Note that this requires each node to have a clock that is roughly (i.e. within `SECONDS_PER_SLOT` seconds) synchronized with the other nodes. ### Beacon chain fork choice rule From 1082c68fef660fc66f078ad5442a0065e03e3e71 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 28 Mar 2019 22:54:39 +0000 Subject: [PATCH 29/30] Separate document for phase 1 custody game (#818) The 1-round custody game has been implemented. Many bugs squashed, and a bunch of polishing done. Miscellaneous known issues (~8 of them) to be resolved in separate, smaller, PRs. --- specs/core/1_custody-game.md | 499 +++++++++++++++++++++++++++++++++++ 1 file changed, 499 insertions(+) create mode 100644 specs/core/1_custody-game.md diff --git a/specs/core/1_custody-game.md b/specs/core/1_custody-game.md new file mode 100644 index 000000000..fd754634e --- /dev/null +++ b/specs/core/1_custody-game.md @@ -0,0 +1,499 @@ +# Ethereum 2.0 Phase 1 -- Custody Game + +**NOTICE**: This spec is a work-in-progress for researchers and implementers. + +## Table of contents + + + +- [Ethereum 2.0 Phase 1 -- Custody Game](#ethereum-20-phase-1----custody-game) + - [Table of contents](#table-of-contents) + - [Introduction](#introduction) + - [Terminology](#terminology) + - [Constants](#constants) + - [Misc](#misc) + - [Time parameters](#time-parameters) + - [Max transactions per block](#max-transactions-per-block) + - [Signature domains](#signature-domains) + - [Data structures](#data-structures) + - [Custody objects](#custody-objects) + - [`CustodyChunkChallenge`](#custodychunkchallenge) + - [`CustodyBitChallenge`](#custodybitchallenge) + - [`CustodyChunkChallengeRecord`](#custodychunkchallengerecord) + - [`CustodyBitChallengeRecord`](#custodybitchallengerecord) + - [`CustodyResponse`](#custodyresponse) + - [`CustodyKeyReveal`](#custodykeyreveal) + - [Phase 0 container updates](#phase-0-container-updates) + - [`Validator`](#validator) + - [`BeaconState`](#beaconstate) + - [`BeaconBlockBody`](#beaconblockbody) + - [Helpers](#helpers) + - [`get_crosslink_chunk_count`](#get_crosslink_chunk_count) + - [`get_custody_chunk_bit`](#get_custody_chunk_bit) + - [`epoch_to_custody_period`](#epoch_to_custody_period) + - [`verify_custody_key`](#verify_custody_key) + - [Per-block processing](#per-block-processing) + - [Transactions](#transactions) + - [Custody reveals](#custody-reveals) + - [Chunk challenges](#chunk-challenges) + - [Bit challenges](#bit-challenges) + - [Custody responses](#custody-responses) + - [Per-epoch processing](#per-epoch-processing) + + + +## Introduction + +This document details the beacon chain additions and changes in Phase 1 of Ethereum 2.0 to support the shard data custody game, building upon the [phase 0](0_beacon-chain.md) specification. + +## Terminology + +* **Custody game**: +* **Custody period**: +* **Custody chunk**: +* **Custody chunk bit**: +* **Custody chunk challenge**: +* **Custody bit**: +* **Custody bit challenge**: +* **Custody key**: +* **Custody key reveal**: +* **Custody key mask**: +* **Custody response**: +* **Custody response deadline**: + +## Constants + +### Misc + +| Name | Value | +| - | - | +| `BYTES_PER_SHARD_BLOCK` | `2**14` (= 16,384) | +| `BYTES_PER_CUSTODY_CHUNK` | `2**9` (= 512) | +| `MINOR_REWARD_QUOTIENT` | `2**8` (= 256) | + +### Time parameters + +| Name | Value | Unit | Duration | +| - | - | :-: | :-: | +| `MAX_CHUNK_CHALLENGE_DELAY` | `2**11` (= 2,048) | epochs | ~9 days | +| `EPOCHS_PER_CUSTODY_PERIOD` | `2**11` (= 2,048) | epochs | ~9 days | +| `CUSTODY_RESPONSE_DEADLINE` | `2**14` (= 16,384) | epochs | ~73 days | + +### Max transactions per block + +| Name | Value | +| - | - | +| `MAX_CUSTODY_KEY_REVEALS` | `2**4` (= 16) | +| `MAX_CUSTODY_CHUNK_CHALLENGES` | `2**2` (= 4) | +| `MAX_CUSTODY_BIT_CHALLENGES` | `2**2` (= 4) | +| `MAX_CUSTODY_RESPONSES` | `2**5` (= 32) | + +### Signature domains + +| Name | Value | +| - | - | +| `DOMAIN_CUSTODY_KEY_REVEAL` | `6` | +| `DOMAIN_CUSTODY_BIT_CHALLENGE` | `7` | + +## Data structures + +### Custody objects + +#### `CustodyChunkChallenge` + +```python +{ + 'responder_index': ValidatorIndex, + 'attestation': Attestation, + 'chunk_index': 'uint64', +} +``` + +#### `CustodyBitChallenge` + +```python +{ + 'responder_index': ValidatorIndex, + 'attestation': Attestation, + 'challenger_index': ValidatorIndex, + 'responder_key': BLSSignature, + 'chunk_bits': Bitfield, + 'signature': BLSSignature, +} +``` + +#### `CustodyChunkChallengeRecord` + +```python +{ + 'challenge_index': 'uint64', + 'challenger_index': ValidatorIndex, + 'responder_index': ValidatorIndex, + 'deadline': Epoch, + 'crosslink_data_root': Hash, + 'depth': 'uint64', + 'chunk_index': 'uint64', +} +``` + +#### `CustodyBitChallengeRecord` + +```python +{ + 'challenge_index': 'uint64', + 'challenger_index': ValidatorIndex, + 'responder_index': ValidatorIndex, + 'deadline': Epoch, + 'crosslink_data_root': Hash, + 'chunk_bits': Bitfield, + 'responder_key': BLSSignature, +} +``` + +#### `CustodyResponse` + +```python +{ + 'challenge_index': 'uint64', + 'chunk_index': 'uint64', + 'chunk': ['byte', BYTES_PER_CUSTODY_CHUNK], + 'branch': [Hash], +} +``` + +#### `CustodyKeyReveal` + +```python +{ + 'revealer_index': ValidatorIndex, + 'period': 'uint64', + 'key': BLSSignature, + 'masker_index': ValidatorIndex, + 'mask': Hash, +} +``` + +### Phase 0 container updates + +Add the following fields to the end of the specified container objects. Fields with underlying type `uint64` are initialized to `0` and list fields are initialized to `[]`. + +#### `Validator` + +```python + 'custody_reveal_index': 'uint64', + 'max_reveal_lateness': 'uint64', +``` + +#### `BeaconState` + +```python + 'custody_chunk_challenge_records': [CustodyChunkChallengeRecord], + 'custody_bit_challenge_records': [CustodyBitChallengeRecord], + 'custody_challenge_index': 'uint64', +``` + +#### `BeaconBlockBody` + +```python + 'custody_key_reveals': [CustodyKeyReveal], + 'custody_chunk_challenges': [CustodyChunkChallenge], + 'custody_bit_challenges': [CustodyBitChallenge], + 'custody_responses': [CustodyResponse], +``` + +## Helpers + +### `get_crosslink_chunk_count` + +```python +def get_custody_chunk_count(attestation: Attestation) -> int: + crosslink_start_epoch = attestation.data.latest_crosslink.epoch + crosslink_end_epoch = slot_to_epoch(attestation.data.slot) + crosslink_crosslink_length = min(MAX_CROSSLINK_EPOCHS, end_epoch - start_epoch) + chunks_per_epoch = 2 * BYTES_PER_SHARD_BLOCK * SLOTS_PER_EPOCH // BYTES_PER_CUSTODY_CHUNK + return crosslink_crosslink_length * chunks_per_epoch +``` + +### `get_custody_chunk_bit` + +```python +def get_custody_chunk_bit(key: BLSSignature, chunk: bytes) -> bool: + # TODO: Replace with something MPC-friendly, e.g. the Legendre symbol + return get_bitfield_bit(hash(challenge.responder_key + chunk), 0) +``` + +### `epoch_to_custody_period` + +```python +def epoch_to_custody_period(epoch: Epoch) -> int: + return epoch // EPOCHS_PER_CUSTODY_PERIOD +``` + +### `verify_custody_key` + +```python +def verify_custody_key(state: BeaconState, reveal: CustodyKeyReveal) -> bool: + # Case 1: non-masked non-punitive non-early reveal + pubkeys = [state.validator_registry[reveal.revealer_index].pubkey] + message_hashes = [hash_tree_root(reveal.period)] + + # Case 2: masked punitive early reveal + # Masking prevents proposer stealing the whistleblower reward + # Secure under the aggregate extraction infeasibility assumption + # See pages 11-12 of https://crypto.stanford.edu/~dabo/pubs/papers/aggreg.pdf + if reveal.mask != ZERO_HASH: + pubkeys.append(state.validator_registry[reveal.masker_index].pubkey) + message_hashes.append(reveal.mask) + + return bls_verify_multiple( + pubkeys=pubkeys, + message_hashes=message_hashes, + signature=reveal.key, + domain=get_domain( + fork=state.fork, + epoch=reveal.period * EPOCHS_PER_CUSTODY_PERIOD, + domain_type=DOMAIN_CUSTODY_KEY_REVEAL, + ), + ) +``` + +## Per-block processing + +### Transactions + +Add the following transactions to the per-block processing, in order the given below and after all other transactions in phase 0. + +#### Custody reveals + +Verify that `len(block.body.custody_key_reveals) <= MAX_CUSTODY_KEY_REVEALS`. + +For each `reveal` in `block.body.custody_key_reveals`, run the following function: + +```python +def process_custody_reveal(state: BeaconState, + reveal: CustodyKeyReveal) -> None: + assert verify_custody_key(state, reveal) + revealer = state.validator_registry[reveal.revealer_index] + current_custody_period = epoch_to_custody_period(get_current_epoch(state)) + + # Case 1: non-masked non-punitive non-early reveal + if reveal.mask == ZERO_HASH: + assert reveal.period == epoch_to_custody_period(revealer.activation_epoch) + revealer.custody_reveal_index + # Revealer is active or exited + assert is_active_validator(revealer, get_current_epoch(state)) or revealer.exit_epoch > get_current_epoch(state) + revealer.custody_reveal_index += 1 + revealer.max_reveal_lateness = max(revealer.max_reveal_lateness, current_custody_period - reveal.period) + proposer_index = get_beacon_proposer_index(state, state.slot) + increase_balance(state, proposer_index, base_reward(state, index) // MINOR_REWARD_QUOTIENT) + + # Case 2: masked punitive early reveal + else: + assert reveal.period > current_custody_period + assert revealer.slashed is False + slash_validator(state, reveal.revealer_index, reveal.masker_index) +``` + +#### Chunk challenges + +Verify that `len(block.body.custody_chunk_challenges) <= MAX_CUSTODY_CHUNK_CHALLENGES`. + +For each `challenge` in `block.body.custody_chunk_challenges`, run the following function: + +```python +def process_chunk_challenge(state: BeaconState, + challenge: CustodyChunkChallenge) -> None: + # Verify the attestation + assert verify_standalone_attestation(state, convert_to_standalone(state, challenge.attestation)) + # Verify it is not too late to challenge + assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_CHUNK_CHALLENGE_DELAY + responder = state.validator_registry[challenge.responder_index] + assert responder.exit_epoch >= get_current_epoch(state) - MAX_CHUNK_CHALLENGE_DELAY + # Verify the responder participated in the attestation + attesters = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) + assert challenge.responder_index in attesters + # Verify the challenge is not a duplicate + for record in state.custody_chunk_challenge_records: + assert ( + record.crosslink_data_root != challenge.attestation.data.crosslink_data_root or + record.chunk_index != challenge.chunk_index + ) + # Verify depth + depth = math.log2(next_power_of_two(get_custody_chunk_count(challenge.attestation))) + assert challenge.chunk_index < 2**depth + # Add new chunk challenge record + state.custody_chunk_challenge_records.append(CustodyChunkChallengeRecord( + challenge_index=state.custody_challenge_index, + challenger_index=get_beacon_proposer_index(state, state.slot), + responder_index=challenge.responder_index + deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE, + crosslink_data_root=challenge.attestation.data.crosslink_data_root, + depth=depth, + chunk_index=challenge.chunk_index, + )) + state.custody_challenge_index += 1 + # Postpone responder withdrawability + responder.withdrawable_epoch = FAR_FUTURE_EPOCH +``` + +#### Bit challenges + +Verify that `len(block.body.custody_bit_challenges) <= MAX_CUSTODY_BIT_CHALLENGES`. + +For each `challenge` in `block.body.custody_bit_challenges`, run the following function: + +```python +def process_bit_challenge(state: BeaconState, + challenge: CustodyBitChallenge) -> None: + # Verify challenge signature + challenger = state.validator_registry[challenge.challenger_index] + assert bls_verify( + pubkey=challenger.pubkey, + message_hash=signed_root(challenge), + signature=challenge.signature, + domain=get_domain(state, get_current_epoch(state), DOMAIN_CUSTODY_BIT_CHALLENGE), + ) + # Verify the challenger is not slashed + assert challenger.slashed is False + # Verify the attestation + assert verify_standalone_attestation(state, convert_to_standalone(state, challenge.attestation)) + # Verify the attestation is eligible for challenging + responder = state.validator_registry[challenge.responder_index] + min_challengeable_epoch = responder.exit_epoch - EPOCHS_PER_CUSTODY_PERIOD * (1 + responder.max_reveal_lateness) + assert min_challengeable_epoch <= slot_to_epoch(challenge.attestation.data.slot) + # Verify the responder participated in the attestation + attesters = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield) + assert challenge.responder_index in attesters + # A validator can be the challenger or responder for at most one challenge at a time + for record in state.custody_bit_challenge_records: + assert record.challenger_index != challenge.challenger_index + assert record.responder_index != challenge.responder_index + # Verify the responder key + assert verify_custody_key(state, CustodyKeyReveal( + revealer_index=challenge.responder_index, + period=epoch_to_custody_period(slot_to_epoch(attestation.data.slot)), + key=challenge.responder_key, + masker_index=0, + mask=ZERO_HASH, + )) + # Verify the chunk count + chunk_count = get_custody_chunk_count(challenge.attestation) + assert verify_bitfield(challenge.chunk_bits, chunk_count) + # Verify the xor of the chunk bits does not equal the custody bit + chunk_bits_xor = 0b0 + for i in range(chunk_count): + chunk_bits_xor ^ get_bitfield_bit(challenge.chunk_bits, i) + custody_bit = get_bitfield_bit(attestation.custody_bitfield, attesters.index(responder_index)) + assert custody_bit != chunk_bits_xor + # Add new bit challenge record + state.custody_bit_challenge_records.append(CustodyBitChallengeRecord( + challenge_index=state.custody_challenge_index, + challenger_index=challenge.challenger_index, + responder_index=challenge.responder_index, + deadline=get_current_epoch(state) + CUSTODY_RESPONSE_DEADLINE + crosslink_data_root=challenge.attestation.crosslink_data_root, + chunk_bits=challenge.chunk_bits, + responder_key=challenge.responder_key, + )) + state.custody_challenge_index += 1 + # Postpone responder withdrawability + responder.withdrawable_epoch = FAR_FUTURE_EPOCH +``` + +#### Custody responses + +Verify that `len(block.body.custody_responses) <= MAX_CUSTODY_RESPONSES`. + +For each `response` in `block.body.custody_responses`, run the following function: + +```python +def process_custody_response(state: BeaconState, + response: CustodyResponse) -> None: + chunk_challenge = next(record for record in state.custody_chunk_challenge_records if record.challenge_index == response.challenge_index, None) + if chunk_challenge is not None: + return process_chunk_challenge_response(state, response, chunk_challenge) + + bit_challenge = next(record for record in state.custody_bit_challenge_records if record.challenge_index == response.challenge_index, None) + if bit_challenge is not None: + return process_bit_challenge_response(state, response, bit_challenge) + + assert False +``` + +```python +def process_chunk_challenge_response(state: BeaconState, + response: CustodyResponse, + challenge: CustodyChunkChallengeRecord) -> None: + # Verify chunk index + assert response.chunk_index == challenge.chunk_index + # Verify the chunk matches the crosslink data root + assert verify_merkle_branch( + leaf=hash_tree_root(response.chunk), + branch=response.branch, + depth=challenge.depth, + index=response.chunk_index, + root=challenge.crosslink_data_root, + ) + # Clear the challenge + state.custody_chunk_challenge_records.remove(challenge) + # Reward the proposer + proposer_index = get_beacon_proposer_index(state, state.slot) + increase_balance(state, proposer_index, base_reward(state, index) // MINOR_REWARD_QUOTIENT) +``` + +```python +def process_bit_challenge_response(state: BeaconState, + response: CustodyResponse, + challenge: CustodyBitChallengeRecord) -> None: + # Verify chunk index + assert response.chunk_index < len(challenge.chunk_bits) + # Verify the chunk matches the crosslink data root + assert verify_merkle_branch( + leaf=hash_tree_root(response.chunk), + branch=response.branch, + depth=math.log2(next_power_of_two(len(challenge.chunk_bits))), + index=response.chunk_index, + root=challenge.crosslink_data_root, + ) + # Verify the chunk bit does not match the challenge chunk bit + assert get_custody_chunk_bit(challenge.responder_key, response.chunk) != get_bitfield_bit(challenge.chunk_bits, response.chunk_index) + # Clear the challenge + state.custody_bit_challenge_records.remove(challenge) + # Slash challenger + slash_validator(state, challenge.challenger_index, challenge.responder_index) +``` + +## Per-epoch processing + +Run `process_challenge_deadlines(state)` immediately after `process_ejections(state)`: + +```python +def process_challenge_deadlines(state: BeaconState) -> None: + for challenge in state.custody_chunk_challenge_records: + if get_current_epoch(state) > challenge.deadline: + slash_validator(state, challenge.responder_index, challenge.challenger_index) + state.custody_chunk_challenge_records.remove(challenge) + + for challenge in state.custody_bit_challenge_records: + if get_current_epoch(state) > challenge.deadline: + slash_validator(state, challenge.responder_index, challenge.challenger_index) + state.custody_bit_challenge_records.remove(challenge) +``` + +In `process_penalties_and_exits`, change the definition of `eligible` to the following (note that it is not a pure function because `state` is declared in the surrounding scope): + +```python +def eligible(index): + validator = state.validator_registry[index] + # Cannot exit if there are still open chunk challenges + if len([record for record in state.custody_chunk_challenge_records if record.responder_index == index]) > 0: + return False + # Cannot exit if you have not revealed all of your custody keys + elif epoch_to_custody_period(revealer.activation_epoch) + validator.custody_reveal_index <= epoch_to_custody_period(validator.exit_epoch): + return False + # Cannot exit if you already have + elif validator.withdrawable_epoch < FAR_FUTURE_EPOCH: + return False + # Return minimum time + else: + return current_epoch >= validator.exit_epoch + MIN_VALIDATOR_WITHDRAWAL_EPOCHS +``` From f5c5c166af0caa7d451e5e74d8c565dca3ea4cee Mon Sep 17 00:00:00 2001 From: vbuterin Date: Thu, 28 Mar 2019 17:56:43 -0500 Subject: [PATCH 30/30] Replace custody challenge game with JABS (#812) See also #818. === * Replace custody challenge game with JABS Replace the existing proof of custody game with a new game ("Justin's Awesome Bit Sum" or JABS) that works as follows: * The data `D` is split up into 512-byte chunks `D[0] .... D[n-1]`, and use a mix function `mix(subkey, data) -> {0,1}` (currently the first bit of the hash of `subkey+data`). We calculate `M[i] = (mix(D[0]) + ... + mix(D[i-1])) % 2`, and set the custody bit to `M[n-1]` * Anyone can challenge by providing the full `M` where `M[n-1]` is not equal to the custody bit * Anyone can respond to a challenge by providing a specific position in `M` along with a branch of the data where `M[i-1] ^ mix(D[i]) != M[i]` The maximum size of data is now `2**6` epochs * `2**6` blocks * `2**14` bytes = `2**26` bytes, so assuming 512-byte mix chunks the maximum mix size is `2**17` bits or `2**14` bytes. The average mix size is `2**8` bytes. --- specs/core/1_shard-data-chains.md | 1085 ++++++++--------------------- 1 file changed, 288 insertions(+), 797 deletions(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 92cee4d19..8f2d12a91 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -1,157 +1,143 @@ # Ethereum 2.0 Phase 1 -- Shard Data Chains -**NOTICE**: This document is a work-in-progress for researchers and implementers. It reflects recent spec changes and takes precedence over the [Python proof-of-concept implementation](https://github.com/ethereum/beacon_chain). +**NOTICE**: This document is a work-in-progress for researchers and implementers. -At the current stage, Phase 1, while fundamentally feature-complete, is still subject to change. Development teams with spare resources may consider starting on the "Shard chains and crosslink data" section; at least basic properties, such as the fact that a shard block can get created every slot and is dependent on both a parent block in the same shard and a beacon chain block at or before that same slot, are unlikely to change, though details are likely to undergo similar kinds of changes to what Phase 0 has undergone since the start of the year. - -## Table of contents +## Table of Contents -- [Ethereum 2.0 Phase 1 -- Shard Data Chains](#ethereum-20-phase-1----shard-data-chains) - - [Table of contents](#table-of-contents) - - [Introduction](#introduction) - - [Terminology](#terminology) - - [Constants](#constants) - - [Misc](#misc) - - [Time parameters](#time-parameters) - - [Max operations per block](#max-operations-per-block) - - [Signature domains](#signature-domains) -- [Shard chains and crosslink data](#shard-chains-and-crosslink-data) - - [Helper functions](#helper-functions) - - [`get_shuffled_committee`](#get_shuffled_committee) - - [`get_persistent_committee`](#get_persistent_committee) - - [`get_shard_proposer_index`](#get_shard_proposer_index) - - [Data Structures](#data-structures) - - [Shard chain blocks](#shard-chain-blocks) - - [Shard block processing](#shard-block-processing) - - [Verifying shard block data](#verifying-shard-block-data) - - [Verifying a crosslink](#verifying-a-crosslink) - - [Shard block fork choice rule](#shard-block-fork-choice-rule) -- [Updates to the beacon chain](#updates-to-the-beacon-chain) +- [Ethereum 2.0 Phase 1 -- Shards Data Chains](#ethereum-20-phase-1----shard-data-chains) + - [Table of Contents](#table-of-contents) + - [Introduction](#introduction) + - [Constants](#constants) + - [Misc](#misc) + - [Time parameters](#time-parameters) + - [Signature domains](#signature-domains) - [Data structures](#data-structures) - - [`Validator`](#validator) - - [`BeaconBlockBody`](#beaconblockbody) - - [`BeaconState`](#beaconstate) - - [`BranchChallenge`](#branchchallenge) - - [`BranchResponse`](#branchresponse) - - [`BranchChallengeRecord`](#branchchallengerecord) - - [`InteractiveCustodyChallengeRecord`](#interactivecustodychallengerecord) - - [`InteractiveCustodyChallengeInitiation`](#interactivecustodychallengeinitiation) - - [`InteractiveCustodyChallengeResponse`](#interactivecustodychallengeresponse) - - [`InteractiveCustodyChallengeContinuation`](#interactivecustodychallengecontinuation) - - [`SubkeyReveal`](#subkeyreveal) - - [Helpers](#helpers) - - [`get_branch_challenge_record_by_id`](#get_branch_challenge_record_by_id) - - [`get_custody_challenge_record_by_id`](#get_custody_challenge_record_by_id) - - [`get_attestation_merkle_depth`](#get_attestation_merkle_depth) - - [`epoch_to_custody_period`](#epoch_to_custody_period) - - [`slot_to_custody_period`](#slot_to_custody_period) - - [`get_current_custody_period`](#get_current_custody_period) - - [`verify_custody_subkey_reveal`](#verify_custody_subkey_reveal) - - [`verify_signed_challenge_message`](#verify_signed_challenge_message) - - [`penalize_validator`](#penalize_validator) - - [Per-slot processing](#per-slot-processing) - - [Operations](#operations) - - [Branch challenges](#branch-challenges) - - [Branch responses](#branch-responses) - - [Subkey reveals](#subkey-reveals) - - [Interactive custody challenge initiations](#interactive-custody-challenge-initiations) - - [Interactive custody challenge responses](#interactive-custody-challenge-responses) - - [Interactive custody challenge continuations](#interactive-custody-challenge-continuations) - - [Per-epoch processing](#per-epoch-processing) - - [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition) + - [`ShardBlockBody`](#shardblockbody) + - [`ShardBlock`](#shardblock) + - [`ShardBlockHeader`](#shardblockheader) + - [`ShardAttestation`](#shardattestation) + - [Helper functions](#helper-functions) + - [`get_period_committee`](#get_period_committee) + - [`get_persistent_committee`](#get_persistent_committee) + - [`get_shard_proposer_index`](#get_shard_proposer_index) + - [`get_shard_header`](#get_shard_header) + - [`verify_shard_attestation_signature`](#verify_shard_attestation_signature) + - [`compute_crosslink_data_root`](#compute_crosslink_data_root) + - [Object validity](#object-validity) + - [Shard blocks](#shard-blocks) + - [Shard attestations](#shard-attestations) + - [Beacon attestations](#beacon-attestations) + - [Shard fork choice rule](#shard-fork-choice-rule) -### Introduction +## Introduction -This document represents the specification for Phase 1 of Ethereum 2.0 -- Shard Data Chains. Phase 1 depends on the implementation of [Phase 0 -- The Beacon Chain](0_beacon-chain.md). +This document describes the shard data layer and the shard fork choice rule in Phase 1 of Ethereum 2.0. -Ethereum 2.0 consists of a central beacon chain along with `SHARD_COUNT` shard chains. Phase 1 is primarily concerned with the construction, validity, and consensus on the _data_ of these shard chains. Phase 1 does not specify shard chain state execution or account balances. This is left for future phases. +## Constants -### Terminology +### Misc -### Constants +| Name | Value | +| - | - | +| `BYTES_PER_SHARD_BLOCK_BODY` | `2**14` (= 16,384) | +| `MAX_SHARD_ATTESTIONS` | `2**4` (= 16) | +| `PHASE_1_GENESIS_EPOCH` | **TBD** | +| `PHASE_1_GENESIS_SLOT` | get_epoch_start_slot(PHASE_1_GENESIS_EPOCH) | -Phase 1 depends upon all of the constants defined in [Phase 0](0_beacon-chain.md#constants) in addition to the following: - -#### Misc - -| Name | Value | Unit | -|-------------------------------|------------------|--------| -| `SHARD_CHUNK_SIZE` | 2**5 (= 32) | bytes | -| `SHARD_BLOCK_SIZE` | 2**14 (= 16,384) | bytes | -| `MINOR_REWARD_QUOTIENT` | 2**8 (= 256) | | -| `MAX_POC_RESPONSE_DEPTH` | 5 | | -| `ZERO_PUBKEY` | int_to_bytes48(0)| | -| `VALIDATOR_NULL` | 2**64 - 1 | | - -#### Time parameters +### Time parameters | Name | Value | Unit | Duration | | - | - | :-: | :-: | -| `CROSSLINK_LOOKBACK` | 2**5 (= 32) | slots | 3.2 minutes | -| `MAX_BRANCH_CHALLENGE_DELAY` | 2**11 (= 2,048) | epochs | 9 days | -| `CUSTODY_PERIOD_LENGTH` | 2**11 (= 2,048) | epochs | 9 days | -| `PERSISTENT_COMMITTEE_PERIOD` | 2**11 (= 2,048) | epochs | 9 days | -| `CHALLENGE_RESPONSE_DEADLINE` | 2**14 (= 16,384) | epochs | 73 days | +| `CROSSLINK_LOOKBACK` | 2**0 (= 1) | epochs | 6.2 minutes | +| `PERSISTENT_COMMITTEE_PERIOD` | 2**11 (= 2,048) | epochs | ~9 days | -#### Max operations per block +### Signature domains -| Name | Value | -|----------------------------------------------------|---------------| -| `MAX_BRANCH_CHALLENGES` | 2**2 (= 4) | -| `MAX_BRANCH_RESPONSES` | 2**4 (= 16) | -| `MAX_EARLY_SUBKEY_REVEALS` | 2**4 (= 16) | -| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_INITIATIONS` | 2 | -| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_RESPONSES` | 16 | -| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_CONTINUTATIONS` | 16 | +| Name | Value | +| - | - | +| `DOMAIN_SHARD_PROPOSER` | `128` | +| `DOMAIN_SHARD_ATTESTER` | `129` | -#### Signature domains +## Data structures -| Name | Value | -|------------------------------|-----------------| -| `DOMAIN_SHARD_PROPOSER` | 129 | -| `DOMAIN_SHARD_ATTESTER` | 130 | -| `DOMAIN_CUSTODY_SUBKEY` | 131 | -| `DOMAIN_CUSTODY_INTERACTIVE` | 132 | +### `ShardBlockBody` -# Shard chains and crosslink data +```python +['byte', BYTES_PER_SHARD_BLOCK_BODY] +``` + +### `ShardBlock` + +```python +{ + 'slot': Slot, + 'shard': Shard, + 'beacon_chain_root': Hash, + 'previous_block_root': Hash, + 'data': ShardBlockBody, + 'state_root': Hash, + 'attestations': [ShardAttestation], + 'signature': BLSSignature, +} +``` + +### `ShardBlockHeader` + +```python +{ + 'slot': Slot, + 'shard': Shard, + 'beacon_chain_root': Hash, + 'previous_block_root': Hash, + 'body_root': Hash, + 'state_root': Hash, + 'attestations': [ShardAttestation], + 'signature': BLSSignature, +} +``` + +### `ShardAttestation` + +```python +{ + 'data': { + 'slot': Slot, + 'shard': Shard, + 'shard_block_root': Hash, + }, + 'aggregation_bitfield': Bitfield, + 'aggregate_signature': BLSSignature, +} +``` ## Helper functions -#### `get_shuffled_committee` +### `get_period_committee` ```python -def get_shuffled_committee(state: BeaconState, - shard: Shard, - committee_start_epoch: Epoch, - index: int, - committee_count: int) -> List[ValidatorIndex]: +def get_period_committee(state: BeaconState, + shard: Shard, + committee_start_epoch: Epoch, + index: int, + committee_count: int) -> List[ValidatorIndex]: """ - Return shuffled committee. + Return committee for a period. Used to construct persistent committees. """ active_validator_indices = get_active_validator_indices(state.validator_registry, committee_start_epoch) - length = len(active_validator_indices) seed = generate_seed(state, committee_start_epoch) - start_offset = get_split_offset( - length, - SHARD_COUNT * committee_count, - shard * committee_count + index, + return compute_committee( + validator_indices=active_validator_indices, + seed=seed, + index=shard * committee_count + index, + total_committees=SHARD_COUNT * committee_count, ) - end_offset = get_split_offset( - length, - SHARD_COUNT * committee_count, - shard * committee_count + index + 1, - ) - return [ - active_validator_indices[get_permuted_index(i, length, seed)] - for i in range(start_offset, end_offset) - ] ``` -#### `get_persistent_committee` +### `get_persistent_committee` ```python def get_persistent_committee(state: BeaconState, @@ -160,7 +146,6 @@ def get_persistent_committee(state: BeaconState, """ Return the persistent committee for the given ``shard`` at the given ``slot``. """ - earlier_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD * 2 later_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD @@ -172,14 +157,11 @@ def get_persistent_committee(state: BeaconState, ) + 1 index = slot % committee_count - earlier_committee = get_shuffled_committee(state, shard, earlier_start_epoch, index, committee_count) - later_committee = get_shuffled_committee(state, shard, later_start_epoch, index, committee_count) + earlier_committee = get_period_committee(state, shard, earlier_start_epoch, index, committee_count) + later_committee = get_period_committee(state, shard, later_start_epoch, index, committee_count) def get_switchover_epoch(index): - return ( - bytes_to_int(hash(earlier_seed + bytes3(index))[0:8]) % - PERSISTENT_COMMITTEE_PERIOD - ) + return bytes_to_int(hash(earlier_seed + bytes3(index))[0:8]) % PERSISTENT_COMMITTEE_PERIOD # Take not-yet-cycled-out validators from earlier committee and already-cycled-in validators from # later committee; return a sorted list of the union of the two, deduplicated @@ -189,723 +171,232 @@ def get_persistent_committee(state: BeaconState, ))) ``` -#### `get_shard_proposer_index` +### `get_shard_proposer_index` ```python def get_shard_proposer_index(state: BeaconState, shard: Shard, slot: Slot) -> ValidatorIndex: - seed = hash( - state.current_shuffling_seed + - int_to_bytes8(shard) + - int_to_bytes8(slot) - ) + # Randomly shift persistent committee persistent_committee = get_persistent_committee(state, shard, slot) - # Default proposer - index = bytes_to_int(seed[0:8]) % len(persistent_committee) - # If default proposer exits, try the other proposers in order; if all are exited - # return None (ie. no block can be proposed) - validators_to_try = persistent_committee[index:] + persistent_committee[:index] - for index in validators_to_try: + seed = hash(state.current_shuffling_seed + int_to_bytes8(shard) + int_to_bytes8(slot)) + random_index = bytes_to_int(seed[0:8]) % len(persistent_committee) + persistent_committee = persistent_committee[random_index:] + persistent_committee[:random_index] + + # Search for an active proposer + for index in persistent_committee: if is_active_validator(state.validator_registry[index], get_current_epoch(state)): return index + + # No block can be proposed if no validator is active return None ``` -## Data Structures - -### Shard chain blocks - -A `ShardBlock` object has the following fields: +### `get_shard_header` ```python -{ - # Slot number - 'slot': 'uint64', - # What shard is it on - 'shard_id': 'uint64', - # Parent block's root - 'parent_root': 'bytes32', - # Beacon chain block - 'beacon_chain_ref': 'bytes32', - # Merkle root of data - 'data_root': 'bytes32' - # State root (placeholder for now) - 'state_root': 'bytes32', - # Block signature - 'signature': 'bytes96', - # Attestation - 'participation_bitfield': 'bytes', - 'aggregate_signature': 'bytes96', -} +def get_shard_header(block: ShardBlock) -> ShardBlockHeader: + return ShardBlockHeader( + slot: block.slot, + shard: block.shard, + beacon_chain_root: block.beacon_chain_root, + previous_block_root: block.previous_block_root, + body_root: hash_tree_root(block.body), + state_root: block.state_root, + attestations: block.attestations, + signature: block.signature, + ) ``` -## Shard block processing - -For a `shard_block` on a shard to be processed by a node, the following conditions must be met: - -* The `ShardBlock` pointed to by `shard_block.parent_root` has already been processed and accepted -* The signature for the block from the _proposer_ (see below for definition) of that block is included along with the block in the network message object - -To validate a block header on shard `shard_block.shard_id`, compute as follows: - -* Verify that `shard_block.beacon_chain_ref` is the hash of a block in the (canonical) beacon chain with slot less than or equal to `slot`. -* Verify that `shard_block.beacon_chain_ref` is equal to or a descendant of the `shard_block.beacon_chain_ref` specified in the `ShardBlock` pointed to by `shard_block.parent_root`. -* Let `state` be the state of the beacon chain block referred to by `shard_block.beacon_chain_ref`. -* Let `persistent_committee = get_persistent_committee(state, shard_block.shard_id, shard_block.slot)`. -* Assert `verify_bitfield(shard_block.participation_bitfield, len(persistent_committee))` -* For every `i in range(len(persistent_committee))` where `is_active_validator(state.validator_registry[persistent_committee[i]], get_current_epoch(state))` returns `False`, verify that `get_bitfield_bit(shard_block.participation_bitfield, i) == 0` -* Let `proposer_index = get_shard_proposer_index(state, shard_block.shard_id, shard_block.slot)`. -* Verify that `proposer_index` is not `None`. -* Let `msg` be the `shard_block` but with `shard_block.signature` set to `[0, 0]`. -* Verify that `bls_verify(pubkey=validators[proposer_index].pubkey, message_hash=hash(msg), signature=shard_block.signature, domain=get_domain(state, slot_to_epoch(shard_block.slot), DOMAIN_SHARD_PROPOSER))` passes. -* Let `group_public_key = bls_aggregate_pubkeys([state.validator_registry[index].pubkey for i, index in enumerate(persistent_committee) if get_bitfield_bit(shard_block.participation_bitfield, i) is True])`. -* Verify that `bls_verify(pubkey=group_public_key, message_hash=shard_block.parent_root, sig=shard_block.aggregate_signature, domain=get_domain(state, slot_to_epoch(shard_block.slot), DOMAIN_SHARD_ATTESTER))` passes. - -### Verifying shard block data - -At network layer, we expect a shard block header to be broadcast along with its `block_body`. - -* Verify that `len(block_body) == SHARD_BLOCK_SIZE` -* Verify that `merkle_root(block_body)` equals the `data_root` in the header. - -### Verifying a crosslink - -A node should sign a crosslink only if the following conditions hold. **If a node has the capability to perform the required level of verification, it should NOT follow chains on which a crosslink for which these conditions do NOT hold has been included, or a sufficient number of signatures have been included that during the next state recalculation, a crosslink will be registered.** - -First, the conditions must recursively apply to the crosslink referenced in `last_crosslink_root` for the same shard (unless `last_crosslink_root` equals zero, in which case we are at the genesis). - -Second, we verify the `shard_chain_commitment`. -* Let `start_slot = state.latest_crosslinks[shard].epoch * SLOTS_PER_EPOCH + SLOTS_PER_EPOCH - CROSSLINK_LOOKBACK`. -* Let `end_slot = attestation.data.slot - attestation.data.slot % SLOTS_PER_EPOCH - CROSSLINK_LOOKBACK`. -* Let `length = end_slot - start_slot`, `headers[0] .... headers[length-1]` be the serialized block headers in the canonical shard chain from the verifer's point of view (note that this implies that `headers` and `bodies` have been checked for validity). -* Let `bodies[0] ... bodies[length-1]` be the bodies of the blocks. -* Note: If there is a missing slot, then the header and body are the same as that of the block at the most recent slot that has a block. - -We define two helpers: +### `verify_shard_attestation_signature` ```python -def pad_to_power_of_2(values: List[bytes]) -> List[bytes]: - zero_shard_block = b'\x00' * SHARD_BLOCK_SIZE - while not is_power_of_two(len(values)): - values = values + [zero_shard_block] - return values +def verify_shard_attestation_signature(state: BeaconState, + attestation: ShardAttestation) -> None: + data = attestation.data + persistent_committee = get_persistent_committee(state, data.shard, data.slot) + assert verify_bitfield(attestation.aggregation_bitfield, len(persistent_committee)) + pubkeys = [] + for i, index in enumerate(persistent_committee): + if get_bitfield_bit(attestation.aggregation_bitfield, i) == 0b1 + validator = state.validator_registry[index] + assert is_active_validator(validator, get_current_epoch(state)) + pubkeys.append(validator.pubkey) + assert bls_verify( + pubkey=bls_aggregate_pubkeys(pubkeys), + message_hash=data.shard_block_root, + signature=attestation.aggregate_signature, + domain=get_domain(state, slot_to_epoch(data.slot), DOMAIN_SHARD_ATTESTER) + ) ``` -```python -def merkle_root_of_bytes(data: bytes) -> bytes: - return merkle_root([data[i:i + 32] for i in range(0, len(data), 32)]) -``` - -We define the function for computing the commitment as follows: +### `compute_crosslink_data_root` ```python -def compute_commitment(headers: List[ShardBlock], bodies: List[bytes]) -> Bytes32: +def compute_crosslink_data_root(blocks: List[ShardBlock]) -> Hash: + def is_power_of_two(value: int) -> bool: + return (value > 0) and (value & (value - 1) == 0) + + def pad_to_power_of_2(values: List[bytes]) -> List[bytes]: + while not is_power_of_two(len(values)): + values += [b'\x00' * BYTES_PER_SHARD_BLOCK_BODY] + return values + + def merkle_root_of_bytes(data: bytes) -> bytes: + return merkle_root([data[i:i + 32] for i in range(0, len(data), 32)]) + return hash( - merkle_root( - pad_to_power_of_2([ - merkle_root_of_bytes(zpad(serialize(h), SHARD_BLOCK_SIZE)) for h in headers - ]) - ) + - merkle_root( - pad_to_power_of_2([ - merkle_root_of_bytes(h) for h in bodies - ]) - ) + merkle_root(pad_to_power_of_2([ + merkle_root_of_bytes(zpad(serialize(get_shard_header(block)), BYTES_PER_SHARD_BLOCK_BODY)) for block in blocks + ])) + + merkle_root(pad_to_power_of_2([ + merkle_root_of_bytes(block.body) for block in blocks + ])) ) ``` -The `shard_chain_commitment` is only valid if it equals `compute_commitment(headers, bodies)`. +## Object validity +### Shard blocks -### Shard block fork choice rule +Let: -The fork choice rule for any shard is LMD GHOST using the shard chain attestations of the persistent committee and the beacon chain attestations of the crosslink committee currently assigned to that shard, but instead of being rooted in the genesis it is rooted in the block referenced in the most recent accepted crosslink (ie. `state.crosslinks[shard].shard_block_root`). Only blocks whose `beacon_chain_ref` is the block in the main beacon chain at the specified `slot` should be considered (if the beacon chain skips a slot, then the block at that slot is considered to be the block in the beacon chain at the highest slot lower than a slot). - -# Updates to the beacon chain - -## Data structures - -### `Validator` - -Add member values to the end of the `Validator` object: +* `beacon_blocks` be the `BeaconBlock` list such that `beacon_blocks[slot]` is the canonical `BeaconBlock` at slot `slot` +* `beacon_state` be the canonical `BeaconState` after processing `beacon_blocks[-1]` +* `valid_shard_blocks` be the list of valid `ShardBlock`, recursively defined +* `unix_time` be the current unix time +* `candidate` be a candidate `ShardBlock` for which validity is to be determined by running `is_valid_shard_block` ```python - 'next_subkey_to_reveal': 'uint64', - 'reveal_max_periods_late': 'uint64', -``` +def is_valid_shard_block(beacon_blocks: List[BeaconBlock], + beacon_state: BeaconState, + valid_shard_blocks: List[ShardBlock], + unix_time: uint64, + candidate: ShardBlock) -> bool + # Check if block is already determined valid + for _, block in enumerate(valid_shard_blocks): + if candidate == block: + return True -And the initializers: + # Check slot number + assert block.slot >= PHASE_1_GENESIS_SLOT + assert unix_time >= beacon_state.genesis_time + (block.slot - GENESIS_SLOT) * SECONDS_PER_SLOT -```python - 'next_subkey_to_reveal': get_current_custody_period(state), - 'reveal_max_periods_late': 0, -``` + # Check shard number + assert block.shard <= SHARD_COUNT -### `BeaconBlockBody` + # Check beacon block + beacon_block = beacon_blocks[block.slot] + assert block.beacon_block_root == signed_root(beacon_block) + assert beacon_block.slot <= block.slot: -Add member values to the `BeaconBlockBody` structure: + # Check state root + assert block.state_root == ZERO_HASH # [to be removed in phase 2] -```python - 'branch_challenges': [BranchChallenge], - 'branch_responses': [BranchResponse], - 'subkey_reveals': [SubkeyReveal], - 'interactive_custody_challenge_initiations': [InteractiveCustodyChallengeInitiation], - 'interactive_custody_challenge_responses': [InteractiveCustodyChallengeResponse], - 'interactive_custody_challenge_continuations': [InteractiveCustodyChallengeContinuation], - -``` - -And initialize to the following: - -```python - 'branch_challenges': [], - 'branch_responses': [], - 'subkey_reveals': [], -``` - -### `BeaconState` - -Add member values to the `BeaconState` structure: - -```python - 'branch_challenge_records': [BranchChallengeRecord], - 'next_branch_challenge_id': 'uint64', - 'custody_challenge_records': [InteractiveCustodyChallengeRecord], - 'next_custody_challenge_id': 'uint64', -``` - -### `BranchChallenge` - -Define a `BranchChallenge` as follows: - -```python -{ - 'responder_index': 'uint64', - 'data_index': 'uint64', - 'attestation': SlashableAttestation, -} -``` - -### `BranchResponse` - -Define a `BranchResponse` as follows: - -```python -{ - 'challenge_id': 'uint64', - 'responding_to_custody_challenge': 'bool', - 'data': 'bytes32', - 'branch': ['bytes32'], -} -``` - -### `BranchChallengeRecord` - -Define a `BranchChallengeRecord` as follows: - -```python -{ - 'challenge_id': 'uint64', - 'challenger_index': 'uint64', - 'responder_index': 'uint64', - 'root': 'bytes32', - 'depth': 'uint64', - 'deadline': 'uint64', - 'data_index': 'uint64', -} -``` - -### `InteractiveCustodyChallengeRecord` - -```python -{ - 'challenge_id': 'uint64', - 'challenger_index': 'uint64', - 'responder_index': 'uint64', - # Initial data root - 'data_root': 'bytes32', - # Initial custody bit - 'custody_bit': 'bool', - # Responder subkey - 'responder_subkey': 'bytes96', - # The hash in the PoC tree in the position that we are currently at - 'current_custody_tree_node': 'bytes32', - # The position in the tree, in terms of depth and position offset - 'depth': 'uint64', - 'offset': 'uint64', - # Max depth of the branch - 'max_depth': 'uint64', - # Deadline to respond (as an epoch) - 'deadline': 'uint64', -} -``` - -### `InteractiveCustodyChallengeInitiation` - -```python -{ - 'attestation': SlashableAttestation, - 'responder_index': 'uint64', - 'challenger_index': 'uint64', - 'responder_subkey': 'bytes96', - 'signature': 'bytes96', -} -``` - -### `InteractiveCustodyChallengeResponse` - -```python -{ - 'challenge_id': 'uint64', - 'hashes': ['bytes32'], - 'signature': 'bytes96', -} -``` - -### `InteractiveCustodyChallengeContinuation` - -```python -{ - 'challenge_id': 'uint64', - 'sub_index': 'uint64', - 'new_custody_tree_node': 'bytes32', - 'proof': ['bytes32'], - 'signature': 'bytes96', -} -``` - -### `SubkeyReveal` - -Define a `SubkeyReveal` as follows: - -```python -{ - 'validator_index': 'uint64', - 'period': 'uint64', - 'subkey': 'bytes96', - 'mask': 'bytes32', - 'revealer_index': 'uint64' -} -``` - -## Helpers - -### `get_branch_challenge_record_by_id` - -```python -def get_branch_challenge_record_by_id(state: BeaconState, id: int) -> BranchChallengeRecord: - return [c for c in state.branch_challenges if c.challenge_id == id][0] -``` - -### `get_custody_challenge_record_by_id` - -```python -def get_custody_challenge_record_by_id(state: BeaconState, id: int) -> BranchChallengeRecord: - return [c for c in state.branch_challenges if c.challenge_id == id][0] -``` - -### `get_attestation_merkle_depth` - -```python -def get_attestation_merkle_depth(attestation: Attestation) -> int: - start_epoch = attestation.data.latest_crosslink.epoch - end_epoch = slot_to_epoch(attestation.data.slot) - chunks_per_slot = SHARD_BLOCK_SIZE // 32 - chunks = (end_epoch - start_epoch) * EPOCH_LENGTH * chunks_per_slot - return log2(next_power_of_two(chunks)) -``` - -### `epoch_to_custody_period` - -```python -def epoch_to_custody_period(epoch: Epoch) -> int: - return epoch // CUSTODY_PERIOD_LENGTH -``` - -### `slot_to_custody_period` - -```python -def slot_to_custody_period(slot: Slot) -> int: - return epoch_to_custody_period(slot_to_epoch(slot)) -``` - -### `get_current_custody_period` - -```python -def get_current_custody_period(state: BeaconState) -> int: - return epoch_to_custody_period(get_current_epoch(state)) -``` - -### `verify_custody_subkey_reveal` - -```python -def verify_custody_subkey_reveal(pubkey: bytes48, - subkey: bytes96, - mask: bytes32, - mask_pubkey: bytes48, - period: int) -> bool: - # Legitimate reveal: checking that the provided value actually is the subkey - if mask == ZERO_HASH: - pubkeys=[pubkey] - message_hashes=[hash(int_to_bytes8(period))] - - # Punitive early reveal: checking that the provided value is a valid masked subkey - # (masking done to prevent "stealing the reward" from a whistleblower by block proposers) - # Secure under the aggregate extraction infeasibility assumption described on page 11-12 - # of https://crypto.stanford.edu/~dabo/pubs/papers/aggreg.pdf + # Check parent block + if block.slot == PHASE_1_GENESIS_SLOT: + assert candidate.previous_block_root == ZERO_HASH else: - pubkeys=[pubkey, mask_pubkey] - message_hashes=[hash(int_to_bytes8(period)), mask] - - return bls_multi_verify( - pubkeys=pubkeys, - message_hashes=message_hashes, - signature=subkey, - domain=get_domain( - fork=state.fork, - epoch=period * CUSTODY_PERIOD_LENGTH, - domain_type=DOMAIN_CUSTODY_SUBKEY, - ) - ) -``` + parent_block = next( + block for block in valid_shard_blocks if + signed_root(block) == candidate.previous_block_root + , None) + assert parent_block != None + assert parent_block.shard == block.shard + assert parent_block.slot < block.slot + assert signed_root(beacon_blocks[parent_block.slot]) == parent_block.beacon_chain_root -### `verify_signed_challenge_message` + # Check attestations + assert len(block.attestations) <= MAX_SHARD_ATTESTIONS + for _, attestation in enumerate(block.attestations): + assert max(GENESIS_SHARD_SLOT, block.slot - SLOTS_PER_EPOCH) <= attestation.data.slot + assert attesation.data.slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY + assert attetation.data.shart == block.shard + verify_shard_attestation_signature(beacon_state, attestation) -```python -def verify_signed_challenge_message(message: Any, pubkey: bytes48) -> bool: - return bls_verify( - message_hash=signed_root(message), - pubkey=pubkey, - signature=message.signature, - domain=get_domain(state, get_current_epoch(state), DOMAIN_CUSTODY_INTERACTIVE) + # Check signature + proposer_index = get_shard_proposer_index(beacon_state, block.shard, block.slot) + assert proposer_index is not None + assert bls_verify( + pubkey=validators[proposer_index].pubkey, + message_hash=signed_root(block), + signature=block.signature, + domain=get_domain(beacon_state, slot_to_epoch(block.slot), DOMAIN_SHARD_PROPOSER) ) + return True ``` -### `penalize_validator` +### Shard attestations -Change the definition of `penalize_validator` as follows: +Let: + +* `valid_shard_blocks` be the list of valid `ShardBlock` +* `beacon_state` be the canonical `BeaconState` +* `candidate` be a candidate `ShardAttestation` for which validity is to be determined by running `is_valid_shard_attestation` ```python -def penalize_validator(state: BeaconState, index: ValidatorIndex, whistleblower_index=None:ValidatorIndex) -> None: - """ - Penalize the validator of the given ``index``. - Note that this function mutates ``state``. - """ - exit_validator(state, index) - validator = state.validator_registry[index] - state.latest_penalized_balances[get_current_epoch(state) % LATEST_PENALIZED_EXIT_LENGTH] += get_effective_balance(state, index) - - block_proposer_index = get_beacon_proposer_index(state, state.slot) - whistleblower_reward = get_effective_balance(state, index) // WHISTLEBLOWER_REWARD_QUOTIENT - if whistleblower_index is None: - state.validator_balances[block_proposer_index] += whistleblower_reward +def is_valid_shard_attestation(valid_shard_blocks: List[ShardBlock], + beacon_state: BeaconState, + candidate: Attestation) -> bool: + # Check shard block + shard_block = next( + block for block in valid_shard_blocks if + signed_root(block) == candidate.attestation.data.shard_block_root + , None) + assert shard_block != None + assert shard_block.slot == attestation.data.slot + assert shard_block.shard == attestation.data.shard + + # Check signature + verify_shard_attestation_signature(beacon_state, attestation) + + return True +``` + +### Beacon attestations + +Let: + +* `shard` be a valid `Shard` +* `shard_blocks` be the `ShardBlock` list such that `shard_blocks[slot]` is the canonical `ShardBlock` for shard `shard` at slot `slot` +* `beacon_state` be the canonical `BeaconState` +* `valid_attestations` be the list of valid `Attestation`, recursively defined +* `candidate` be a candidate `Attestation` which is valid under phase 0 rules, and for which validity is to be determined under phase 1 rules by running `is_valid_beacon_attestation` + +```python +def is_valid_beacon_attestation(shard: Shard, + shard_blocks: List[ShardBlock], + beacon_state: BeaconState, + valid_attestations: List[Attestation], + candidate: Attestation) -> bool: + # Check if attestation is already determined valid + for _, attestation in enumerate(valid_attestations): + if candidate == attestation: + return True + + # Check previous attestation + if candidate.data.previous_crosslink.epoch <= PHASE_1_GENESIS_EPOCH: + assert candidate.data.previous_crosslink.crosslink_data_root == ZERO_HASH else: - state.validator_balances[whistleblower_index] += ( - whistleblower_reward * INCLUDER_REWARD_QUOTIENT / (INCLUDER_REWARD_QUOTIENT + 1) - ) - state.validator_balances[block_proposer_index] += whistleblower_reward / (INCLUDER_REWARD_QUOTIENT + 1) - state.validator_balances[index] -= whistleblower_reward - validator.penalized_epoch = get_current_epoch(state) - validator.withdrawable_epoch = get_current_epoch(state) + LATEST_PENALIZED_EXIT_LENGTH + previous_attestation = next( + attestation for attestation in valid_attestations if + attestation.data.crosslink_data_root == candidate.data.previous_crosslink.crosslink_data_root + , None) + assert previous_attestation != None + assert candidate.data.previous_attestation.epoch < slot_to_epoch(candidate.data.slot) + + # Check crosslink data root + start_epoch = state.latest_crosslinks[shard].epoch + end_epoch = min(slot_to_epoch(candidate.data.slot) - CROSSLINK_LOOKBACK, start_epoch + MAX_CROSSLINK_EPOCHS) + blocks = [] + for slot in range(start_epoch * SLOTS_PER_EPOCH, end_epoch * SLOTS_PER_EPOCH): + blocks.append(shard_blocks[slot]) + assert candidate.data.crosslink_data_root == compute_crosslink_data_root(blocks) + + return True ``` -The only change is that this introduces the possibility of a penalization where the "whistleblower" that takes credit is NOT the block proposer. +## Shard fork choice rule -## Per-slot processing - -### Operations - -Add the following operations to the per-slot processing, in order the given below and _after_ all other operations (specifically, right after exits). - -#### Branch challenges - -Verify that `len(block.body.branch_challenges) <= MAX_BRANCH_CHALLENGES`. - -For each `challenge` in `block.body.branch_challenges`, run: - -```python -def process_branch_challenge(state: BeaconState, - challenge: BranchChallenge) -> None: - # Check that it's not too late to challenge - assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY - assert state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY - # Check the attestation is valid - assert verify_slashable_attestation(state, challenge.attestation) - # Check that the responder participated - assert challenger.responder_index in challenge.attestation.validator_indices - # Check the challenge is not a duplicate - assert [ - c for c in state.branch_challenge_records if c.root == challenge.attestation.data.crosslink_data_root and - c.data_index == challenge.data_index - ] == [] - # Check validity of depth - depth = get_attestation_merkle_depth(challenge.attestation) - assert c.data_index < 2**depth - # Add new challenge - state.branch_challenge_records.append(BranchChallengeRecord( - challenge_id=state.next_branch_challenge_id, - challenger_index=get_beacon_proposer_index(state, state.slot), - root=challenge.attestation.data.shard_chain_commitment, - depth=depth, - deadline=get_current_epoch(state) + CHALLENGE_RESPONSE_DEADLINE, - data_index=challenge.data_index - )) - state.next_branch_challenge_id += 1 -``` - -#### Branch responses - -Verify that `len(block.body.branch_responses) <= MAX_BRANCH_RESPONSES`. - -For each `response` in `block.body.branch_responses`, if `response.responding_to_custody_challenge == False`, run: - -```python -def process_branch_exploration_response(state: BeaconState, - response: BranchResponse) -> None: - challenge = get_branch_challenge_record_by_id(response.challenge_id) - assert verify_merkle_branch( - leaf=response.data, - branch=response.branch, - depth=challenge.depth, - index=challenge.data_index, - root=challenge.root - ) - # Must wait at least ENTRY_EXIT_DELAY before responding to a branch challenge - assert get_current_epoch(state) >= challenge.inclusion_epoch + ENTRY_EXIT_DELAY - state.branch_challenge_records.pop(challenge) - # Reward the proposer - proposer_index = get_beacon_proposer_index(state, state.slot) - state.validator_balances[proposer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT -``` - -If `response.responding_to_custody_challenge == True`, run: - -```python -def process_branch_custody_response(state: BeaconState, - response: BranchResponse) -> None: - challenge = get_custody_challenge_record_by_id(response.challenge_id) - responder = state.validator_registry[challenge.responder_index] - # Verify we're not too late - assert get_current_epoch(state) < responder.withdrawable_epoch - # Verify the Merkle branch *of the data tree* - assert verify_merkle_branch( - leaf=response.data, - branch=response.branch, - depth=challenge.max_depth, - index=challenge.offset, - root=challenge.data_root - ) - # Responder wins - if hash(challenge.responder_subkey + response.data) == challenge.current_custody_tree_node: - penalize_validator(state, challenge.challenger_index, challenge.responder_index) - # Challenger wins - else: - penalize_validator(state, challenge.responder_index, challenge.challenger_index) - state.custody_challenge_records.pop(challenge) -``` - -#### Subkey reveals - -Verify that `len(block.body.early_subkey_reveals) <= MAX_EARLY_SUBKEY_REVEALS`. - -For each `reveal` in `block.body.early_subkey_reveals`: - -* Verify that `verify_custody_subkey_reveal(state.validator_registry[reveal.validator_index].pubkey, reveal.subkey, reveal.period, reveal.mask, state.validator_registry[reveal.revealer_index].pubkey)` returns `True`. -* Let `is_early_reveal = reveal.period > get_current_custody_period(state) or (reveal.period == get_current_custody_period(state) and state.validator_registry[reveal.validator_index].exit_epoch > get_current_epoch(state))` (ie. either the reveal is of a future period, or it's of the current period and the validator is still active) -* Verify that one of the following is true: - * (i) `is_early_reveal` is `True` - * (ii) `is_early_reveal` is `False` and `reveal.period == state.validator_registry[reveal.validator_index].next_subkey_to_reveal` (revealing a past subkey, or a current subkey for a validator that has exited) and `reveal.mask == ZERO_HASH` - -In case (i): - -* Verify that `state.validator_registry[reveal.validator_index].penalized_epoch > get_current_epoch(state). -* Run `penalize_validator(state, reveal.validator_index, reveal.revealer_index)`. -* Set `state.validator_balances[reveal.revealer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT` - -In case (ii): - -* Determine the proposer `proposer_index = get_beacon_proposer_index(state, state.slot)` and set `state.validator_balances[proposer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT`. -* Set `state.validator_registry[reveal.validator_index].next_subkey_to_reveal += 1` -* Set `state.validator_registry[reveal.validator_index].reveal_max_periods_late = max(state.validator_registry[reveal.validator_index].reveal_max_periods_late, get_current_period(state) - reveal.period)`. - -#### Interactive custody challenge initiations - -Verify that `len(block.body.interactive_custody_challenge_initiations) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_INITIATIONS`. - -For each `initiation` in `block.body.interactive_custody_challenge_initiations`, use the following function to process it: - -```python -def process_initiation(state: BeaconState, - initiation: InteractiveCustodyChallengeInitiation) -> None: - challenger = state.validator_registry[initiation.challenger_index] - responder = state.validator_registry[initiation.responder_index] - # Verify the signature - assert verify_signed_challenge_message(initiation, challenger.pubkey) - # Verify the attestation - assert verify_slashable_attestation(initiation.attestation, state) - # Check that the responder actually participated in the attestation - assert initiation.responder_index in attestation.validator_indices - # Any validator can be a challenger or responder of max 1 challenge at a time - for c in state.custody_challenge_records: - assert c.challenger_index != initiation.challenger_index - assert c.responder_index != initiation.responder_index - # Can't challenge if you've been penalized - assert challenger.penalized_epoch == FAR_FUTURE_EPOCH - # Make sure the revealed subkey is valid - assert verify_custody_subkey_reveal( - pubkey=state.validator_registry[responder_index].pubkey, - subkey=initiation.responder_subkey, - period=slot_to_custody_period(attestation.data.slot) - ) - # Verify that the attestation is still eligible for challenging - min_challengeable_epoch = responder.exit_epoch - CUSTODY_PERIOD_LENGTH * (1 + responder.reveal_max_periods_late) - assert min_challengeable_epoch <= slot_to_epoch(initiation.attestation.data.slot) - # Create a new challenge object - state.branch_challenge_records.append(InteractiveCustodyChallengeRecord( - challenge_id=state.next_branch_challenge_id, - challenger_index=initiation.challenger_index, - responder_index=initiation.responder_index, - data_root=attestation.custody_commitment, - custody_bit=get_bitfield_bit(attestation.custody_bitfield, attestation.validator_indices.index(responder_index)), - responder_subkey=responder_subkey, - current_custody_tree_node=ZERO_HASH, - depth=0, - offset=0, - max_depth=get_attestation_data_merkle_depth(initiation.attestation.data), - deadline=get_current_epoch(state) + CHALLENGE_RESPONSE_DEADLINE - )) - state.next_branch_challenge_id += 1 - # Responder can't withdraw yet! - state.validator_registry[responder_index].withdrawable_epoch = FAR_FUTURE_EPOCH -``` - -#### Interactive custody challenge responses - -A response provides 32 hashes that are under current known proof of custody tree node. Note that at the beginning the tree node is just one bit of the custody root, so we ask the responder to sign to commit to the top 5 levels of the tree and therefore the root hash; at all other stages in the game responses are self-verifying. - -Verify that `len(block.body.interactive_custody_challenge_responses) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_RESPONSES`. - -For each `response` in `block.body.interactive_custody_challenge_responses`, use the following function to process it: - -```python -def process_response(state: BeaconState, - response: InteractiveCustodyChallengeResponse) -> None: - challenge = get_custody_challenge_record_by_id(state, response.challenge_id) - responder = state.validator_registry[challenge.responder_index] - # Check that the right number of hashes was provided - expected_depth = min(challenge.max_depth - challenge.depth, MAX_POC_RESPONSE_DEPTH) - assert 2**expected_depth == len(response.hashes) - # Must make some progress! - assert expected_depth > 0 - # Check the hashes match the previously provided root - root = merkle_root(response.hashes) - # If this is the first response check the bit and the signature and set the root - if challenge.depth == 0: - assert get_bitfield_bit(root, 0) == challenge.custody_bit - assert verify_signed_challenge_message(response, responder.pubkey) - challenge.current_custody_tree_node = root - # Otherwise just check the response against the root - else: - assert root == challenge_data.current_custody_tree_node - # Update challenge data - challenge.deadline=FAR_FUTURE_EPOCH - responder.withdrawable_epoch = get_current_epoch(state) + MAX_POC_RESPONSE_DEPTH -``` - -#### Interactive custody challenge continuations - -Once a response provides 32 hashes, the challenger has the right to choose any one of them that they feel is constructed incorrectly to continue the game. Note that eventually, the game will get to the point where the `new_custody_tree_node` is a leaf node. - -Verify that `len(block.body.interactive_custody_challenge_continuations) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_CONTINUATIONS`. - -For each `continuation` in `block.body.interactive_custody_challenge_continuations`, use the following function to process it: - -```python -def process_continuation(state: BeaconState, - continuation: InteractiveCustodyChallengeContinuation) -> None: - challenge = get_custody_challenge_record_by_id(state, continuation.challenge_id) - challenger = state.validator_registry[challenge.challenger_index] - responder = state.validator_registry[challenge.responder_index] - expected_depth = min(challenge_data.max_depth - challenge_data.depth, MAX_POC_RESPONSE_DEPTH) - # Verify we're not too late - assert get_current_epoch(state) < responder.withdrawable_epoch - # Verify the Merkle branch (the previous custody response provided the next level of hashes so the - # challenger has the info to make any Merkle branch) - assert verify_merkle_branch( - leaf=new_custody_tree_node, - branch=continuation.proof, - depth=expected_depth, - index=sub_index, - root=challenge_data.current_custody_tree_node - ) - # Verify signature - assert verify_signed_challenge_message(continuation, challenger.pubkey) - # Update the challenge data - challenge.current_custody_tree_node = continuation.new_custody_tree_node - challenge.depth += expected_depth - challenge.deadline = get_current_epoch(state) + MAX_POC_RESPONSE_DEPTH - responder.withdrawable_epoch = FAR_FUTURE_EPOCH - challenge.offset = challenge_data.offset * 2**expected_depth + sub_index -``` - -## Per-epoch processing - -Add the following loop immediately below the `process_ejections` loop: - -```python -def process_challenge_absences(state: BeaconState) -> None: - """ - Iterate through the challenge list - and penalize validators with balance that did not answer challenges. - """ - for c in state.branch_challenge_records: - if get_current_epoch(state) > c.deadline: - penalize_validator(state, c.responder_index, c.challenger_index) - - for c in state.custody_challenge_records: - if get_current_epoch(state) > c.deadline: - penalize_validator(state, c.responder_index, c.challenger_index) - if get_current_epoch(state) > state.validator_registry[c.responder_index].withdrawable_epoch: - penalize_validator(state, c.challenger_index, c.responder_index) -``` - -In `process_penalties_and_exits`, change the definition of `eligible` to the following (note that it is not a pure function because `state` is declared in the surrounding scope): - -```python -def eligible(index): - validator = state.validator_registry[index] - # Cannot exit if there are still open branch challenges - if [c for c in state.branch_challenge_records if c.responder_index == index] != []: - return False - # Cannot exit if you have not revealed all of your subkeys - elif validator.next_subkey_to_reveal <= epoch_to_custody_period(validator.exit_epoch): - return False - # Cannot exit if you already have - elif validator.withdrawable_epoch < FAR_FUTURE_EPOCH: - return False - # Return minimum time - else: - return current_epoch >= validator.exit_epoch + MIN_VALIDATOR_WITHDRAWAL_EPOCHS -``` - -## One-time phase 1 initiation transition - -Run the following on the fork block after per-slot processing and before per-block and per-epoch processing. - -For all `validator` in `ValidatorRegistry`, update it to the new format and fill the new member values with: - -```python - 'next_subkey_to_reveal': get_current_custody_period(state), - 'reveal_max_periods_late': 0, -``` - -Update the `BeaconState` to the new format and fill the new member values with: - -```python - 'branch_challenge_records': [], - 'next_branch_challenge_id': 0, - 'custody_challenge_records': [], - 'next_custody_challenge_id': 0, -``` +The fork choice rule for any shard is LMD GHOST using the shard attestations of the persistent committee and the beacon chain attestations of the crosslink committee currently assigned to that shard, but instead of being rooted in the genesis it is rooted in the block referenced in the most recent accepted crosslink (i.e. `state.crosslinks[shard].shard_block_root`). Only blocks whose `beacon_chain_root` is the block in the main beacon chain at the specified `slot` should be considered. (If the beacon chain skips a slot, then the block at that slot is considered to be the block in the beacon chain at the highest slot lower than a slot.)