diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index ede369b95..7d6df51aa 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -10,60 +10,37 @@ - [Introduction](#introduction) - [Glossary](#glossary) -- [Custom types](#custom-types) - [Constants](#constants) - [Misc](#misc) - [Domain types](#domain-types) - - [Shard Work Status](#shard-work-status) - - [Misc](#misc-1) - - [Participation flag indices](#participation-flag-indices) - - [Incentivization weights](#incentivization-weights) - [Preset](#preset) - - [Misc](#misc-2) + - [Misc](#misc-1) + - [Time parameters](#time-parameters) - [Shard blob samples](#shard-blob-samples) - - [Precomputed size verification points](#precomputed-size-verification-points) - - [Gwei values](#gwei-values) - [Configuration](#configuration) -- [Updated containers](#updated-containers) - - [`AttestationData`](#attestationdata) - - [`BeaconBlockBody`](#beaconblockbody) - - [`BeaconState`](#beaconstate) -- [New containers](#new-containers) - - [`Builder`](#builder) - - [`DataCommitment`](#datacommitment) - - [`AttestedDataCommitment`](#attesteddatacommitment) - - [`ShardBlobBody`](#shardblobbody) - - [`ShardBlobBodySummary`](#shardblobbodysummary) - - [`ShardBlob`](#shardblob) - - [`ShardBlobHeader`](#shardblobheader) - - [`SignedShardBlob`](#signedshardblob) - - [`SignedShardBlobHeader`](#signedshardblobheader) - - [`PendingShardHeader`](#pendingshardheader) - - [`ShardBlobReference`](#shardblobreference) - - [`ShardProposerSlashing`](#shardproposerslashing) - - [`ShardWork`](#shardwork) + - [Time parameters](#time-parameters-1) +- [Containers](#containers) + - [New Containers](#new-containers) + - [`BuilderBlockBid`](#builderblockbid) + - [`BuilderBlockBidWithRecipientAddress`](#builderblockbidwithrecipientaddress) + - [`ShardedCommitmentsContainer`](#shardedcommitmentscontainer) + - [`ShardSample`](#shardsample) + - [Extended Containers](#extended-containers) + - [`BeaconState`](#beaconstate) + - [`BuilderBlockData`](#builderblockdata) + - [`BeaconBlockBody`](#beaconblockbody) - [Helper functions](#helper-functions) - - [Misc](#misc-3) - - [`next_power_of_two`](#next_power_of_two) - - [`compute_previous_slot`](#compute_previous_slot) - - [`compute_updated_sample_price`](#compute_updated_sample_price) - - [`compute_committee_source_epoch`](#compute_committee_source_epoch) - - [`batch_apply_participation_flag`](#batch_apply_participation_flag) - - [Beacon state accessors](#beacon-state-accessors) - - [Updated `get_committee_count_per_slot`](#updated-get_committee_count_per_slot) - - [`get_active_shard_count`](#get_active_shard_count) - - [`get_shard_proposer_index`](#get_shard_proposer_index) - - [`get_start_shard`](#get_start_shard) - - [`compute_shard_from_committee_index`](#compute_shard_from_committee_index) - - [`compute_committee_index_from_shard`](#compute_committee_index_from_shard) - [Block processing](#block-processing) - - [Operations](#operations) - - [Extended Attestation processing](#extended-attestation-processing) - - [`process_shard_header`](#process_shard_header) - - [`process_shard_proposer_slashing`](#process_shard_proposer_slashing) - - [Epoch transition](#epoch-transition) - - [`process_pending_shard_confirmations`](#process_pending_shard_confirmations) - - [`reset_pending_shard_work`](#reset_pending_shard_work) + - [`is_builder_block_slot`](#is_builder_block_slot) + - [Beacon state accessors](#beacon-state-accessors) + - [`get_active_shard_count`](#get_active_shard_count) +- [Beacon chain state transition function](#beacon-chain-state-transition-function) + - [Block processing](#block-processing-1) + - [`process_block`](#process_block) + - [Block header](#block-header) + - [Builder Block Bid](#builder-block-bid) + - [Sharded data](#sharded-data) + - [Execution payload](#execution-payload) @@ -72,26 +49,14 @@ ## Introduction This document describes the extensions made to the Phase 0 design of The Beacon Chain to support data sharding, -based on the ideas [here](https://hackmd.io/G-Iy5jqyT7CXWEz8Ssos8g) and more broadly [here](https://arxiv.org/abs/1809.09044), +based on the ideas [here](https://notes.ethereum.org/@dankrad/new_sharding) and more broadly [here](https://arxiv.org/abs/1809.09044), using KZG10 commitments to commit to data to remove any need for fraud proofs (and hence, safety-critical synchrony assumptions) in the design. ### Glossary - **Data**: A list of KZG points, to translate a byte string into - **Blob**: Data with commitments and meta-data, like a flattened bundle of L2 transactions. -- **Builder**: Independent actor that builds blobs and bids for proposal slots via fee-paying blob-headers, responsible for availability. -- **Shard proposer**: Validator taking bids from blob builders for shard data opportunity, co-signs with builder to propose the blob. -## Custom types - -We define the following Python custom types for type hinting and readability: - -| Name | SSZ equivalent | Description | -| - | - | - | -| `Shard` | `uint64` | A shard number | -| `BLSCommitment` | `Bytes48` | A G1 curve point | -| `BLSPoint` | `uint256` | A number `x` in the range `0 <= x < MODULUS` | -| `BuilderIndex` | `uint64` | Builder registry index | ## Constants @@ -101,48 +66,13 @@ The following values are (non-configurable) constants used throughout the specif | Name | Value | Notes | | - | - | - | -| `PRIMITIVE_ROOT_OF_UNITY` | `7` | Primitive root of unity of the BLS12_381 (inner) modulus | -| `DATA_AVAILABILITY_INVERSE_CODING_RATE` | `2**1` (= 2) | Factor by which samples are extended for data availability encoding | -| `POINTS_PER_SAMPLE` | `uint64(2**3)` (= 8) | 31 * 8 = 248 bytes | -| `MODULUS` | `0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001` (curve order of BLS12_381) | +| `FIELD_ELEMENTS_PER_SAMPLE` | `uint64(2**4)` (= 16) | 31 * 16 = 496 bytes | ### Domain types | Name | Value | | - | - | -| `DOMAIN_SHARD_BLOB` | `DomainType('0x80000000')` | - -### Shard Work Status - -| Name | Value | Notes | -| - | - | - | -| `SHARD_WORK_UNCONFIRMED` | `0` | Unconfirmed, nullified after confirmation time elapses | -| `SHARD_WORK_CONFIRMED` | `1` | Confirmed, reduced to just the commitment | -| `SHARD_WORK_PENDING` | `2` | Pending, a list of competing headers | - -### Misc - -TODO: `PARTICIPATION_FLAG_WEIGHTS` backwards-compatibility is difficult, depends on usage. - -| Name | Value | -| - | - | -| `PARTICIPATION_FLAG_WEIGHTS` | `[TIMELY_SOURCE_WEIGHT, TIMELY_TARGET_WEIGHT, TIMELY_HEAD_WEIGHT, TIMELY_SHARD_WEIGHT]` | - -### Participation flag indices - -| Name | Value | -| - | - | -| `TIMELY_SHARD_FLAG_INDEX` | `3` | - -### Incentivization weights - -TODO: determine weight for shard attestations - -| Name | Value | -| - | - | -| `TIMELY_SHARD_WEIGHT` | `uint64(8)` | - -TODO: `WEIGHT_DENOMINATOR` needs to be adjusted, but this breaks a lot of Altair code. +| `DOMAIN_SHARD_SAMPLE` | `DomainType('0x10000000')` | ## Preset @@ -150,341 +80,140 @@ TODO: `WEIGHT_DENOMINATOR` needs to be adjusted, but this breaks a lot of Altair | Name | Value | Notes | | - | - | - | -| `MAX_SHARDS` | `uint64(2**10)` (= 1,024) | Theoretical max shard count (used to determine data structure sizes) | -| `INITIAL_ACTIVE_SHARDS` | `uint64(2**6)` (= 64) | Initial shard count | -| `SAMPLE_PRICE_ADJUSTMENT_COEFFICIENT` | `uint64(2**3)` (= 8) | Sample price may decrease/increase by at most exp(1 / this value) *per epoch* | -| `MAX_SHARD_PROPOSER_SLASHINGS` | `2**4` (= 16) | Maximum amount of shard proposer slashing operations per block | -| `MAX_SHARD_HEADERS_PER_SHARD` | `4` | | -| `SHARD_STATE_MEMORY_SLOTS` | `uint64(2**8)` (= 256) | Number of slots for which shard commitments and confirmation status is directly available in the state | -| `BLOB_BUILDER_REGISTRY_LIMIT` | `uint64(2**40)` (= 1,099,511,627,776) | shard blob builders | +| `MAX_SHARDS` | `uint64(2**12)` (= 4,096) | Theoretical max shard count (used to determine data structure sizes) | +| `ACTIVE_SHARDS` | `uint64(2**8)` (= 256) | Initial shard count | +| `MAX_PROPOSER_BLOCKS_BETWEEN_BUILDER_BLOCKS` | `uint64(2**4)` (= 16) | TODO: Need to define what happens if there were more blocks without builder blocks | + +### Time parameters + +With the introduction of builder blocks the number of slots per epoch is doubled (it counts beacon blocks and builder blocks). + +| Name | Value | Unit | Duration | +| - | - | :-: | :-: | +| `SLOTS_PER_EPOCH` | `uint64(2**6)` (= 64) | slots | 8:32 minutes | ### Shard blob samples | Name | Value | Notes | | - | - | - | -| `MAX_SAMPLES_PER_BLOB` | `uint64(2**11)` (= 2,048) | 248 * 2,048 = 507,904 bytes | -| `TARGET_SAMPLES_PER_BLOB` | `uint64(2**10)` (= 1,024) | 248 * 1,024 = 253,952 bytes | - -### Precomputed size verification points - -| Name | Value | -| - | - | -| `G1_SETUP` | Type `List[G1]`. The G1-side trusted setup `[G, G*s, G*s**2....]`; note that the first point is the generator. | -| `G2_SETUP` | Type `List[G2]`. The G2-side trusted setup `[G, G*s, G*s**2....]` | -| `ROOT_OF_UNITY` | `pow(PRIMITIVE_ROOT_OF_UNITY, (MODULUS - 1) // int(MAX_SAMPLES_PER_BLOB * POINTS_PER_SAMPLE), MODULUS)` | - -### Gwei values - -| Name | Value | Unit | Description | -| - | - | - | - | -| `MAX_SAMPLE_PRICE` | `Gwei(2**33)` (= 8,589,934,592) | Gwei | Max sample charged for a TARGET-sized shard blob | -| `MIN_SAMPLE_PRICE` | `Gwei(2**3)` (= 8) | Gwei | Min sample price charged for a TARGET-sized shard blob | +| `SAMPLES_PER_BLOB` | `uint64(2**9)` (= 512) | 248 * 512 = 126,976 bytes | ## Configuration Note: Some preset variables may become run-time configurable for testnets, but default to a preset while the spec is unstable. -E.g. `INITIAL_ACTIVE_SHARDS`, `MAX_SAMPLES_PER_BLOB` and `TARGET_SAMPLES_PER_BLOB`. +E.g. `ACTIVE_SHARDS` and `SAMPLES_PER_BLOB`. -## Updated containers +### Time parameters -The following containers have updated definitions to support Sharding. +| Name | Value | Unit | Duration | +| - | - | :-: | :-: | +| `SECONDS_PER_SLOT` | `uint64(8)` | seconds | 8 seconds | -### `AttestationData` +## Containers + +### New Containers + +#### `BuilderBlockBid` ```python -class AttestationData(Container): +class BuilderBlockBid(Container): slot: Slot - index: CommitteeIndex - # LMD GHOST vote - beacon_block_root: Root - # FFG vote - source: Checkpoint - target: Checkpoint - # Hash-tree-root of ShardBlob - shard_blob_root: Root # [New in Sharding] + parent_block_root: Root + + execution_payload_root: Root + + sharded_data_commitment_root: Root # Root of the sharded data (only data, not beacon/builder block commitments) + + sharded_data_commitment_count: uint64 # Count of sharded data commitments + + bid: Gwei # Block builder bid paid to proposer + + validator_index: ValidatorIndex # Validator index for this bid + + # Block builders use an Eth1 address -- need signature as + # block bid and data gas base fees will be charged to this address + signature_y_parity: bool + signature_r: uint256 + signature_s: uint256 ``` -### `BeaconBlockBody` +#### `BuilderBlockBidWithRecipientAddress` ```python -class BeaconBlockBody(bellatrix.BeaconBlockBody): # [extends Bellatrix block body] - shard_proposer_slashings: List[ShardProposerSlashing, MAX_SHARD_PROPOSER_SLASHINGS] - shard_headers: List[SignedShardBlobHeader, MAX_SHARDS * MAX_SHARD_HEADERS_PER_SHARD] +class BuilderBlockBidWithRecipientAddress(Container): + builder_block_bid: Union[None, BuilderBlockBid] + recipient_address: ExecutionAddress # Address to receive the block builder bid ``` -### `BeaconState` +#### `ShardedCommitmentsContainer` + +```python +class ShardedCommitmentsContainer(Container): + sharded_commitments: List[KZGCommitment, 2 * MAX_SHARDS] + + # Aggregate degree proof for all sharded_commitments + degree_proof: KZGCommitment + + # The sizes of the blocks encoded in the commitments (last builder and all beacon blocks since) + included_block_sizes: List[uint64, MAX_PROPOSER_BLOCKS_BETWEEN_BUILDER_BLOCKS + 1] + + # Number of commitments that are for sharded data (no blocks) + included_sharded_data_commitments: uint64 + + # Random evaluation of beacon blocks + execution payload (this helps with quick verification) + block_verification_kzg_proof: KZGCommitment +``` + +#### `ShardSample` + +```python +class ShardSample(Container): + slot: Slot + row: uint64 + column: uint64 + data: Vector[BLSFieldElement, FIELD_ELEMENTS_PER_SAMPLE] + proof: KZGCommitment + builder: ValidatorIndex + signature: BLSSignature +``` + +### Extended Containers + +#### `BeaconState` ```python class BeaconState(bellatrix.BeaconState): - # Blob builder registry. - blob_builders: List[Builder, BLOB_BUILDER_REGISTRY_LIMIT] - blob_builder_balances: List[Gwei, BLOB_BUILDER_REGISTRY_LIMIT] - # A ring buffer of the latest slots, with information per active shard. - shard_buffer: Vector[List[ShardWork, MAX_SHARDS], SHARD_STATE_MEMORY_SLOTS] - shard_sample_price: uint64 + blocks_since_builder_block: List[BeaconBlock, MAX_PROPOSER_BLOCKS_BETWEEN_BUILDER_BLOCKS] ``` -## New containers - -### `Builder` +#### `BuilderBlockData` ```python -class Builder(Container): - pubkey: BLSPubkey - # TODO: fields for either an expiry mechanism (refunding execution account with remaining balance) - # and/or a builder-transaction mechanism. -``` +class BuilderBlockData(Container): + execution_payload: ExecutionPayload + sharded_commitments_container: ShardedCommitmentsContainer +``` -### `DataCommitment` +#### `BeaconBlockBody` ```python -class DataCommitment(Container): - # KZG10 commitment to the data - point: BLSCommitment - # Length of the data in samples - samples_count: uint64 -``` - -### `AttestedDataCommitment` - -```python -class AttestedDataCommitment(Container): - # KZG10 commitment to the data, and length - commitment: DataCommitment - # hash_tree_root of the ShardBlobHeader (stored so that attestations can be checked against it) - root: Root - # The proposer who included the shard-header - includer_index: ValidatorIndex -``` - -### `ShardBlobBody` - -Unsigned shard data, bundled by a shard-builder. -Unique, signing different bodies as shard proposer for the same `(slot, shard)` is slashable. - -```python -class ShardBlobBody(Container): - # The actual data commitment - commitment: DataCommitment - # Proof that the degree < commitment.samples_count * POINTS_PER_SAMPLE - degree_proof: BLSCommitment - # The actual data. Should match the commitment and degree proof. - data: List[BLSPoint, POINTS_PER_SAMPLE * MAX_SAMPLES_PER_BLOB] - # fee payment fields (EIP 1559 like) - # TODO: express in MWei instead? - max_priority_fee_per_sample: Gwei - max_fee_per_sample: Gwei -``` - -### `ShardBlobBodySummary` - -Summary version of the `ShardBlobBody`, omitting the data payload, while preserving the data-commitments. - -The commitments are not further collapsed to a single hash, -to avoid an extra network roundtrip between proposer and builder, to include the header on-chain more quickly. - -```python -class ShardBlobBodySummary(Container): - # The actual data commitment - commitment: DataCommitment - # Proof that the degree < commitment.samples_count * POINTS_PER_SAMPLE - degree_proof: BLSCommitment - # Hash-tree-root as summary of the data field - data_root: Root - # fee payment fields (EIP 1559 like) - # TODO: express in MWei instead? - max_priority_fee_per_sample: Gwei - max_fee_per_sample: Gwei -``` - -### `ShardBlob` - -`ShardBlobBody` wrapped with the header data that is unique to the shard blob proposal. - -```python -class ShardBlob(Container): - slot: Slot - shard: Shard - # Builder of the data, pays data-fee to proposer - builder_index: BuilderIndex - # Proposer of the shard-blob - proposer_index: ValidatorIndex - # Blob contents - body: ShardBlobBody -``` - -### `ShardBlobHeader` - -Header version of `ShardBlob`. - -```python -class ShardBlobHeader(Container): - slot: Slot - shard: Shard - # Builder of the data, pays data-fee to proposer - builder_index: BuilderIndex - # Proposer of the shard-blob - proposer_index: ValidatorIndex - # Blob contents, without the full data - body_summary: ShardBlobBodySummary -``` - -### `SignedShardBlob` - -Full blob data, signed by the shard builder (ensuring fee payment) and shard proposer (ensuring a single proposal). - -```python -class SignedShardBlob(Container): - message: ShardBlob - signature: BLSSignature -``` - -### `SignedShardBlobHeader` - -Header of the blob, the signature is equally applicable to `SignedShardBlob`. -Shard proposers can accept `SignedShardBlobHeader` as a data-transaction by co-signing the header. - -```python -class SignedShardBlobHeader(Container): - message: ShardBlobHeader - # Signature by builder. - # Once accepted by proposer, the signatures is the aggregate of both. - signature: BLSSignature -``` - -### `PendingShardHeader` - -```python -class PendingShardHeader(Container): - # The commitment that is attested - attested: AttestedDataCommitment - # Who voted for the header - votes: Bitlist[MAX_VALIDATORS_PER_COMMITTEE] - # Sum of effective balances of votes - weight: Gwei - # When the header was last updated, as reference for weight accuracy - update_slot: Slot -``` - -### `ShardBlobReference` - -Reference version of `ShardBlobHeader`, substituting the body for just a hash-tree-root. - -```python -class ShardBlobReference(Container): - slot: Slot - shard: Shard - # Builder of the data - builder_index: BuilderIndex - # Proposer of the shard-blob - proposer_index: ValidatorIndex - # Blob hash-tree-root for slashing reference - body_root: Root -``` - -### `ShardProposerSlashing` - -```python -class ShardProposerSlashing(Container): - slot: Slot - shard: Shard - proposer_index: ValidatorIndex - builder_index_1: BuilderIndex - builder_index_2: BuilderIndex - body_root_1: Root - body_root_2: Root - signature_1: BLSSignature - signature_2: BLSSignature -``` - -### `ShardWork` - -```python -class ShardWork(Container): - # Upon confirmation the data is reduced to just the commitment. - status: Union[ # See Shard Work Status enum - None, # SHARD_WORK_UNCONFIRMED - AttestedDataCommitment, # SHARD_WORK_CONFIRMED - List[PendingShardHeader, MAX_SHARD_HEADERS_PER_SHARD] # SHARD_WORK_PENDING - ] +class BeaconBlockBody(altair.BeaconBlockBody): + payload_data: Union[BuilderBlockBid, BuilderBlockData] ``` ## Helper functions -### Misc +### Block processing -#### `next_power_of_two` +#### `is_builder_block_slot` ```python -def next_power_of_two(x: int) -> int: - return 2 ** ((x - 1).bit_length()) -``` - -#### `compute_previous_slot` - -```python -def compute_previous_slot(slot: Slot) -> Slot: - if slot > 0: - return Slot(slot - 1) - else: - return Slot(0) -``` - -#### `compute_updated_sample_price` - -```python -def compute_updated_sample_price(prev_price: Gwei, samples_length: uint64, active_shards: uint64) -> Gwei: - adjustment_quotient = active_shards * SLOTS_PER_EPOCH * SAMPLE_PRICE_ADJUSTMENT_COEFFICIENT - if samples_length > TARGET_SAMPLES_PER_BLOB: - delta = max(1, prev_price * (samples_length - TARGET_SAMPLES_PER_BLOB) // TARGET_SAMPLES_PER_BLOB // adjustment_quotient) - return min(prev_price + delta, MAX_SAMPLE_PRICE) - else: - delta = max(1, prev_price * (TARGET_SAMPLES_PER_BLOB - samples_length) // TARGET_SAMPLES_PER_BLOB // adjustment_quotient) - return max(prev_price, MIN_SAMPLE_PRICE + delta) - delta -``` - -#### `compute_committee_source_epoch` - -```python -def compute_committee_source_epoch(epoch: Epoch, period: uint64) -> Epoch: - """ - Return the source epoch for computing the committee. - """ - source_epoch = Epoch(epoch - epoch % period) - if source_epoch >= period: - source_epoch -= period # `period` epochs lookahead - return source_epoch -``` - -#### `batch_apply_participation_flag` - -```python -def batch_apply_participation_flag(state: BeaconState, bits: Bitlist[MAX_VALIDATORS_PER_COMMITTEE], - epoch: Epoch, full_committee: Sequence[ValidatorIndex], flag_index: int): - if epoch == get_current_epoch(state): - epoch_participation = state.current_epoch_participation - else: - epoch_participation = state.previous_epoch_participation - for bit, index in zip(bits, full_committee): - if bit: - epoch_participation[index] = add_flag(epoch_participation[index], flag_index) +def is_builder_block_slot(slot: Slot) -> bool: + return slot % 2 == 1 ``` ### Beacon state accessors -#### Updated `get_committee_count_per_slot` - -```python -def get_committee_count_per_slot(state: BeaconState, epoch: Epoch) -> uint64: - """ - Return the number of committees in each slot for the given ``epoch``. - """ - return max(uint64(1), min( - get_active_shard_count(state, epoch), - uint64(len(get_active_validator_indices(state, epoch))) // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE, - )) -``` - #### `get_active_shard_count` ```python @@ -493,396 +222,195 @@ def get_active_shard_count(state: BeaconState, epoch: Epoch) -> uint64: Return the number of active shards. Note that this puts an upper bound on the number of committees per slot. """ - return INITIAL_ACTIVE_SHARDS -``` - -#### `get_shard_proposer_index` - -```python -def get_shard_proposer_index(state: BeaconState, slot: Slot, shard: Shard) -> ValidatorIndex: - """ - Return the proposer's index of shard block at ``slot``. - """ - epoch = compute_epoch_at_slot(slot) - seed = hash(get_seed(state, epoch, DOMAIN_SHARD_BLOB) + uint_to_bytes(slot) + uint_to_bytes(shard)) - indices = get_active_validator_indices(state, epoch) - return compute_proposer_index(state, indices, seed) -``` - -#### `get_start_shard` - -```python -def get_start_shard(state: BeaconState, slot: Slot) -> Shard: - """ - Return the start shard at ``slot``. - """ - epoch = compute_epoch_at_slot(Slot(slot)) - committee_count = get_committee_count_per_slot(state, epoch) - active_shard_count = get_active_shard_count(state, epoch) - return committee_count * slot % active_shard_count -``` - -#### `compute_shard_from_committee_index` - -```python -def compute_shard_from_committee_index(state: BeaconState, slot: Slot, index: CommitteeIndex) -> Shard: - active_shards = get_active_shard_count(state, compute_epoch_at_slot(slot)) - assert index < active_shards - return Shard((index + get_start_shard(state, slot)) % active_shards) -``` - -#### `compute_committee_index_from_shard` - -```python -def compute_committee_index_from_shard(state: BeaconState, slot: Slot, shard: Shard) -> CommitteeIndex: - epoch = compute_epoch_at_slot(slot) - active_shards = get_active_shard_count(state, epoch) - index = CommitteeIndex((active_shards + shard - get_start_shard(state, slot)) % active_shards) - assert index < get_committee_count_per_slot(state, epoch) - return index + return ACTIVE_SHARDS ``` +## Beacon chain state transition function ### Block processing +#### `process_block` + ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) - # is_execution_enabled is omitted, execution is enabled by default. - process_execution_payload(state, block.body.execution_payload, EXECUTION_ENGINE) - process_randao(state, block.body) + verify_builder_block_bid(state, block) + process_sharded_data(state, block) + if is_execution_enabled(state, block.body): + process_execution_payload(state, block, EXECUTION_ENGINE) + + if not is_builder_block_slot(block.slot): + process_randao(state, block.body) + process_eth1_data(state, block.body) - process_operations(state, block.body) # [Modified in Sharding] + process_operations(state, block.body) process_sync_aggregate(state, block.body.sync_aggregate) + + if is_builder_block_slot(block.slot): + state.blocks_since_builder_block = [] + state.blocks_since_builder_block.append(block) ``` -#### Operations +#### Block header ```python -def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: - # Verify that outstanding deposits are processed up to the maximum number of deposits - assert len(body.deposits) == min(MAX_DEPOSITS, state.eth1_data.deposit_count - state.eth1_deposit_index) - - def for_ops(operations: Sequence[Any], fn: Callable[[BeaconState, Any], None]) -> None: - for operation in operations: - fn(state, operation) - - for_ops(body.proposer_slashings, process_proposer_slashing) - for_ops(body.attester_slashings, process_attester_slashing) - # New shard proposer slashing processing - for_ops(body.shard_proposer_slashings, process_shard_proposer_slashing) - - # Limit is dynamic: based on active shard count - assert len(body.shard_headers) <= MAX_SHARD_HEADERS_PER_SHARD * get_active_shard_count(state, get_current_epoch(state)) - for_ops(body.shard_headers, process_shard_header) - - # New attestation processing - for_ops(body.attestations, process_attestation) - for_ops(body.deposits, process_deposit) - for_ops(body.voluntary_exits, process_voluntary_exit) - - # TODO: to avoid parallel shards racing, and avoid inclusion-order problems, - # update the fee price per slot, instead of per header. - # state.shard_sample_price = compute_updated_sample_price(state.shard_sample_price, ?, shard_count) -``` - -##### Extended Attestation processing - -```python -def process_attestation(state: BeaconState, attestation: Attestation) -> None: - altair.process_attestation(state, attestation) - process_attested_shard_work(state, attestation) -``` - -```python -def process_attested_shard_work(state: BeaconState, attestation: Attestation) -> None: - attestation_shard = compute_shard_from_committee_index( - state, - attestation.data.slot, - attestation.data.index, - ) - full_committee = get_beacon_committee(state, attestation.data.slot, attestation.data.index) - - buffer_index = attestation.data.slot % SHARD_STATE_MEMORY_SLOTS - committee_work = state.shard_buffer[buffer_index][attestation_shard] - - # Skip attestation vote accounting if the header is not pending - if committee_work.status.selector != SHARD_WORK_PENDING: - # If the data was already confirmed, check if this matches, to apply the flag to the attesters. - if committee_work.status.selector == SHARD_WORK_CONFIRMED: - attested: AttestedDataCommitment = committee_work.status.value - if attested.root == attestation.data.shard_blob_root: - batch_apply_participation_flag(state, attestation.aggregation_bits, - attestation.data.target.epoch, - full_committee, TIMELY_SHARD_FLAG_INDEX) - return - - current_headers: Sequence[PendingShardHeader] = committee_work.status.value - - # Find the corresponding header, abort if it cannot be found - header_index = len(current_headers) - for i, header in enumerate(current_headers): - if attestation.data.shard_blob_root == header.attested.root: - header_index = i - break - - # Attestations for an unknown header do not count towards shard confirmations, but can otherwise be valid. - if header_index == len(current_headers): - # Note: Attestations may be re-included if headers are included late. - return - - pending_header: PendingShardHeader = current_headers[header_index] - - # The weight may be outdated if it is not the initial weight, and from a previous epoch - if pending_header.weight != 0 and compute_epoch_at_slot(pending_header.update_slot) < get_current_epoch(state): - pending_header.weight = sum(state.validators[index].effective_balance for index, bit - in zip(full_committee, pending_header.votes) if bit) - - pending_header.update_slot = state.slot - - full_committee_balance = Gwei(0) - # Update votes bitfield in the state, update weights - for i, bit in enumerate(attestation.aggregation_bits): - weight = state.validators[full_committee[i]].effective_balance - full_committee_balance += weight - if bit: - if not pending_header.votes[i]: - pending_header.weight += weight - pending_header.votes[i] = True - - # Check if the PendingShardHeader is eligible for expedited confirmation, requiring 2/3 of balance attesting - if pending_header.weight * 3 >= full_committee_balance * 2: - # participants of the winning header are remembered with participation flags - batch_apply_participation_flag(state, pending_header.votes, attestation.data.target.epoch, - full_committee, TIMELY_SHARD_FLAG_INDEX) - - if pending_header.attested.commitment == DataCommitment(): - # The committee voted to not confirm anything - state.shard_buffer[buffer_index][attestation_shard].status.change( - selector=SHARD_WORK_UNCONFIRMED, - value=None, - ) - else: - state.shard_buffer[buffer_index][attestation_shard].status.change( - selector=SHARD_WORK_CONFIRMED, - value=pending_header.attested, - ) -``` - -##### `process_shard_header` - -```python -def process_shard_header(state: BeaconState, signed_header: SignedShardBlobHeader) -> None: - header: ShardBlobHeader = signed_header.message - slot = header.slot - shard = header.shard - - # Verify the header is not 0, and not from the future. - assert Slot(0) < slot <= state.slot - header_epoch = compute_epoch_at_slot(slot) - # Verify that the header is within the processing time window - assert header_epoch in [get_previous_epoch(state), get_current_epoch(state)] - # Verify that the shard is valid - shard_count = get_active_shard_count(state, header_epoch) - assert shard < shard_count - # Verify that a committee is able to attest this (slot, shard) - start_shard = get_start_shard(state, slot) - committee_index = (shard_count + shard - start_shard) % shard_count - committees_per_slot = get_committee_count_per_slot(state, header_epoch) - assert committee_index <= committees_per_slot - - # Check that this data is still pending - committee_work = state.shard_buffer[slot % SHARD_STATE_MEMORY_SLOTS][shard] - assert committee_work.status.selector == SHARD_WORK_PENDING - - # Check that this header is not yet in the pending list - current_headers: List[PendingShardHeader, MAX_SHARD_HEADERS_PER_SHARD] = committee_work.status.value - header_root = hash_tree_root(header) - assert header_root not in [pending_header.attested.root for pending_header in current_headers] - - # Verify proposer matches - assert header.proposer_index == get_shard_proposer_index(state, slot, shard) - - # Verify builder and proposer aggregate signature - blob_signing_root = compute_signing_root(header, get_domain(state, DOMAIN_SHARD_BLOB)) - builder_pubkey = state.blob_builders[header.builder_index].pubkey - proposer_pubkey = state.validators[header.proposer_index].pubkey - assert bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_header.signature) - - # Verify the length by verifying the degree. - body_summary = header.body_summary - points_count = body_summary.commitment.samples_count * POINTS_PER_SAMPLE - if points_count == 0: - assert body_summary.degree_proof == G1_SETUP[0] - assert ( - bls.Pairing(body_summary.degree_proof, G2_SETUP[0]) - == bls.Pairing(body_summary.commitment.point, G2_SETUP[-points_count]) +def process_block_header(state: BeaconState, block: BeaconBlock) -> None: + # Verify that the slots match + assert block.slot == state.slot + # Verify that the block is newer than latest block header + assert block.slot > state.latest_block_header.slot + # Verify that proposer index is the correct index + if not is_builder_block_slot(block.slot): + assert block.proposer_index == get_beacon_proposer_index(state) + # Verify that the parent matches + assert block.parent_root == hash_tree_root(state.latest_block_header) + # Cache current block as the new latest block + state.latest_block_header = BeaconBlockHeader( + slot=block.slot, + proposer_index=block.proposer_index, + parent_root=block.parent_root, + state_root=Bytes32(), # Overwritten in the next process_slot call + body_root=hash_tree_root(block.body), ) - # Charge EIP 1559 fee, builder pays for opportunity, and is responsible for later availability, - # or fail to publish at their own expense. - samples = body_summary.commitment.samples_count - # TODO: overflows, need bigger int type - max_fee = body_summary.max_fee_per_sample * samples - - # Builder must have sufficient balance, even if max_fee is not completely utilized - assert state.blob_builder_balances[header.builder_index] >= max_fee - - base_fee = state.shard_sample_price * samples - # Base fee must be paid - assert max_fee >= base_fee - - # Remaining fee goes towards proposer for prioritizing, up to a maximum - max_priority_fee = body_summary.max_priority_fee_per_sample * samples - priority_fee = min(max_fee - base_fee, max_priority_fee) - - # Burn base fee, take priority fee - # priority_fee <= max_fee - base_fee, thus priority_fee + base_fee <= max_fee, thus sufficient balance. - state.blob_builder_balances[header.builder_index] -= base_fee + priority_fee - # Pay out priority fee - increase_balance(state, header.proposer_index, priority_fee) - - # Initialize the pending header - index = compute_committee_index_from_shard(state, slot, shard) - committee_length = len(get_beacon_committee(state, slot, index)) - initial_votes = Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length) - pending_header = PendingShardHeader( - attested=AttestedDataCommitment( - commitment=body_summary.commitment, - root=header_root, - includer_index=get_beacon_proposer_index(state), - ), - votes=initial_votes, - weight=0, - update_slot=state.slot, - ) - - # Include it in the pending list - current_headers.append(pending_header) + # Verify proposer is not slashed + proposer = state.validators[block.proposer_index] + assert not proposer.slashed ``` -The degree proof works as follows. For a block `B` with length `l` (so `l` values in `[0...l - 1]`, seen as a polynomial `B(X)` which takes these values), -the length proof is the commitment to the polynomial `B(X) * X**(MAX_DEGREE + 1 - l)`, -where `MAX_DEGREE` is the maximum power of `s` available in the setup, which is `MAX_DEGREE = len(G2_SETUP) - 1`. -The goal is to ensure that a proof can only be constructed if `deg(B) < l` (there are not hidden higher-order terms in the polynomial, which would thwart reconstruction). - -##### `process_shard_proposer_slashing` +#### Builder Block Bid ```python -def process_shard_proposer_slashing(state: BeaconState, proposer_slashing: ShardProposerSlashing) -> None: - slot = proposer_slashing.slot - shard = proposer_slashing.shard - proposer_index = proposer_slashing.proposer_index +def verify_builder_block_bid(state: BeaconState, block: BeaconBlock) -> None: + if is_builder_block_slot(block.slot): + # Get last builder block bid + assert state.blocks_since_builder_block[-1].body.payload_data.selector == 0 + builder_block_bid = state.blocks_since_builder_block[-1].body.payload_data.value.builder_block_bid + assert builder_block_bid.slot + 1 == block.slot - reference_1 = ShardBlobReference(slot=slot, shard=shard, - proposer_index=proposer_index, - builder_index=proposer_slashing.builder_index_1, - body_root=proposer_slashing.body_root_1) - reference_2 = ShardBlobReference(slot=slot, shard=shard, - proposer_index=proposer_index, - builder_index=proposer_slashing.builder_index_2, - body_root=proposer_slashing.body_root_2) + assert block.body.payload_data.selector == 1 # Verify that builder block does not contain bid - # Verify the signed messages are different - assert reference_1 != reference_2 + builder_block_data = block.body.payload_data.value - # Verify the proposer is slashable - proposer = state.validators[proposer_index] - assert is_slashable_validator(proposer, get_current_epoch(state)) + assert builder_block_bid.execution_payload_root == hash_tree_root(builder_block_data.execution_payload) - # The builders are not slashed, the proposer co-signed with them - builder_pubkey_1 = state.blob_builders[proposer_slashing.builder_index_1].pubkey - builder_pubkey_2 = state.blob_builders[proposer_slashing.builder_index_2].pubkey - domain = get_domain(state, DOMAIN_SHARD_PROPOSER, compute_epoch_at_slot(slot)) - signing_root_1 = compute_signing_root(reference_1, domain) - signing_root_2 = compute_signing_root(reference_2, domain) - assert bls.FastAggregateVerify([builder_pubkey_1, proposer.pubkey], signing_root_1, proposer_slashing.signature_1) - assert bls.FastAggregateVerify([builder_pubkey_2, proposer.pubkey], signing_root_2, proposer_slashing.signature_2) + assert builder_block_bid.sharded_data_commitment_count == builder_block_data.included_sharded_data_commitments - slash_validator(state, proposer_index) + assert builder_block_bid.sharded_data_commitment_root == hash_tree_root(builder_block_data.sharded_commitments[-builder_block_bid.included_sharded_data_commitments:]) + + assert builder_block_bid.validator_index == block.proposer_index + + else: + assert block.body.payload_data.selector == 0 + + builder_block_bid = block.body.payload_data.value.builder_block_bid + assert builder_block_bid.slot == block.slot + assert builder_block_bid.parent_block_root == block.parent_root + # We do not check that the builder address exists or has sufficient balance here. + # If it does not have sufficient balance, the block proposer loses out, so it is their + # responsibility to check. + + # Check that the builder is a slashable validator. We can probably reduce this requirement and only + # ensure that they have 1 ETH in their account as a DOS protection. + builder = state.validators[builder_block_bid.validator_index] + assert is_slashable_validator(builder, get_current_epoch(state)) ``` -### Epoch transition - -This epoch transition overrides Bellatrix epoch transition: +#### Sharded data ```python -def process_epoch(state: BeaconState) -> None: - # Sharding pre-processing - process_pending_shard_confirmations(state) - reset_pending_shard_work(state) +def process_sharded_data(state: BeaconState, block: BeaconBlock) -> None: + if is_builder_block_slot(block.slot): + assert block.body.payload_data.selector == 1 + sharded_commitments_container = block.body.payload_data.value.sharded_commitments_container - # Base functionality - process_justification_and_finalization(state) - process_inactivity_updates(state) - process_rewards_and_penalties(state) # Note: modified, see new TIMELY_SHARD_FLAG_INDEX - process_registry_updates(state) - process_slashings(state) - process_eth1_data_reset(state) - process_effective_balance_updates(state) - process_slashings_reset(state) - process_randao_mixes_reset(state) - process_historical_roots_update(state) - process_participation_flag_updates(state) - process_sync_committee_updates(state) + # Verify not too many commitments + assert len(sharded_commitments_container.sharded_commitments) // 2 <= get_active_shard_count(state, get_current_epoch(state)) + + # Verify the degree proof + r = hash_to_bls_field(sharded_commitments_container.sharded_commitments, 0) + r_powers = compute_powers(r, len(sharded_commitments_container.sharded_commitments)) + combined_commitment = elliptic_curve_lincomb(sharded_commitments_container.sharded_commitments, r_powers) + + payload_field_elements_per_blob = SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE // 2 + + verify_degree_proof(combined_commitment, payload_field_elements_per_blob, sharded_commitments_container.degree_proof) + + # Verify that the 2*N commitments lie on a degree < N polynomial + low_degree_check(sharded_commitments_container.sharded_commitments) + + # Verify that blocks since the last builder block have been included + blocks_chunked = [bytes_to_field_elements(ssz_serialize(block)) for block in state.blocks_since_builder_block] + block_vectors = [] + + for block_chunked in blocks_chunked: + for i in range(0, len(block_chunked), payload_field_elements_per_blob): + block_vectors.append(block_chunked[i:i + payload_field_elements_per_blob]) + + number_of_blobs = len(block_vectors) + r = hash_to_bls_field(sharded_commitments_container.sharded_commitments[:number_of_blobs], 0) + x = hash_to_bls_field(sharded_commitments_container.sharded_commitments[:number_of_blobs], 1) + + r_powers = compute_powers(r, number_of_blobs) + combined_vector = vector_lincomb(block_vectors, r_powers) + combined_commitment = elliptic_curve_lincomb(sharded_commitments_container.sharded_commitments[:number_of_blobs], r_powers) + y = evaluate_polynomial_in_evaluation_form(combined_vector, x) + + verify_kzg_proof(combined_commitment, x, y, sharded_commitments_container.block_verification_kzg_proof) + + # Verify that number of sharded data commitments is correctly indicated + assert 2 * (number_of_blobs + included_sharded_data_commitments) == len(sharded_commitments_container.sharded_commitments) ``` -#### `process_pending_shard_confirmations` +#### Execution payload ```python -def process_pending_shard_confirmations(state: BeaconState) -> None: - # Pending header processing applies to the previous epoch. - # Skip if `GENESIS_EPOCH` because no prior epoch to process. - if get_current_epoch(state) == GENESIS_EPOCH: - return +def process_execution_payload(state: BeaconState, block: BeaconBlock, execution_engine: ExecutionEngine) -> None: + if is_builder_block_slot(block.slot): + assert block.body.payload_data.selector == 1 + payload = block.body.payload_data.value.execution_payload - previous_epoch = get_previous_epoch(state) - previous_epoch_start_slot = compute_start_slot_at_epoch(previous_epoch) + # Verify consistency of the parent hash with respect to the previous execution payload header + if is_merge_transition_complete(state): + assert payload.parent_hash == state.latest_execution_payload_header.block_hash + # Verify random + assert payload.random == get_randao_mix(state, get_current_epoch(state)) + # Verify timestamp + assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) - # Mark stale headers as unconfirmed - for slot in range(previous_epoch_start_slot, previous_epoch_start_slot + SLOTS_PER_EPOCH): - buffer_index = slot % SHARD_STATE_MEMORY_SLOTS - for shard_index in range(len(state.shard_buffer[buffer_index])): - committee_work = state.shard_buffer[buffer_index][shard_index] - if committee_work.status.selector == SHARD_WORK_PENDING: - winning_header = max(committee_work.status.value, key=lambda header: header.weight) - if winning_header.attested.commitment == DataCommitment(): - committee_work.status.change(selector=SHARD_WORK_UNCONFIRMED, value=None) - else: - committee_work.status.change(selector=SHARD_WORK_CONFIRMED, value=winning_header.attested) -``` + # Get sharded data commitments + sharded_commitments_container = block.body.sharded_commitments_container + sharded_data_commitments = sharded_commitments_container.sharded_commitments[-sharded_commitments_container.included_sharded_data_commitments:] -#### `reset_pending_shard_work` + # Get all unprocessed builder block bids + unprocessed_builder_block_bid_with_recipient_addresses = [] + for block in state.blocks_since_builder_block[1:]: + unprocessed_builder_block_bid_with_recipient_addresses.append(block.body.builder_block_bid_with_recipient_address.value) -```python -def reset_pending_shard_work(state: BeaconState) -> None: - # Add dummy "empty" PendingShardHeader (default vote if no shard header is available) - next_epoch = get_current_epoch(state) + 1 - next_epoch_start_slot = compute_start_slot_at_epoch(next_epoch) - committees_per_slot = get_committee_count_per_slot(state, next_epoch) - active_shards = get_active_shard_count(state, next_epoch) + # Verify the execution payload is valid + # The execution engine gets two extra payloads: One for the sharded data commitments (these are needed to verify type 3 transactions) + # and one for all so far unprocessed builder block bids: + # * The execution engine needs to transfer the balance from the bidder to the proposer. + # * The execution engine needs to deduct data gas fees from the bidder balances + assert execution_engine.execute_payload(payload, + sharded_data_commitments, + unprocessed_builder_block_bid_with_recipient_addresses) - for slot in range(next_epoch_start_slot, next_epoch_start_slot + SLOTS_PER_EPOCH): - buffer_index = slot % SHARD_STATE_MEMORY_SLOTS - - # Reset the shard work tracking - state.shard_buffer[buffer_index] = [ShardWork() for _ in range(active_shards)] - - start_shard = get_start_shard(state, slot) - for committee_index in range(committees_per_slot): - shard = (start_shard + committee_index) % active_shards - # a committee is available, initialize a pending shard-header list - committee_length = len(get_beacon_committee(state, slot, CommitteeIndex(committee_index))) - state.shard_buffer[buffer_index][shard].status.change( - selector=SHARD_WORK_PENDING, - value=List[PendingShardHeader, MAX_SHARD_HEADERS_PER_SHARD]( - PendingShardHeader( - attested=AttestedDataCommitment(), - votes=Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length), - weight=0, - update_slot=slot, - ) - ) - ) - # a shard without committee available defaults to SHARD_WORK_UNCONFIRMED. -``` + # Cache execution payload header + state.latest_execution_payload_header = ExecutionPayloadHeader( + parent_hash=payload.parent_hash, + fee_recipient=payload.fee_recipient, + state_root=payload.state_root, + receipt_root=payload.receipt_root, + logs_bloom=payload.logs_bloom, + random=payload.random, + block_number=payload.block_number, + gas_limit=payload.gas_limit, + gas_used=payload.gas_used, + timestamp=payload.timestamp, + extra_data=payload.extra_data, + base_fee_per_gas=payload.base_fee_per_gas, + block_hash=payload.block_hash, + transactions_root=hash_tree_root(payload.transactions), + ) +``` \ No newline at end of file diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index ab32c37fa..3b627a339 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -13,17 +13,14 @@ - [Misc](#misc) - [Gossip domain](#gossip-domain) - [Topics and messages](#topics-and-messages) - - [Shard blob subnets](#shard-blob-subnets) - - [`shard_blob_{subnet_id}`](#shard_blob_subnet_id) - - [Global topics](#global-topics) - - [`shard_blob_header`](#shard_blob_header) - - [`shard_blob_tx`](#shard_blob_tx) - - [`shard_proposer_slashing`](#shard_proposer_slashing) + - [Builder block bid](#builder-block-bid) + - [`builder_block_bid`](#builder_block_bid) + - [Shard sample subnets](#shard-sample-subnets) + - [`shard_row_{subnet_id}`](#shard_row_subnet_id) - ## Introduction The specification of these changes continues in the same format as the network specifications of previous upgrades, and assumes them as pre-requisite. @@ -33,12 +30,10 @@ The adjustments and additions for Shards are outlined in this document. ### Misc -| Name | Value | Description | -| ---- | ----- | ----------- | -| `SHARD_BLOB_SUBNET_COUNT` | `64` | The number of `shard_blob_{subnet_id}` subnets used in the gossipsub protocol. | -| `SHARD_TX_PROPAGATION_GRACE_SLOTS` | `4` | The number of slots for a late transaction to propagate | -| `SHARD_TX_PROPAGATION_BUFFER_SLOTS` | `8` | The number of slots for an early transaction to propagate | - +| Name | Value | Description | +| --------------------------- | ----- | -------------------------------------------------------------------------------- | +| `SHARD_ROW_SUBNET_COUNT` | `512` | The number of `shard_row_{subnet_id}` subnets used in the gossipsub protocol. | +| `SHARD_COLUMN_SUBNET_COUNT` | `512` | The number of `shard_column_{subnet_id}` subnets used in the gossipsub protocol. | ## Gossip domain @@ -48,130 +43,49 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |---------------------------------|--------------------------| -| `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_blob_header` | `SignedShardBlobHeader` | -| `shard_blob_tx` | `SignedShardBlobHeader` | -| `shard_proposer_slashing` | `ShardProposerSlashing` | +| `shard_row_{subnet_id}` | `SignedShardSample` | +| `shard_column_{subnet_id}` | `SignedShardSample` | +| `builder_block_bid` | `BuilderBlockBid` | The [DAS network specification](./das-p2p.md) defines additional topics. -#### Shard blob subnets +#### Builder block bid -Shard blob subnets are used by builders to make their blobs available after selection by shard proposers. +##### `builder_block_bid` -##### `shard_blob_{subnet_id}` +- _[IGNORE]_ The `bid` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- + i.e. validate that `bid.slot <= current_slot + 1` + (a client MAY queue future samples for propagation at the appropriate slot). +- _[IGNORE]_ The `bid` is for the current or next block + i.e. validate that `bid.slot >= current_slot` +- _[IGNORE]_ The `bid` is the first `bid` valid bid for `bid.slot`, or the bid is at least 1% higher than the previous known `bid` +- _[REJECT]_ The validator defined by `bid.validator_index` exists and is slashable. +- _[REJECT]_ The bid signature, which is an Eth1 signature, needs to be valid and the address needs to contain enough Ether to cover the bid and the data gas base fee. -Shard blob data, in the form of a `SignedShardBlob` is published to the `shard_blob_{subnet_id}` subnets. +#### Shard sample subnets -```python -def compute_subnet_for_shard_blob(state: BeaconState, slot: Slot, shard: Shard) -> uint64: - """ - Compute the correct subnet for a shard blob publication. - Note, this mimics compute_subnet_for_attestation(). - """ - committee_index = compute_committee_index_from_shard(state, slot, shard) - committees_per_slot = get_committee_count_per_slot(state, compute_epoch_at_slot(slot)) - slots_since_epoch_start = Slot(slot % SLOTS_PER_EPOCH) - committees_since_epoch_start = committees_per_slot * slots_since_epoch_start +Shard sample (row/column) subnets are used by builders to make their samples available as part of their intermediate block release after selection by beacon block proposers. - return uint64((committees_since_epoch_start + committee_index) % SHARD_BLOB_SUBNET_COUNT) -``` +##### `shard_row_{subnet_id}` -The following validations MUST pass before forwarding the `signed_blob`, -on the horizontal subnet or creating samples for it. Alias `blob = signed_blob.message`. +Shard sample data, in the form of a `SignedShardSample` is published to the `shard_row_{subnet_id}` and `shard_column_{subnet_id}` subnets. -- _[IGNORE]_ The `blob` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- - i.e. validate that `blob.slot <= current_slot + 1` - (a client MAY queue future blobs for propagation at the appropriate slot). -- _[IGNORE]_ The `blob` is new enough to still be processed -- - i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)` -- _[REJECT]_ The shard blob is for an active shard -- - i.e. `blob.shard < get_active_shard_count(state, compute_epoch_at_slot(blob.slot))` -- _[REJECT]_ The `blob.shard` MUST have a committee at the `blob.slot` -- - i.e. validate that `compute_committee_index_from_shard(state, blob.slot, blob.shard)` doesn't raise an error -- _[REJECT]_ The shard blob is for the correct subnet -- - i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id` -- _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination. -- _[REJECT]_ The blob is not too large -- the data MUST NOT be larger than the SSZ list-limit, and a client MAY apply stricter bounds. -- _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid. -- _[REJECT]_ The blob builder defined by `blob.builder_index` exists and has sufficient balance to back the fee payment. -- _[REJECT]_ The blob signature, `signed_blob.signature`, is valid for the aggregate of proposer and builder -- - i.e. `bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_blob.signature)`. -- _[REJECT]_ The blob is proposed by the expected `proposer_index` for the blob's `slot` and `shard`, - in the context of the current shuffling (defined by the current node head state and `blob.slot`). - If the `proposer_index` cannot immediately be verified against the expected shuffling, - the blob MAY be queued for later processing while proposers for the blob's branch are calculated -- - in such a case _do not_ `REJECT`, instead `IGNORE` this message. +The following validations MUST pass before forwarding the `sample`. -#### Global topics +- _[IGNORE]_ The `sample` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- + i.e. validate that `sample.slot <= current_slot + 1` + (a client MAY queue future samples for propagation at the appropriate slot). +- _[IGNORE]_ The `sample` is new enough to still be processed -- + i.e. validate that `compute_epoch_at_slot(sample.slot) >= get_previous_epoch(state)` +- _[REJECT]_ The shard sample is for the correct subnet -- + i.e. `sample.row == subnet_id` for `shard_row_{subnet_id}` and `sample.column == subnet_id` for `shard_column_{subnet_id}` +- _[IGNORE]_ The sample is the first sample with valid signature received for the `(sample.builder, sample.slot, sample.row, sample.column)` combination. +- _[REJECT]_ The `sample.data` MUST NOT contain any point `x >= BLS_MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid. +- _[REJECT]_ The validator defined by `sample.builder` exists and is slashable. +- _[REJECT]_ The sample is proposed by the expected `builder` for the sample's `slot`. + i.e., the beacon block at `sample.slot - 1` according to the node's fork choice contains an `IntermediateBlockBid` + with `intermediate_block_bid.validator_index == sample.builder` +- _[REJECT]_ The sample signature, `sample.signature`, is valid for the builder -- + i.e. `bls.Verify(builder_pubkey, sample_signing_root, sample.signature)` OR `sample.signature == Bytes96(b"\0" * 96)` AND + the sample verification `verify_sample` passes -There are three additional global topics for Sharding. - -- `shard_blob_header`: co-signed headers to be included on-chain and to serve as a signal to the builder to publish full data. -- `shard_blob_tx`: builder-signed headers, also known as "data transaction". -- `shard_proposer_slashing`: slashings of duplicate shard proposals. - -##### `shard_blob_header` - -Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet. -Shard blob headers select shard blob bids by builders -and should be timely to ensure builders can publish the full shard blob before subsequent attestations. - -The following validations MUST pass before forwarding the `signed_blob_header` on the network. Alias `header = signed_blob_header.message`. - -- _[IGNORE]_ The `header` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- - i.e. validate that `header.slot <= current_slot + 1` - (a client MAY queue future headers for propagation at the appropriate slot). -- _[IGNORE]_ The header is new enough to still be processed -- - i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)` -- _[REJECT]_ The shard header is for an active shard -- - i.e. `header.shard < get_active_shard_count(state, compute_epoch_at_slot(header.slot))` -- _[REJECT]_ The `header.shard` MUST have a committee at the `header.slot` -- - i.e. validate that `compute_committee_index_from_shard(state, header.slot, header.shard)` doesn't raise an error. -- _[IGNORE]_ The header is the first header with valid signature received for the `(header.proposer_index, header.slot, header.shard)` combination. -- _[REJECT]_ The blob builder defined by `blob.builder_index` exists and has sufficient balance to back the fee payment. -- _[REJECT]_ The header signature, `signed_blob_header.signature`, is valid for the aggregate of proposer and builder -- - i.e. `bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_blob_header.signature)`. -- _[REJECT]_ The header is proposed by the expected `proposer_index` for the blob's `header.slot` and `header.shard` - in the context of the current shuffling (defined by the current node head state and `header.slot`). - If the `proposer_index` cannot immediately be verified against the expected shuffling, - the blob MAY be queued for later processing while proposers for the blob's branch are calculated -- - in such a case _do not_ `REJECT`, instead `IGNORE` this message. - -##### `shard_blob_tx` - -Shard data-transactions in the form of a `SignedShardBlobHeader` are published to the global `shard_blob_tx` subnet. -These shard blob headers are signed solely by the blob-builder. - -The following validations MUST pass before forwarding the `signed_blob_header` on the network. Alias `header = signed_blob_header.message`. - -- _[IGNORE]_ The header is not propagating more than `SHARD_TX_PROPAGATION_BUFFER_SLOTS` slots ahead of time -- - i.e. validate that `header.slot <= current_slot + SHARD_TX_PROPAGATION_BUFFER_SLOTS`. -- _[IGNORE]_ The header is not propagating later than `SHARD_TX_PROPAGATION_GRACE_SLOTS` slots too late -- - i.e. validate that `header.slot + SHARD_TX_PROPAGATION_GRACE_SLOTS >= current_slot` -- _[REJECT]_ The shard header is for an active shard -- - i.e. `header.shard < get_active_shard_count(state, compute_epoch_at_slot(header.slot))` -- _[REJECT]_ The `header.shard` MUST have a committee at the `header.slot` -- - i.e. validate that `compute_committee_index_from_shard(state, header.slot, header.shard)` doesn't raise an error. -- _[IGNORE]_ The header is not stale -- i.e. the corresponding shard proposer has not already selected a header for `(header.slot, header.shard)`. -- _[IGNORE]_ The header is the first header with valid signature received for the `(header.builder_index, header.slot, header.shard)` combination. -- _[REJECT]_ The blob builder, define by `header.builder_index`, exists and has sufficient balance to back the fee payment. -- _[IGNORE]_ The header fee SHOULD be higher than previously seen headers for `(header.slot, header.shard)`, from any builder. - Propagating nodes MAY increase fee increments in case of spam. -- _[REJECT]_ The header signature, `signed_blob_header.signature`, is valid for ONLY the builder -- - i.e. `bls.Verify(builder_pubkey, blob_signing_root, signed_blob_header.signature)`. The signature is not an aggregate with the proposer. -- _[REJECT]_ The header is designated for proposal by the expected `proposer_index` for the blob's `header.slot` and `header.shard` - in the context of the current shuffling (defined by the current node head state and `header.slot`). - If the `proposer_index` cannot immediately be verified against the expected shuffling, - the blob MAY be queued for later processing while proposers for the blob's branch are calculated -- - in such a case _do not_ `REJECT`, instead `IGNORE` this message. - -##### `shard_proposer_slashing` - -Shard proposer slashings, in the form of `ShardProposerSlashing`, are published to the global `shard_proposer_slashing` topic. - -The following validations MUST pass before forwarding the `shard_proposer_slashing` on to the network. -- _[IGNORE]_ The shard proposer slashing is the first valid shard proposer slashing received - for the proposer with index `proposer_slashing.proposer_index`. - The `proposer_slashing.slot` and `proposer_slashing.shard` are ignored, there are no repeated or per-shard slashings. -- _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation. diff --git a/specs/sharding/polynomial-commitments.md b/specs/sharding/polynomial-commitments.md new file mode 100644 index 000000000..e2a4285ca --- /dev/null +++ b/specs/sharding/polynomial-commitments.md @@ -0,0 +1,396 @@ +# Sharding -- Polynomial Commitments + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Constants](#constants) + - [BLS Field](#bls-field) + - [KZG Trusted setup](#kzg-trusted-setup) +- [Custom types](#custom-types) +- [Helper functions](#helper-functions) + - [`next_power_of_two`](#next_power_of_two) + - [`reverse_bit_order`](#reverse_bit_order) + - [`list_to_reverse_bit_order`](#list_to_reverse_bit_order) +- [Field operations](#field-operations) + - [Generic field operations](#generic-field-operations) + - [`bls_modular_inverse`](#bls_modular_inverse) + - [`roots_of_unity`](#roots_of_unity) + - [Field helper functions](#field-helper-functions) + - [`compute_powers`](#compute_powers) + - [`low_degree_check`](#low_degree_check) + - [`vector_lincomb`](#vector_lincomb) + - [`bytes_to_field_elements`](#bytes_to_field_elements) +- [Polynomial operations](#polynomial-operations) + - [`add_polynomials`](#add_polynomials) + - [`multiply_polynomials`](#multiply_polynomials) + - [`interpolate_polynomial`](#interpolate_polynomial) + - [`evaluate_polynomial_in_evaluation_form`](#evaluate_polynomial_in_evaluation_form) +- [KZG Operations](#kzg-operations) + - [Elliptic curve helper functoins](#elliptic-curve-helper-functoins) + - [`elliptic_curve_lincomb`](#elliptic_curve_lincomb) + - [Hash to field](#hash-to-field) + - [`hash_to_bls_field`](#hash_to_bls_field) + - [KZG operations](#kzg-operations) + - [`verify_kzg_proof`](#verify_kzg_proof) + - [`verify_kzg_multiproof`](#verify_kzg_multiproof) + - [`verify_degree_proof`](#verify_degree_proof) + + + + + +## Introduction + +This document specifies basic polynomial operations and KZG polynomial commitment operations as they are needed for the sharding specification. The implementations are not optimized for performance, but readability. All practical implementations should optimize the polynomial operations, and hints what the best known algorithms for these implementations are are included below. + +## Constants + +### BLS Field + +| Name | Value | Notes | +| - | - | - | +| `BLS_MODULUS` | `0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001` (curve order of BLS12_381) | +| `PRIMITIVE_ROOT_OF_UNITY` | `7` | Primitive root of unity of the BLS12_381 (inner) BLS_MODULUS | + +### KZG Trusted setup + +| Name | Value | +| - | - | +| `G1_SETUP` | Type `List[G1]`. The G1-side trusted setup `[G, G*s, G*s**2....]`; note that the first point is the generator. | +| `G2_SETUP` | Type `List[G2]`. The G2-side trusted setup `[G, G*s, G*s**2....]` | + +## Custom types + +We define the following Python custom types for type hinting and readability: + +| Name | SSZ equivalent | Description | +| - | - | - | +| `KZGCommitment` | `Bytes48` | A G1 curve point | +| `BLSFieldElement` | `uint256` | A number `x` in the range `0 <= x < BLS_MODULUS` | +| `BLSPolynomialByCoefficients` | `List[BLSFieldElement]` | A polynomial over the BLS field, given in coefficient form | +| `BLSPolynomialByEvaluations` | `List[BLSFieldElement]` | A polynomial over the BLS field, given in evaluation form | + +## Helper functions + +#### `next_power_of_two` + +```python +def next_power_of_two(x: int) -> int: + assert x > 0 + return 2 ** ((x - 1).bit_length()) +``` + +#### `reverse_bit_order` + +```python +def reverse_bit_order(n: int, order: int) -> int: + """ + Reverse the bit order of an integer n + """ + assert is_power_of_two(order) + # Convert n to binary with the same number of bits as "order" - 1, then reverse its bit order + return int(('{:0' + str(order.bit_length() - 1) + 'b}').format(n)[::-1], 2) +``` + +#### `list_to_reverse_bit_order` + +```python +def list_to_reverse_bit_order(l: List[int]) -> List[int]: + """ + Convert a list between normal and reverse bit order. This operation is idempotent. + """ + return [l[reverse_bit_order(i, len(l))] for i in range(len(l))] +``` + +## Field operations + +### Generic field operations + +#### `bls_modular_inverse` + +```python +def bls_modular_inverse(x: BLSFieldElement) -> BLSFieldElement: + """ + Compute the modular inverse of x, i.e. y such that x * y % BLS_MODULUS == 1 and return 1 for x == 0 + """ + lm, hm = 1, 0 + low, high = x % BLS_MODULUS, BLS_MODULUS + while low > 1: + r = high // low + nm, new = hm - lm * r, high - low * r + lm, low, hm, high = nm, new, lm, low + return lm % BLS_MODULUS +``` + +#### `roots_of_unity` + +```python +def roots_of_unity(order: uint64) -> List[BLSFieldElement]: + """ + Compute a list of roots of unity for a given order. + The order must divide the BLS multiplicative group order, i.e. BLS_MODULUS - 1 + """ + assert (BLS_MODULUS - 1) % order == 0 + roots = [] + root_of_unity = pow(PRIMITIVE_ROOT_OF_UNITY, (BLS_MODULUS - 1) // order, BLS_MODULUS) + + current_root_of_unity = 1 + for i in range(SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE): + roots.append(current_root_of_unity) + current_root_of_unity = current_root_of_unity * root_of_unity % BLS_MODULUS + return roots +``` + +### Field helper functions + +#### `compute_powers` + +```python +def compute_powers(x: BLSFieldElement, n: uint64) -> List[BLSFieldElement]: + current_power = 1 + powers = [] + for _ in range(n): + powers.append(BLSFieldElement(current_power)) + current_power = current_power * int(x) % BLS_MODULUS + return powers +``` + +#### `low_degree_check` + +```python +def low_degree_check(commitments: List[KZGCommitment]): + """ + Checks that the commitments are on a low-degree polynomial. + If there are 2*N commitments, that means they should lie on a polynomial + of degree d = K - N - 1, where K = next_power_of_two(2*N) + (The remaining positions are filled with 0, this is to make FFTs usable) + + For details see here: https://notes.ethereum.org/@dankrad/barycentric_low_degree_check + """ + assert len(commitments) % 2 == 0 + N = len(commitments) // 2 + r = hash_to_bls_field(commitments, 0) + K = next_power_of_two(2 * N) + d = K - N - 1 + r_to_K = pow(r, N, K) + roots = list_to_reverse_bit_order(roots_of_unity(K)) + + # For an efficient implementation, B and Bprime should be precomputed + def B(z): + r = 1 + for w in roots[:d + 1]: + r = r * (z - w) % BLS_MODULUS + return r + + def Bprime(z): + r = 0 + for i in range(d + 1): + m = 1 + for w in roots[:i] + roots[i + 1:d + 1]: + m = m * (z - w) % BLS_MODULUS + r = (r + m) % BLS_MODULUS + return r + + coefs = [] + for i in range(K): + coefs.append( - (r_to_K - 1) * bls_modular_inverse(K * roots[i * (K - 1) % K] * (r - roots[i])) % BLS_MODULUS) + for i in range(d + 1): + coefs[i] = (coefs[i] + B(r) * bls_modular_inverse(Bprime(r) * (r - roots[i]))) % BLS_MODULUS + + assert elliptic_curve_lincomb(commitments, coefs) == bls.inf_G1() +``` + +#### `vector_lincomb` + +```python +def vector_lincomb(vectors: List[List[BLSFieldElement]], scalars: List[BLSFieldElement]) -> List[BLSFieldElement]: + """ + Compute a linear combination of field element vectors. + """ + r = [0]*len(vectors[0]) + for v, a in zip(vectors, scalars): + for i, x in enumerate(v): + r[i] = (r[i] + a * x) % BLS_MODULUS + return [BLSFieldElement(x) for x in r] +``` + +#### `bytes_to_field_elements` + +```python +def bytes_to_field_elements(block: bytes) -> List[BLSFieldElement]: + """ + Slices a block into 31-byte chunks that can fit into field elements. + """ + sliced_block = [block[i:i + 31] for i in range(0, len(bytes), 31)] + return [BLSFieldElement(int.from_bytes(x, "little")) for x in sliced_block] +``` + +## Polynomial operations + +#### `add_polynomials` + +```python +def add_polynomials(a: BLSPolynomialByCoefficients, b: BLSPolynomialByCoefficients) -> BLSPolynomialByCoefficients: + """ + Sum the polynomials ``a`` and ``b`` given by their coefficients. + """ + a, b = (a, b) if len(a) >= len(b) else (b, a) + return [(a[i] + (b[i] if i < len(b) else 0)) % BLS_MODULUS for i in range(len(a))] +``` + +#### `multiply_polynomials` + +```python +def multiply_polynomials(a: BLSPolynomialByCoefficients, b: BLSPolynomialByCoefficients) -> BLSPolynomialByCoefficients: + """ + Multiplies the polynomials `a` and `b` given by their coefficients + """ + r = [0] + for power, coef in enumerate(a): + summand = [0] * power + [coef * x % BLS_MODULUS for x in b] + r = add_polynomials(r, summand) + return r +``` + + +#### `interpolate_polynomial` + +```python +def interpolate_polynomial(xs: List[BLSFieldElement], ys: List[BLSFieldElement]) -> BLSPolynomialByCoefficients: + """ + Lagrange interpolation + """ + assert len(xs) == len(ys) + r = [0] + + for i in range(len(xs)): + summand = [ys[i]] + for j in range(len(ys)): + if j != i: + weight_adjustment = bls_modular_inverse(xs[j] - xs[i]) + summand = multiply_polynomials( + summand, [weight_adjustment, ((BLS_MODULUS - weight_adjustment) * xs[i])] + ) + r = add_polynomials(r, summand) + + return r +``` + +#### `evaluate_polynomial_in_evaluation_form` + +```python +def evaluate_polynomial_in_evaluation_form(poly: BLSPolynomialByEvaluations, x: BLSFieldElement) -> BLSFieldElement: + """ + Evaluates a polynomial (in evaluation form) at an arbitrary point + """ + field_elements_per_blob = SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE + roots = roots_of_unity(field_elements_per_blob) + + def A(z): + r = 1 + for w in roots: + r = r * (z - w) % BLS_MODULUS + return r + + def Aprime(z): + return field_elements_per_blob * pow(z, field_elements_per_blob - 1, BLS_MODULUS) + + r = 0 + inverses = [bls_modular_inverse(z - x) for z in roots] + for i, x in enumerate(inverses): + r += poly[i] * bls_modular_inverse(Aprime(roots[i])) * x % BLS_MODULUS + r = r * A(x) % BLS_MODULUS + return r +``` + +## KZG Operations + +We are using the KZG10 polynomial commitment scheme (Kate, Zaverucha and Goldberg, 2010: https://www.iacr.org/archive/asiacrypt2010/6477178/6477178.pdf). + +### Elliptic curve helper functoins + +#### `elliptic_curve_lincomb` + +```python +def elliptic_curve_lincomb(points: List[KZGCommitment], scalars: List[BLSFieldElement]) -> KZGCommitment: + """ + BLS multiscalar multiplication. This function can be optimized using Pippenger's algorithm and variants. + This is a non-optimized implementation. + """ + r = bls.inf_G1() + for x, a in zip(points, scalars): + r = r.add(x.mult(a)) + return r +``` + +### Hash to field + +#### `hash_to_bls_field` + +```python +def hash_to_bls_field(x: Container, challenge_number: uint64) -> BLSFieldElement: + """ + This function is used to generate Fiat-Shamir challenges. The output is not uniform over the BLS field. + """ + return ( + (int.from_bytes(hash(hash_tree_root(x) + int.to_bytes(challenge_number, 32, "little")), "little")) + % BLS_MODULUS + ) +``` + +### KZG operations + +#### `verify_kzg_proof` + +```python +def verify_kzg_proof(commitment: KZGCommitment, x: BLSFieldElement, y: BLSFieldElement, proof: KZGCommitment) -> None: + """ + Check that `proof` is a valid KZG proof for the polynomial committed to by `commitment` evaluated + at `x` equals `y`. + """ + zero_poly = G2_SETUP[1].add(G2_SETUP[0].mult(x).neg()) + + assert ( + bls.Pairing(proof, zero_poly) + == bls.Pairing(commitment.add(G1_SETUP[0].mult(y).neg), G2_SETUP[0]) + ) +``` + +#### `verify_kzg_multiproof` + +```python +def verify_kzg_multiproof(commitment: KZGCommitment, + xs: List[BLSFieldElement], + ys: List[BLSFieldElement], + proof: KZGCommitment) -> None: + """ + Verify a KZG multiproof. + """ + zero_poly = elliptic_curve_lincomb(G2_SETUP[:len(xs)], interpolate_polynomial(xs, [0] * len(ys))) + interpolated_poly = elliptic_curve_lincomb(G2_SETUP[:len(xs)], interpolate_polynomial(xs, ys)) + + assert ( + bls.Pairing(proof, zero_poly) + == bls.Pairing(commitment.add(interpolated_poly.neg()), G2_SETUP[0]) + ) +``` + +#### `verify_degree_proof` + +```python +def verify_degree_proof(commitment: KZGCommitment, degree_bound: uint64, proof: KZGCommitment): + """ + Verifies that the commitment is of polynomial degree < degree_bound. + """ + + assert ( + bls.Pairing(proof, G2_SETUP[0]) + == bls.Pairing(commitment, G2_SETUP[-degree_bound]) + ) +``` \ No newline at end of file diff --git a/specs/sharding/validator.md b/specs/sharding/validator.md new file mode 100644 index 000000000..38914095f --- /dev/null +++ b/specs/sharding/validator.md @@ -0,0 +1,141 @@ +# Sharding -- Honest Validator + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + + - [Introduction](#introduction) + - [Prerequisites](#prerequisites) + - [Constants](#constants) + - [Sample counts](#sample-counts) + - [Helpers](#helpers) + - [`get_validator_row_subnets`](#get_validator_row_subnets) + - [`get_validator_column_subnets`](#get_validator_column_subnets) + - [`reconstruct_polynomial`](#reconstruct_polynomial) + - [Sample verification](#sample-verification) + - [`verify_sample`](#verify_sample) +- [Beacon chain responsibilities](#beacon-chain-responsibilities) + - [Validator assignments](#validator-assignments) + - [Attesting](#attesting) +- [Sample reconstruction](#sample-reconstruction) + - [Minimum online validator requirement](#minimum-online-validator-requirement) + + + + +## Introduction + +This document represents the changes to be made in the code of an "honest validator" to implement executable beacon chain proposal. + +## Prerequisites + +This document is an extension of the [Bellatrix -- Honest Validator](../bellatrix/validator.md) guide. +All behaviors and definitions defined in this document, and documents it extends, carry over unless explicitly noted or overridden. + +All terminology, constants, functions, and protocol mechanics defined in the updated Beacon Chain doc of [Sharding](./beacon-chain.md) are requisite for this document and used throughout. +Please see related Beacon Chain doc before continuing and use them as a reference throughout. + +## Constants + +### Sample counts + +| Name | Value | +| - | - | +| `VALIDATOR_SAMPLE_ROW_COUNT` | `2` | +| `VALIDATOR_SAMPLE_COLUMN_COUNT` | `2` | + +## Helpers + +### `get_validator_row_subnets` + +TODO: Currently the subnets are public (i.e. anyone can derive them.) This is good for a proof of custody with public verifiability, but bad for validator privacy. + +```python +def get_validator_row_subnets(validator: Validator, epoch: Epoch) -> List[uint64]: + return [int.from_bytes(hash_tree_root([validator.pubkey, 0, i])) for i in range(VALIDATOR_SAMPLE_ROW_COUNT)] +``` + +### `get_validator_column_subnets` + +```python +def get_validator_column_subnets(validator: Validator, epoch: Epoch) -> List[uint64]: + return [int.from_bytes(hash_tree_root([validator.pubkey, 1, i])) for i in range(VALIDATOR_SAMPLE_COLUMN_COUNT)] +``` + +### `reconstruct_polynomial` + +```python +def reconstruct_polynomial(samples: List[SignedShardSample]) -> List[SignedShardSample]: + """ + Reconstructs one full row/column from at least 1/2 of the samples + """ + +``` + +## Sample verification + +### `verify_sample` + +```python +def verify_sample(state: BeaconState, block: BeaconBlock, sample: SignedShardSample): + assert sample.row < 2 * get_active_shard_count(state, get_current_epoch(block.slot)) + assert sample.column < 2 * SAMPLES_PER_BLOB + assert block.slot == sample.slot + + # Verify builder signature. + # TODO: We should probably not do this. This should only be done by p2p to verify samples *before* intermediate block is in + # builder = state.validators[signed_block.message.proposer_index] + # signing_root = compute_signing_root(sample, get_domain(state, DOMAIN_SHARD_SAMPLE)) + # assert bls.Verify(sample.builder, signing_root, sample.signature) + + roots_in_rbo = list_to_reverse_bit_order(roots_of_unity(SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE)) + + # Verify KZG proof + verify_kzg_multiproof(block.body.payload_data.value.sharded_commitments_container.sharded_commitments[sample.row], + roots_in_rbo[sample.column * FIELD_ELEMENTS_PER_SAMPLE:(sample.column + 1) * FIELD_ELEMENTS_PER_SAMPLE] + sample.data, + sample.proof) +``` + +# Beacon chain responsibilities + +## Validator assignments + +### Attesting + +Every attester is assigned `VALIDATOR_SAMPLE_ROW_COUNT` rows and `VALIDATOR_SAMPLE_COLUMN_COUNT` columns of shard samples. As part of their validator duties, they should subscribe to the subnets given by `get_validator_row_subnets` and `get_validator_column_subnets`, for the whole epoch. + +A row or column is *available* for a `slot` if at least half of the total number of samples were received on the subnet and passed `verify_sample`. Otherwise it is called unavailable. + +If a validator is assigned to an attestation at slot `attestation_slot` and had his previous attestation duty at `previous_attestation_slot`, then they should only attest under the following conditions: + + * For all intermediate blocks `block` with `previous_attestation_slot < block.slot <= attestation_slot`: All sample rows and columns assigned to the validator were available. + +If this condition is not fulfilled, then the validator should instead attest to the last block for which the condition holds. + +This leads to the security property that a chain that is not fully available cannot have more than 1/16th of all validators voting for it. TODO: This claim is for an "infinite number" of validators. Compute the concrete security due to sampling bias. + +# Sample reconstruction + +A validator that has received enough samples of a row or column to mark it as available, should reconstruct all samples in that row/column (if they aren't all available already.) The function `reconstruct_polynomial` gives an example implementation for this. + +Once they have run the reconstruction function, they should distribute the samples that they reconstructed on all pubsub that +the local node is subscribed to, if they have not already received that sample on that pubsub. As an example: + + * The validator is subscribed to row `2` and column `5` + * The sample `(row, column) = (2, 5)` is missing in the column `5` pubsub + * After they have reconstruction of row `2`, the validator should send the sample `(2, 5)` on to the row `2` pubsub (if it was missing) as well as the column `5` pubsub. + +TODO: We need to verify the total complexity of doing this and make sure this does not cause too much load on a validator + +## Minimum online validator requirement + +The data availability construction guarantees that reconstruction is possible if 75% of all samples are available. In this case, at least 50% of all rows and 50% of all columns are independently available. In practice, it is likely that some supernodes will centrally collect all samples and fill in any gaps. However, we want to build a system that reliably reconstructs even absent all supernodes. Any row or column with 50% of samples will easily be reconstructed even with only 100s of validators online; so the only question is how we get to 50% of samples for all rows and columns, when some of them might be completely unseeded. + +Each validator will transfer 4 samples between rows and columns where there is overlap. Without loss of generality, look at row 0. Each validator has 1/128 chance of having a sample in this row, and we need 256 samples to reconstruct it. So we expect that we need ~256 * 128 = 32,768 validators to have a fair chance of reconstructing it if it was completely unseeded. + +A more elaborate estimate [here](https://notes.ethereum.org/@dankrad/minimum-reconstruction-validators) needs about 55,000 validators to be online for high safety that each row and column will be reconstructed. \ No newline at end of file