mirror of
https://github.com/vacp2p/rfc-index.git
synced 2026-01-10 08:08:17 -05:00
Update slot-builder with fixes
This commit is contained in:
Jimmy Debe
@@ -5,18 +5,19 @@ status: raw
|
||||
tags: codex
|
||||
editor:
|
||||
contributors:
|
||||
- Jimmy Debe <jimmy@status.im>
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
This document describes the Codex slot builder mechanism.
|
||||
Slots used in the Codex protocol is an important component to node collaboration in the network.
|
||||
Slots used in the Codex protocol are an important component of node collaboration in the network.
|
||||
|
||||
## Background
|
||||
|
||||
The Codex protocol places a dataset into blocks before sending a storage request to the network.
|
||||
Slots control and facilitates the distrubtion of the data blocks to participating storage providers.
|
||||
The mechanism builds individual Merkle trees for each slot enabling cell-level proof generation, and
|
||||
Slots control and facilitate the distribution of the data blocks to participating storage providers.
|
||||
The mechanism builds individual Merkle trees for each slot, enabling cell-level proof generation, and
|
||||
constructs a root verification tree over all slot roots.
|
||||
|
||||
## Specification
|
||||
@@ -28,32 +29,38 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL
|
||||
A Codex client wanting to present a dataset to the network will present a set of erasure encoded data blocks,
|
||||
as described in the [CODEX-ERASURE-CODING](./erasure-coding.md) specification.
|
||||
These data blocks will be placed into slots for storage providers to access.
|
||||
The slot building process MUST to contruct a block digest merkle tree from the data blocks.
|
||||
The root hashes from this tree is used as the leaves in a slot merkle tree.
|
||||
The slot building process MUST construct a block digest Merkle tree from the data blocks.
|
||||
The root hashes from this tree are used as the leaves in a slot merkle tree.
|
||||
|
||||
A prepared dataset will be presented to storage providers in the form of slots.
|
||||
Slot represents a data block cell to be stored based on a storage contract terms.
|
||||
Based on the storage contract,
|
||||
storage providers SHOULD be able to locate a specifc data block needing to be stored.
|
||||
The prepared dataset is presented to storage providers in the form of slots.
|
||||
A slot represents the location of a data block cell with an open storage contract.
|
||||
Storage providers SHOULD be able to locate a specific data block and
|
||||
all the details of the storage contract.
|
||||
See, the [CODEX-MARKETPLACE](./marketplace.md) specification.
|
||||
|
||||
### Contruct the Slot Tree
|
||||
### Construct a Slot Tree
|
||||
|
||||
A slot stores a list of root hashes that help in the retrieval of a dataset.
|
||||
The block digest tree SHOULD be contructed before building any slots.
|
||||
A block is divided into cells that are then hashed and
|
||||
those hashes are used to create a [Posieden2](https://eprint.iacr.org/2023/323) based merkle tree.
|
||||
The block size must be divisible by the cell size for block tree constuction.
|
||||
#### Block Digest Tree
|
||||
|
||||
A slot stores a list of root hashes that help with the retrieval of a dataset.
|
||||
The block digest tree SHOULD be constructed before building any slots.
|
||||
A data block is divided into cells that are hashed.
|
||||
The block size MUST be divisible by the cell size for the block digest tree construction.
|
||||
|
||||
$$
|
||||
\text{Cell size} \mid \text{Block size (in bytes)}
|
||||
$$
|
||||
|
||||
A block digest tree SHOULD contain the unqiue root hashes of blocks of the entire dataset,
|
||||
which MAY also be based on the Posiden2 algorithm.
|
||||
A block digest tree SHOULD contain the unique root hashes of blocks of the entire dataset,
|
||||
which MAY be based on the [Posieden2](https://eprint.iacr.org/2023/323) algorithm.
|
||||
The result of one digest tree will be represented by the root hash of the tree.
|
||||
|
||||
#### Slot Tree
|
||||
|
||||
#### Slot Tree Contruction
|
||||
|
||||
A slot tree represents one slot,
|
||||
which includes the list of digest root hashes.
|
||||
If a block is empty,
|
||||
the slot branch SHOULD be a hash of an empty block.
|
||||
Some slots MAY be empty,
|
||||
depending on the size of the dataset.
|
||||
|
||||
@@ -61,44 +68,59 @@ $$
|
||||
\text{Blocks per slot} = \frac{\text{Total blocks}}{\text{Number of slots}}
|
||||
$$
|
||||
|
||||
The cells per slot tree branch MUST be padded to a power of two.
|
||||
This will ensure a balanced slot Merkle tree.
|
||||
|
||||
$$
|
||||
\text{Cells per slot} = \text{Blocks per slot} \times \text{Cells per block}
|
||||
$$
|
||||
|
||||
Below are the REQUIRED values to build a slot.
|
||||
|
||||
``` nim
|
||||
|
||||
type SlotsBuilder*[T, H] = ref object of RootObj
|
||||
store: BlockStore # Storage backend for blocks
|
||||
manifest: Manifest # Current dataset manifest
|
||||
strategy: IndexingStrategy # Block indexing strategy
|
||||
cellSize: NBytes # Size of each cell in bytes
|
||||
numSlotBlocks: Natural # Blocks per slot (including padding)
|
||||
slotRoots: seq[H] # Computed slot root hashes
|
||||
emptyBlock: seq[byte] # Pre-allocated empty block data
|
||||
verifiableTree: ?T # Optional verification tree
|
||||
emptyDigestTree: T # Pre-computed empty block tree
|
||||
store: BlockStore # Storage backend for blocks
|
||||
manifest: Manifest # Current dataset manifest
|
||||
strategy: IndexingStrategy # Block indexing strategy
|
||||
cellSize: NBytes # Size of each cell in bytes
|
||||
numSlotBlocks: Natural # Blocks per slot (including padding)
|
||||
slotRoots: seq[H] # Computed slot root hashes
|
||||
emptyBlock: seq[byte] # Pre-allocated empty block data
|
||||
verifiableTree: ?T # Optional verification tree
|
||||
emptyDigestTree: T # Pre-computed empty block tree
|
||||
|
||||
```
|
||||
|
||||
### Verification Tree
|
||||
|
||||
Nodes within the network are REQUIRED to verify a dataset before retrieving it.
|
||||
A verification tree is a merkle proof derived from the `slotRoot`.
|
||||
The entire dataset is not REQUIRED to contruct the tree.
|
||||
|
||||
The following parameters SHOULD be presented:
|
||||
A verification tree is a Merkle proof derived from the `slotRoot`.
|
||||
The entire dataset is not REQUIRED to construct the tree.
|
||||
|
||||
The following are the inputs to verify a proof:
|
||||
|
||||
```nim
|
||||
|
||||
type
|
||||
H = array[32, byte]
|
||||
Natural = uint64
|
||||
|
||||
type ProofInputs*[H] = object
|
||||
entropy*: H # Randomness value
|
||||
datasetRoot*: H # Dataset root hash
|
||||
slotIndex*: Natural # Slot identifier
|
||||
slotRoot*: H # Root hash of slot
|
||||
nCellsPerSlot*: Natural # Cell count per slot
|
||||
nSlotsPerDataSet*: Natural # Total slot count
|
||||
slotProof*: seq[H] # Inclusion proof for slot in dataset
|
||||
samples*: seq[Sample[H]] # Cell inclusion proofs
|
||||
entropy*: H # Randomness value
|
||||
datasetRoot*: H # Dataset root hash
|
||||
slotIndex*: Natural # Slot identifier
|
||||
slotRoot*: H # Root hash of slot
|
||||
nCellsPerSlot*: Natural # Cell count per slot
|
||||
nSlotsPerDataSet*: Natural # Total slot count
|
||||
slotProof*: seq[H] # Inclusion proof for slot in dataset
|
||||
samples*: seq[Sample[H]] # Cell inclusion proofs
|
||||
|
||||
```
|
||||
|
||||
To verify, a node MUST recompute the root hash,
|
||||
based on `slotProof` and the hash of the `slotIndex`,
|
||||
to confirm that the `slotIndex` is a member of the dataset represented by `datasetRoot`.
|
||||
|
||||
## Copyright
|
||||
|
||||
@@ -107,5 +129,5 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
|
||||
## References
|
||||
|
||||
- [CODEX-ERASURE-CODING](./erasure-coding.md)
|
||||
- [CODEX-MARKETPLACE](./marketplace.md)
|
||||
- [Posieden2](https://eprint.iacr.org/2023/323)
|
||||
-
|
||||
|
||||
Reference in New Issue
Block a user