Updates

Based on recent learnings.

---------

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>

codex/raw/slot-buIlder.md

@@ -0,0 +1,133 @@
+---
+title: CODEX-SLOT-BUILDER
+name: Codex Slot Builder
+status: raw
+tags: codex
+editor:
+contributors:
+- Jimmy Debe <jimmy@status.im>
+---
+
+## Abstract
+
+This document describes the Codex slot builder mechanism.
+Slots used in the Codex protocol are an important component of node collaboration in the network.
+
+## Background
+
+The Codex protocol places a dataset into blocks before sending a storage request to the network.
+Slots control and facilitate the distribution of the data blocks to participating storage providers.
+The mechanism builds individual Merkle trees for each slot, enabling cell-level proof generation, and
+constructs a root verification tree over all slot roots.
+
+## Specification
+
+The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
+“SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and
+“OPTIONAL” in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
+
+A Codex client wanting to present a dataset to the network will present a set of erasure encoded data blocks,
+as described in the [CODEX-ERASURE-CODING](./erasure-coding.md) specification.
+These data blocks will be placed into slots for storage providers to access.
+The slot building process MUST construct a block digest Merkle tree from the data blocks.
+The root hashes from this tree are used as the leaves in a slot merkle tree.
+
+The prepared dataset is presented to storage providers in the form of slots.
+A slot represents the location of a data block cell with an open storage contract.
+Storage providers SHOULD be able to locate a specific data block and
+all the details of the storage contract.
+See, the [CODEX-MARKETPLACE](./marketplace.md) specification.
+
+### Construct a Slot Tree
+
+#### Block Digest Tree
+
+A slot stores a list of root hashes that help with the retrieval of a dataset.
+The block digest tree SHOULD be constructed before building any slots.
+A data block is divided into cells that are hashed.
+The block size MUST be divisible by the cell size for the block digest tree construction.
+
+$$
+\text{Cell size} \mid \text{Block size (in bytes)}
+$$
+
+A block digest tree SHOULD contain the unique root hashes of blocks of the entire dataset,
+which MAY be based on the [Poseidon2](https://eprint.iacr.org/2023/323) algorithm.
+The result of one digest tree will be represented by the root hash of the tree.
+
+#### Slot Tree
+
+A slot tree represents one slot,
+which includes the list of digest root hashes.
+If a block is empty,
+the slot branch SHOULD be a hash of an empty block.
+Some slots MAY be empty,
+depending on the size of the dataset.
+
+$$
+\text{Blocks per slot} = \frac{\text{Total blocks}}{\text{Number of slots}}
+$$
+
+The cells per slot tree branch MUST be padded to a power of two.
+This will ensure a balanced slot Merkle tree.
+
+$$
+\text{Cells per slot} = \text{Blocks per slot} \times \text{Cells per block}
+$$
+
+Below are the REQUIRED values to build a slot.
+
+``` nim
+
+type SlotsBuilder*[T, H] = ref object of RootObj
+ store: BlockStore              # Storage backend for blocks
+ manifest: Manifest             # Current dataset manifest
+ strategy: IndexingStrategy     # Block indexing strategy
+ cellSize: NBytes               # Size of each cell in bytes
+ numSlotBlocks: Natural         # Blocks per slot (including padding)
+ slotRoots: seq[H]              # Computed slot root hashes
+ emptyBlock: seq[byte]          # Pre-allocated empty block data
+ verifiableTree: ?T             # Optional verification tree
+ emptyDigestTree: T             # Pre-computed empty block tree
+
+```
+
+### Verification Tree
+
+Nodes within the network are REQUIRED to verify a dataset before retrieving it.
+A verification tree is a Merkle proof derived from the `slotRoot`.
+The entire dataset is not REQUIRED to construct the tree.
+
+The following are the inputs to verify a proof:
+
+```nim
+
+type
+ H = array[32, byte]
+ Natural = uint64
+
+type ProofInputs*[H] = object
+ entropy*: H                    # Randomness value
+ datasetRoot*: H                # Dataset root hash
+ slotIndex*: Natural            # Slot identifier
+ slotRoot*: H                   # Root hash of slot
+ nCellsPerSlot*: Natural        # Cell count per slot
+ nSlotsPerDataSet*: Natural     # Total slot count
+ slotProof*: seq[H]             # Inclusion proof for slot in dataset
+ samples*: seq[Sample[H]]       # Cell inclusion proofs
+
+```
+
+To verify, a node MUST recompute the root hash,
+based on `slotProof` and the hash of the `slotIndex`,
+to confirm that the `slotIndex` is a member of the dataset represented by `datasetRoot`.
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
+
+## References
+
+- [CODEX-ERASURE-CODING](./erasure-coding.md)
+- [CODEX-MARKETPLACE](./marketplace.md)
+- [Poseidon2](https://eprint.iacr.org/2023/323)

waku/standards/core/66/metadata.md

@@ -3,9 +3,10 @@ slug: 66
 title: 66/WAKU2-METADATA
 name: Waku Metadata Protocol
 status: draft
-editor: Alvaro Revuelta <alrevuelta@status.im>
+editor: Franck Royer <franck@status.im>
 contributors:
- - Filip Dimitrijevic <filip@status.im>
+  - Filip Dimitrijevic <filip@status.im>
+  - Alvaro Revuelta <alrevuelta@status.im>
 ---
 
 ## Abstract
@@ -15,16 +16,19 @@ that can be associated with a [10/WAKU2](/waku/standards/core/10/waku2.md) node.
 
 ## Metadata Protocol
 
-Waku specifies a req/resp protocol that provides information about the node's medatadata.
-Such metadata is meant to be used by the node to decide if a peer is worth connecting
-or not.
+The keywords “MUST”, // List style “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”,
+“NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
+
+Waku specifies a req/resp protocol that provides information about the node's capabilities.
+Such metadata MAY be used by other peers for subsequent actions such as light protocol requests or disconnection.
+
 The node that makes the request,
 includes its metadata so that the receiver is aware of it,
-without requiring an extra interaction.
+without requiring another round trip.
 The parameters are the following:
 
 * `clusterId`: Unique identifier of the cluster that the node is running in.
-* `shards`: Shard indexes that the node is subscribed to.
+* `shards`: Shard indexes that the node is subscribed to via [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md).
 
 ***Protocol Identifier***
 
@@ -48,6 +52,51 @@ message WakuMetadataResponse {
 }
 ```
 
+## Implementation Suggestions
+
+### Triggering Metadata Request
+
+A node SHOULD proceed with metadata request upon first connection to a remote node.
+A node SHOULD use the remote node's libp2p peer id as identifier for this heuristic.
+
+A node MAY proceed with metadata request upon reconnection to a remote peer.
+
+A node SHOULD store the remote peer's metadata information for future reference.
+A node MAY implement a TTL regarding a remote peer's metadata, and refresh it upon expiry by initiating another metadata request.
+It is RECOMMENDED to set the TTL to 6 hours.
+
+A node MAY trigger a metadata request after receiving an error response from a remote note
+stating they do not support a specific cluster or shard.
+For example, when using a request-response service such as [`19/WAKU2-LIGHTPUSH`](/waku/standards/core/19/lightpush.md).
+
+### Providing Cluster Id
+
+A node MUST include their cluster id into their metadata payload.
+It is RECOMMENDED for a node to operate on a single cluster id.
+
+### Providing Shard Information
+
+* Nodes that mount [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md) MAY include the shards they are subscribed to in their metadata payload.
+* Shard-relevant services are message related services,
+  such as [`13/WAKU2-STORE`](/waku/standards/core/13/store.md), [12/WAKU2-FILTER](/waku/standards/core/12/filter.md)
+  and [`19/WAKU2-LIGHTPUSH`](/waku/standards/core/19/lightpush.md)
+  but not [`34/WAKU2-PEER-EXCHANGE`](/waku/standards/core/34/peer-exchange.md)
+* Nodes that mount [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md) and a shard-relevant service SHOULD include the shards they are subscribed to in their metadata payload.
+* Nodes that do not mount [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md) SHOULD NOT include any shard information
+
+### Using Cluster Id
+
+When reading the cluster id of a remote peer, the local node MAY disconnect if their cluster id is different from the remote peer.
+
+### Using Shard Information
+
+It is NOT RECOMMENDED to disconnect from a peer based on the fact that their shard information is different from the local node.
+
+Ahead of doing a shard-relevant request,
+a node MAY use the previously received metadata shard information to select a peer that support the targeted shard.
+
+For non-shard-relevant requests, a node SHOULD NOT discriminate a peer based on medata shard information.
+
 ## Copyright
 
 Copyright and related rights waived via