Compare commits

...

7 Commits

Author SHA1 Message Date
Cofson
85b3200ed9 Update purchase spec metadata: rename to PURCHASING and add contributors 2025-10-25 11:35:08 +02:00
Cofson
f32711607d 2nd refactor: Separate protocol requirements from implementation details 2025-10-13 21:55:12 +02:00
Cofson
1f73c9190d Refactor: Separate protocol requirements from implementation details 2025-10-13 21:50:14 +02:00
Cofson
652adbc85f Improved Abstract section 2025-10-10 09:09:43 +02:00
Cofson
eec236a39f Added Filip in contributors 2025-10-08 19:54:52 +02:00
Cofson
e49e587dd5 Created codex/raw/codex-purchase.md file 2025-10-08 19:10:03 +02:00
AkshayaMani
36be428cdd RFC Refactor: Sphinx Packet Format (#173) 2025-10-05 20:21:16 -04:00
2 changed files with 918 additions and 28 deletions

415
codex/raw/codex-purchase.md Normal file
View File

@@ -0,0 +1,415 @@
---
slug: codex-purchase
title: CODEX-PURCHASING
name: Codex Purchasing Module
status: raw
category: Standards Track
tags: codex, storage, marketplace, state-machine
editor: Codex Team and Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Uhlíř <adam.u@status.im>
- Arnaud DeVille <arnaud@status.im>
- Mark Spanbroek <mark@spanbroek.net>
- Eric Mastro <ericmastro@status.im>
- Filip Dimitrijevic <filip@status.im>
---
## Abstract
This specification defines the client role in the [Codex](https://github.com/codex-storage/nim-codex) marketplace protocol. A client is a Codex node that purchases storage by creating storage requests, submitting them to the marketplace smart contract, and managing the request lifecycle until completion or failure.
The specification outlines the protocol-level requirements for clients, including storage request creation, marketplace interactions, fund management, and request state monitoring. Implementation approaches, such as state machines and recovery mechanisms, are provided as suggestions.
## Background
In the Codex marketplace protocol, clients purchase storage by creating and submitting storage requests to the marketplace smart contract. Each storage request specifies the data to be stored (identified by a CID), storage parameters (duration, collateral requirements, proof parameters), and the number of storage slots needed.
The marketplace smart contract manages the storage request lifecycle, including request fulfillment, slot management, and fund distribution. Clients must interact with the marketplace to submit requests, monitor request state, and withdraw funds when requests complete or fail.
This specification defines what the protocol requires from clients. The marketplace smart contract governs these requirements—how individual Codex nodes implement them internally (state machines, recovery mechanisms, etc.) is an implementation detail covered in the Implementation Suggestions section.
## Client Protocol Requirements
This section defines the normative requirements for client implementations to participate in the Codex marketplace.
### Storage Request Creation
Clients MUST create storage requests with the following parameters:
- **client**: The client's address (identifying the requester)
- **ask**: Storage parameters including proof probability, pricing, collateral, slots, slot size, duration, and maximum slot loss
- **content**: Content identification using CID and merkle root
- **expiry**: Expiration timestamp for the request (uint64)
- **nonce**: Unique nonce value to ensure request uniqueness
### Marketplace Interactions
Clients MUST implement the following smart contract interactions:
- **requestStorage(request)**: Submit a new storage request to the marketplace along with required payment. Returns a unique `requestId` for tracking the request.
- **getRequest(requestId)**: Retrieve the full `StorageRequest` data from the marketplace. Used for recovery and state verification.
- **requestState(requestId)**: Query the current state of a storage request. Used for recovery and monitoring request progress.
- **withdrawFunds(requestId)**: Withdraw funds locked by the marketplace when a request completes, is cancelled, or fails. Clients MUST call this function to release locked funds.
### Event Subscriptions
Clients MUST subscribe to the following marketplace events:
- **RequestFulfilled(requestId)**: Emitted when a storage request has enough filled slots to start. Clients monitor this event to determine when their request becomes active.
- **RequestFailed(requestId)**: Emitted when a storage request fails due to proof failures or other reasons. Clients observe this event to detect failed requests and initiate fund withdrawal.
### Request Lifecycle Management
Clients MUST monitor storage requests through their lifecycle:
- After submitting a request with `requestStorage`, clients wait for the `RequestFulfilled` event or expiry timeout
- Once fulfilled, clients monitor for the request duration to complete successfully or for a `RequestFailed` event
- When a request completes (finished), is cancelled (expired before fulfillment), or fails, clients MUST call `withdrawFunds` to release locked funds
### Fund Management
Clients MUST:
- Provide sufficient payment when calling `requestStorage` to cover the storage request cost (based on slots, duration, and pricing parameters)
- Call `withdrawFunds` when a request completes, is cancelled, or fails to release any locked funds
## Implementation Suggestions
This section describes implementation approaches used in the nim-codex reference implementation. These are suggestions and not normative requirements.
The nim-codex implementation uses a state machine pattern to manage purchase lifecycles, providing deterministic state transitions, explicit terminal states, and recovery support. The state machine definitions (state identifiers, transitions, state descriptions, requirements, data models, and interfaces) are documented in the subsections below.
> **Note**: The Purchase module terminology and state machine design are specific to the nim-codex implementation. The protocol only requires that clients interact with the marketplace smart contract as specified in the Client Protocol Requirements section.
### State Identifiers
- PurchasePending: `pending`
- PurchaseSubmitted: `submitted`
- PurchaseStarted: `started`
- PurchaseFinished: `finished`
- PurchaseErrored: `errored`
- PurchaseCancelled: `cancelled`
- PurchaseFailed: `failed`
- PurchaseUnknown: `unknown`
### General Rules for All States
- If a `CancelledError` is raised, the state machine logs the cancellation message and takes no further action.
- If a `CatchableError` is raised, the state machine moves to `errored` with the error message.
### State Transitions
```text
|
v
------------------------- unknown
| / /
v v /
pending ----> submitted ----> started ---------> finished <----/
\ \ /
\ ------------> failed <----/
\ /
--> cancelled <-----------------------
```
**Note:**
Any state can transition to errored upon a `CatchableError`.
`failed` is an intermediate state before `errored`.
`finished`, `cancelled`, and `errored` are terminal states.
### State Descriptions
**Pending State (`pending`)**
A storage request is being created by making a call `on-chain`. If the storage request creation fails, the state machine moves to the `errored` state with the corresponding error.
**Submitted State (`submitted`)**
The storage request has been created and the purchase waits for the request to start. When it starts, an `on-chain` event `RequestFulfilled` is emitted, triggering the subscription callback, and the state machine moves to the `started` state. If the expiry is reached before the callback is called, the state machine moves to the `cancelled` state.
**Started State (`started`)**
The purchase is active and waits until the end of the request, defined by the storage request parameters, before moving to the `finished` state. A subscription is made to the marketplace to be notified about request failure. If a request failure is notified, the state machine moves to `failed`.
Marketplace subscription signature:
```nim
method subscribeRequestFailed*(market: Market, requestId: RequestId, callback: OnRequestFailed): Future[Subscription] {.base, async.}
```
**Finished State (`finished`)**
The purchase is considered successful and cleanup routines are called. The purchase module calls `marketplace.withdrawFunds` to release the funds locked by the marketplace:
```nim
method withdrawFunds*(market: Market, requestId: RequestId) {.base, async: (raises: [CancelledError, MarketError]).}
```
After that, the purchase is done; no more states are called and the state machine stops successfully.
**Failed State (`failed`)**
If the marketplace emits a `RequestFailed` event, the state machine moves to the `failed` state and the purchase module calls `marketplace.withdrawFunds` (same signature as above) to release the funds locked by the marketplace. After that, the state machine moves to `errored`.
**Cancelled State (`cancelled`)**
The purchase is cancelled and the purchase module calls `marketplace.withdrawFunds` to release the funds locked by the marketplace (same signature as above). After that, the purchase is terminated; no more states are called and the state machine stops with the reason of failure as error.
**Errored State (`errored`)**
The purchase is terminated; no more states are called and the state machine stops with the reason of failure as error.
**Unknown State (`unknown`)**
The purchase is in recovery mode, meaning that the state has to be determined. The purchase module calls the marketplace to get the request data (`getRequest`) and the request state (`requestState`):
```nim
method getRequest*(market: Market, id: RequestId): Future[?StorageRequest] {.base, async: (raises: [CancelledError]).}
method requestState*(market: Market, requestId: RequestId): Future[?RequestState] {.base, async.}
```
Based on this information, it moves to the corresponding next state.
> **Note**: Functional and non-functional requirements for the client role are summarized in the [Codex Marketplace Specification](https://github.com/codex-storage/codex-spec/blob/master/specs/marketplace.md). The requirements listed below are specific to the nim-codex Purchase module implementation.
### Functional Requirements
#### Purchase Definition
- Every purchase MUST represent exactly one `StorageRequest`
- The purchase MUST have a unique, deterministic identifier `PurchaseId` derived from `requestId`
- It MUST be possible to restore any purchase from its `requestId` after a restart
- A purchase is considered expired when the expiry timestamp in its `StorageRequest` is reached before the request start, i.e, an event `RequestFulfilled` is emitted by the `marketplace`
#### State Machine Progression
- New purchases MUST start in the `pending` state (submission flow)
- Recovered purchases MUST start in the `unknown` state (recovery flow)
- The state machine MUST progress step-by-step until a deterministic terminal state is reached
- The choice of terminal state MUST be based on the `RequestState` returned by the `marketplace`
#### Failure Handling
- On marketplace failure events, the purchase MUST immediately transition to `errored` without retries
- If a `CancelledError` is raised, the state machine MUST log the cancellation and stop further processing
- If a `CatchableError` is raised, the state machine MUST transition to `errored` and record the error
### Non-Functional Requirements
#### Execution Model
A purchase MUST be handled by a single thread; only one worker SHOULD process a given purchase instance at a time.
#### Reliability
`load` supports recovery after process restarts.
#### Performance
State transitions should be non-blocking; all I/O is async.
#### Logging
All state transitions and errors should be clearly logged for traceability.
#### Safety
- Avoid side effects during `new` other than initialising internal fields; `on-chain` interactions are delegated to states using `marketplace` dependency.
- Retry policy for external calls.
#### Testing
- Unit tests check that each state handles success and error properly.
- Integration tests check that a full purchase flows correctly through states.
### Data Models (Wire Format)
#### Purchase
```nim
Purchase* = ref object of Machine
future*: Future[void]
market*: Market
clock*: Clock
requestId*: RequestId
request*: ?StorageRequest
```
Extends the `Machine` type from the asyncstatemachine framework. Contains a `future` for async state machine execution, references to the `market` and `clock` dependencies, the `requestId` identifier, and an optional `request` field for the `StorageRequest`.
#### StorageRequest
```nim
StorageRequest* = object
client* {.serialize.}: Address
ask* {.serialize.}: StorageAsk
content* {.serialize.}: StorageContent
expiry* {.serialize.}: uint64
nonce*: Nonce
```
Contains the `client` address, storage parameters in `ask`, content identification in `content`, an `expiry` timestamp (uint64), and a unique `nonce`. Fields are marked with `{.serialize.}` for serialization support.
#### StorageAsk
```nim
StorageAsk* = object
proofProbability* {.serialize.}: UInt256
pricePerBytePerSecond* {.serialize.}: UInt256
collateralPerByte* {.serialize.}: UInt256
slots* {.serialize.}: uint64
slotSize* {.serialize.}: uint64
duration* {.serialize.}: uint64
maxSlotLoss* {.serialize.}: uint64
```
Defines storage agreement parameters: `proofProbability` and pricing in `pricePerBytePerSecond`, `collateralPerByte` for collateral requirements, `slots` and `slotSize` for storage distribution, `duration` for agreement length, and `maxSlotLoss` for fault tolerance. Numeric fields use `UInt256` for large values or `uint64` for counts and durations.
#### StorageContent
```nim
StorageContent* = object
cid* {.serialize.}: Cid
merkleRoot*: array[32, byte]
```
Identifies content to be stored using a `cid` (Content Identifier) and a `merkleRoot` (32-byte array) for verification purposes.
#### RequestId
```nim
RequestId* = distinct array[32, byte]
```
A distinct 32-byte array type used as a unique identifier for storage requests. Provides type safety through Nim's distinct types.
#### Nonce
```nim
Nonce* = distinct array[32, byte]
```
A distinct 32-byte array type used as a unique nonce value in `StorageRequest` to ensure request uniqueness.
### Interfaces
| Interface (Nim) | Description | Input | Output |
| --------------- | ----------- | ----- | ------ |
| `func new(_: type Purchase, requestId: RequestId, market: Market, clock: Clock): Purchase` | Construct a purchase from a storage request identifier. Used to recover a purchase. | `requestId: RequestId`, `market: Market`, `clock: Clock` | `Purchase` |
| `func new(_: type Purchase, request: StorageRequest, market: Market, clock: Clock): Purchase` | Create a purchase from a full `StorageRequest`. | `request: StorageRequest`, `market: Market`, `clock: Clock` | `Purchase` |
| `proc start*(purchase: Purchase)` | Start the state machine in pending mode (new on-chain submission flow). | `purchase: Purchase` | `void` |
| `proc load*(purchase: Purchase)` | Start the state machine in unknown mode (restore/recover after restart). | `purchase: Purchase` | `void` |
| `proc wait*(purchase: Purchase) {.async.}` | Await terminal state: completes on success or raises on failure. | `purchase: Purchase` | `Future[void]` |
| `func id*(purchase: Purchase): PurchaseId` | Stable identifier derived from `requestId`. | `purchase: Purchase` | `PurchaseId` |
| `func finished*(purchase: Purchase): bool` | Check whether the purchase completed successfully. | `purchase: Purchase` | `bool` |
| `func error*(purchase: Purchase): ?(ref CatchableError)` | Get error if the purchase failed. | `purchase: Purchase` | `Option[ref CatchableError]` |
| `func state*(purchase: Purchase): ?string` | Query current state name via the state machine. | `purchase: Purchase` | `Option[string]` |
| `proc hash*(x: PurchaseId): Hash` | Compute hash for `PurchaseId`. | `x: PurchaseId` | `Hash` |
| `proc ==*(x, y: PurchaseId): bool` | Equality comparison for `PurchaseId`. | `x: PurchaseId`, `y: PurchaseId` | `bool` |
| `proc toHex*(x: PurchaseId): string` | Hex string representation of `PurchaseId`. | `x: PurchaseId` | `string` |
| `method run*(state: PurchasePending, machine: Machine): Future[?State] {.async: (raises: []).}` | Submit request to market. | `state: PurchasePending`, `machine: Machine` | `Future[Option[State]]` |
| `method run*(state: PurchaseSubmitted, machine: Machine): Future[?State] {.async: (raises: []).}` | Await purchase start. | `state: PurchaseSubmitted`, `machine: Machine` | `Future[Option[State]]` |
| `method run*(state: PurchaseStarted, machine: Machine): Future[?State] {.async: (raises: []).}` | Run the purchase. | `state: PurchaseStarted`, `machine: Machine` | `Future[Option[State]]` |
| `method run*(state: PurchaseFinished, machine: Machine): Future[?State] {.async: (raises: []).}` | Purchase completed. | `state: PurchaseFinished`, `machine: Machine` | `Future[Option[State]]` |
| `method run*(state: PurchaseErrored, machine: Machine): Future[?State] {.async: (raises: []).}` | Purchase failed. | `state: PurchaseErrored`, `machine: Machine` | `Future[Option[State]]` |
| `method run*(state: PurchaseCancelled, machine: Machine): Future[?State] {.async: (raises: []).}` | Purchase cancelled or timed out. | `state: PurchaseCancelled`, `machine: Machine` | `Future[Option[State]]` |
| `method run*(state: PurchaseUnknown, machine: Machine): Future[?State] {.async: (raises: []).}` | Recover a purchase. | `state: PurchaseUnknown`, `machine: Machine` | `Future[Option[State]]` |
### Dependencies
The Purchase module requires the following dependencies:
- **marketplace** - Used for `requestStorage`, `withdrawFunds`, and subscriptions for request events
- **clock** - Provides timing utilities for expiry and scheduling logic
- **nim-chronos** - Async runtime for futures, awaiting I/O, and cancellation handling
- **asyncstatemachine** - Base state machine framework for implementing the purchase lifecycle
- **hashes** - Standard Nim hashing for `PurchaseId`
- **questionable** - Used for optional fields like `request`
### State Machine Implementation
Based on the specification, implementations should use the `asyncstatemachine` framework as the base for the Purchase state machine. The `Purchase` object extends `Machine` from this framework.
### Recovery Implementation
The `unknown` state implements recovery by:
1. Calling `marketplace.getRequest` to retrieve the `StorageRequest` data
2. Calling `marketplace.requestState` to determine the current `RequestState`
3. Transitioning to the appropriate state based on the marketplace response
### Fund Withdrawal
Terminal states (`finished`, `failed`, `cancelled`) all call `marketplace.withdrawFunds` to release locked funds. Implementations must ensure this call is made before the state machine stops.
## Security/Privacy Considerations
### Safety Requirements
The specification defines the following safety requirements:
- Implementations must avoid side effects during `new` other than initialising internal fields
- `on-chain` interactions must be delegated to states using `marketplace` dependency
- A retry policy for external calls should be implemented
### State Machine Safety
The deterministic state machine design ensures:
- Every purchase reaches exactly one terminal state (`finished`, `cancelled`, or `errored`)
- The `failed` state is an intermediate state that always transitions to `errored`
- State transitions follow the defined state diagram without cycles in terminal states
### Fund Management Safety
All terminal paths (`finished`, `failed`, `cancelled`) call `marketplace.withdrawFunds` as specified in the state descriptions. This ensures locked funds are released when purchases complete or fail.
### Recovery Safety
The `unknown` state enables recovery after restarts by querying the marketplace for current state. The specification requires that it must be possible to restore any purchase from its `requestId` after a restart.
## Rationale
This specification is based on the Purchase module component specification from the Codex project.
### State Machine Pattern
The specification uses a state machine pattern for managing purchase lifecycles. This design provides:
- **Deterministic state transitions**: As documented in the state diagram, purchases follow well-defined paths
- **Explicit terminal states**: Three terminal states (`finished`, `cancelled`, `errored`) clearly indicate final outcomes
- **Recovery support**: The `unknown` state enables restoration after process restarts as specified in the functional requirements
### Marketplace Abstraction
The marketplace abstraction allows for different backend implementations (on-chain or custom storage systems). The specification notes that "the abstraction could be replaceable: it could point to a different backend or custom storage system in future implementations."
This design separates purchase lifecycle management from marketplace implementation details, with all on-chain interactions delegated to states using the `marketplace` dependency as specified in the safety requirements.
### Async State Machine Framework
The specification requires the `asyncstatemachine` dependency for the base state machine framework and `nim-chronos` for async runtime. The `Purchase` object extends `Machine` from this framework, and state transitions are non-blocking with async I/O as specified in the performance requirements.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
### normative
- **Codex Marketplace Specification**: [Marketplace Spec](https://github.com/codex-storage/codex-spec/blob/master/specs/marketplace.md) - Defines the client role and protocol requirements
- **Codex Purchase Implementation**: [GitHub - codex-storage/nim-codex](https://github.com/codex-storage/nim-codex)
- **Codex Documentation**: [Codex Docs - Component Specification - Purchase](https://github.com/codex-storage/codex-docs-obsidian/blob/main/10%20Notes/Specs/Component%20Specification%20-%20Purchase.md)
### informative
- **Nim Chronos**: [GitHub - status-im/nim-chronos](https://github.com/status-im/nim-chronos) - Async/await framework for Nim
- **RFC 2119**: Key words for use in RFCs to Indicate Requirement Levels
- **Codex Marketplace**: Smart contract or backend system managing storage agreements

View File

@@ -120,6 +120,18 @@ contents as packets are forwarded hop-by-hop.
Sphinx packets are fixed-size and indistinguishable from one another, providing
unlinkability and metadata protection.
- **Initialization Vector (IV)**
A fixed-length input used to initialize block ciphers to add randomness to the
encryption process. It ensures that encrypting the same plaintext with the same
key produces different ciphertexts. The IV is not secret but must be unique for
each encryption.
- **Single-Use Reply Block (SURB)**
A pre-computed Sphinx header that encodes a return path back to the sender.
SURBs are generated by the sender and included in the Sphinx packet sent to the recipient.
It enables the recipient to send anonymous replies,
without learning the senders identity, the return path, or the forwarding delays.
## 3. Motivation and Background
libp2p enables modular peer-to-peer applications, but it lacks built-in support
@@ -194,7 +206,7 @@ describe the core components of a mix network.
The Mix Protocol relies on two core design elements to achieve sender
unlinkability and metadata
protection: a mixing strategy and a a cryptographically secure mix packet
protection: a mixing strategy and a cryptographically secure mix packet
format.
### 4.1 Mixing Strategy
@@ -297,7 +309,7 @@ layered encryption and per-hop delay provides resistance to traffic analysis and
enables message-level
unlinkability.
Unlike typical custom libp2p protocols, the Mix protocol is stateless&mdash;it
Unlike typical custom libp2p protocols, the Mix Protocol is stateless&mdash;it
does not establish
persistent streams, negotiate protocols, or maintain sessions. Each message is
self-contained
@@ -458,8 +470,8 @@ multistream handshake for protocol negotiation.
While libp2p supports multiplexing multiple streams over a single transport
connection using
stream muxers such as mplex and yamux, it does not natively support reusing a
stream over multiple
stream muxers such as mplex and yamux, it does not natively support reusing the
same stream over multiple
message transmissions. However, stream reuse may be desirable in the mixnet
setting to reduce overhead
and avoid hitting per protocol stream limits between peers.
@@ -512,9 +524,9 @@ adversarial routing bias.
While no existing mechanism provides unbiased sampling by default,
[Wakus ambient discovery](https://rfc.vac.dev/waku/standards/core/33/discv5/)
&mdash; an extension
&mdash;an extension
over [Discv5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)
&mdash; demonstrates
&mdash;demonstrates
an approximate solution. It combines topic-based capability advertisement with
periodic
peer sampling. A similar strategy could potentially be adapted for the Mix
@@ -618,7 +630,7 @@ _unobservability_ where
a passive adversary cannot determine whether a node is sending real messages or
not.
In the Mix Protocol, cover traffic is limited to _loop messages_ &mdash; dummy
In the Mix Protocol, cover traffic is limited to _loop messages_&mdash;dummy
Sphinx packets
that follow a valid mix path and return to the originating node. These messages
carry no application
@@ -627,7 +639,7 @@ routing behavior.
Cover traffic MAY be generated by either mix nodes or senders. The strategy for
generating
such traffic &mdash; such as timing and frequency &mdash; is pluggable and not
such traffic&mdash;such as timing and frequency&mdash;is pluggable and not
specified
in this document.
@@ -667,9 +679,9 @@ deter participation by compromised or transient peers.
The Mix Protocol does not mandate any form of payment, token exchange, or
accounting. More
sophisticated economic models &mdash; such as stake-based participation,
sophisticated economic models&mdash;such as stake-based participation,
credentialed relay networks,
or zero-knowledge proof-of-contribution systems &mdash; MAY be layered on top of
or zero-knowledge proof-of-contribution systems&mdash;MAY be layered on top of
the protocol or
enforced via external coordination.
@@ -779,6 +791,11 @@ destination origin protocol instance.
The node MUST NOT retain decrypted content after forwarding.
The routing behavior described in this section relies on the use of
Sphinx packets to preserve unlinkability and confidentiality across
hops. The next section specifies their structure, cryptographic
components, and construction.
## 8. Sphinx Packet Format
The Mix Protocol uses the Sphinx packet format to enable unlinkable, multi-hop
@@ -789,8 +806,10 @@ encapsulated in a Sphinx packet constructed by the initiating mix node. The
packet is encrypted in
layers such that each hop in the mix path can decrypt exactly one layer and
obtain the next-hop
routing information and delay value, without learning the complete path or the
routing information and forwarding delay, without learning the complete path or the
message origin.
Only the final hop learns the destination, which is encoded in the innermost
routing layer.
Sphinx packets are self-contained and indistinguishable on the wire, providing
strong metadata
@@ -820,12 +839,10 @@ subsections.
### 8.1 Packet Structure Overview
Each Sphinx packet consists of three fixed-length header fields &mdash; $α$,
$β$, and $γ$ &mdash;
followed by a fixed-length encrypted payload $δ$. Together, these components
enable per-hop message
processing with strong confidentiality and integrity guarantees in a stateless
and unlinkable manner.
Each Sphinx packet consists of three fixed-length header fields&mdash; $α$,
$β$, and $γ$ &mdash;followed by a fixed-length encrypted payload $δ$.
Together, these components enable per-hop message processing with strong
confidentiality and integrity guarantees in a stateless and unlinkable manner.
- **$α$ (Alpha)**: An ephemeral public value. Each mix node uses its private key
and $α$ to
@@ -833,8 +850,9 @@ derive a shared session key for that hop. This session key is used to decrypt
and process
one layer of the packet.
- **$β$ (Beta)**: The nested encrypted routing information. It encodes the next
hop address, the forwarding delay,
integrity check $γ$ for the next hop, and the $β$ for subsequent hops.
hop address, the forwarding delay, integrity check $γ$ for the next hop, and
the $β$ for subsequent hops. At the final hop, $β$ encodes the destination
address and fixed-length zero padding to preserve uniform size.
- **$γ$ (Gamma)**: A message authentication code computed over $β$ using the
session key derived
from $α$. It ensures header integrity at each hop.
@@ -843,12 +861,12 @@ fixed maximum length and
encrypted in layers corresponding to each hop in the mix path.
At each hop, the mix node derives the session key from $α$, verifies the header
integrity
using $γ$, decrypts one layer of $β$ to extract the next hop and delay, and
decrypts one layer
of $δ$. It then constructs a new packet with updated values of $α$, $β$, $γ$,
and $δ$, and
forwards it to the next hop.
integrity using $γ$, decrypts one layer of $β$ to extract the next hop and
delay, and decrypts one layer of $δ$. It then constructs a new packet with
updated values of $α$, $β$, $γ$, and $δ$, and forwards it to the next hop. At
the final hop, the mix node decrypts the innermost layer of $β$ and $δ$, which
yields the destination address and the original application message
respectively.
All Sphinx packets are fixed in size and indistinguishable on the wire. This
uniform format,
@@ -866,20 +884,29 @@ This section defines the cryptographic primitives used in Sphinx packet
construction and processing.
- **Security Parameter**: All cryptographic operations target a minimum of
$\kappa = 128$ bits of
$κ = 128$ bits of
security, balancing performance with resistance to modern attacks.
- **Elliptic Curve Group $\mathbb{G}$**:
- **Curve**: Curve25519
- **Notation**: Let $g$ denote the canonical base point (generator) of $\mathbb{G}$.
- **Purpose**: Used for deriving DiffieHellman-style shared key at each hop
using $α$.
- **Representation**: Small 32-byte group elements, efficient for both
encryption and key exchange.
- **Scalar Field**: The curve is defined over the finite field
$\mathbb{Z}_q$, where $q = 2^{252} + 27742317777372353535851937790883648493$.
Ephemeral exponents used in Sphinx packet construction are selected uniformly
at random from $\mathbb{Z}_q^*$, the multiplicative subgroup of $\mathbb{Z}_q$.
- **Hash Function**:
- **Construction**: SHA-256
- **Notation**: The hash function is denoted by $H(\cdot)$ in subsequent sections.
- **Key Derivation Function (KDF)**:
- **Purpose**: To derive encryption keys, IVs, and MAC key from the shared
session key at each hop.
- **Construction**: SHA-256 hash with output truncated to 128 bits.
- **Construction**: SHA-256 hash with output truncated to $128$ bits.
- **Key Derivation**: The KDF key separation labels (_e.g.,_ `"aes_key"`,
`"mac_key"`)
are fixed strings and MUST be agreed upon across implementations.
@@ -889,9 +916,457 @@ are fixed strings and MUST be agreed upon across implementations.
- **Keys and IVs**: Each derived from the session key for the hop using the KDF.
- **Message Authentication Code (MAC)**:
- **Construction**: HMAC-SHA-256 with output truncated to 128 bits.
- **Construction**: HMAC-SHA-256 with output truncated to $128$ bits.
- **Purpose**: To compute $γ$ for each hop.
- **Key**: Derived using KDF from the session key for the hop.
These primitives are used consistently throughout packet construction and
decryption, as described in the following sections.
### 8.3 Packet Component Sizes
This section defines the size of each component in a Sphinx packet, deriving them
from the security parameter and protocol parameters introduced earlier. All Sphinx
packets MUST be fixed in length to ensure uniformity and indistinguishability on
the wire. The serialized packet is structured as follows:
```text
+--------+----------+--------+----------+
| α | β | γ | δ |
| 32 B | variable | 16 B | variable |
+--------+----------+--------+----------+
```
#### 8.3.1 Header Field Sizes
The header consists of the fields $α$, $β$, and $γ$, totaling a fixed size per
maximum path length:
- **$α$ (Alpha)**: 32 bytes
The size of $α$ is determined by the elliptic curve group representation used
(Curve25519), which encodes group elements as 32-byte values.
- **$β$ (Beta)**: $((t + 1)r + 1)κ$ bytes
The size of $β$ depends on:
- **Maximum path length ($r$)**: The recommended value of $r=5$ balances
bandwidth versus anonymity tradeoffs.
- **Combined address and delay width ($tκ$)**: The recommended $t=6$
accommodates standard libp2p relay multiaddress representations plus a
2-byte delay field. While the actual multiaddress and delay fields may be
shorter, they are padded to $tκ$ bytes to maintain fixed field size. The
structure and rationale for the $tκ$ block and its encoding are specified in
[Section 8.4](#84-address-and-delay-encoding).
Note: This expands on the original
[Sphinx packet format]((https://cypherpunks.ca/~iang/pubs/Sphinx_Oakland09.pdf)),
which embeds a fixed $κ$-byte mix node identifier per hop in $β$.
The Mix Protocol generalizes this to $tκ$ bytes to accommodate libp2p
multiaddresses and forwarding delays while preserving the cryptographic
properties of the original design.
- **Per-hop $γ$ size ($κ$)** (defined below): Accounts for the integrity tag
included with each hops routing information.
Using the recommended value of $r=5$ and $t=6$, the resulting $β$ size is
$576$ bytes. At the final hop, $β$ encodes the destination address in the
first $tκ-2$ bytes and the remaining bytes are zero-padded.
- **$γ$ (Gamma)**: $16$ bytes
The size of $γ$ equals the security parameter $κ$, providing a $κ$-bit integrity
tag at each hop.
Thus, the total header length is:
$`
\begin{aligned}
|Header| &= α + β + γ \\
&= 32 + ((t + 1)r + 1)κ + 16
\end{aligned}
`$
Notation: $|x|$ denotes the size (in bytes) of field $x$.
Using the recommended value of $r = 5$ and $t = 6$, the header size is:
$`
\begin{aligned}
|Header| &= 32 + 576 + 16 \\
&= 624 \ bytes
\end{aligned}
`$
#### 8.3.2 Payload Size
This subsection defines the size of the encrypted payload $δ$ in a Sphinx packet.
$δ$ contains the application message, padded to a fixed maximum length to ensure
all packets are indistinguishable on the wire. The size of $δ$ is calculated as:
$`
\begin{aligned}
|δ| &= TotalPacketSize - HeaderSize
\end{aligned}
`$
The recommended total packet size is $4608$ bytes, chosen to:
- Accommodate larger libp2p application messages, such as those commonly
observed in Status chat using Waku (typically ~4KB payloads),
- Allow inclusion of additional data such as SURBs without requiring fragmentation,
- Maintain reasonable per-hop processing and bandwidth overhead.
This recommended total packet size of \$4608\$ bytes yields:
$`
\begin{aligned}
Payload &= 4608 - 624 \\
&= 3984\ bytes
\end{aligned}
`$
Implementations MUST account for payload extensions, such as SURBs,
when determining the maximum message size that can be encapsulated in a
single Sphinx packet. Details on SURBs are defined in
[Section X.X].
The following subsection defines the padding and fragmentation requirements for
ensuring this fixed-size constraint.
#### 8.3.3 Padding and Fragmentation
Implementations MUST ensure that all messages shorter than the maximum payload size
are padded before Sphinx encapsulation to ensure that all packets are
indistinguishable on the wire. Messages larger than the maximum payload size MUST
be fragmented by the origin protocol or top-level application before being passed
to the Mix Protocol. Reassembly is the responsibility of the consuming application,
not the Mix Protocol.
#### 8.3.4 Anonymity Set Considerations
The fixed maximum packet size is a configurable parameter. Protocols or
applications that choose to configure a different packet size (either larger or
smaller than the default) MUST be aware that using unique or uncommon packet sizes
can reduce their effective anonymity set to only other users of the same size.
Implementers SHOULD align with widely used defaults to maximize anonymity set size.
Similarly, parameters such as $r$ and $t$ are configurable. Changes to these
parameters affect header size and therefore impact payload size if the total packet
size remains fixed. However, if such changes alter the total packet size on the
wire, the same anonymity set considerations apply.
The following subsection defines how the next-hop or destination address and
forwarding delay are encoded within $β$ to enable correct routing and mixing
behavior.
### 8.4 Address and Delay Encoding
Each hops $β$ includes a fixed-size block containing the next-hop address and
the forwarding delay, except for the final hop, which encodes the destination
address and a delay-sized zero padding. This section defines the structure and
encoding of that block.
The combined address and delay block MUST be exactly $tκ$ bytes in length,
as defined in [Section 8.3.1](#831-header-field-sizes), regardless of the
actual address or delay values. The first $(tκ - 2)$ bytes MUST encode the
address, and the final $2$ bytes MUST encode the forwarding delay.
This fixed-length encoding ensures that packets remain indistinguishable on
the wire and prevents correlation attacks based on routing metadata structure.
Implementations MAY use any address and delay encoding format agreed upon
by all participating mix nodes, as long as the combined length is exactly
$tκ$ bytes. The encoding format MUST be interpreted consistently by all
nodes within a deployment.
For interoperability, a recommended default encoding format involves:
- Encoding the next-hop or destination address as a libp2p multi-address:
- To keep the address block compact while allowing relay connectivity, each mix
node is limited to one IPv4 circuit relay multiaddress. This ensures that most
nodes can act as mix nodes, including those behind NATs or firewalls.
- In libp2p terms, this combines transport addresses with multiple peer
identities to form an address that describes a relay circuit:
`
/ip4/<ipv4>/tcp/<port>/p2p/<relayPeerID>/p2p-circuit/p2p/<relayedPeerID>
`
Variants may include directly reachable peers and transports such as
`/quic-v1`, depending on the mix node's supported stack.
- IPv6 support is deferred, as it adds $16$ bytes just for the IP field.
- Future revisions may extend this format to support IPv6 or DNS-based
multiaddresses.
With these constraints, the recommended encoding layout is:
- IPv4 address (4 bytes)
- Protocol identifier _e.g.,_ TCP or QUIC (1 byte)
- Port number (2 bytes)
- Peer IDs (39 bytes, post-Base58 decoding)
- Encoding the forwarding delay as an unsigned 16-bit integer (2 bytes) in
milliseconds, using big endian network byte order.
If the encoded address or delay is shorter than its respective allocated
field, it MUST be padded with zeros. If it exceeds the allocated size, it
MUST be rejected or truncated according to the implementation policy.
Note: Future versions of the Mix Protocol may support address compression by
encoding only the peer identifier and relying on external peer discovery
mechanisms to retrieve full multiaddresses at runtime. This would allow for
more compact headers and greater address flexibility, but requires fast and
reliable lookup support across deployments. This design is out of scope for
the current version.
With the field sizes and encoding conventions established, the next section describes
how a mix node constructs a complete Sphinx packet when initiating the Mix Protocol.
### 8.5 Packet Construction
This section defines how a mix node constructs a Sphinx packet when initiating
the Mix Protocol on behalf of a local origin protocol instance.
The construction process wraps the message in a sequence of encryption
layers&mdash;one for each hop&mdash;such that only the corresponding mix node
can decrypt its layer and retrieve the routing instructions for that hop.
#### 8.5.1 Inputs
To initiate the Mix Protocol, the origin protocol instance submits a message
to the Mix Entry Layer on the same node. This layer forwards it to the local
Mix Protocol instance, which constructs a Sphinx packet
using the following REQUIRED inputs:
- **Application message**: The serialized message provided by the origin
protocol instance. The Mix Protocol instance applies any configured spam
protection mechanism and attaches one or two SURBs prior to encapsulating
the message in the Sphinx packet. The initiating node MUST ensure that
the resulting payload size does not exceed the maximum supported size
defined in [Section 8.3.2](#832-payload-size).
- **Origin protocol codec**: The libp2p protocol string corresponding to the
origin protocol instance. This is included in the payload so that
the exit node can route the message to the intended destination protocol
after decryption.
- **Mix Path length $L$**: The number of mix nodes to include in the path.
The mix path MUST consist of at least three hops, each representing a
distinct mix node.
- **Destination address $Δ$**: The routing address of the intended recipient
of the message. This address is encoded in $(tκ - 2)$ bytes as defined in
[Section 8.4](#84-address-and-delay-encoding) and revealed only at the last hop.
#### 8.5.2 Construction Steps
This subsection defines how the initiating mix node constructs a complete
Sphinx packet using the inputs defined in
[Section 8.5.1](#851-inputs). The construction MUST
follow the cryptographic structure defined in
[Section 8.1](#81-packet-structure-overview), use the primitives specified in
[Section 8.2](#82-cryptographic-primitives), and adhere to the component sizes
and encoding formats from [Section 8.3](#83-packet-component-sizes) and
[Section 8.4](#84-address-and-delay-encoding).
The construction MUST proceed as follows:
1. **Prepare Application Message**
- Apply any configured spam protection mechanism (_e.g.,_ PoW, VDF, RLN)
to the serialized message. Spam protection mechanisms are pluggable as defined
in [Section 6.3](#63-spam-protection).
- Attach one or more SURBs, if required. Their format and processing are
specified in [Section X.X].
- Append the origin protocol codec.
- Pad the result to the maximum application message length of $3968$ bytes
using a deterministic padding scheme. This value is derived from the fixed
payload size in [Section 8.3.2](#832-payload-size) ($3984$ bytes) minus the
security parameter $κ = 16$ bytes defined in
[Section 8.2](#82-cryptographic-primitives). The chosen scheme MUST yield a
fixed-size padded output and MUST be consistent across all mix nodes to
ensure correct interpretation during unpadding. For example, schemes that
explicitly encode the padding length and prepend zero-valued padding bytes
MAY be used.
- Let the resulting message be $m$.
2. **Select A Mix Path**
- First obtain an unbiased random sample of live, routable mix nodes using
some discovery mechanism. The choice of discovery mechanism is
deployment-specific as defined in [Section 6.1](#61-discovery). The
discovery mechanism MUST be unbiased and provide, at a minimum, the
multiaddress and X25519 public key of each mix node.
- From this sample, choose a random mix path of length $L \geq 3$. As defined
in [Section 2](#2-terminology), a mix path is a non-repeating sequence of
mix nodes.
- For each hop $i \in \{0 \ldots L-1\}$:
- Retrieve the multiaddress and corresponding X25519 public key $y_i$ of
the $i$-th mix node.
- Encode the multiaddress in $(tκ - 2)$ bytes as defined in
[Section 8.4](#84-address-and-delay-encoding). Let the resulting encoded
multiaddress be $\mathrm{addr\_i}$.
3. **Wrap Plaintext Payload In Sphinx Packet**
a. **Compute Ephemeral Secrets**
- Choose a random private exponent $x \in_R \mathbb{Z}_q^*$.
- Initialize:
$`
\begin{aligned}
α_0 &= g^x \\
s_0 &= y_0^x \\
b_0 &= H(α_0\ |\ s_0)
\end{aligned}
`$
- For each hop $i$ (from $1$ to $L-1$), compute:
$`
\begin{aligned}
α_i &= α_{i-1}^{b_{i-1}} \\
s_i &= y_{i}^{x\prod_{\text{j=0}}^{\text{i-1}} b_{j}} \\
b_i &= H(α_i\ |\ s_i)
\end{aligned}
`$
Note that the length of $α_i$ is $32$ bytes as defined in
[Section 8.3](#83-packet-component-sizes).
b. **Compute Per-Hop Filler Strings**
Filler strings are encrypted strings that are appended to the header during
encryption. They ensure that the header length remains constant across hops,
regardless of the position of a node in the mix path.
To compute the sequence of filler strings, perform the following steps:
- Initialize $Φ_0 = \epsilon$ (empty string).
- For each $i$ (from $1$ to $L-1$):
- Derive per-hop AES key and IV:
$`
\begin{array}{l}
Φ_{\mathrm{aes\_key}_{i-1}} =
\mathrm{KDF}(\text{"aes\_key"} \mid s_{i-1})\\
Φ_{\mathrm{iv}_{i-1}} =
\mathrm{KDF}(\text{"iv"} \mid s_{i-1})
\end{array}
`$
- Compute the filler string $Φ_i$ using $\text{AES-CTR}^\prime_i$,
which is AES-CTR encryption with the keystream starting from
index $((t+1)(r-i)+t+2)\kappa$ :
$`
\begin{array}{l}
Φ_i = \mathrm{AES\text{-}CTR}'_i\bigl(Φ_{\mathrm{aes\_key}_{i-1}},
Φ_{\mathrm{iv}_{i-1}}, Φ_{i-1} \mid 0_{(t+1)\kappa} \bigr),
\text{where notation $0_x$ defines the string of $0$ bits of length $x$.}
\end{array}
`$
Note that the length of $Φ_i$ is $(t+1)i\kappa$.
c. **Construct Routing Header**
The routing header as defined in
[Section 8.1](#81-packet-structure-overview) is the encrypted structure
that carries the forwarding instructions for each hop. It ensures that a
mix node can learn only its immediate next hop and forwarding delay without
inferring the full path.
Filler strings computed in the previous step are appended during encryption
to ensure that the header length remains constant across hops. This prevents
a node from distinguishing its position in the path based on header size.
To construct the routing header, perform the following steps for each hop
$i = L-1$ down to $0$, recursively:
- Derive per-hop AES key, MAC key, and IV:
$`
\begin{array}{l}
β_{\mathrm{aes\_key}_i} =
\mathrm{KDF}(\text{"aes\_key"} \mid s_i)\\
\mathrm{mac\_key}_i =
\mathrm{KDF}(\text{"mac\_key"} \mid s_{i})\\
β_{\mathrm{iv}_i} =
\mathrm{KDF}(\text{"iv"} \mid s_i)
\end{array}
`$
- Set the per hop two-byte encoded delay $\mathrm{delay}_i$ as defined in
[Section 8.4](#84-address-and-delay-encoding):
- If final hop (_i.e.,_ $i = L - 1$), encode two byte zero padding.
- For all other hop $i,\ i < L - 1$, sample a forwarding delay
using the delay strategy configured by the application and encode it in two bytes.
The delay strategy is pluggable as defined in [Section 6.2](#62-delay-strategy).
- Using the derived keys and encoded forwarding delay, compute the nested
encrypted routing information $β_i$:
- If $i = L-1$ (_i.e.,_ exit node):
$`
\begin{array}{l}
β_i = \mathrm{AES\text{-}CTR}\bigl(β_{\mathrm{aes\_key}_i},
β_{\mathrm{iv}_i}, Δ \mid \mathrm{delay}_i \mid 0_{((t+1)(r-L)+2)\kappa}
\bigr) \bigm| Φ_{L-1}
\end{array}
`$
- Otherwise (_i.e.,_ intermediary node):
$`
\begin{array}{l}
β_i = \mathrm{AES\text{-}CTR}\bigl(β_{\mathrm{aes\_key}_i},
β_{\mathrm{iv}_i}, \mathrm{addr}_{i+1} \mid $\mathrm{delay}_i$
\mid γ_{i+1} \mid β_{i+1 \, [0 \ldots (r(t+1) - t)\kappa - 1]} \bigr)
\end{array}
`$
Note that the length of $\beta_i$ is $(r(t+1)+1)\kappa$, $0 \leq i \leq L-1$
as defined in [Section 8.3](#83-packet-component-sizes).
- Compute the message authentication code $γ_i$:
$`
\begin{array}{l}
γ_i = \mathrm{HMAC\text{-}SHA\text{-}256}\bigl(\mathrm{mac\_key}_i,
β_i \bigr)
\end{array}
`$
Note that the length of $\gamma_i$ is $\kappa$ as defined in
[Section 8.3](#83-packet-component-sizes).
d. **Encrypt Payload**
The encrypted payload $δ$ contains the message $m$ defined in Step 1,
prepended with a $κ$-byte string of zeros. It is encrypted in layers such that
each hop in the mix path removes exactly one layer using the per-hop session
key. This ensures that only the final hop (_i.e.,_ the exit node) can fully
recover $m$, validate its integrity, and forward it to the destination.
To compute the encrypted payload, perform the following steps for each hop
$i = L-1$ down to $0$, recursively:
- Derive per-hop AES key and IV:
$`
\begin{array}{l}
δ_{\mathrm{aes\_key}_i} =
\mathrm{KDF}(\text{"δ\_aes\_key"} \mid s_i)\\
δ_{\mathrm{iv}_i} =
\mathrm{KDF}(\text{"δ\_iv"} \mid s_i)
\end{array}
`$
- Using the derived keys, compute the encrypted payload $δ_i$:
- If $i = L-1$ (_i.e.,_ exit node):
$`
\begin{array}{l}
δ_i = \mathrm{AES\text{-}CTR}\bigl(δ_{\mathrm{aes\_key}_i},
δ_{\mathrm{iv}_i}, 0_{\kappa} \mid m
\bigr)
\end{array}
`$
- Otherwise (_i.e.,_ intermediary node):
$`
\begin{array}{l}
δ_i = \mathrm{AES\text{-}CTR}\bigl(δ_{\mathrm{aes\_key}_i},
δ_{\mathrm{iv}_i}, δ_{i+1} \bigr)
\end{array}
`$