title: CODEX-BLOCK-EXCHANGE -name: Codex Block Exchange Protocol -status: raw -category: Standards Track -tags: codex, block-exchange, p2p, data-distribution -editor: Codex Team -contributors:
b1a5783
d03e699
b2f3564
63107d3
This specification contains a mix of:
slug: codex-marketplace -title: CODEX-MARKETPLACE -name: Codex Storage Marketplace -status: raw -category: Standards Track -tags: codex, storage, marketplace, smart-contract -editor: Codex Team and Dmitriy Ryajov dryajov@status.im -contributors:
d2df7e0
Codex Marketplace and its interactions are defined by a smart contract deployed on an EVM-compatible blockchain. This specification describes these interactions for the various roles within the network.
The document is intended for implementors of Codex nodes.
Early-stage Codex specifications collected before reaching draft status.
NOTE: This repo is WIP. We are currently restructuring the RFC process.
This repository contains specifications from the Waku, Nomos, -Codex, and -Status projects that are part of the IFT portfolio. -Vac is an -IFT service that will manage the RFC, -Request for Comments, -process within this repository.
This repository replaces the previous rfc.vac.dev resource. -Each project will maintain initial specifications in separate repositories, -which may be considered as a raw specification. -All Vac raw specifications and -discussions will live in the Vac subdirectory. -When projects have reached some level of maturity -for a specification living in their repository, -the process of updating the status to draft may begin in this repository. -Specifications will adhere to -1/COSS before obtaining draft status.
rfc.vac.dev
Implementations should follow specifications as described, -and all contributions will be discussed before the stable status is obtained. -The goal of this RFC process will to engage all interseted parities and -reach a rough consensus for techcinal specifications.
Please see 1/COSS for general guidelines and specification lifecycle.
Feel free to join the Vac discord.
Here's the project board used by core contributors and maintainers: Projects
The repository for each project raw specifications:
An IETF-style index of Vac-managed RFCs across Waku, Nomos, Codex, and Status. Use the filters below to jump straight to a specification.
JavaScript is required to load the RFC index table.
title: CONSENSUS-CLARO -name: Claro Consensus Protocol -status: deprecated -category: Standards Track -tags:
1ddddc7
3ab314d
f52c54f
01e2781
This document specifies Claro: a Byzantine, fault-tolerant, binary decision agreement algorithm that utilizes bounded memory for its execution. @@ -852,11 +856,11 @@ Move: a Language for Writing DAG Abstractions
Deprecated Nomos specifications kept for archival and reference purposes.
Early-stage Nomos specifications that have not yet progressed beyond raw status.
title: NOMOSDA-ENCODING -name: NomosDA Encoding Protocol -status: raw -category: -tags: data-availability -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors:
39d6f07
This document describes the encoding and verification processes of NomosDA, which is the data availability (DA) solution used by the Nomos blockchain. NomosDA provides an assurance that all data from Nomos blobs are accessible and verifiable by every network participant.
This document presents an implementation specification describing how:
title: NOMOS-DA-NETWORK -name: NomosDA Network -status: raw -category: -tags: network, data-availability, da-nodes, executors, sampling -editor: Daniel Sanchez Quiros danielsq@status.im -contributors:
51ef4cd
NomosDA is the scalability solution protocol for data availability within the Nomos network. This document delineates the protocol's structure at the network level, @@ -447,6 +449,7 @@ ensures the overlay node distribution is scalable for networks of any size.
title: P2P-HARDWARE-REQUIREMENTS -name: Nomos p2p Network Hardware Requirements Specification -status: raw -category: infrastructure -tags: [hardware, requirements, nodes, validators, services] -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors:
34bbd7a
This specification defines the hardware requirements for running various types of Nomos blockchain nodes. Hardware needs vary significantly based on the node's role, from lightweight verification nodes to high-performance Zone Executors. The requirements are designed to support diverse participation levels while ensuring network security and performance.
title: P2P-NAT-SOLUTION -name: Nomos P2P Network NAT Solution Specification -status: raw -category: networking -tags: [nat, traversal, autonat, upnp, pcp, nat-pmp] -editor: Antonio Antonino antonio@status.im -contributors:
cfb3b78
This specification defines a comprehensive NAT (Network Address Translation) traversal solution for the Nomos P2P network. The solution enables nodes to automatically determine their NAT status and establish both outbound and inbound connections regardless of network configuration. The strategy combines AutoNAT, dynamic port mapping protocols, and continuous verification to maximize public reachability while maintaining decentralized operation.
title: P2P-NETWORK-BOOTSTRAPPING -name: Nomos P2P Network Bootstrapping Specification -status: raw -category: networking -tags: [p2p, networking, bootstrapping, peer-discovery, libp2p] -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors:
aa8a3b0
Nomos network bootstrapping is the process by which a new node discovers peers and synchronizes with the existing decentralized network. It ensures that a node can:
title: NOMOS-P2P-NETWORK -name: Nomos P2P Network Specification -status: draft -category: networking -tags: [p2p, networking, libp2p, kademlia, gossipsub, quic] -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors:
a3a5b91
This specification defines the peer-to-peer (P2P) network layer for Nomos blockchain nodes. The network serves as the comprehensive communication infrastructure enabling transaction dissemination through mempool and block propagation. The specification leverages established libp2p protocols to ensure robust, scalable performance with low bandwidth requirements and minimal latency while maintaining accessibility for diverse hardware configurations and network environments.
title: NOMOS-SDP -name: Nomos Service Declaration Protocol Specification -status: raw -category: -tags: participation, validators, declarations -editor: Marcin Pawlowski marcin@status.im -contributors:
53dfb97
This document defines a mechanism enabling validators to declare their participation in specific protocols that require a known and agreed-upon list of participants. Some examples of this are Data Availability and the Blend Network. We create a single repository of identifiers which is used to establish secure communication between validators and provide services. Before being admitted to the repository, the validator proves that it locked at least a minimum stake.
nonce
Vac builds public good protocols for the decentralised web. +Vac acts as a custodian for the protocols that live in the RFC-Index repository. +With the goal of widespread adoption, +Vac will make sure the protocols adhere to a set of principles, +including but not limited to liberty, security, privacy, decentralisation and inclusivity.
To learn more, visit Vac Research
dd397ad
d5e0072
ed2c68f
3eaccf9
990d940
6495074
bab16a8
a9162f2
slug: 1 -title: 1/COSS -name: Consensus-Oriented Specification System -status: draft -category: Best Current Practice -editor: Daniel Kaiser danielkaiser@status.im -contributors:
This document describes a consensus-oriented specification system (COSS) for building interoperable technical specifications. COSS is based on a lightweight editorial process that @@ -489,24 +509,35 @@ Color coded specifications SHOULD use the following color scheme:
slug: 2 -title: 2/MVDS -name: Minimum Viable Data Synchronization -status: stable -editor: Sanaz Taheri sanaz@status.im -contributors:
a5b24ac
0253d53
70326d1
472a7fd
4362a7b
In this specification, we describe a minimum viable protocol for -data synchronization inspired by the Bramble Synchronization Protocol1. +data synchronization inspired by the Bramble Synchronization Protocol (BSP). This protocol is designed to ensure reliable messaging between peers across an unreliable peer-to-peer (P2P) network where they may be unreachable or unresponsive.
We present a reference implementation2 +
We present a reference implementation1 including a simulation to demonstrate its performance.
MESSAGE
Copyright and related rights waived via CC0.
akwizgran et al. BSP. Briar.
https://github.com/vacp2p/mvds
slug: 3 -title: 3/REMOTE-LOG -name: Remote log specification -status: draft -editor: Oskar Thorén oskarth@titanproxy.com -contributors:
3fd8b5a
dce61fe
A remote log is a replication of a local log. This means a node can read data that originally came from a node that is offline.
This specification is complemented by a proof of concept implementation1.
https://github.com/vacp2p/research/tree/master/remote_log
slug: 4 -title: 4/MVDS-META -name: MVDS Metadata Field -status: draft -editor: Sanaz Taheri sanaz@status.im -contributors:
3a396b5
2e80c3b
In this specification, we describe a method to construct message history that will aid the consistency guarantees of 2/MVDS. Additionally, @@ -907,13 +951,25 @@ As their delivery is not needed to be guaranteed.
2eaa794
a3ad14e
25/LIBP2P-DNS-DISCOVERY specifies a scheme to implement libp2p peer discovery via DNS for Waku v2. The generalised purpose is to retrieve an arbitrarily long, authenticated, @@ -1024,22 +1080,30 @@ and verify that the content matches the hash.
25/LIBP2P-DNS-DISCOVERY
libp2p
slug: 32 -title: 32/RLN-V1 -name: Rate Limit Nullifier -status: draft -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
eb25cd0
cbefa48
94db406
a23299f
539575b
The following specification covers the RLN construct as well as some auxiliary libraries useful for interacting with it. @@ -1695,14 +1759,24 @@ assert!(verified);
All Vac specifications that have not reached draft status will live in this repository. To learn more about raw specifications, take a look at 1/COSS.
f051117
This document specifies a scalable, decentralized, and Byzantine Fault Tolerant (BFT) consensus mechanism inspired by Hashgraph, designed for binary decision-making in P2P networks.
517b639
c655980
7e3a625
This document introduces a decentralized group messaging protocol using Ethereum adresses as identifiers. @@ -2630,15 +2717,25 @@ opens the door to some novel research.
title: ETH-MLS-OFFCHAIN -name: Secure channel setup using decentralized MLS and Ethereum accounts -status: raw -category: Standards Track -tags: -editor: Ugur Sen ugur@status.im -contributors: seemenkina ekaterina@status.im
e39d288
3b968cc
The following document specifies Ethereum authenticated scalable and decentralized secure group messaging application by @@ -3006,14 +3103,32 @@ This design choice is intentional, in order to preserve the efficiency advantage
13aaae3
e234e9d
b842725
f2e1b4c
22bb331
5b8ce46
The need for secure communications has become paramount. Traditional centralized messaging protocols are susceptible to various security threats, @@ -3904,14 +4019,32 @@ control mechanisms to restrict who can call the set and get functions.
36caaa6
db90adc
The content of this specification has been split between ETH-DEMLS and NOISE-X3DH-RATCHET @@ -5101,14 +5234,26 @@ the prospective key package has not been already used.
99be3b9
cd8c9f4
0db60c1
This document extends the libp2p gossipsub specification specifying gossipsub Tor Push, @@ -5277,15 +5422,24 @@ For the best protection, these contexts should run on separate physical machines
title: LOGOS-CAPABILITY-DISCOVERY -name: Logos Capability Discovery Protocol -status: raw -category: Standards Track -tags: -editor: Arunima Chaudhuri arunima@status.im -contributors: Ugur Sen ugur@status.im
aaf158a
Note: This specification is currently a WIP and undergoing a high rate of changes.
ad
dabc317
4f54254
7f1df32
e742cd5
9d11a22
36be428
5e3b478
38fce27
7f5276e
The Mix Protocol defines a decentralized anonymous message routing layer for libp2p networks. It enables sender anonymity by routing each message through a decentralized mix overlay network composed of participating libp2p nodes, known as mix nodes. @@ -7466,14 +7639,24 @@ However, these protections do not prevent adversaries from injecting large volum
DoS protection—such as admission control, rate-limiting, or resource-bound access—MUST be implemented outside the core protocol. Any such mechanism MUST preserve sender unlinkability and SHOULD be evaluated carefully to avoid introducing correlation risks.
Defending against large-scale DoS attacks is considered a deployment-level responsibility and is out of scope for this specification.
The need for secure communications has become paramount. This specification outlines a protocol describing a @@ -7748,14 +7931,27 @@ communication.
860bae2
3f722d9
ea62398
This spec integrates Interep into the RLN spec. @@ -7861,16 +8057,25 @@ previously in proof verification.
title: RLN-STEALTH-COMMITMENTS -name: RLN Stealth Commitment Usage -category: Standards Track -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
0b0e00f
This specification describes the usage of stealth commitments to add prospective users to a network-governed @@ -7965,16 +8170,26 @@ which also includes a test to generate a stealth commitment for a given user.
title: RLN-V2 -name: Rate Limit Nullifier V2 -status: raw -editor: Rasul Ibragimov curryrasul@gmail.com -contributors:
8342636
d7e84b4
The protocol specified in this document is an improvement of 32/RLN-V1, being more general construct, that allows to set various limits for an epoch @@ -8155,16 +8370,34 @@ this spec inherits all the security considerations of 2
title: SDS -name: Scalable Data Sync protocol for distributed logs -status: raw -editor: Hanno Cornelius hanno@status.im -contributors:
6980237
171e934
6672c5b
b1da703
3505da6
536d31b
8ee2a6d
235c1d5
7182459
7a01711
08b363d
bee78c4
This specification introduces the Scalable Data Sync (SDS) protocol to achieve end-to-end reliability @@ -8590,16 +8823,17 @@ if there are any eligible items in the outgoing repair request buffer, regardless of whether other participants have also recently broadcast a Periodic Sync message.
This section contains meta info about writing RFCs. This section (including its subsections) MUST be removed.
COSS explains the Vac RFC process.
Contributors can visit Waku RFCs for new Waku specifications under discussion.
slug: 10 -title: 10/WAKU2 -name: Waku v2 -status: draft -editor: Hanno Cornelius hanno@status.im -contributors:
Core Waku protocol specifications, including messaging, peer discovery, and network primitives.
34aa3f3
cafa04f
ff87c84
8e14d58
6cf68fd
6734b16
356649a
550238c
eef961b
d6651b7
6e98666
9b740d8
330c35b
Waku is a family of modular peer-to-peer protocols for secure communication. The protocols are designed to be secure, privacy-preserving, censorship-resistant @@ -9223,581 +9476,30 @@ for more details on this piece of work.
21/WAKU2-FAULT-TOLERANT-STORE
Waku is a family of modular peer-to-peer protocols for secure communication. -The protocols are designed to be secure, privacy-preserving, censorship-resistant -and being able to run in resource-restricted environments. -At a high level, it implements Pub/Sub over libp2p -and adds a set of capabilities to it. -These capabilities are things such as: -(i) retrieving historical messages for mostly-offline devices -(ii) adaptive nodes, allowing for heterogeneous nodes to contribute to the network -(iii) preserving bandwidth usage for resource-restriced devices
This makes Waku ideal for running a p2p protocol on mobile devices and -other similar restricted environments.
Historically, it has its roots in 6/WAKU1, -which stems from Whisper, -originally part of the Ethereum stack. -However, Waku acts more as a thin wrapper for Pub/Sub and has a different API. -It is implemented in an iterative manner where initial focus -is on porting essential functionality to libp2p. -See rough road map (2020) for more historical context.
Waku, as a family of protocols, is designed to have a set of properties -that are useful for many applications:
1.Useful for generalized messaging.
Many applications require some form of messaging protocol to communicate -between different subsystems or different nodes. -This messaging can be human-to-human, machine-to-machine or a mix. -Waku is designed to work for all these scenarios.
2.Peer-to-peer.
Applications sometimes have requirements that make them suitable -for peer-to-peer solutions:
3.Runs anywhere.
Applications often run in restricted environments, -where resources or the environment is restricted in some fashion. -For example:
4.Privacy-preserving.
Applications often have a desire for some privacy guarantees, such as:
5.Modular design.
Applications often have different trade-offs when it comes to what properties they -and their users value. -Waku is designed in a modular fashion where an application protocol or -node can choose what protocols they run. -We call this concept adaptive nodes.
For example:
For more on the concept of adaptive nodes and what this means in practice, -please see the 30/ADAPTIVE-NODES spec.
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, -“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and -“OPTIONAL” in this document are to be interpreted as described in 2119.
While Waku is best thought of as a single cohesive thing, -there are three network interaction domains:
(a) gossip domain -(b) discovery domain -(c) request/response domain
Since Waku is built on top of libp2p, many protocols have a libp2p protocol identifier. -The current main protocol identifiers -are:
/vac/waku/relay/2.0.0
/vac/waku/store-query/3.0.0
/vac/waku/filter/2.0.0-beta1
/vac/waku/lightpush/2.0.0-beta1
This is in addition to protocols that specify messages, payloads, and -recommended usages. -Since these aren't negotiated libp2p protocols, -they are referred to by their RFC ID. -For example:
There are also more experimental libp2p protocols such as:
/vac/waku/waku-rln-relay/2.0.0-alpha1
/vac/waku/peer-exchange/2.0.0-alpha1
The semantics of these protocols are referred to by RFC ID 17/WAKU2-RLN-RELAY and 34/WAKU2-PEER-EXCHANGE.
Unless otherwise specified, -all protocols are implemented over libp2p and use Protobuf by default. -Since messages are exchanged over a bi-directional binary stream, -as a convention, -libp2p protocols prefix binary message payloads with -the length of the message in bytes. -This length integer is encoded as a protobuf varint.
Waku is using gossiping to disseminate messages throughout the network.
Protocol identifier: /vac/waku/relay/2.0.0
See 11/WAKU2-RELAY specification for more details.
For an experimental privacy-preserving economic spam protection mechanism, -see 17/WAKU2-RLN-RELAY.
See 23/WAKU2-TOPICS -for more information about the recommended topic usage.
In addition to /vac/waku/* protocols, -Waku MAY directly use the following libp2p protocols:
/vac/waku/*
/ipfs/ping/1.0.0 -
for liveness checks between peers, or -to keep peer-to-peer connections alive.
/ipfs/id/1.0.0 -
and
/ipfs/id/push/1.0.0 -
respectively, as basic means for capability discovery. -These protocols are anyway used by the libp2p connection -establishment layer Waku is built on. -We plan to introduce a new Vac capability discovery protocol -with better anonymity properties and more functionality.
Waku is built in top of libp2p, and like libp2p it strives to be transport agnostic. -We define a set of recommended transports in order to achieve a baseline of -interoperability between clients. -This section describes these recommended transports.
Waku client implementations SHOULD support the TCP transport. -Where TCP is supported it MUST be enabled for both dialing and listening, -even if other transports are available.
Waku nodes running in environments that do not allow the use of TCP directly, -MAY use other transports.
A Waku node SHOULD support secure websockets for bidirectional communication streams, -for example in a web browser context.
A node MAY support unsecure websockets if required by the application or -running environment.
Waku can retrieve a list of nodes to connect to using DNS-based discovery -as per EIP-1459. -While this is a useful way of bootstrapping connection to a set of peers, -it MAY be used in conjunction with an ambient peer discovery -procedure to find other nodes to connect to, -such as Node Discovery v5. -It is possible to bypass the discovery domain by specifying static nodes.
WAKU2-ENR -describes the usage of EIP-778 ENR (Ethereum Node Records) -for Waku discovery purposes. -It introduces two new ENR fields, multiaddrs and -waku2, that a Waku node MAY use for discovery purposes. -These fields MUST be used under certain conditions, as set out in the specification. -Both EIP-1459 DNS-based discovery and Node Discovery v5 operate on ENR, -and it's reasonable to expect even wider utility for ENR in Waku networks in the future.
multiaddrs
waku2
In addition to the Gossip domain, -Waku provides a set of request/response protocols. -They are primarily used in order to get Waku to run in resource restricted environments, -such as low bandwidth or being mostly offline.
Protocol identifier*: /vac/waku/store-query/3.0.0
This is used to fetch historical messages for mostly offline devices. -See 13/WAKU2-STORE spec specification for more details.
There is also an experimental fault-tolerant addition to the store protocol -that relaxes the high availability requirement. -See 21/WAKU2-FAULT-TOLERANT-STORE
Protocol identifier*: /vac/waku/filter/2.0.0-beta1
This is used to preserve more bandwidth when fetching a subset of messages. -See 12/WAKU2-FILTER specification for more details.
Protocol identifier*: /vac/waku/lightpush/2.0.0-beta1
This is used for nodes with short connection windows and -limited bandwidth to publish messages into the Waku network. -See 19/WAKU2-LIGHTPUSH specification for more details.
The above is a non-exhaustive list, -and due to the modular design of Waku, -there may be other protocols here that provide a useful service to the Waku network.
See the sequence diagram below for an overview of how different protocols interact.
We have six nodes, A-F. -The protocols initially mounted are indicated as such. -The PubSub topics pubtopic1 and -pubtopic2 is used for routing and -indicates that it is subscribed to messages on that topic for relay, -see 11/WAKU2-RELAY for details. -Ditto for 13/WAKU2-STORE -where it indicates that these messages are persisted on that node.
pubtopic1
pubtopic2
Node A creates a WakuMessage msg1 with a ContentTopic contentTopic1. -See 14/WAKU2-MESSAGE for more details. -If WakuMessage version is set to 1, -we use the 6/WAKU1 compatible data field with encryption. -See 7/WAKU-DATA for more details.
msg1
contentTopic1
data
Node F requests to get messages filtered by PubSub topic pubtopic1 and -ContentTopic contentTopic1. -Node D subscribes F to this filter and -will in the future forward messages that match that filter. -See 12/WAKU2-FILTER for more details.
Node A publishes msg1 on pubtopic1 and -subscribes to that relay topic. -It then gets relayed further from B to D, but -not C since it doesn't subscribe to that topic. -See 11/WAKU2-RELAY.
Node D saves msg1 for possible later retrieval by other nodes. -See 13/WAKU2-STORE.
Node D also pushes msg1 to F, -as it has previously subscribed F to this filter. -See 12/WAKU2-FILTER.
At a later time, Node E comes online. -It then requests messages matching pubtopic1 and -contentTopic1 from Node D. -Node D responds with messages meeting this (and possibly other) criteria. -See 13/WAKU2-STORE.
6/WAKU1 and Waku are different protocols all together. -They use a different transport protocol underneath; -6/WAKU1 is devp2p RLPx based while Waku uses libp2p. -The protocols themselves also differ as does their data format. -Compatibility can be achieved only by using a bridge -that not only talks both devp2p RLPx and libp2p, -but that also transfers (partially) the content of a packet from one version -to the other.
See 15/WAKU-BRIDGE for details on a bidirectional bridge mode.
Each protocol layer of Waku provides a distinct service and -is associated with a separate set of security features and concerns. -Therefore, the overall security of Waku -depends on how the different layers are utilized. -In this section, -we overview the security properties of Waku protocols -against a static adversarial model which is described below. -Note that a more detailed security analysis of each Waku protocol -is supplied in its respective specification as well.
In the primary adversarial model, -we consider adversary as a passive entity that attempts to collect information -from others to conduct an attack, -but it does so without violating protocol definitions and instructions.
The following are not considered as part of the adversarial model:
Waku by default guarantees pseudonymity for all of the protocol layers -since parties do not have to disclose their true identity -and instead they utilize libp2p PeerID as their identifiers. -While pseudonymity is an appealing security feature, -it does not guarantee full anonymity since the actions taken under the same pseudonym -i.e., PeerID can be linked together and -potentially result in the re-identification of the true actor.
PeerID
At a high level, -anonymity is the inability of an adversary in linking an actor -to its data/performed action (the actor and action are context-dependent). -To be precise about linkability, -we use the term Personally Identifiable Information (PII) -to refer to any piece of data that could potentially -be used to uniquely identify a party. -For example, the signature verification key, and -the hash of one's static IP address are unique for each user and -hence count as PII. -Notice that users' actions can be traced through their PIIs -(e.g., signatures) and hence result in their re-identification risk. -As such, we seek anonymity by avoiding linkability between actions and -the actors / actors' PII. Concerning anonymity, Waku provides the following features:
Publisher-Message Unlinkability: -This feature signifies the unlinkability of a publisher -to its published messages in the 11/WAKU2-RELAY protocol. -The Publisher-Message Unlinkability -is enforced through the StrictNoSign policy due to which the data fields -of pubsub messages that count as PII for the publisher must be left unspecified.
StrictNoSign
Subscriber-Topic Unlinkability: -This feature stands for the unlinkability of the subscriber -to its subscribed topics in the 11/WAKU2-RELAY protocol. -The Subscriber-Topic Unlinkability -is achieved through the utilization of a single PubSub topic. -As such, subscribers are not re-identifiable from their subscribed topic IDs -as the entire network is linked to the same topic ID. -This level of unlinkability / anonymity is known as k-anonymity -where k is proportional to the system size (number of subscribers). -Note that there is no hard limit on the number of the pubsub topics, however, -the use of one topic is recommended for the sake of anonymity.
This property indicates that no adversary can flood the system -(i.e., publishing a large number of messages in a short amount of time), -either accidentally or deliberately, with any kind of message -i.e. even if the message content is valid or useful. -Spam protection is partly provided in 11/WAKU2-RELAY -through the scoring mechanism -provided for by GossipSub v1.1. -At a high level, -peers utilize a scoring function to locally score the behavior -of their connections and remove peers with a low score.
11/WAKU2-RELAY
Confidentiality can be addressed through data encryption whereas integrity and -authenticity are achievable through digital signatures. -These features are provided for in 14/WAKU2-MESSAGE (version 1)` -through payload encryption as well as encrypted signatures.
Lack of anonymity/unlinkability in the protocols involving direct connections -including 13/WAKU2-STORE and 12/WAKU2-FILTER protocols:
13/WAKU2-STORE
12/WAKU2-FILTER
The anonymity/unlinkability is not guaranteed in the protocols like 13/WAKU2-STORE -and 12/WAKU2-FILTER where peers need to have direct connections -to benefit from the designated service. -This is because during the direct connections peers utilize PeerID -to identify each other, -therefore the service obtained in the protocol is linkable -to the beneficiary's PeerID (which counts as PII). -For 13/WAKU2-STORE, -the queried node would be able to link the querying node's PeerID -to its queried topics. -Likewise, in the 12/WAKU2-FILTER, -a full node can link the light node's PeerIDs to its content filter.
There are multiple implementations of Waku and its protocols:
Below you can find an overview of the specifications that they implement -as they relate to Waku. -This includes Waku legacy specifications, as they are used for bridging between the two networks.
*js-waku implements 13/WAKU2-STORE as a querying node only. -**js-waku only implements 19/WAKU2-LIGHTPUSH requests.
To implement a minimal Waku client, -we recommend implementing the following subset in the following order:
b346ad2
0904a8b
8ff46fa
4c4591c
6874961
To get compatibility with Waku Legacy:
7/WAKU-DATA
For an interoperable keep-alive mechanism:
The following features are currently experimental, -under research and initial implementations:
Economic Spam Resistance:
We aim to enable an incentivized spam protection technique -to enhance 11/WAKU2-RELAY by using rate limiting nullifiers. -More details on this can be found in 17/WAKU2-RLN-RELAY. -In this advanced method, -peers are limited to a certain rate of messaging per epoch and -an immediate financial penalty is enforced for spammers who break this rate.
Prevention of Denial of Service (DoS) and Node Incentivization: -Denial of service signifies the case where an adversarial node -exhausts another node's service capacity (e.g., by making a large number of requests) -and makes it unavailable to the rest of the system. -DoS attack is to be mitigated through the accounting model as described in 18/WAKU2-SWAP. -In a nutshell, peers have to pay for the service they obtain from each other. -In addition to incentivizing the service provider, -accounting also makes DoS attacks costly for malicious peers. -The accounting model can be used in 13/WAKU2-STORE and -12/WAKU2-FILTER to protect against DoS attacks.
Additionally, this gives node operators who provide a useful service to the network -an incentive to perform that service. -See 18/WAKU2-SWAP -for more details on this piece of work.
libp2p specs
6/WAKU1
Whisper spec (EIP627)
Waku v2 plan
30/ADAPTIVE-NODES
Protocol Identifiers
14/WAKU2-MESSAGE
26/WAKU-PAYLOAD
23/WAKU2-TOPICS
27/WAKU2-PEERS
bi-directional binary stream
Protobuf varint encoding
11/WAKU2-RELAY spec
17/WAKU2-RLN-RELAY
EIP-1459
Ambient peer discovery
Node Discovery v5
WAKU2-ENR
EIP-778 ENR (Ethereum Node Records)
13/WAKU2-STORE spec
21/WAKU2-FT-STORE
19/WAKU2-LIGHTPUSH
15/WAKU-BRIDGE
k-anonymity
GossipSub v1.1
nim-waku (Nim)
go-waku (Go)
js-waku (NodeJS and Browser)
8/WAKU-MAIL
9/WAKU-RPC
16/WAKU2-RPC
18/WAKU2-SWAP spec
slug: 11 -title: 11/WAKU2-RELAY -name: Waku v2 Relay -status: stable -tags: waku-core -editor: Hanno Cornelius hanno@status.im -contributors:
11/WAKU2-RELAY specifies a Publish/Subscribe approach to peer-to-peer messaging with a strong focus on privacy, censorship-resistance, security and scalability. @@ -10040,10 +9742,10 @@ on behalf of the group as such the true signer is indistinguishable from other group members. -
10/WAKU2
slug: 12 -title: 12/WAKU2-FILTER -name: Waku v2 Filter -status: draft -tags: waku-core -version: 01 -editor: Hanno Cornelius hanno@status.im -contributors:
e8a3f8a
e4d8f27
046a3b7
57124a7
940d795
420adf1
previous versions: 00
Protocol identifiers:
/vac/waku/filter-push/2.0.0-beta1
This specification describes the 12/WAKU2-FILTER protocol, which enables a client to subscribe to a subset of real-time messages from a Waku peer. This is a more lightweight version of 11/WAKU2-RELAY, @@ -10134,7 +9845,7 @@ query for a recent time window, provided it is acceptable to do frequent polling
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in 2119.
Content filtering is a way to do message-based filtering. Currently the only content filter being applied is on contentTopic.
contentTopic
WakuFilter
Note that while using WakuFilter allows light nodes to save bandwidth, it comes with a privacy cost in the sense that they need to disclose their liking topics to the full nodes to retrieve the relevant messages. @@ -10360,10 +10071,10 @@ Examples of such 2PC protocols are Oblivious Transfers and one-way Private Set Intersections (PSI). -
slug: 13 -title: 13/WAKU2-STORE -name: Waku Store Query -status: draft -tags: waku-core -version: 01 -editor: Hanno Cornelius hanno@status.im -contributors:
1b8b2ac
a60a2c4
755be94
3baed07
51e2879
Previous version: 00
This specification explains the WAKU2-STORE protocol, which enables querying of 14/WAKU2-MESSAGEs.
WAKU2-STORE
slug: 14 -title: 14/WAKU2-MESSAGE -name: Waku v2 Message -status: stable -category: Standards Track -tags: waku/core-protocol -editor: Hanno Cornelius hanno@status.im -contributors:
8052808
8e70159
88df5d8
9cd48a8
10/WAKU2 is a family of modular peer-to-peer protocols for secure communication. These protocols are designed to be secure, privacy-preserving, @@ -10958,7 +10689,7 @@ message.timestamp = 0x175789bfa23f8400 message_hash = 0x483ea950cb63f9b9d6926b262bb36194d3f40a0463ce8446228350bd44e96de4 -
The level of confidentiality, integrity, and authenticity of the WakuMessage payload is discretionary. @@ -10985,9 +10716,9 @@ for network participants to behave correctly, this attribute is inherently insecure. A malicious actor can tamper with the value of a Waku message’s ephemeral attribute, and the receiver would not be able to verify the integrity of the message.
WakuMessage
ephemeral
c3d5fe6
d637b10
4bf2f6e
f883f26
This specification describes how 6/WAKU1 traffic can be used with 10/WAKU2 networks.
payload
contentFilter
topic
expiry
ttl
As mentioned above, a bridge will be posting new 6/WAKU1 envelopes, which requires doing the PoW nonce calculation.
slug: 17 -title: 17/WAKU2-RLN-RELAY -name: Waku v2 RLN Relay -status: draft -tags: waku-core -editor: Alvaro Revuelta alvaro@status.im -contributors:
776c1b7
5064ded
7b443c1
244ea55
7bcefac
1ed5919
4b3b4e3
This specification describes the 17/WAKU2-RLN-RELAY protocol, which is an extension of 11/WAKU2-RELAY to provide spam protection using Rate Limiting Nullifiers (RLN).
The acceptable_root_window_size should indicate how many blocks may have been mined during the time it takes for a peer to receive a message. This formula represents a lower bound of the number of acceptable roots.
acceptable_root_window_size
slug: 19 -title: 19/WAKU2-LIGHTPUSH -name: Waku v2 Light Push -status: draft -editor: Hanno Cornelius hanno@status.im -contributors:
c88680a
f9efd29
c90013b
Protocol identifier: /vac/waku/lightpush/2.0.0-beta1
Light nodes with short connection windows and limited bandwidth wish to publish messages into the Waku network. Additionally, @@ -11467,29 +11238,38 @@ or forward the PushRequest via 19/LIGHTPUSH on a Security Considerations +
PushRequest
Since this can introduce an amplification factor, it is RECOMMENDED for the node relaying to the rest of the network to take extra precautions. This can be done by rate limiting via 17/WAKU2-RLN-RELAY.
Note that the above is currently not fully implemented.
e4f5f28
This specification describes the usage of the ENR (Ethereum Node Records) format for 10/WAKU2 purposes. The ENR format is defined in EIP-778 [3].
true
slug: 33 -title: 33/WAKU2-DISCV5 -name: Waku v2 Discv5 Ambient Peer Discovery -status: draft -editor: Daniel Kaiser danielkaiser@status.im -contributors:
5971166
87d4ff7
dcc579c
38d68ce
b8f8d20
c6ef447
037474d
33/WAKU2-DISCV5 specifies a modified version of Ethereum's Node Discovery Protocol v5 as a means for ambient node discovery. @@ -11784,7 +11581,7 @@ as in go-waku. -
33/WAKU2-DISCV5
Implementations should limit the number of bucket entries that have the same network parameters (IP address / port) to mitigate Sybil attacks.
Properly protecting against eclipse attacks is challenging and raises research questions that we will address in future stages of our discv5 roadmap.
34/WAKU2-PEER-EXCHANGE
slug: 34 -title: 34/WAKU2-PEER-EXCHANGE -name: Waku2 Peer Exchange -status: draft -category: Standards Track -tags: waku/core-protocol -editor: Hanno Cornelius hanno@status.im -contributors:
2b297d5
This document specifies a simple request-response peer exchange protocol. Responders send information about a requested number of peers. The main purpose of this protocol is providing resource restricted devices with peers.
The peer exchange protocol specified in this document comes with anonymity and security implications. @@ -12015,9 +11820,9 @@ are not propagated in the relay domain. However, there might still be some form of leakage: ENRs could be used to track peers and facilitate linking attacks. We will investigate this further in our Waku anonymity analysis.
slug: 36 -title: 36/WAKU2-BINDINGS-API -name: Waku v2 C Bindings API -status: draft -tags: waku-core -editor: Richard Ramos richard@status.im -contributors:
cb56103
e9469d0
76e514a
6bd686d
Native applications that wish to integrate Waku may not be able to use nwaku and its JSON RPC API due to constraints @@ -13408,19 +13225,31 @@ enr and peerID each node found. onErrCb will be executed with the reason the function execution failed.
onErrCb
0277fd0
77029a2
e5b859a
This specification describes an opinionated deployment of 10/WAKU2 protocols to form a coherent and shared decentralized messaging network that is open-access, @@ -13503,7 +13332,7 @@ as described in Transports +
Relay nodes MUST follow 10/WAKU2 specifications with regards to supporting different transports. If TCP transport is available, @@ -13717,9 +13546,9 @@ and SHOULD use the short length format:
slug: 66 -title: 66/WAKU2-METADATA -name: Waku Metadata Protocol -status: draft -editor: Franck Royer franck@status.im -contributors:
4361e29
f829b12
d82eacc
This specification describes the metadata that can be associated with a 10/WAKU2 node.
Application-layer specifications built on top of Waku core protocols.
3b152e4
89a94a5
c4ff509
8841f49
a16a2b4
Content Topics:
/eth-pm/1/public-key/proto
/eth-pm/1/private-message/proto
This specification explains the Toy Ethereum Private Message protocol which enables a peer to send an encrypted message to another peer over the Waku network using the peer's Ethereum address.
M
B'
Alice SHOULD now publish this message on the Private Message content topic.
f08de10
33cf551
29acb80
7bd0712
This specification explains the Toy Ethereum Private Message protocol -which enables a peer to send an encrypted message to another peer -over the Waku network using the peer's Ethereum address.
Alice wants to send an encrypted message to Bob, -where only Bob can decrypt the message. -Alice only knows Bob's Ethereum Address.
The goal of this specification -is to demonstrate how Waku can be used for encrypted messaging purposes, -using Ethereum accounts for identity. -This protocol caters to Web3 wallet restrictions, -allowing it to be implemented using standard Web3 API. -In the current state, -Toy Ethereum Private Message, ETH-PM, has privacy and features limitations, -has not been audited and hence is not fit for production usage. -We hope this can be an inspiration for developers -wishing to build on top of Waku.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, -“SHOULD NOT”, “RECOMMENDED”, “MAY”, and -“OPTIONAL” in this document are to be interpreted as described in 2119.
Here are the variables used in the protocol and their definition:
B
b
b'
The proposed protocol MUST adhere to the following design requirements:
Alice's details are not included in the message's structure, -meaning that there is no programmatic way for Bob to reply to Alice -or verify her identity.
Private messages are sent on the same content topic for all users. -As the recipient data is encrypted, -all participants must decrypt all messages which can lead to scalability issues.
This protocol does not guarantee Perfect Forward Secrecy nor Future Secrecy: -If Bob's private key is compromised, past and future messages could be decrypted. -A solution combining regular X3DH -bundle broadcast with Double Ratchet -encryption would remove these limitations; -See the Status secure transport specification -for an example of a protocol that achieves this in a peer-to-peer setting.
Bob MUST decide to participate in the protocol before Alice can send him a message. -This is discussed in more detail in -Consideration for a non-interactive/uncoordinated protocol
First, Bob needs to generate a keypair for Encryption purposes.
Bob SHOULD get 32 bytes from a secure random source as Encryption Private Key, b'. -Then Bob can compute the corresponding SECP-256k1 Public Key, B'.
For Alice to encrypt messages for Bob, -Bob SHOULD broadcast his Encryption Public Key B'. -To prove that the Encryption Public Key B' -is indeed owned by the owner of Bob's Ethereum Account B, -Bob MUST sign B' using B.
To prove ownership of the Encryption Public Key, -Bob must sign it using EIP-712 v3, -meaning calling eth_signTypedData_v3 on his wallet's API.
eth_signTypedData_v3
Note: While v4 also exists, it is not available on all wallets and -the features brought by v4 is not needed for the current use case.
The TypedData to be passed to eth_signTypedData_v3 MUST be as follows, where:
TypedData
encryptionPublicKey
0x
bobAddress
const typedData = { - domain: { - chainId: 1, - name: 'Ethereum Private Message over Waku', - version: '1', - }, - message: { - encryptionPublicKey: encryptionPublicKey, - ownerAddress: bobAddress, - }, - primaryType: 'PublishEncryptionPublicKey', - types: { - EIP712Domain: [ - { name: 'name', type: 'string' }, - { name: 'version', type: 'string' }, - { name: 'chainId', type: 'uint256' }, - ], - PublishEncryptionPublicKey: [ - { name: 'encryptionPublicKey', type: 'string' }, - { name: 'ownerAddress', type: 'string' }, - ], - }, - } -
The resulting signature is then included in a PublicKeyMessage, where
PublicKeyMessage
encryption_public_key
eth_address
signature
syntax = "proto3"; - -message PublicKeyMessage { - bytes encryption_public_key = 1; - bytes eth_address = 2; - bytes signature = 3; -} -
This MUST be wrapped in a 14/WAKU-MESSAGE version 0, -with the Public Key Broadcast content topic. -Finally, Bob SHOULD publish the message on Waku.
Alice has to get Bob's public Key to send a message to Bob. -Because an Ethereum Address is part of the hash of the public key's account, -it is not enough in itself to deduce Bob's Public Key.
This is why the protocol dictates that Bob MUST send his Public Key first, -and Alice MUST receive it before she can send him a message.
Moreover, nwaku, the reference implementation of 13/WAKU2-STORE, -stores messages for a maximum period of 30 days. -This means that Bob would need to broadcast his public key -at least every 30 days to be reachable.
Below we are reviewing possible solutions to mitigate this "sign up" step.
If Bob has signed at least one transaction with his account -then his Public Key can be extracted from the transaction's ECDSA signature. -The challenge with this method is that standard Web3 Wallet API -does not allow Alice to specifically retrieve all/any transaction sent by Bob.
Alice would instead need to use the eth.getBlock API -to retrieve Ethereum blocks one by one. -For each block, she would need to check the from value of each transaction -until she finds a transaction sent by Bob.
eth.getBlock
from
This process is resource intensive and -can be slow when using services such as Infura due to rate limits in place, -which makes it inappropriate for a browser or mobile phone environment.
An alternative would be to either run a backend -that can connect directly to an Ethereum node, -use a centralized blockchain explorer -or use a decentralized indexing service such as The Graph.
Note that these would resolve a UX issue -only if a sender wants to proceed with air drops.
Indeed, if Bob does not publish his Public Key in the first place -then it MAY be an indication that he does not participate in this protocol -and hence will not receive messages.
However, these solutions would be helpful -if the sender wants to proceed with an air drop of messages: -Send messages over Waku for users to retrieve later, -once they decide to participate in this protocol. -Bob may not want to participate first but may decide to participate at a later stage -and would like to access previous messages. -This could make sense in an NFT offer scenario: -Users send offers to any NFT owner, -NFT owner may decide at some point to participate in the protocol and -retrieve previous offers.
Another improvement would be for Bob not having to re-publish his public key -every 30 days or less. -Similarly to above, -if Bob stops publishing his public key -then it MAY be an indication that he does not participate in the protocol anymore.
In any case, -the protocol could be modified to store the Public Key in a more permanent storage, -such as a dedicated smart contract on the blockchain.
Alice MAY monitor the Waku network to collect Ethereum Address and -Encryption Public Key tuples. -Alice SHOULD verify that the signatures of PublicKeyMessages she receives -are valid as per EIP-712. -She SHOULD drop any message without a signature or with an invalid signature.
Using Bob's Encryption Public Key, -retrieved via 10/WAKU2, -Alice MAY now send an encrypted message to Bob.
If she wishes to do so, -Alice MUST encrypt her message M using Bob's Encryption Public Key B', -as per 26/WAKU-PAYLOAD Asymmetric Encryption specs.
slug: 26 -title: 26/WAKU2-PAYLOAD -name: Waku Message Payload Encryption -status: draft -editor: Oskar Thoren oskarth@titanproxy.com -contributors:
This specification describes how Waku provides confidentiality, authenticity, and integrity, as well as some form of unlinkability. Specifically, it describes how encryption, decryption and @@ -14277,7 +13929,7 @@ but written in a way that is agnostic and self-contained for EIP-627: Whisper spec as well from RLPx Transport Protocol spec (ECIES encryption) with some modifications.
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in 2119.
padding
In order to decode a message, a node SHOULD try to apply both symmetric and asymmetric decryption operations. This is because the type of encryption is not included in the message.
slug: 53 -title: 53/WAKU2-X3DH -name: X3DH usage for Waku payload encryption -status: draft -category: Standards Track -tags: waku-application -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
b60abdb
51567b1
9fd3266
5e95a1a
555eb20
This document describes a method that can be used to provide a secure channel between two peers, and thus provide confidentiality, integrity, authenticity and forward secrecy. @@ -14466,7 +14127,7 @@ with some adaptations to operate in a decentralized environment.
Nodes on a network may want to communicate with each other in a secure manner, without other nodes network being able to read their messages.
The message key MUST be derived from a single ratchet step in the symmetric-key ratchet as described in Symmetric key ratchet
The message key MUST be used to encrypt the next message to be sent.
Inherits the security considerations of X3DH @@ -14718,10 +14379,10 @@ In this case, the server is trusted.
slug: 54 -title: 54/WAKU2-X3DH-SESSIONS -name: Session management for Waku X3DH -status: draft -category: Standards Track -tags: waku-application -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
db365cb
0e490d4
7f8b187
a22c2a0
484df92
This document specifies how to manage sessions based on an X3DH key exchange. This includes how to establish new sessions, how to re-establish them, how to maintain them, and how to close them.
installation-id
slug: 6 -title: 6/WAKU1 -name: Waku v1 -status: stable -editor: Oskar Thorén oskarth@titanproxy.com -contributors:
Legacy Waku standards retained for reference and historical compatibility.
be052c8
7d83b3d
161b35a
4c666c6
61f7641
This specification describes the format of Waku packets within the ÐΞVp2p Wire Protocol. This spec substitutes EIP-627. Waku is a fork of the original Whisper protocol @@ -15483,13 +15165,13 @@ This can be mitigated somewhat by running on e.g. port 80 or but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
80
but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
Notes useful for implementing Waku mode.
1.Avoid duplicate envelopes
To avoid duplicate envelopes, @@ -15592,723 +15274,39 @@ confirmations-enabled and rate-limits
Felix Lange et al. The RLPx Transport Protocol. Ethereum.
This specification describes the format of Waku packets within the ÐΞVp2p Wire Protocol. -This spec substitutes EIP-627. -Waku is a fork of the original Whisper protocol -that enables better usability for resource restricted devices, -such as mostly-offline bandwidth-constrained smartphones. -It does this through (a) light node support, (b) historic envelopes -(with a mailserver) (c) expressing topic interest for better bandwidth usage -and (d) basic rate limiting.
Waku was created to incrementally improve in areas that Whisper is lacking in, -with special attention to resource restricted devices. -We specify the standard for Waku packets in order to ensure forward compatibility -of different Waku clients, backwards compatibility with Whisper clients, -as well as to allow multiple implementations of Waku and its capabilities. -We also modify the language to be more unambiguous, concise and consistent.
waku-envelope
For nodes to communicate, they MUST implement devp2p and run RLPx. -They MUST have some way of connecting to other nodes. -Node discovery is largely out of scope for this spec, -but see the appendix for some suggestions on how to do this.
This protocol needs to advertise the waku/1 capability.
waku/1
In Whisper, envelopes are gossiped between peers. -Whisper is a form of rumor-mongering protocol -that works by flooding to its connected peers based on some factors. -Envelopes are eligible for retransmission until their TTL expires. -A node SHOULD relay envelopes to all connected nodes -if an envelope matches their PoW and bloom filter settings. -If a node works in light mode, it MAY choose not to forward envelopes. -A node MUST NOT send expired envelopes, -unless the envelopes are sent as a 8/WAKU-MAIL response. -A node SHOULD NOT send an envelope to a peer that it has already sent before.
Nodes SHOULD limit the maximum size of both packets and envelopes. -If a packet or envelope exceeds its limit, it MUST be dropped.
a57d7b4
900a3e9
662eb12
93c3896
Clients MAY use their own maximum packet and envelope sizes. -The default values are 1.5mb for the RLPx Packet and 1mb for a Waku envelope.
1.5mb
1mb
All Waku packets are sent as devp2p RLPx transport protocol, version 51 packets. -These packets MUST be RLP-encoded arrays of data containing two objects: -packet code followed by another object (whose type depends on the packet code). -See informal RLP spec and -the Ethereum Yellow Paper, appendix B -for more details on RLP.
Waku is a RLPx subprotocol called waku with version 0. -The version number corresponds to the major version in the header spec. -Minor versions should not break compatibility of waku, -this would result in a new major. -(Some exceptions to this apply in the Draft stage -of where client implementation is rapidly change).
waku
0
Using Augmented Backus-Naur form (ABNF) -we have the following format:
; Packet codes 0 - 127 are reserved for Waku protocol -packet-code = 1*3DIGIT - -; rate limits per packet -packet-limit-ip = 1*DIGIT -packet-limit-peerid = 1*DIGIT -packet-limit-topic = 1*DIGIT - -; rate limits by size in bytes -bytes-limit-ip = 1*DIGIT -bytes-limit-peerid = 1*DIGIT -bytes-limit-topic = 1*DIGIT - -packet-rate-limits = "[" packet-limit-ip packet-limit-peerid packet-limit-topic "]" -bytes-rate-limits = "[" bytes-limit-ip bytes-limit-peerid bytes-limit-topic "]" - -pow-requirement-key = 0 -bloom-filter-key = 1 -light-node-key = 2 -confirmations-enabled-key = 3 -packet-rate-limits-key = 4 -topic-interest-key = 5 -bytes-rate-limits-key = 6 - -status-options = "[" - [ pow-requirement-key pow-requirement ] - [ bloom-filter-key bloom-filter ] - [ light-node-key light-node ] - [ confirmations-enabled-key confirmations-enabled ] - [ packet-rate-limits-key packet-rate-limits ] - [ topic-interest-key topic-interest ] - [ bytes-limits-key bytes-rate-limits ] -"]" - -status = status-options - -status-update = status-options - -confirmations-enabled = BIT - -light-node = BIT - -; pow is "a single floating point value of PoW. -; This value is the IEEE 754 binary representation -; of a 64-bit floating point number packed as a uint64. -; Values of qNAN, sNAN, INF and -INF are not allowed. -; Negative values are also not allowed." -pow = 1*DIGIT "." 1*DIGIT -pow-requirement = pow - -; bloom filter is "a byte array" -bloom-filter = *OCTET - -waku-envelope = "[" expiry ttl topic data nonce "]" - -; List of topics interested in -topic-interest = "[" *10000topic "]" - -; 4 bytes (UNIX time in seconds) -expiry = 4OCTET - -; 4 bytes (time-to-live in seconds) -ttl = 4OCTET - -; 4 bytes of arbitrary data -topic = 4OCTET - -; byte array of arbitrary size -; (contains encrypted payload) -data = *OCTET - -; 8 bytes of arbitrary data -; (used for PoW calculation) -nonce = 8OCTET - -messages = 1*waku-envelope - -; version of the confirmation packet -version = 1*DIGIT - -; keccak256 hash of the envelopes batch data (raw bytes) -; for which the confirmation is sent -hash = *OCTET - -hasherror = *OCTET - -; error code -code = 1*DIGIT - -; a descriptive error message -description = *ALPHA - -error = "[" hasherror code description "]" -errors = *error - -response = "[" hash errors "]" - -confirmation = "[" version response "]" - -; message confirmation packet types -batch-ack = confirmation -message-response = confirmation - -; mail server / client specific -p2p-request = waku-envelope -p2p-message = 1*waku-envelope -p2p-request-complete = *OCTET - -; packet-format needs to be paired with its -; corresponding packet-format -packet-format = "[" packet-code packet-format "]" - -required-packet = 0 status / - 1 messages / - 22 status-update / - -optional-packet = 11 batch-ack / - 12 message-response / - 126 p2p-request-complete / - 126 p2p-request / - 127 p2p-message - -packet = "[" required-packet [ optional-packet ] "]" -
All primitive types are RLP encoded. Note that, per RLP specification, -integers are encoded starting from 0x00.
0x00
The packet codes reserved for Waku protocol: 0 - 127.
Packets with unknown codes MUST be ignored without generating any error, -for forward compatibility of future versions.
The Waku sub-protocol MUST support the following packet codes:
The following message codes are optional, but they are reserved for specific purpose.
The Status packet serves as a Waku handshake and peers MUST exchange this -packet upon connection. It MUST be sent after the RLPx handshake and prior to -any other Waku packets.
A Waku node MUST await the Status packet from a peer -before engaging in other Waku protocol activity with that peer. -When a node does not receive the Status packet from a peer, -before a configurable timeout, it SHOULD disconnect from that peer.
Upon retrieval of the Status packet, the node SHOULD validate the packet -received and validated the Status packet. Note that its peer might not be in -the same state.
When a node is receiving other Waku packets from a peer before a Status -packet is received, the node MUST ignore these packets and -SHOULD disconnect from that peer. -Status packets received after the handshake is completed MUST also be ignored.
The Status packet MUST contain an association list containing various options. -All options within this association list are OPTIONAL, -ordering of the key-value pairs is not guaranteed and -therefore MUST NOT be relied on. -Unknown keys in the association list SHOULD be ignored.
This packet is used for sending the standard Waku envelopes.
The Status Update packet is used to communicate an update -of the settings of the node. -The format is the same as the Status packet, all fields are optional. -If none of the options are specified the packet MUST be ignored and -considered a noop. -Fields that are omitted are considered unchanged, -fields that haven't changed SHOULD not be transmitted.
When PoW Requirement is updated, -peers MUST NOT deliver envelopes with PoW lower than the PoW Requirement specified.
PoW is defined as average number of iterations, -required to find the current BestBit -(the number of leading zero bits in the hash), divided by envelope size and TTL:
PoW = (2**BestBit) / (size * TTL)
PoW calculation:
fn short_rlp(envelope) = rlp of envelope, excluding env_nonce field. -fn pow_hash(envelope, env_nonce) = sha3(short_rlp(envelope) ++ env_nonce) -fn pow(pow_hash, size, ttl) = 2**leading_zeros(pow_hash) / (size * ttl)
where size is the size of the RLP-encoded envelope, -excluding env_nonce field (size of short_rlp(envelope)).
env_nonce
short_rlp(envelope)
The bloom filter is used to identify a number of topics to a peer without compromising -(too much) privacy over precisely what topics are of interest. -Precise control over the information content (and thus efficiency of the filter) -may be maintained through the addition of bits.
Blooms are formed by the bitwise OR operation on a number of bloomed topics. -The bloom function takes the topic and projects them onto a 512-bit slice. -At most, three bits are marked for each bloomed topic.
The projection function is defined as a mapping -from a 4-byte slice S to a 512-bit slice D; for ease of explanation, -S will dereference to bytes, whereas D will dereference to bits.
LET D[*] = 0 -FOREACH i IN { 0, 1, 2 } DO -LET n = S[i] -IF S[3] & (2 ** i) THEN n += 256 -D[n] = 1 -END FOR
A full bloom filter (all the bits set to 1) -means that the node is to be considered a Full Node and it will accept any topic.
Full Node
If both topic interest and bloom filter are specified, -topic interest always takes precedence and bloom filter MUST be ignored.
If only bloom filter is specified, -the current topic interest MUST be discarded and -only the updated bloom filter MUST be used when forwarding or posting envelopes.
A bloom filter with all bits set to 0 signals that the node is not currently -interested in receiving any envelope.
Topic interest is used to share a node's interest in envelopes with specific topics. -It does this in a more bandwidth considerate way, -at the expense of some metadata protection. -Peers MUST only send envelopes with specified topics.
It is currently bounded to a maximum of 10000 topics. -If you are interested in more topics than that, this is currently underspecified -and likely requires updating it. The constant is subject to change.
If only topic interest is specified, the current bloom filter MUST be discarded and -only the updated topic interest MUST be used when forwarding or posting envelopes.
An empty array signals that the node is not currently interested in receiving -any envelope.
Rate limits is used to inform other nodes of their self defined rate limits.
In order to provide basic Denial-of-Service attack protection, -each node SHOULD define its own rate limits. -The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
Each node MAY decide to whitelist, i.e. do not rate limit, selected IPs or peer IDs.
If a peer exceeds node's rate limits, the connection between them MAY be dropped.
Each node SHOULD broadcast its rate limits to its peers -using the status-update packet. -The rate limits MAY also be sent as an optional parameter in the handshake.
status-update
Each node SHOULD respect rate limits advertised by its peers. -The number of packets SHOULD be throttled in order not to exceed peer's rate limits. -If the limit gets exceeded, the connection MAY be dropped by the peer.
Two rate limits strategies are applied:
Both strategies SHOULD be applied per IP address, peer id and topic.
The size limit SHOULD be greater or equal than the maximum packet size.
When the node's light-node field is set to true, -the node SHOULD NOT forward Envelopes from its peers.
light-node
A node connected to a peer with the light-node field set to true -MUST NOT depend on the peer for forwarding Envelopes.
When the node's confirmations-enabled field is set to true, -the node SHOULD send message confirmations -to its peers.
confirmations-enabled
Message confirmations tell a node that an envelope -originating from it has been received by its peers, -allowing a node to know whether an envelope has or has not been received.
A node MAY send a message confirmation for any batch of envelopes -received with a Messages packet (0x01).
0x01
A message confirmation is sent using Batch Ack packet (0x0B) or -Message Response packet (0x0C). -The message confirmation is specified in the ABNF specification.
0x0B
0x0C
The current version in the confirmation is 1.
version
confirmation
1
The supported error codes:
The drawback of sending message confirmations is that it increases the noise -in the network because for each sent envelope, -a corresponding confirmation is broadcast by one or more peers.
This packet is used for sending Dapp-level peer-to-peer requests, -e.g. Waku Mail Client requesting historic (expired) -envelopes from the Waku Mail Server.
This packet is used for sending the peer-to-peer envelopes, -which are not supposed to be forwarded any further. -E.g. it might be used by the Waku Mail Server for delivery of historic (expired) -envelopes, which is otherwise not allowed.
This packet is used to indicate that all envelopes, -requested earlier with a P2P Request packet (0x7E), -have been sent via one or more P2P Message packets (0x7F).
0x7E
0x7F
The content of the packet is explained in the Waku Mail Server specification.
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme -with SECP-256k1 public key.
Symmetric encryption uses AES GCM algorithm with random 96-bit nonce.
Packet codes 0x00 and 0x01 are already used in all Waku / Whisper versions. -Packet code 0x02 and 0x03 were previously used in Whisper but -are deprecated as of Waku v0.4
0x02
0x03
Packet code 0x22 is used to dynamically change the settings of a node.
0x22
Packet codes 0x7E and 0x7F may be used to implement Waku Mail Server and -Client. -Without the P2P Message packet it would be impossible to deliver the historic envelopes, -since they will be recognized as expired, and -the peer will be disconnected for violating the Waku protocol. -They might be useful for other purposes -when it is not possible to spend time on PoW, -e.g. if a stock exchange will want to provide live feed about the latest trades.
Waku supports multiple capabilities. -These include light node, rate limiting and bridging of traffic. -Here we list these capabilities, how they are identified, -what properties they have and what invariants they must maintain.
Additionally, -there is the capability of a mailserver which is documented in its on specification.
The rationale for light nodes is to allow for interaction with waku -on resource restricted devices as bandwidth can often be an issue.
Light nodes MUST NOT forward any incoming envelopes, -they MUST only send their own envelopes. -When light nodes happen to connect to each other, they SHOULD disconnect. -As this would result in envelopes being dropped between the two.
Light nodes are identified by the light_node value in the Status packet.
light_node
Nodes MAY implement accounting, keeping track of resource usage. -It is heavily inspired by Swarm's SWAP protocol, -and works by doing pairwise accounting for resources.
Each node keeps track of resource usage with all other nodes. -Whenever an envelope is received from a node that is expected -(fits bloom filter or topic interest, is legal, etc) this is tracked.
Every epoch (say, every minute or every time an event happens) -statistics SHOULD be aggregated and saved by the client:
In later versions this will be amended by nodes communication thresholds, -settlements and disconnect logic.
The currently advertised capability is waku/1. -This needs to be advertised in the hello ÐΞVp2p packet. -If a node supports multiple versions of waku, those needs to be explicitly advertised. -For example if both waku/0 and waku/1 are supported, -both waku/0 and waku/1 MUST be advertised.
hello
ÐΞVp2p
waku/0
These are policies that guide how we make decisions when it comes to upgradability, -compatibility, and extensibility:
Waku aims to be compatible with previous and future versions.
In cases where we want to break this compatibility, -we do so gracefully and as a single decision point.
To achieve this, we employ the following two general strategies:
Examples:
shh/6
Waku is a different subprotocol from Whisper so it isn't directly compatible. -However, the data format is the same, -so compatibility can be achieved by the use of a bridging mode as described below. -Any client which does not implement certain packet codes -should gracefully ignore the packets with those codes. -This will ensure the forward compatibility.
waku/1 and shh/6 are different DevP2P subprotocols, -however they share the same data format making their envelopes compatible. -This means we can bridge the protocols naively, this works as follows.
Roles:
Flow:
Note: This flow means if another bridge C1 is active, -we might get duplicate relaying for a envelope between C1 and C2. -I.e. Whisper(<>Waku<>Whisper)<>Waku, A-C1-C2-B. -Theoretically this bridging chain can get as long as TTL permits.
It is desirable to have a strategy for maintaining forward compatibility -between waku/1 and future version of waku. -Here we outline some concerns and strategy for this.
status-options
status
There are several security considerations to take into account when running Waku. -Chief among them are: scalability, DDoS-resistance and privacy. -These also vary depending on what capabilities are used. -The security considerations for extra capabilities, -such as mailservers -can be found in their respective specifications.
In version 0 of Waku, bandwidth usage is likely to be an issue. -For more investigation into this, -see the theoretical scaling model described -here.
Use of gossip-based routing doesn't necessarily scale. -It means each node can see an envelope multiple times, and -having too many light nodes can cause propagation probability that is too low. -See Whisper vs PSS -for more and a possible Kademlia based alternative.
Waku currently lacks incentives to run nodes, -which means node operators are more likely to create centralized choke points.
The main privacy concern with a light node -is that it has to reveal its topic interests -(in addition to its IP/ID) to its directed peers. -This is because when a light node publishes an envelope, -its directed peers will know that the light node owns that envelope -(as light nodes do not relay other envelopes). -Therefore, the directed peers of a light node can make assumptions about what envelopes -(topics) the light node is interested in.
A mailserver client fetches archival envelopes from a mailserver -through a direct connection. -In this direct connection, -the client discloses its IP/ID as well as the topics/ bloom filter -it is interested in to the mailserver. -The collection of such information allows the mailserver to link clients' IP/IDs -to their topic interests and build a profile for each client over time. -As such, the mailserver client has to trust the mailserver with this level of information.
By having a bloom filter where only the topics you are interested in are set, -you reveal which envelopes you are interested in. -This is a fundamental tradeoff between bandwidth usage and privacy, -though the tradeoff space is likely suboptimal in terms of the Anonymity -trilemma.
Privacy for Whisper / Waku haven't been studied rigorously for various threat models -like global passive adversary, local active attacker, etc. -This is unlike e.g. Tor and mixnets.
Similar to bloom filter privacy, -if you use a very specific topic you reveal more information. -See scalability model linked above.
PoW bad for heterogeneous devices:
Proof of work is a poor spam prevention mechanism. -A mobile device can only have a very low PoW -in order not to use too much CPU / burn up its phone battery. -This means someone can spin up a powerful node and overwhelm the network.
Devp2p TCP port blockable:
By default Devp2p runs on port 30303, -which is not commonly used for any other service. -This means it is easy to censor, e.g. airport WiFi. -This can be mitigated somewhat by running on e.g. port 80 or 443, -but there are still outstanding issues. -See libp2p and Tor's Pluggable Transport for how this can be improved.
30303
443
To avoid duplicate envelopes, -only connect to one Waku node. -Benign duplicate envelopes is an intrinsic property of Whisper -which often leads to a N factor increase in traffic, -where N is the number of peers you are connected to.
2.Topic specific recommendations
Consider partition topics based on some usage, -to avoid too much traffic on a single topic.
Resource restricted devices SHOULD use EIP-1459 -to discover nodes.
Known static nodes MAY also be used.
Released June 09, 2020
Released April 21,2020
RLP
Released March 17,2020
Released February 21, 2020.
0x20
0x21
topic-interest
Released February 13, 2020.
Released December 10, 2019.
Initial version. Released November 21, 2019.
Summary of main differences between this spec and Whisper v6, as described in EIP-627:
slug: 7 -title: 7/WAKU-DATA -name: Waku Envelope data field -status: stable -editor: Oskar Thorén oskarth@titanproxy.com -contributors:
This specification describes the encryption, decryption and signing of the content in the data field used in Waku.
The data field is used within the waku envelope, the field MUST contain the encrypted payload of the envelope.
waku envelope
The fields that are concatenated and encrypted as part of the data field are:
slug: 8 -title: 8/WAKU-MAIL -name: Waku Mailserver -status: stable -editor: Andrea Maria Piana andreap@status.im -contributors:
fa77279
0c8e39b
de5cfa2
28e7862
In this specification, we describe Mailservers. These are nodes responsible for archiving envelopes and delivering them to peers on-demand.
A node which wants to provide mailserver functionality MUST store envelopes from incoming Messages packets (Waku packet-code 0x01). The envelopes can be stored in any format, @@ -16468,7 +15477,7 @@ payload = "[" request-id last-envelope-hash [ cursor ] "]" it means that not all envelopes were sent due to the set Limit in the request. One or more consecutive requests MAY be sent with Cursor field filled in order to receive the rest of the envelopes.
Limit
Cursor
There are several security considerations to take into account when running or interacting with Mailservers. Chief among them are: scalability, DDoS-resistance and privacy.
A mailserver has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning mailserver can overwhelm an individual node.
topics
slug: 9 -title: 9/WAKU-RPC -name: Waku RPC API -status: stable -editor: Andrea Maria Piana andreap@status.im -contributors:
5eb393f
9617146
75705cd
e808e36
This specification describes the RPC API that Waku nodes MAY adhere to. The unified API allows clients to easily be able to connect to any node implementation. @@ -16870,25 +15890,36 @@ It can not be both.
slug: 22 -title: 22/TOY-CHAT -name: Waku v2 Toy Chat -status: draft -tags: waku/application -editor: Franck Royer franck@status.im -contributors:
Informational Waku documents covering guidance, examples, and supporting material.
722c3d2
5c5ea36
411e135
Content Topic: /toy-chat/2/huilong/proto.
/toy-chat/2/huilong/proto
This specification explains a toy chat example using Waku v2. This protocol is mainly used to:
nick
4df2d5f
dc7497a
e63d8a0
b8f088c
2b693e8
055c525
a11dfed
This specification explains a toy chat example using Waku v2. -This protocol is mainly used to:
Currently, all main Waku v2 implementations support the toy chat protocol: -nim-waku, -js-waku (NodeJS -and web) -and go-waku.
Note that this is completely separate from the protocol the Status app -is using for its chat functionality.
The chat protocol enables sending and receiving messages in a chat room. -There is currently only one chat room, which is tied to the content topic. -The messages SHOULD NOT be encrypted.
The contentTopic MUST be set to /toy-chat/2/huilong/proto.
syntax = "proto3"; - -message Chat2Message { - uint64 timestamp = 1; - string nick = 2; - bytes payload = 3; -} -
timestamp
slug: 23 -title: 23/WAKU2-TOPICS -name: Waku v2 Topic Usage Recommendations -status: draft -category: Informational -editor: Oskar Thoren oskarth@titanproxy.com -contributors:
This document outlines recommended usage of topic names in Waku v2. In 10/WAKU2 spec there are two types of topics:
/waku/1/0x007f80ff/rfc26
slug: 27 -title: 27/WAKU2-PEERS -name: Waku v2 Client Peer Management Recommendations -status: draft -editor: Hanno Cornelius hanno@status.im -contributors:
af7c413
4b77d10
e65c359
4a78cac
7daec2f
27/WAKU2-PEERS describes a recommended minimal set of peer storage and peer management features to be implemented by Waku v2 clients.
In this context, peer storage refers to a client's ability to keep track of discovered @@ -17222,10 +16235,10 @@ For example, idle TCP connections often times out after 10 to 15 minutes.
nim-waku
5 minutes
10 minutes
Peer ID
slug: 29 -title: 29/WAKU2-CONFIG -name: Waku v2 Client Parameter Configuration Recommendations -status: draft -editor: Hanno Cornelius hanno@status.im -contributors:
7408956
c506eac
930f84d
e6396b9
29/WAKU2-CONFIG describes the RECOMMENDED values to assign to configurable parameters for Waku v2 clients. Since Waku v2 is built on libp2p, @@ -17298,10 +16323,10 @@ MAY choose to implement.
29/WAKU2-CONFIG
IHaveMaxLength
slug: 30 -title: 30/ADAPTIVE-NODES -name: Adaptive nodes -status: draft -editor: Oskar Thorén oskarth@titanproxy.com -contributors:
91c9679
b35846a
04036ad
This is an informational spec that show cases the concept of adaptive nodes.
We can look at node types as a continuum, @@ -17394,10 +16430,10 @@ and 15/WAKU- This one shows a cross-section of nodes in different dimensions and shows how the connections look different for different protocols. -Copyright +Copyright Copyright and related rights waived via CC0. -References +References 11/WAKU2-RELAY 25/LIBP2P-DNS-DISCOVERY @@ -17409,20 +16445,27 @@ This subdirectory is for achrive purpose and should not be used in production ready implementations. Visit Waku RFCs for new Waku specifications under discussion.
This one shows a cross-section of nodes in different dimensions and shows how the connections look different for different protocols.
slug: 5 -title: 5/WAKU0 -name: Waku v0 -status: deprecated -editor: Oskar Thorén oskarth@titanproxy.com -contributors:
9770963
ac8fe6d
This specification describes the format of Waku messages within the ÐΞVp2p Wire Protocol. This spec substitutes EIP-627. Waku is a fork of the original Whisper protocol that enables better usability @@ -17432,7 +16475,7 @@ It does this through (a) light node support, (b) historic messages (with a mailserver) (c) expressing topic interest for better bandwidth usage and (d) basic rate limiting.
Waku was created to incrementally improve in areas that Whisper is lacking in, with special attention to resource restricted devices. We specify the standard for Waku messages @@ -17440,20 +16483,20 @@ in order to ensure forward compatibility of different Waku clients, backwards compatibility with Whisper clients, as well as to allow multiple implementations of Waku and its capabilities. We also modify the language to be more unambiguous, concise and consistent.
For nodes to communicate, they MUST implement devp2p and run RLPx. They MUST have some way of connecting to other nodes. Node discovery is largely out of scope for this spec, but see the appendix for some suggestions on how to do this.
In Whisper, messages are gossiped between peers. Whisper is a form of rumor-mongering protocol that works by flooding to its connected peers based on some factors. @@ -17464,8 +16507,8 @@ If a node works in light mode, it MAY choose not to forward envelopes. A node MUST NOT send expired envelopes, unless the envelopes are sent as a mailserver response. A node SHOULD NOT send a message to a peer that it has already sent before.
All Waku messages are sent as devp2p RLPx transport protocol, version 51 packets. These packets MUST be RLP-encoded arrays of data containing two objects: @@ -17479,7 +16522,7 @@ Minor versions should not break compatibility of waku, this would result in a new major. (Some exceptions to this apply in the Draft stage of where client implementation is rapidly change).
Using Augmented Backus-Naur form (ABNF) we have the following format:
; Packet codes 0 - 127 are reserved for Waku protocol @@ -17572,7 +16615,7 @@ packet = "[" required-packet [ optional-packet ] "]"
All primitive types are RLP encoded. Note that, per RLP specification, integers are encoded starting from 0x00.
The message codes reserved for Waku protocol: 0 - 127.
Messages with unknown codes MUST be ignored without generating any error, for forward compatibility of future versions.
The Status message serves as a Waku handshake and peers MUST exchange this message upon connection. It MUST be sent after the RLPx handshake and prior to any other Waku messages.
The Status Update message is used to communicate an update of the settings of the node. The format is the same as the Status message, all fields are optional. @@ -17734,19 +16777,19 @@ created in the future (the root cause is no time sync between nodes).
The drawback of sending message confirmations is that it increases the noise in the network because for each sent message, a corresponding confirmation is broadcast by one or more peers.
This packet is used for sending Dapp-level peer-to-peer requests, e.g. Waku Mail Client requesting old messages from the Waku Mail Server.
This packet is used for sending the peer-to-peer messages, which are not supposed to be forwarded any further. E.g. it might be used by the Waku Mail Server for delivery of old (expired) messages, which is otherwise not allowed.
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme with SECP-256k1 public key.
Packet codes 0x00 and 0x01 are already used in all Waku / Whisper versions. Packet code 0x02 and 0x03 were previously used in Whisper but are deprecated as of Waku v0.4
Waku supports multiple capabilities. These include light node, rate limiting and bridging of traffic. Here we list these capabilities, how they are identified, what properties they have and what invariants they must maintain.
Additionally there is the capability of a mailserver which is documented in its on specification.
The rationale for light nodes is to allow for interaction with waku on resource restricted devices as bandwidth can often be an issue.
Light nodes MUST NOT forward any incoming messages, @@ -17774,7 +16817,7 @@ When light nodes happen to connect to each other, they SHOULD disconnect. As this would result in messages being dropped between the two.
Light nodes are identified by the light_node value in the status message.
Nodes MAY implement accounting, keeping track of resource usage. It is heavily inspired by Swarm's SWAP protocol, @@ -17791,8 +16834,8 @@ statistics SHOULD be aggregated and saved by the client:
In later versions this will be amended by nodes communication thresholds, settlements and disconnect logic.
These are policies that guide how we make decisions when it comes to upgradability, compatibility, and extensibility:
Waku is a different subprotocol from Whisper so it isn't directly compatible. However, the data format is the same, so compatibility can be achieved by the use of a bridging mode as described below. Any client which does not implement certain packet codes should gracefully ignore the packets with those codes. This will ensure the forward compatibility.
waku/0 and shh/6 are different DevP2P subprotocols, however they share the same data format making their envelopes compatible. This means we can bridge the protocols naively, this works as follows.
It is desirable to have a strategy for maintaining forward compatibility between waku/0 and future version of waku. Here we outline some concerns and strategy for this.
There are several security considerations to take into account when running Waku. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used. The security considerations for extra capabilities such as mailservers can be found in their respective specifications.
Bandwidth usage:
In version 0 of Waku, bandwidth usage is likely to be an issue. For more investigation into this, @@ -17910,7 +16953,7 @@ for more and a possible Kademlia based alternative.
Lack of incentives:
Waku currently lacks incentives to run nodes, which means node operators are more likely to create centralized choke points.
Light node privacy:
The main privacy concern with light nodes is that directly connected peers will know that a message originates from them @@ -17931,13 +16974,13 @@ This is unlike e.g. Tor and mixnets.
Similar to bloom filter privacy, if you use a very specific topic you reveal more information. See scalability model linked above.
Proof of work is a poor spam prevention mechanism. A mobile device can only have a very low PoW in order not to use too much CPU / burn up its phone battery. This means someone can spin up a powerful node and overwhelm the network.
By default Devp2p runs on port 30303, which is not commonly used for any other service. @@ -17945,14 +16988,14 @@ This means it is easy to censor, e.g. airport WiFi. This can be mitigated somewhat by running on e.g. port 80 or 443, but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
Consider partition topics based on some usage, to avoid too much traffic on a single topic.
Resource restricted devices SHOULD use EIP-1459 to discover nodes.
8b552ba
87b56de
9042acf
8a53f24
This specification describes the JSON-RPC API that Waku v2 nodes MAY adhere to. Refer to the Waku v2 specification @@ -18640,9 +17699,9 @@ using the cursor received above; 2 messages per page, backward direction.
8f94e97
0d8ad08
3c8410c
This specification outlines how we do accounting and settlement based on the provision and usage of resources, most immediately bandwidth usage and/or storing and retrieving of Waku message. @@ -18665,7 +17738,7 @@ This enables nodes to cooperate and efficiently share resources, and in the case of unequal nodes to settle the difference through a relaxed payment mechanism in the form of sending cheques.
Protocol identifier*: /vac/waku/swap/2.0.0-beta1
/vac/waku/swap/2.0.0-beta1
The Waku network makes up a service network, and some nodes provide a useful service to other nodes. We want to account for that, and when imbalances arise, settle this. @@ -18851,9 +17924,9 @@ pick another node.
In the hard phase, in addition to sending cheques and activating policy, this is done with blockchain integration on a public testnet. More details TBD.
This also includes settlements where cheques can be redeemed.
cb4d0de
5da8a11
206133e
The reliability of 13/WAKU2-STORE protocol heavily relies on the fact that full nodes i.e., those who persist messages have high availability and @@ -18908,10 +17995,10 @@ while using this method is that a querying node has to reveal its offline time to the queried node. This will gradually result in the extraction of the node's activity pattern which can lead to inference attacks.
We extend the HistoryQuery protobuf message with two fields of start_time and end_time to signify the time range to be queried.
start_time
end_time
syntax = "proto3"; message HistoryQuery { @@ -18961,10 +18048,10 @@ As such, the messages field of the corresponding HistoryResponse MUST contain historical waku messages that satisfy the indicated pubsubtopic AND contentFilters AND the time range [start_time, end_time]. -Copyright +Copyright Copyright and related rights waived via CC0. -References +References 13/WAKU2-STORE timestamp @@ -18974,22 +18061,25 @@ MUST contain historical waku messages that satisfy the indicated pubsubto scalable infrastructure for developers creating applications for the network state. Published Specifications are currently available here, Nomos Specifications. - -title: NOMOSDA-ENCODING -name: NomosDA Encoding Protocol -status: raw -category: -tags: data-availability -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors: +Nomos Raw Specifications +Early-stage Nomos specifications that have not yet progressed beyond raw status. +NOMOSDA-ENCODING + + +NameNomosDA Encoding Protocol +Statusraw +EditorDaniel Sanchez-Quiros <danielsq@status.im> +ContributorsDaniel Kashepava <danielkashepava@status.im>Álvaro Castro-Castilla <alvaro@status.im>Filip Dimitrijevic <filip@status.im>Thomas Lavaur <thomaslavaur@status.im>Mehmet Gonen <mehmet@status.im> + + + +Timeline -Daniel Kashepava danielkashepava@status.im -Álvaro Castro-Castilla alvaro@status.im -Filip Dimitrijevic filip@status.im -Thomas Lavaur thomaslavaur@status.im -Mehmet Gonen mehmet@status.im +2025-12-22 — b1a5783 — Chore/mdbook updates (#237) +2025-12-18 — d03e699 — ci: add mdBook configuration (#233) +2025-09-25 — 39d6f07 — added nomos/raw/nomosda-encoding.md draft (#156) - + Introduction This document describes the encoding and verification processes of NomosDA, which is the data availability (DA) solution used by the Nomos blockchain. NomosDA provides an assurance that all data from Nomos blobs are accessible and verifiable by every network participant. This document presents an implementation specification describing how: @@ -18997,7 +18087,7 @@ contributors: Encoders encode blobs they want to upload to the Data Availability layer. Other nodes implement the verification of blobs that were already uploaded to DA. -Definitions +Definitions Encoder: An encoder is any actor who performs the encoding process described in this document. This involves committing to the data, generating proofs, and submitting the result to the DA layer. @@ -19119,183 +18209,32 @@ b. The amount of rows depends on the size of the data. Details The encoder and verifier processes described above make use of a variety of cryptographic functions to facilitate the correct verification of column data by verifiers. These functions rely on primitives such as polynomial commitments and Reed-Solomon erasure codes, the details of which are outside the scope of this document. These details, as well as introductions to the cryptographic primitives being used, can be found in the NomosDA Cryptographic Protocol: NomosDA Cryptographic Protocol -References +References Encoder Specification: GitHub/encoder.py Verifier Specification: GitHub/verifier.py Cryptographic protocol: NomosDA Cryptographic Protocol -Copyright +Copyright Copyright and related rights waived via CC0. - -title: NOMOSDA-ENCODING -name: NomosDA Encoding Protocol -status: raw -category: -tags: data-availability -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors: +NOMOS-DA-NETWORK + + +NameNomosDA Network +Statusraw +EditorDaniel Sanchez Quiros <danielsq@status.im> +ContributorsÁlvaro Castro-Castilla <alvaro@status.im>Daniel Kashepava <danielkashepava@status.im>Gusto Bacvinka <augustinas@status.im>Filip Dimitrijevic <filip@status.im> + + + +Timeline -Daniel Kashepava danielkashepava@status.im -Álvaro Castro-Castilla alvaro@status.im -Filip Dimitrijevic filip@status.im -Thomas Lavaur thomaslavaur@status.im -Mehmet Gonen mehmet@status.im +2025-12-22 — b1a5783 — Chore/mdbook updates (#237) +2025-12-18 — d03e699 — ci: add mdBook configuration (#233) +2025-09-25 — 51ef4cd — added nomos/raw/nomosda-network.md (#160) - + Introduction -This document describes the encoding and verification processes of NomosDA, which is the data availability (DA) solution used by the Nomos blockchain. NomosDA provides an assurance that all data from Nomos blobs are accessible and verifiable by every network participant. -This document presents an implementation specification describing how: - -Encoders encode blobs they want to upload to the Data Availability layer. -Other nodes implement the verification of blobs that were already uploaded to DA. - -Definitions - - -Encoder: An encoder is any actor who performs the encoding process described in this document. This involves committing to the data, generating proofs, and submitting the result to the DA layer. -In the Nomos architecture, the rollup sequencer typically acts as the encoder, but the role is not exclusive and any actor in the DA layer can also act as encoders. - - -Verifier: Verifies its portion of the distributed blob data as per the verification protocol. In the Nomos architecture, the DA nodes act as the verifiers. - - -Overview -In the encoding stage, the encoder takes the DA parameters and the padded blob data and creates an initial matrix of data chunks. This matrix is expanded using Reed-Solomon coding and various commitments and proofs are created for the data. -When a verifier receives a sample, it verifies the data it receives from the encoder and broadcasts the information if the data is verified. Finally, the verifier stores the sample data for the required length of time. -Construction -The encoder and verifier use the NomosDA cryptographic protocol to carry out their respective functions. These functions are implemented as abstracted and configurable software entities that allow the original data to be encoded and verified via high-level operations. -Glossary -NameDescriptionRepresentation -CommitmentCommitment as per the NomosDA Cryptographic Protocolbytes -ProofProof as per the NomosDA Cryptographic Protocolbytes -ChunksMatrixMatrix of chunked data. Each chunk is 31 bytes. Row and Column sizes depend on the encoding necessities.List[List[bytes]] - - -Encoder -An encoder takes a set of parameters and the blob data, and creates a matrix of chunks that it uses to compute the necessary cryptographic data. It produces the set of Reed-Solomon (RS) encoded data, the commitments, and the proofs that are needed prior to dispersal. -flowchart LR - A[DaEncoderParams] -->|Input| B(Encoder) - I[31bytes-padded-input] -->|Input| B - B -->|Creates| D[Chunks matrix] - D --> |Input| C[NomosDA encoding] - C --> E{Encoded data📄} - -Encoding Process -The encoder executes the encoding process as follows: - - -The encoder takes the following input parameters: -class DAEncoderParams: - column_count: usize - bytes_per_field_element: usize - -NameDescriptionRepresentation -column_countThe number of subnets available for dispersal in the systemusize, int in Python -bytes_per_field_elementThe amount of bytes per data chunk. This is set to 31 bytes. Each chunk has 31 bytes rather than 32 to ensure that the chunk value does not exceed the maximum value on the BLS12-381 elliptic curve.usize, int in Python - - - -The encoder also includes the blob data to be encoded, which must be of a size that is a multiple of bytes_per_field_element bytes. Clients are responsible for padding the data so it fits this constraint. - - -The encoder splits the data into bytes_per_field_element-sized chunks. It also arranges these chunks into rows and columns, creating a matrix. -a. The amount of columns of the matrix needs to fit with the column_count parameter, taking into account the rs_expansion_factor (currently fixed to 2). -i. This means that the size of each row in this matrix is (bytes_per_field_element*column_count)/rs_expansion_factor. -b. The amount of rows depends on the size of the data. - - -The data is encoded as per the cryptographic details. - - -The encoder provides the encoded data set: -NameDescriptionRepresentation -dataOriginal databytes -chunked_dataMatrix before RS expansionChunksMatrix -extended_matrixMatrix after RS expansionChunksMatrix -row_commitmentsCommitments for each matrix rowList[Commitment] -combined_column_proofsProofs for each matrix columnList[Proof] - - -class EncodedData: - data: bytes - chunked_data: ChunksMatrix - extended_matrix: ChunksMatrix - row_commitments: List[Commitment] - combined_column_proofs: List[Proof] - - - -Encoder Limits -NomosDA does not impose a fixed limit on blob size at the encoding level. However, protocols that involve resource-intensive operations must include upper bounds to prevent abuse. In the case of NomosDA, blob size limits are expected to be enforced, as part of the protocol's broader responsibility for resource management and fairness. -Larger blobs naturally result in higher computational and bandwidth costs, particularly for the encoder, who must compute a proof for each column. Without size limits, malicious clients could exploit the system by attempting to stream unbounded data to DA nodes. Since payment is provided before blob dispersal, DA nodes are protected from performing unnecessary work. This enables the protocol to safely accept very large blobs, as the primary computational cost falls on the encoder. The protocol can accommodate generous blob sizes in practice, while rejecting only absurdly large blobs, such as those exceeding 1 GB, to prevent denial-of-service attacks and ensure network stability. -To mitigate this, the protocol define acceptable blob size limits, and DA implementations enforce local mitigation strategies, such as flagging or blacklisting clients that violate these constraints. -Verifier -A verifier checks the proper encoding of data blobs it receives. A verifier executes the verification process as follows: - - -The verifier receives a DAShare with the required verification data: -NameDescriptionRepresentation -columnColumn chunks (31 bytes) from the encoded matrixList[bytes] -column_idxColumn id (0..2047). It is directly related to the subnetworks in the network specification.u16, unsigned int of 16 bits. int in Python -combined_column_proofProof of the random linear combination of the column elements.Proof -row_commitmentsCommitments for each matrix rowList[Commitment] -blob_idThis is computed as the hash (blake2b) of row_commitmentsbytes - - - -Upon receiving the above data it verifies the column data as per the cryptographic details. If the verification is successful, the node triggers the replication protocol and stores the blob. -class DAShare: - column: Column - column_idx: u16 - combined_column_proof: Proof - row_commitments: List[Commitment] - - def blob_id(self) -> BlobId: - hasher = blake2b(digest_size=32) - for c in self.row_commitments: - hasher.update(bytes(c)) - return hasher.digest() - - - -Verification Logic -sequenceDiagram - participant N as Node - participant S as Subnetwork Column N - loop For each incoming blob column - N-->>N: If blob is valid - N-->>S: Replication - N->>N: Stores blob - end - -Details -The encoder and verifier processes described above make use of a variety of cryptographic functions to facilitate the correct verification of column data by verifiers. These functions rely on primitives such as polynomial commitments and Reed-Solomon erasure codes, the details of which are outside the scope of this document. These details, as well as introductions to the cryptographic primitives being used, can be found in the NomosDA Cryptographic Protocol: -NomosDA Cryptographic Protocol -References - -Encoder Specification: GitHub/encoder.py -Verifier Specification: GitHub/verifier.py -Cryptographic protocol: NomosDA Cryptographic Protocol - -Copyright -Copyright and related rights waived via CC0. - -title: NOMOS-DA-NETWORK -name: NomosDA Network -status: raw -category: -tags: network, data-availability, da-nodes, executors, sampling -editor: Daniel Sanchez Quiros danielsq@status.im -contributors: - -Álvaro Castro-Castilla alvaro@status.im -Daniel Kashepava danielkashepava@status.im -Gusto Bacvinka augustinas@status.im -Filip Dimitrijevic filip@status.im - - -Introduction NomosDA is the scalability solution protocol for data availability within the Nomos network. This document delineates the protocol's structure at the network level, identifies participants, @@ -19338,7 +18277,7 @@ and the data is split into a fixed number of portions (see the Overview +Overview Network Participants The NomosDA network includes three categories of participants: @@ -19368,7 +18307,7 @@ and replicated data. Indexing: Tracks and exposes blob metadata on-chain. NomosDA Indexing -Construction +Construction NomosDA Network Registration Entities wishing to participate in NomosDA must declare their role via SDP (Service Declaration Protocol). Once declared, they're accounted for in the subnetwork construction. @@ -19466,7 +18405,7 @@ subgraph Dispersal Executor -->|Disperse| N10 end
messages
HistoryResponse
pubsubtopic
contentFilters
pubsubto scalable infrastructure for developers creating applications for the network state. Published Specifications are currently available here, Nomos Specifications. - -title: NOMOSDA-ENCODING -name: NomosDA Encoding Protocol -status: raw -category: -tags: data-availability -editor: Daniel Sanchez-Quiros danielsq@status.im -contributors: +Nomos Raw Specifications +Early-stage Nomos specifications that have not yet progressed beyond raw status. +NOMOSDA-ENCODING + + +NameNomosDA Encoding Protocol +Statusraw +EditorDaniel Sanchez-Quiros <danielsq@status.im> +ContributorsDaniel Kashepava <danielkashepava@status.im>Álvaro Castro-Castilla <alvaro@status.im>Filip Dimitrijevic <filip@status.im>Thomas Lavaur <thomaslavaur@status.im>Mehmet Gonen <mehmet@status.im> + + + +Timeline -Daniel Kashepava danielkashepava@status.im -Álvaro Castro-Castilla alvaro@status.im -Filip Dimitrijevic filip@status.im -Thomas Lavaur thomaslavaur@status.im -Mehmet Gonen mehmet@status.im +2025-12-22 — b1a5783 — Chore/mdbook updates (#237) +2025-12-18 — d03e699 — ci: add mdBook configuration (#233) +2025-09-25 — 39d6f07 — added nomos/raw/nomosda-encoding.md draft (#156) - + Introduction This document describes the encoding and verification processes of NomosDA, which is the data availability (DA) solution used by the Nomos blockchain. NomosDA provides an assurance that all data from Nomos blobs are accessible and verifiable by every network participant. This document presents an implementation specification describing how: @@ -18997,7 +18087,7 @@ contributors: Encoders encode blobs they want to upload to the Data Availability layer. Other nodes implement the verification of blobs that were already uploaded to DA.
Encoder: An encoder is any actor who performs the encoding process described in this document. This involves committing to the data, generating proofs, and submitting the result to the DA layer.
The encoder and verifier processes described above make use of a variety of cryptographic functions to facilitate the correct verification of column data by verifiers. These functions rely on primitives such as polynomial commitments and Reed-Solomon erasure codes, the details of which are outside the scope of this document. These details, as well as introductions to the cryptographic primitives being used, can be found in the NomosDA Cryptographic Protocol:
NomosDA Cryptographic Protocol
In the Nomos architecture, the rollup sequencer typically acts as the encoder, but the role is not exclusive and any actor in the DA layer can also act as encoders.
Verifier: Verifies its portion of the distributed blob data as per the verification protocol. In the Nomos architecture, the DA nodes act as the verifiers.
In the encoding stage, the encoder takes the DA parameters and the padded blob data and creates an initial matrix of data chunks. This matrix is expanded using Reed-Solomon coding and various commitments and proofs are created for the data.
When a verifier receives a sample, it verifies the data it receives from the encoder and broadcasts the information if the data is verified. Finally, the verifier stores the sample data for the required length of time.
The encoder and verifier use the NomosDA cryptographic protocol to carry out their respective functions. These functions are implemented as abstracted and configurable software entities that allow the original data to be encoded and verified via high-level operations.
Commitment
bytes
Proof
ChunksMatrix
List[List[bytes]]
An encoder takes a set of parameters and the blob data, and creates a matrix of chunks that it uses to compute the necessary cryptographic data. It produces the set of Reed-Solomon (RS) encoded data, the commitments, and the proofs that are needed prior to dispersal.
flowchart LR - A[DaEncoderParams] -->|Input| B(Encoder) - I[31bytes-padded-input] -->|Input| B - B -->|Creates| D[Chunks matrix] - D --> |Input| C[NomosDA encoding] - C --> E{Encoded data📄} -
The encoder executes the encoding process as follows:
The encoder takes the following input parameters:
class DAEncoderParams: - column_count: usize - bytes_per_field_element: usize -
column_count
usize
int
bytes_per_field_element
The encoder also includes the blob data to be encoded, which must be of a size that is a multiple of bytes_per_field_element bytes. Clients are responsible for padding the data so it fits this constraint.
The encoder splits the data into bytes_per_field_element-sized chunks. It also arranges these chunks into rows and columns, creating a matrix. -a. The amount of columns of the matrix needs to fit with the column_count parameter, taking into account the rs_expansion_factor (currently fixed to 2). -i. This means that the size of each row in this matrix is (bytes_per_field_element*column_count)/rs_expansion_factor. -b. The amount of rows depends on the size of the data.
rs_expansion_factor
(bytes_per_field_element*column_count)/rs_expansion_factor
The data is encoded as per the cryptographic details.
The encoder provides the encoded data set:
chunked_data
extended_matrix
row_commitments
List[Commitment]
combined_column_proofs
List[Proof]
class EncodedData: - data: bytes - chunked_data: ChunksMatrix - extended_matrix: ChunksMatrix - row_commitments: List[Commitment] - combined_column_proofs: List[Proof] -
NomosDA does not impose a fixed limit on blob size at the encoding level. However, protocols that involve resource-intensive operations must include upper bounds to prevent abuse. In the case of NomosDA, blob size limits are expected to be enforced, as part of the protocol's broader responsibility for resource management and fairness.
Larger blobs naturally result in higher computational and bandwidth costs, particularly for the encoder, who must compute a proof for each column. Without size limits, malicious clients could exploit the system by attempting to stream unbounded data to DA nodes. Since payment is provided before blob dispersal, DA nodes are protected from performing unnecessary work. This enables the protocol to safely accept very large blobs, as the primary computational cost falls on the encoder. The protocol can accommodate generous blob sizes in practice, while rejecting only absurdly large blobs, such as those exceeding 1 GB, to prevent denial-of-service attacks and ensure network stability.
To mitigate this, the protocol define acceptable blob size limits, and DA implementations enforce local mitigation strategies, such as flagging or blacklisting clients that violate these constraints.
A verifier checks the proper encoding of data blobs it receives. A verifier executes the verification process as follows:
The verifier receives a DAShare with the required verification data:
DAShare
column
List[bytes]
column_idx
0..2047
subnetworks
u16
combined_column_proof
blob_id
Upon receiving the above data it verifies the column data as per the cryptographic details. If the verification is successful, the node triggers the replication protocol and stores the blob.
class DAShare: - column: Column - column_idx: u16 - combined_column_proof: Proof - row_commitments: List[Commitment] - - def blob_id(self) -> BlobId: - hasher = blake2b(digest_size=32) - for c in self.row_commitments: - hasher.update(bytes(c)) - return hasher.digest() -
sequenceDiagram - participant N as Node - participant S as Subnetwork Column N - loop For each incoming blob column - N-->>N: If blob is valid - N-->>S: Replication - N->>N: Stores blob - end -
NomosDA is the scalability solution protocol for data availability within the Nomos network. This document delineates the protocol's structure at the network level, identifies participants, @@ -19338,7 +18277,7 @@ and the data is split into a fixed number of portions (see the Overview +
The NomosDA network includes three categories of participants:
Entities wishing to participate in NomosDA must declare their role via SDP (Service Declaration Protocol). Once declared, they're accounted for in the subnetwork construction.
The NomosDA network is engineered for connection efficiency. Executors manage numerous open connections, @@ -19485,7 +18424,7 @@ triggering the specific protocol once established:
The Nomos network is designed to be inclusive and accessible across a wide range of hardware configurations. By defining clear hardware requirements for different node types, we enable:
Important Notice: These hardware requirements are preliminary and subject to revision based on implementation testing and real-world network performance data.
Hardware requirements vary based on the node's role and services:
Light Nodes provide network verification with minimal resource requirements, suitable for resource-constrained environments.
Target Use Cases:
Network Address Translation presents a critical challenge for Nomos participants, particularly those operating on consumer hardware without technical expertise. The Nomos network requires a NAT traversal solution that:
The solution must ensure that nodes can both establish outbound connections and accept inbound connections from other peers, maintaining network connectivity across diverse NAT configurations.
The Nomos P2P network bootstrapping strategy relies on a designated subset of bootstrap nodes to facilitate secure and efficient node onboarding. These nodes serve as the initial entry points for new network participants.
Bootstrap nodes operate under a minimal trust model:
The Nomos blockchain requires a reliable, scalable P2P network that can:
The Nomos P2P network addresses three critical challenges:
Original working document, from Nomos Notion: P2P Network Specification.
The requirements for the protocol are defined as follows:
The SDP enables nodes to declare their eligibility to serve a specific service in the system, and withdraw their declarations.
The protocol defines the following actions:
💡 The protocol messages are subject to a finality that means messages become part of the immutable ledger after a delay. The delay at which it happens is defined by the consensus.
In this section, we present the main constructions of the protocol. First, we start with data definitions. Second, we describe the protocol actions. Finally, we present part of the Bedrock Mantle design responsible for storing and processing SDP-related messages and data.
In this section, we discuss and define data types, messages, and their storage.
Refer to the Mantle Specification for a list of potential improvements to the protocol.
This document specifies Claro: a Byzantine, fault-tolerant, binary decision agreement algorithm that utilizes bounded memory for its execution. Claro is a novel variant of the Snow family providing a probabilistic @@ -20736,7 +19694,7 @@ in order to disambiguate from a previously released research endeavor by Amores-Sesar, Cachin, and Tedeschi. Their naming was coincidentally named the same as our work but is sufficiently differentiated from how ours works.
This work is a part of a larger research endeavor to explore highly scalable Byzantine Fault Tolerant (BFT) consensus protocols. Consensus lies at the heart of many decentralized protocols, and @@ -21271,8 +20229,8 @@ three possible values of the opinion:
the parity summations across network invariants often become easier to manipulate.
In practice, each honest node gossips its current opinion which reduces the number of messages that need to be gossiped for a given proposal. The resulting impact on the privacy of the node's opinion @@ -21371,695 +20329,34 @@ Move: a Language for Writing DAG Abstractions
json-ld
Copyright and related rights waived via -CC0
This document specifies Claro: a Byzantine, fault-tolerant, binary decision -agreement algorithm that utilizes bounded memory for its execution. -Claro is a novel variant of the Snow family providing a probabilistic -leaderless BFT consensus algorithm that achieves metastablity via -network sub-sampling. We present an application context of the use of -Claro in an efficient, leaderless, probabilistic permission-less -consensus mechanism. We outline a simple taxonomy of Byzantine -adversaries, leaving explicit explorations of to subsequent -publication.
NOTE: We have renamed this variant to Claro from Glacier -in order to disambiguate from a previously released research endeavor by -Amores-Sesar, Cachin, and Tedeschi. -Their naming was coincidentally named the same as our work but -is sufficiently differentiated from how ours works.
Claro
Glacier
This work is a part of a larger research endeavor to -explore highly scalable Byzantine Fault Tolerant (BFT) consensus protocols. -Consensus lies at the heart of many decentralized protocols, and -thus its characteristics and properties are inherited by applications built on top. -Thus, we seek to improve upon the current state of the art in two main directions: -base-layer scalability and censorship resistance.
Avalanche has shown to exibit the former in a production environment in a way -that is differentiated from Nakamoto consensus and -other Proof of Stake (PoS) protocols based in practical Byzantine Fault Tolerant -(pBFT) methodologies. -We aim to understand its limitations and improve upon them.
Our starting point is Avalanche’s Binary Byzantine Agreement algorithm, -called Snowball. -As long as modifications allow a DAG to be constructed later on, -this simplifies the design significantly. -The DAG stays the same in principle: it supports confidence, -but the core algorithm can be modeled without.
The concept of the Snowball algorithm is relatively simple. -Following is a simplified description (lacking some details, but giving an overview). -For further details, please refer to the Avalanche paper.
alpha
beta
Next, we will proceed to describe our new algorithm, based on Snowball.
We have identified a shortcoming of the Snowball algorithm -that was a perfect starting point for devising improvements. -The scenario is as follows:
This document only outlines the specification to Claro. -Subsequent analysis work on Claro -(both on its performance and how it differentiates with Snowball) -will be published shortly and this document will be updated.
The Claro consensus algorithm computes a boolean decision on a -proposition via a set of distributed computational nodes. Claro is -a leaderless, probabilistic, binary consensus algorithm with fast -finality that provides good reliability for network and Byzantine -fault tolerance.
Claro is an evolution of the Snowball Byzantine Binary Agreement (BBA) algorithm, -in which we tackle specifically the perceived weakness described above. -The main focus is going to be the counter and the triggering of the reset. -Following, we elaborate the different modifications and -features that have been added to the reference algorithm:
It’s worth delving a bit into the way the data is interpreted -in order to reach a decision. -Our approach is based conceptually on the paper Confidence as Higher-Order Uncertainty, -which describes a frequentist approach to decision certainty. -The first-order certainty, measured by frequency, -is caused by known positive evidence, and -the higher-order certainty is caused by potential positive evidence. -Because confidence is a relative measurement defined on evidence, -it naturally follows comparing the amount of evidence the system knows -with the amount that it will know in the near future (defining “near” as a constant).
Intuitively, we are looking for a function of evidence, w, -call it c for confidence, that satisfies the following conditions:
w
c
w = 0
c = 0
The paper describes also a set of operations for the evidence/confidence pairs, -so that different sources of knowledge could be combined. -However, we leave here the suggestion of a possible research line in the future -combining an algebra of evidence/confidence pairs with -swarm-propagation algorithm like the one described in -this paper.
A proposal is formulated to which consensus of truth or falsity is -desired. Each node that participates starts the protocol with an -opinion on the proposal, represented in the sequel as NO, NONE, -and YES.
NO
NONE
YES
A new proposition is discovered either by local creation or in -response to a query, a node checks its local opinion. If the node can -compute a justification of the proposal, it sets its opinion to one of -YES or NO. If it cannot form an opinion, it leaves its opinion as -NONE.
For now, we will ignore the proposal dissemination process and -assume all nodes participating have an initial opinion -to respond to within a given request. -Further research will relax this assumption and -analyze timing attacks on proposal propagation through the network.
The node then participates in a number of query rounds in which it -solicits other node's opinion in query rounds. Given a set of N -leaderless computational nodes, a gossip-based protocol is presumed to -exist which allows members to discover, join, and leave a weakly -transitory maximally connected graph. Joining this graph allows each -node to view a possibly incomplete node membership list of all other -nodes. This view may change as the protocol advances, as nodes join -and leave. Under generalized Internet conditions, the membership of -the graph would experience a churn rate varying across different -time-scales, as the protocol rounds progress. As such, a given node -may not have a view on the complete members participating in the -consensus on a proposal in a given round.
N
The algorithm is divided into 4 phases:
confidence
evidence
accumulated evidence
The node initializes the following integer ratios as constants:
# The following values are constants chosen with justification from experiments -# performed with the adversarial models - -# -confidence_threshold - <-- 1 - -# constant look ahead for number of rounds we expect to finalize a -# decision. Could be set dependent on number of nodes -# visible in the current gossip graph. -look_ahead - <-- 19 - -# the confidence weighting parameter (aka alpha_1) -certainty - <-- 4 / 5 -doubt ;; the lack of confidence weighting parameter (aka alpha_2) - <-- 2 / 5 - -k_multiplier ;; neighbor threshold multiplier - <-- 2 - -;;; maximal threshold multiplier, i.e. we will never exceed -;;; questioning k_initial * k_multiplier ^ max_k_multiplier_power peers -max_k_multiplier_power - <-- 4 - -;;; Initial number of nodes queried in a round -k_initial - <-- 7 - -;;; maximum query rounds before termination -max_rounds ;; placeholder for simulation work, no justification yet - <-- 100 -
The following variables are needed to keep the state of Claro:
;; current number of nodes to attempt to query in a round -k - <-- k_original - -;; total number of votes examined over all rounds -total_votes - <-- 0 -;; total number of YES (i.e. positive) votes for the truth of the proposal -total_positive - <-- 0 -;; the current query round, an integer starting from zero -round - <-- 0 -
A node selects k nodes randomly from the complete pool of peers in the -network. This query is can optionally be weighted, so the probability -of selecting nodes is proportional to their
k
Node Weighting -$$ -P(i) = \frac{w_i}{\sum_{j=0}^{j=N} w_j} -$$
where w is evidence. -The list of nodes is maintained by a separate protocol (the network layer), -and eventual consistency of this knowledge in the network -suffices. Even if there are slight divergences in the network view -from different nodes, the algorithm is resilient to those.
A query is sent to each neighbor with the node's current opinion of -the proposal.
opinion
Each node replies with their current opinion on the proposal.
See the wire protocol Interoperability section for -details on the semantics and syntax of the "on the wire" -representation of this query.
Adaptive querying. An additional optimization in the query -consists of adaptively growing the k constant in the event of -high confusion. We define high confusion as the situation in -which neither opinion is strongly held in a query (i.e. a -threshold is not reached for either yes or no). For this, we will -use the alpha threshold defined below. This adaptive growth of -the query size is done as follows:
Every time the threshold is not reached, we multiply k by a -constant. In our experiments, we found that a constant of 2 works -well, but what really matters is that it stays within that order of -magnitude.
The growth is capped at 4 times the initial k value. Again, this -is an experimental value, and could potentially be increased. This -depends mainly on complex factors such as the size of the query -messages, which could saturate the node bandwidth if the number of -nodes queried is too high.
When the query finishes, the node now initializes the following two -values:
new_votes - <-- |total vote replies received in this round to the current query| - positive_votes - <-- |YES votes received from the query| -
When the query returns, three ratios are used later on to compute the -transition function and the opinion forming. Confidence encapsulates -the notion of how much we know (as a node) in relation to how much we -will know in the near future (this being encoded in the look-ahead -parameter l.) Evidence accumulated keeps the ratio of total positive -votes vs the total votes received (positive and negative), whereas the -evidence per round stores the ratio of the current round only.
l
Parameters -$$ -\begin{array}{lc} -\text{Look-ahead parameter} & l = 20 \newline -\text{First evidence parameter} & \alpha_1 = 0.8 \newline -\text{Second evidence parameter} & \alpha_2 = 0.5 \newline -\end{array} -$$
Computation -$$ -\begin{array}{lc} -\text{Confidence} & c_{accum} \impliedby \frac{total\ votes} -{total\ votes + l} \newline -\text{Total accumulated evidence}& e_{accum} \impliedby \frac{total\ positive -votes}{total\ votes} \newline -\text{Evidence per round} & e_{round} \impliedby \frac{round\ positive -votes}{round\ votes} \newline -\end{array} -$$
The node runs the new_votes and positive_votes parameters received -in the query round through the following algorithm:
new_votes
positive_votes
- total_votes - +== new_votes - total_positive - +== positive_votes - confidence - <-- total_votes / (total_votes + look_ahead) - total_evidence - <-- total_positive / total_votes - new_evidence - <-- positive_votes / new_votes - evidence - <-- new_evidence * ( 1 - confidence ) + total_evidence * confidence - alpha - <-- doubt * ( 1 - confidence ) + certainty * confidence -
In order to eliminate the need for a step function (a conditional in -the code), we introduce a transition function from one regime to the -other. Our interest in removing the step function is twofold:
Simplify the algorithm. With this change the number of branches is -reduced, and everything is expressed as a set of equations.
The transition function makes the regime switch smooth, -making it harder to potentially exploit the sudden regime change in -some unforeseen manner. Such a swift change in operation mode could -potentially result in a more complex behavior than initially -understood, opening the door to elaborated attacks. The transition -function proposed is linear with respect to the confidence.
Transition Function -$$ -\begin{array}{cl} -evidence & \impliedby e_{round} (1 - c_{accum}) + e_{accum} c_{accum} \newline -\alpha & \impliedby \alpha_1 (1 - c_{accum}) + \alpha_2 c_{accum} \newline -\end{array} -$$
Since the confidence is modeled as a ratio that depends on the -constant l, we can visualize the transition function at -different values of l. Recall that this constant encapsulates -the idea of “near future” in the frequentist certainty model: the -higher it is, the more distant in time we consider the next -valuable input of evidence to happen.
We have observed via experiment that for a transition function to be -useful, we need establish two requirements:
The change has to be balanced and smooth, giving an -opportunity to the first regime to operate and not jump directly -to the second regime.
The convergence to 1.0 (fully operating in the second regime) -should happen within a reasonable time-frame. We’ve set this -time-frame experimentally at 1000 votes, which is in the order of -~100 queries given a k of 9.
[[ Note: Avalanche uses k = 20, as an experimental result from their -deployment. Due to the fundamental similarities between the -algorithms, it’s a good start for us. ]]
The node updates its local opinion on the consensus proposal by -examining the relationship between the evidence accumulated for a -proposal with the confidence encoded in the alpha parameter:
IF - evidence > alpha - THEN - opinion <-- YES - ELSE IF - evidence < 1 - alpha - THEN - opinion <-- NO -
If the opinion of the node is NONE after evaluating the relation -between evidence and alpha, adjust the number of uniform randomly -queried nodes by multiplying the neighbors k by the k_multiplier -up to the limit of k_max_multiplier_power query size increases.
k_multiplier
k_max_multiplier_power
- ;; possibly increase number nodes to uniformly randomly query in next round - WHEN - opinion is NONE - AND - k < k_original * k_multiplier ^ max_k_multiplier_power - THEN - k <-- k * k_multiplier -
The next step is a simple one: change our opinion if the threshold -alpha is reached. This needs to be done separately for the YES/NO -decision, checking both boundaries. The last step is then to decide -on the current opinion. For that, a confidence threshold is -employed. This threshold is derived from the network size, and is -directly related to the number of total votes received.
YES/NO
decide
Decision -$$ -\begin{array}{cl} -evidence > \alpha & \implies \text{opinion YES} \newline -evidence < 1 - \alpha & \implies \text{opinion NO} \newline -if\ \text{confidence} > c_{target} & THEN \ \text{finalize decision} \newline -\end{array} -$$
After the OPINION phase is executed, the current value of confidence -is considered: if confidence exceeds a threshold derived from the -network size and directly related to the total votes received, an -honest node marks the decision as final, and always returns this -opinion is response to further queries from other nodes on the -network.
OPINION
- IF - confidence > confidence_threshold - OR - round > max_rounds - THEN - finalized <-- T - QUERY LOOP TERMINATES - ELSE - round +== 1 - QUERY LOOP CONTINUES -
Thus, after the decision phase, either a decision has been finalized -and the local node becomes quiescent never initiating a new query, or -it initiates a new query.
A local round of Claro terminates in one of the following -execution model considerations:
No queries are received for any newly initiated round for temporal -periods observed via a locally computed passage of time. See the -following point on local time.
The confidence on the proposal exceeds our threshold for -finalization.
The number of rounds executed would be greater than -max_rounds.
rounds
max_rounds
After a local node has finalized an opinion into a decision, it enters a quiescent -state whereby it never solicits new votes on the proposal. The local -node MUST reply with the currently finalized decision.
decision
The algorithm only requires that nodes have computed the drift of -observation of the passage of local time, not that that they have -coordinated an absolute time with their peers. For an implementation -of a phase locked-loop feedback to measure local clock drift see -NTP.
In the query step, the node is envisioned as packing information into -the query to cut down on the communication overhead a query to each of -this k nodes containing the node's own current opinion on the -proposal (YES, NO, or NONE). The algorithm does not currently -specify how a given node utilizes this incoming information. A -possible use may be to count unsolicited votes towards a currently -active round, and discard the information if the node is in a -quiescent state.
If the view of other nodes is incomplete, then the sum of the optional -weighting will not be a probability distribution normalized to 1.
The current algorithm doesn't describe how the initial opinions are formed.
The following implementations have been created for various testing and -simulation purposes:
For interoperability we present a wire protocol semantics by requiring -the validity of the following statements expressed in Notation3 (aka -n3) about any query performed by a query node:
n3
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . -@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . -@prefix xsd: <http://www.w3.org/2001/XMLSchema#> . - -@prefix Claro <https://rdf.logos.co/protocol/Claro#> . - -Claro:query - :holds ( - :_0 [ rdfs:label "round"; - a xsd:postitiveInteger; ], - rdfs:comment """ -The current round of this query - -A value of zero corresponds to the initial round. -""" ; - - :_1 [ rdfs:label "uri"; - rdfs:comment """ -A unique URI for the proposal. - -It MAY be possible to examine the proposal by resolving this resource, -and its associated URIs. -""" ; - a xsd:anyURI ], - - :_2 [ rdfs:label "opinion"; - rdfs:comment """ -The opinion on the proposal - -One of the strings "YES" "NO" or "NONE". -""" ; - # TODO constrain as an enumeration on three values efficiently - a xsd:string ] - ) . -
Nodes are advised to use Waku messages to include their own -metadata in serializations as needed.
The semantic description presented above can be reliably round-tripped -through a suitable serialization mechanism. JSON-LD provides a -canonical mapping to UTF-8 JSON.
At their core, the query messages are a simple enumeration of the -three possible values of the opinion:
-{ NO, NONE, YES } -
{ NO, NONE, YES }
When represented via integers, such as choosing
-{ -1, 0, +1 } -
{ -1, 0, +1 }
the parity summations across network invariants often become easier to -manipulate.
In practice, each honest node gossips its current opinion which -reduces the number of messages that need to be gossiped for a given -proposal. The resulting impact on the privacy of the node's opinion -is not currently analyzed.
Adversarial models have been tested for which the values for current -parameters of Claro have been tuned. Exposition of the -justification of this tuning need to be completed.
A random adversary optionally chooses to respond to all queries with a -random decision. Note that this adversary may be in some sense -Byzantine but not malicious. The random adversary also models some -software defects involved in not "understanding" how to derive a truth -value for a given proposition.
Like a petulant child, an infantile adversary responds with the -opposite vote of the honest majority on an opinion.
Omniscient adversaries have somehow gained an "unfair" participation in -consensus by being able to control f of N nodes with a out-of-band -"supra-liminal" coordination mechanism. Such adversaries use this -coordinated behavior to delay or sway honest majority consensus.
f
The passive network omniscient adversary is fully aware at all times -of the network state. Such an adversary can always chose to vote in -the most efficient way to block the distributed consensus from -finalizing.
An omniscient gossip adversary somehow not only controls f of N -nodes, but has also has corrupted communications between nodes such -that she may inspect, delay, and drop arbitrary messages. Such an -adversary uses capability to corrupt consensus away from honest -decisions to ones favorable to itself. This adversary will, of -course, choose to participate in an honest manner until defecting is -most advantageous.
Although we have proposed a normative description of the -implementation of the underlying binary consensus algorithm (Claro), -we believe we have prepared for analysis its adversarial performance -in a manner that is amenable to replacement by another member of the -snow* family.
We have presumed the existence of a general family of algorithms that -can be counted on to vote on nodes in the DAG in a fair manner. -Avalanche provides an example of the construction of votes on UTXO -transactions. One can express all state machine, i.e. account-based -models as checkpoints anchored in UTXO trust, so we believe that this -presupposition has some justification. We can envision a need for -tooling abstraction that allow one to just program the DAG itself, as -they should be of stable interest no matter if Claro isn't.
Logos
On BFT Consensus Evolution: From Monolithic to -DAG
snow-ipfs
snow* The Snow family of -algorithms
Move -Move: a Language for Writing DAG Abstractions
rdf
rdfs
xsd
n3-w3c-notes
ntp
Copyright and related rights waived via CC0
Specifications related the Codex decentralised data storage platform. Visit Codex specs to view the new Codex specifications currently under discussion.
Sections marked with notes indicate areas where implementation details may differ from this specification.
The Block Exchange (BE) is a core Codex component responsible for peer-to-peer content distribution across the network. It manages the sending and receiving of data blocks between nodes, @@ -22083,7 +20380,7 @@ fixed-length chunks of arbitrary data.
The Block Exchange module serves as the fundamental layer for content distribution in the Codex network. It provides primitives for requesting and delivering blocks of data @@ -23431,7 +21728,7 @@ to callers only after system-level retries are exhausted
Protocol Buffers: Using Protocol Buffers provides efficient serialization, forward compatibility, and wide language support.
The Block Exchange (BE) is a core Codex component responsible for -peer-to-peer content distribution across the network. -It manages the sending and receiving of data blocks between nodes, -enabling efficient data sharing and retrieval. -This specification defines both an internal service interface and a -network protocol for referring to and providing data blocks. -Blocks are uniquely identifiable by means of an address and represent -fixed-length chunks of arbitrary data.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", -"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this -document are to be interpreted as described in -RFC 2119.
The Block Exchange module serves as the fundamental layer for content -distribution in the Codex network. -It provides primitives for requesting and delivering blocks of data -between peers, supporting both standalone blocks and blocks that are -part of larger datasets. -The protocol is designed to work over libp2p streams and integrates -with Codex's discovery, storage, and payment systems.
When a peer wishes to obtain a block, it registers its unique address -with the Block Exchange, and the Block Exchange will then be in charge -of procuring it by finding a peer that has the block, if any, and then -downloading it. -The Block Exchange will also accept requests from peers which might -want blocks that the node has, and provide them.
Discovery Separation: Throughout this specification we assume that -if a peer wants a block, then the peer has the means to locate and -connect to peers which either: (1) have the block; or (2) are -reasonably expected to obtain the block in the future. -In practical implementations, the Block Exchange will typically require -the support of an underlying discovery service, e.g., the Codex DHT, -to look up such peers, but this is beyond the scope of this document.
The protocol supports two distinct block types to accommodate different -use cases: standalone blocks for independent data chunks and dataset -blocks for ordered collections of data that form larger structures.
The Block Exchange protocol supports two types of blocks:
Standalone blocks are self-contained pieces of data addressed by their -SHA256 content identifier (CID). -These blocks are independent and do not reference any larger structure.
Properties:
Dataset blocks are part of ordered sets and are addressed by a -(datasetCID, index) tuple. -The datasetCID refers to the Merkle tree root of the entire dataset, -and the index indicates the block's position within that dataset.
(datasetCID, index)
Formally, we can define a block as a tuple consisting of raw data and -its content identifier: (data: seq[byte], cid: Cid), where standalone -blocks are addressed by cid, and dataset blocks can be addressed -either by cid or a (datasetCID, index) tuple.
(data: seq[byte], cid: Cid)
cid
(treeCID, index)
All blocks in the Codex Block Exchange protocol adhere to the -following specifications:
codex-block
sha2-256
Note: *The multicodec value 0xCD02 is not currently registered in the official multiformats multicodec table. This may be a reserved/private code pending official registration.
To ensure network stability and prevent resource exhaustion, implementations -SHOULD enforce reasonable limits. The following are recommended values -(actual implementation limits may vary):
Note: These values are not verified from implementation and serve as -reasonable guidelines. Actual implementations MAY use different limits based -on their resource constraints and deployment requirements.
Enforcement:
The Block Exchange module exposes two core primitives for -block management:
requestBlock
async def requestBlock(address: BlockAddress) -> Block -
Registers a block address for retrieval and returns the block data -when available. -This function can be awaited by the caller until the block is retrieved -from the network or local storage.
Parameters:
address
Returns:
Block
cancelRequest
async def cancelRequest(address: BlockAddress) -> bool -
Cancels a previously registered block request.
bool
The Block Exchange module depends on and interacts with several other -Codex components:
The Block Exchange protocol uses the following libp2p protocol -identifier:
/codex/blockexc/1.0.0 -
The protocol version is negotiated through libp2p's multistream-select -protocol during connection establishment. The following describes standard -libp2p version negotiation behavior; actual Codex implementation details -may vary.
Version Format: /codex/blockexc/<major>.<minor>.<patch>
/codex/blockexc/<major>.<minor>.<patch>
Current Version: 1.0.0
1.0.0
1. Initiator opens stream -2. Initiator proposes: "/codex/blockexc/1.0.0" -3. Responder checks supported versions -4. If supported: - Responder accepts: "/codex/blockexc/1.0.0" - → Connection established -5. If not supported: - Responder rejects with: "na" (not available) - → Try fallback version or close connection -
Major Version Compatibility:
1.x.x
2.x.x
Minor Version Compatibility:
1.1.0
Patch Version Compatibility:
Implementations MAY support multiple protocol versions simultaneously:
Supported protocols (in preference order): - 1. /codex/blockexc/1.2.0 (preferred, latest features) - 2. /codex/blockexc/1.1.0 (fallback, stable) - 3. /codex/blockexc/1.0.0 (legacy support) -
Negotiation Strategy:
For optional features within same major.minor version:
Method 1: Message field presence - - Send message with optional field - - Peer ignores if not supported (protobuf default) - -Method 2: Capability exchange (future extension) - - Exchange capability bitmask in initial message - - Enable features only if both peers support -
Backward Compatibility:
Forward Compatibility:
The protocol operates over libp2p streams. -When a node wants to communicate with a peer:
The protocol handles peer lifecycle events:
This section illustrates typical message exchange sequences for common -block exchange scenarios.
Scenario: Node A requests a standalone block from Node B
Node A Node B - | | - |--- Message(wantlist) ------------------>| - | wantlist.entries[0]: | - | address.cid = QmABC123 | - | wantType = wantBlock | - | priority = 0 | - | | - |<-- Message(blockPresences, payload) ----| - | blockPresences[0]: | - | address.cid = QmABC123 | - | type = presenceHave | - | payload[0]: | - | cid = QmABC123 | - | data = <64 KiB block data> | - | address.cid = QmABC123 | - | | -
Steps:
wantType = wantBlock
Scenario: Node A requests a dataset block from Node B
Node A Node B - | | - |--- Message(wantlist) ------------------>| - | wantlist.entries[0]: | - | address.leaf = true | - | address.treeCid = QmTree456 | - | address.index = 42 | - | wantType = wantBlock | - | | - |<-- Message(payload) ---------------------| - | payload[0]: | - | cid = QmBlock789 | - | data = <64 KiB zero-padded data> | - | address.leaf = true | - | address.treeCid = QmTree456 | - | address.index = 42 | - | proof = <CodexProof bytes> | - | | -
Scenario: Node A checks if Node B has a block without requesting full data
Node A Node B - | | - |--- Message(wantlist) ------------------>| - | wantlist.entries[0]: | - | address.cid = QmCheck999 | - | wantType = wantHave | - | sendDontHave = true | - | | - |<-- Message(blockPresences) -------------| - | blockPresences[0]: | - | address.cid = QmCheck999 | - | type = presenceHave | - | price = 0x00 (free) | - | | -
wantType = wantHave
Scenario: Node A requests block Node B doesn't have
Node A Node B - | | - |--- Message(wantlist) ------------------>| - | wantlist.entries[0]: | - | address.cid = QmMissing111 | - | wantType = wantBlock | - | sendDontHave = true | - | | - |<-- Message(blockPresences) -------------| - | blockPresences[0]: | - | address.cid = QmMissing111 | - | type = presenceDontHave | - | | -
sendDontHave = true
presenceDontHave
Scenario: Node A cancels a previous block request
Node A Node B - | | - |--- Message(wantlist) ------------------>| - | wantlist.entries[0]: | - | address.cid = QmCancel222 | - | cancel = true | - | | -
cancel = true
Scenario: Node A adds requests to existing WantList
Node A Node B - | | - |--- Message(wantlist) ------------------>| - | wantlist.full = false | - | wantlist.entries[0]: | - | address.cid = QmNew1 | - | wantType = wantBlock | - | wantlist.entries[1]: | - | address.cid = QmNew2 | - | wantType = wantBlock | - | | -
full = false
These diagrams illustrate the complete flow of block exchange operations -including service interface, peer discovery, and network protocol interactions.
The protocol supports two strategies for WantBlock requests, -each with different trade-offs. -Implementations may choose the strategy based on network conditions, -peer availability, and resource constraints.
In this strategy, the requester sends wantType = wantBlock to all -discovered peers simultaneously. -This minimizes latency as the first peer to respond with the block -data wins, but it wastes bandwidth since multiple peers may send -the same block data.
Trade-offs:
sequenceDiagram - participant Client - participant BlockExchange - participant LocalStore - participant Discovery - participant PeerA - participant PeerB - participant PeerC - - Client->>BlockExchange: requestBlock(address) - BlockExchange->>LocalStore: checkBlock(address) - LocalStore-->>BlockExchange: Not found - - BlockExchange->>Discovery: findPeers(address) - Discovery-->>BlockExchange: [PeerA, PeerB, PeerC] - - par Send wantBlock to all peers - BlockExchange->>PeerA: Message(wantlist: wantBlock) - BlockExchange->>PeerB: Message(wantlist: wantBlock) - BlockExchange->>PeerC: Message(wantlist: wantBlock) - end - - Note over PeerA,PeerC: All peers start preparing block data - - PeerB-->>BlockExchange: Message(payload: BlockDelivery) - Note over BlockExchange: First response wins - - BlockExchange->>BlockExchange: Verify block - BlockExchange->>LocalStore: Store block - - par Cancel requests to other peers - BlockExchange->>PeerA: Message(wantlist: cancel) - BlockExchange->>PeerC: Message(wantlist: cancel) - end - - Note over PeerA,PeerC: May have already sent data (wasted bandwidth) - - BlockExchange-->>Client: Return block -
In this strategy, the requester first sends wantType = wantHave to -discover which peers have the block, then sends wantType = wantBlock -only to a single selected peer. -This conserves bandwidth but adds an extra round-trip of latency.
sequenceDiagram - participant Client - participant BlockExchange - participant LocalStore - participant Discovery - participant PeerA - participant PeerB - participant PeerC - - Client->>BlockExchange: requestBlock(address) - BlockExchange->>LocalStore: checkBlock(address) - LocalStore-->>BlockExchange: Not found - - BlockExchange->>Discovery: findPeers(address) - Discovery-->>BlockExchange: [PeerA, PeerB, PeerC] - - Note over BlockExchange: Phase 1: Discovery - - par Send wantHave to all peers - BlockExchange->>PeerA: Message(wantlist: wantHave) - BlockExchange->>PeerB: Message(wantlist: wantHave) - BlockExchange->>PeerC: Message(wantlist: wantHave) - end - - PeerA-->>BlockExchange: BlockPresence(presenceDontHave) - PeerB-->>BlockExchange: BlockPresence(presenceHave, price=X) - PeerC-->>BlockExchange: BlockPresence(presenceHave, price=Y) - - BlockExchange->>BlockExchange: Select best peer (PeerB: lower price) - - Note over BlockExchange: Phase 2: Retrieval - - BlockExchange->>PeerB: Message(wantlist: wantBlock) - PeerB-->>BlockExchange: Message(payload: BlockDelivery) - - BlockExchange->>BlockExchange: Verify block - BlockExchange->>LocalStore: Store block - BlockExchange-->>Client: Return block -
Implementations MAY combine both strategies:
flowchart TD - A[Block Request] --> B{Block Size?} - B -->|Small < 64 KiB| C[Parallel Strategy] - B -->|Large >= 64 KiB| D{Paid Content?} - D -->|Yes| E[Two-Phase Discovery] - D -->|No| F{Network Condition?} - F -->|Reliable| E - F -->|Unreliable| C - C --> G[Return Block] - E --> G -
sequenceDiagram - participant Requester - participant Provider - participant Verifier - - Requester->>Provider: WantList(leaf=true, treeCid, index) - Provider->>Provider: Load block at index - Provider->>Provider: Generate CodexProof - Provider->>Requester: BlockDelivery(data, proof) - - Requester->>Verifier: Verify proof - - alt Proof valid - Verifier-->>Requester: Valid - Requester->>Requester: Verify CID - alt CID matches - Requester->>Requester: Store block - Requester-->>Requester: Success - else CID mismatch - Requester->>Requester: Reject block - Requester->>Provider: Disconnect - end - else Proof invalid - Verifier-->>Requester: Invalid - Requester->>Requester: Reject block - Requester->>Provider: Disconnect - end -
sequenceDiagram - participant Buyer - participant Seller - participant StateChannel - - Buyer->>Seller: Message(wantlist) - Seller->>Seller: Check block availability - Seller->>Buyer: BlockPresence(price) - - alt Buyer accepts price - Buyer->>StateChannel: Create update - StateChannel-->>Buyer: Signed state - Buyer->>Seller: Message(payment: StateChannelUpdate) - Seller->>StateChannel: Verify update - - alt Payment valid - StateChannel-->>Seller: Valid - Seller->>Buyer: BlockDelivery(data) - Buyer->>Buyer: Verify block - Buyer->>StateChannel: Finalize - else Payment invalid - StateChannel-->>Seller: Invalid - Seller->>Buyer: BlockPresence(price) - end - else Buyer rejects price - Buyer->>Seller: Message(wantlist.cancel) - end -
sequenceDiagram - participant Network - participant BlockExchange - participant PeerStore - participant Peer - - Network->>BlockExchange: PeerJoined(Peer) - BlockExchange->>PeerStore: AddPeer(Peer) - BlockExchange->>Peer: Open stream - - loop Active exchange - BlockExchange->>Peer: Message(wantlist/payload) - Peer->>BlockExchange: Message(payload/presence) - end - - alt Graceful disconnect - Peer->>BlockExchange: Close stream - BlockExchange->>PeerStore: RemovePeer(Peer) - else Connection failure - Network->>BlockExchange: PeerDropped(Peer) - BlockExchange->>PeerStore: RemovePeer(Peer) - BlockExchange->>BlockExchange: Requeue pending requests - end -
All messages use Protocol Buffers encoding for serialization. -The main message structure supports multiple operation types in a -single message.
message Message { - Wantlist wantlist = 1; - // Field 2 reserved for future use - repeated BlockDelivery payload = 3; - repeated BlockPresence blockPresences = 4; - int32 pendingBytes = 5; - AccountMessage account = 6; - StateChannelUpdate payment = 7; -} -
Fields:
wantlist
blockPresences
pendingBytes
account
payment
Note on Missing Field 2:
Field number 2 is intentionally skipped in the Message protobuf definition. -This is a common protobuf practice for several reasons:
The gap does not affect protocol operation. Protobuf field numbers need not be -sequential, and skipping numbers is standard practice for protocol evolution.
The BlockAddress structure supports both standalone and dataset -block addressing:
message BlockAddress { - bool leaf = 1; - bytes treeCid = 2; // Present when leaf = true - uint64 index = 3; // Present when leaf = true - bytes cid = 4; // Present when leaf = false -} -
leaf
treeCid
leaf = true
index
leaf = false
Addressing Modes:
The WantList communicates which blocks a peer desires to receive:
message Wantlist { - enum WantType { - wantBlock = 0; - wantHave = 1; - } - - message Entry { - BlockAddress address = 1; - int32 priority = 2; - bool cancel = 3; - WantType wantType = 4; - bool sendDontHave = 5; - } - - repeated Entry entries = 1; - bool full = 2; -} -
WantType Values:
wantBlock (0)
wantHave (1)
Entry Fields:
priority
cancel
wantType
sendDontHave
Priority Field Clarification:
The priority field is currently fixed at 0 in all implementations and is -reserved for future protocol extensions. Originally intended for request -prioritization, this feature is not yet implemented.
Current Behavior:
priority = 0
Future Extensions:
The priority field is reserved for:
Implementation Notes:
WantList Fields:
entries
full
Delta Updates:
WantLists support delta updates for efficiency. -When full = false, entries represent additions or modifications to -the existing WantList rather than a complete replacement.
Block deliveries contain the actual block data along with verification -information:
message BlockDelivery { - bytes cid = 1; - bytes data = 2; - BlockAddress address = 3; - bytes proof = 4; -} -
proof
Merkle Proof Verification:
When delivering dataset blocks (address.leaf = true):
address.leaf = true
Block presence messages indicate whether a peer has or does not have a -requested block:
enum BlockPresenceType { - presenceHave = 0; - presenceDontHave = 1; -} - -message BlockPresence { - BlockAddress address = 1; - BlockPresenceType type = 2; - bytes price = 3; -} -
type
price
UInt256 Price Format:
The price field encodes a 256-bit unsigned integer representing the cost in -wei (the smallest Ethereum denomination, where 1 ETH = 10^18 wei).
Encoding Specification:
0x0000000000000000000000000000000000000000000000000000000000000000
Free (0 wei): - 0x0000000000000000000000000000000000000000000000000000000000000000 - -1 wei: - 0x0000000000000000000000000000000000000000000000000000000000000001 - -1 gwei (10^9 wei): - 0x000000000000000000000000000000000000000000000000000000003b9aca00 - -0.001 ETH (10^15 wei): - 0x00000000000000000000000000000000000000000000000000038d7ea4c68000 - -1 ETH (10^18 wei): - 0x0000000000000000000000000000000000000000000000000de0b6b3a7640000 - -Maximum (2^256 - 1): - 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff -
Conversion Logic:
# Wei to bytes (big-endian) -def wei_to_bytes(amount_wei: int) -> bytes: - return amount_wei.to_bytes(32, byteorder='big') - -# Bytes to wei -def bytes_to_wei(price_bytes: bytes) -> int: - return int.from_bytes(price_bytes, byteorder='big') - -# ETH to wei to bytes -def eth_to_price_bytes(amount_eth: float) -> bytes: - amount_wei = int(amount_eth * 10**18) - return wei_to_bytes(amount_wei) -
Payment-related messages for micropayments using Nitro state channels.
Account Message:
message AccountMessage { - bytes address = 1; // Ethereum address to which payments should be made -} -
This section provides real-world examples of protobuf messages for different -block exchange scenarios.
Scenario: Request a single standalone block
Protobuf (wire format representation):
Message { - wantlist: Wantlist { - entries: [ - Entry { - address: BlockAddress { - leaf: false - cid: 0x0155a0e40220b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9 // CID bytes - } - priority: 0 - cancel: false - wantType: wantBlock // 0 - sendDontHave: true - } - ] - full: true - } -} -
Hex representation (sample):
0a2e 0a2c 0a24 0001 5512 2012 20b9 4d27 -b993 4d3e 08a5 2e52 d7da 7dab fac4 84ef -e37a 5380 ee90 88f7 ace2 efcd e910 0018 -0020 0028 011201 01 -
Scenario: Request block at index 100 from dataset
Protobuf:
Message { - wantlist: Wantlist { - entries: [ - Entry { - address: BlockAddress { - leaf: true - treeCid: 0x0155a0e40220c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 // Tree CID - index: 100 - } - priority: 0 - cancel: false - wantType: wantBlock - sendDontHave: true - } - ] - full: false // Delta update - } -} -
Scenario: Provider sends dataset block with Merkle proof
Message { - payload: [ - BlockDelivery { - cid: 0x0155a0e40220a1b2c3d4e5f6071829... // Block CID - data: <65536 bytes of block data> // 64 KiB - address: BlockAddress { - leaf: true - treeCid: 0x0155a0e40220c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 - index: 100 - } - proof: <CodexProof bytes> // Merkle proof data - // Contains: path indices, sibling hashes, tree height - // Format: Implementation-specific (e.g., [height][index][hash1][hash2]...[hashN]) - // Size varies by tree depth (illustrative: ~1KB for depth-10 tree) - } - ] -} -
Scenario: Provider indicates block availability with price
Message { - blockPresences: [ - BlockPresence { - address: BlockAddress { - leaf: false - cid: 0x0155a0e40220b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9 - } - type: presenceHave // 0 - price: 0x00000000000000000000000000000000000000000000000000038d7ea4c68000 // 0.001 ETH in wei - } - ] -} -
Scenario: Send payment via state channel update
Message { - account: AccountMessage { - address: 0x742d35Cc6634C0532925a3b844200a717C48D6d9 // 20 bytes Ethereum address - } - payment: StateChannelUpdate { - update: <JSON bytes> - // Contains signed Nitro state as UTF-8 JSON string - // Example: {"channelId":"0x1234...","nonce":42,...} - } -} -
Scenario: Combined WantList, BlockPresence, and Delivery
Message { - wantlist: Wantlist { - entries: [ - Entry { - address: BlockAddress { - leaf: false - cid: 0x0155a0e40220... // Requesting new block - } - wantType: wantBlock - priority: 0 - cancel: false - sendDontHave: true - } - ] - full: false - } - blockPresences: [ - BlockPresence { - address: BlockAddress { - leaf: false - cid: 0x0155a0e40220... // Response to previous request - } - type: presenceHave - price: 0x00 // Free - } - ] - payload: [ - BlockDelivery { - cid: 0x0155a0e40220... // Delivering another block - data: <65536 bytes> - address: BlockAddress { - leaf: false - cid: 0x0155a0e40220... - } - } - ] - pendingBytes: 131072 // 128 KiB more data pending -} -
Scenario: Cancel multiple pending requests
Message { - wantlist: Wantlist { - entries: [ - Entry { - address: BlockAddress { - leaf: false - cid: 0x0155a0e40220abc123... - } - cancel: true // Cancellation flag - }, - Entry { - address: BlockAddress { - leaf: true - treeCid: 0x0155a0e40220def456... - index: 50 - } - cancel: true - } - ] - full: false - } -} -
CID Structure:
CID v1 format (multibase + multicodec + multihash): -[0x01] [0x55] [0xa0] [0xe4] [0x02] [0x20] [<32 bytes SHA256 hash>] - │ │ │ │ │ │ │ - │ │ │ │ │ │ └─ Hash digest - │ │ │ │ │ └───────── Hash length (32) - │ │ │ │ └──────────────── Hash algorithm (SHA2-256) - │ │ │ └─────────────────────── Codec size - │ │ └────────────────────────────── Codec (raw = 0x55) - │ └───────────────────────────────────── Multicodec prefix - └──────────────────────────────────────────── CID version (1) - -Actual: 0x01 55 a0 e4 02 20 <hash bytes> -
Example Block CID Breakdown:
Full CID: 0x0155a0e40220b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9 - -Parts: - Version: 0x01 (CID v1) - Multicodec: 0x55 (raw) - Codec Size: 0xa0e402 (codex-block = 0xCD02, varint encoded)* - Hash Type: 0x20 (SHA2-256) - Hash Len: 0x12 (20) (32 bytes) - Hash: b94d27b993... (32 bytes SHA256) -
State Channel Update:
message StateChannelUpdate { - bytes update = 1; // Signed Nitro state, serialized as JSON -} -
update
The Block Exchange protocol integrates with Nitro state channels to enable -micropayments for block delivery.
When Payment is Required:
Free Blocks:
price = 0x00
Initial Price Advertisement:
Price Format:
price: bytes (32 bytes, big-endian UInt256) -Example: 0x0000000000000000000000000000000000000000000000000de0b6b3a7640000 - represents 1 ETH = 10^18 wei -
Requester → Provider: Message(wantlist: wantHave) -Provider → Requester: BlockPresence(type=presenceHave, price=<amount>) -
Requester evaluates price:
If accepted:
Requester: - 1. Load existing state channel with Provider - 2. Create new state with updated balance - 3. Sign state update - 4. Encode as JSON - -Requester → Provider: Message(payment: StateChannelUpdate(update=<signed JSON>)) -
Provider: - 1. Decode state channel update - 2. Verify signatures - 3. Check balance increase matches price - 4. Verify state channel validity - 5. Check nonce/sequence number - -If valid: - Provider → Requester: BlockDelivery(data, proof) -Else: - Provider → Requester: BlockPresence(price) // Retry with correct payment -
Requester: - 1. Receive and verify block - 2. Store block locally - 3. Finalize state channel update - 4. Update peer credit balance -
Note: The following state machine represents a design specification for -payment flow logic. Actual implementation may differ.
State: INIT - → Send wantHave - → Transition to PRICE_DISCOVERY - -State: PRICE_DISCOVERY - ← Receive BlockPresence(price) - → If price acceptable: Transition to PAYMENT_CREATION - → If price rejected: Transition to CANCELLED - -State: PAYMENT_CREATION - → Create state channel update - → Send payment message - → Transition to PAYMENT_PENDING - -State: PAYMENT_PENDING - ← Receive BlockDelivery: Transition to DELIVERY_VERIFICATION - ← Receive BlockPresence(price): Transition to PAYMENT_FAILED - -State: PAYMENT_FAILED - → Retry with corrected payment: Transition to PAYMENT_CREATION - → Abort: Transition to CANCELLED - -State: DELIVERY_VERIFICATION - → Verify block - → If valid: Transition to COMPLETED - → If invalid: Transition to DISPUTE - -State: COMPLETED - → Finalize state channel - → End - -State: CANCELLED - → Send cancellation - → End - -State: DISPUTE - → Reject block - → Dispute state channel update - → End -
Account Message Usage:
Sent early in connection to establish payment address:
Message { - account: AccountMessage { - address: 0x742d35Cc6634C0532925a3b8... // Ethereum address - } -} -
State Channel Update Format:
{ - "channelId": "0x1234...", - "nonce": 42, - "balances": { - "0x742d35Cc...": "1000000000000000000", // Seller balance - "0x8ab5d2F3...": "500000000000000000" // Buyer balance - }, - "signatures": [ - "0x789abc...", // Buyer signature - "0x456def..." // Seller signature - ] -} -
Insufficient Funds:
Invalid Signature:
Nonce Mismatch:
Channel Expired:
The Block Exchange protocol defines error handling for common failure scenarios:
Merkle Proof Verification Failure:
CID Mismatch:
Stream Disconnection:
Missing Blocks:
Request Timeout:
Oversized Messages:
Invalid WantList:
Payment Failures:
The protocol defines a clear separation between system-level and caller-level -retry responsibilities:
System-Level Retry (Automatic):
The Block Exchange module automatically retries in these scenarios:
The caller's requestBlock call remains pending during system-level retries. -This is transparent to the caller.
Caller-Level Retry (Manual):
The caller is responsible for retry decisions when:
In these cases, requestBlock returns an error and the caller decides -whether to retry, perhaps after waiting or refreshing the peer list -via discovery.
Retry Flow:
requestBlock(address) - │ - ├─► System tries Peer A ──► Fails - │ │ - │ └─► System tries Peer B ──► Fails (automatic, transparent) - │ │ - │ └─► System tries Peer C ──► Success ──► Return block - │ - └─► All peers failed ──► Return error to caller - │ - └─► Caller decides: retry? wait? abort? -
Peer Rotation:
When a peer fails to deliver blocks:
Graceful Degradation:
Error Propagation:
Two-Tier Block Addressing: -The protocol supports both standalone and dataset blocks to accommodate -different use cases. -Standalone blocks are simpler and don't require Merkle proofs, while -dataset blocks enable efficient verification of large datasets without -requiring the entire dataset.
WantList Delta Updates: -Supporting delta updates reduces bandwidth consumption when peers only -need to modify a small portion of their wants, which is common in -long-lived connections.
Separate Presence Messages: -Decoupling presence information from block delivery allows peers to -quickly assess availability without waiting for full block transfers.
Fixed Block Size: -The 64 KiB default block size balances efficient network transmission -with manageable memory overhead.
Zero-Padding Requirement: -Requiring zero-padding for incomplete dataset blocks ensures uniform -block sizes within datasets, simplifying Merkle tree construction and -verification.
Protocol Buffers: -Using Protocol Buffers provides efficient serialization, forward -compatibility, and wide language support.
Copyright and related rights waived via -CC0.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in 2119.
The Codex network aims to create a peer-to-peer storage engine with robust data durability, data persistence guarantees, and a comprehensive incentive structure.
The marketplace is a critical component of the Codex network, serving as a platform where all involved parties interact to ensure data persistence. It provides mechanisms to enforce agreements and facilitate data repair when SPs fail to fulfill their duties.
Implemented as a smart contract on an EVM-compatible blockchain, the marketplace enables various scenarios where nodes assume one or more roles to maintain a reliable persistence layer for users. This specification details these interactions.
CancelledError
nim-chronos
CatchableError
SaleErrored
Protocol Compliance Note: The Client implementation described above is specific to nim-codex. The only normative requirements for Clients are defined in the Client Role section of Part I. Implementations must satisfy those protocol requirements but may use completely different internal designs.
Status is a communication tool providing privacy features for the user. Specifications can also be viewed at Status.
60fcdd5
de29833
This specification is a voting protocol for peers to submit votes to a smart contract. Voting is immutable, this will help avoid sabotage from malicious peers.
In open p2p protocol there is an issue with voting off-chain as there is much room for malicious peers to only include votes that support their case when submitting votes to chain.
Once the vote deadline has expired, the smart contract will not accept votes anymore. Also directory will be updated according to vote results (community added to directory, removed etc.)
1255162
86cafd8
This specification describes a voting method to feature different active Status Communities.
When there is a active community that is seeking new members, current users of community should be able to feature their community so that it will be accessible to larger audience. @@ -25746,28 +22612,36 @@ only the vote with highest timestamp is considered valid. it can't be featured in current week. - In a current week top 5 (or 10) communities with highest amount of SNT votes up to previous Sunday 23:59:59 UTC are considered featured.
slug: 55 -title: 55/STATUS-1TO1-CHAT -name: Status 1-to-1 Chat -status: draft -category: Standards Track -tags: waku-application -description: A chat protocol to send public and private messages to a single recipient by the Status app. -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
0b4d151
e95c5c6
44baefd
e105bea
4a8585b
This specification describes how the Status 1-to-1 chat protocol is implemented on top of the Waku v2 protocol. This protocol can be used to send messages to a single recipient.
This document describes how 2 peers communicate with each other to send messages in a 1-to-1 chat, with privacy and authenticity guarantees.
This protocol MAY use any key-exchange mechanism previously discussed -
COLOR_CHANGED
To change the display image of the group chat, group admins MUST use an IMAGE_CHANGED event.
IMAGE_CHANGED
slug: 56 -title: 56/STATUS-COMMUNITIES -name: Status Communities that run over Waku v2 -status: draft -category: Standards Track -tags: waku-application -description: Status Communities allow multiple users to communicate in a discussion space. This is a key feature of the Status application. -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
f4b34af
9bed57e
11bdc3b
8350742
a786356
This document describes the design of Status Communities over Waku v2, allowing for multiple users to communicate in a discussion space. This is a key feature for the Status messaging app.
Due to the nature of communities, the following requirements are necessary for the design of communities -
The following cryptographic primitives are used in the design -
The clock used in the wire format refers to the Lamport timestamp of the message. The Lamport timestamp is a logical clock that is used to determine the order of events in a distributed system. This allows ordering of messages in an asynchronous network where messages may be received out of order.
The Community owner is a single point of failure. @@ -26507,9 +23392,9 @@ The following document describes this method in more detail - since members of the community don't need to receive all the control messages.
slug: 61 -title: 61/STATUS-Community-History-Service -name: Status Community History Service -status: draft -category: Standards Track -description: Explains how new members of a Status community can request historical messages from archive nodes. -editor: r4bbit r4bbit@status.im -contributors:
d8ba50e
ad72c49
Messages are stored permanently by store nodes (13/WAKU2-STORE) for up to a certain configurable period of time, @@ -26602,7 +23495,7 @@ older than 30 days.
These assumptions are less than ideal and will be enhanced in future work. This forum discussion provides more details.
The following is a high-level overview of the user flow and features this specification describes. For more detailed descriptions, read the dedicated sections in this specification.
slug: 62 -title: 62/STATUS-PAYLOADS -name: Status Message Payloads -status: draft -description: Describes the payload of each message in Status. -editor: r4bbit r4bbit@status.im -contributors:
ad80a59
1abf2c9
e2b9e32
20e9a66
488ce9b
This specification describes how the payload of each message in Status looks like. It is primarily centered around chat and chat-related use cases.
Released August 25, 2020
Released July 16, 2020
Released May 22, 2020
slug: 63 -title: 63/STATUS-Keycard-Usage -name: Status Keycard Usage -status: draft -category: Standards Track -description: Describes how an application can use the Status Keycard to create, store and transact with different account addresses. -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
928173d
68e6d45
100e0e8
This specification describes how an application can use the Status Keycard to -
More documentation on the Status Keycard can be found here
The Status Keycard is a hardware wallet that can be used to store and sign transactions. For the purpose of the Status App, @@ -28231,37 +25142,45 @@ MAY implement the following flows according to the actions listed above.
/unblock-pin
Inherits the security considerations of Status Keycard
Inherits the privacy considerations of Status Keycard
slug: 65 -title: 65/STATUS-ACCOUNT-ADDRESS -name: Status Account Address -status: draft -category: Standards Track -description: Details of what a Status account address is and how account addresses are created and used. -editor: Aaryamann Challani p1ge0nh8er@proton.me -contributors:
67a1221
91874da
ae04f02
This specification details what a Status account address is and how account addresses are created and used.
The core concept of an account in Status is a set of cryptographic keypairs. Namely, the combination of the following:
The above payload is broadcasted when 2 devices that belong to a user need to be paired.
slug: 71 -title: 71/STATUS-PUSH-NOTIFICATION-SERVER -name: Push Notification Server -status: draft -category: Standards Track -description: A set of methods to allow Status clients to use push notification services in mobile environments. -editor: Jimmy Debe jimmy@status.im -contributors:
3e21cc0
5bce327
8df3f00
eb7c5bf
A push notification server implementation for IOS devices and Android devices. This specification provides a set of methods that allow clients to use push notification services in mobile environments.
Push notification for iOS and Android devices can only be implemented by relying on APN, @@ -28420,11 +25349,11 @@ Status provides a set of methods to acheive push notification services.
Since this can not be safely implemented in a privacy-preserving manner, clients need to be given an option to opt-in to receive and send push notifications. They are disabled by default.
shake-256
chat_id
title: STATUS-SIMPLE-SCALING -name: Status Simple Scaling -status: raw -category: Informational -tags: waku/application -description: Describes how to scale Status Communities and Status 1-to-1 chats using Waku v2 protocol and components. -editor: Daniel Kaiser danielkaiser@status.im -contributors:
Early-stage Status specifications that precede draft or stable status.
9b5e219
a189a72
61a39e2
18a16ae
f19a136
27fae4f
fa8c750
5919040
f7a51b9
This document describes how to scale 56/STATUS-COMMUNITIES as well as 55/STATUS-1TO1-CHAT using Waku v2 protocol and components. @@ -29417,7 +26363,7 @@ one register query per shard.
A discovery query for nodes that are part of this shard would look like
DISCOVER{ns: "rs/16/2"}
Hereunder we describe the "opt-in message signing for DoS prevention" solution, designed ad hoc for Status MVP.
Since publishing messages to pubsub topics has no limits, @@ -29575,9 +26521,9 @@ It could be rate-limited with RLN.
This document describes how to scale -56/STATUS-COMMUNITIES as well as 55/STATUS-1TO1-CHAT -using Waku v2 protocol and components. -It also adds a few new aspects, -where more sophisticated components are not yet researched and evaluated.
-Note: (Parts of) this RFC will be deprecated in the future -as we continue research to scale specific components -in a way that aligns better with our principles of decentralization and -protecting anonymity. -This document informs about scaling at the current stage of research and -shows it is practically possible. -Practical feasibility is also a core goal for us. -We believe in incremental improvement, i.e. -having a working decentralized scaling solution with trade-offs -is better than a fully centralized solution. -
Note: (Parts of) this RFC will be deprecated in the future -as we continue research to scale specific components -in a way that aligns better with our principles of decentralization and -protecting anonymity. -This document informs about scaling at the current stage of research and -shows it is practically possible. -Practical feasibility is also a core goal for us. -We believe in incremental improvement, i.e. -having a working decentralized scaling solution with trade-offs -is better than a fully centralized solution.
56/STATUS-COMMUNITIES as well as -55/STATUS-1TO1-CHAT use Waku v2 protocols. -Both use Waku content topics -(see 23/WAKU2-TOPICS) -for content based filtering.
Waku v2 currently has scaling limitations in two dimensions:
Messages that are part of a specific content topic -have to be disseminated in a single mesh network (i.e. pubsub topic). -This limits scaling the number of messages disseminated in a specific content topic, -and by extension, the number of active nodes that are part of this content topic.
Scaling a large set of content topics requires distributing these over several -mesh networks (which this document refers to as pubsub topic shards).
This document focuses on the second scaling dimension. -With the scaling solutions discussed in this document, -each content topics can have a large set of active users, -but still has to fit in a single pubsub mesh.
-Note: While it is possible to use the same content topic name on several shards, -each node that is interested in this content topic -has to be subscribed to all respective shards, which does not scale. -Splitting content topics in a more sophisticated and -efficient way will be part of a future document. -
Note: While it is possible to use the same content topic name on several shards, -each node that is interested in this content topic -has to be subscribed to all respective shards, which does not scale. -Splitting content topics in a more sophisticated and -efficient way will be part of a future document.
Sharding the 11/WAKU2-RELAY -network is an integral part of scaling the Status app.
WAKU2-RELAY-SHARDING -specifies shards clusters, which are sets of 1024 shards -(separate pubsub mesh networks). -Content topics specified by application protocols can be distributed over these shards. -The Status app protocols are assigned to shard cluster 16, -as defined in WAKU2-RELAY-STATIC-SHARD-ALLOC.
1024
16
WAKU2-RELAY-SHARDING -specifies three sharding methods. -This document uses static sharding, -which leaves the distribution of content topics to application protocols, -but takes care of shard discovery.
The 1024 shards within the main Status shard cluster are allocated as follows.
Shard indices are mapped to pubsub topic names as follows (specified in WAKU2-RELAY-SHARDING).
/waku/2/rs/<cluster_id>/<shard_number>
an example for the shard with index 18 in the Status shard cluster:
18
/waku/2/rs/16/18
In other words, the mesh network with the pubsub topic name /waku/2/rs/16/18, -carries messages associated with shard 18 in the Status shard cluster.
The Waku implementation should offer an interface that -allows Status nodes to subscribe to Status specific content topics like
subscribe("/status/xyz", 16, 18) -
The shard cluster index 16 can be kept in the Status app configuration, -so that Status nodes can simply use
subscribe("/status/xyz", 18) -
which means: connect to the "status/xyz" content topic on shard 18 -within the Status shard cluster.
"status/xyz"
In order to associate a community with a shard, -the community description protobuf is extended by the field -uint16 relay_shard_index = 15:
uint16 relay_shard_index = 15
-syntax = "proto3"; - -message CommunityDescription { - // The Lamport timestamp of the message - uint64 clock = 1; - // A mapping of members in the community to their roles - map<string,CommunityMember> members = 2; - // The permissions of the Community - CommunityPermissions permissions = 3; - // The metadata of the Community - ChatIdentity identity = 5; - // A mapping of chats to their details - map<string,CommunityChat> chats = 6; - // A list of banned members - repeated string ban_list = 7; - // A mapping of categories to their details - map<string,CommunityCategory> categories = 8; - // The admin settings of the Community - CommunityAdminSettings admin_settings = 10; - // If the community is encrypted - bool encrypted = 13; - // The list of tags - repeated string tags = 14; - // index of the community's shard within the Status shard cluster - uint16 relay_shard_index = 15 -} -
-Note: Currently, Status app has allocated shared cluster 16 in WAKU2-RELAY-STATIC-SHARD-ALLOC. -Status app could allocate more shard clusters, for instance to establish a test net. -We could add the shard cluster index to the community description as well. -The recommendation for now, -is to keep it as a configuration option of the Status app. -Note: Once this RFC moves forward, -the new community description protobuf fields should be mentioned in 56/STATUS-COMMUNITIES. -
Note: Currently, Status app has allocated shared cluster 16 in WAKU2-RELAY-STATIC-SHARD-ALLOC. -Status app could allocate more shard clusters, for instance to establish a test net. -We could add the shard cluster index to the community description as well. -The recommendation for now, -is to keep it as a configuration option of the Status app. -Note: Once this RFC moves forward, -the new community description protobuf fields should be mentioned in 56/STATUS-COMMUNITIES.
Status communities can be mapped to shards in two ways: static, and owner-based.
With static mapping, -communities are assigned a specific shard index within the Status shard cluster. -This mapping is similar in nature to the shard cluster allocation in WAKU2-RELAY-STATIC-SHARD-ALLOC. -Shard indices allocated in that way are in the range 16 - 127. -The Status CC community uses index 16 -(not to confuse with shard cluster index 16, which is the Status shard cluster).
16 - 127
-Note: This way of mapping will be specified post-MVP. -
Note: This way of mapping will be specified post-MVP.
Community owners can choose to map their communities to any shard within -the index range 128 - 767.
128 - 767
55/STATUS-1TO1-CHAT -uses partitioned topics to map 1:1 chats to a set of 5000 content topics. -This document extends this mapping to 8192 content topics that are, in turn, -mapped to 128 shards in the index range of 768 - 895.
768 - 895
contentPartitionsNum = 8192 -contentPartition = mod(publicKey, contentPartitionsNum) -partitionContentTopic = "contact-discovery-" + contentPartition - -partitionContentTopic = keccak256(partitionContentTopic) - -shardNum = 128 -shardIndex = 768 + mod(publicKey, shardNum) - -
As described in 30/ADAPTIVE-NODES, -Waku supports a continuum of node types with respect to available resources. -Infrastructure nodes are powerful nodes that have a high bandwidth connection and -a high up-time.
This document, which informs about simple ways of scaling Status over Waku, -assumes the presence of a set of such infrastructure nodes in each shard. -Infrastructure nodes are especially important for -providing connectivity in the roll-out phase.
Infrastructure nodes are not limited to Status fleets, or -nodes run by community owners. -Anybody can run infrastructure nodes.
Infrastructure nodes are provided by the community owner, -or by members of the respective community.
Infrastructure nodes are part of a subset of the shards in the range 128 - 767. -Recommendations on choosing this subset will be added -in a future version of this document.
Status fleet nodes make up a part of these infrastructure nodes.
Infrastructure nodes are part of a subset of the shards in the range 768 - 985 -(similar to owner-mapped communities). -Recommendations on choosing this subset will be added -in a future version of this document.
768 - 985
Desktop clients can choose to only use filter and lightpush.
-Note: Discussion: I'd suggest to set this as the default for the MVP. -The load on infrastructure nodes would not be higher, because -they have to receive and relay each message anyways. -This comes as a trade-off to anonymity and decentralization, -but can significantly improve scaling. -We still have k-anonymity because -several chat pairs are mapped into one content topic. -We could improve on this in the future, and research the applicability of PIR -(private information retrieval) techniques in the future. -
Note: Discussion: I'd suggest to set this as the default for the MVP. -The load on infrastructure nodes would not be higher, because -they have to receive and relay each message anyways. -This comes as a trade-off to anonymity and decentralization, -but can significantly improve scaling. -We still have k-anonymity because -several chat pairs are mapped into one content topic. -We could improve on this in the future, and research the applicability of PIR -(private information retrieval) techniques in the future.
Waku messages are typically relayed in larger mesh networks -comprised of nodes with varying resource profiles -(see 30/ADAPTIVE-NODES). -To maximise scaling, relaying of specific message types can be dedicated to shards -where only infrastructure nodes with very strong resource profiles relay messages. -This comes as a trade-off to decentralization.
To get the maximum scaling for select large communities for the Status scaling MVP, -specific control messages that cause significant load -(at a high user number) SHOULD be moved to a separate control message shard. -These control messages comprise:
37b3edf
The relay functionality of control messages shards SHOULD -be provided by infrastructure nodes. -Desktop clients should use light protocols as the default for control message shards. -Strong Desktop clients MAY opt in to support the relay network.
Each large community (in the index range of 16 - 127) -can get its dedicated control message shard -(in the index range 896 - 1023) if deemed necessary. -The Status CC community uses shard 896 as its control message shard. -This comes with trade-offs to decentralization and anonymity -(see Security Considerations section).
896 - 1023
896
Similar to control messages, media-heavy communities should use separate media shards -(in the index range 896 - 1023) for disseminating messages with large media data. -The Status CC community uses shard 897 as its media shard.
897
Large communities MAY choose to mainly rely on infrastructure nodes -for all message transfers (not limited to control, and media messages). -Desktop clients of such communities should use light protocols as the default. -Strong Desktop clients MAY opt in to support the relay network.
-Note: This is not planned for the MVP. -
Note: This is not planned for the MVP.
Light protocols may be used to save bandwidth, -at the (global) cost of not contributing to the network. -Using light protocols is RECOMMENDED for resource restricted nodes, -e.g. browsers, -and devices that (temporarily) -have a low bandwidth connection or a connection with usage-based billing.
Light protocols comprise
Archive nodes are Waku nodes that offer the Waku archive service via -the Waku store protocol (13/WAKU2-STORE). -They are part of a set of shards and store all messages disseminated in these shards. -Nodes can request history messages via the 13/WAKU2-STORE.
The store service is not limited to a Status fleet. -Anybody can run a Waku Archive node in the Status shards.
-Note: There is no specification for discovering archive nodes -associated with specific shards yet. -Nodes expect archive nodes to store all messages, regardless of shard association. -
Note: There is no specification for discovering archive nodes -associated with specific shards yet. -Nodes expect archive nodes to store all messages, regardless of shard association.
The recommendation for the allocation of archive nodes to shards is similar to the -allocation of infrastructure nodes to shards described above. -In fact, the archive service can be offered by infrastructure nodes.
Shard discovery is covered by WAKU2-RELAY-SHARDING. -This allows the Status app to abstract from the discovery process and -simply address shards by their index.
To make nodes behind restrictive NATs discoverable, -this document suggests using libp2p rendezvous. -Nodes can check whether they are behind a restrictive NAT using the -libp2p AutoNAT protocol.
-Note: The following will move into WAKU2-RELAY-SHARDING, -or 33/WAKU2-DISCV5: -Nodes behind restrictive NATs SHOULD not announce their publicly unreachable address -via 33/WAKU2-DISCV5 discovery. -
Note: The following will move into WAKU2-RELAY-SHARDING, -or 33/WAKU2-DISCV5: -Nodes behind restrictive NATs SHOULD not announce their publicly unreachable address -via 33/WAKU2-DISCV5 discovery.
It is RECOMMENDED that nodes that are part of the relay network also -act as rendezvous points. -This includes accepting register queries from peers, -as well as answering rendezvous discover queries. -Nodes MAY opt-out of the rendezvous functionality.
To allow nodes to initiate connections to peers behind restrictive NATs -(after discovery via rendezvous), -it is RECOMMENDED that nodes that are part of the Waku relay network also offer -libp2p circuit relay -functionality.
To minimize the load on circuit-relay nodes, nodes SHOULD
Nodes that do not announce themselves at all and only plan to use light protocols, -MAY use rendezvous discovery instead of or along-side WAKU2-PEER-EXCHANGE. -For these nodes, rendezvous and -WAKU2-PEER-EXCHANGE -offer the same functionality, -but return node sets sampled in different ways. -Using both can help increasing connectivity.
Nodes that are not behind restrictive NATs MAY register at rendezvous points, too; -this helps increasing discoverability, and by extension connectivity. -Such nodes SHOULD, however, not register at circuit relays.
Registering a namespace via lib-p2p rendezvous -is done via a register query:
REGISTER{my-app, {QmA, AddrA}} -
The app name, my-app contains the encoding of a single shard in string form:
my-app
"rs/"| to_string(<2-byte shard cluster index>) | "/" | to_string(<2-byte shard index>) -
The string conversion SHOULD remove leading zeros.
-Note: Since the ns -field is of type string, a more efficient byte encoding is not utilized. -
Note: Since the ns -field is of type string, a more efficient byte encoding is not utilized.
Registering shard 2 in the Status shard cluster (with shard cluster index 16, -see WAKU2-RELAY-STATIC-SHARD-ALLOC, -the register query would look like
REGISTER{"rs/16/2", {QmA, AddrA}} -
Participation in further shards is registered with further queries; -one register query per shard.
DISCOVER{ns: "rs/16/2"} -
Hereunder we describe the "opt-in message signing for DoS prevention" solution, -designed ad hoc for Status MVP.
Since publishing messages to pubsub topics has no limits, -anyone can publish messages at a very high rate and DoS the network. -This would elevate the bandwidth consumption of all nodes subscribed -to said pubsub topic, making it prohibitive (in terms of bandwidth) -to be subscribed to it. -In order to scale, we need some mechanism to prevent this from happening, -otherwise all scaling efforts will be in vain. -Since RLN is not ready yet, -hereunder we describe a simpler approach designed ad hoc for Status use case, -feasible to implement for the MVP and -that validates some of the ideas that will evolve to solutions such as RLN.
With this approach, certain pubsub topics can be optionally configured -to only accept messages signed with a given key, -that only trusted entities know. -This key can be pre-shared among a set of participants, -that are trusted to make fair usage of the network, -publishing messages at a reasonable rate/size. -Note that this key can be shared/reused among multiple participants, and -only one key is whitelisted per pubsub topic. -This is an opt-in solution that operators can choose to deploy in their shards -(i.e. pubsub topics), but it's not enforced in the default one. -Operators can freely choose how they want to generate, and -distribute the public keys. -It's also their responsibility to handle the private key, -sharing it with only trusted parties and keeping proper custody of it.
The following concepts are introduced:
private-key-topic
protected-pubsub-topic
app-message-hash
sha256(concat(pubsubTopic, payload, contentTopic, timestamp, ephemeral))
message-signature
application-message-hash
public-key-topic
verify(message-signature, app-message-hash, public-key-topic)
This solution introduces two roles:
A publisher that wants to send messages -that are relayed in the network for a given protected-pubsub-topic shall:
meta
The app-message-hash of the message shall be calculated as the sha256 hash -of the following fields of the message:
sha256
sha256(concat(pubsubTopic, payload, contentTopic, timestamp, ephemeral)) -
Where fields are serialized into bytes using little-endian. -Note that ephemeral is a boolean that is serialized to 0 if false and -1 if true.
false
Requirements for the relay are listed below:
Requirements on the gossipsub validator:
Accept
Reject
abs(current_timestamp-timestamp) > MessageWindowInSec
Other requirements:
P4
This solution is designed to be backward compatible so -that nodes validating messages can coexist in the same topic -with other nodes that don't perform validation. -But note that only nodes that perform message validation -will be protected against DoS. -Nodes wishing to opt-in this DoS protection feature shall:
Relay nodes complying with this specification -shall accept the following message in the configured pubsub topic.
Given the following key pair:
private-key-topic = 5526a8990317c9b7b58d07843d270f9cd1d9aaee129294c1c478abf7261dd9e6 -public-key-topic = 049c5fac802da41e07e6cdf51c3b9a6351ad5e65921527f2df5b7d59fd9b56ab02bab736cdcfc37f25095e78127500da371947217a8cd5186ab890ea866211c3f6 -
And the following message to send:
protected-pubsub-topic = pubsub-topic -contentTopic = content-topic -payload = 1A12E077D0E89F9CAC11FBBB6A676C86120B5AD3E248B1F180E98F15EE43D2DFCF62F00C92737B2FF6F59B3ABA02773314B991C41DC19ADB0AD8C17C8E26757B -timestamp = 1683208172339052800 -ephemeral = true -
The message hash and meta (aka signature) are calculated as follows.
app-message-hash = 662F8C20A335F170BD60ABC1F02AD66F0C6A6EE285DA2A53C95259E7937C0AE9 -message.meta = 127FA211B2514F0E974A055392946DC1A14052182A6ABEFB8A6CD7C51DA1BF2E40595D28EF1A9488797C297EED3AAC45430005FB3A7F037BDD9FC4BD99F59E63 -
Using message.meta, the relay node shall calculate the app-message-hash -of the received message using public-key-topic, -and with the values above, the signature should be verified, -making the node Accept the message and relaying it to other nodes in the network.
message.meta
Basic idea: -Tokenized load.
An idea we plan to explore in the future: -Map 1:1 chats to community shards, if both A and B are part of the respective community. -This increases k-anonymity and benefits from community DoS protection. -It could be rate-limited with RLN.
This document makes several trade-offs to privacy and anonymity. -Todo: elaborate. -See WAKU2-ADVERSARIAL-MODELS -for information on Waku Anonymity.
title: STATUS-PROTOCOLS -name: Status Protocol Stack -status: raw -category: Standards Track -description: Specifies the Status application protocol stack. -editor: Hanno Cornelius hanno@status.im -contributors:
This specification describes the Status Application protocol stack. It focuses on elements and features in the protocol stack for all application-level functions:
For the purposes of the rest of this document, clients in full mode will be referred to as "full clients" and clients in light mode will be referred to as "light clients".
The application MUST make use of at least one discovery method to discover and connect to Waku peers useful for the user functions specific to that instance of the application.
The specific Waku discovery protocol used for discovery depends on the use case and resource-availability of the client.
751a01e
This document lists the types of messages that are using MVDS in the Status application.
Status app uses MVDS to ensure messages going through Waku are acknolwedged by the recipient. This is to ensure that the messages are not missed by any interested parties.
title: STATUS-URL-DATA -name: Status URL Data -status: raw -category: Standards Track -tags: -editor: Felicio Mununga felicio@status.im -contributors:
36f64f0
This document specifies serialization, compression, and encoding techniques used to transmit data within URLs in the context of Status protocols.
When sharing URLs, link previews often expose metadata to the websites behind those links. To reduce reliance on external servers for providing appropriate link previews, @@ -30712,9 +27150,9 @@ raw_data = deserialized_data.content
a519e67
89cac77
This document describes URL scheme for previewing and deep linking content as well as for triggering actions.
title: 3RD-PARTY -name: 3rd party -status: deprecated -description: This specification discusses 3rd party APIs that Status relies on. -editor: Filip Dimitrijevic filip@status.im -contributors:
Deprecated Status specifications maintained for archival purposes.
614348a
This specification discusses 3rd party APIs that Status relies on. These APIs provide various capabilities, including:
Concerns If Iubenda becomes unavailable, users will be unable to view the app's privacy policy.
This specification discusses 3rd party APIs that Status relies on. -These APIs provide various capabilities, including:
Relying on 3rd party APIs conflicts with Status’s censorship-resistance principle. -Since Status aims to avoid suppression of information, -it is important to minimize reliance on 3rd parties that are critical to app functionality.
What is it? -Infura hosts a collection of Ethereum full nodes and provides an API -to access the Ethereum and IPFS networks without requiring a full node.
How Status Uses It -Since Status operates on mobile devices, -it cannot rely on a local node. -Therefore, all Ethereum network communication happens via Infura.
Concerns -Making an HTTP request can reveal user metadata, -which could be exploited in attacks if Infura is compromised. -Infura uses centralized hosting providers; -if these providers fail or cut off service, -Ethereum-dependent features in Status would be affected.
What is it? -Etherscan is a service that allows users to explore the Ethereum blockchain -for transactions, addresses, tokens, prices, -and other blockchain activities.
How Status Uses It -The Status Wallet allows users to view address and transaction details on Etherscan.
Concerns -If Etherscan becomes unavailable, -users won’t be able to view address or transaction details through Etherscan. -However, in-app information will still be accessible.
What is it? -CryptoCompare provides live crypto prices, charts, and analysis from major exchanges.
How Status Uses It -Status regularly fetches crypto prices from CryptoCompare, -using this information to calculate fiat values -for transactions or wallet assets.
Concerns -HTTP requests can reveal metadata, -which could be exploited if CryptoCompare is compromised. -If CryptoCompare becomes unavailable, -Status won’t be able to show fiat equivalents for crypto in the wallet.
Various services provide information on collectibles:
Concerns -HTTP requests can reveal metadata, -which could be exploited if these services are compromised.
What is it? -Iubenda helps create compliance documents for websites and apps across jurisdictions.
How Status Uses It -Status’s privacy policy is hosted on Iubenda.
Concerns -If Iubenda becomes unavailable, -users will be unable to view the app's privacy policy.
title: ACCOUNT -name: Account -status: deprecated -description: This specification explains what a Status account is, and how a node establishes trust. -editor: Filip Dimitrijevic filip@status.im -contributors:
This specification explains what a Status account is, and how a node establishes trust.
For the user, the deserialization process is exactly the same as serialization with the exception that the user MUST provide a serialized public key for deserialization. Else the deserialization algorithm will fail.
For further guidance on the implementation of public key de/serialization consult the status-go implementation and tests.
status-go
Released June 24, 2020
Mailserver
title: CLIENT -name: Client -status: deprecated -description: This specification describes how to write a Status client for communicating with other Status clients. -editor: Filip Dimitrijevic filip@status.im -contributors:
This specification describes how to write a Status client for communicating with other Status clients. This specification presents a reference implementation of the protocol @@ -31419,7 +27779,7 @@ used in a command-line client and a mobile app.
This document consists of two parts. The first outlines the specifications required to be a full Status client. The second provides a design rationale and answers some common questions.
Implementing a Status clients largely means implementing the following layers. Additionally, there are separate specifications for things like key management and account lifecycle.
These bootstrap nodes MAY change and are not guaranteed to stay this way forever and at some point circumstances might force them to change.
A Status client MUST discover or have a list of peers to connect to. Status uses a light discovery mechanism based on a combination of Discovery v5 and Rendezvous Protocol, @@ -31547,7 +27907,7 @@ various degrees of standardization. Please refer to BIPs and EIPs Standards support
For a list of EIPs and BIPs that SHOULD be supported by Status client, please see EIPS.
See Appendix A
P2P Overlay
There are several security considerations to take into account when running Status. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used, such as Mailserver, light node, and so on.
In version 1 of Status, bandwidth usage is likely to be an issue. In Status version 1.1 this is partially addressed with Waku usage, see the theoretical scaling model.
Status currently lacks incentives to run nodes, which means node operators are more likely to create centralized choke points.
The main privacy concern with light nodes is that directly connected peers will know that a message originates from them (as it are the only ones it sends). This means nodes can make assumptions about what messages (topics) their peers are interested in.
Bloom filter privacy:
Mailserver trusted connection:
A Mailserver has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning Mailserver can overwhelm an individual node.
By default Devp2p runs on port 30303, which is not commonly used for any other service. This means it is easy to censor, e.g. airport WiFi. This can be mitigated somewhat by running on e.g. port 80 or 443, but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
See https://github.com/status-im/status-mobile/issues/6351 for some discussion.
Jacek Sieka
This document describes requirements that an application must fulfill in order to provide a proper environment for Dapps running inside a browser. A description of the Status Dapp API is provided, along with an overview of bidirectional communication underlying the API implementation. The document also includes a list of EIPs that this API implements.
window.ethereum
The application should expose an Ethereum Provider object (window.ethereum) to JS code running inside the browser. It is important to have the window.ethereum object available before the page loads, otherwise Dapps might not work correctly.
Additionally, the browser component should also provide bidirectional communication between JS code and the application.
isStatus
Returns true. Can be used by the Dapp to find out whether it's running inside Status.
Returns a StatusAPI object. For now it supports one method: getContactCode that sends a contact-code request to Status.
StatusAPI
getContactCode
contact-code
isConnected
eth_requestAccounts
connect
disconnect
chainChanged
accountsChanged
This specification describes how Status relates with EIPs.
Status should follow all standards as possible. Whenever the Status app needs a feature, it should be first checked if there is a standard for that, if not, Status should propose a standard.
This specification documents all the interactions that the Status client has with the Ethereum blockchain.
All the interactions are made through JSON-RPC. Currently Infura is used. The client assumes high-availability, @@ -32203,9 +28590,9 @@ It is used in status to check if a token transfer was made to the user address.<
ENS names are propagated through ChatMessage and ContactUpdate payload. A client SHOULD verify ens names against the public key of the sender on receiving the message against the ENS contract
ChatMessage
ContactUpdate
title: GROUP-CHAT -name: Group Chat -status: deprecated -description: This document describes the group chat protocol used by the Status application. -editor: Filip Dimitrijevic filip@status.im -contributors:
This document describes the group chat protocol used by the Status application. The node uses pairwise encryption among members so a message is exchanged between each participant, similarly to a one-to-one message.
chatId
If the event is valid a client MUST remove the member from the list of admins of the chat.
admins
title: IPFS-gateway-for-Sticker-Pack -name: IPFS gateway for Sticker Pack -status: deprecated -description: This specification describes how Status uses the IPFS gateway to store stickers. -editor: Filip Dimitrijevic filip@status.im -contributors:
This specification describes how Status uses the IPFS gateway to store stickers. The specification explores image format, @@ -32372,7 +28771,7 @@ and how an end user can see them inside the Status app.
Accepted image file types are PNG, JPG/JPEG and GIF, with a maximum allowed size of 300kb. @@ -32440,9 +28839,9 @@ This method will return an uint256 which is the id of the owned sti
PNG
JPG/JPEG
GIF
uint256
To buy a sticker pack call approveAndCall(address,uint256,bytes) where address is the address of buyer,uint256 is the price and third parameters bytes is the callback called if approved. In the callback, call buyToken(uint256,address,uint256), first parameter is sticker pack id, second buyers address, and the last is the price.
approveAndCall(address,uint256,bytes)
buyToken(uint256,address,uint256)
title: Keycard Usage for Wallet and Chat Keys -name: Keycard Usage for Wallet and Chat Keys -status: deprecated -description: In this specification, we describe how Status communicates with Keycard to create, store and use multiaccount. -editor: Filip Dimitrijevic filip@status.im -contributors:
In this specification, we describe how Status communicates with Keycard to create, store and use multiaccount.
encryption-public-key
status-im.hardwallet.card/get-keys
status-im.hardwallet.card/generate-and-load-keys
title: NOTIFICATIONS -name: Notifications -status: deprecated -description: A client should implement local notifications to offer notifications for any event in the app without the privacy cost and dependency on third party services. -editor: Filip Dimitrijevic filip@status.im -contributors:
A client should implement local notifications to offer notifications for any event in the app without the privacy cost and dependency on third party services. @@ -32766,32 +29177,36 @@ The system decides when the background updates are triggered and the heuristics
Push Notifications, as offered by Apple and Google are a privacy concern, they require a centralized service that is aware of who the notification needs to be delivered to.
title: PAYLOADS -name: Payloads -status: deprecated -description: Payload of messages in Status, regarding chat and chat-related use cases. -editor: Filip Dimitrijevic filip@status.im -contributors:
The payloads aims to be flexible enough to support messaging but also cases described in the Status Whitepaper as well as various clients created using different technologies.
This document describes the payload format and some special considerations.
The node wraps all payloads in a protobuf record:
Status Whitepaper protobuf record Protobuf @@ -33084,17 +29499,23 @@ The details are in the Lamport timestamps Group chats specs May 22, 2020 change commit
title: PUSH-NOTIFICATION-SERVER -name: Push notification server -status: deprecated -description: Status provides a set of Push notification services that can be used to achieve this functionality. -editor: Filip Dimitrijevic filip@status.im -contributors:
Push notifications for iOS devices and some Android devices can only be implemented by relying on APN service for iOS or Firebase.
This is useful for Android devices that do not support foreground services @@ -33118,7 +29539,7 @@ The party releasing the app, Status in this case, needs to run its own Gorush instance
A gorush instance MUST be publicly available, this will be used only by push notification servers.
A push notification server used by clients to register for receiving and sending push notifications.
A Status client that wants to receive push notifications
This can be mitigated by device syncing, but not completely addressed.
If no anonymous mode is used, when registering with a push notification service a client discloses:
Push notification servers can be run by anyone. Gorush can be run by anyone I take, but we are in charge of the certificate, so they would not be able to notify status-clients.
Released
title: SECURE-TRANSPORT -name: Secure Transport -status: deprecated -description: This document describes how Status provides a secure channel between two peers, providing confidentiality, integrity, authenticity, and forward secrecy. -editor: Filip Dimitrijevic filip@status.im -contributors:
This document describes how Status provides a secure channel between two peers, and thus provide confidentiality, integrity, authenticity and forward secrecy. It is transport-agnostic and works over asynchronous networks.
It builds on the X3DH and Double Ratchet specifications, with some adaptations to operate in a decentralized environment.
This document describes how nodes establish a secure channel, and how various conversational security properties are achieved.
Perfect Forward Secrecy is a feature of specific key-agreement protocols @@ -33679,7 +30102,7 @@ Specifically, past messages cannot be decrypted by a third-party who manages to
Secret channel describes a communication channel where Double Ratchet algorithm is in use.
DirectMessageProtocol
installa - otherwise, payload encrypted with output key of DH exchange (no Perfect Forward Secrecy).
The same considerations apply as in section 4 of the X3DH spec and section 6 of the Double Ratchet spec, with some additions detailed below.
Being mostly offline is an intrinsic property of mobile clients. They need to save network transfer and battery consumption to avoid spending too much money or constant charging. Waku protocol, on the other hand, is an online protocol. @@ -34228,7 +30655,7 @@ it SHOULD handle P2P Request Complete code (0x7d). This code is fol
0x7d
If Cursor is not empty, it means that not all messages were sent due to the set Limit in the request. One or more consecutive requests MAY be sent with Cursor field filled in order to receive the rest of the messages.
The node encrypts all Waku envelopes. A Mailserver node can not inspect their contents.
Since a Mailserver is delivering expired envelopes and has a direct TCP connection with the recipient, the recipient is vulnerable to DoS attacks from a malicious Mailserver node.
MailserverChange to keep Mailserver term consistent Replaced Whisper references with Waku
title: WAKU-USAGE -name: Waku Usage -status: deprecated -description: Status uses Waku to provide privacy-preserving routing and messaging on top of devP2P. -editor: Filip Dimitrijevic filip@status.im -contributors:
Status uses Waku to provide privacy-preserving routing and messaging on top of devP2P. Waku uses topics to partition its messages, @@ -34322,7 +30752,7 @@ Hence, all clients MUST use the following Whisper node settings:
0.000002
10
Handshake is a RLP-encoded packet sent to a newly connected peer. It MUST start with a Status Code (0x00) and follow up with items:
[ [ pow-requirement-key pow-requirement ] @@ -34542,8 +30972,8 @@ To move further back in time, use cursor and limit.Boolean - returns true if the request was sent. The above topics is then converted into a bloom filter and then and sent to the Mailserver. -Changelog -Version 0.1 +Changelog +Version 0.1 Released May 22, 2020
[ [ pow-requirement-key pow-requirement ] @@ -34542,8 +30972,8 @@ To move further back in time, use cursor and limit.Boolean
cursor
limit
The above topics is then converted into a bloom filter and then and sent to the Mailserver.
title: WHISPER-MAILSERVER -name: Whisper mailserver -status: deprecated -description: Whisper Mailserver is a Whisper extension that allows to store messages permanently and deliver them to the clients even though they are already not available in the network and expired. -editor: Filip Dimitrijevic filip@status.im -contributors:
Being mostly offline is an intrinsic property of mobile clients. They need to save network transfer and battery consumption to avoid spending too much money or constant charging. @@ -34655,7 +31090,7 @@ it SHOULD handle P2P Request Complete code (0x7d). This code is fol Cursor: an array of a cursor returned from the previous request (optional)
The node encrypts all Whisper envelopes. A Mailserver node can not inspect their contents.
MailserverWaku V1 May 22, 2020 change commit
title: WHISPER-USAGE -name: Whisper Usage -status: deprecated -description: Status uses Whisper to provide privacy-preserving routing and messaging on top of devP2P. -editor: Filip Dimitrijevic filip@status.im -contributors:
Status uses Whisper to provide privacy-preserving routing and messaging on top of devP2P. Whisper uses topics to partition its messages, @@ -34993,16 +31431,16 @@ To move further back in time, use cursor and limit.