Compare commits

...

65 Commits

Author SHA1 Message Date
Cofson
e72b95578e Merge branch 'main' into 36/bindings-api-draft 2025-07-25 17:10:25 +02:00
Cofson
7365003f19 Improved based on the comments 2025-07-10 20:55:32 +02:00
Cofson
b60abdb2ff update waku/standards/application/53/x3dh.md (#150)
Improved x3dh.md in the waku/standards/application/53 folder

---------

Co-authored-by: Jimmy Debe <91767824+jimstir@users.noreply.github.com>
2025-07-01 00:16:08 +02:00
Cofson
e9dd746034 Merge branch 'main' into 36/bindings-api-draft 2025-07-01 00:15:33 +02:00
AkshayaMani
5e3b4788ec RFC Refactor PR: Modular Rewrite of Mix Protocol Specification (#158)
This PR rewrites the Mix Protocol RFC for clarity, layering, and
implementability. It reorganizes the spec into a modular structure
aligned with RFC 7322 and RFC 2119, with clean separation between
protocol logic, integration, and pluggable components.

### Goals

- Clearly define Mix as a **message-based routing protocol**, not a
transport or application-layer solution.
- Support **per-message anonymity** via a `mixify` flag and **external
integration** through Entry and Exit layers.
- Adopt a clean, layered structure: motivation → integration → routing
model → cryptographic structure → node behavior.
- Use correct RFC formatting, keywords, and SEMBR for implementation
clarity.

### Changes Completed in This PR

#### Sections 0–5: Foundational Structure and Protocol Layering

* [x] **Abstract**: Concise, non-repetitive summary of protocol purpose
and scope.
* [x] **§1 Introduction**: Cleanly describes protocol role, document
scope, and integration model.
* [x] **§2 Terminology**: Defines all key terms; includes correct use of
`MUST`, `SHOULD`, and informal variants.
* [x] **§3 Motivation and Background**: Articulates need for sender
anonymity in libp2p; includes `§3.1 Comparison with Tor`.

#### Section 4 Mixing Strategy and Packet Format

* [x] **§4.1 Mixing Strategy**: Defines continuous-time mixing;
justifies choice over batching.
* [x] **§4.2 Packet Format Overview**: Defines what a mix packet must
achieve; outlines Sphinx format properties and rationale.

#### Section 5 Protocol Overview and Integration

* [x] **§5 Protocol Overview**: Clean, layered walkthrough of core
protocol behavior and layering in libp2p.
* [x] **§5.1 Integration with Origin Protocols**: External interface
components (Mix Entry/Exit layers).
* [x] **§5.2 Mixify Option**: Per-message flag defined.
* [x] **§5.3 Why a Protocol, Not a Transport**: Explains why Mix is
layered as a libp2p protocol, not a transport.
* [x] **§5.4 Protocol Interaction Flow**: Three-phase diagram and
explanation (entry → routing → exit).

#### Section 6 Pluggable Components

* [x] Discovery: advertise Mix support via ENR, X25519 key
* [x] Delay strategy: sender-defined per-hop delay
* [x] Spam protection: PoW/VDF/RLN options, exit node validation
* [x] Cover traffic: periodic loops for unobservability
* [x] Incentivization: nodes MUST participate to send

#### Section 7 Core Mix Protocol Responsibilities

* [x] Define Sender, Intermediary, Exit node roles
* [x] Specify lifecycle and message flow per role

#### Section 8 Sphinx Packet Format (Detailed Spec)

* [ ] Full field definitions (α, β, γ, δ, delays, MACs)
* [ ] Encoding and padding behavior

#### Section 9 Node Behavior (Mix Protocol Handler)

* [ ] Describe sender node logic: path selection, packet wrapping
* [ ] Describe intermediary/exit node behavior: decryption, delay,
forwarding

#### Section 10 Limitations and Future Work

* [ ] SURB-based reply support (unimplemented)
* [ ] DoS/Sybil attack surface
* [ ] Path overlap, message reordering, end-to-end acks

#### Appendices

* [ ] Appendix A: Proof-of-Work Example
* [ ] Appendix B: ENR-Based Discovery Example
2025-06-27 14:03:10 -04:00
Cofson
658734ba4c fixed 2 issues from comments 2025-06-15 22:55:34 +02:00
Cofson
132701c8c8 Merge branch 'main' into 36/bindings-api-draft 2025-06-13 00:35:01 +02:00
Jimmy Debe
36caaa621a Fix Errors rfc.vac.dev (#165)
Two errors causing build problems on rfc.vac.dev. This should be fix.
2025-06-05 10:13:21 -04:00
Jimmy Debe
db90adc94e Fix LaTeX errors (#163)
LaTeX are causing build errors on rfc.vac.dev. Potential fixes..
2025-06-02 17:33:46 -04:00
Cofson
f829b12517 waku/standards/core/66/metadata.md update (#148)
Improved metadata.md in waku/standards/core/66 folder
2025-05-13 01:52:53 +02:00
Cofson
cc6794812d Merge branch 'main' into 36/bindings-api-draft 2025-05-05 15:57:22 +02:00
Cofson
91c9679bc8 update waku/informational/30/adaptive-nodes.md (#147)
Improved adaptive-nodes.md file in waku/informational/30 folder

---------

Co-authored-by: Jimmy Debe <91767824+jimstir@users.noreply.github.com>
2025-04-29 10:25:54 +02:00
Cofson
614348a498 Status deprecated update2 (#134)
These are from the previous PR:
https://github.com/vacp2p/rfc-index/pull/109

---------

Co-authored-by: Jimmy Debe <91767824+jimstir@users.noreply.github.com>
2025-04-29 10:24:09 +02:00
Cofson
5971166405 Update discv5.md (#139)
Updates to discv5 links.
2025-04-29 10:22:57 +02:00
Cofson
db365cb756 update waku/standards/application/54/x3dh-sessions.md (#151)
Improved x3dh-sessions.md file in the waku/standards/application/54
folder
2025-04-24 15:04:57 +02:00
Cofson
4df2d5f787 update waku/informational/23/topics.md (#144)
Updates to topics.md in waku/informational/23 folder
2025-04-22 12:56:38 +02:00
Cofson
1cb6d8adb2 Merge branch 'main' into 36/bindings-api-draft 2025-04-22 12:55:55 +02:00
Cofson
af7c413e64 update waku/informational/27/peers.md (#145)
Improved peers.md file in waku/informational/27 folder
2025-04-22 12:55:32 +02:00
Cofson
7408956616 update waku/informational/29/config.md (#146)
Improved config.md file in waku/informational/29 folder
2025-04-22 12:55:15 +02:00
Cofson
6654f255d6 Merge branch 'main' into 36/bindings-api-draft 2025-04-16 15:09:39 +02:00
Jimmy Debe
34aa3f3647 Fix links 10/WAKU2 (#153)
Update links that were not working.
2025-04-15 19:35:08 -04:00
Jimmy Debe
1b8b2ac70b Add missing status to 13/WAKU-STORE (#149)
13/WAKU-STORE was missing the status.
2025-04-15 19:32:24 -04:00
Cofson
230ccbf322 update waku/standards/core/36/bindings-api.md 2025-04-13 20:14:02 +02:00
Cofson
eba089277d merge 2025-04-11 14:25:43 +02:00
Jimmy Debe
3b152e44b5 20/TOY-ETH-PM: Update (#141)
Removing Waku terminology (Waku v2). Updating links, grammar and some
formatting changes. A path to stable should be considered as
implementation has not been revisited for two years.

---------

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2025-04-09 17:20:00 -04:00
Jimmy Debe
cafa04fb93 10/WAKU2: Update (#125)
Updates include changing links, removing deprecated specs, and adding
updated specs.

---------

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2025-04-09 17:19:40 -04:00
Jimmy Debe
805280880a 14/WAKU2-MESSAGE: Move to Stable (#120)
Updating 14/WAKU2-MESSAGE rfc. Suggesting a move to stable.

---------

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2025-04-09 17:19:10 -04:00
Jimmy Debe
517b63984c Update the RFCs: Vac Raw RFC (#143)
Updated a few Vac raw RFCs noise-x3dh-double-ratchet, eth-mls-on-chain,
eth-secpm, eth-dcgka.
2025-04-04 17:04:00 +02:00
Jimmy Debe
f08de10845 26/WAKU2-PAYLOADS: Update (#136)
26/WAKU2-PAYLOADS updating links and some information in sections.
2025-03-31 19:06:58 -04:00
Jimmy Debe
e8a3f8a77d 12/WAKU2-FILTER: Update (#119)
An update of the 12/WAKU2-FILTER RFC. Some rearrange some sections,
updated links and terms.

---------

Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2025-03-25 11:48:27 -04:00
Arseniy Klempner
8ee2a6d6b2 docs: add optional retrieval hint to causal history in sds (#130) 2025-03-07 18:32:46 -08:00
Jimmy Debe
f4b34afd1a Fix Linting Errors (#135)
Fix the current markdown-linting errors.
2025-03-07 16:36:06 +01:00
Cofson
3902d40266 Additional changes based on codebase
- ContentFilter
- ContentTopic
- FilterSubscription
2025-03-07 16:16:38 +01:00
Jazz Turner-Baggs
0277fd0c4d docs: update dead links in 64/Network (#133)
This PR updates dead links in 64/NETWORK.
-
`https://github.com/waku-org/specs/blob/master/standards/core/metadata.md`
is repalced with
`https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/66/metadata.md`
- Updates the name `WAKU-METADATA` to `66/WAKU2-METADATA` to match style
2025-02-25 09:24:45 +00:00
Hanno Cornelius
9bed57e4ad docs: define basic sharding for Communities (#132)
Changes here address the bulk of what is necessary for
https://github.com/waku-org/pm/issues/266 and
https://github.com/status-im/status-go/issues/5821 to define the new
shared-shard strategy for Communities.

This PR should be enough unblock implementation and, once merged, we can
go ahead with the pubsub topic changes in status-go.
I'm not yet closing the accompanying deliverable, as the spec still
requires a full list of all Communities-related messages with functional
scope explicitly assigned.
2025-02-21 12:38:27 +00:00
Hanno Cornelius
235c1d5aa6 docs: clarify receiving sync messages (#131)
Minor adds to "Receive message" procedure to reflect changes introduced
in https://github.com/vacp2p/rfc-index/pull/129
2025-02-20 11:03:49 +00:00
Arseniy Klempner
718245979f docs: update sds sync message requirements (#129) 2025-02-18 21:10:31 -08:00
Cofson
1ddddc7994 update to tree structure (#128)
closed the previous due to some signature errors and fixed it in this
one (for reference: https://github.com/vacp2p/rfc-index/pull/121)

---------

Co-authored-by: Jimmy Debe <91767824+jimstir@users.noreply.github.com>
2025-02-15 04:03:46 +01:00
Cofson
891a9dd7a2 Testing commit signing 2025-02-14 18:53:03 +01:00
Cofson
dfb5b167e2 Update bindings-api.md 2025-02-14 05:02:23 +01:00
Jimmy Debe
a60a2c45de 13/WAKU-STORE: Update (#124)
Update 13/WAKU-STORE RFC to store v3 spec. Previous draft has been moved
to `previous-versions` version number 00.
2025-02-03 03:52:35 -05:00
Jimmy Debe
c3d5fe6f88 15/WAKU2-BRIDGE: Update (#113)
Updating format, adding references and some information.

---------

Co-authored-by: kaiserd <1684595+kaiserd@users.noreply.github.com>
Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2025-01-28 20:21:01 -05:00
Arseniy Klempner
7a01711ffc fix(sds): remove optional from causal history field in Message protobuf (#123) 2025-01-28 12:02:54 +02:00
Prem Chaitanya Prathi
dc7497a312 add usage guidelines for waku content topics (#117)
1. add usage guidelines for waku content topics
2. update communities spec as per content topic changes (#118)  
    - Update content topic usage as per
    https://forum.vac.dev/t/status-communities-review-and-proposed-usage-of-waku-content-topics/335
    - Update symmetric encryption done at content topic level before
    messages are handed over to waku
    - Update about shard or pubsub topic usage
2025-01-02 17:45:06 +05:30
Jimmy Debe
08b363d67e Update SDS.md: Remove Errors (#115)
Remove markdown error from the SDS rfc.
2024-12-18 03:16:50 +01:00
Jinho Jang
2b297d597f Update peer-exchange.md to fix a build error (#114)
<img width="1714" alt="Screenshot 2024-12-06 at 8 29 20 PM"
src="https://github.com/user-attachments/assets/b3fce6c4-48e8-44a6-8f7d-64e42002e121">


Images should be defined using relative paths to successfully build
https://rfc.vac.dev/, which is powered by Docusaurus.

I have verified that this update resolves the build issue locally.
2024-12-07 00:49:08 +09:00
Hanno Cornelius
bee78c40b9 docs: add SDS protocol for scalable e2e reliability (#108)
Introduces the "Scalable Data Sync" protocol, based on the e2e
reliability mechanisms discussed in the [Vac forum
post](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/10).

The initial aim is to provide an end-to-end reliability protocol for
Status chat protocols, specifically within the context of Status
Communities.

---------

Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>
Co-authored-by: Jimmy Debe <91767824+jimstir@users.noreply.github.com>
2024-11-28 16:22:36 +00:00
Jimmy Debe
87d4ff74d0 Workflow Fix: markdown-lint (#111)
Fixing some errors that still occur in markdown-lint.
2024-11-20 18:32:58 -05:00
Jimmy Debe
dcc579c47a Update WAKU2-PEER-EXCHANGE: Move to draft (#7)
Move 34/WAKU2-PEER-EXCHANGE to draft.
2024-11-20 12:36:40 -05:00
Jimmy Debe
ff87c84dc7 Update Waku Links (#104)
Change Waku links that do not point to master waku/specs repo. Also
update title for template, to look better on the rfc website.
2024-11-20 12:32:56 -05:00
Jimmy Debe
e60519bc80 Update .markdownlint.json (#106)
Remove 80 characters line length requirement
- Semantic breaks must still be enforced. 
Remove requirements to remove headings with the same content. 
- If multiple headings have the same headings, those headings should not
be referenced within the document.
2024-11-20 12:23:13 -05:00
Jimmy Debe
776c1b76cd rfc-index: Update (#110)
Removing .DS_Store files and updating some links.
2024-11-20 12:22:29 -05:00
ksr
38fce27d33 typo fix 2024-11-08 08:42:02 +01:00
Hanno Cornelius
37b3edfba3 docs: add spec for status protocol stack, deprecate waku-usage spec (#105)
This adds a new raw specification covering the Status protocol stack,
including:
- common app-level features (content topics, functional scopes,
ephemerality)
- e2e reliability layer
- encryption layer (TBD)
- Waku transport layer

It deprecates/replaces the previous status waku-usage spec.

---------

Co-authored-by: Jimmy Debe <91767824+jimstir@users.noreply.github.com>
2024-10-25 17:33:35 +01:00
0xbathang
b717c0aec9 Update README.md (#101) 2024-10-11 16:49:21 +02:00
0xbathang
869e12f06a Update README.md (#102) 2024-10-11 16:49:06 +02:00
Afri Schoedon
5b5e972259 cosmetic: fix typo in readme (#99) 2024-10-09 17:54:02 +02:00
Afri Schoedon
d5e0072498 cosmetic: fix external links in 1/COSS (#100) 2024-10-09 17:53:44 +02:00
ramsesfv
c655980494 Eth secpm splitted (#91)
This branch contains 2 new RFCs: 
1. An RFC describing the secure 1-to-1 communication channels;
2. An RFC describing the MLS and the decentralized MLS.

---------

Co-authored-by: Ekaterina Broslavskaya <seemenkina@gmail.com>
Co-authored-by: kaiserd <1684595+kaiserd@users.noreply.github.com>
2024-10-03 16:20:18 +02:00
Jimmy Debe
9b5e2194cf Update simple-scaling (#93)
Remove slug within title of spec.
2024-09-17 13:14:00 -04:00
Jimmy Debe
a519e67752 Move Status-URL-scheme (#98)
Move url-scheme to `status/raw`.
2024-09-17 13:11:46 -04:00
Jimmy Debe
36f64f01f0 feat(59/STATUS-URL-DATA): initial draft (#13)
Moved from https://github.com/status-im/specs/pull/169

Reference pull request: https://github.com/vacp2p/rfc/pull/600

---------

Co-authored-by: Felicio Mununga <felicio@users.noreply.github.com>
2024-09-17 13:11:30 -04:00
Jimmy Debe
ad80a59e2b 62/STATUS-PAYLOAD: Add Description (#95)
62/STATUS-PAYLOAD missing description.
2024-09-16 12:51:00 -04:00
AkshayaMani
7f5276e18c libp2p Mix Protocol Spec Draft (#97)
Old PR can be found here: [Mix Protocol Spec
#85](https://github.com/vacp2p/rfc-index/pull/85)
2024-09-16 08:05:47 -04:00
Jimmy Debe
3ab314d87d Fix Files for Linting (#94) 2024-09-13 16:51:09 +02:00
98 changed files with 16725 additions and 4219 deletions

1
.github/workflows/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
.DS_Store

4
.github/workflows/.markdownlint.json vendored Normal file
View File

@@ -0,0 +1,4 @@
{
"MD013": false,
"MD024": false
}

View File

@@ -19,5 +19,5 @@ jobs:
- name: Markdown Linter
uses: DavidAnson/markdownlint-cli2-action@v15
with:
configFile: .github/workflows/markdownlint.json
config: .github/workflows/.markdownlint.json
globs: '**/*.md'

View File

@@ -1 +0,0 @@

View File

@@ -2,38 +2,46 @@
*NOTE*: This repo is WIP. We are currently restructuring the RFC process.
This repository contains specifications from the [Waku](https://waku.org/), [Nomos](https://nomos.tech/),
[Codex](https://codex.storage/), and [Status](https://status.app/) projects that are part of the [IFT portfolio](https://free.technology/).
[Vac](https://vac.dev) is an [IFT service](https://free.technology/services) that will manage the RFC, [Request for Comments](https://en.wikipedia.org/wiki/Request_for_Comments), process within this repository.
This repository contains specifications from the [Waku](https://waku.org/), [Nomos](https://nomos.tech/),
[Codex](https://codex.storage/), and
[Status](https://status.app/) projects that are part of the [IFT portfolio](https://free.technology/).
[Vac](https://vac.dev) is an
[IFT service](https://free.technology/services) that will manage the RFC,
[Request for Comments](https://en.wikipedia.org/wiki/Request_for_Comments),
process within this repository.
## New RFC Process
This repository replaces the previous `rfc.vac.dev` resource.
Each project will maintain initial specifications in separate repositories,
Each project will maintain initial specifications in separate repositories,
which may be considered as a **raw** specification.
All [Vac](https://vac.dev) **raw** specifications and discussions will live in the Vac subdirectory.
When projects have reached some level of maturity for a specification living in their repository,
All [Vac](https://vac.dev) **raw** specifications and
discussions will live in the Vac subdirectory.
When projects have reached some level of maturity
for a specification living in their repository,
the process of updating the status to **draft** may begin in this repository.
Specifications will adhere to [1/COSS](./vac/1/coss.md) before obtaining **draft** status.
Specifications will adhere to
[1/COSS](./vac/1/coss.md) before obtaining **draft** status.
Implementations should follow specifications as described,
and all contributions will be discussed before the **stable** status is obtained.
The goal of this RFC process will to engage all interseted parities and
The goal of this RFC process will to engage all interseted parities and
reach a rough consensus for techcinal specifications.
## Contributing
Please see [1/COSS](./vac/1/coss.md) for general guidelines and specification lifecycle.
Feel free to join the [Vac discord](https://discord.gg/Vy54fEWuqC).
Feel free to join the [Vac discord](https://discord.gg/Vy54fEWuqC).
Here's the project board used by core contributors and maintainers: [Projects](https://github.com/orgs/vacp2p/projects/5)
## IFT Projects' Raw Specifications
The repository for each project **raw** specifications:
- [Vac Raw Specifications](./vac/raw)
- [Status Raw Specifications](./status/raw)
- [Waku Raw Specificiations](https://github.com/waku-org/specs/tree/master)
- [Codex Raw Specifications]()
- [Codex Raw Specifications](none)
- [Nomos Raw Specifications](https://github.com/logos-co/nomos-specs)

View File

@@ -1,4 +1,5 @@
# Codex RFCs
Specifications related the Codex decentralised data storage platform.
Visit [Codex specs](https://github.com/codex-storage/codex-spec) to view the new Codex specifications currently under discussion.
Visit [Codex specs](https://github.com/codex-storage/codex-spec)
to view the new Codex specifications currently under discussion.

View File

@@ -1,6 +1,6 @@
# Nomos RFCs
Nomos is building a secure, flexible, and
Nomos is building a secure, flexible, and
scalable infrastructure for developers creating applications for the network state.
To learn more about Nomos current protocols under discussion,
head over to [Nomos Specs](https://github.com/logos-co/nomos-specs).

View File

@@ -1,17 +1,17 @@
---
slug: 38
title: 38/CONSENSUS-CLARO
title: CONSENSUS-CLARO
name: Claro Consensus Protocol
status: raw
status: deprecated
category: Standards Track
tags: logos/consensus
tags:
- logos/consensus
editor: Corey Petty <corey@status.im>
created: 01-JUL-2022
revised: <2022-08-26 Fri 13:11Z>
uri: <https://rdf.logos.co/protocol/Claro/1/0/0#<2022-08-26%20Fri$2013:11Z>
contributors:
- Álvaro Castro-Castilla
- Mark Evenson
- Álvaro Castro-Castilla
- Mark Evenson
---
## Abstract
@@ -26,33 +26,80 @@ consensus mechanism. We outline a simple taxonomy of Byzantine
adversaries, leaving explicit explorations of to subsequent
publication.
NOTE: We have renamed this variant to `Claro` from `Glacier` in order to disambiguate from a previously released research endeavor by [Amores-Sesar, Cachin, and Tedeschi](https://arxiv.org/pdf/2210.03423.pdf). Their naming was coincidentally named the same as our work but is sufficiently differentiated from how ours works.
NOTE: We have renamed this variant to `Claro` from `Glacier`
in order to disambiguate from a previously released research endeavor by
[Amores-Sesar, Cachin, and Tedeschi](https://arxiv.org/pdf/2210.03423.pdf).
Their naming was coincidentally named the same as our work but
is sufficiently differentiated from how ours works.
## Motivation
This work is a part of a larger research endeavor to explore highly scalable Byzantine Fault Tolerant (BFT) consensus protocols. Consensus lies at the heart of many decentralized protocols, and thus its characteristics and properties are inherited by applications built on top. Thus, we seek to improve upon the current state of the art in two main directions: base-layer scalability and censorship resistance.
## Motivation
Avalanche has shown to exibit the former in a production environment in a way that is differentiated from Nakamoto consensus and other Proof of Stake (PoS) protocols based in practical Byzantine Fault Tolerant (pBFT) methodologies. We aim to understand its limitations and improve upon them.
This work is a part of a larger research endeavor to
explore highly scalable Byzantine Fault Tolerant (BFT) consensus protocols.
Consensus lies at the heart of many decentralized protocols, and
thus its characteristics and properties are inherited by applications built on top.
Thus, we seek to improve upon the current state of the art in two main directions:
base-layer scalability and censorship resistance.
Avalanche has shown to exibit the former in a production environment in a way
that is differentiated from Nakamoto consensus and
other Proof of Stake (PoS) protocols based in practical Byzantine Fault Tolerant
(pBFT) methodologies.
We aim to understand its limitations and improve upon them.
## Background
Our starting point is Avalanches Binary Byzantine Agreement algorithm, called Snowball. As long as modifications allow a DAG to be constructed later on, this simplifies the design significantly. The DAG stays the same in principle: it supports confidence, but the core algorithm can be modeled without.
The concept of the Snowball algorithm is relatively simple. Following is a simplified description (lacking some details, but giving an overview). For further details, please refer to the [Avalanche paper](https://assets.website-files.com/5d80307810123f5ffbb34d6e/6009805681b416f34dcae012_Avalanche%20Consensus%20Whitepaper.pdf).
Our starting point is Avalanches Binary Byzantine Agreement algorithm,
called Snowball.
As long as modifications allow a DAG to be constructed later on,
this simplifies the design significantly.
The DAG stays the same in principle: it supports confidence,
but the core algorithm can be modeled without.
1. The objective is to vote yes/no on a decision (this decision could be a single bit, or, in our DAG use case, whether a vertex should be included or not).
2. Every node has an eventually-consistent complete view of the network. It will select at random k nodes, and will ask their opinion on the decision (yes/no).
3. After this sampling is finished, if there is a vote that has more than an `alpha` threshold, it accumulates one count for this opinion, as well as changes its opinion to this one. But, if a different opinion is received, the counter is reset to 1. If no threshold `alpha` is reached, the counter is reset to 0 instead.
4. After several iterations of this algorithm, we will reach a threshold `beta`, and decide on that as final.
The concept of the Snowball algorithm is relatively simple.
Following is a simplified description (lacking some details, but giving an overview).
For further details, please refer to the [Avalanche paper](https://assets.website-files.com/5d80307810123f5ffbb34d6e/6009805681b416f34dcae012_Avalanche%20Consensus%20Whitepaper.pdf).
Next, we will proceed to describe our new algorithm, based on Snowball.
1. The objective is to vote yes/no on a decision
(this decision could be a single bit or,
in our DAG use case, whether a vertex should be included or not).
2. Every node has an eventually-consistent complete view of the network.
It will select at random k nodes, and
will ask their opinion on the decision (yes/no).
3. After this sampling is finished,
if there is a vote that has more than an `alpha` threshold,
it accumulates one count for this opinion,
as well as changes its opinion to this one.
But, if a different opinion is received, the counter is reset to 1.
If no threshold `alpha` is reached, the counter is reset to 0 instead.
4. After several iterations of this algorithm,
we will reach a threshold `beta`, and decide on that as final.
We have identified a shortcoming of the Snowball algorithm that was a perfect starting point for devising improvements. The scenario is as follows:
Next, we will proceed to describe our new algorithm, based on Snowball.
- There is a powerful adversary in the network, that controls a large percentage of the node population: 10% to ~50%.
- This adversary follows a strategy that allows them to rapidly change the decision bit (possibly even in a coordinated way) so as to maximally confuse the honest nodes.
- Under normal conditions, honest nodes will accumulate supermajorities soon enough, and reach the `beta` threshold. However, when an honest node performs a query and does not reach the threshold `alpha` of responses, the counter will be set to 0.
- The highest threat to Snowball is an adversary that keeps it from reaching the `beta` threshold, managing to continuously reset the counter, and steering Snowball away from making a decision.
We have identified a shortcoming of the Snowball algorithm
that was a perfect starting point for devising improvements.
The scenario is as follows:
This document only outlines the specification to Claro. Subsequent analysis work on Claro (both on its performance and how it differentiates with Snowball) will be published shortly and this document will be updated.
- There is a powerful adversary in the network,
that controls a large percentage of the node population: 10% to ~50%.
- This adversary follows a strategy that allows them to
rapidly change the decision bit
(possibly even in a coordinated way) so as to maximally confuse the honest nodes.
- Under normal conditions,
honest nodes will accumulate supermajorities soon enough, and
reach the `beta` threshold.
However, when an honest node performs a query and does not reach the threshold
`alpha` of responses, the counter will be set to 0.
- The highest threat to Snowball is an adversary
that keeps it from reaching the `beta` threshold,
managing to continuously reset the counter, and
steering Snowball away from making a decision.
This document only outlines the specification to Claro.
Subsequent analysis work on Claro
(both on its performance and how it differentiates with Snowball)
will be published shortly and this document will be updated.
## Claro Algorithm Specification
@@ -63,29 +110,64 @@ finality that provides good reliability for network and Byzantine
fault tolerance.
### Algorithmic concept
Claro is an evolution of the Snowball Byzantine Binary Agreement (BBA) algorithm, in which we tackle specifically the perceived weakness described above. The main focus is going to be the counter and the triggering of the reset. Following, we elaborate the different modifications and features that have been added to the reference algorithm:
1. Instead of allowing the latest evidence to change the opinion completely, we take into account all accumulated evidence, to reduce the impact of high variability when there is already a large amount of evidence collected.
2. Eliminate the counter and threshold scheme, and introduce instead two regimes of operation:
- One focused on grabbing opinions and reacting as soon as possible. This part is somewhat closer conceptually to the reference algorithm.
- Another one focused on interpreting the accumulated data instead of reacting to the latest information gathered.
3. Finally, combine those two phases via a transition function. This avoids the creation of a step function, or a sudden change in behavior that could complicate analysis and understanding of the dynamics. Instead, we can have a single algorithm that transfers weight from one operation to the other as more evidence is gathered.
4. Additionally, we introduce a function for weighted sampling. This will allow the combination of different forms of weighting:
Claro is an evolution of the Snowball Byzantine Binary Agreement (BBA) algorithm,
in which we tackle specifically the perceived weakness described above.
The main focus is going to be the counter and the triggering of the reset.
Following, we elaborate the different modifications and
features that have been added to the reference algorithm:
1. Instead of allowing the latest evidence to change the opinion completely,
we take into account all accumulated evidence,
to reduce the impact of high variability when there is already a
large amount of evidence collected.
2. Eliminate the counter and threshold scheme,
and introduce instead two regimes of operation:
- One focused on grabbing opinions and reacting as soon as possible.
This part is somewhat closer conceptually to the reference algorithm.
- Another one focused on interpreting the accumulated data
instead of reacting to the latest information gathered.
3. Finally, combine those two phases via a transition function.
This avoids the creation of a step function, or
a sudden change in behavior that could complicate analysis and
understanding of the dynamics.
Instead, we can have a single algorithm that transfers weight
from one operation to the other as more evidence is gathered.
4. Additionally, we introduce a function for weighted sampling.
This will allow the combination of different forms of weighting:
- Staking
- Heuristic reputation
- Manual reputation.
Its worth delving a bit into the way the data is interpreted in order to reach a decision. Our approach is based conceptually on the paper [Confidence as Higher-Order Uncertainty](https://cis.temple.edu/~pwang/Publication/confidence.pdf), which describes a frequentist approach to decision certainty. The first-order certainty, measured by frequency, is caused by known positive evidence, and the higher-order certainty is caused by potential positive evidence. Because confidence is a relative measurement defined on evidence, it naturally follows comparing the amount of evidence the system knows with the amount that it will know in the near future (defining “near” as a constant).
Its worth delving a bit into the way the data is interpreted
in order to reach a decision.
Our approach is based conceptually on the paper [Confidence as Higher-Order Uncertainty](https://cis.temple.edu/~pwang/Publication/confidence.pdf),
which describes a frequentist approach to decision certainty.
The first-order certainty, measured by frequency,
is caused by known positive evidence, and
the higher-order certainty is caused by potential positive evidence.
Because confidence is a relative measurement defined on evidence,
it naturally follows comparing the amount of evidence the system knows
with the amount that it will know in the near future (defining “near” as a constant).
Intuitively, we are looking for a function of evidence, **`w`**, call it **`c`** for confidence, that satisfies the following conditions:
Intuitively, we are looking for a function of evidence, **`w`**,
call it **`c`** for confidence, that satisfies the following conditions:
1. Confidence `c` is a continuous and monotonically increasing function of `w`. (More evidence, higher confidence.)
1. Confidence `c` is a continuous and monotonically increasing function of `w`.
(More evidence, higher confidence.)
2. When `w = 0`, `c = 0`. (Without any evidence, confidence is minimum.)
3. When `w` goes to infinity, `c` converges to 1. (With infinite evidence, confidence is maximum.)
3. When `w` goes to infinity, `c` converges to 1.
(With infinite evidence, confidence is maximum.)
The paper describes also a set of operations for the evidence/confidence pairs, so that different sources of knowledge could be combined. However, we leave here the suggestion of a possible research line in the future combining an algebra of evidence/confidence pairs with swarm-propagation algorithm like the one described in [this paper](http://replicated.cc/files/schmebulock.pdf).
The paper describes also a set of operations for the evidence/confidence pairs,
so that different sources of knowledge could be combined.
However, we leave here the suggestion of a possible research line in the future
combining an algebra of evidence/confidence pairs with
swarm-propagation algorithm like the one described in
[this paper](http://replicated.cc/files/schmebulock.pdf).
### Initial opinion
A proposal is formulated to which consensus of truth or falsity is
desired. Each node that participates starts the protocol with an
opinion on the proposal, represented in the sequel as `NO`, `NONE`,
@@ -97,8 +179,11 @@ compute a justification of the proposal, it sets its opinion to one of
`YES` or `NO`. If it cannot form an opinion, it leaves its opinion as
`NONE`.
For now, we will ignore the proposal dissemination process and assume all nodes participating have an initial opinion to respond to within a given request. Further research will relax this assumption and analyze timing attacks on proposal propagation through the network.
For now, we will ignore the proposal dissemination process and
assume all nodes participating have an initial opinion
to respond to within a given request.
Further research will relax this assumption and
analyze timing attacks on proposal propagation through the network.
The node then participates in a number of query rounds in which it
solicits other node's opinion in query rounds. Given a set of `N`
@@ -114,6 +199,7 @@ may not have a view on the complete members participating in the
consensus on a proposal in a given round.
The algorithm is divided into 4 phases:
1. Querying
2. Computing `confidence`, `evidence`, and `accumulated evidence`
3. Transition function
@@ -141,7 +227,8 @@ final opinion on the truth of the proposal. -->
### Setup Parameters
The node initializes the following integer ratios as constants:
```
``` markdown
# The following values are constants chosen with justification from experiments
# performed with the adversarial models
@@ -177,10 +264,10 @@ k_initial
max_rounds ;; placeholder for simulation work, no justification yet
<-- 100
```
The following variables are needed to keep the state of Claro:
```
``` markdown
;; current number of nodes to attempt to query in a round
k
<-- k_original
@@ -196,20 +283,20 @@ round
<-- 0
```
### Phase One: Query
### Phase One: Query
A node selects `k` nodes randomly from the complete pool of peers in the
network. This query is can optionally be weighted, so the probability
of selecting nodes is proportional to their
of selecting nodes is proportional to their
Node Weighting
$$
P(i) = \frac{w_i}{\sum_{j=0}^{j=N} w_j}
$$
P(i) = \frac{w_i}{\sum_{j=0}^{j=N} w_j}
$$
where `w` is evidence. The list of nodes is maintained by a separate protocol (the network
layer), and eventual consistency of this knowledge in the network
where `w` is evidence.
The list of nodes is maintained by a separate protocol (the network layer),
and eventual consistency of this knowledge in the network
suffices. Even if there are slight divergences in the network view
from different nodes, the algorithm is resilient to those.
@@ -244,12 +331,15 @@ nodes queried is too high.
When the query finishes, the node now initializes the following two
values:
```markdown
new_votes
<-- |total vote replies received in this round to the current query|
positive_votes
<-- |YES votes received from the query|
```
### Phase Two: Computation
When the query returns, three ratios are used later on to compute the
transition function and the opinion forming. Confidence encapsulates
the notion of how much we know (as a node) in relation to how much we
@@ -270,15 +360,20 @@ $$
Computation
$$
\begin{array}{lc}
\text{Confidence} & c_{accum} \impliedby \frac{total\ votes}{total\ votes + l} \newline
\text{Total accumulated evidence}& e_{accum} \impliedby \frac{total\ positive\ votes}{total\ votes} \newline
\text{Evidence per round} & e_{round} \impliedby \frac{round\ positive\ votes}{round\ votes} \newline
\text{Confidence} & c_{accum} \impliedby \frac{total\ votes}
{total\ votes + l} \newline
\text{Total accumulated evidence}& e_{accum} \impliedby \frac{total\ positive\
votes}{total\ votes} \newline
\text{Evidence per round} & e_{round} \impliedby \frac{round\ positive\
votes}{round\ votes} \newline
\end{array}
$$
The node runs the `new_votes` and `positive_votes` parameters received
in the query round through the following algorithm:
```markdown
total_votes
+== new_votes
total_positive
@@ -293,8 +388,10 @@ in the query round through the following algorithm:
<-- new_evidence * ( 1 - confidence ) + total_evidence * confidence
alpha
<-- doubt * ( 1 - confidence ) + certainty * confidence
```
### Phase Three: Computation
In order to eliminate the need for a step function (a conditional in
the code), we introduce a transition function from one regime to the
other. Our interest in removing the step function is twofold:
@@ -314,9 +411,9 @@ $$
\begin{array}{cl}
evidence & \impliedby e_{round} (1 - c_{accum}) + e_{accum} c_{accum} \newline
\alpha & \impliedby \alpha_1 (1 - c_{accum}) + \alpha_2 c_{accum} \newline
\end{array}
\end{array}
$$
Since the confidence is modeled as a ratio that depends on the
constant *`l`*, we can visualize the transition function at
different values of *`l`*. Recall that this constant encapsulates
@@ -327,11 +424,11 @@ valuable input of evidence to happen.
We have observed via experiment that for a transition function to be
useful, we need establish two requirements:
1. The change has to be balanced and smooth, giving an
1. The change has to be balanced and smooth, giving an
opportunity to the first regime to operate and not jump directly
to the second regime.
2. The convergence to 1.0 (fully operating in the second regime)
2. The convergence to 1.0 (fully operating in the second regime)
should happen within a reasonable time-frame. Weve set this
time-frame experimentally at 1000 votes, which is in the order of
~100 queries given a *`k`* of 9.
@@ -344,6 +441,7 @@ The node updates its local opinion on the consensus proposal by
examining the relationship between the evidence accumulated for a
proposal with the confidence encoded in the `alpha` parameter:
```php
IF
evidence > alpha
THEN
@@ -352,12 +450,15 @@ proposal with the confidence encoded in the `alpha` parameter:
evidence < 1 - alpha
THEN
opinion <-- NO
```
If the opinion of the node is `NONE` after evaluating the relation
between `evidence` and `alpha`, adjust the number of uniform randomly
queried nodes by multiplying the neighbors `k` by the `k_multiplier`
up to the limit of `k_max_multiplier_power` query size increases.
```php
;; possibly increase number nodes to uniformly randomly query in next round
WHEN
opinion is NONE
@@ -365,8 +466,10 @@ up to the limit of `k_max_multiplier_power` query size increases.
k < k_original * k_multiplier ^ max_k_multiplier_power
THEN
k <-- k * k_multiplier
```
### Decision
### Decision
The next step is a simple one: change our opinion if the threshold
*`alpha`* is reached. This needs to be done separately for the `YES/NO`
decision, checking both boundaries. The last step is then to *`decide`*
@@ -389,7 +492,9 @@ network size and directly related to the total votes received, an
honest node marks the decision as final, and always returns this
opinion is response to further queries from other nodes on the
network.
```php
IF
confidence > confidence_threshold
OR
@@ -400,27 +505,27 @@ network.
ELSE
round +== 1
QUERY LOOP CONTINUES
```
Thus, after the decision phase, either a decision has been finalized
and the local node becomes quiescent never initiating a new query, or
it initiates a [new query](#query).
it initiates a [new query](query).
### Termination
A local round of Claro terminates in one of the following
execution model considerations:
1. No queries are received for any newly initiated round for temporal
1. No queries are received for any newly initiated round for temporal
periods observed via a locally computed passage of time. See [the
following point on local time](#clock).
2. The `confidence` on the proposal exceeds our threshold for
2. The `confidence` on the proposal exceeds our threshold for
finalization.
3. The number of `rounds` executed would be greater than
`max_rounds`.
3. The number of `rounds` executed would be greater than
`max_rounds`.
#### Quiescence
After a local node has finalized an `opinion` into a `decision`, it enters a quiescent
@@ -438,6 +543,7 @@ of a phase locked-loop feedback to measure local clock drift see
## Further points
### Node receives information during round
In the query step, the node is envisioned as packing information into
the query to cut down on the communication overhead a query to each of
this `k` nodes containing the node's own current opinion on the
@@ -448,24 +554,27 @@ active round, and discard the information if the node is in a
quiescent state.
#### Problems with Weighting Node Value of Opinions
If the view of other nodes is incomplete, then the sum of the optional
weighting will not be a probability distribution normalized to 1.
The current algorithm doesn't describe how the initial opinions are formed.
## Implementation status
The following implementations have been created for various testing and simulation purposes:
- [Rust](https://github.com/logos-co/consensus-research)
- [Python]() - FILL THIS IN WITH NEWLY CREATED REPO
- [Common Lisp]() - FILL THIS IN WITH NEWLY CREATED REPO
## Wire Protocol
The following implementations have been created for various testing and
simulation purposes:
- [Rust](https://github.com/logos-co/consensus-research)
- [Python](none) - FILL THIS IN WITH NEWLY CREATED REPO
- [Common Lisp](none) - FILL THIS IN WITH NEWLY CREATED REPO
## Wire Protocol
For interoperability we present a wire protocol semantics by requiring
the validity of the following statements expressed in Notation3 (aka
`n3`) about any query performed by a query node:
```n3
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@@ -515,18 +624,17 @@ canonical mapping to UTF-8 JSON.
At their core, the query messages are a simple enumeration of the
three possible values of the opinion:
{ NO, NONE, YES }
> { NO, NONE, YES }
When represented via integers, such as choosing
{ -1, 0, +1 }
When represented via integers, such as choosing
> { -1, 0, +1 }
the parity summations across network invariants often become easier to
manipulate.
## Security Considerations
### Privacy
In practice, each honest node gossips its current opinion which
@@ -534,7 +642,7 @@ reduces the number of messages that need to be gossiped for a given
proposal. The resulting impact on the privacy of the node's opinion
is not currently analyzed.
### Security with respect to various Adversarial Models
### Security with respect to various Adversarial Models
Adversarial models have been tested for which the values for current
parameters of Claro have been tuned. Exposition of the
@@ -585,7 +693,7 @@ Although we have proposed a normative description of the
implementation of the underlying binary consensus algorithm (Claro),
we believe we have prepared for analysis its adversarial performance
in a manner that is amenable to replacement by another member of the
[snow*](#snow*) family.
[snow*](snow) family.
We have presumed the existence of a general family of algorithms that
can be counted on to vote on nodes in the DAG in a fair manner.
@@ -594,7 +702,7 @@ transactions. One can express all state machine, i.e. account-based
models as checkpoints anchored in UTXO trust, so we believe that this
presupposition has some justification. We can envision a need for
tooling abstraction that allow one to just program the DAG itself, as
they should be of stable interest no matter if Claro isn't.
they should be of stable interest no matter if Claro isn't.
## Informative References
@@ -607,15 +715,15 @@ they should be of stable interest no matter if Claro isn't.
3. [snow*](<https://www.avalabs.org/whitepapers>) The Snow family of
algorithms
4. [Move](<https://cloud.google.com/composer/docs/how-to/using/writing-dags>)
Move: a Language for Writing DAG Abstractions
Move: a Language for Writing DAG Abstractions
5. [rdf](<http://www.w3.org/1999/02/22-rdf-syntax-ns#>)
6. [rdfs](<http://www.w3.org/2000/01/rdf-schema#>)
7. [xsd](<http://www.w3.org/2001/XMLSchema#>)
7. [xsd](<http://www.w3.org/2001/XMLSchema#>)
8. [n3-w3c-notes](<https://www.w3.org/TeamSubmission/n3/>)
@@ -632,4 +740,4 @@ they should be of stable interest no matter if Claro isn't.
## Copyright
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
[CC0](https://creativecommons.org/public)

View File

@@ -9,44 +9,61 @@ editor: Szymon Szlachtowicz <szymon.s@ethworks.io>
---
## Abstract
This specification is a voting protocol for peers to submit votes to a smart contract. Voting is immutable,
This specification is a voting protocol for peers to submit votes to a smart contract.
Voting is immutable,
this will help avoid sabotage from malicious peers.
## Motivation
In open p2p protocol there is an issue with voting off-chain as there is much room for malicious peers to only include votes that support their case when submitting votes to chain.
In open p2p protocol there is an issue with voting off-chain
as there is much room for malicious peers to only include votes that support
their case when submitting votes to chain.
Proposed solution is to aggregate votes over waku and allow users to submit votes to smart contract that aren't already submitted.
Proposed solution is to aggregate votes over waku and
allow users to submit votes to smart contract that aren't already submitted.
### Smart contract
Voting should be finalized on chain so that the finished vote is immutable.
Because of that, smart contract needs to be deployed.
When votes are submitted smart contract has to verify what votes are properly signed and that sender has correct amount of SNT.
When Vote is verified the amount of SNT voted on specific topic by specific sender is saved on chain.
When votes are submitted
smart contract has to verify what votes are properly signed and
that sender has correct amount of SNT.
When Vote is verified
the amount of SNT voted on specific topic by specific sender is saved on chain.
### Double voting
Smart contract should also keep a list of all signatures so that no one can send the same vote twice.
Smart contract should also keep a list of all signatures so
that no one can send the same vote twice.
Another possibility is to allow each sender to only vote once.
### Initializing Vote
When someone wants to initialize vote he has to send a transaction to smart contract that will create a new voting session.
When initializing a user has to specify type of vote (Addition, Deletion), amount of his initial SNT to submit and public key of community under vote.
When someone wants to initialize vote
he has to send a transaction to smart contract that will create a new voting session.
When initializing a user has to specify type of vote (Addition, Deletion),
amount of his initial SNT to submit and public key of community under vote.
Smart contract will return a ID which is identifier of voting session.
Also there will be function on Smart Contract that when given community public key it will return voting session ID or undefined if community isn't under vote.
Also there will be function on Smart Contract that
when given community public key it will return voting session ID or
undefined if community isn't under vote.
## Voting
### Sending votes
Sending votes is simple every peer is able to send a message to Waku topic specific to given application:
```
Sending votes is simple every peer is able to send a message to Waku topic
specific to given application:
```json
/status-community-directory-curation-vote/1/{voting-session-id}/json
```
vote object that is sent over waku should contain information about:
vote object that is sent over waku should contain information about:
```ts
type Vote = {
@@ -54,28 +71,36 @@ type Vote = {
vote: string // vote sent eg. 'yes' 'no'
sntAmount: BigNumber //number of snt cast on vote
sign: string // cryptographic signature of a transaction (signed fields: sender,vote,sntAmount,nonce,sessionID)
nonce: number // number of votes cast from this address on current vote (only if we allow multiple votes from the same sender)
nonce: number // number of votes cast from this address on current vote
// (only if we allow multiple votes from the same sender)
sessionID: number // ID of voting session
}
```
### Aggregating votes
Every peer that is opening specific voting session will listen to votes sent over p2p network, and aggregate them for a single transaction to chain.
Every peer that is opening specific voting session
will listen to votes sent over p2p network, and
aggregate them for a single transaction to chain.
### Submitting to chain
Every peer that has aggregated at least one vote will be able to send them to smart contract.
When someone votes he will aggregate his own vote and will be able to immediately send it.
Every peer that has aggregated at least one vote
will be able to send them to smart contract.
When someone votes he will aggregate his own vote and
will be able to immediately send it.
Peer doesn't need to vote to be able to submit the votes to the chain.
Smart contract needs to verify that all votes are valid (eg. all senders had enough SNT, all votes are correctly signed) and that votes aren't duplicated on smart contract.
Smart contract needs to verify that all votes are valid
(eg. all senders had enough SNT, all votes are correctly signed) and
that votes aren't duplicated on smart contract.
### Finalizing
### Finalizing
Once the vote deadline has expired, the smart contract will not accept votes anymore.
Also directory will be updated according to vote results (community added to directory, removed etc.)
Also directory will be updated according to vote results
(community added to directory, removed etc.)
## Copyright

View File

@@ -9,26 +9,32 @@ editor: Szymon Szlachtowicz <szymon.s@ethworks.io>
---
## Abstract
This specification describes a voting method to feature different active Status Communities.
## Overview
When there is a active community that is seeking new members, current users of community should be able to feature their community so that it will be accessible to larger audience.
When there is a active community that is seeking new members,
current users of community should be able to feature their community so
that it will be accessible to larger audience.
Status community curation DApp should provide such a tool.
Rules of featuring:
- Given community can't be featured twice in a row.
- Only one vote per user per community (single user can vote on multiple communities)
- Voting will be done off-chain
- If community hasn't been featured votes for given community are still valid for the next 4 weeks
- If community hasn't been featured
votes for given community are still valid for the next 4 weeks
Since voting for featuring is similar to polling solutions proposed in this spec could be also used for different applications.
Since voting for featuring is similar to polling solutions proposed
in this spec could be also used for different applications.
### Voting
Voting for featuring will be done through waku v2.
Voting for featuring will be done through waku v2.
Payload of waku message will be :
```ts
type FeatureVote = {
voter: string // address of a voter
@@ -44,12 +50,16 @@ timestamp is necessary so that votes can't be reused after 4 week period
### Counting Votes
Votes will be counted by the DApp itself.
DApp will aggregate all the votes in the last 4 weeks and calculate which communities should be displayed in the Featured tab of DApp.
DApp will aggregate all the votes in the last 4 weeks and
calculate which communities should be displayed in the Featured tab of DApp.
Rules of counting:
- When multiple votes from the same address on the same community are encountered only the vote with highest timestamp is considered valid.
- If a community has been featured in a previous week it can't be featured in current week.
- In a current week top 5 (or 10) communities with highest amount of SNT votes up to previous Sunday 23:59:59 UTC are considered featured.
- When multiple votes from the same address on the same community are encountered
only the vote with highest timestamp is considered valid.
- If a community has been featured in a previous week
it can't be featured in current week.
- In a current week top 5 (or 10) communities with highest amount of SNT votes
up to previous Sunday 23:59:59 UTC are considered featured.
## Copyright

View File

@@ -17,7 +17,8 @@ contributors:
## Abstract
This specification describes how the Status 1-to-1 chat protocol is implemented on top of the Waku v2 protocol.
This specification describes how the Status 1-to-1 chat protocol is implemented
on top of the Waku v2 protocol.
This protocol can be used to send messages to a single recipient.
## Terminology
@@ -27,11 +28,13 @@ This protocol can be used to send messages to a single recipient.
- **Public chat**: A chat where any participant can join and read messages.
- **Private chat**: A chat where only invited participants can join and read messages.
- **Group chat**: A chat where multiple select participants can join and read messages.
- **Group admin**: A participant that is able to add/remove participants from a group chat.
- **Group admin**: A participant that is able to
add/remove participants from a group chat.
## Background
This document describes how 2 peers communicate with each other to send messages in a 1-to-1 chat, with privacy and authenticity guarantees.
This document describes how 2 peers communicate with each other
to send messages in a 1-to-1 chat, with privacy and authenticity guarantees.
## Specification
@@ -39,47 +42,61 @@ This document describes how 2 peers communicate with each other to send messages
This protocol MAY use any key-exchange mechanism previously discussed -
1. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md)
1. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md)
2. [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md)
This protocol can provide end-to-end encryption to give peers a strong degree of privacy and security.
Public chat messages are publicly readable by anyone since there's no permission model for who is participating in a public chat.
This protocol can provide end-to-end encryption
to give peers a strong degree of privacy and security.
Public chat messages are publicly readable by anyone since
there's no permission model for who is participating in a public chat.
## Flow
## Chat Flow
### Negotiation of a 1:1 chat
There are two phases in the initial negotiation of a 1:1 chat:
1. **Identity verification** (e.g., face-to-face contact exchange through QR code, Identicon matching).
A QR code serves two purposes simultaneously - identity verification and initial key material retrieval;
1. **Identity verification**
(e.g., face-to-face contact exchange through QR code, Identicon matching).
A QR code serves two purposes simultaneously -
identity verification and initial key material retrieval;
1. **Asynchronous initial key exchange**
For more information on account generation and trust establishment, see [65/STATUS-ACCOUNT-ADDRESS](../65/account-address.md)
For more information on account generation and trust establishment, see [65/ACCOUNT-ADDRESS](../65/account-address.md)
### Post Negotiation
After the peers have shared their public key material, a 1:1 chat can be established using the methods described in the key-exchange protocols mentioned above.
After the peers have shared their public key material,
a 1:1 chat can be established using the methods described in the
key-exchange protocols mentioned above.
### Session management
The 1:1 chat is made robust by having sessions between peers.
It is handled by the key-exchange protocol used. For example,
1. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md), the session management is described in [54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md)
1. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md),
the session management is described in [54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md)
2. [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md), the session management is described in [WAKU2-NOISE-SESSIONS](https://github.com/waku-org/specs/blob/master/standards/application/noise-sessions.md)
2. [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md),
the session management is described in [WAKU2-NOISE-SESSIONS](https://github.com/waku-org/specs/blob/master/standards/application/noise-sessions.md)
## Negotiation of a 1:1 chat amongst multiple participants (group chat)
A small, private group chat can be constructed by having multiple participants negotiate a 1:1 chat amongst each other.
Each participant MUST maintain a session with all other participants in the group chat.
A small, private group chat can be constructed by having multiple participants
negotiate a 1:1 chat amongst each other.
Each participant MUST
maintain a session with all other participants in the group chat.
This allows for a group chat to be created with a small number of participants.
However, this method does not scale as the number of participants increases, for the following reasons -
However, this method does not scale as the number of participants increases,
for the following reasons -
1. The number of messages sent over the network increases with the number of participants.
2. Handling the X3DH key exchange for each participant is computationally expensive.
The above issues are addressed in [56/STATUS-COMMUNITIES](../56/communities.md), with other trade-offs.
The above issues are addressed in [56/STATUS-COMMUNITIES](../56/communities.md),
with other trade-offs.
### Flow
@@ -96,7 +113,8 @@ message MembershipUpdateMessage {
// chat_id = hex(chat_creator_public_key) + "-" + random_uuid
// This chat_id MUST be validated by all participants
string chat_id = 1;
// A list of events for this group chat, first 65 bytes are the signature, then is a
// A list of events for this group chat, first 65 bytes are the signature,
then is a
// protobuf encoded MembershipUpdateEvent
repeated bytes events = 2;
oneof chat_entity {
@@ -108,7 +126,8 @@ message MembershipUpdateMessage {
}
```
Note that in `events`, the first element is the signature, and all other elements after are encoded `MembershipUpdateEvent`'s.
Note that in `events`, the first element is the signature, and
all other elements after are encoded `MembershipUpdateEvent`'s.
where `MembershipUpdateEvent` is defined as follows:
@@ -141,69 +160,102 @@ message MembershipUpdateEvent {
}
}
```
<!-- Note: I don't like defining wire formats which are out of the scope of the rfc this way. Should explore alternatives -->
Note that the definitions for `ChatMessage` and `EmojiReaction` can be found in [chat_message.proto](https://github.com/status-im/status-go/blob/5fd9e93e9c298ed087e6716d857a3951dbfb3c1e/protocol/protobuf/chat_message.proto#L1) and [emoji_reaction.proto](https://github.com/status-im/status-go/blob/5fd9e93e9c298ed087e6716d857a3951dbfb3c1e/protocol/protobuf/emoji_reaction.proto).
<!-- Note:
I don't like defining wire formats which are out of the scope of the rfc this way.
Should explore alternatives -->
Note that the definitions for `ChatMessage` and
`EmojiReaction` can be found in
[chat_message.proto](https://github.com/status-im/status-go/blob/5fd9e93e9c298ed087e6716d857a3951dbfb3c1e/protocol/protobuf/chat_message.proto#L1)
and [emoji_reaction.proto](https://github.com/status-im/status-go/blob/5fd9e93e9c298ed087e6716d857a3951dbfb3c1e/protocol/protobuf/emoji_reaction.proto).
##### Chat Created
When creating a group chat, this is the first event that MUST be sent.
Any event with a clock value lower than this MUST be discarded.
Upon receiving this event a client MUST validate the `chat_id` provided with the update and create a chat with identified by `chat_id`.
When creating a group chat, this is the first event that MUST be sent.
Any event with a clock value lower than this MUST be discarded.
Upon receiving this event a client MUST validate the `chat_id`
provided with the update and
create a chat with identified by `chat_id`.
By default, the creator of the group chat is the only group admin.
##### Name Changed
To change the name of the group chat, group admins MUST use a `NAME_CHANGED` event.
Upon receiving this event a client MUST validate the `chat_id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
Upon receiving this event,
a client MUST validate the `chat_id` provided with the updates and
MUST ensure the author of the event is an admin of the chat,
otherwise the event MUST be ignored.
If the event is valid the chat name SHOULD be changed according to the provided message.
##### Members Added
To add members to the chat, group admins MUST use a `MEMBERS_ADDED` event.
Upon receiving this event a participant MUST validate the `chat_id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
If the event is valid, a participant MUST update the list of members of the chat who have not joined, adding the members received.
To add members to the chat, group admins MUST use a `MEMBERS_ADDED` event.
Upon receiving this event,
a participant MUST validate the `chat_id` provided with the updates and
MUST ensure the author of the event is an admin of the chat,
otherwise the event MUST be ignored.
If the event is valid,
a participant MUST update the list of members of the chat who have not joined,
adding the members received.
##### Member Joined
To signal the intent to start receiving messages from a given chat, new participants MUST use a `MEMBER_JOINED` event.
Upon receiving this event a participant MUST validate the `chat_id` provided with the updates.
If the event is valid a participant MUST add the new participant to the list of participants stored locally.
To signal the intent to start receiving messages from a given chat,
new participants MUST use a `MEMBER_JOINED` event.
Upon receiving this event,
a participant MUST validate the `chat_id` provided with the updates.
If the event is valid a participant,
a participant MUST add the new participant to the list of participants stored locally.
Any message sent to the group chat MUST now include the new participant.
##### Member Removed
There are two ways in which a member MAY be removed from a group chat:
- A member MAY leave the chat by sending a `MEMBER_REMOVED` event, with the `members` field containing their own public key.
- An admin MAY remove a member by sending a `MEMBER_REMOVED` event, with the `members` field containing the public key of the member to be removed.
Each participant MUST validate the `chat_id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
- A member MAY leave the chat by sending a `MEMBER_REMOVED` event,
with the `members` field containing their own public key.
- An admin MAY remove a member by sending a `MEMBER_REMOVED` event,
with the `members` field containing the public key of the member to be removed.
Each participant MUST validate the `chat_id` provided with the updates and
MUST ensure the author of the event is an admin of the chat,
otherwise the event MUST be ignored.
If the event is valid, a participant MUST update the local list of members accordingly.
##### Admins Added
To promote participants to group admin, group admins MUST use an `ADMINS_ADDED` event.
Upon receiving this event, a participant MUST validate the `chat_id` provided with the updates, MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
If the event is valid, a participant MUST update the list of admins of the chat accordingly.
Upon receiving this event,
a participant MUST validate the `chat_id` provided with the updates,
MUST ensure the author of the event is an admin of the chat,
otherwise the event MUST be ignored.
If the event is valid,
a participant MUST update the list of admins of the chat accordingly.
##### Admin Removed
Group admins MUST NOT be able to remove other group admins.
An admin MAY remove themselves by sending an `ADMIN_REMOVED` event, with the `members` field containing their own public key.
Each participant MUST validate the `chat_id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
An admin MAY remove themselves by sending an `ADMIN_REMOVED` event,
with the `members` field containing their own public key.
Each participant MUST validate the `chat_id` provided with the updates and
MUST ensure the author of the event is an admin of the chat,
otherwise the event MUST be ignored.
If the event is valid, a participant MUST update the list of admins of the chat accordingly.
##### Color Changed
To change the text color of the group chat name, group admins MUST use a `COLOR_CHANGED` event.
To change the text color of the group chat name,
group admins MUST use a `COLOR_CHANGED` event.
##### Image Changed
To change the display image of the group chat, group admins MUST use an `IMAGE_CHANGED` event.
To change the display image of the group chat,
group admins MUST use an `IMAGE_CHANGED` event.
## Security Considerations
1. Inherits the security considerations of the key-exchange mechanism used, e.g., [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) or [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md)
1. Inherits the security considerations of the key-exchange mechanism used,
e.g., [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) or [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md)
## Copyright
@@ -213,7 +265,7 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
1. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md)
2. [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md)
3. [65/STATUS-ACCOUNT-ADDRESS](../65/account-address.md)
3. [65/STATUS-ACCOUNT](../65/account-address.md)
4. [54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md)
5. [WAKU2-NOISE-SESSIONS](https://github.com/waku-org/specs/blob/master/standards/application/noise-sessions.md)
6. [56/STATUS-COMMUNITIES](../56/communities.md)

View File

@@ -9,53 +9,71 @@ description: Status Communities allow multiple users to communicate in a discuss
editor: Aaryamann Challani <p1ge0nh8er@proton.me>
contributors:
- Andrea Piana <andreap@status.im>
- Prem Chaitanya Prathi <prem@waku.org>
---
## Abstract
This document describes the design of Status Communities over Waku v2, allowing for multiple users to communicate in a discussion space.
This is a key feature for the Status messaging app.
This document describes the design of Status Communities over Waku v2,
allowing for multiple users to communicate in a discussion space.
This is a key feature for the Status messaging app.
## Background and Motivation
The purpose of Status communities, as specified in this document, is allowing for large group chats.
The purpose of Status communities, as specified in this document,
is allowing for large group chats.
Communities can have further substructure, e.g. specific channels.
Smaller group chats, on the other hand, are out of scope for this document and can be built over [55/STATUS-1TO1-CHAT](../55/1to1-chat.md).
We refer to these smaller group chats simply as "group chats", to differentiate them from Communities.
Smaller group chats, on the other hand,
are out of scope for this document and
can be built over [55/STATUS-1TO1-CHAT](../55/1to1-chat.md).
We refer to these smaller group chats simply as "group chats",
to differentiate them from Communities.
For group chats based on [55/STATUS-1TO1-CHAT](../55/1to1-chat.md), the key exchange mechanism MUST be X3DH, as described in [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md).
For group chats based on [55/STATUS-1TO1-CHAT](../55/1to1-chat.md),
the key exchange mechanism MUST be X3DH,
as described in [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md).
However, this method does not scale as the number of participants increases,
for the following reasons -
However, this method does not scale as the number of participants increases, for the following reasons -
1. The number of messages sent over the network increases with the number of participants.
2. Handling the X3DH key exchange for each participant is computationally expensive.
Having multicast channels reduces the overhead of a group chat based on 1:1 chat.
Additionally, if all the participants of the group chat have a shared key, then the number of messages sent over the network is reduced to one per message.
Additionally, if all the participants of the group chat have a shared key,
then the number of messages sent over the network is reduced to one per message.
## Terminology
- **Community**: A group of peers that can communicate with each other.
- **Member**: A peer that is part of a community.
- **Admin**: A member that has administrative privileges. Used interchangeably with "owner".
- **Admin**: A member that has administrative privileges.
Used interchangeably with "owner".
- **Channel**: A designated subtopic for a community. Used interchangeably with "chat".
## Design Requirements
Due to the nature of communities, the following requirements are necessary for the design of communities -
Due to the nature of communities,
the following requirements are necessary for the design of communities -
1. The creator of the Community is the owner of the Community.
2. The Community owner is trusted.
3. The Community owner can add or remove members from the Community.
This extends to banning and kicking members.
4. The Community owner can add, edit and remove channels.
5. Community members can send/receive messages to the channels which they have access to.
5. Community members can send/receive messages
to the channels which they have access to.
6. Communities may be encrypted (private) or unencrypted (public).
7. A Community is uniquely identified by a public key.
8. The public key of the Community is shared out of band.
9. The metadata of the Community can be found by listening on a content topic derived from the public key of the Community.
10. Community members run their own Waku nodes, with the configuration described in [Waku-Protocols](#waku-protocols).
Light nodes solely implementing [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md) may not be able to run their own Waku node with the configuration described.
9. The metadata of the Community can be found by listening on a content topic
derived from the public key of the Community.
10. Community members run their own Waku nodes,
with the configuration described in [Waku-Protocols](#waku-protocols).
Light nodes solely implementing
[19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md)
may not be able to run their own Waku node with the configuration described.
## Design
@@ -64,16 +82,17 @@ Light nodes solely implementing [19/WAKU2-LIGHTPUSH](../../waku/standards/core/1
The following cryptographic primitives are used in the design -
- X3DH
- Single Ratchet
- The single ratchet is used to encrypt the messages sent to the Community.
- The single ratchet is re-keyed when a member is added/removed from the Community.
- Single Ratchet
- The single ratchet is used to encrypt the messages sent to the Community.
- The single ratchet is re-keyed when a member is added/removed from the Community.
## Wire format
<!--
The wire format is described first to give an overview of the protocol.
It is referenced in the flow of community creation and community management.
More or less an intersection of https://github.com/status-im/specs/blob/403b5ce316a270565023fc6a1f8dec138819f4b0/docs/raw/organisation-channels.md and https://github.com/status-im/status-go/blob/6072bd17ab1e5d9fc42cf844fcb8ad18aa07760c/protocol/protobuf/communities.proto,
More or less an intersection of https://github.com/status-im/specs/blob/403b5ce316a270565023fc6a1f8dec138819f4b0/docs/raw/organisation-channels.md
and https://github.com/status-im/status-go/blob/6072bd17ab1e5d9fc42cf844fcb8ad18aa07760c/protocol/protobuf/communities.proto,
-->
@@ -88,7 +107,8 @@ message IdentityImage {
SourceType source_type = 2;
// image_type signals the image type and method of parsing the payload
ImageType image_type = 3;
// encryption_keys is a list of encrypted keys that can be used to decrypt an encrypted payload
// encryption_keys is a list of encrypted keys that can be used to decrypt an
// encrypted payload
repeated bytes encryption_keys = 4;
// encrypted signals the encryption state of the payload, default is false.
bool encrypted = 5;
@@ -101,7 +121,8 @@ message IdentityImage {
// ENS_AVATAR uses the ENS record's resolver get-text-data.avatar data
// The `payload` field will be ignored if ENS_AVATAR is selected
// The application will read and parse the ENS avatar data as image payload data, URLs will be ignored
// The application will read and
// parse the ENS avatar data as image payload data, URLs will be ignored
// The parent `ChatMessageIdentity` must have a valid `ens_name` set
ENS_AVATAR = 2;
}
@@ -129,7 +150,8 @@ message ChatIdentity {
string color = 6;
string emoji = 7;
repeated SocialLink social_links = 8;
// first known message timestamp in seconds (valid only for community chats for now)
// first known message timestamp in seconds
// (valid only for community chats for now)
// 0 - unknown
// 1 - no messages
uint32 first_message_timestamp = 9;
@@ -285,16 +307,54 @@ message CommunityDescription {
Note: The usage of the clock is described in the [Clock](#clock) section.
### Functional scope and shard assignment
We define two special [functional scopes](../raw/status-app-protocols.md#functional-scope) for messages related to Status Communities:
1. Global community control
2. Global community content
All messages that relate to controlling communities MUST be assigned the _global community control_ scope.
All messages that carry user-generated content for communities MUST be assigned the _global community content_ scope.
> **Note:** a previous iteration of Status Communities defined separate community-wide scopes for each community.
However, this model was deprecated and all communities now operate on a global, shared scope.
This implies that different communities will share shards on the routing layer.
The following [Waku transport layer](../raw/status-app-protocols.md#waku-transport-layer) allocations are reserved for communities:
As per [STATUS-SIMPLE-SCALING](https://rfc.vac.dev/status/raw/simple-scaling/#relay-shards), communities use the default cluster ID `16`
set aside for all Status app protocols.
Within this cluster, the following [shards](../raw/status-app-protocols.md#pubsub-topics-and-sharding) are reserved for the community functional scopes:
1. All messages with a _global community control_ scope MUST be published to shard `128`
2. All messages with a _global community content_ scope MUST be published to shard `256`
### Content topic level encryption
-a universal chat identifier is used for all community chats.
<!-- Don't enforce any constraints on the unique id generation -->
All messages are encrypted before they are handed over to waku ir-respective of the encryption explained above.
All community chats are encrypted using a symmetric key generated from universal chat id using pbkdf2.
```js
symKey = pbkdf2(password:universalChatID, salt:nil, iteration-count:65356,key-length:32, hash-func: random-sha256)
```
### Content topic usage
"Content topic" refers to the field in [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md/#message-attributes), further elaborated in [10/WAKU2](../../waku/standards/core/10/waku2.md/#overview-of-protocol-interaction).
"Content topic" refers to the field in [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md/#message-attributes),
further elaborated in [10/WAKU2](../../waku/standards/core/10/waku2.md/#overview-of-protocol-interaction).
The content-topic usage follows the guidelines specified at [23/topics](../../waku/informational/23/topics.md#content-topic-usage-guidelines)
#### Advertising a Community
The content topic that the community is advertised on MUST be derived from the public key of the community.
The content topic MUST be the first four bytes of the keccak-256 hash of the compressed (33 bytes) public key of the community encoded into a hex string.
The content topic that the community is advertised on
MUST be derived from the public key of the community.
The content topic MUST be the first four bytes of the keccak-256 hash
of the compressed (33 bytes) public key of the community encoded into a hex string.
```
```js
hash = hex(keccak256(encodeToHex(compressedPublicKey)))
topicLen = 4
@@ -310,34 +370,14 @@ for i = 0; i < topicLen; i++ {
contentTopic = "/waku/1/0x" + topic + "/rfc26"
```
#### Community channels/chats
The unique identifier for a community channel/chat is the chat id.
<!-- Don't enforce any constraints on the unique id generation -->
The content topic that Community channels/chats use MUST be the hex-encoded keccak-256 hash of the public key of the community concatenated with the chat id.
```
hash = hex(keccak256(encodeToHex(compressedPublicKey + chatId)))
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
contentTopic = "/waku/1/0x" + topic + "/rfc26"
```
#### Community event messages
Requests to leave, join, kick and ban, as well as key exchange messages, MUST be sent to the content topic derived from the public key of the community.
The content topic MUST be the hex-encoded keccak-256 hash of the public key of the community.
Message such as community description
MUST be sent to the content topic derived from the public key of the community.
The content topic
MUST be the hex-encoded keccak-256 hash of the public key of the community.
```
```js
hash = hex(keccak256(encodeToHex(publicKey)))
topicLen = 4
@@ -350,7 +390,54 @@ for i = 0; i < topicLen; i++ {
}
contentTopic = "/waku/1/0x" + topic + "/rfc26"
```
```
#### Community Requests
Requests to leave, join, kick and ban, as well as key exchange messages, MUST be sent to the content topic derived from the public key of the community on the common shard.
The content topic
MUST be the keccak-256 hash of hex-encoded universal chat id (public key appended with fixed string) of the community omitting the first 2 bytes.
```js
universalChatId = publicKey+"-memberUpdate"
hash = hex(keccak256(encodeToHex(universalChatId))[2:])
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
contentTopic = "/waku/1/0x" + topic + "/rfc26"
```
#### Community Shard Info
If a community is assigned a dedicated shard then the shard info for that community is published on a content topic derived from a specialized key. This is useful for users joining the new community so that they can subscribe to this specific content topic.
```js
chatID = publicKey+"-shard-info"
hash = hex(keccak256(encodeToHex(chatID))[2:])
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
contentTopic = "/waku/1/0x" + topic + "/rfc26"
```
#### Community channels/chats
All channels/chats shall use a single content-topic which is derived from a universal chat id irrespective of their individual unique chat ids.
### Community Management
@@ -359,84 +446,122 @@ The flows for Community management are as described below.
#### Community Creation Flow
1. The Community owner generates a public/private key pair.
2. The Community owner configures the Community metadata, according to the wire format "CommunityDescription".
3. The Community owner publishes the Community metadata on a content topic derived from the public key of the Community.
the Community metadata SHOULD be encrypted with the public key of the Community. <!-- TODO: Verify this-->
The Community metadata MAY be sent during fixed intervals, to ensure that the Community metadata is available to members.
2. The Community owner configures the Community metadata,
according to the wire format "CommunityDescription".
3. The Community owner publishes the Community metadata on a content topic
derived from the public key of the Community.
the Community metadata SHOULD be encrypted with the public key of the Community.
The Community metadata is sent during fixed intervals,
to ensure that the Community metadata is available to members.
The Community metadata SHOULD be sent every time the Community metadata is updated.
4. The Community owner MAY advertise the Community out of band, by sharing the public key of the Community on other mediums of communication.
4. The Community owner MAY advertise the Community out of band,
by sharing the public key of the Community on other mediums of communication.
#### Community Join Flow (peer requests to join a Community)
1. A peer and the Community owner establish a 1:1 chat as described in [55/STATUS-1TO1-CHAT](../55/1to1-chat.md).
2. The peer requests to join a Community by sending a "CommunityRequestToJoin" message to the Community.
At this point, the peer MAY send a "CommunityCancelRequestToJoin" message to cancel the request.
2. The peer requests to join a Community by sending a
"CommunityRequestToJoin" message to the Community.
At this point, the peer MAY send a
"CommunityCancelRequestToJoin" message to cancel the request.
3. The Community owner MAY accept or reject the request.
4. If the request is accepted, the Community owner sends a "CommunityRequestToJoinResponse" message to the peer.
5. The Community owner then adds the member to the Community metadata, and publishes the updated Community metadata.
4. If the request is accepted,
the Community owner sends a "CommunityRequestToJoinResponse" message to the peer.
5. The Community owner then adds the member to the Community metadata, and
publishes the updated Community metadata.
#### Community Join Flow (peer is invited to join a Community)
1. The Community owner and peer establish a 1:1 chat as described in [55/STATUS-1TO1-CHAT](../55/1to1-chat.md).
2. The peer is invited to join a Community by the Community owner, by sending a "CommunityInvitation" message.
2. The peer is invited to join a Community by the Community owner,
by sending a "CommunityInvitation" message.
3. The peer decrypts the "CommunityInvitation" message, and verifies the signature.
4. The peer requests to join a Community by sending a "CommunityRequestToJoin" message to the Community.
4. The peer requests to join a Community by sending a
"CommunityRequestToJoin" message to the Community.
5. The Community owner MAY accept or reject the request.
6. If the request is accepted, the Community owner sends a "CommunityRequestToJoinResponse" message to the peer.
7. The Community owner then adds the member to the Community metadata, and publishes the updated Community metadata.
6. If the request is accepted,
the Community owner sends a "CommunityRequestToJoinResponse" message to the peer.
7. The Community owner then adds the member to the Community metadata, and
publishes the updated Community metadata.
#### Community Leave Flow
1. A member requests to leave a Community by sending a "CommunityRequestToLeave" message to the Community.
1. A member requests to leave a Community by sending a
"CommunityRequestToLeave" message to the Community.
2. The Community owner MAY accept or reject the request.
3. If the request is accepted, the Community owner removes the member from the Community metadata, and publishes the updated Community metadata.
3. If the request is accepted,
the Community owner removes the member from the Community metadata,
and publishes the updated Community metadata.
#### Community Ban Flow
1. The Community owner adds a member to the ban list, revokes their grants, and publishes the updated Community metadata.
2. If the Community is Private, Re-keying is performed between the members of the Community, to ensure that the banned member is unable to decrypt any messages.
1. The Community owner adds a member to the ban list, revokes their grants,
and publishes the updated Community metadata.
2. If the Community is Private,
Re-keying is performed between the members of the Community,
to ensure that the banned member is unable to decrypt any messages.
### Waku Protocols
### Waku Protocols
The following Waku protocols SHOULD be used to implement Status Communities -
1. [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md) - To send and receive messages
2. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) - To encrypt and decrypt messages
3. [54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md) - To handle session keys
4. [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) - To wrap community messages in a Waku message
5. [13/WAKU2-STORE](../../waku/standards/core/13/store.md) - To store and retrieve messages for offline devices
1. [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md) -
To send and receive messages
2. [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) -
To encrypt and decrypt messages
3. [54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md)-
To handle session keys
4. [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) -
To wrap community messages in a Waku message
5. [13/WAKU2-STORE](../../waku/standards/core/13/store.md) -
To store and retrieve messages for offline devices
The following Waku protocols MAY be used to implement Status Communities -
1. [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md) - Content filtering for resource restricted devices
2. [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md) - Allows Light clients to participate in the network
1. [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md) -
Content filtering for resource restricted devices
2. [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md) -
Allows Light clients to participate in the network
### Backups
The member MAY back up their local settings, by encrypting it with their public key, and sending it to a given content topic.
The member MAY then rely on this backup to restore their local settings, in case of a data loss.
This feature relies on [13/WAKU2-STORE](../../waku/standards/core/13/store.md) for storing and retrieving messages.
The member MAY back up their local settings,
by encrypting it with their public key, and
sending it to a given content topic.
The member MAY then rely on this backup to restore their local settings,
in case of a data loss.
This feature relies on
[13/WAKU2-STORE](../../waku/standards/core/13/store.md)
for storing and retrieving messages.
### Clock
The clock used in the wire format refers to the Lamport timestamp of the message.
The Lamport timestamp is a logical clock that is used to determine the order of events in a distributed system.
This allows ordering of messages in an asynchronous network where messages may be received out of order.
The Lamport timestamp is a logical clock that is used to determine the order of events
in a distributed system.
This allows ordering of messages in an asynchronous network
where messages may be received out of order.
## Security Considerations
1. The Community owner is a single point of failure. If the Community owner is compromised, the Community is compromised.
1. The Community owner is a single point of failure.
If the Community owner is compromised, the Community is compromised.
2. Follows the same security considerations as the [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) protocol.
2. Follows the same security considerations as the
[53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) protocol.
## Future work
1. To scale and optimize the Community management, the Community metadata should be stored on a decentralized storage system, and only the references to the Community metadata should be broadcasted. The following document describes this method in more detail - [Optimizing the `CommunityDescription` dissemination](https://hackmd.io/rD1OfIbJQieDe3GQdyCRTw)
1. To scale and optimize the Community management,
the Community metadata should be stored on a decentralized storage system, and
only the references to the Community metadata should be broadcasted.
The following document describes this method in more detail -
[Optimizing the `CommunityDescription` dissemination](https://hackmd.io/rD1OfIbJQieDe3GQdyCRTw)
2. Token gating for communities
3. Sharding the content topic used for [#Community Event Messages](#community-event-messages), since members of the community don't need to receive all the control messages.
3. Sharding the content topic used for [#Community Event Messages](#community-event-messages),
since members of the community don't need to receive all the control messages.
## Copyright
@@ -455,5 +580,6 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
- [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md)
### informative
- [community.go](https://github.com/status-im/status-go/blob/6072bd17ab1e5d9fc42cf844fcb8ad18aa07760c/protocol/communities/community.go)
- [organisation-channels.md](https://github.com/status-im/specs/blob/403b5ce316a270565023fc6a1f8dec138819f4b0/docs/raw/organisation-channels.md)

View File

@@ -13,16 +13,31 @@ contributors:
## Abstract
Messages are stored permanently by store nodes ([13/WAKU2-STORE](../../waku/standards/core/13/store.md)) for up to a certain configurable period of time, limited by the overall storage provided by a store node.
Messages older than that period are no longer provided by store nodes, making it impossible for other nodes to request historical messages that go beyond that time range.
This raises issues in the case of Status communities, where recently joined members of a community are not able to request complete message histories of the community channels.
Messages are stored permanently by store nodes
([13/WAKU2-STORE](../../waku/standards/core/13/store.md))
for up to a certain configurable period of time,
limited by the overall storage provided by a store node.
Messages older than that period are no longer provided by store nodes,
making it impossible for other nodes to request historical messages
that go beyond that time range.
This raises issues in the case of Status communities,
where recently joined members of a community
are not able to request complete message histories of the community channels.
This specification describes how **Control Nodes** (which are specific nodes in Status communities) archive historical message data of their communities, beyond the time range limit provided by Store Nodes using the [BitTorrent](https://bittorrent.org) protocol.
It also describes how the archives are distributed to community members via the Status network, so they can fetch them and gain access to a complete message history.
This specification describes how **Control Nodes**
(which are specific nodes in Status communities)
archive historical message data of their communities,
beyond the time range limit provided by Store Nodes using
the [BitTorrent](https://bittorrent.org) protocol.
It also describes how the archives are distributed to community members via
the Status network,
so they can fetch them and gain access to a complete message history.
## Terminology
The following terminology is used throughout this specification. Notice that some actors listed here are nodes that operate in Waku networks only, while others operate in the Status communities layer):
The following terminology is used throughout this specification.
Notice that some actors listed here are nodes that operate in Waku networks only,
while others operate in the Status communities layer):
| Name | References |
| -------------------- | --- |
@@ -43,7 +58,9 @@ The following terminology is used throughout this specification. Notice that som
This specification has the following assumptions:
- Store nodes([13/WAKU2-STORE](../../waku/standards/core/13/store.md)) are available 24/7, ensuring constant live message availability.
- Store nodes,
([13/WAKU2-STORE](../../waku/standards/core/13/store.md)),
are available 24/7 ensuring constant live message availability.
- The storage time range limit is 30 days.
- Store nodes have enough storage to persist historical messages for up to 30 days.
- No store nodes have storage to persist historical messages older than 30 days.
@@ -52,95 +69,163 @@ This specification has the following assumptions:
Furthermore, it assumes that:
- Control nodes have enough storage to persist historical messages older than 30 days.
- Control nodes have enough storage to persist historical messages
older than 30 days.
- Control nodes provide archives with historical messages **at least** every 30 days.
- Control nodes receive all community messages.
- Control nodes are honest.
- Control nodes know at least one store node from which it can query historical messages.
These assumptions are less than ideal and will be enhanced in future work. This [forum discussion](https://forum.vac.dev/t/status-communities-protocol-and-product-point-of-view/114) provides more details.
These assumptions are less than ideal and will be enhanced in future work.
This [forum discussion](https://forum.vac.dev/t/status-communities-protocol-and-product-point-of-view/114)
provides more details.
## Overview
The following is a high-level overview of the user flow and features this specification describes. For more detailed descriptions, read the dedicated sections in this specification.
The following is a high-level overview of the user flow and
features this specification describes.
For more detailed descriptions, read the dedicated sections in this specification.
### Serving community history archives
Control nodes go through the following (high level) process to provide community members with message histories:
Control nodes go through the following
(high level) process to provide community members with message histories:
1. Community owner creates a Status community (previously known as [org channels](https://github.com/status-im/specs/pull/151)) which makes its node a Control node.
2. Community owner enables message archive capabilities (on by default but can be turned off as well - see [UI feature spec](https://github.com/status-im/feature-specs/pull/36)).
3. A special type of channel to exchange metadata about the archival data is created, this channel is not visible in the user interface.
1. Community owner creates a Status community
(previously known as [org channels](https://github.com/status-im/specs/pull/151))
which makes its node a Control node.
2. Community owner enables message archive capabilities
(on by default but can be turned off as well - see [UI feature spec](https://github.com/status-im/feature-specs/pull/36)).
3. A special type of channel to exchange metadata about the archival data is created,
this channel is not visible in the user interface.
4. Community owner invites community members.
5. Control node receives messages published in channels and stores them into a local database.
6. After 7 days, the control node exports and compresses last 7 days worth of messages from database and bundles it together with a [message archive index](#waku-message-archive-index) into a torrent, from which it then creates a magnet link ([Magnet URI scheme](https://en.wikipedia.org/wiki/Magnet_URI_scheme), [Extensions for Peers to Send Metadata Files](https://www.bittorrent.org/beps/bep_0009.html)).
7. Control node sends the magnet link created in step 6 to community members via special channel created in step 3 through the Waku network.
8. Every subsequent 7 days, steps 6 and 7 are repeated and the new message archive data is appended to the previously created message archive data.
5. Control node receives messages published in channels and
stores them into a local database.
6. After 7 days, the control node exports and
compresses last 7 days worth of messages from database and
bundles it together with a
[message archive index](#wakumessagearchiveindex) into a torrent,
from which it then creates a magnet link ([Magnet URI scheme](https://en.wikipedia.org/wiki/Magnet_URI_scheme),
[Extensions for Peers to Send Metadata Files](https://www.bittorrent.org/beps/bep_0009.html)).
7. Control node sends the magnet link created in step 6 to community members via
special channel created in step 3 through the Waku network.
8. Every subsequent 7 days,
steps 6 and 7 are repeated and
the new message archive data
is appended to the previously created message archive data.
### Serving archives for missed messages
If the control node goes offline (where "offline" means, the control node's main process is no longer running), it MUST go through the following process:
If the control node goes offline
(where "offline" means, the control node's main process is no longer running),
it MUST go through the following process:
1. Control node restarts
2. Control node requests messages from store nodes for the missed time range for all channels in their community
2. Control node requests messages from store nodes
for the missed time range for all channels in their community
3. All missed messages are stored into control node's local message database
4. If 7 or more days have elapsed since the last message history torrent was created, the control node will perform step 6 and 7 of [Serving community history archives](#serving-community-history-archives) for every 7 days worth of messages in the missed time range (e.g. if the node was offline for 30 days, it will create 4 message history archives)
4. If 7 or more days have elapsed since the last message history torrent was created,
the control node will perform step 6 and
7 of [Serving community history archives](#serving-community-history-archives)
for every 7 days worth of messages in the missed time range
(e.g. if the node was offline for 30 days, it will create 4 message history archives)
### Receiving community history archives
Community member nodes go through the following (high level) process to fetch and restore community message histories:
Community member nodes go through the following (high level) process to fetch and
restore community message histories:
1. User joins community and becomes community member (see [org channels spec](../56/communities.md))
2. By joining a community, member nodes automatically subscribe to special channel for message archive metadata exchange provided by the community
3. Member node requests live message history (last 30 days) of all the community channels including the special channel from store nodes
4. Member node receives Waku message ([14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md)) that contains the metadata magnet link from the special channel
5. Member node extracts the magnet link from the Waku message and passes it to torrent client
6. Member node downloads [message archive index](#message-history-archive-index) file and determines which message archives are not downloaded yet (all or some)
2. By joining a community,
member nodes automatically subscribe to special channel for
message archive metadata exchange provided by the community
3. Member node requests live message history
(last 30 days) of all the community channels,
including the special channel from store nodes
4. Member node receives Waku message
([14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md))
that contains the metadata magnet link from the special channel
5. Member node extracts the magnet link from the Waku message and
passes it to torrent client
6. Member node downloads
[message archive index](#message-history-archive-index) file and
determines which message archives are not downloaded yet (all or some)
7. Member node fetches missing message archive data via torrent
8. Member node unpacks and decompresses message archive data to then hydrate its local database, deleting any messages for that community that the database previously stored in the same time range as covered by the message history archive
8. Member node unpacks and
decompresses message archive data to then hydrate its local database,
deleting any messages,
for that community that the database previously stored in the same time range,
as covered by the message history archive
## Storing live messages
For archival data serving, the control node MUST store live messages as [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md).
This is in addition to their database of application messages.
This is required to provide confidentiality, authenticity, and integrity of message data distributed via the BitTorrent layer, and later validated by community members when they unpack message history archives.
This is in addition to their database of application messages.
This is required to provide confidentiality, authenticity,
and integrity of message data distributed via the BitTorrent layer, and
later validated by community members when they unpack message history archives.
Control nodes SHOULD remove those messages from their local databases once they are older than 30 days and after they have been turned into message archives and distributed to the BitTorrent network.
Control nodes SHOULD remove those messages from their local databases
once they are older than 30 days and
after they have been turned into message archives and
distributed to the BitTorrent network.
### Exporting messages for bundling
Control nodes export Waku messages from their local database for creating and bundling history archives using the following criteria:
Control nodes export Waku messages from their local database for creating and
bundling history archives using the following criteria:
- Waku messages to be exported MUST have a `contentTopic` that match any of the topics of the community channels
- Waku messages to be exported MUST have a `timestamp` that lies within a time range of 7 days
- Waku messages to be exported MUST have a `contentTopic`
that match any of the topics of the community channels
- Waku messages to be exported MUST have a `timestamp`
that lies within a time range of 7 days
The `timestamp` is determined by the context in which the control node attempts to create a message history archives as described below:
The `timestamp` is determined by the context in which the control node attempts
to create a message history archives as described below:
1. The control node attempts to create an archive periodically for the past seven days (including the current day). In this case, the `timestamp` has to lie within those 7 days.
2. The control node has been offline (control node's main process has stopped and needs restart) and attempts to create archives for all the live messages it has missed since it went offline. In this case, the `timestamp` has to lie within the day the latest message was received and the current day.
1. The control node attempts to create an archive periodically
for the past seven days (including the current day).
In this case, the `timestamp` has to lie within those 7 days.
2. The control node has been offline
(control node's main process has stopped and needs restart) and
attempts to create archives for all the live messages it has missed
since it went offline.
In this case,
the `timestamp` has to lie within the day the latest message was received and
the current day.
Exported messages MUST be restored as [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) for bundling. Waku messages that are older than 30 days and have been exported for bundling can be removed from the control node's database (control nodes still maintain a database of application messages).
Exported messages MUST be restored as
[14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) for bundling.
Waku messages that are older than 30 days and
have been exported for bundling can be removed from the control node's database
(control nodes still maintain a database of application messages).
## Message history archives
Message history archives are represented as `WakuMessageArchive` and created from Waku messages exported from the local database.
Message history archives are represented as `WakuMessageArchive` and
created from Waku messages exported from the local database.
Message history archives are implemented using the following protocol buffer.
### WakuMessageHistoryArchive
The `from` field SHOULD contain a timestamp of the time range's lower bound.
The type parallels the `timestamp` of [WakuMessage](../../waku/standards/core/14/message.md/#payloads).
The type parallels the `timestamp` of [WakuMessage](../../waku/standards/core/14/message.md/).
The `to` field SHOULD contain a timestamp of the time range's the higher bound.
The `contentTopic` field MUST contain a list of all communiity channel topics.
The `messages` field MUST contain all messages that belong into the archive given its `from`, `to` and `contentTopic` fields.
The `messages` field MUST contain all messages that belong into the archive
given its `from`, `to` and `contentTopic` fields.
The `padding` field MUST contain the amount of zero bytes needed so that the overall byte size of the protobuf encoded `WakuMessageArchive` is a multiple of the `pieceLength` used to divide the message archive data into pieces.
This is needed for seamless encoding and decoding of archival data in interation with BitTorrent as explained in [creating message archive torrents](#creating-message-archive-torrents).
The `padding` field MUST contain the amount of zero bytes needed so
that the overall byte size of the protobuf encoded `WakuMessageArchive`
is a multiple of the `pieceLength` used to divide the message archive data into pieces.
This is needed for seamless encoding and
decoding of archival data in interation with BitTorrent,
as explained in [creating message archive torrents](#creating-message-archive-torrents).
```
```protobuf
syntax = "proto3"
message WakuMessageArchiveMetadata {
@@ -158,21 +243,32 @@ message WakuMessageArchive {
}
```
## Message history archive index
## Message History Archive Index
Control nodes MUST provide message archives for the entire community history.
The entirey history consists of a set of `WakuMessageArchive`'s where each archive contains a subset of historical `WakuMessage`s for a time range of seven days.
All the `WakuMessageArchive`s are concatenated into a single file as a byte string (see [Ensuring reproducible data pieces](#ensuring-reproducible-data-pieces)).
The entirey history consists of a set of `WakuMessageArchive`'s
where each archive contains a subset of historical `WakuMessage`s
for a time range of seven days.
All the `WakuMessageArchive`s are concatenated into a single file as a byte string
(see [Ensuring reproducible data pieces](#ensuring-reproducible-data-pieces)).
Control nodes MUST create a message history archive index (`WakuMessageArchiveIndex`) with metadata that allows receiving nodes to only fetch the message history archives they are interested in.
Control nodes MUST create a message history archive index
(`WakuMessageArchiveIndex`) with metadata that allows receiving nodes
to only fetch the message history archives they are interested in.
### WakuMessageArchiveIndex
A `WakuMessageArchiveIndex` is a map where the key is the KECCAK-256 hash of the `WakuMessageArchiveIndexMetadata` derived from a 7-day archive and the value is an instance of that `WakuMessageArchiveIndexMetadata` corresponding to that archive.
A `WakuMessageArchiveIndex` is a map where the key is the KECCAK-256 hash of
the `WakuMessageArchiveIndexMetadata` derived from a 7-day archive and
the value is an instance of that `WakuMessageArchiveIndexMetadata`
corresponding to that archive.
The `offset` field MUST contain the position at which the message history archive starts in the byte string of the total message archive data. This MUST be the sum of the length of all previously created message archives in bytes (see [Creating message archive torrents](#creating-message-archive-torrents)).
The `offset` field MUST contain the position at which the message history archive
starts in the byte string of the total message archive data.
This MUST be the sum of the length of all previously created message archives
in bytes (see [Creating message archive torrents](#creating-message-archive-torrents)).
```
```protobuf
syntax = "proto3"
message WakuMessageArchiveIndexMetadata {
@@ -187,45 +283,67 @@ message WakuMessageArchiveIndex {
}
```
The control node MUST update the `WakuMessageArchiveIndex` every time it creates one or more `WakuMessageArchive`s and bundle it into a new torrent.
For every created `WakuMessageArchive`, there MUST be a `WakuMessageArchiveIndexMetadata` entry in the `archives` field `WakuMessageArchiveIndex`.
The control node MUST update the `WakuMessageArchiveIndex`
every time it creates one or
more `WakuMessageArchive`s and bundle it into a new torrent.
For every created `WakuMessageArchive`,
there MUST be a `WakuMessageArchiveIndexMetadata` entry in the `archives` field `WakuMessageArchiveIndex`.
# Creating message archive torrents
## Creating message archive torrents
Control nodes MUST create a torrent file ("torrent") containing metadata to all message history archives.
To create a torrent file, and later serve the message archive data in the BitTorrent network, control nodes MUST store the necessary data in dedicated files on the file system.
Control nodes MUST create a torrent file ("torrent")
containing metadata to all message history archives.
To create a torrent file, and
later serve the message archive data in the BitTorrent network,
control nodes MUST store the necessary data in dedicated files on the file system.
A torrent's source folder MUST contain the following two files:
- `data` - Contains all protobuf encoded `WakuMessageArchive`'s (as bit strings) concatenated in ascending order based on their time
- `data` - Contains all protobuf encoded `WakuMessageArchive`'s (as bit strings)
concatenated in ascending order based on their time
- `index` - Contains the protobuf encoded `WakuMessageArchiveIndex`
Control nodes SHOULD store these files in a dedicated folder that is identifiable via the community id.
Control nodes SHOULD store these files in a dedicated folder that is identifiable,
via the community id.
### Ensuring reproducible data pieces
The control node MUST ensure that the byte string resulting from the protobuf encoded `data` is equal to the byte string `data` from the previously generated message archive torrent, plus the data of the latest 7 days worth of messages encoded as `WakuMessageArchive`.
The control node MUST ensure that the byte string resulting from
the protobuf encoded `data` is equal to the byte string `data`
from the previously generated message archive torrent,
plus the data of the latest 7 days worth of messages encoded as `WakuMessageArchive`.
Therefore, the size of `data` grows every seven days as it's append only.
The control nodes also MUST ensure that the byte size of every individual `WakuMessageArchive` encoded protobuf is a multiple of `pieceLength: ???` (**TODO**) using the `padding` field.
If the protobuf encoded 'WakuMessageArchive` is not a multiple of `pieceLength`, its `padding` field MUST be filled with zero bytes and the `WakuMessageArchive` MUST be re-encoded until its size becomes multiple of `pieceLength`.
The control nodes also MUST ensure that the byte size of every individual `WakuMessageArchive`
encoded protobuf is a multiple of `pieceLength: ???` (**TODO**)
using the `padding` field.
If the protobuf encoded `WakuMessageArchive` is not a multiple of `pieceLength`,
its `padding` field MUST be filled with zero bytes and
the `WakuMessageArchive` MUST be re-encoded until its size becomes multiple of `pieceLength`.
This is necessary because the content of the `data` file will be split into pieces of `pieceLength` when the torrent file is created, and the SHA1 hash of every piece is then stored in the torrent file and later used by other nodes to request the data for each individual data piece.
This is necessary because the content of the `data` file
will be split into pieces of `pieceLength` when the torrent file is created,
and the SHA1 hash of every piece is then stored in the torrent file and
later used by other nodes to request the data for each individual data piece.
By fitting message archives into a multiple of `pieceLength` and ensuring they fill possible remaining space with zero bytes, control nodes prevent the **next** message archive to occupy that remaining space of the last piece, which will result in a different SHA1 hash for that piece.
By fitting message archives into a multiple of `pieceLength` and
ensuring they fill possible remaining space with zero bytes,
control nodes prevent the **next** message archive to
occupy that remaining space of the last piece,
which will result in a different SHA1 hash for that piece.
#### **Example: Without padding**
Let `WakuMessageArchive` "A1" be of size 20 bytes:
```
```json
0 11 22 33 44 55 66 77 88 99
10 11 12 13 14 15 16 17 18 19
```
With a `pieceLength` of 10 bytes, A1 will fit into `20 / 10 = 2` pieces:
```
```json
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
```
@@ -234,32 +352,37 @@ With a `pieceLength` of 10 bytes, A1 will fit into `20 / 10 = 2` pieces:
Let `WakuMessageArchive` "A2" be of size 21 bytes:
```
```json
0 11 22 33 44 55 66 77 88 99
10 11 12 13 14 15 16 17 18 19
20
```
With a `pieceLength` of 10 bytes, A2 will fit into `21 / 10 = 2` pieces. The remainder will introduce a third piece:
With a `pieceLength` of 10 bytes, A2 will fit into `21 / 10 = 2` pieces.
The remainder will introduce a third piece:
```
```json
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
20 // piece[2] SHA1: 0x789
```
The next `WakuMessageArchive` "A3" will be appended ("#3") to the existing data and occupy the remaining space of the third data piece. The piece at index 2 will now produce a different SHA1 hash:
The next `WakuMessageArchive` "A3" will be appended ("#3") to the existing data
and occupy the remaining space of the third data piece.
The piece at index 2 will now produce a different SHA1 hash:
```
```json
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
20 #3 #3 #3 #3 #3 #3 #3 #3 #3 // piece[2] SHA1: 0xeef
#3 #3 #3 #3 #3 #3 #3 #3 #3 #3 // piece[3]
```
By filling up the remaining space of the third piece with A2 using its `padding` field, it is guaranteed that its SHA1 will stay the same:
By filling up the remaining space of the third piece
with A2 using its `padding` field,
it is guaranteed that its SHA1 will stay the same:
```
```json
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
20 0 0 0 0 0 0 0 0 0 // piece[2] SHA1: 0x999
@@ -269,82 +392,144 @@ By filling up the remaining space of the third piece with A2 using its `padding`
### Seeding message history archives
The control node MUST seed the [generated torrent](#creating-message-archive-torrents) until a new `WakuMessageArchive` is created.
The control node MUST seed the
[generated torrent](#creating-message-archive-torrents)
until a new `WakuMessageArchive` is created.
The control node SHOULD NOT seed torrents for older message history archives. Only one torrent at a time should be seeded.
The control node SHOULD NOT seed torrents for older message history archives.
Only one torrent at a time should be seeded.
### Creating magnet links
Once a torrent file for all message archives is created, the control node MUST derive a magnet link following the [Magnet URI scheme](https://en.wikipedia.org/wiki/Magnet_URI_scheme) using the underlying BitTorrent protocol client.
Once a torrent file for all message archives is created,
the control node MUST derive a magnet link following the
[Magnet URI scheme](https://en.wikipedia.org/wiki/Magnet_URI_scheme)
using the underlying BitTorrent protocol client.
### Message archive distribution
Message archives are available via the BitTorrent network as they are being [seeded by the control node](#seeding-message-history-archives).
Other community member nodes will download the message archives from the BitTorrent network once they receive a magnet link that contains a message archive index.
Message archives are available via the BitTorrent network as they are being
[seeded by the control node](#seeding-message-history-archives).
Other community member nodes will download the message archives
from the BitTorrent network once they receive a magnet link
that contains a message archive index.
The control node MUST send magnet links containing message archives and the message archive index to a special community channel.
The control node MUST send magnet links containing message archives and
the message archive index to a special community channel.
The topic of that special channel follows the following format:
```
```text
/{application-name}/{version-of-the-application}/{content-topic-name}/{encoding}
```
All messages sent with this topic MUST be instances of `ApplicationMetadataMessage` ([62/STATUS-PAYLOAD](../62/payload.md)) with a `payload` of `CommunityMessageArchiveIndex`.
All messages sent with this topic MUST be instances of `ApplicationMetadataMessage`
([62/STATUS-PAYLOADS](../62/payloads.md)) with a `payload` of `CommunityMessageArchiveIndex`.
Only the control node MAY post to the special channel. Other messages on this specified channel MUST be ignored by clients.
Only the control node MAY post to the special channel.
Other messages on this specified channel MUST be ignored by clients.
Community members MUST NOT have permission to send messages to the special channel.
However, community member nodes MUST subscribe to special channel to receive Waku messages containing magnet links for message archives.
However, community member nodes MUST subscribe to special channel
to receive Waku messages containing magnet links for message archives.
### Canonical message histories
Only control nodes are allowed to distribute messages with magnet links via the special channel for magnet link exchange.
Only control nodes are allowed to distribute messages with magnet links via
the special channel for magnet link exchange.
Community members MUST NOT be allowed to post any messages to the special channel.
Status nodes MUST ensure that any message that isn't signed by the control node in the special channel is ignored.
Status nodes MUST ensure that any message
that isn't signed by the control node in the special channel is ignored.
Since the magnet links are created from the control node's database (and previously distributed archives), the message history provided by the control node becomes the canonical message history and single source of truth for the community.
Since the magnet links are created from the control node's database
(and previously distributed archives),
the message history provided by the control node becomes the canonical message history
and single source of truth for the community.
Community member nodes MUST replace messages in their local databases with the messages extracted from archives within the same time range.
Messages that the control node didn't receive MUST be removed and are no longer part of the message history of interest, even if it already existed in a community member node's database.
Community member nodes MUST replace messages in their local databases
with the messages extracted from archives within the same time range.
Messages that the control node didn't receive MUST be removed and
are no longer part of the message history of interest,
even if it already existed in a community member node's database.
## Fetching message history archives
Generally, fetching message history archives is a three step process:
1. Receive [message archive index](#message-history-archive-index) magnet link as described in [Message archive distribution], download `index` file from torrent, then determine which message archives to download
1. Receive [message archive index](#message-history-archive-index)
magnet link as described in [Message archive distribution],
download `index` file from torrent, then determine which message archives to download
2. Download individual archives
Community member nodes subscribe to the special channel that control nodes publish magnet links for message history archives to.
There are two scenarios in which member nodes can receive such a magnet link message from the special channel:
Community member nodes subscribe to the special channel
that control nodes publish magnet links for message history archives to.
There are two scenarios in which member nodes can receive such a magnet link message
from the special channel:
1. The member node receives it via live messages, by listening to the special channel
2. The member node requests messages for a time range of up to 30 days from store nodes (this is the case when a new community member joins a community)
1. The member node receives it via live messages, by listening to the special channel
2. The member node requests messages for a time range of up to 30 days
from store nodes (this is the case when a new community member joins a community)
### Downloading message archives
When member nodes receive a message with a `CommunityMessageHistoryArchive` ([62/STATUS-PAYLOADS](../62/payloads.md)) from the aforementioned channnel, they MUST extract the `magnet_uri` and pass it to their underlying BitTorrent client so they can fetch the latest message history archive index, which is the `index` file of the torrent (see [Creating message archive torrents](#creating-message-archive-torrents)).
Due to the nature of distributed systems, there's no guarantee that a received message is the "last" message. This is especially true when member nodes request historical messages from store nodes.
When member nodes receive a message with a `CommunityMessageHistoryArchive`
([62/STATUS-PAYLOADS](../62/payloads.md)) from the aforementioned channnel,
they MUST extract the `magnet_uri` and
pass it to their underlying BitTorrent client
so they can fetch the latest message history archive index,
which is the `index` file of the torrent (see [Creating message archive torrents](#creating-message-archive-torrents)).
Therefore, member nodes MUST wait for 20 seconds after receiving the last `CommunityMessageArchive` before they start extracting the magnet link to fetch the latest archive index.
Due to the nature of distributed systems,
there's no guarantee that a received message is the "last" message.
This is especially true
when member nodes request historical messages from store nodes.
Once a message history archive index is downloaded and parsed back into `WakuMessageArchiveIndex`, community member nodes use a local lookup table to determine which of the listed archives are missing using the KECCAK-256 hashes stored in the index.
Therefore, member nodes MUST wait for 20 seconds
after receiving the last `CommunityMessageArchive`
before they start extracting the magnet link to fetch the latest archive index.
For this lookup to work, member nodes MUST store the KECCAK-256 hashes of the `WakuMessageArchiveIndexMetadata` provided by the `index` file for all of the message history archives that have been downlaoded in their local database.
Once a message history archive index is downloaded and
parsed back into `WakuMessageArchiveIndex`,
community member nodes use a local lookup table
to determine which of the listed archives are missing
using the KECCAK-256 hashes stored in the index.
Given a `WakuMessageArchiveIndex`, member nodes can access individual `WakuMessageArchiveIndexMetadata` to download individual archives.
For this lookup to work,
member nodes MUST store the KECCAK-256 hashes
of the `WakuMessageArchiveIndexMetadata` provided by the `index` file
for all of the message history archives that have been downlaoded
in their local database.
Given a `WakuMessageArchiveIndex`,
member nodes can access individual `WakuMessageArchiveIndexMetadata`
to download individual archives.
Community member nodes MUST choose one of the following options:
1. **Download all archives** - Request and download all data pieces for `data` provided by the torrent (this is the case for new community member nodes that haven't downloaded any archives yet)
2. **Download only the latest archive** - Request and download all pieces starting at the `offset` of the latest `WakuMessageArchiveIndexMetadata` (this the case for any member node that already has downloaded all previous history and is now interested in only the latst archive)
3. **Download specific archives** - Look into `from` and `to` fields of every `WakuMessageArchiveIndexMetadata` and determine the pieces for archives of a specific time range (can be the case for member nodes that have recently joined the network and are only interested in a subset of the complete history)
1. **Download all archives** - Request and
download all data pieces for `data` provided by the torrent
(this is the case for new community member nodes
that haven't downloaded any archives yet)
2. **Download only the latest archive** -
Request and download all pieces starting at the `offset` of the latest `WakuMessageArchiveIndexMetadata`
(this the case for any member node
that already has downloaded all previous history and
is now interested in only the latst archive)
3. **Download specific archives** -
Look into `from` and
`to` fields of every `WakuMessageArchiveIndexMetadata` and
determine the pieces for archives of a specific time range
(can be the case for member nodes that have recently joined the network and
are only interested in a subset of the complete history)
### Storing historical messages
When message archives are fetched, community member nodes MUST unwrap the resulting `WakuMessage` instances into `ApplicationMetadataMessage` instances and store them in their local database.
When message archives are fetched,
community member nodes MUST unwrap the resulting `WakuMessage` instances
into `ApplicationMetadataMessage` instances and store them in their local database.
Community member nodes SHOULD NOT store the wrapped `WakuMessage` messages.
All message within the same time range MUST be replaced with the messages provided by the message history archive.
All message within the same time range
MUST be replaced with the messages provided by the message history archive.
Community members nodes MUST ignore the expiration state of each archive message.
@@ -354,39 +539,58 @@ The following are things to cosider when implementing this specification.
## Control node honesty
This spec assumes that all control nodes are honest and behave according to the spec. Meaning they don't inject their own messages into, or remove any messages from historic archives.
This spec assumes that all control nodes are honest and behave according to the spec.
Meaning they don't inject their own messages into, or
remove any messages from historic archives.
## Bandwidth consumption
Community member nodes will download the latest archive they've received from the archive index, which includes messages from the last seven days. Assuming that community members nodes were online for that time range, they have already downloaded that message data and will now download an archive that contains the same.
Community member nodes will download the latest archive
they've received from the archive index,
which includes messages from the last seven days.
Assuming that community members nodes were online for that time range,
they have already downloaded that message data and
will now download an archive that contains the same.
This means there's a possibility member nodes will download the same data at least twice.
This means there's a possibility member nodes
will download the same data at least twice.
## Multiple community owners
It is possible for control nodes to export the private key of their owned community and pass it to other users so they become control nodes as well.
It is possible for control nodes
to export the private key of their owned community and
pass it to other users so they become control nodes as well.
This means, it's possible for multiple control nodes to exist.
This might conflict with the assumption that the control node serves as a single source of thruth. Multiple control nodes can have different message histories.
This might conflict with the assumption that the control node
serves as a single source of thruth.
Multiple control nodes can have different message histories.
Not only will multiple control nodes multiply the amount of archive index messages being distributed to the network, they might also contain different sets of magnet links and their corresponding hashes.
Not only will multiple control nodes
multiply the amount of archive index messages being distributed to the network,
they might also contain different sets of magnet links and their corresponding hashes.
Even if just a single message is missing in one of the histories, the hashes presented in archive indices will look completely different, resulting in the community member node to download the corresponding archive (which might be identical to an archive that was already downloaded, except for that one message).
Even if just a single message is missing in one of the histories,
the hashes presented in archive indices will look completely different,
resulting in the community member node to download the corresponding archive
(which might be identical to an archive that was already downloaded,
except for that one message).
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
* [13/WAKU2-STORE](../../waku/standards/core/13/store.md)
* [BitTorrent](https://bittorrent.org)
* [10/WAKU2](../../waku/standards/core/10/waku2.md)
* [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md)
* [Magnet URI scheme](https://en.wikipedia.org/wiki/Magnet_URI_scheme)
* [forum discussion](https://forum.vac.dev/t/status-communities-protocol-and-product-point-of-view/114)
* [org channels](https://github.com/status-im/specs/pull/151)
* [UI feature spec](https://github.com/status-im/feature-specs/pull/36)
* [Extensions for Peers to Send Metadata Files](https://www.bittorrent.org/beps/bep_0009.html)
* [org channels spec](../56/communities.md)
* [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md)
* [62/STATUS-PAYLOADS](../62/payloads.md)
- [13/WAKU2-STORE](../../waku/standards/core/13/store.md)
- [BitTorrent](https://bittorrent.org)
- [10/WAKU2](../../waku/standards/core/10/waku2.md)
- [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md)
- [Magnet URI scheme](https://en.wikipedia.org/wiki/Magnet_URI_scheme)
- [forum discussion](https://forum.vac.dev/t/status-communities-protocol-and-product-point-of-view/114)
- [org channels](https://github.com/status-im/specs/pull/151)
- [UI feature spec](https://github.com/status-im/feature-specs/pull/36)
- [Extensions for Peers to Send Metadata Files](https://www.bittorrent.org/beps/bep_0009.html)
- [org channels spec](../56/communities.md)
- [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md)
- [62/STATUS-PAYLOADS](../62/payloads.md)

View File

@@ -1,8 +1,9 @@
---
slug: 62
title: 62/STATUS-Payloads
title: 62/STATUS-PAYLOADS
name: Status Message Payloads
status: draft
description: Describes the payload of each message in Status.
editor: r4bbit <r4bbit@status.im>
contributors:
- Adam Babik <adam@status.im>
@@ -35,9 +36,13 @@ message StatusProtocolMessage {
}
```
`signature` is the bytes of the signed `SHA3-256` of the payload, signed with the key of the author of the message.
The node needs the signature to validate authorship of the message, so that the message can be relayed to third parties.
If a signature is not present, but an author is provided by a layer below, the message is not to be relayed to third parties, and it is considered plausibly deniable.
`signature` is the bytes of the signed `SHA3-256` of the payload,
signed with the key of the author of the message.
The node needs the signature to validate authorship of the message,
so that the message can be relayed to third parties.
If a signature is not present, but an author is provided by a layer below,
the message is not to be relayed to third parties,
and it is considered plausibly deniable.
### Encoding
@@ -49,7 +54,7 @@ The node encodes the payload using [Protobuf](https://developers.google.com/prot
The type `ChatMessage` represents a chat message exchanged between clients.
#### Payload
Payload
The protobuf description is:
@@ -134,7 +139,7 @@ message ChatMessage {
| 5 | ens_name | `string` | The ENS name of the user sending the message |
| 6 | chat_id | `string` | The local ID of the chat the message is sent to |
| 7 | message_type | `MessageType` | The type of message, different for one-to-one, public or group chats |
| 8 | content_type | `ContentType` | The type of the content of the message |
| 8 | content_type | `ContentType` | The type of the content of the message |
| 9 | payload | `Sticker` I `Image` I `Audio` I `DiscordMessage` I `bytes` I nil` | The payload of the message based on the content type |
| 13 | grant | `bytes` | Grant for community chat messages |
| 14 | display_name | `string` | The message author's display name |
@@ -143,12 +148,15 @@ message ChatMessage {
#### Content types
A node requires content types for a proper interpretation of incoming messages. Not each message is plain text but may carry different information.
A node requires content types for a proper interpretation of incoming messages.
Not each message is plain text but may carry different information.
The following content types MUST be supported:
* `TEXT_PLAIN` identifies a message which content is a plaintext.
There are other content types that MAY be implemented by the client:
* `STICKER`
* `STATUS`
* `EMOJI`
@@ -160,19 +168,29 @@ There are other content types that MAY be implemented by the client:
* `DISCORD_MESSAGE`
* `IDENTITY_VERIFICATION`
##### Mentions
##### Mentions
A mention MUST be represented as a string with the `@0xpk` format, where `pk` is the public key of the [user account](https://specs.status.im/spec/2) to be mentioned, within the `text` field of a message with content_type `TEXT_PLAIN`.
A mention MUST be represented as a string with the `@0xpk` format,
where `pk` is the public key of the
[user account](https://specs.status.im/spec/2) to be mentioned,
within the `text` field of a message with content_type `TEXT_PLAIN`.
A message MAY contain more than one mention.
This specification RECOMMENDs that the application does not require the user to enter the entire pk.
This specification RECOMMENDs that the application allows the user to create a mention by typing @ followed by the related ENS or 3-word pseudonym.
This specification RECOMMENDs that the application provides the user auto-completion functionality to create a mention.
For better user experience, the client SHOULD display a known [ens name or the 3-word pseudonym corresponding to the key](https://specs.status.im/spec/2#contact-verification) instead of the `pk`.
This specification RECOMMENDs that the application
does not require the user to enter the entire pk.
This specification RECOMMENDs that the application
allows the user to create a mention by typing @ followed by the related ENS or
3-word pseudonym.
This specification RECOMMENDs that the application
provides the user auto-completion functionality to create a mention.
For better user experience,
the client SHOULD display a known
[ens name or the 3-word pseudonym corresponding to the key](https://specs.status.im/spec/2#contact-verification)
instead of the `pk`.
##### Sticker content type
A `ChatMessage` with `STICKER` `Content/Type` MUST also specify the ID of the `Pack` and
the `Hash` of the pack, in the `Sticker` field of `ChatMessage`
A `ChatMessage` with `STICKER` `Content/Type` MUST also specify the ID of the `Pack`
and the `Hash` of the pack, in the `Sticker` field of `ChatMessage`
```protobuf
message StickerMessage {
@@ -183,15 +201,17 @@ message StickerMessage {
##### Image content type
A `ChatMessage` with `IMAGE` `Content/Type` MUST also specify the `payload` of the image
and the `type`.
A `ChatMessage` with `IMAGE` `Content/Type` MUST also
specify the `payload` of the image and the `type`.
Clients MUST sanitize the payload before accessing its content, in particular:
- Clients MUST choose a secure decoder
- Clients SHOULD strip metadata if present without parsing/decoding it
- Clients SHOULD NOT add metadata/exif when sending an image file for privacy and security reasons
- Clients MUST make sure that the transport layer constraints the size of the payload to limit they are able to handle securely
Clients MUST sanitize the payload before accessing its content, in particular:
* Clients MUST choose a secure decoder
* Clients SHOULD strip metadata if present without parsing/decoding it
* Clients SHOULD NOT add metadata/exif when sending an image file for privacy
and security reasons
* Clients MUST make sure that the transport layer constraints the size of the payload
to limit they are able to handle securely
```protobuf
message ImageMessage {
@@ -209,14 +229,18 @@ message ImageMessage {
##### Audio content type
A `ChatMessage` with `AUDIO` `Content/Type` MUST also specify the `payload` of the audio,
A `ChatMessage` with `AUDIO`,
`Content/Type` MUST also specify the `payload` of the audio,
the `type` and the duration in milliseconds (`duration_ms`).
Clients MUST sanitize the payload before accessing its content, in particular:
- Clients MUST choose a secure decoder
- Clients SHOULD strip metadata if present without parsing/decoding it
- Clients SHOULD NOT add metadata/exif when sending an audio file for privacy and security reasons
- Clients MUST make sure that the transport layer constraints the size of the payload to limit they are able to handle securely
Clients MUST sanitize the payload before accessing its content, in particular:
* Clients MUST choose a secure decoder
* Clients SHOULD strip metadata if present without parsing/decoding it
* Clients SHOULD NOT add metadata/exif when sending an audio file for privacy
and security reasons
* Clients MUST make sure that the transport layer constraints the size
of the payload to limit they are able to handle securely
```protobuf
message AudioMessage {
@@ -231,11 +255,13 @@ message AudioMessage {
##### Community content type
A `ChatMessage` with `COMMUNITY` `Content/Type` MUST also specify the `payload` of the community as bytes from a [CommunityDescription](#communitydescription).
A `ChatMessage` with `COMMUNITY` `Content/Type`,
MUST also specify the `payload` of the community as bytes from a [CommunityDescription](#communitydescription).
##### DiscordMessage content type
A `ChatMessage` with `DISCORD_MESSAGE` `Content/Type` MUST also specify the `payload` of the `DiscordMessage`.
A `ChatMessage` with `DISCORD_MESSAGE` `Content/Type`,
MUST also specify the `payload` of the `DiscordMessage`.
```protobuf
message DiscordMessage {
@@ -279,13 +305,16 @@ message DiscordMessageAttachment {
#### Message types
A node requires message types to decide how to encrypt a particular message and what metadata needs to be attached when passing a message to the transport layer.
A node requires message types to decide how to encrypt a particular message and
what metadata needs to be attached when passing a message to the transport layer.
For more on this, see [10/WAKU2](../../waku/standards/core/10/waku2.md).
<!-- TODO: This reference is a bit odd, considering the layer payloads should interact with is Secure Transport, and not Whisper/Waku. This requires more detail -->
<!-- TODO: This reference is a bit odd,
considering the layer payloads should interact with is Secure Transport, and
not Whisper/Waku. This requires more detail -->
The following messages types MUST be supported:
* `ONE_TO_ONE` is a message to the public group
* `PUBLIC_GROUP` is a private message
* `PRIVATE_GROUP` is a message to the private group.
@@ -307,34 +336,56 @@ enum MessageType {
#### Clock vs Timestamp and message ordering
If a user sends a new message before the messages sent while the user was offline are received, the new message is supposed to be displayed last in a chat.
This is where the basic algorithm of Lamport timestamp would fall short as it's only meant to order causally related events.
If a user sends a new message,
before the messages sent while the user was offline are received,
the new message is supposed to be displayed last in a chat.
This is where the basic algorithm of Lamport timestamp would fall short as
it's only meant to order causally related events.
The status client therefore makes a "bid", speculating that it will beat the current chat-timestamp, s.t. the status client's Lamport timestamp format is: `clock = `max({timestamp}, chat_clock + 1)`
The status client therefore makes a "bid",
speculating that it will beat the current chat-timestamp,
s.t. the status client's Lamport timestamp format is:
`clock = max({timestamp}, chat_clock + 1)`
This will satisfy the Lamport requirement, namely: a -> b then T(a) < T(b)
`timestamp` MUST be Unix time calculated, when the node creates the message, in milliseconds.
`timestamp` MUST be Unix time calculated, when the node creates the message,
in milliseconds.
This field SHOULD not be relied upon for message ordering.
`clock` SHOULD be calculated using the algorithm of [Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps).
When there are messages available in a chat, the node calculates `clock`'s value based on the last received message in a particular chat: `max(timeNowInMs, last-message-clock-value + 1)`.
`clock` SHOULD be calculated using the algorithm of
[Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps).
When there are messages available in a chat,
the node calculates `clock`'s value
based on the last received message in a particular chat:
`max(timeNowInMs, last-message-clock-value + 1)`.
If there are no messages, `clock` is initialized with `timestamp`'s value.
Messages with a `clock` greater than `120` seconds over the Whisper/Waku timestamp SHOULD be discarded, in order to avoid malicious users to increase the `clock` of a chat arbitrarily.
Messages with a `clock` greater than `120` seconds over the Whisper/Waku timestamp
SHOULD be discarded,
in order to avoid malicious users to increase the `clock` of a chat arbitrarily.
Messages with a `clock` less than `120` seconds under the Whisper/Waku timestamp might indicate an attempt to insert messages in the chat history which is not distinguishable from a `datasync` layer re-transit event.
Messages with a `clock` less than `120` seconds under the Whisper/Waku timestamp
might indicate an attempt to insert messages in the chat history,
which is not distinguishable from a `datasync` layer re-transit event.
A client MAY mark this messages with a warning to the user, or discard them.
The node uses `clock` value for the message ordering. The algorithm used, and the distributed nature of the system produces casual ordering, which might produce counter-intuitive results in some edge cases.
For example, when a user joins a public chat and sends a message before receiving the exist messages, their message `clock` value might be lower and the message will end up in the past when the historical messages are fetched.
The node uses `clock` value for the message ordering.
The algorithm used, and
the distributed nature of the system produces casual ordering,
which might produce counter-intuitive results in some edge cases.
For example, when a user joins a public chat and
sends a message before receiving the exist messages,
their message `clock` value might be lower and
the message will end up in the past when the historical messages are fetched.
#### Chats
Chat is a structure that helps organize messages.
It's usually desired to display messages only from a single recipient, or a group of recipients at a time and chats help to achieve that.
Chat is a structure that helps organize messages.
It's usually desired to display messages only from a single recipient,
or a group of recipients at a time and chats help to achieve that.
All incoming messages can be matched against a chat.
All incoming messages can be matched against a chat.
The below table describes how to calculate a chat ID for each message type.
|Message Type|Chat ID Calculation|Direction|Comment|
@@ -347,7 +398,9 @@ The below table describes how to calculate a chat ID for each message type.
### ContactUpdate
`ContactUpdate` is a message exchange to notify peers that either the user has been added as a contact, or that information about the sending user have changed.
`ContactUpdate` is a message exchange to notify peers
that either the user has been added as a contact, or
that information about the sending user have changed.
```protobuf
message ContactUpdate {
@@ -368,7 +421,7 @@ message ContactRequestPropagatedState {
}
```
#### Payload
#### Payload Fields
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
@@ -384,31 +437,38 @@ message ContactRequestPropagatedState {
A client SHOULD send a `ContactUpdate` to all the contacts each time:
- The ens_name has changed
- A user edits the profile image
* The ens_name has changed
* A user edits the profile image
A client SHOULD also periodically send a `ContactUpdate` to all the contacts, the interval is up to the client, the Status official client sends these updates every 48 hours.
A client SHOULD also periodically send a `ContactUpdate` to all the contacts,
the interval is up to the client,
the Status official client sends these updates every 48 hours.
### EmojiReaction
`EmojiReaction`s represents a user's "reaction" to a specific chat message.
`EmojiReaction`s represents a user's "reaction" to a specific chat message.
For more information about the concept of emoji reactions see [Facebook Reactions](https://en.wikipedia.org/wiki/Facebook_like_button#Use_on_Facebook).
This specification RECOMMENDS that the UI/UX implementation of sending `EmojiReactions` requires only a single click operation, as users have an expectation that emoji reactions are effortless and simple to perform.
This specification RECOMMENDS that the UI/UX implementation of sending `EmojiReactions`
requires only a single click operation,
as users have an expectation that emoji reactions are effortless
and simple to perform.
```protobuf
message EmojiReaction {
// clock Lamport timestamp of the chat message
uint64 clock = 1;
// chat_id the ID of the chat the message belongs to, for query efficiency the chat_id is stored in the db even though the
// chat_id the ID of the chat the message belongs to, for query efficiency the
// chat_id is stored in the db even though the
// target message also stores the chat_id
string chat_id = 2;
// message_id the ID of the target message that the user wishes to react to
string message_id = 3;
// message_type is (somewhat confusingly) the ID of the type of chat the message belongs to
// message_type
// is (somewhat confusingly) the ID of the type of chat the message belongs to
MessageType message_type = 4;
// type the ID of the emoji the user wishes to react with
@@ -431,20 +491,23 @@ message EmojiReaction {
Clients MUST specify `clock`, `chat_id`, `message_id`, `type` and `message_type`.
This specification RECOMMENDS that the UI/UX implementation of retracting an `EmojiReaction`s requires only a single click operation, as users have an expectation that emoji reaction removals are effortless and simple to perform.
This specification RECOMMENDS that the UI/UX implementation of retracting an `EmojiReaction`s
requires only a single click operation,
as users have an expectation that emoji reaction removals are effortless and
simple to perform.
### MembershipUpdateMessage and MembershipUpdateEvent
`MembershipUpdateEvent` is a message used to propagate information about group membership changes in a group chat.
`MembershipUpdateEvent` is a message used to propagate information
about group membership changes in a group chat.
The details are in the [Group chats specs](../56/communities.md).
```protobuf
message MembershipUpdateMessage {
// The chat id of the private group chat
string chat_id = 1;
// A list of events for this group chat, first x bytes are the signature, then is a
// protobuf encoded MembershipUpdateEvent
// A list of events for this group chat, first x bytes are the signature,
// then is a protobuf encoded MembershipUpdateEvent
repeated bytes events = 2;
// An optional chat message
@@ -483,7 +546,7 @@ message MembershipUpdateEvent {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
@@ -496,7 +559,8 @@ A `MembershipUpdateMessage` includes either a `ChatMessage` or `EmojiReaction`.
### SyncInstallationContactV2
The node uses `SyncInstallationContact` messages to synchronize in a best-effort the contacts to other devices.
The node uses `SyncInstallationContact` messages to synchronize
in a best-effort the contacts to other devices.
```protobuf
message SyncInstallationContactV2 {
@@ -522,16 +586,15 @@ message SyncInstallationContactV2 {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | last_updated_locally | `uint64` | Timestamp of last local update |
| 1 | last_updated_locally | `uint64` | Timestamp of last local update |
| 2 | id | `string` | id of the contact synced |
| 3 | profile_image | `string` | `base64` encoded profile picture of the user |
| 4 | ens_name | `string` | ENS name of the contact |
| 5 | `array[string]` | Array of `system_tags` for the user, this can currently be: `":contact/added", ":contact/blocked", ":contact/request-received"`|
| 5 | |`array[string]` | Array of `system_tags` for the user, this can currently be: `":contact/added", ":contact/blocked", ":contact/request-received"` |
| 7 | local_nickname | `string` | Local display name of the contact |
| 9 | added | `bool` | Wether the contact is added |
| 10 | blocked | `bool` | Wether the contact is blocked |
@@ -547,7 +610,8 @@ message SyncInstallationContactV2 {
### SyncInstallationPublicChat
The node uses `SyncInstallationPublicChat` message to synchronize in a best-effort the public chats to other devices.
The node uses `SyncInstallationPublicChat` message to synchronize
in a best-effort the public chats to other devices.
```protobuf
message SyncInstallationPublicChat {
@@ -556,16 +620,17 @@ message SyncInstallationPublicChat {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | clock value of the chat |
| 1 | clock | `uint64` | clock value of the chat |
| 2 | id | `string` | id of the chat synced |
### SyncPairInstallation
The node uses `PairInstallation` messages to propagate information about a device to its paired devices.
The node uses `PairInstallation` messages to propagate information
about a device to its paired devices.
```protobuf
message SyncPairInstallation {
@@ -578,18 +643,19 @@ message SyncPairInstallation {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | clock value of the chat |
| 1 | clock | `uint64` | clock value of the chat |
| 2| installation_id | `string` | A randomly generated id that identifies this device |
| 3 | device_type | `string` | The OS of the device `ios`,`android` or `desktop` |
| 4 | name | `string` | The self-assigned name of the device |
### ChatIdentity
`ChatIdentity` represents the user defined identity associated with their public chat key.
`ChatIdentity` represents the user defined identity associated
with their public chat key.
```protobuf
message ChatIdentity {
@@ -601,18 +667,19 @@ message ChatIdentity {
string color = 6;
string emoji = 7;
repeated SocialLink social_links = 8;
// first known message timestamp in seconds (valid only for community chats for now)
// first known message timestamp in seconds
// (valid only for community chats for now)
// 0 - unknown
// 1 - no messages
uint32 first_message_timestamp = 9;
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2| ens_name | `string` | A valid ENS associated with the chat key |
| 3 | images | `map<string, IdentityImage>` | Image data associated with the chat key |
| 4 | display_name | `string` | The self-assigned display_name of the chat key |
@@ -624,8 +691,8 @@ message ChatIdentity {
### CommunityDescription
`CommunityDescription` represents a community metadata that is used to discover communities and share community updates.
`CommunityDescription` represents a community metadata
that is used to discover communities and share community updates.
```protobuf
message CommunityDescription {
@@ -670,17 +737,20 @@ message CommunityPermissions {
}
bool ens_only = 1;
// https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/megolm.md is a candidate for the algorithm to be used in case we want to have private communityal chats, lighter than pairwise encryption using the DR, less secure, but more efficient for large number of participants
// https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/megolm.md is a
// candidate for the algorithm to be used in case we want to have private
// communityal chats, lighter than pairwise encryption using the DR, less secure,
// but more efficient for large number of participants
bool private = 2;
Access access = 3;
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2| members | `map<string, CommunityMember>` | The members of the community |
| 3 | permissions | `CommunityPermissions` | Image data associated with the chat key |
| 4 | display_name | `string` | The self-assigned display_name of the chat key |
@@ -692,9 +762,11 @@ message CommunityPermissions {
### CommunityRequestToJoin
A `CommunityRequestToJoin` represents a request to join a community, sent by a user that is not yet a member of that community.
A `CommunityRequestToJoin` represents a request to join a community,
sent by a user that is not yet a member of that community.
A request to join a community includes a list of `RevealedAccount`.
These are wallet addresses that users are willing to reveal with the community's control node and admins.
These are wallet addresses that users are willing to reveal
with the community's control node and admins.
```protobuf
message CommunityRequestToJoin {
@@ -714,11 +786,11 @@ message RevealedAccount {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2| ens_name | `string` | The ENS of the user sending the request |
| 3 | chat_id | `string` | The id of the chat to request access to |
| 4 | community_id | `bytes` | The public key of the community |
@@ -727,7 +799,8 @@ message RevealedAccount {
### PinMessage
A `PinMessage` is a signal that tells a client that a specific message has to be marked as pinned.
A `PinMessage` is a signal that tells a client that a specific message
has to be marked as pinned.
```protobuf
message PinMessage {
@@ -739,22 +812,20 @@ message PinMessage {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2| message_id | `string` | The id of the message to be pinned |
| 3 | chat_id | `string` | The id of the chat of the message to be pinned |
| 4 | pinned | `bool` | Whether the message should be pinned or unpinned |
| 5 | message_type | `MessageType` | The type of message (public/one-to-one/private-group-chat) |
### EditMessage
A `EditMessage` represents an update to an existing message.
```protobuf
message EditMessage {
uint64 clock = 1;
@@ -773,11 +844,11 @@ message EditMessage {
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2| text | `string` | The updated message text |
| 3 | chat_id | `string` | The id of the chat of the message |
| 4 | message_id | `string` | The id of the message to be edited |
@@ -786,10 +857,10 @@ message EditMessage {
| 7 | content_type | `ChatMessage.ContentType` | The updated content type of the message |
| 8 | unfurled_links | `array<UnfurledLink>` | Updated link metadata |
### DeleteMessage
A `DeleteMessage` represents a signal to delete a message from the local database of a client.
A `DeleteMessage` represents a signal to delete a message
from the local database of a client.
```protobuf
message DeleteMessage {
@@ -808,21 +879,21 @@ message DeleteMessage {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | chat_id | `string` | The id of the chat of the message |
| 3 | message_id | `string` | The id of the message to delete |
| 4 | grant | `bytes` | A grant for a community edit messages |
| 5 | message_type | `MessageType` | The type of message |
| 6 | deleted_by | `string` | The public key of the user who deleted the message |
### CommunityMessageArchiveLink
A `CommunityMessageArchiveLink` contains a magnet uri for a community's message archive, created using [61/STATUS-Community-History-Archives](../61/community-history-service.md).
A `CommunityMessageArchiveLink` contains a magnet uri for a community's message archive,
created using [61/STATUS-Community-History-Archives](../61/community-history-service.md).
```protobuf
message CommunityMessageArchiveMagnetlink {
@@ -831,11 +902,11 @@ message CommunityMessageArchiveMagnetlink {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | magnet_uri | `string` | The magnet uri of the community archive torrent |
### AcceptContactRequest
@@ -850,16 +921,17 @@ message AcceptContactRequest {
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | id | `string` | The id of the contact request |
| 2 | clock | `uint64` | Clock value of the message |
| 2 | clock | `uint64` | Clock value of the message |
### RetractContactRequest
A `RetractContractRequest` message signals to the reiver of a request that the request was retracted.
A `RetractContractRequest` message signals to the reiver, of a request,
that the request was retracted.
```protobuf
message RetractContactRequest {
@@ -869,12 +941,12 @@ message RetractContactRequest {
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | id | `string` | The id of the contact request |
| 2 | clock | `uint64` | Clock value of the message |
| 2 | clock | `uint64` | Clock value of the message |
### CommunityRequestToJoinResponse
@@ -891,11 +963,11 @@ message CommunityRequestToJoinResponse {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | community | `CommunityDescription` | The community metadata |
| 3 | accepted | `bool` | Whether the request was accepted |
| 4 | grant | `bytes` | The grant |
@@ -904,7 +976,8 @@ message CommunityRequestToJoinResponse {
### CommunityRequestToLeave
A `CommunityRequestToLeave` represents a signal to a community that a user wants to be removed from the community's member list.
A `CommunityRequestToLeave` represents a signal to a community
that a user wants to be removed from the community's member list.
```protobuf
message CommunityRequestToLeave {
@@ -912,17 +985,19 @@ message CommunityRequestToLeave {
bytes community_id = 2;
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | community_id | `bytes` | The id of the community |
### RequestContactVerification
A `RequestContactVerification` is a request to verify a contact using a "challenge", which can by any string message and typically involves questions that only the contact should know.
A `RequestContactVerification` is a request to verify a contact using a "challenge",
which can by any string message and
typically involves questions that only the contact should know.
```protobuf
message RequestContactVerification {
@@ -930,17 +1005,18 @@ message RequestContactVerification {
string challenge = 3;
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | challenge | `string` | The challenge message used for verification |
### AcceptContactVerification
A `AcceptContactVerification` signals that a verification request was accepted and includes a response to the challenge.
A `AcceptContactVerification` signals that a verification request was accepted and
includes a response to the challenge.
```protobuf
message AcceptContactVerification {
@@ -950,11 +1026,11 @@ message AcceptContactVerification {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | id | `string` | The verification request id |
| 3 | response | `string` | The response for the challenge |
@@ -969,11 +1045,11 @@ message DeclineContactVerification {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | id | `string` | The verification request id |
### CancelContactVerification
@@ -987,14 +1063,13 @@ message CancelContactVerification {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | id | `string` | The verification request id |
### CommunityCancelRequestToJoin
A `CommunityCancelRequestToJoin` cancels a pending request to join.
@@ -1009,11 +1084,11 @@ message CommunityCancelRequestToJoin {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | ens_name | `string` | The ENS name of the account cancelling the request |
| 3 | chat_id | `string` | The id of the chat |
| 4 | community_id | `bytes` | The id of the community |
@@ -1021,7 +1096,8 @@ message CommunityCancelRequestToJoin {
### CommunityEditSharedAddresses
A `CommunityEditSharedAddresses` message allows users to edit the shared accounts they've revealed when requesting to join a community.
A `CommunityEditSharedAddresses` message allows users to edit the shared accounts
they've revealed when requesting to join a community.
```protobuf
message CommunityEditSharedAddresses {
@@ -1031,11 +1107,11 @@ message CommunityEditSharedAddresses {
}
```
#### Payload
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | Clock value of the message |
| 1 | clock | `uint64` | Clock value of the message |
| 2 | community_id | `bytes` | The id of the community |
| 3 | revealed_accounts | `array<RevealedAccount>` | A list of revealed accounts |
@@ -1043,33 +1119,32 @@ message CommunityEditSharedAddresses {
There are two ways to upgrade the protocol without breaking compatibility:
- A node always supports accretion
- A node does not support deletion of existing fields or messages, which might break compatibility
* A node always supports accretion
* A node does not support deletion of existing fields or messages,
which might break compatibility
## Security Considerations
-
## Changelog
### Version 0.5
Released [August 25, 2020](https://github.com/status-im/specs/commit/968fafff23cdfc67589b34dd64015de29aaf41f0)
- Added support for emoji reactions
* Added support for emoji reactions
### Version 0.4
Released [July 16, 2020](https://github.com/status-im/specs/commit/ad45cd5fed3c0f79dfa472253a404f670dd47396)
- Added support for images
- Added support for audio
* Added support for images
* Added support for audio
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
- Added language to include Waku in all relevant places
* Added language to include Waku in all relevant places
## Copyright

View File

@@ -12,7 +12,9 @@ contributors:
## Terminology
- **Account**: A valid [BIP-32](https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki) compliant key.
- **Account**: A valid
[BIP-32](https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki)
compliant key.
- **Multiaccount**: An account from which multiple Accounts can be derived.
## Abstract
@@ -28,8 +30,11 @@ More documentation on the Status Keycard can be found [here](https://keycard.tec
## Motivation
The Status Keycard is a hardware wallet that can be used to store and sign transactions.
For the purpose of the Status App, this specification describes how the Keycard SHOULD be used to store and sign transactions.
The Status Keycard is a hardware wallet that can be used to store and
sign transactions.
For the purpose of the Status App,
this specification describes how the Keycard SHOULD be used to store and
sign transactions.
## Usage
@@ -40,7 +45,7 @@ For the purpose of the Status App, this specification describes how the Keycard
To initialize the keycard for use with the application.
The keycard is locked with a 6 digit pin.
#### Request wire format
Request wire format
```json
{
@@ -48,7 +53,7 @@ The keycard is locked with a 6 digit pin.
}
```
#### Response wire format
Response wire format
```json
{
@@ -65,18 +70,19 @@ The application SHOULD provide a way to recover the keycard in case the pin is f
To fetch if the keycard is ready to be used by the application.
#### Request wire format
Request wire format
The requester MAY add a `pairing` field to filter through the generated keys
```json
{
"pairing": <shared_secret>/<pairing_index>/<256_bit_salt> OR null
}
```
#### Response wire format
Response wire format
##### If the keycard is not initialized yet
#### If the keycard is not initialized yet
```json
{
@@ -84,7 +90,7 @@ The requester MAY add a `pairing` field to filter through the generated keys
}
```
##### If the keycard is initialized
#### If the keycard is initialized
```json
{
@@ -101,9 +107,10 @@ The requester MAY add a `pairing` field to filter through the generated keys
### 3. Pairing the Keycard to the Client device (`/pair`)
To establish a secure communication channel described [here](https://keycard.tech/docs/apdu/opensecurechannel.html), the keycard and the client device need to be paired.
To establish a secure communication channel described [here](https://keycard.tech/docs/apdu/opensecurechannel.html),
the keycard and the client device need to be paired.
#### Request wire format
Request wire format
```json
{
@@ -111,7 +118,7 @@ To establish a secure communication channel described [here](https://keycard.tec
}
```
#### Response wire format
Response wire format
```json
"<shared_secret>/<pairing_index>/<256_bit_salt>"
@@ -121,7 +128,7 @@ To establish a secure communication channel described [here](https://keycard.tec
To generate a new set of keys and load them onto the keycard.
#### Request wire format
Request wire format
```json
{
@@ -131,7 +138,7 @@ To generate a new set of keys and load them onto the keycard.
}
```
#### Response wire format
Response wire format
```json
{
@@ -154,7 +161,7 @@ To generate a new set of keys and load them onto the keycard.
To fetch the keys that are currently loaded on the keycard.
#### Request wire format
Request wire format
```json
{
@@ -163,7 +170,7 @@ To fetch the keys that are currently loaded on the keycard.
}
```
#### Response wire format
Response wire format
```json
{
@@ -184,9 +191,10 @@ To fetch the keys that are currently loaded on the keycard.
### 6. Sign a transaction (`/sign`)
To sign a transaction using the keycard, passing in the pairing information and the transaction to be signed.
To sign a transaction using the keycard, passing in the pairing information and
the transaction to be signed.
#### Request wire format
Request wire format
```json
{
@@ -197,7 +205,7 @@ To sign a transaction using the keycard, passing in the pairing information and
}
```
#### Response wire format
Response wire format
```json
<256_bit_signature>
@@ -205,9 +213,10 @@ To sign a transaction using the keycard, passing in the pairing information and
### 7. Export a key (`/export-key`)
To export a key from the keycard, passing in the pairing information and the path to the key to be exported.
To export a key from the keycard, passing in the pairing information and
the path to the key to be exported.
#### Request wire format
Request wire format
```json
{
@@ -217,17 +226,17 @@ To export a key from the keycard, passing in the pairing information and the pat
}
```
#### Response wire format
Response wire format
```json
<256_bit_public_key>
```
```
### 8. Verify a pin (`/verify-pin`)
To verify the pin of the keycard.
#### Request wire format
Request wire format
```json
{
@@ -235,22 +244,22 @@ To verify the pin of the keycard.
}
```
#### Response wire format
Response wire format
```json
1_digit_status_code
```
Status code reference:
- 3: PIN is valid
<!--TODO: what are the other status codes?-->
### 9. Change the pin (`/change-pin`)
To change the pin of the keycard.
#### Request wire format
Request wire format
```json
{
@@ -260,15 +269,15 @@ To change the pin of the keycard.
}
```
#### Response wire format
Response wire format
##### If the operation was successful
#### If the operation was successful
```json
true
```
##### If the operation was unsuccessful
#### If the operation was unsuccessful
```json
false
@@ -276,9 +285,10 @@ false
### 10. Unblock the keycard (`/unblock-pin`)
If the Keycard is blocked due to too many incorrect pin attempts, it can be unblocked using the PUK.
If the Keycard is blocked due to too many incorrect pin attempts,
it can be unblocked using the PUK.
#### Request wire format
Request wire format
```json
{
@@ -288,15 +298,15 @@ If the Keycard is blocked due to too many incorrect pin attempts, it can be unbl
}
```
#### Response wire format
Response wire format
##### If the operation was successful
If the operation was successful
```json
true
```
##### If the operation was unsuccessful
If the operation was unsuccessful
```json
false
@@ -304,7 +314,8 @@ false
## Flows
Any application that uses the Status Keycard MAY implement the following flows according to the actions listed above.
Any application that uses the Status Keycard
MAY implement the following flows according to the actions listed above.
### 1. A new user wants to use the Keycard with the application
@@ -335,7 +346,6 @@ Any application that uses the Status Keycard MAY implement the following flows a
1. The user unblocks the Keycard using the `/unblock-pin` endpoint.
## Security Considerations
Inherits the security considerations of [Status Keycard](https://keycard.tech/docs/)
@@ -344,7 +354,6 @@ Inherits the security considerations of [Status Keycard](https://keycard.tech/do
Inherits the privacy considerations of [Status Keycard](https://keycard.tech/docs/)
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

View File

@@ -14,42 +14,64 @@ contributors:
## Abstract
This specification details what a Status account address is and how account addresses are created and used.
This specification details what a Status account address is and
how account addresses are created and used.
## Background
The core concept of an account in Status is a set of cryptographic keypairs. Namely, the combination of the following:
The core concept of an account in Status is a set of cryptographic keypairs.
Namely, the combination of the following:
1. a Waku chat identity keypair
1. a set of cryptocurrency wallet keypairs
The Status node verifies or derives everything else associated with the contact from the above items, including:
The Status node verifies or
derives everything else associated with the contact from the above items, including:
- Ethereum address (future verification, currently the same base keypair)
- identicon
- message signatures
## Initial Key Generation
### Public/Private Keypairs
- An ECDSA (secp256k1 curve) public/private keypair MUST be generated via a [BIP43](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki) derived path from a [BIP39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic seed phrase.
### Public/Private Keypairs
- An ECDSA (secp256k1 curve) public/private keypair MUST be generated via a
[BIP43](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki)
derived path from a
[BIP39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki)
mnemonic seed phrase.
- The default paths are defined as such:
- Waku Chat Key (`IK`): `m/43'/60'/1581'/0'/0` (post Multiaccount integration)
- following [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md)
- Status Wallet paths: `m/44'/60'/0'/0/i` starting at `i=0`
- following [BIP44](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
- NOTE: this (`i=0`) is also the current (and only) path for Waku key before Multiaccount integration
- Waku Chat Key (`IK`): `m/43'/60'/1581'/0'/0` (post Multiaccount integration)
- following [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md)
- Status Wallet paths: `m/44'/60'/0'/0/i` starting at `i=0`
- following [BIP44](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
- NOTE: this (`i=0`) is also the current (and only)
path for Waku key before Multiaccount integration
## Account Broadcasting
- A user is responsible for broadcasting certain information publicly so that others may contact them.
- A user is responsible for broadcasting certain information publicly so
that others may contact them.
### X3DH Prekey bundles
- Refer to [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) for details on the X3DH prekey bundle broadcasting, as well as regeneration.
- Refer to [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md)
for details on the X3DH prekey bundle broadcasting, as well as regeneration.
## Optional Account additions
### ENS Username
- A user MAY register a public username on the Ethereum Name System (ENS). This username is a user-chosen subdomain of the `stateofus.eth` ENS registration that maps to their Waku identity key (`IK`).
- A user MAY register a public username on the Ethereum Name System (ENS).
This username is a user-chosen subdomain of the `stateofus.eth`
ENS registration that maps to their Waku identity key (`IK`).
### User Profile Picture
- An account MAY edit the `IK` generated identicon with a chosen picture. This picture will become part of the publicly broadcasted profile of the account.
- An account MAY edit the `IK` generated identicon with a chosen picture.
This picture will become part of the publicly broadcasted profile of the account.
<!-- TODO: Elaborate on wallet account and multiaccount -->
@@ -88,11 +110,14 @@ message MultiAccount {
}
```
The above payload is broadcasted when 2 devices that belong to a user need to be paired.
The above payload is broadcasted when 2 devices
that belong to a user need to be paired.
## Security Considerations
- This specification inherits security considerations of [53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) and [54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md).
- This specification inherits security considerations of
[53/WAKU2-X3DH](../../waku/standards/application/53/x3dh.md) and
[54/WAKU2-X3DH-SESSIONS](../../waku/standards/application/54/x3dh-sessions.md).
## Copyright

View File

@@ -11,27 +11,40 @@ contributors:
---
## Abstract
A push notification server implementation for IOS devices and Android devices.
This specification provides a set of methods that allow clients to use push notification services in mobile environments.
A push notification server implementation for IOS devices and Android devices.
This specification provides a set of methods that allow clients
to use push notification services in mobile environments.
## Background
Push notification for iOS and Android devices can only be implemented by relying on
[APN](https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html#//apple_ref/doc/uid/TP40008194-CH8-SW1),
Apple Push Notification, service for iOS or
[Firebase](https://firebase.google.com/) for Android.
For some Android devices, foreground services are restricted, requiring a user to grant authorization to applications to use foreground notifications.
Apple iOS devices restrict notifications to a few internal functions that every application can not use.
Applications on iOS can request execution time when they are in the background. This has a limited set of use cases for example,
it will not schedule any time if the application was closed with force quit.
Requesting execution time is not responsive enough to implement a push notification system.
Push notification for iOS and
Android devices can only be implemented by relying on
[APN](https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html#//apple_ref/doc/uid/TP40008194-CH8-SW1),
Apple Push Notification, service for iOS or
[Firebase](https://firebase.google.com/) for Android.
For some Android devices, foreground services are restricted,
requiring a user to grant authorization to applications
to use foreground notifications.
Apple iOS devices restrict notifications to
a few internal functions that every application can not use.
Applications on iOS can request execution time when they are in the background.
This has a limited set of use cases for example,
it will not schedule any time if the application was closed with force quit.
Requesting execution time is not responsive enough to
implement a push notification system.
Status provides a set of methods to acheive push notification services.
Since this can not be safely implemented in a privacy-preserving manner, clients need to be given an option to opt-in to receive and send push notifications.
Since this can not be safely implemented in a privacy-preserving manner,
clients need to be given an option to opt-in to receive and send push notifications.
They are disabled by default.
## Specification
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”,
“SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
### Definitions
@@ -42,8 +55,7 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL
| server | A service that performs push notifications. |
| Waku-Store | A Waku node that decides to provide functionality to store messages permanently and deliver the messages to requesting clients. As described in [13/WAKU-STORE](../../waku/standards/core/13/store.md) specification.|
### Server Components
### Server Components
| Components | Description |
| --------------- | --------- |
@@ -52,37 +64,45 @@ The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL
| Registering Client | A client that wants to receive push notifications. |
| Sending Client | A client that wants to send push notifications. |
### Requirements
### Requirements:
The party releasing the app MUST possess a certificate for the Apple Push Notification service and it MUST run a
[gorush](https://github.com/appleboy/gorush) publicly accessible server for sending the actual notification.
The party releasing the app MUST possess a certificate
for the Apple Push Notification service and
it MUST run a [gorush](https://github.com/appleboy/gorush)
publicly accessible server for sending the actual notification.
The party releasing the app MUST run its own [gorush](https://github.com/appleboy/gorush).
### Push Notification Server Flow
#### Registration Process:
#### Registration Process
![registration](./images/registration.png)
#### Sending and Receiving Notification Process:
#### Sending and Receiving Notification Process
![notification](./images/notification.png)
### Registering Client
Registering a client with a push notification service.
- A client MAY register with one or more push notification services in order to increase availability.
- A client MAY register with one or
more push notification services in order to increase availability.
- A client SHOULD make sure that all the notification services they registered with have the same information about their tokens.
- A client SHOULD make sure that all the notification services they registered with
have the same information about their tokens.
- A `PNR message` (Push Notification Registration) MUST be sent to the
[partitioned topic](../../waku/standards/application/54/x3dh-sessions.md) for the public key of the node, encrypted with this key.
[partitioned topic](../../waku/standards/application/54/x3dh-sessions.md)
for the public key of the node, encrypted with this key.
- The message MUST be wrapped in a [`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_REGISTRATION`.
- The message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_REGISTRATION`.
- The marshaled protobuf payload MUST also be encrypted with AES-GCM using the DiffieHellman key generated from the client and server identity.
This is done in order to ensure that the extracted key from the signature will be considered invalid if it cant decrypt the payload.
- The marshaled protobuf payload MUST also be encrypted with AES-GCM
using the DiffieHellman key generated from the client and server identity.
This is done in order to ensure that the extracted key from the signature
will be considered invalid if it cant decrypt the payload.
The content of the message MUST contain the following [protobuf record](https://developers.google.com/protocol-buffers/):
@@ -111,7 +131,9 @@ message PushNotificationRegistration {
```
A push notification server will handle the message according to the following rules:
- it MUST extract the public key of the sender from the signature and verify that the payload can be decrypted successfully.
- it MUST extract the public key of the sender from the signature and
verify that the payload can be decrypted successfully.
- it MUST verify that `token_type` is supported.
@@ -119,7 +141,9 @@ A push notification server will handle the message according to the following ru
- it MUST verify that `installation_id` is non empty.
- it MUST verify that `version` is non-zero and greater than the currently stored version for the public key and `installation_id` of the sender, if any.
- it MUST verify that `version` is non-zero and
greater than the currently stored version for the public key and
`installation_id` of the sender, if any.
- it MUST verify that `grant` is non empty and according to the Grant Server specs.
@@ -127,7 +151,8 @@ A push notification server will handle the message according to the following ru
- it MUST verify that `apn_topic` is set if token_type is APN_TOKEN.
- The message MUST be wrapped in a [`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_REGISTRATION_RESPONSE`.
- The message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_REGISTRATION_RESPONSE`.
The payload of the response is:
@@ -148,69 +173,97 @@ message PushNotificationRegistrationResponse {
```
A client SHOULD listen for a response sent on the [partitioned topic](../../waku/standards/application/54/x3dh-sessions.md) that the key used to register.
A client SHOULD listen for a response sent on the
[partitioned topic](../../waku/standards/application/54/x3dh-sessions.md)
that the key used to register.
If success is true the client has registered successfully.
If `success` is `false`:
> If `MALFORMED_MESSAGE` is returned, the request SHOULD NOT be retried without ensuring that it is correctly formed.
> If `INTERNAL_ERROR` is returned, the request MAY be retried, but the client MUST backoff exponentially.
> If `MALFORMED_MESSAGE` is returned,
the request SHOULD NOT be retried without ensuring that it is correctly formed.
> If `INTERNAL_ERROR` is returned, the request MAY be retried,
but the client MUST backoff exponentially.
#### Handle Errors
#### Handle Errors:
- If the message cant be decrypted, the message MUST be discarded.
- If `token_type` is not supported, a response MUST be sent with `error` set to `UNSUPPORTED_TOKEN_TYPE`.
- If `token`, `installation_id`, `device_tokens`, `version` are empty, a response MUST be sent with `error` set to `MALFORMED_MESSAGE`.
- If `token`, `installation_id`, `device_tokens`, `version` are empty,
a response MUST be sent with `error` set to `MALFORMED_MESSAGE`.
- If the `version` is equal or less than the currently stored `version`, a response MUST be sent with `error` set to `VERSION_MISMATCH`.
- If the `version` is equal or less than the currently stored `version`,
a response MUST be sent with `error` set to `VERSION_MISMATCH`.
- If any other error occurs the `error` SHOULD be set to `INTERNAL_ERROR`.
- If the response is successful `success` MUST be set to `true` otherwise a response MUST be sent with `success` set to `false`.
- If the response is successful `success` MUST be set to `true` otherwise
a response MUST be sent with `success` set to `false`.
- `request_id` SHOULD be set to the [`SHAKE-256`](https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.202.pdf) of the encrypted payload.
- `request_id` SHOULD be set to the
[`SHAKE-256`](https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.202.pdf)
of the encrypted payload.
- The response MUST be sent on the [partitioned topic](../../waku/standards/application/54/x3dh-sessions.md) of the sender and
MUST not be encrypted using the secure transport to facilitate the usage of ephemeral keys.
- The response MUST be sent on the
[partitioned topic](../../waku/standards/application/54/x3dh-sessions.md)
of the sender and MUST not be encrypted using
the secure transport to facilitate the usage of ephemeral keys.
- If no response is returned, the request SHOULD be considered failed and
MAY be retried with the same server or a different one, but clients
MUST exponentially backoff after each trial.
## Push Notification Server
A node that handles receiving and sending push notifications for clients.
### Query Topic:
### Query Topic
On successful registration the server MUST be listening to the topic derived from:
> `0x` + HexEncode(Shake256(CompressedClientPublicKey))
Using the topic derivation algorithm described here and listen for client queries.
#### Server Grant:
A client MUST authorize a push notification server to send them push notifications.
This is done by building a grant which is specific to a given client-server pair.
When receiving a grant, the server MUST validate that the signature matches the registering client.
#### Server Grant
The grant is built as:<br />
`Signature(Keccak256(CompressedPublicKeyOfClient . CompressedPublicKeyOfServer . AccessToken), PrivateKeyOfClient)`
A client MUST authorize a push notification server to send them push notifications.
This is done by building a grant which is specific to a given client-server pair.
When receiving a grant,
the server MUST validate that the signature matches the registering client.
#### Unregistering with a Server:
- To unregister a client MUST send a `PushNotificationRegistration` request as described above with `unregister` set
to `true`, or removing their device information.
- The server MUST remove all data about this user if `unregistering` is `true`, apart from the `hash` of the public key and
the `version` of the last options, in order to make sure that old messages are not processed.
- A client MAY unregister from a server on explicit logout if multiple chat keys are used on a single device.
The grant is built as:
```js
`Signature(Keccak256(CompressedPublicKeyOfClient.CompressedPublicKeyOfServer.AccessToken), PrivateKeyOfClient)`
```
#### Unregistering with a Server
- To unregister a client MUST send a `PushNotificationRegistration`
request as described above with `unregister` set to `true`,
or removing their device information.
- The server MUST remove all data about this user if `unregistering` is `true`,
apart from the `hash` of the public key and
the `version` of the last options,
in order to make sure that old messages are not processed.
- A client MAY unregister from a server on explicit logout
if multiple chat keys are used on a single device.
#### Re-registering with a Server
#### Re-registering with a Server:
- A client SHOULD re-register with the node if the APN or FIREBASE token changes.
- When re-registering a client SHOULD ensure that it has the most up-to-date `PushNotificationRegistration` and
- When re-registering a client SHOULD ensure
that it has the most up-to-date `PushNotificationRegistration` and
increment `version` if necessary.
- Once re-registered, a client SHOULD advertise the changes.
Changing options is handled the same as re-registering.
#### Advertising a Server:
Each user registered with one or more push notification servers
SHOULD advertise periodically the push notification services they have registered with for each device they own.
#### Advertising a Server
Each user registered with one or more push notification servers
SHOULD advertise periodically the push notification services
they have registered with for each device they own.
```protobuf
message PushNotificationQueryInfo {
@@ -229,22 +282,33 @@ message ContactCodeAdvertisement {
```
#### Handle Advertisement Message:
- The message MUST be wrapped in a [`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_QUERY_INFO`.
- If no filtering is done based on public keys, the access token SHOULD be included in the advertisement.
#### Handle Advertisement Message
- The message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_QUERY_INFO`.
- If no filtering is done based on public keys,
the access token SHOULD be included in the advertisement.
Otherwise it SHOULD be left empty.
- This SHOULD be advertised on the [contact code topic](../../waku/standards/application/53/x3dh.md) and
- This SHOULD be advertised on the
[contact code topic](../../waku/standards/application/53/x3dh.md) and
SHOULD be coupled with normal contact-code advertisement.
- When a user register or re-register with a push notification service, their contact-code SHOULD be re-advertised.
- Multiple servers MAY be advertised for the same installation_id for redundancy reasons.
- When a user register or re-register with a push notification service,
their contact-code SHOULD be re-advertised.
- Multiple servers MAY be advertised for the same installation_id
 for redundancy reasons.
#### Discovering a Server:
To discover a push notification service for a given user, their 
[contact code topic](../../waku/standards/application/53/x3dh.md) SHOULD be listened to.
A Waku-Store node can be queried for the specific topic to retrieve the most up-to-date contact code.
#### Discovering a Server
#### Querying a Server:
If a token is not present in the latest advertisement for a user, the server SHOULD be queried directly.
To discover a push notification service for a given user, their
[contact code topic](../../waku/standards/application/53/x3dh.md)
SHOULD be listened to.
A Waku-Store node can be queried for the specific topic
to retrieve the most up-to-date contact code.
#### Querying a Server
If a token is not present in the latest advertisement for a user,
the server SHOULD be queried directly.
To query a server a message:
@@ -255,9 +319,12 @@ message PushNotificationQuery {
```
#### Handle Query Message:
- The message MUST be wrapped in a [`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_QUERY`.
- it MUST be sent to the server on the topic derived from the hashed public key of the key we are querying,
#### Handle Query Message
- The message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_QUERY`.
- it MUST be sent to the server on the topic derived from the hashed public key
of the key we are querying,
[as described above](#query-topic).
- An ephemeral key SHOULD be used and SHOULD NOT be encrypted using the [secure transport](../../waku/standards/application/53/x3dh.md).
@@ -282,32 +349,48 @@ message PushNotificationQueryResponse {
```
#### Handle Query Response:
#### Handle Query Response
- A `PushNotificationQueryResponse` message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_QUERY_RESPONSE`.
Otherwise a response MUST NOT be sent.
- If `allowed_key_list` is not set `access_token` MUST be set and `allowed_key_list` MUST NOT be set.
- If `allowed_key_list` is not set `access_token` MUST be set
and `allowed_key_list` MUST NOT be set.
- If `allowed_key_list` is set `allowed_key_list` MUST be set and `access_token` MUST NOT be set.
- If `allowed_key_list` is set `allowed_key_list` MUST be set and
`access_token` MUST NOT be set.
- If `access_token` is returned, the `access_token` SHOULD be used to send push notifications.
- If `access_token` is returned,
the `access_token` SHOULD be used to send push notifications.
- If `allowed_key_list` are returned, the client SHOULD decrypt each token by generating an `AES-GCM` symmetric key from the DiffieHellman between the target client and itself.
If AES decryption succeeds it will return a valid `uuid` which is what is used for access_token.
- If `allowed_key_list` are returned,
the client SHOULD decrypt each token by generating an `AES-GCM` symmetric key
from the DiffieHellman between the target client and itself.
If AES decryption succeeds,
it will return a valid `uuid` which is what is used for access_token.
The token SHOULD be used to send push notifications.
- The response MUST be sent on the [partitioned topic](../../waku/standards/application/54/x3dh-sessions.md) of the sender and
MUST NOT be encrypted using the [secure transport](../../waku/standards/application/53/x3dh.md) to facilitate the usage of ephemeral keys.
- The response MUST be sent on the
[partitioned topic](../../waku/standards/application/54/x3dh-sessions.md)
of the sender and
MUST NOT be encrypted using the
[secure transport](../../waku/standards/application/53/x3dh.md)
to facilitate the usage of ephemeral keys.
- On receiving a response a client MUST verify `grant` to ensure that the server has been authorized to send push notification to a given client.
- On receiving a response,
a client MUST verify `grant` to ensure that the server
has been authorized to send push notification to a given client.
### Sending Client
Sending a push notification
- When sending a push notification, only the `installation_id` for the devices targeted by the message SHOULD be used.
- When sending a push notification,
only the `installation_id` for the devices targeted by the message SHOULD be used.
- If a message is for all the user devices, all the `installation_id` known to the client MAY be used.
- If a message is for all the user devices,
all the `installation_id` known to the client MAY be used.
- The number of devices MAY be capped in order to reduce resource consumption.
@@ -315,7 +398,8 @@ Sending a push notification
- For any device that a token is available, or that
a token is successfully queried,
a push notification message SHOULD be sent to the corresponding push notification server.
a push notification message SHOULD be sent to the corresponding
push notification server.
```protobuf
message PushNotification {
@@ -339,14 +423,19 @@ message PushNotificationRequest {
}
```
#### Handle Notification Request:
#### Handle Notification Request
- A `PushNotificationRequest` message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_REQUEST`.
- Where `message` is the encrypted payload of the message and `chat_id` is the `SHAKE-256` of the `chat_id`.
`message_id` is the id of the message `author` is the `SHAKE-256` of the public key of the sender.
- Where `message` is the encrypted payload of the message and
`chat_id` is the `SHAKE-256` of the `chat_id`.
`message_id` is the id of the message,
`author` is the `SHAKE-256` of the public key of the sender.
- If multiple server are available for a given push notification, only one notification MUST be sent.
- If multiple server are available for a given push notification,
only one notification MUST be sent.
- If no response is received a client SHOULD wait at least 3 seconds,
after which the request MAY be retried against a different server.
@@ -354,7 +443,7 @@ after which the request MAY be retried against a different server.
- This message SHOULD be sent using an ephemeral key.
On receiving the message, the push notification server MUST validate the access token.
If the access token is valid, a notification MUST be sent to the
If the access token is valid, a notification MUST be sent to the
[gorush](https://github.com/appleboy/gorush) instance with the following data:
```yaml
@@ -405,39 +494,67 @@ message PushNotificationResponse {
Where `message_id` is the `message_id` sent by the client.
#### Handle Notification Response:
- A `PushNotificationResponse` message MUST be wrapped in a [`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_RESPONSE`.
#### Handle Notification Response
- The response MUST be sent on the [partitioned topic](../../waku/standards/application/54/x3dh-sessions.md) of the sender and
MUST not be encrypted using the [secure transport](../../waku/standards/application/53/x3dh.md) to facilitate the usage of ephemeral keys.
- A `PushNotificationResponse` message MUST be wrapped in a
[`ApplicationMetadataMessage`](../62/payloads.md) with type set to `PUSH_NOTIFICATION_RESPONSE`.
- If the request is accepted `success` MUST be set to `true`. Otherwise `success` MUST be set to `false`.
- The response MUST be sent on the
[partitioned topic](../../waku/standards/application/54/x3dh-sessions.md)
of the sender and
MUST not be encrypted using the
[secure transport](../../waku/standards/application/53/x3dh.md)
to facilitate the usage of ephemeral keys.
- If `error` is `BAD_TOKEN` the client MAY query again the server for the token and retry the request.
- If the request is accepted `success` MUST be set to `true`.
Otherwise `success` MUST be set to `false`.
- If `error` is `BAD_TOKEN` the client MAY query again the server for the token and
retry the request.
- If `error` is `INTERNAL_ERROR` the client MAY retry the request.
### Protobuf Description
#### PushNotificationRegistration:
`token_type`: the type of token. Currently supported is `APN_TOKEN` for Apple Push.<br />
`device_token`: the actual push notification token sent by `Firebase` or `APN` and `FIREBASE_TOKEN` for firebase.<br />
`installation_id`: the `installation_id` of the device.<br />
`access_token`: the access token that will be given to clients to send push notifications.<br />
`enabled`: whether the device wants to be sent push notifications.<br />
`version`: a monotonically increasing number identifying the current `PushNotificationRegistration`.
Any time anything is changed in the record it MUST be increased by the client, otherwise the request will not be accepted.<br />
`allowed_key_list`: a list of `access_token` encrypted with the AES key generated by DiffieHellman between the publisher and the
allowed contact.<br />
`blocked_chat_list`: a list of `SHA2-256` hashes of chat ids. Any chat id in this list will not trigger a notification.<br />
`unregister`: whether the account should be unregistered.<br />
`grant`: the grant for this specific server.<br />
`allow_from_contacts_only`: whether the client only wants push notifications from contacts.<br />
`apn_topic`: the APN topic for the push notification.<br />
`block_mentions`: whether the client does not want to be notified on mentions.<br />
`allowed_mentions_chat_list`: a list of SHA2-256 hashes of chat ids where we want to receive mentions.<br />
#### PushNotificationRegistration
`token_type`: the type of token. Currently supported is `APN_TOKEN` for Apple Push.
`device_token`: the actual push notification token sent by `Firebase` or
`APN` and `FIREBASE_TOKEN` for firebase.
`installation_id`: the `installation_id` of the device.
`access_token`: the access token that will be given to clients to send push notifications.
`enabled`: whether the device wants to be sent push notifications.
`version`: a monotonically increasing number identifying the current `PushNotificationRegistration`.
Any time anything is changed in the record it MUST be increased by the client,
otherwise the request will not be accepted.
`allowed_key_list`: a list of `access_token` encrypted with the AES key
generated by DiffieHellman between the publisher and the allowed contact.
`blocked_chat_list`: a list of `SHA2-256` hashes of chat ids.
Any chat id in this list will not trigger a notification.
`unregister`: whether the account should be unregistered.
`grant`: the grant for this specific server.
`allow_from_contacts_only`: whether the client only wants
push notifications from contacts.
`apn_topic`: the APN topic for the push notification.
`block_mentions`: whether the client does not want to be notified on mentions.
`allowed_mentions_chat_list`: a list of SHA2-256 hashes of chat ids
where we want to receive mentions.
DATA DISCLOSED
- Type of device owned by a given user.
- The `FIREBASE` or `APN` push notification token,
@@ -448,48 +565,71 @@ DATA DISCLOSED
- The number of contacts a client has, in case `allowed_key_list` is set.
#### PushNotificationRegistrationResponse:
`success`: whether the registration was successful<br />
`error`: the error type, if any<br />
`request_id`: the `SHAKE-256` hash of the `signature` of the request<br />
`preferences`: the server stored preferences in case of an error<br />
#### PushNotificationRegistrationResponse
`success`: whether the registration was successful
`error`: the error type, if any
`request_id`: the `SHAKE-256` hash of the `signature` of the request
`preferences`: the server stored preferences in case of an error
#### ContactCodeAdvertisement
#### ContactCodeAdvertisement:
`push_notification_info`: the information for each device advertised
DATA DISCLOSED
- The chat key of the sender
#### PushNotificationQuery:
#### PushNotificationQuery
`public_keys`: the `SHAKE-256` of the public keys the client is interested in
DATA DISCLOSED
- The hash of the public keys the client is interested in
#### PushNotificationQueryInfo:
`access_token`: the access token used to send a push notification<br />
`installation_id`: the `installation_id` of the device associated with the `access_token`<br />
`public_key`: the `SHAKE-256` of the public key associated with this `access_token` and `installation_id`.<br />
`allowed_key_list`: a list of encrypted access tokens to be returned to the client in case theres any filtering on public keys in place.<br />
`grant`: the grant used to register with this server.<br />
`version`: the version of the registration on the server.<br />
`server_public_key`: the compressed public key of the server.<br />
#### PushNotificationQueryInfo
#### PushNotificationQueryResponse:
`info`: a list of `PushNotificationQueryInfo`.<br />
`message_id`: the message id of the `PushNotificationQueryInfo` the server is replying to.<br />
`success`: whether the query was successful.<br />
`access_token`: the access token used to send a push notification
#### PushNotification:
`access_token`: the access token used to send a push notification.<br />
`chat_id`: the `SHAKE-256` of the `chat_id`.<br />
`public_key`: the `SHAKE-256` of the compressed public key of the receiving client.<br />
`installation_id`: the `installation_id` of the receiving client.<br />
`message`: the encrypted message that is being notified on.<br />
`type`: the type of the push notification, either `MESSAGE` or `MENTION`<br />
`installation_id`: the `installation_id` of the device associated with the `access_token`.
`public_key`: the `SHAKE-256` of the public key associated with this `access_token`
and `installation_id`.
`allowed_key_list`: a list of encrypted access tokens to be returned
to the client in case theres any filtering on public keys in place.
`grant`: the grant used to register with this server.
`version`: the version of the registration on the server.
`server_public_key`: the compressed public key of the server.
#### PushNotificationQueryResponse
`info`: a list of `PushNotificationQueryInfo`.
`message_id`: the message id of the `PushNotificationQueryInfo`
the server is replying to.
`success`: whether the query was successful.
#### PushNotification
`access_token`: the access token used to send a push notification.
`chat_id`: the `SHAKE-256` of the `chat_id`.
`public_key`: the `SHAKE-256` of the compressed public key of the receiving client.
`installation_id`: the `installation_id` of the receiving client.
`message`: the encrypted message that is being notified on.
`type`: the type of the push notification, either `MESSAGE` or `MENTION`
`author`: the `SHAKE-256` of the public key of the sender
Data disclosed
- The `SHAKE-256` hash of the `chat_id` the notification is to be sent for
- The cypher text of the message
@@ -498,55 +638,75 @@ Data disclosed
- The type of notification
#### PushNotificationRequest:
`requests`: a list of PushNotification<br />
#### PushNotificationRequest
`requests`: a list of PushNotification
`message_id`: the [Status message id](../62/payloads.md)
Data disclosed
- The status `message_id` for which the notification is for
#### PushNotificationResponse:
`message_id`: the `message_id` being notified on.<br />
#### PushNotificationResponse
`message_id`: the `message_id` being notified on.
`reports`: a list of `PushNotificationReport`
#### PushNotificationReport:
`success`: whether the push notification was successful.<br />
`error`: the type of the error in case of failure.<br />
`public_key`: the public key of the user being notified.<br />
#### PushNotificationReport
`success`: whether the push notification was successful.
`error`: the type of the error in case of failure.
`public_key`: the public key of the user being notified.
`installation_id`: the `installation_id` of the user being notified.
### Anonymous Mode
In order to preserve privacy, the client MAY provide anonymous mode of operations to propagate information about the user.
A client in anonymous mode can register with the server using a key that is different from their chat key.
This will hide their real chat key. This public key is effectively a secret and
SHOULD only be disclosed to clients approved to notify a user.
- A client MAY advertise the access token on the [contact-code topic](../../waku/standards/application/53/x3dh.md) of the key generated.
In order to preserve privacy, the client MAY provide anonymous mode of operations
to propagate information about the user.
A client in anonymous mode can register with the server
using a key that is different from their chat key.
This will hide their real chat key. This public key is effectively a secret and
SHOULD only be disclosed to clients approved to notify a user.
- A client MAY share their public key contact updates in the [protobuf record](https://developers.google.com/protocol-buffers/).
- A client MAY advertise the access token on the
[contact-code topic](../../waku/standards/application/53/x3dh.md)
of the key generated.
- A client receiving a push notification public key SHOULD listen to the contact code topic of the push notification public key for updates.
- A client MAY share their public key contact updates in the
[protobuf record](https://developers.google.com/protocol-buffers/).
The method described above effectively does not share the identity of the sender nor the receiver to the server, but
MAY result in missing push notifications as the propagation of the secret is left to the client.
This can be mitigated by [device syncing](../62/payloads.md), but not completely addressed.
- A client receiving a push notification public key
SHOULD listen to the contact code topic of the push notification public key for updates.
The method described above effectively does not share the identity of the sender
nor the receiver to the server, but
MAY result in missing push notifications
as the propagation of the secret is left to the client.
This can be mitigated by [device syncing](../62/payloads.md),
but not completely addressed.
## Security/Privacy Considerations
If anonymous mode is not used, when registering with a push notification service a client will disclose:
If anonymous mode is not used,
when registering with a push notification service a client will disclose:
- The devices that will receive notifications.
- The chat key.
A client MAY disclose:
- The hash of the `chat_id` they want to filter out.
When running in anonymous mode, the clients chat key is not disclosed.
When running in anonymous mode, the clients chat key is not disclosed.
When querying a push notification server a client will disclose:
- That it is interested in sending push notification to another client, but
querying clients chat key is not disclosed.
- That it is interested in sending push notification to another client,
but querying clients chat key is not disclosed.
When sending a push notification a client will disclose:
- The `shake-256` of the `chat_id`.
## Copyright
@@ -554,6 +714,7 @@ When sending a push notification a client will disclose:
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [PUSH-NOTIFICATION-SERVER, Initial Specification](https://github.com/status-im/specs/blob/master/docs/raw/push-notification-server.md)
2. [Push Notification, Apple Developer](https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html#//apple_ref/doc/uid/TP40008194-CH8-SW1)
3. [Firebase](https://firebase.google.com)

View File

@@ -1,4 +1,4 @@
# Status RFCs
Status is a communitication tool providing privacy features for the user.
Specifcations can also be viewd at [Status](https://status.app/specs).
Status is a communication tool providing privacy features for the user.
Specifications can also be viewed at [Status](https://status.app/specs).

View File

@@ -0,0 +1,127 @@
---
title: 3RD-PARTY
name: 3rd party
status: deprecated
description: This specification discusses 3rd party APIs that Status relies on.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Volodymyr Kozieiev <volodymyr@status.im>
---
## Abstract
This specification discusses 3rd party APIs that Status relies on.
These APIs provide various capabilities, including:
- communicating with the Ethereum network,
- allowing users to view address and transaction details on external websites,
- retrieving fiat/crypto exchange rates,
- obtaining information about collectibles,
- hosting the privacy policy.
## Definitions
| Term | Description |
|-------------------|-------------------------------------------------------------------------------------------------------|
| Fiat money | Currency established as money, often by government regulation, but without intrinsic value. |
| Full node | A computer, connected to the Ethereum network, that enforces all Ethereum consensus rules. |
| Crypto-collectible| A unique, non-fungible digital asset, distinct from cryptocurrencies where tokens are identical. |
## Why 3rd Party APIs Can Be a Problem
Relying on 3rd party APIs conflicts with Statuss censorship-resistance principle.
Since Status aims to avoid suppression of information,
it is important to minimize reliance on 3rd parties that are critical to app functionality.
## 3rd Party APIs Used by the Current Status App
### Infura
**What is it?**
Infura hosts a collection of Ethereum full nodes and provides an API
to access the Ethereum and IPFS networks without requiring a full node.
**How Status Uses It**
Since Status operates on mobile devices,
it cannot rely on a local node.
Therefore, all Ethereum network communication happens via Infura.
**Concerns**
Making an HTTP request can reveal user metadata,
which could be exploited in attacks if Infura is compromised.
Infura uses centralized hosting providers;
if these providers fail or cut off service,
Ethereum-dependent features in Status would be affected.
### Etherscan
**What is it?**
Etherscan is a service that allows users to explore the Ethereum blockchain
for transactions, addresses, tokens, prices,
and other blockchain activities.
**How Status Uses It**
The Status Wallet allows users to view address and transaction details on Etherscan.
**Concerns**
If Etherscan becomes unavailable,
users wont be able to view address or transaction details through Etherscan.
However, in-app information will still be accessible.
### CryptoCompare
**What is it?**
CryptoCompare provides live crypto prices, charts, and analysis from major exchanges.
**How Status Uses It**
Status regularly fetches crypto prices from CryptoCompare,
using this information to calculate fiat values
for transactions or wallet assets.
**Concerns**
HTTP requests can reveal metadata,
which could be exploited if CryptoCompare is compromised.
If CryptoCompare becomes unavailable,
Status wont be able to show fiat equivalents for crypto in the wallet.
### Collectibles
Various services provide information on collectibles:
- [Service 1](https://api.pixura.io/graphql)
- [Service 2](https://www.etheremon.com/api)
- [Service 3](https://us-central1-cryptostrikers-prod.cloudfunctions.net/cards/)
- [Service 4](https://api.cryptokitties.co/)
**Concerns**
HTTP requests can reveal metadata,
which could be exploited if these services are compromised.
### Iubenda
**What is it?**
Iubenda helps create compliance documents for websites and apps across jurisdictions.
**How Status Uses It**
Statuss privacy policy is hosted on Iubenda.
**Concerns**
If Iubenda becomes unavailable,
users will be unable to view the app's privacy policy.
## Changelog
| Version | Comment |
|---------|-----------------|
| 0.1.0 | Initial release |
## Copyright
Copyright and related rights waived via CC0.
## References
- [GraphQL](https://api.pixura.io/graphql)
- [Etheremon](https://www.etheremon.com/api)
- [Cryptostrikers](https://us-central1-cryptostrikers-prod.cloudfunctions.net/cards/)
- [Cryptokitties](https://api.cryptokitties.co/)

View File

@@ -0,0 +1,138 @@
---
title: IPFS-gateway-for-Sticker-Pack
name: IPFS gateway for Sticker Pack
status: deprecated
description: This specification describes how Status uses the IPFS gateway to store stickers.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Gheorghe Pinzaru <gheorghe@status.im>
---
## Abstract
This specification describes how Status uses the IPFS gateway
to store stickers.
The specification explores image format,
how a user uploads stickers,
and how an end user can see them inside the Status app.
## Definition
| Term | Description |
|------------------|----------------------------------------------------------------------------------------|
| **Stickers** | A set of images which can be used to express emotions |
| **Sticker Pack** | ERC721 token which includes the set of stickers |
| **IPFS** | P2P network used to store and share data, in this case, the images for the stickerpack |
## Specification
### Image format
Accepted image file types are `PNG`, `JPG/JPEG` and `GIF`,
with a maximum allowed size of 300kb.
The minimum sticker image resolution is 512x512,
and its background SHOULD be transparent.
### Distribution
The node implements sticker packs as [ERC721 token](https://eips.ethereum.org/EIPS/eip-721)
and contain a set of stickers.
The node stores these stickers inside the sticker pack as a set of hyperlinks pointing to IPFS storage.
These hyperlinks are publicly available and can be accessed by any user inside the status chat.
Stickers can be sent in chat only by accounts that own the sticker pack.
### IPFS gateway
At the moment of writing, the current main Status app uses the [Infura](https://infura.io/) gateway.
However, clients could choose a different gateway or to run own IPFS node.
Infura gateway is an HTTPS gateway,
which based on an HTTP GET request with the multihash block will return the stored content at that block address.
The node requires the use of a gateway to enable easy access to the resources over HTTP.
The node stores each image of a sticker inside IPFS using a unique address that is
derived from the hash of the file.
This ensures that a file can't be overridden,
and an end-user of the IPFS will receive the same file at a given address.
### Security
The IPFS gateway acts as an end-user of the IPFS
and allows users of the gateway to access IPFS without connection to the P2P network.
Usage of a gateway introduces potential risk for the users of that gateway provider.
In case of a compromise in the security of the provider, meta information such as IP address,
User-Agent and other of its users can be leaked.
If the provider servers are unavailable the node loses access through the gateway to the IPFS network.
### Status sticker usage
When the app shows a sticker, the Status app makes an HTTP GET request to IPFS gateway using the hyperlink.
To send a sticker in chat, a user of Status should buy or install a sticker pack.
To be available for installation a Sticker Pack should be submitted to Sticker market by an author.
#### Submit a sticker
To submit a sticker pack, the author should upload all assets to IPFS.
Then generate a payload including name, author, thumbnail,
preview and a list of stickers in the [EDN format](https://github.com/edn-format/edn). Following this structure:
``
{meta {:name "Sticker pack name"
:author "Author Name"
:thumbnail "e30101701220602163b4f56c747333f43775fdcbe4e62d6a3e147b22aaf6097ce0143a6b2373"
:preview "e30101701220ef54a5354b78ef82e542bd468f58804de71c8ec268da7968a1422909357f2456"
:stickers [{:hash "e301017012207737b75367b8068e5bdd027d7b71a25138c83e155d1f0c9bc5c48ff158724495"}
{:hash "e301017012201a9cdea03f27cda1aede7315f79579e160c7b2b6a2eb51a66e47a96f47fe5284"}]}}
``
All asset fields, are contenthash fields as per [EIP 1577](https://eips.ethereum.org/EIPS/eip-1577).
The node also uploads this payload to IPFS, and the node uses the IPFS address in the content field of the Sticker Market contract.
See [Sticker Market spec](https://github.com/status-im/sticker-market/blob/651e88e5f38c690e57ecaad47f46b9641b8b1e27/docs/specification.md) for a detailed description of the contract.
#### Install a sticker pack
To install a sticker pack, the node fetches all sticker packs available in Sticker Market.
The node needs the following steps to fetch all sticker packs:
#### 1. Get total number of sticker packs
Call `packCount()` on the sticker market contract, will return number of sticker pack registered as `uint256`.
#### 2. Get sticker pack by id
ID's are represented as `uint256` and are incremental from `0` to total number of sticker packs in the contract,
received in the previous step.
To get a sticker pack call `getPackData(sticker-pack-id)`, the return type is `["bytes4[]" "address" "bool" "uint256" "uint256" "bytes"]`
which represents the following fields: `[category owner mintable timestamp price contenthash]`.
Price is the SNT value in wei set by sticker pack owner.
The contenthash is the IPFS address described in the [submit description](#submit-a-sticker) above.
Other fields specification could be found in [Sticker Market spec](https://github.com/status-im/sticker-market/blob/651e88e5f38c690e57ecaad47f46b9641b8b1e27/docs/specification.md)
##### 3. Get owned sticker packs
The current Status app fetches owned sticker packs during the open of any sticker view
(a screen which shows a sticker pack, or the list of sticker packs).
To get owned packs, get all owned tokens for the current account address,
by calling `balanceOf(address)` where address is the address for the current account.
This method returns a `uint256` representing the count of available tokens. Using `tokenOfOwnerByIndex(address,uint256)` method,
with the address of the user and ID in form of a `uint256`
which is an incremented int from 0 to the total number of tokens, gives the token id.
To get the sticker pack id from a token call`tokenPackId(uint256)` where `uint256` is the token id.
This method will return an `uint256` which is the id of the owned sticker pack.
##### 4. Buy a sticker pack
To buy a sticker pack call `approveAndCall(address,uint256,bytes)`
where `address` is the address of buyer,`uint256` is the price and third parameters `bytes` is the callback called if approved.
In the callback, call `buyToken(uint256,address,uint256)`, first parameter is sticker pack id, second buyers address, and the last is the price.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [ERC721 Token Standard](https://eips.ethereum.org/EIPS/eip-721)
- [Infura](https://infura.io/)
- [EDN Format](https://github.com/edn-format/edn)
- [EIP 1577](https://eips.ethereum.org/EIPS/eip-1577)
- [Sticker Market Specification](https://github.com/status-im/sticker-market/blob/651e88e5f38c690e57ecaad47f46b9641b8b1e27/docs/specification.md)

View File

@@ -0,0 +1,461 @@
---
title: ACCOUNT
name: Account
status: deprecated
description: This specification explains what a Status account is, and how a node establishes trust.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Corey Petty <corey@status.im>
- Oskar Thorén <oskar@status.im>
- Samuel Hawksby-Robinson <samuel@status.im>
---
## Abstract
This specification explains what a Status account is,
and how a node establishes trust.
## Introduction
The core concept of an account in Status is a set of cryptographic keypairs.
Namely, the combination of the following:
1. a Whisper/Waku chat identity keypair
1. a set of cryptocurrency wallet keypairs
The node verifies or derives everything else associated with the contact from the above items, including:
- Ethereum address (future verification, currently the same base keypair)
- 3 word mnemonic name
- identicon
- message signatures
## Initial Key Generation
### Public/Private Keypairs
- An ECDSA (secp256k1 curve) public/private keypair MUST be generated via a [BIP43](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki) derived path from a [BIP39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic seed phrase.
- The default paths are defined as such:
- Whisper/Waku Chat Key (`IK`): `m/43'/60'/1581'/0'/0` (post Multiaccount integration)
- following [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md)
<!-- WE CURRENTLY DO NOT IMPLEMENT ENCRYPTION KEY, FOR FUTURE - C.P. -->
<!-- - DB encryption Key (`DBK`): `m/43'/60'/1581'/1'/0` (post Multiaccount integration) -->
<!-- - following [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md) -->
- Status Wallet paths: `m/44'/60'/0'/0/i` starting at `i=0`
- following [BIP44](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
- NOTE: this (`i=0`) is also the current (and only) path for Whisper/Waku key before Multiaccount integration
### X3DH Prekey bundle creation
- Status follows the X3DH prekey bundle scheme that [Open Whisper Systems](https://en.wikipedia.org/wiki/Signal_Messenger#2013%E2%80%932018:_Open_Whisper_Systems) (not to be confused with the Whisper sub-protocol) outlines [in their documentation](https://signal.org/docs/specifications/x3dh/#the-x3dh-protocol) with the following exceptions:
- Status does not publish one-time keys `OPK` or perform DH including them, because there are no central servers in the Status implementation.
- A client MUST create X3DH prekey bundles, each defined by the following items:
- Identity Key: `IK`
- Signed prekey: `SPK`
- Prekey signature: `Sig(IK, Encode(SPK))`
- Timestamp
- These bundles are made available in a variety of ways, as defined in section 2.1.
## Account Broadcasting
- A user is responsible for broadcasting certain information publicly so that others may contact them.
### X3DH Prekey bundles
- A client SHOULD regenerate a new X3DH prekey bundle every 24 hours. This MAY be done in a lazy way, such that a client that does not come online past this time period does not regenerate or broadcast bundles.
- The current bundle SHOULD be broadcast on a Whisper/Waku topic specific to his Identity Key, `{IK}-contact-code`, intermittently. This MAY be done every 6 hours.
- A bundle SHOULD accompany every message sent.
- TODO: retrieval of long-time offline users bundle via `{IK}-contact-code`
## Optional Account additions
### ENS Username
- A user MAY register a public username on the Ethereum Name System (ENS). This username is a user-chosen subdomain of the `stateofus.eth` ENS registration that maps to their Whisper/Waku identity key (`IK`).
<!-- ### User Profile Picture
- An account MAY edit the `IK` generated identicon with a chosen picture. This picture will become part of the publicly broadcast profile of the account. -->
<!-- TODO: Elaborate on wallet account and multiaccount -->
<!-- TODO: Elaborate on security implications -->
## Trust establishment
**Trust establishment deals with users verifying they are communicating with who they think they are.**
### Terms Glossary
| term | description |
| ------------------------- | ----------- |
| privkey | ECDSA secp256k1 private key |
| pubkey | ECDSA secp256k1 public key |
| Whisper/Waku key | pubkey for chat with HD derivation path m/43'/60'/1581'/0'/0 |
### Contact Discovery
#### Public channels
- Public group channels in Status are a broadcast/subscription system. All public messages are encrypted with a symmetric key derived from the channel name, `K_{pub,sym}`, which is publicly known.
- A public group channel's symmetric key MUST creation must follow the [web3 API](https://web3js.readthedocs.io/en/1.0/web3-shh.html#generatesymkeyfrompassword)'s `web3.ssh.generateSymKeyFromPassword` function
- In order to post to a public group channel, a client MUST have a valid account created.
- In order to listen to a public group channel, a client must subscribe to the channel name.
The sender of a message is derived from the message's signature.
- Discovery of channel names is not currently part of the protocol, and is typically done out of band.
If a channel name is used that has not been used, it will be created.
- A client MUST sign the message otherwise it will be discarded by the recipients.
- channel name specification:
- matches `[a-z0-9\-]`
- is not a public key
#### Private 1:1 messages
This can be done in the following ways:
1. scanning a user generated QR code
1. discovery through the Status app
1. asynchronous X3DH key exchange
1. public key via public channel listening
- `status-mobile/src/status_im/contact_code/core.cljs`
1. contact codes
1. decentralized storage (not implemented)
1. Whisper/Waku
### Initial Key Exchange
#### Bundles
- An X3DH prekey bundle is defined as ([code](https://github.com/status-im/status-go/messaging/chat/protobuf/encryption.pb.go)):
```golang
Identity // Identity key
SignedPreKeys // a map of installation id to array of signed prekeys by that installation id
Signature // Prekey signature
Timestamp // When the bundle was lasted created locally
```
- include BundleContainer
- a new bundle SHOULD be created at least every 12 hours
- a node only generates a bundle when it is used
- a bundle SHOULD be distributed on the contact code channel. This is the Whisper and Waku topic `{IK}-contact-code`,
where `IK` is the hex encoded public key of the user, prefixed with `0x`.
The node encrypts the channel in the same way it encrypted public chats.
### Contact Verification
To verify that contact key information is as it should be, use the following.
#### Identicon
A low-poly identicon is deterministically generated from the Whisper/Waku chat public key.
This can be compared out of band to ensure the receiver's public key is the one stored locally.
#### 3 word pseudonym / Whisper/Waku key fingerprint
Status generates a deterministic 3-word random pseudonym from the Whisper/Waku chat public key.
This pseudonym acts as a human readable fingerprint to the Whisper/Waku chat public key.
This name also shows when viewing a contact's public profile and in the chat UI.
- implementation: [gfycat](https://github.com/status-im/status-mobile/tree/develop/src/status_im/utils/gfycat)
#### ENS name
Status offers the ability to register a mapping of a human readable subdomain of `stateofus.eth` to their Whisper/Waku chat public key.
The user purchases this registration (currently by staking 10 SNT)
and the node stores it on the Ethereum mainnet blockchain for public lookup.
<!-- TODO: Elaborate on security implications -->
<!-- TODO: Incorporate or cut below into proper spec
### Possible Connection Breakdown
possible connections
- client - client (not really ever, this is facilitated through all other connections)
- personal chat
- ratcheted with X3DH
- private group chat
- pairwise ratcheted with X3DH
- public chat
- client - mailserver (statusd + ???)
- a mailserver identifies itself by an [enode address](https://github.com/ethereum/wiki/wiki/enode-url-format)
- client - Whisper/Waku node (statusd)
- a node identifies itself by an enode address
- client - bootnode (go-ethereum)
- a bootnode identifies itself by
- an enode address
- `NOTE: redezvous information here`
- client - ENS registry (ethereum blockchain -> default to infura)
- client - Ethereum RPC (custom go-ethereum RPC API -> default to infura API)
- client - IPFS (Status hosted IPFS gateway -> defaults to ???)
- we have a status hosted IPFS gateway for pinning but it currently isn't used much.
### Notes
A user in the system is a public-private key pair using the Elliptic-Curve Cryptography secp256k1 that Ethereum uses.
- A 3-word random name is derived from the public key using the following package
- `NOTE: need to find package`
- This provides an associated human-readble fingerprint to the user's public key
- A user can optionally add additional layers on top of this keypair
- Chosen username
- ENS username
All messages sent are encrypted with the public key of the destination and signed by the private key of the given user using the following scheme:
- private chat
- X3DH is used to define shared secrets which is then double ratcheted
- private group chat
- considered pairwise private chats
- public group chat
- the message is encrypted with a symmetric key derived from the chat name
-->
## Public Key Serialization
Idiomatically known as "public key compression" and "public key decompression".
The node SHOULD provide functionality for the serialization and deserialization of public / chat keys.
For maximum flexibility, when implementing this functionality, the node MUST support public keys encoded in a range of encoding formats, detailed below.
### Basic Serialization Example
In the example of a typical hexadecimal encoded elliptical curve (EC) public key (such as a secp256k1 pk),
```text
0x04261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc3cd4e22200c68daf7493e1f8da6a190a68a671e2d3977809612424c7c3888bc6
```
minor modification for compatibility and flexibility makes the key self-identifiable and easily parsable,
```text
fe70104261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc3cd4e22200c68daf7493e1f8da6a190a68a671e2d3977809612424c7c3888bc6
```
EC serialization and compact encoding produces a much smaller string representation of the original key.
```text
zQ3shPyZJnxZK4Bwyx9QsaksNKDYTPmpwPvGSjMYVHoXHeEgB
```
### Public Key "Compression" Rationale
Serialized and compactly encoded ("compressed") public keys have a number of UI / UX advantages
over non-serialized less densely encoded public keys.
Compressed public keys are smaller, and users may perceive them as less intimidating and less unnecessarily large.
Compare the "compressed" and "uncompressed" version of the same public key from above example:
- `0xe70104261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc3cd4e22200c68daf7493e1f8da6a190a68a671e2d3977809612424c7c3888bc6`
- `zQ3shPyZJnxZK4Bwyx9QsaksNKDYTPmpwPvGSjMYVHoXHeEgB`
The user can transmit and share the same data, but at one third of the original size.
136 characters uncompressed vs 49 characters compressed, giving a significant character length reduction of 64%.
The user client app MAY use the compressed public keys throughout the user interface.
For example in the `status-mobile` implementation of the user interface
the following places could take advantage of a significantly smaller public key:
- `Onboarding` > `Choose a chat name`
- `Profile` > `Header`
- `Profile` > `Share icon` > `QR code popover`
- `Invite friends` url from `Invite friends` button and `+ -button` > `Invite friends`
- Other user `Profile details`
- `Profile details` > `Share icon` > `QR code popover`
In the case of QR codes a compressed public key can reduce the complexity of the derived codes:
| Uncompressed |
| --- |
|![image](/status/deprecated/images/qr-code1-accountmd.png) |
| Compressed |
| --- |
| ![image](/status/deprecated/images/qr-code2-accountmd.png)|
### Key Encoding
When implementing the pk de/serialization functionality, the node MUST use the [multiformats/multibase](https://github.com/multiformats/multibase)
encoding protocol to interpret incoming key data and to return key data in a desired encoding.
The node SHOULD support the following `multibase` encoding formats.
```csv
encoding, code, description, status
identity, 0x00, 8-bit binary (encoder and decoder keeps data unmodified), default
base2, 0, binary (01010101), candidate
base8, 7, octal, draft
base10, 9, decimal, draft
base16, f, hexadecimal, default
base16upper, F, hexadecimal, default
base32hex, v, rfc4648 case-insensitive - no padding - highest char, candidate
base32hexupper, V, rfc4648 case-insensitive - no padding - highest char, candidate
base32hexpad, t, rfc4648 case-insensitive - with padding, candidate
base32hexpadupper, T, rfc4648 case-insensitive - with padding, candidate
base32, b, rfc4648 case-insensitive - no padding, default
base32upper, B, rfc4648 case-insensitive - no padding, default
base32pad, c, rfc4648 case-insensitive - with padding, candidate
base32padupper, C, rfc4648 case-insensitive - with padding, candidate
base32z, h, z-base-32 (used by Tahoe-LAFS), draft
base36, k, base36 [0-9a-z] case-insensitive - no padding, draft
base36upper, K, base36 [0-9a-z] case-insensitive - no padding, draft
base58btc, z, base58 bitcoin, default
base58flickr, Z, base58 flicker, candidate
base64, m, rfc4648 no padding, default
base64pad, M, rfc4648 with padding - MIME encoding, candidate
base64url, u, rfc4648 no padding, default
base64urlpad, U, rfc4648 with padding, default
```
**Note** this specification RECOMMENDs that implementations extend the standard `multibase` protocol
to parse strings prepended with `0x` as `f` hexadecimal encoded bytes.
Implementing this recommendation will allow the node to correctly interpret traditionally identified hexadecimal strings (e.g. `0x1337c0de`).
*Example:*
`0xe70102261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc`
SHOULD be interpreted as
`fe70102261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc`
This specification RECOMMENDs that the consuming service of the node uses a compact encoding type,
such as base64 or base58 to allow for as short representations of the key as possible.
### Public Key Types
When implementing the pk de/serialization functionality, The node MUST support the [multiformats/multicodec](https://github.com/multiformats/multicodec) key type identifiers for the following public key type.
| Name | Tag | Code | Description |
| ------------------ | --- | ------ | ------------------------------------ |
| `secp256k1-pub` | key | `0xe7` | Secp256k1 public key |
For a public key to be identifiable to the node the public key data MUST be prepended with the relevant [multiformats/unsigned-varint](https://github.com/multiformats/unsigned-varint) formatted code.
*Example:*
Below is a representation of an deserialized secp256k1 public key.
```text
04
26 | 1c | 55 | 67 | 5e | 55 | ff | 25
ed | b5 | 0b | 34 | 5c | fb | 3a | 3f
35 | f6 | 07 | 12 | d2 | 51 | cb | aa
ab | 97 | bd | 50 | 05 | 4c | 6e | bc
3c | d4 | e2 | 22 | 00 | c6 | 8d | af
74 | 93 | e1 | f8 | da | 6a | 19 | 0a
68 | a6 | 71 | e2 | d3 | 97 | 78 | 09
61 | 24 | 24 | c7 | c3 | 88 | 8b | c6
```
The `multicodec` code for a secp256k1 public key is `0xe7`.
After parsing the code `0xe7` as a `multiformats/uvarint`, the byte value is `0xe7 0x01`, prepending this to the public key results in the below representation.
```text
e7 | 01 | 04
26 | 1c | 55 | 67 | 5e | 55 | ff | 25
ed | b5 | 0b | 34 | 5c | fb | 3a | 3f
35 | f6 | 07 | 12 | d2 | 51 | cb | aa
ab | 97 | bd | 50 | 05 | 4c | 6e | bc
3c | d4 | e2 | 22 | 00 | c6 | 8d | af
74 | 93 | e1 | f8 | da | 6a | 19 | 0a
68 | a6 | 71 | e2 | d3 | 97 | 78 | 09
61 | 24 | 24 | c7 | c3 | 88 | 8b | c6
```
### De/Serialization Process Flow
When implementing the pk de/serialization functionality, the node MUST be passed a `multicodec` identified public key,
of the above supported types, encoded with a valid `multibase` identifier.
This specification RECOMMENDs that the node also accept an encoding type parameter to encode the output data.
This provides for the case where the user requires the de/serialization key to be in a different encoding to the encoding of the given key.
#### Serialization Example
A hexadecimal encoded secp256k1 public chat key typically is represented as below:
```text
0x04261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc3cd4e22200c68daf7493e1f8da6a190a68a671e2d3977809612424c7c3888bc6
```
To be properly interpreted by the node for serialization the public key MUST be prepended with the `multicodec` `uvarint` code `0xea 0x01`
and encoded with a valid `multibase` encoding, therefore giving the following:
```text
fea0104261c55675e55ff25edb50b345cfb3a3f35f60712d251cbaaab97bd50054c6ebc3cd4e22200c68daf7493e1f8da6a190a68a671e2d3977809612424c7c3888bc6
```
If adhering to the specification recommendation to provide the user with an output encoding parameter,
the above string would be passed to the node with the following `multibase` encoding identifier.
In this example the output encoding is defined as `base58 bitcoin`.
```text
z
```
The return value in this case would be
```text
zQ3shPyZJnxZK4Bwyx9QsaksNKDYTPmpwPvGSjMYVHoXHeEgB
```
Which after `multibase` decoding can be represented in bytes as below:
```text
e7 | 01 | 02
26 | 1c | 55 | 67 | 5e | 55 | ff | 25
ed | b5 | 0b | 34 | 5c | fb | 3a | 3f
35 | f6 | 07 | 12 | d2 | 51 | cb | aa
ab | 97 | bd | 50 | 05 | 4c | 6e | bc
```
#### Deserialization Example
For the user, the deserialization process is exactly the same as serialization with the exception
that the user MUST provide a serialized public key for deserialization. Else the deserialization algorithm will fail.
For further guidance on the implementation of public key de/serialization consult the [`status-go` implementation and tests](https://github.com/status-im/status-go/blob/c9772325f2dca76b3504191c53313663ca2efbe5/api/utils_test.go).
## Security Considerations
-
## Changelog
### Version 0.4
Released [June 24, 2020](https://github.com/status-im/specs/commit/e98a9b76b7d4e1ce93e0b692e1521c2d54f72c59)
- Added details of public key serialization and deserialization
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
- Added language to include Waku in all relevant places
- Change to keep `Mailserver` term consistent
- Added clarification to Open Whisper Systems
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [BIP43](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki)
- [BIP39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki)
- [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md)
- [BIP44](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
- [Open Whisper Systems](https://en.wikipedia.org/wiki/Signal_Messenger#2013%E2%80%932018:_Open_Whisper_Systems)
- [X3DH](https://signal.org/docs/specifications/x3dh/#the-x3dh-protocol)
- [web3 API](https://web3js.readthedocs.io/en/1.0/web3-shh.html#generatesymkeyfrompassword)
- [Protobuf encryption](https://github.com/status-im/status-go/messaging/chat/protobuf/encryption.pb.go)
- [gfycat in Status](https://github.com/status-im/status-mobile/tree/develop/src/status_im/utils/gfycat)
- [multiformats](https://github.com/multiformats/)
- [status-go implementation and tests](https://github.com/status-im/status-go/blob/c9772325f2dca76b3504191c53313663ca2efbe5/api/utils_test.go)
- [June 24, 2020 change commit](https://github.com/status-im/specs/commit/e98a9b76b7d4e1ce93e0b692e1521c2d54f72c59)
- [May 22, 2020 change commit](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)

427
status/deprecated/client.md Normal file
View File

@@ -0,0 +1,427 @@
---
title: CLIENT
name: Client
status: deprecated
description: This specification describes how to write a Status client for communicating with other Status clients.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Babik <adam@status.im>
- Andrea Maria Piana <andreap@status.im>
- Dean Eigenmann <dean@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskar@status.im>
- Samuel Hawksby-Robinson <samuel@status.im>
---
## Abstract
This specification describes how to write a Status client for communicating
with other Status clients.
This specification presents a reference implementation of the protocol
used in a command-line client and a mobile app.
This document consists of two parts.
The first outlines the specifications required to be a full Status client.
The second provides a design rationale and answers some common questions.
## Introduction
### Protocol layers
Implementing a Status clients largely means implementing the following layers.
Additionally, there are separate specifications for things like key management and account lifecycle.
Other aspects, such as how a node uses IPFS for stickers or how the browser works, are currently underspecified.
These specifications facilitate the implementation of a Status client for basic private communication.
| Layer | Purpose | Technology |
| ----------------- | ------------------------------ | ---------------------------- |
| Data and payloads | End user functionality | 1:1, group chat, public chat |
| Data sync | Data consistency | MVDS. |
| Secure transport | Confidentiality, PFS, etc | Double Ratchet |
| Transport privacy | Routing, Metadata protection | Waku / Whisper |
| P2P Overlay | Overlay routing, NAT traversal | devp2p |
### Protobuf
[`protobuf`](https://developers.google.com/protocol-buffers/) is used in different layers, version `proto3` used is unless stated otherwise.
## Components
### P2P Overlay
Status clients run on a public, permissionless peer-to-peer network, as specified by the devP2P
network protocols. devP2P provides a protocol for node discovery which is in
draft mode
[here](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md). See
more on node discovery and management in the next section.
To communicate between Status nodes, the [RLPx Transport
Protocol, v5](https://github.com/ethereum/devp2p/blob/master/rlpx.md) is used, which
allows for TCP-based communication between nodes.
On top of this RLPx-based subprotocols are ran, the client
SHOULD NOT use [Whisper V6](https://eips.ethereum.org/EIPS/eip-627), the client
SHOULD use [Waku V1](/waku/standards/legacy/6/waku1.md)
for privacy-preserving messaging and efficient usage of a node's bandwidth.
#### Node discovery and roles
There are four types of node roles:
1. `Bootstrap node`
1. `Whisper/Waku relayer`
1. `Mailserver` (servers and clients)
1. `Mobile node` (Status Clients)
A standard Status client MUST implement both `Whisper/Waku relayer` and `Mobile node` node types. The
other node types are optional, but it is RECOMMEND to implement a `Mailserver`
client mode, otherwise the user experience is likely to be poor.
#### Bootstrapping
Bootstrap nodes allow Status nodes to discover and connect to other Status nodes
in the network.
Currently, Status Gmbh provides the main bootstrap nodes, but anyone can
run these provided they are connected to the rest of the Whisper/Waku network.
Status maintains a list of production fleet bootstrap nodes in the following locations:
**Hong Kong:**
- `enode://6e6554fb3034b211398fcd0f0082cbb6bd13619e1a7e76ba66e1809aaa0c5f1ac53c9ae79cf2fd4a7bacb10d12010899b370c75fed19b991d9c0cdd02891abad@47.75.99.169:443`
- `enode://23d0740b11919358625d79d4cac7d50a34d79e9c69e16831c5c70573757a1f5d7d884510bc595d7ee4da3c1508adf87bbc9e9260d804ef03f8c1e37f2fb2fc69@47.52.106.107:443`
**Amsterdam:**
- `enode://436cc6f674928fdc9a9f7990f2944002b685d1c37f025c1be425185b5b1f0900feaf1ccc2a6130268f9901be4a7d252f37302c8335a2c1a62736e9232691cc3a@178.128.138.128:443`
- `enode://5395aab7833f1ecb671b59bf0521cf20224fe8162fc3d2675de4ee4d5636a75ec32d13268fc184df8d1ddfa803943906882da62a4df42d4fccf6d17808156a87@178.128.140.188:443`
**Central US:**
- `enode://32ff6d88760b0947a3dee54ceff4d8d7f0b4c023c6dad34568615fcae89e26cc2753f28f12485a4116c977be937a72665116596265aa0736b53d46b27446296a@34.70.75.208:443`
- `enode://5405c509df683c962e7c9470b251bb679dd6978f82d5b469f1f6c64d11d50fbd5dd9f7801c6ad51f3b20a5f6c7ffe248cc9ab223f8bcbaeaf14bb1c0ef295fd0@35.223.215.156:443`
These bootstrap nodes MAY change and are not guaranteed to stay this way forever
and at some point circumstances might force them to change.
#### Discovery
A Status client MUST discover or have a list of peers to connect to. Status uses a
light discovery mechanism based on a combination of [Discovery v5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md) and
[Rendezvous Protocol](https://github.com/libp2p/specs/tree/master/rendezvous),
(with some [modifications](https://github.com/status-im/rendezvous#differences-with-original-rendezvous)).
Additionally, some static nodes MAY also be used.
A Status client MUST use at least one discovery method or use static nodes
to communicate with other clients.
Discovery V5 uses bootstrap nodes to discover other peers. Bootstrap nodes MUST support
Discovery V5 protocol as well in order to provide peers. It is kademlia-based discovery mechanism
and it might consume significant (at least on mobile) amount of network traffic to operate.
In order to take advantage from simpler and more mobile-friendly peers discovery mechanism,
i.e. Rendezvous protocol, one MUST provide a list of Rendezvous nodes which speak
Rendezvous protocol. Rendezvous protocol is request-response discovery mechanism.
It uses Ethereum Node Records (ENR) to report discovered peers.
Both peers discovery mechanisms use topics to provide peers with certain capabilities.
There is no point in returning peers that do not support a particular protocol.
Status nodes that want to be discovered MUST register to Discovery V5 and/or Rendezvous
with the `whisper` topic. Status nodes that are `Mailservers` and want to
be discoverable MUST additionally register with the `whispermail` topic.
It is RECOMMENDED to use both mechanisms but at the same time implement a structure
called `PeerPool`. `PeerPool` is responsible for maintaining an optimal number of peers.
For mobile nodes, there is no significant advantage to have more than 2-3 peers and one `Mailserver`.
`PeerPool` can notify peers discovery protocol implementations that they should suspend
their execution because the optimal number of peers is found. They should resume
if the number of connected peers drops or a `Mailserver` disconnects.
It is worth noticing that an efficient caching strategy MAY be of great use, especially,
on mobile devices. Discovered peers can be cached as they rarely change and used
when the client starts again. In such a case, there might be no need to even start
peers discovery protocols because cached peers will satisfy the optimal number of peers.
Alternatively, a client MAY rely exclusively on a list of static peers. This is the most efficient
way because there are no peers discovery algorithm overhead introduced. The disadvantage
is that these peers might be gone and without peers discovery mechanism, it won't be possible to find
new ones.
The current list of static peers is published on <https://fleets.status.im/>. `eth.prod` is the current
group of peers the official Status client uses. The others are test networks.
Finally, Waku node addresses can be retrieved by traversing
the merkle tree found at [`fleets.status.im`](https://fleets.status.im), as described in [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459#client-protocol).
#### Mobile nodes
A `Mobile node` is a Whisper and/or Waku node which connects to part of the respective Whisper
and/or Waku network(s). A `Mobile node` MAY relay messages. See next section for more details on how
to use Whisper and/or Waku to communicate with other Status nodes.
### Transport privacy and Whisper / Waku usage
Once a Whisper and/or Waku node is up and running there are some specific settings required
to communicate with other Status nodes.
See [WHISPER-USAGE](/status/deprecated/whisper-usage.md) and [WAKU-USAGE](/status/deprecated/waku-usage.md) for more details.
For providing an offline inbox, see the complementary [WHISPER-MAILSERVER](/status/deprecated/whisper-mailserver.md) and [WAKU-MAILSERVER](/status/deprecated/waku-mailserver.md).
### Secure Transport
In order to provide confidentiality, integrity, authentication and forward
secrecy of messages the node implements a secure transport on top of Whisper and Waku. This is
used in 1:1 chats and group chats, but not for public chats. See [SECURE-TRANSPORT](/status/deprecated/secure-transport.md) for more.
### Data Sync
[MVDS](/vac/2/mvds.md) is used for 1:1 and group chats, however it is currently not in use for public chats.
[Status payloads](#payloads-and-clients) are serialized and then wrapped inside an
MVDS message which is added to an [MVDS payload](/vac/2/mvds.md#payloads),
the node encrypts this payload (if necessary for 1-to-1 / group-chats) and sends it using
Whisper or Waku which also encrypts it.
### Payloads and clients
On top of secure transport, various types of data sync clients and
the node uses payload formats for things like 1:1 chat, group chat and public chat. These have
various degrees of standardization. Please refer to [PAYLOADS](/status/deprecated/payloads.md) for more details.
### BIPs and EIPs Standards support
For a list of EIPs and BIPs that SHOULD be supported by Status client, please
see [EIPS](/status/deprecated/eips.md).
## Security Considerations
See [Appendix A](#appendix-a-security-considerations)
## Design Rationale
P2P Overlay
### Why devp2p? Why not use libp2p?
At the time Status developed the main Status clients, devp2p was the most
mature. However, in the future libp2p is likely to be used, as it'll
provide us with multiple transports, better protocol negotiation, NAT traversal,
etc.
For very experimental bridge support, see the bridge between libp2p and devp2p
in [Murmur](https://github.com/status-im/murmur).
### What about other RLPx subprotocols like LES, and Swarm?
Status is primarily optimized for resource restricted devices, and at present
time light client support for these protocols are suboptimal. This is a work in
progress.
For better Ethereum light client support, see [Re-enable LES as
option](https://github.com/status-im/status-go/issues/1025). For better Swarm
support, see [Swarm adaptive
nodes](https://github.com/ethersphere/SWIPs/pull/12).
For transaction support, Status clients currently have to rely on Infura.
Status clients currently do not offer native support for file storage.
### Why do you use Whisper?
Whisper is one of the [three parts](http://gavwood.com/dappsweb3.html) of the
vision of Ethereum as the world computer, Ethereum and Swarm being the other
two. Status was started as an encapsulation of and a clear window to this world
computer.
### Why do you use Waku?
Waku is a direct upgrade and replacement for Whisper, the main motivation for
developing and implementing Waku can be found in the [Waku specs](/waku/).
>Waku was created to incrementally improve in areas that Whisper is lacking in,
>with special attention to resource restricted devices. We specify the standard for
>Waku messages in order to ensure forward compatibility of different Waku clients,
>backwards compatibility with Whisper clients, as well as to allow multiple
>implementations of Waku and its capabilities. We also modify the language to be more
>unambiguous, concise and consistent.
Considerable work has gone into the active development of Ethereum, in contrast Whisper
is not currently under active development, and it has several drawbacks. Among others:
- Whisper is very wasteful bandwidth-wise and doesn't appear to be scalable
- Proof of work is a poor spam protection mechanism for heterogeneous devices
- The privacy guarantees provided are not rigorous
- There are no incentives to run a node
Finding a more suitable transport privacy is an ongoing research effort,
together with [Vac](https://vac.dev/vac-overview) and other teams in the space.
### Why is PoW for Waku set so low?
A higher PoW would be desirable, but this kills the battery on mobile phones,
which is a prime target for Status clients.
This means the network is currently vulnerable to DDoS attacks. Alternative
methods of spam protection are currently being researched.
### Why do you not use Discovery v5 for node discovery?
At the time of implementing dynamic node discovery, Discovery v5 wasn't completed
yet. Additionally, running a DHT on a mobile leads to slow node discovery, bad
battery and poor bandwidth usage. Instead, each client can choose to turn on
Discovery v5 for a short period until the node populates their peer list.
For some further investigation, see
[here](https://github.com/status-im/swarms/blob/master/ideas/092-disc-v5-research.md).
### I heard something about `Mailservers` being trusted somehow?
In order to use a `Mailserver`, a given node needs to connect to it directly, i.e. add the `Mailserver`
as its peer and mark it as trusted.
This means that the `Mailserver` is able to send direct p2p messages to the node instead of broadcasting them.
Effectively, it knows the bloom filter of the topics the node is interested in,
when it is online as well as many metadata like IP address.
### Data sync
#### Why is MVDS not used for public chats?
Currently, public chats are broadcast-based, and there's no direct way of finding
out who is receiving messages. Hence there's no clear group sync state context
whereby participants can sync. Additionally, MVDS is currently not optimized for
large group contexts, which means bandwidth usage will be a lot higher than
reasonable. See [P2P Data Sync for Mobile](https://vac.dev/p2p-data-sync-for-mobile) for more.
This is an active area of research.
## Footnotes
1. <https://github.com/status-im/status-protocol-go/>
2. <https://github.com/status-im/status-console-client/>
3. <https://github.com/status-im/status-mobile/>
## Appendix A: Security considerations
There are several security considerations to take into account when running Status.
Chief among them are: scalability, DDoS-resistance and privacy.
These also vary depending on what capabilities are used, such as `Mailserver`, light node, and so on.
### Scalability and UX
**Bandwidth usage:**
In version 1 of Status, bandwidth usage is likely to be an issue.
In Status version 1.1 this is partially addressed with Waku usage, see [the theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
**`Mailserver` High Availability requirement:**
A `Mailserver` has to be online to receive messages for other nodes, this puts a high availability requirement on it.
**Gossip-based routing:**
Use of gossip-based routing doesn't necessarily scale.
It means each node can see a message multiple times,
and having too many light nodes can cause propagation probability that is too low.
See [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/) for more and a possible Kademlia based alternative.
**Lack of incentives:**
Status currently lacks incentives to run nodes, which means node operators are more likely to create centralized choke points.
### Privacy
**Light node privacy:**
The main privacy concern with light nodes is that directly connected peers will know that a message originates from them (as it are the only ones it sends). This means nodes can make assumptions about what messages (topics) their peers are interested in.
**Bloom filter privacy:**
A user reveals which messages they are interested in, by setting only the topics they are interested in on the bloom filter.
This is a fundamental trade-off between bandwidth usage and privacy,
though the trade-off space is likely suboptimal in terms of the [Anonymity](https://eprint.iacr.org/2017/954.pdf) [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
**`Mailserver client` privacy:**
A `Mailserver client` has to trust a `Mailserver`, which means they can send direct traffic. This reveals what topics / bloom filter a node is interested in, along with its peerID (with IP).
**Privacy guarantees not rigorous:**
Privacy for Whisper or Waku hasn't been studied rigorously for various threat models like global passive adversary, local active attacker, etc. This is unlike e.g. Tor and mixnets.
**Topic hygiene:**
Similar to bloom filter privacy, using a very specific topic reveals more information. See scalability model linked above.
### Spam resistance
**PoW bad for heterogeneous devices:**
Proof of work is a poor spam prevention mechanism. A mobile device can only have a very low PoW in order not to use too much CPU / burn up its phone battery. This means someone can spin up a powerful node and overwhelm the network.
**`Mailserver` trusted connection:**
A `Mailserver` has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning `Mailserver` can overwhelm an individual node.
### Censorship resistance
**Devp2p TCP port blockable:**
By default Devp2p runs on port `30303`, which is not commonly used for any other service. This means it is easy to censor, e.g. airport WiFi. This can be mitigated somewhat by running on e.g. port `80` or `443`, but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
See <https://github.com/status-im/status-mobile/issues/6351> for some discussion.
## Acknowledgments
Jacek Sieka
## Changelog
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
- Added that Waku SHOULD be used
- Added that Whisper SHOULD NOT be used
- Added language to include Waku in all relevant places
- Change to keep `Mailserver` term consistent
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [Protobuf](https://developers.google.com/protocol-buffers/)
- [Discv5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)
- [RLPx Transport Protocol, v5](https://github.com/ethereum/devp2p/blob/master/rlpx.md)
- [Whisper V6](https://eips.ethereum.org/EIPS/eip-627)
- [Waku V1](/waku/standards/legacy/6/waku1.md)
- [Rendezvous Protocol](https://github.com/libp2p/specs/tree/master/rendezvous)
- [Rendezvous Protocol modifications](https://github.com/status-im/rendezvous#differences-with-original-rendezvous)
- [Fleets Status](https://fleets.status.im)
- [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459#client-protocol)
- [WHISPER-USAGE](/status/deprecated/whisper-usage.md)
- [WAKU-USAGE](/status/deprecated/waku-usage.md)
- [WHISPER-MAILSERVER](/status/deprecated/whisper-mailserver.md)
- [WAKU-MAILSERVER](/status/deprecated/waku-mailserver.md)
- [SECURE-TRANSPORT](/status/deprecated/secure-transport.md)
- [MVDS](/vac/2/mvds.md)
- [PAYLOADS](/status/deprecated/payloads.md)
- [EIPS](/status/deprecated/eips.md)
- [Murmur](https://github.com/status-im/murmur)
- [Re-enable LES as option](https://github.com/status-im/status-go/issues/1025)
- [Swarm adaptive nodes](https://github.com/ethersphere/SWIPs/pull/12)
- [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/)
- [Waku specs](/waku/)
- [Vac](https://vac.dev/vac-overview)
- [theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability)
- [Anonymity](https://eprint.iacr.org/2017/954.pdf)
- [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf)
- [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/)
- [Discovery v5 research](https://github.com/status-im/swarms/blob/master/ideas/092-disc-v5-research.md)
- [P2P Data Sync for Mobile](https://vac.dev/p2p-data-sync-for-mobile)
- [Status protocol go](https://github.com/status-im/status-protocol-go/)
- [Status console client](https://github.com/status-im/status-console-client/)
- [Status mobile](https://github.com/status-im/status-mobile/)
- [Status mobile issue 6351](https://github.com/status-im/status-mobile/issues/6351)

View File

@@ -0,0 +1,135 @@
---
title: Dapp browser API usage
name: Dapp browser API usage
status: deprecated
description: This document describes requirements that an application must fulfill in order to provide a proper environment for Dapps running inside a browser.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
---
## Abstract
This document describes requirements that an application must fulfill in order to provide a proper environment for Dapps running inside a browser.
A description of the Status Dapp API is provided, along with an overview of bidirectional communication underlying the API implementation.
The document also includes a list of EIPs that this API implements.
## Definitions
| Term | Description |
|------------|-------------------------------------------------------------------------------------|
| **Webview** | Platform-specific browser core implementation. |
| **Ethereum Provider** | A JS object (`window.ethereum`) injected into each web page opened in the browser providing web3 compatible provider. |
| **Bridge** | A set of facilities allow bidirectional communication between JS code and the application. |
## Overview
The application should expose an Ethereum Provider object (`window.ethereum`) to JS code running inside the browser.
It is important to have the `window.ethereum` object available before the page loads, otherwise Dapps might not work correctly.
Additionally, the browser component should also provide bidirectional communication between JS code and the application.
## Usage in Dapps
Dapps can use the below properties and methods of `window.ethereum` object.
### Properties
#### `isStatus`
Returns true. Can be used by the Dapp to find out whether it's running inside Status.
#### `status`
Returns a `StatusAPI` object. For now it supports one method: `getContactCode` that sends a `contact-code` request to Status.
### Methods
#### `isConnected`
Similarly to Ethereum JS API [docs](https://github.com/ethereum/wiki/wiki/JavaScript-API#web3isconnected),
it should be called to check if connection to a node exists. On Status, this fn always returns true, as once Status is up and running, node is automatically started.
#### `scanQRCode`
Sends a `qr-code` Status API request.
#### `request`
`request` method as defined by EIP-1193.
### Unused
Below are some legacy methods that some Dapps might still use.
#### `enable` (DEPRECATED)
Sends a `web3` Status API request. It returns a first entry in the list of available accounts.
Legacy `enable` method as defined by [EIP1102](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1102.md).
#### `send` (DEPRECATED)
Legacy `send` method as defined by [EIP1193](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1193.md).
#### `sendAsync` (DEPRECATED)
Legacy `sendAsync` method as defined by [EIP1193](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1193.md).
#### `sendSync` (DEPRECATED)
Legacy `send` method.
## Implementation
Status uses a [forked version](https://github.com/status-im/react-native-webview) of [react-native-webview](https://github.com/react-native-community/react-native-webview) to display web or dapps content.
The fork provides an Android implementation of JS injection before page load.
It is required in order to properly inject Ethereum Provider object.
Status injects two JS scripts:
- [provider.js](https://github.com/status-im/status-mobile/blob/develop/resources/js/provider.js): `window.ethereum` object
- [webview.js](https://github.com/status-im/status-mobile/blob/develop/resources/js/webview.js): override for `history.pushState` used internally
Dapps running inside a browser communicate with Status Ethereum node by means of a *bridge* provided by react-native-webview library.
The bridge allows for bidirectional communication between browser and Status. In order to do so, it injects a special `ReactNativeWebview` object into each page it loads.
On Status (React Native) end, `react-native-webview` library provides `WebView.injectJavascript` function
on a webview component that allows to execute arbitrary code inside the webview.
Thus it is possible to inject a function call passing Status node response back to the Dapp.
Below is the table briefly describing what functions/properties are used. More details available in package [docs](https://github.com/react-native-community/react-native-webview/blob/master/docs/Guide.md#communicating-between-js-and-native).
| Direction | Side | Method |
|-----------|------|-----------|
| Browser->Status | JS | `ReactNativeWebView.postMessage()`|
| Browser->Status | RN | `WebView.onMessage()`|
| Status->Browser | JS | `ReactNativeWebView.onMessage()`|
| Status->Browser | RN | `WebView.injectJavascript()`|
## Compatibility
Status browser supports the following EIPs:
- [EIP1102](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1102.md): `eth_requestAccounts` support
- [EIP1193](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1193.md): `connect`, `disconnect`, `chainChanged`, and `accountsChanged` event support is not implemented
## Changelog
| Version | Comment |
| :-----: | ------- |
| 0.1.0 | Initial Release |
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [Ethereum JS API docs](https://github.com/ethereum/wiki/wiki/JavaScript-API#web3isconnected)
- [EIP1102](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1102.md)
- [EIP1193](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1193.md)
- [forked version](https://github.com/status-im/react-native-webview)
- [react-native-webview](https://github.com/react-native-community/react-native-webview)
- [provider.js](https://github.com/status-im/status-mobile/blob/develop/resources/js/provider.js)
- [webview.js](https://github.com/status-im/status-mobile/blob/develop/resources/js/webview.js)
- [docs](https://github.com/react-native-community/react-native-webview/blob/master/docs/Guide.md#communicating-between-js-and-native)

286
status/deprecated/eips.md Normal file
View File

@@ -0,0 +1,286 @@
---
title: EIPS
name: EIPS
status: deprecated
description: Status relation with the EIPs
editor: Ricardo Guilherme Schmidt <ricardo3@status.im>
contributors:
-
---
## Abstract
This specification describes how Status relates with EIPs.
## Introduction
Status should follow all standards as possible.
Whenever the Status app needs a feature, it should be first checked if there is a standard for that,
if not, Status should propose a standard.
### Support table
| | Status v0 | Status v1 | Other | State |
|----------|-----------|-----------|----------| -------- |
| BIP32 | N | Y | N | `stable` |
| BIP39 | Y | Y | Y | `stable` |
| BIP43 | N | Y | N | `stable` |
| BIP44 | N | Y | N | `stable` |
| EIP20 | Y | Y | Y | `stable` |
| EIP55 | Y | Y | Y | `stable` |
| EIP67 | P | P | N | `stable` |
| EIP137 | P | P | N | `stable` |
| EIP155 | Y | Y | Y | `stable` |
| EIP165 | P | N | N | `stable` |
| EIP181 | P | N | N | `stable` |
| EIP191 | Y? | N | Y | `stable` |
| EIP627 | Y | Y | N | `stable` |
| EIP681 | Y | N | Y | `stable` |
| EIP712 | P | P | Y | `stable` |
| EIP721 | P | P | Y | `stable` |
| EIP831 | N | Y | N | `stable` |
| EIP945 | Y | Y | N | `stable` |
| EIP1102 | Y | Y | Y | `stable` |
| EIP1193 | Y | Y | Y | `stable` |
| EIP1577 | Y | P | N | `stable` |
| EIP1581 | N | Y | N | `stable` |
| EIP1459 | N | | N | `raw` |
## Components
### BIP32 - Hierarchical Deterministic Wallets
Support: Dependency.
[Reference](https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki)
Description: Enable wallets to derive multiple private keys from the same seed.
Used for: Dependency of BIP39 and BIP43.
### BIP39 - Mnemonic code for generating deterministic keys
Support: Dependency.
[Reference](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki)
Description: Enable wallet to create private key based on a safe seed phrase.
Used for: Security and user experience.
### BIP43 - Purpose Field for Deterministic Wallets
Support: Dependency.
[Reference](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki)
Description: Enable wallet to create private keys branched for a specific purpose.
Used for: Dependency of BIP44, uses "ethereum" coin.
### BIP44 - Multi-Account Hierarchy for Deterministic Wallets
Support: Dependency.
[Reference](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
Description: Enable wallet to derive multiple accounts in top of BIP39.
Used for: Privacy.
[Source code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/constants.cljs#L240)
Observation: BIP44 don't solve privacy issues regarding the transparency of transactions, therefore directly connected addresses through a transactions can be identifiable by a "network reconnaissance attack" over transaction history, this attack together with leakage of information from centralized services, such as exchanges, would be fatal against the whole privacy of users, regardless of BIP44.
### EIP20 - Fungible Token
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-20)
Description: Enable wallets to use tokens based on smart contracts compliant with this standard.
Used for: Wallet feature.
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/tokens.cljs)
### EIP55 - Mixed-case checksum address encoding
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-55)
Description: Checksum standard that uses lowercase and uppercase inside address hex value.
Used for: Sanity check of forms using ethereum address.
[Related](https://github.com/status-im/status-mobile/issues/4959) [Also](https://github.com/status-im/status-mobile/issues/8707)
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/eip55.cljs)
### EIP67 - Standard URI scheme with metadata, value and byte code
Support: Partial.
[Reference](https://github.com/ethereum/EIPs/issues/67)
Description: A standard way of creating Ethereum URIs for various use-cases.
Used for: Legacy support.
[Issue](https://github.com/status-im/status-mobile/issues/875)
### EIP137 - Ethereum Domain Name Service - Specification
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-137)
Description: Enable wallets to lookup ENS names.
Used for: User experience, as a wallet and identity feature, usernames.
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/ens.cljs#L86)
### EIP155 - Simple replay attack protection
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-155)
Description: Defined chainId parameter in the singed ethereum transaction payload.
Used for: Signing transactions, crucial to safety of users against replay attacks.
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/core.cljs)
### EIP165 - Standard Interface Detection
Support: Dependency/Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-165)
Description: Standard interface for contract to answer if it supports other interfaces.
Used for: Dependency of ENS and EIP721.
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/eip165.cljs)
### EIP181 - ENS support for reverse resolution of Ethereum addresses
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-181)
Description: Enable wallets to render reverse resolution of Ethereum addresses.
Used for: Wallet feature.
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/ens.cljs#L86)
### EIP191 - Signed Message
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-191)
Description: Contract signature standard, adds an obligatory padding to signed message to differentiate from Ethereum Transaction messages.
Used for: Dapp support, security, dependency of ERC712.
### EIP627 - Whisper Specification
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-627)
Description: format of Whisper messages within the ÐΞVp2p Wire Protocol.
Used for: Chat protocol.
### EIP681 - URL Format for Transaction Requests
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-681)
Description: A link that pop up a transaction in the wallet.
Used for: Useful as QR code data for transaction requests, chat transaction requests and for dapp links to transaction requests.
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/eip681.cljs)
Related: [Issue #9183: URL Format for Transaction Requests (EIP681) is poorly supported](https://github.com/status-im/status-mobile/issues/9183) [Issue #9240](https://github.com/status-im/status-mobile/pull/9240) [Issue #9238](https://github.com/status-im/status-mobile/issues/9238) [Issue #7214](https://github.com/status-im/status-mobile/issues/7214) [Issue #7325](https://github.com/status-im/status-mobile/issues/7325) [Issue #8150](https://github.com/status-im/status-mobile/issues/8150)
### EIP712 - Typed Signed Message
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-712)
Description: Standardize types for contract signature, allowing users to easily inspect whats being signed.
Used for: User experience, security.
Related: [Isse #5461](https://github.com/status-im/status-mobile/issues/5461) [Commit](https://github.com/status-im/status-mobile/commit/ba37f7b8d029d3358c7b284f6a2383b9ef9526c9)
### EIP721 - Non Fungible Token
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-721)
Description: Enable wallets to use tokens based on smart contracts compliant with this standard.
Used for: Wallet feature.
Related: [Issue #8909](https://github.com/status-im/status-mobile/issues/8909)
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/erc721.cljs) [Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/tokens.cljs)
### EIP945 - Web 3 QR Code Scanning API
Support: Full.
[Reference](https://github.com/ethereum/EIPs/issues/945)
Used for: Sharing contactcode, reading transaction requests.
Related: [Issue #5870](https://github.com/status-im/status-mobile/issues/5870)
### EIP1102 - Opt-in account exposure
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-1102)
Description: Allow users to opt-in the exposure of their ethereum address to dapps they browse.
Used for: Privacy, DApp support.
Related: [Issue #7985](https://github.com/status-im/status-mobile/issues/7985)
### EIP1193 - Ethereum Provider JavaScript API
Support: Full.
[Reference](https://eips.ethereum.org/EIPS/eip-1193)
Description: Allows dapps to recognize event changes on wallet.
Used for: DApp support.
Related: [Issue #7246](https://github.com/status-im/status-mobile/pull/7246)
### EIP1577 - contenthash field for ENS
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-1577)
Description: Allows users browse ENS domains using contenthash standard.
Used for: Browser, DApp support.
Related: [Isse #6688](https://github.com/status-im/status-mobile/issues/6688)
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/utils/contenthash.cljs) [Sourcecode](https://github.com/status-im/status-mobile/blob/develop/test/cljs/status_im/test/utils/contenthash.cljs#L5)
### EIP1581 - Non-wallet usage of keys derived from BIP-32 trees
Support: Partial.
[Reference](https://eips.ethereum.org/EIPS/eip-1581)
Description: Allow wallet to derive keys that are less sensible (non wallet).
Used for: Security (don't reuse wallet key) and user experience (don't request keycard every login).
Related: [Issue #9096](https://github.com/status-im/status-mobile/issues/9088) [Issue #9096](https://github.com/status-im/status-mobile/pull/9096)
[Sourcecode](https://github.com/status-im/status-mobile/blob/develop/src/status_im/constants.cljs#L242)
### EIP1459 - Node Discovery via DNS
Support: -
[Reference](https://eips.ethereum.org/EIPS/eip-1459)
Description: Allows the storing and retrieving of nodes through merkle trees stored in TXT records of a domain.
Used for: Finding Waku nodes.
Related: -
Sourcecode: -
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [BIP32 - Hierarchical Deterministic Wallets](https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki)
- [BIP39 - Mnemonic code for generating deterministic keys](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki)
- [BIP43 - Purpose Field for Deterministic Wallets](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki)
- [BIP44 - Multi-Account Hierarchy for Deterministic Wallets](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
- [BIP44 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/constants.cljs#L240)
- [EIP20 - Fungible Token](https://eips.ethereum.org/EIPS/eip-20)
- [EIP20 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/tokens.cljs)
- [EIP55 - Mixed-case checksum address encoding](https://eips.ethereum.org/EIPS/eip-55)
- [EIP55 Related Issue 4959](https://github.com/status-im/status-mobile/issues/4959)
- [EIP55 Related Issue 8707](https://github.com/status-im/status-mobile/issues/8707)
- [EIP55 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/eip55.cljs)
- [EIP67 - Standard URI scheme with metadata, value and byte code](https://github.com/ethereum/EIPs/issues/67)
- [EIP67 Related Issue 875](https://github.com/status-im/status-mobile/issues/875)
- [EIP137 - Ethereum Domain Name Service - Specification](https://eips.ethereum.org/EIPS/eip-137)
- [EIP137 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/ens.cljs#L86)
- [EIP155 - Simple replay attack protection](https://eips.ethereum.org/EIPS/eip-155)
- [EIP155 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/core.cljs)
- [EIP165 - Standard Interface Detection](https://eips.ethereum.org/EIPS/eip-165)
- [EIP165 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/eip165.cljs)
- [EIP181 - ENS support for reverse resolution of Ethereum addresses](https://eips.ethereum.org/EIPS/eip-181)
- [EIP181 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/ens.cljs#L86)
- [EIP191 - Signed Message](https://eips.ethereum.org/EIPS/eip-191)
- [EIP627 - Whisper Specification](https://eips.ethereum.org/EIPS/eip-627)
- [EIP681 - URL Format for Transaction Requests](https://eips.ethereum.org/EIPS/eip-681)
- [EIP681 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/eip681.cljs)
- [EIP681 Related Issue 9183](https://github.com/status-im/status-mobile/issues/9183)
- [EIP681 Related Issue 9240](https://github.com/status-im/status-mobile/pull/9240)
- [EIP681 Related Issue 9238](https://github.com/status-im/status-mobile/issues/9238)
- [EIP681 Related Issue 7214](https://github.com/status-im/status-mobile/issues/7214)
- [EIP681 Related Issue 7325](https://github.com/status-im/status-mobile/issues/7325)
- [EIP681 Related Issue 8150](https://github.com/status-im/status-mobile/issues/8150)
- [EIP712 - Typed Signed Message](https://eips.ethereum.org/EIPS/eip-712)
- [EIP712 Related Issue 5461](https://github.com/status-im/status-mobile/issues/5461)
- [EIP712 Related Commit](https://github.com/status-im/status-mobile/commit/ba37f7b8d029d3358c7b284f6a2383b9ef9526c9)
- [EIP721 - Non Fungible Token](https://eips.ethereum.org/EIPS/eip-721)
- [EIP721 Related Issue 8909](https://github.com/status-im/status-mobile/issues/8909)
- [EIP721 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/erc721.cljs)
- [EIP721 Source Code (Tokens)](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/tokens.cljs)
- [EIP945 - Web 3 QR Code Scanning API](https://github.com/ethereum/EIPs/issues/945)
- [EIP945 Related Issue 5870](https://github.com/status-im/status-mobile/issues/5870)
- [EIP1102 - Opt-in account exposure](https://eips.ethereum.org/EIPS/eip-1102)
- [EIP1102 Related Issue 7985](https://github.com/status-im/status-mobile/issues/7985)
- [EIP1193 - Ethereum Provider JavaScript API](https://eips.ethereum.org/EIPS/eip-1193)
- [EIP1193 Related Issue 7246](https://github.com/status-im/status-mobile/pull/7246)
- [EIP1577 - contenthash field for ENS](https://eips.ethereum.org/EIPS/eip-1577)
- [EIP1577 Related Issue 6688](https://github.com/status-im/status-mobile/issues/6688)
- [EIP1577 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/utils/contenthash.cljs)
- [EIP1577 Test Source Code](https://github.com/status-im/status-mobile/blob/develop/test/cljs/status_im/test/utils/contenthash.cljs#L5)
- [EIP1581 - Non-wallet usage of keys derived from BIP-32 trees](https://eips.ethereum.org/EIPS/eip-1581)
- [EIP1581 Related Issue 9088](https://github.com/status-im/status-mobile/issues/9088)
- [EIP1581 Related Issue 9096](https://github.com/status-im/status-mobile/pull/9096)
- [EIP1581 Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/constants.cljs#L242)
- [EIP1459 - Node Discovery via DNS](https://eips.ethereum.org/EIPS/eip-1459)

View File

@@ -0,0 +1,237 @@
---
title: ETHEREUM-USAGE
name: Status interactions with the Ethereum blockchain
status: deprecated
description: All interactions that the Status client has with the Ethereum blockchain.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Andrea Maria Piana <andreap@status.im>
---
## Abstract
This specification documents all the interactions that the Status client has
with the [Ethereum](https://ethereum.org/developers/) blockchain.
## Background
All the interactions are made through [JSON-RPC](https://github.com/ethereum/wiki/wiki/JSON-RPC).
Currently [Infura](https://infura.io/) is used.
The client assumes high-availability,
otherwise it will not be able to interact with the Ethereum blockchain.
Status nodes rely on these Infura nodes
to validate the integrity of the transaction and report a consistent history.
Key handling is described [here](/status/deprecated/account.md)
1. [Wallet](#wallet)
2. [ENS](#ens)
## Wallet
The wallet in Status has two main components:
1) Sending transactions
2) Fetching balance
In the section below are described the `RPC` calls made the nodes, with a brief
description of their functionality and how it is used by Status.
1.[Sending transactions](#sending-transactions)
- [EstimateGas](#estimategas)
- [PendingNonceAt](#pendingnonceat)
- [SuggestGasPrice](#suggestgasprice)
- [SendTransaction](#sendtransaction)
2.[Fetching balance](#fetching-balance)
- [BlockByHash](#blockbyhash)
- [BlockByNumber](#blockbynumber)
- [FilterLogs](#filterlogs)
- [HeaderByNumber](#headerbynumber)
- [NonceAt](#nonceat)
- [TransactionByHash](#transactionbyhash)
- [TransactionReceipt](#transactionreceipt)
### Sending transactions
#### EstimateGas
EstimateGas tries to estimate the gas needed to execute a specific transaction
based on the current pending state of the backend blockchain.
There is no guarantee that this is the true gas limit requirement
as other transactions may be added or removed by miners,
but it should provide a basis for setting a reasonable default.
```go
func (ec *Client) EstimateGas(ctx context.Context, msg ethereum.CallMsg) (uint64, error)
```
[L499](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L499)
#### PendingNonceAt
`PendingNonceAt` returns the account nonce of the given account in the pending state.
This is the nonce that should be used for the next transaction.
```go
func (ec *Client) PendingNonceAt(ctx context.Context, account common.Address) (uint64, error)
```
[L440](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L440)
#### SuggestGasPrice
`SuggestGasPrice` retrieves the currently suggested gas price to allow a timely
execution of a transaction.
```go
func (ec *Client) SuggestGasPrice(ctx context.Context) (*big.Int, error)
```
[L487](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L487)
#### SendTransaction
`SendTransaction` injects a signed transaction into the pending pool for execution.
If the transaction was a contract creation use the TransactionReceipt method to get the
contract address after the transaction has been mined.
```go
func (ec *Client) SendTransaction(ctx context.Context, tx *types.Transaction) error
```
[L512](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L512)
### Fetching balance
A Status node fetches the current and historical [ECR20](https://eips.ethereum.org/EIPS/eip-20) and ETH balance for the user wallet address.
Collectibles following the [ERC-721](https://eips.ethereum.org/EIPS/eip-721) are also fetched if enabled.
A Status node supports by default the following [tokens](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/tokens.cljs). Custom tokens can be added by specifying the `address`, `symbol` and `decimals`.
#### BlockByHash
`BlockByHash` returns the given full block.
It is used by status to fetch a given block which will then be inspected
for transfers to the user address, both tokens and ETH.
```go
func (ec *Client) BlockByHash(ctx context.Context, hash common.Hash) (*types.Block, error)
```
[L78](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L78)
#### BlockByNumber
`BlockByNumber` returns a block from the current canonical chain. If number is nil, the
latest known block is returned.
```go
func (ec *Client) BlockByNumber(ctx context.Context, number *big.Int) (*types.Block, error)
```
[L82](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L82)
#### FilterLogs
`FilterLogs` executes a filter query.
Status uses this function to filter out logs, using the hash of the block
and the address of interest, both inbound and outbound.
```go
func (ec *Client) FilterLogs(ctx context.Context, q ethereum.FilterQuery) ([]types.Log, error)
```
[L377](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L377)
#### NonceAt
`NonceAt` returns the account nonce of the given account.
```go
func (ec *Client) NonceAt(ctx context.Context, account common.Address, blockNumber *big.Int) (uint64, error)
```
[L366](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L366)
#### TransactionByHash
`TransactionByHash` returns the transaction with the given hash,
used to inspect those transactions made/received by the user.
```go
func (ec *Client) TransactionByHash(ctx context.Context, hash common.Hash) (tx *types.Transaction, isPending bool, err error)
```
[L202](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L202)
#### HeaderByNumber
`HeaderByNumber` returns a block header from the current canonical chain.
```go
func (ec *Client) HeaderByNumber(ctx context.Context, number *big.Int) (*types.Header, error)
```
[L172](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L172)
#### TransactionReceipt
`TransactionReceipt` returns the receipt of a transaction by transaction hash.
It is used in status to check if a token transfer was made to the user address.
```go
func (ec *Client) TransactionReceipt(ctx context.Context, txHash common.Hash) (*types.Receipt, error)
```
[L270](https://github.com/ethereum/go-ethereum/blob/26d271dfbba1367326dec38068f9df828d462c61/ethclient/ethclient.go#L270)
## ENS
All the interactions with `ENS` are made through the [ENS contract](https://github.com/ensdomains/ens)
For the `stateofus.eth` username, one can be registered through these [contracts](https://github.com/status-im/ens-usernames)
### Registering, releasing and updating
- [Registering a username](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L113)
- [Releasing a username](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L131)
- [Updating a username](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L174)
### Slashing
Usernames MUST be in a specific format, otherwise they MAY be slashed:
- They MUST only contain alphanumeric characters
- They MUST NOT be in the form `0x[0-9a-f]{5}.*` and have more than 12 characters
- They MUST NOT be in the [reserved list](https://github.com/status-im/ens-usernames/blob/47c4c6c2058be0d80b7d678e611e166659414a3b/config/ens-usernames/reservedNames.js)
- They MUST NOT be too short, this is dynamically set in the contract and can be checked against the [contract](https://github.com/status-im/ens-usernames/blob/master/contracts/registry/UsernameRegistrar.sol#L26)
- [Slash a reserved username](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L237)
- [Slash an invalid username](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L261)
- [Slash a username too similar to an address](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L215)
- [Slash a username that is too short](https://github.com/status-im/ens-usernames/blob/77d9394d21a5b6213902473b7a16d62a41d9cd09/contracts/registry/UsernameRegistrar.sol#L200)
ENS names are propagated through `ChatMessage` and `ContactUpdate` [payload](/status/deprecated/payloads.md).
A client SHOULD verify ens names against the public key of the sender on receiving the message against the [ENS contract](https://github.com/ensdomains/ens)
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [Ethereum Developers](https://ethereum.org/developers/)
- [JSON-RPC](https://github.com/ethereum/wiki/wiki/JSON-RPC)
- [Infura](https://infura.io/)
- [Key Handling](/status/deprecated/account.md)
- [ERC-20 Token Standard](https://eips.ethereum.org/EIPS/eip-20)
- [ERC-721 Non-Fungible Token Standard](https://eips.ethereum.org/EIPS/eip-721)
- [Supported Tokens Source Code](https://github.com/status-im/status-mobile/blob/develop/src/status_im/ethereum/tokens.cljs)
- [go-ethereum](https://github.com/ethereum/go-ethereum/)
- [ENS Contract](https://github.com/ensdomains/ens)

View File

@@ -0,0 +1,162 @@
---
title: GROUP-CHAT
name: Group Chat
status: deprecated
description: This document describes the group chat protocol used by the Status application.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Andrea Piana <andreap@status.im>
---
## Abstract
This document describes the group chat protocol used by the Status application.
The node uses pairwise encryption among members so a message is exchanged
between each participant, similarly to a one-to-one message.
## Membership updates
The node uses membership updates messages to propagate group chat membership changes.
The protobuf format is described in the [PAYLOADS](/status/deprecated/payloads.md).
Below describes each specific field.
The protobuf messages are:
```protobuf
// MembershipUpdateMessage is a message used to propagate information
// about group membership changes.
message MembershipUpdateMessage {
// The chat id of the private group chat
string chat_id = 1;
// A list of events for this group chat, first 65 bytes are the signature, then is a
// protobuf encoded MembershipUpdateEvent
repeated bytes events = 2;
// An optional chat message
ChatMessage message = 3;
}
message MembershipUpdateEvent {
// Lamport timestamp of the event as described in [Status Payload Specs](status-payload-specs.md#clock-vs-timestamp-and-message-ordering)
uint64 clock = 1;
// List of public keys of the targets of the action
repeated string members = 2;
// Name of the chat for the CHAT_CREATED/NAME_CHANGED event types
string name = 3;
// The type of the event
EventType type = 4;
enum EventType {
UNKNOWN = 0;
CHAT_CREATED = 1; // See [CHAT_CREATED](#chat-created)
NAME_CHANGED = 2; // See [NAME_CHANGED](#name-changed)
MEMBERS_ADDED = 3; // See [MEMBERS_ADDED](#members-added)
MEMBER_JOINED = 4; // See [MEMBER_JOINED](#member-joined)
MEMBER_REMOVED = 5; // See [MEMBER_REMOVED](#member-removed)
ADMINS_ADDED = 6; // See [ADMINS_ADDED](#admins-added)
ADMIN_REMOVED = 7; // See [ADMIN_REMOVED](#admin-removed)
}
}
```
### Payload
`MembershipUpdateMessage`:
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | chat-id | `string` | The chat id of the chat where the change is to take place |
| 2 | events | See details | A list of events that describe the membership changes, in their encoded protobuf form |
| 3 | message | `ChatMessage` | An optional message, described in [Message](/status/deprecated/payloads.md/#message) |
`MembershipUpdateEvent`:
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | The clock value of the event |
| 2 | members | `[]string` | An optional list of hex encoded (prefixed with `0x`) public keys, the targets of the action |
| 3 | name | `name` | An optional name, for those events that make use of it |
| 4 | type | `EventType` | The type of event sent, described below |
### Chat ID
Each membership update MUST be sent with a corresponding `chatId`.
The format of this chat ID MUST be a string of [UUID](https://tools.ietf.org/html/rfc4122),
concatenated with the hex-encoded public key of the creator of the chat, joined by `-`.
This chatId MUST be validated by all clients, and MUST be discarded if it does not follow these rules.
### Signature
The node calculates the signature for each event by encoding each `MembershipUpdateEvent` in its protobuf representation
and prepending the bytes of the chatID, lastly the node signs the `Keccak256` of the bytes
using the private key by the author and added to the `events` field of MembershipUpdateMessage.
### Group membership event
Any `group membership` event received MUST be verified by calculating the signature as per the method described above.
The author MUST be extracted from it, if the verification fails the event MUST be discarded.
#### CHAT_CREATED
Chat `created event` is the first event that needs to be sent.
Any event with a clock value lower than this MUST be discarded.
Upon receiving this event a client MUST validate the `chatId`
provided with the updates and create a chat with identified by `chatId` and named `name`.
#### NAME_CHANGED
`admins` use a `name changed` event to change the name of the group chat.
Upon receiving this event a client MUST validate the `chatId` provided with the updates
and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
If the event is valid the chat name SHOULD be changed to `name`.
#### MEMBERS_ADDED
`admins` use a `members added` event to add members to the chat.
Upon receiving this event a client MUST validate the `chatId`
provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored.
If the event is valid a client MUST update the list of members of the chat who have not joined, adding the `members` received.
`members` is an array of hex encoded public keys.
#### MEMBER_JOINED
`members` use a `members joined` event to signal that they want to start receiving messages from this chat.
Upon receiving this event a client MUST validate the `chatId` provided with the updates.
If the event is valid a client MUST update the list of members of the chat who joined, adding the signer.
Any `message` sent to the group chat should now include the newly joined member.
#### ADMINS_ADDED
`admins` use an `admins added` event to add make other admins in the chat.
Upon receiving this event a client MUST validate the `chatId` provided with the updates,
MUST ensure the author of the event is an admin of the chat
and MUST ensure all `members` are already `members` of the chat, otherwise the event MUST be ignored.
If the event is valid a client MUST update the list of admins of the chat, adding the `members` received.
`members` is an array of hex encoded public keys.
#### MEMBER_REMOVED
`members` and/or `admins` use a `member-removed` event to leave or kick members of the chat.
Upon receiving this event a client MUST validate the `chatId` provided with the updates, MUST ensure that:
- If the author of the event is an admin, target can only be themselves or a non-admin member.
- If the author of the event is not an admin, the target of the event can only be themselves.
If the event is valid a client MUST remove the member from the list of `members`/`admins` of the chat,
and no further message should be sent to them.
#### ADMIN_REMOVED
`Admins` use an `admin-removed` event to drop admin privileges.
Upon receiving this event a client MUST validate the `chatId` provided with the updates,
MUST ensure that the author of the event is also the target of the event.
If the event is valid a client MUST remove the member from the list of `admins` of the chat.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [PAYLOADS](/status/deprecated/payloads.md)
- [UUID](https://tools.ietf.org/html/rfc4122)

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

View File

@@ -0,0 +1,303 @@
---
title: Keycard Usage for Wallet and Chat Keys
name: Keycard Usage for Wallet and Chat Keys
status: deprecated
description: In this specification, we describe how Status communicates with Keycard to create, store and use multiaccount.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Roman Volosovskyi <roman@status.im>
---
## Abstract
In this specification, we describe how Status communicates with Keycard to create, store and use multiaccount.
## Definitions
| Term | Description |
| ------------------ | -------------------------------------------------------- |
| Keycard Hardwallet | [https://keycard.tech/docs/](https://keycard.tech/docs/) |
| | |
## Multiaccount creation/restoring
### Creation and restoring via mnemonic
1. `status-im.hardwallet.card/get-application-info`
request: `nil`
response: `{"initialized?" false}`
2. `status-im.hardwallet.card/init-card`
request: `{:pin 123123}`
response:
```clojure
{"password" "nEJXqf6VWbqeC5oN",
"puk" "411810112887",
"pin" "123123"}
```
3. `status-im.hardwallet.card/get-application-info`
request: `nil`
response:
```clojure
{"free-pairing-slots" 5,
"app-version" "2.2",
"secure-channel-pub-key" "04e70d7af7d91b8cd23adbefdfc242c096adee6c1b5ad27a4013a8f926864c1a4f816b338238dc4a04226ab42f23672585c6dca03627885530643f1656ee69b025",
"key-uid" "",
"instance-uid" "9f149d438988a7af5e1a186f650c9328",
"paired?" false,
"has-master-key?" false,
"initialized?" true}
```
4. `status-im.hardwallet.card/pair`
params: `{:password "nEJXqf6VWbqeC5oN"}`
response: `AAVefVX0kPGsxnvQV5OXRbRTLGI3k8/S27rpsq/lZrVR` (`pairing`)
5. `status-im.hardwallet.card/generate-and-load-keys`
```clojure
{:mnemonic "lift mansion moment version card type uncle sunny lock gather nerve math",
:pairing "AAVefVX0kPGsxnvQV5OXRbRTLGI3k8/S27rpsq/lZrVR",
:pin "123123"}
```
response:
```clojure
{"whisper-address" "1f29a1a60c8a12f80c397a91c6ae0323f420e609",
"whisper-private-key" "123123123123123",
"wallet-root-public-key" "04eb9d01990a106a65a6dfaa48300f72aecfeabe502d9f4f7aeaccb146dc2f16e2dec81dcec0a1a52c1df4450f441a48c210e1a73777c0161030378df22e4ae015",
"encryption-public-key" "045ee42f012d72be74b31a28ce320df617e0cd5b9b343fad34fcd61e2f5dfa89ab23d880473ba4e95401a191764c7f872b7af92ea0d8c39462147df6f3f05c2a11",
"wallet-root-address" "132dd67ff47cc1c376879c474fd2afd0f1eee6de",
"whisper-public-key" "0450ad84bb95f32c64f4e5027cc11d1b363a0566a0cfc475c5653e8af9964c5c9b0661129b75e6e1bc6e96ba2443238e53e7f49f2c5f2d16fcf04aca4826765d46",
"address" "bf93eb43fea2ce94bf3a6463c16680b56aa4a08a",
"wallet-address" "7eee1060d8e4722d36c99f30ff8291caa3cfc40c",
"key-uid" "472d8436ccedb64bcbd897bed5895ec3458b306352e1bcee377df87db32ef2c2",
"wallet-public-key" "0495ab02978ea1f8b059140e0be5a87aad9b64bb7d9706735c47dda6e182fd5ca41744ca37583b9a10c316b01d4321d6c85760c61301874089acab041037246294",
"public-key" "0465d452d12171711f32bb931f9ea26fe1b88fe2511a7909a042b914fde10a99719136365d506e2d1694fc14627f9d557da33865efc6001da3942fc1d4d2469ca1",
"instance-uid" "9f149d438988a7af5e1a186f650c9328"}
```
### Multiaccount restoring via pairing
This flow is required in case if a user want to pair a card with an existing multiaccount on it.
1. `status-im.hardwallet.card/get-application-info`
request: `nil`
response:
```clojure
{"free-pairing-slots" 4,
"app-version" "2.2",
"secure-channel-pub-key" "04e70d7af7d91b8cd23adbefdfc242c096adee6c1b5ad27a4013a8f926864c1a4f816b338238dc4a04226ab42f23672585c6dca03627885530643f1656ee69b025",
"key-uid" "",
"instance-uid" "9f149d438988a7af5e1a186f650c9328",
"paired?" false,
"has-master-key?" false,
"initialized?" true}
```
2. `status-im.hardwallet.card/pair`
params: `{:password "nEJXqf6VWbqeC5oN"}`
response: `AAVefVX0kPGsxnvQV5OXRbRTLGI3k8/S27rpsq/lZrVR` (`pairing`)
3. `status-im.hardwallet.card/generate-and-load-keys`
```clojure
{:mnemonic "lift mansion moment version card type uncle sunny lock gather nerve math",
:pairing "AAVefVX0kPGsxnvQV5OXRbRTLGI3k8/S27rpsq/lZrVR",
:pin "123123"}
```
response:
```clojure
{"whisper-address" "1f29a1a60c8a12f80c397a91c6ae0323f420e609",
"whisper-private-key" "123123123123123123123",
"wallet-root-public-key" "04eb9d01990a106a65a6dfaa48300f72aecfeabe502d9f4f7aeaccb146dc2f16e2dec81dcec0a1a52c1df4450f441a48c210e1a73777c0161030378df22e4ae015",
"encryption-public-key" "045ee42f012d72be74b31a28ce320df617e0cd5b9b343fad34fcd61e2f5dfa89ab23d880473ba4e95401a191764c7f872b7af92ea0d8c39462147df6f3f05c2a11",
"wallet-root-address" "132dd67ff47cc1c376879c474fd2afd0f1eee6de",
"whisper-public-key" "0450ad84bb95f32c64f4e5027cc11d1b363a0566a0cfc475c5653e8af9964c5c9b0661129b75e6e1bc6e96ba2443238e53e7f49f2c5f2d16fcf04aca4826765d46",
"address" "bf93eb43fea2ce94bf3a6463c16680b56aa4a08a",
"wallet-address" "7eee1060d8e4722d36c99f30ff8291caa3cfc40c",
"key-uid" "472d8436ccedb64bcbd897bed5895ec3458b306352e1bcee377df87db32ef2c2",
"wallet-public-key" "0495ab02978ea1f8b059140e0be5a87aad9b64bb7d9706735c47dda6e182fd5ca41744ca37583b9a10c316b01d4321d6c85760c61301874089acab041037246294",
"public-key" "0465d452d12171711f32bb931f9ea26fe1b88fe2511a7909a042b914fde10a99719136365d506e2d1694fc14627f9d557da33865efc6001da3942fc1d4d2469ca1",
"instance-uid" "9f149d438988a7af5e1a186f650c9328"}
```
## Multiaccount unlocking
1. `status-im.hardwallet.card/get-application-info`
params:
```clojure
{:pairing nil, :on-success nil}
```
response:
```clojure
{"free-pairing-slots" 4,
"app-version" "2.2",
"secure-channel-pub-key" "04b079ac513d5e0ebbe9becbae1618503419f5cb59edddc7d7bb09ce0db069a8e6dec1fb40c6b8e5454f7e1fcd0bb4a0b9750256afb4e4390e169109f3ea3ba91d",
"key-uid" "a5424fb033f5cc66dce9cbbe464426b6feff70ca40aa952c56247aaeaf4764a9",
"instance-uid" "2268254e3ed7898839abe0b40e1b4200",
"paired?" false,
"has-master-key?" true,
"initialized?" true}
```
2. `status-im.hardwallet.card/get-keys`
params:
```clojure
{:pairing "ACEWbvUlordYWOE6M1Narn/AXICRltjyuKIAn4kkPXQG",
:pin "123123"}
```
response:
```clojure
{"whisper-address" "ec83f7354ca112203d2ce3e0b77b47e6e33258aa",
"whisper-private-key" "123123123123123123123123",
"wallet-root-public-key" "0424a93fe62a271ad230eb2957bf221b4644670589f5c0d69bd11f3371034674bf7875495816095006c2c0d5f834d628b87691a8bbe3bcc2225269020febd65a19",
"encryption-public-key" "0437eef85e669f800570f444e64baa2d0580e61cf60c0e9236b4108455ec1943f385043f759fcb5bd8348e32d6d6550a844cf24e57f68e9397a0f7c824a8caee2d",
"wallet-root-address" "6ff915f9f31f365511b1b8c1e40ce7f266caa5ce",
"whisper-public-key" "04b195df4336c596cca1b89555dc55dd6bb4c5c4491f352f6fdfae140a2349213423042023410f73a862aa188f6faa05c80b0344a1e39c253756cb30d8753f9f8324",
"address" "73509a1bb5f3b74d0dba143705cd9b4b55b8bba1",
"wallet-address" "2f0cc0e0859e7a05f319d902624649c7e0f48955",
"key-uid" "a5424fb033f5cc66dce9cbbe464426b6feff70ca40aa952c56247aaeaf4764a9",
"wallet-public-key" "04d6fab73772933215872c239787b2281f3b10907d099d04b88c861e713bd2b95883e0b1710a266830da29e76bbf6b87ed034ab139e36cc235a1b2a5b5ddfd4e91",
"public-key" "0437eef85e669f800570f444e64baa2d0580e61cf60c0e9236b4108455ec1943f385043f759fcb5bd8348e32d6d6550a844cf24e57f68e9397a0f7c824a8caee2d",
"instance-uid" "2268254e3ed7898839abe0b40e1b4200"}
```
3. `status-im.hardwallet.card/get-application-info`
params:
```clojure
{:pairing "ACEWbvUlordYWOE6M1Narn/AXICRltjyuKIAn4kkPXQG"}
```
response:
```clojure
{"paired?" true,
"has-master-key?" true,
"app-version" "2.2",
"free-pairing-slots" 4,
"pin-retry-counter" 3,
"puk-retry-counter" 5,
"initialized?" true,
"secure-channel-pub-key" "04b079ac513d5e0ebbe9becbae1618503419f5cb59edddc7d7bb09ce0db069a8e6dec1fb40c6b8e5454f7e1fcd0bb4a0b9750256afb4e4390e169109f3ea3ba91d",
"key-uid" "a5424fb033f5cc66dce9cbbe464426b6feff70ca40aa952c56247aaeaf4764a9",
"instance-uid" "2268254e3ed7898839abe0b40e1b4200"}
```
## Transaction signing
1. `status-im.hardwallet.card/get-application-info`
params:
```clojure
{:pairing "ALecvegKyOW4szknl01yYWx60GLDK5gDhxMgJECRZ+7h",
:on-success :hardwallet/sign}
```
response:
```clojure
{"paired?" true,
"has-master-key?" true,
"app-version" "2.2",
"free-pairing-slots" 4,
"pin-retry-counter" 3,
"puk-retry-counter" 5,
"initialized?" true,
"secure-channel-pub-key" "0476d11f2ccdad4e7779b95a1ce063d7280cb6c09afe2c0e48ca0c64ab9cf2b3c901d12029d6c266bfbe227c73a802561302b2330ac07a3270fc638ad258fced4a",
"key-uid" "d5c8cde8085e7a3fcf95aafbcbd7b3cfe32f61b85c2a8f662f60e76bdc100718",
"instance-uid" "e20e27bfee115b431e6e81b8e9dcf04c"}
```
2. `status-im.hardwallet.card/sign`
params:
```clojure
{:hash "92fc7ef54c3e0c42de256b93fbf2c49dc6948ee089406e204dec943b7a0142a9",
:pairing "ALecvegKyOW4szknl01yYWx60GLDK5gDhxMgJECRZ+7h",
:pin "123123",
:path "m/44'/60'/0'/0/0"}
```
response: `5d2ca075593cf50aa34007a0a1df7df14a369534450fce4a2ae8d023a9d9c0e216b5e5e3f64f81bee91613318d01601573fdb15c11887a3b8371e3291e894de600`
## Account derivation
1. `status-im.hardwallet.card/verify-pin`
params:
```clojure
{:pin "123123",
:pairing "ALecvegKyOW4szknl01yYWx60GLDK5gDhxMgJECRZ+7h"}
```
response: `3`
1. `status-im.hardwallet.card/export-key`
params:
```clojure
{:pin "123123",
:pairing "ALecvegKyOW4szknl01yYWx60GLDK5gDhxMgJECRZ+7h",
:path "m/44'/60'/0'/0/1"}
```
response: `046d1bcd2310a5e0094bc515b0ec995a8cb59e23d564094443af10011b6c00bdde44d160cdd32b4b6341ddd7dc83a4f31fdf60ec2276455649ccd7a22fa4ea01d8` (account's `public-key`)
## Reset pin
1. `status-im.hardwallet.card/change-pin`
params:
```clojure
{:new-pin "111111",
:current-pin "222222",
:pairing "AA0sKxPkN+jMHXZZeI8Rgz04AaY5Fg0CzVbm9189Khob"}
```
response:
`true`
## Unblock pin
If user enters a wrong pin three times in a row a card gets blocked. The user can use puk code then to unblock the card and set a new pin.
1. `status-im.hardwallet.card/unblock-pin`
params:
```clojure
{:puk "120702722103",
:new-pin "111111",
:pairing "AIoQl0OtCL0/uSN7Ct1/FHRMEk/eM2Lrhn0bw7f8sgOe"}
```
response
`true`
## Status go calls
In order to use the card in the app we need to use some parts of status-go API, specifically:
1. [`SaveAccountAndLoginWithKeycard`](https://github.com/status-im/status-go/blob/b33ad8147d23a932064f241e575511d70a601dcc/mobile/status.go#L337) after multiaccount creation/restoring to store multiaccount and login into it
2. [`LoginWithKeycard`](https://github.com/status-im/status-go/blob/b33ad8147d23a932064f241e575511d70a601dcc/mobile/status.go#L373) to log into already existing account
3. [`HashTransaction`](https://github.com/status-im/status-go/blob/b33ad8147d23a932064f241e575511d70a601dcc/mobile/status.go#L492) and [`HashMessage`](https://github.com/status-im/status-go/blob/b33ad8147d23a932064f241e575511d70a601dcc/mobile/status.go#L520) for hashing transaction/message before signing
4. [`SendTransactionWithSignature`](https://github.com/status-im/status-go/blob/b33ad8147d23a932064f241e575511d70a601dcc/mobile/status.go#L471) to send transaction
## Where are the keys stored?
1. When we create a regular multiaccount all its keys are stored on device and are encrypted via key which is derived from user's password. In case if account was created using keycard all keys are stored on the card and are retrieved from it during signing into multiaccount.
2. When we create a regular multiaccount we also create a separate database for it and this database is encrypted using key which is derived from user's password. For a keycard account we use `encryption-public-key` (returned by `status-im.hardwallet.card/get-keys`/`status-im.hardwallet.card/generate-and-load-keys`) as a password.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [Keycard Hardwallet Documentation](https://keycard.tech/docs/)
- [Keycard Codebase](https://github.com/status-im/status-go/blob/b33ad8147d23a932064f241e575511d70a601dcc/mobile/status.go)

View File

@@ -0,0 +1,52 @@
---
title: NOTIFICATIONS
name: Notifications
status: deprecated
description: A client should implement local notifications to offer notifications for any event in the app without the privacy cost and dependency on third party services.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Eric Dvorsak <eric@status.im>
---
## Local Notifications
A client should implement local notifications to offer notifications
for any event in the app without the privacy cost and dependency on third party services.
This means that the client should run a background service to continuously or periodically check for updates.
### Android
Android allows running services on the device. When the user enables notifications,
the client may start a ``Foreground Service`,
and display a permanent notification indicating that the service is running,
as required by Android guidelines.
The service will simply keep the app from being killed by the system when it is in the background.
The client will then be able to run in the background
and display local notifications on events such as receiving a message in a one to one chat.
To facilitate the implementation of local notifications,
a node implementation such as `status-go` may provide a specific `notification` signal.
Notifications are a separate process in Android, and interaction with a notification generates an `Intent`.
To handle intents, the `NewMessageSignalHandler` may use a `BroadcastReceiver`,
in order to update the state of local notifications when the user dismisses or tap a notification.
If the user taps on a notification, the `BroadcastReceiver` generates a new intent to open the app should use universal links to get the user to the right place.
### iOS
We are not able to offer local notifications on iOS because there is no concept of services in iOS.
It offers background updates but theyre not consistently triggered, and cannot be relied upon.
The system decides when the background updates are triggered and the heuristics aren't known.
## Why is there no Push Notifications?
Push Notifications, as offered by Apple and Google are a privacy concern,
they require a centralized service that is aware of who the notification needs to be delivered to.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- None

View File

@@ -0,0 +1,386 @@
---
title: PAYLOADS
name: Payloads
status: deprecated
description: Payload of messages in Status, regarding chat and chat-related use cases.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Babik <adam@status.im>
- Andrea Maria Piana <andreap@status.im>
- Oskar Thorén <oskar@status.im>
---
## Abstract
This specification describes how the payload of each message in Status looks like.
It is primarily centered around chat and chat-related use cases.
The payloads aims to be flexible enough to support messaging but also cases
described in the [Status Whitepaper](https://status.im/whitepaper.pdf)
as well as various clients created using different technologies.
## Introduction
This document describes the payload format and some special considerations.
## Payload wrapper
The node wraps all payloads in a [protobuf record](https://developers.google.com/protocol-buffers/):
```protobuf
message ApplicationMetadataMessage {
bytes signature = 1;
bytes payload = 2;
Type type = 3;
enum Type {
UNKNOWN = 0;
CHAT_MESSAGE = 1;
CONTACT_UPDATE = 2;
MEMBERSHIP_UPDATE_MESSAGE = 3;
PAIR_INSTALLATION = 4;
SYNC_INSTALLATION = 5;
REQUEST_ADDRESS_FOR_TRANSACTION = 6;
ACCEPT_REQUEST_ADDRESS_FOR_TRANSACTION = 7;
DECLINE_REQUEST_ADDRESS_FOR_TRANSACTION = 8;
REQUEST_TRANSACTION = 9;
SEND_TRANSACTION = 10;
DECLINE_REQUEST_TRANSACTION = 11;
SYNC_INSTALLATION_CONTACT = 12;
SYNC_INSTALLATION_ACCOUNT = 13;
SYNC_INSTALLATION_PUBLIC_CHAT = 14;
CONTACT_CODE_ADVERTISEMENT = 15;
PUSH_NOTIFICATION_REGISTRATION = 16;
PUSH_NOTIFICATION_REGISTRATION_RESPONSE = 17;
PUSH_NOTIFICATION_QUERY = 18;
PUSH_NOTIFICATION_QUERY_RESPONSE = 19;
PUSH_NOTIFICATION_REQUEST = 20;
PUSH_NOTIFICATION_RESPONSE = 21;
}
}
```
`signature` is the bytes of the signed `SHA3-256` of the payload,
signed with the key of the author of the message.
The node needs the signature to validate authorship of the message,
so that the message can be relayed to third parties.
If a signature is not present, but an author is provided by a layer below,
the message is not to be relayed to third parties, and it is considered plausibly deniable.
`payload` is the protobuf encoded content of the message, with the corresponding `type` set.
## Encoding
The node encodes the payload using [Protobuf](https://developers.google.com/protocol-buffers)
## Types of messages
### Message
The type `ChatMessage` represents a chat message exchanged between clients.
#### Payload
The protobuf description is:
```protobuf
message ChatMessage {
// Lamport timestamp of the chat message
uint64 clock = 1;
// Unix timestamps in milliseconds, currently not used as we use Whisper/Waku as more reliable, but here
// so that we don't rely on it
uint64 timestamp = 2;
// Text of the message
string text = 3;
// Id of the message that we are replying to
string response_to = 4;
// Ens name of the sender
string ens_name = 5;
// Chat id, this field is symmetric for public-chats and private group chats,
// but asymmetric in case of one-to-ones, as the sender will use the chat-id
// of the received, while the receiver will use the chat-id of the sender.
// Probably should be the concatenation of sender-pk & receiver-pk in alphabetical order
string chat_id = 6;
// The type of message (public/one-to-one/private-group-chat)
MessageType message_type = 7;
// The type of the content of the message
ContentType content_type = 8;
oneof payload {
StickerMessage sticker = 9;
}
enum MessageType {
UNKNOWN_MESSAGE_TYPE = 0;
ONE_TO_ONE = 1;
PUBLIC_GROUP = 2;
PRIVATE_GROUP = 3;
// Only local
SYSTEM_MESSAGE_PRIVATE_GROUP = 4;}
enum ContentType {
UNKNOWN_CONTENT_TYPE = 0;
TEXT_PLAIN = 1;
STICKER = 2;
STATUS = 3;
EMOJI = 4;
TRANSACTION_COMMAND = 5;
// Only local
SYSTEM_MESSAGE_CONTENT_PRIVATE_GROUP = 6;
}
}
```
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | The clock of the chat|
| 2 | timestamp | `uint64` | The sender timestamp at message creation |
| 3 | text | `string` | The content of the message |
| 4 | response_to | `string` | The ID of the message replied to |
| 5 | ens_name | `string` | The ENS name of the user sending the message |
| 6 | chat_id | `string` | The local ID of the chat the message is sent to |
| 7 | message_type | `MessageType` | The type of message, different for one-to-one, public or group chats |
| 8 | content_type | `ContentType` | The type of the content of the message |
| 9 | payload | `Sticker\|nil` | The payload of the message based on the content type |
#### Content types
A node requires content types for a proper interpretation of incoming messages. Not each message is plain text but may carry different information.
The following content types MUST be supported:
* `TEXT_PLAIN` identifies a message which content is a plaintext.
There are other content types that MAY be implemented by the client:
* `STICKER`
* `STATUS`
* `EMOJI`
* `TRANSACTION_COMMAND`
##### Mentions
A mention MUST be represented as a string with the `@0xpk` format, where `pk` is the public key of the [user account](/status/deprecated/account.md) to be mentioned,
within the `text` field of a message with content_type `TEXT_PLAIN`.
A message MAY contain more than one mention.
This specification RECOMMENDs that the application does not require the user to enter the entire pk.
This specification RECOMMENDs that the application allows the user to create a mention
by typing @ followed by the related ENS or 3-word pseudonym.
This specification RECOMMENDs that the application provides the user auto-completion functionality to create a mention.
For better user experience, the client SHOULD display a known [ens name or the 3-word pseudonym corresponding to the key](/status/deprecated/account.md#contact-verification) instead of the `pk`.
##### Sticker content type
A `ChatMessage` with `STICKER` `Content/Type` MUST also specify the ID of the `Pack` and
the `Hash` of the pack, in the `Sticker` field of `ChatMessage`
```protobuf
message StickerMessage {
string hash = 1;
int32 pack = 2;
}
```
#### Message types
A node requires message types to decide how to encrypt a particular message
and what metadata needs to be attached when passing a message to the transport layer.
For more on this, see [WHISPER-USAGE](/status/deprecated/whisper-usage.md)
and [WAKU-USAGE](/status/deprecated/waku-usage.md).
<!-- TODO: This reference is a bit odd, considering the layer payloads should interact with is Secure Transport, and not Whisper/Waku. This requires more detail -->
The following messages types MUST be supported:
* `ONE_TO_ONE` is a message to the public group
* `PUBLIC_GROUP` is a private message
* `PRIVATE_GROUP` is a message to the private group.
#### Clock vs Timestamp and message ordering
If a user sends a new message before the messages sent
while the user was offline are received,
the newmessage is supposed to be displayed last in a chat.
This is where the basic algorithm of Lamport timestamp would fall short
as it's only meant to order causally related events.
The status client therefore makes a "bid", speculating that it will beat the current chat-timestamp, s.t. the status client's
Lamport timestamp format is: `clock = max({timestamp}, chat_clock + 1)`
This will satisfy the Lamport requirement, namely: a -> b then T(a) < T(b)
`timestamp` MUST be Unix time calculated, when the node creates the message, in milliseconds.
This field SHOULD not be relied upon for message ordering.
`clock` SHOULD be calculated using the algorithm of [Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps).
When there are messages available in a chat,
the node calculates `clock`'s value based on the last received message in a particular chat: `max(timeNowInMs, last-message-clock-value + 1)`.
If there are no messages, `clock` is initialized with `timestamp`'s value.
Messages with a `clock` greater than `120` seconds over the Whisper/Waku timestamp SHOULD be discarded,
in order to avoid malicious users to increase the `clock` of a chat arbitrarily.
Messages with a `clock` less than `120` seconds under the Whisper/Waku timestamp
might indicate an attempt to insert messages in the chat history which is not distinguishable from a `datasync` layer re-transit event.
A client MAY mark this messages with a warning to the user, or discard them.
The node uses `clock` value for the message ordering.
The algorithm used, and the distributed nature of the system produces casual ordering, which might produce counter-intuitive results in some edge cases.
For example, when a user joins a public chat and sends a message
before receiving the exist messages, their message `clock` value might be lower
and the message will end up in the past when the historical messages are fetched.
#### Chats
Chat is a structure that helps organize messages.
It's usually desired to display messages only from a single recipient,
or a group of recipients at a time and chats help to achieve that.
All incoming messages can be matched against a chat.
The below table describes how to calculate a chat ID for each message type.
|Message Type|Chat ID Calculation|Direction|Comment|
|------------|-------------------|---------|-------|
|PUBLIC_GROUP|chat ID is equal to a public channel name; it should equal `chatId` from the message|Incoming/Outgoing||
|ONE_TO_ONE|let `P` be a public key of the recipient; `hex-encode(P)` is a chat ID; use it as `chatId` value in the message|Outgoing||
|user-message|let `P` be a public key of message's signature; `hex-encode(P)` is a chat ID; discard `chat-id` from message|Incoming|if there is no matched chat, it might be the first message from public key `P`; the node MAY discard the message or MAY create a new chat; Status official clients create a new chat|
|PRIVATE_GROUP|use `chatId` from the message|Incoming/Outgoing|find an existing chat by `chatId`; if none is found, the user is not a member of that chat or the user hasn't joined that chat, the message MUST be discarded |
### Contact Update
`ContactUpdate` is a message exchange to notify peers that either the
user has been added as a contact, or that information about the sending user have
changed.
```protobuf
message ContactUpdate {
uint64 clock = 1;
string ens_name = 2;
string profile_image = 3;
}
```
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | The clock of the chat with the user |
| 2 | ens_name | `string` | The ENS name if set |
| 3 | profile_image | `string` | The base64 encoded profile picture of the user |
#### Contact update
A client SHOULD send a `ContactUpdate` to all the contacts each time:
* The ens_name has changed
* A user edits the profile image
A client SHOULD also periodically send a `ContactUpdate` to all the contacts, the interval is up to the client,
the Status official client sends these updates every 48 hours.
### SyncInstallationContact
The node uses `SyncInstallationContact` messages to synchronize in a best-effort the contacts to other devices.
```protobuf
message SyncInstallationContact {
uint64 clock = 1;
string id = 2;
string profile_image = 3;
string ens_name = 4;
uint64 last_updated = 5;
repeated string system_tags = 6;
}
```
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | clock value of the chat |
| 2 | id | `string` | id of the contact synced |
| 3 | profile_image | `string` | `base64` encoded profile picture of the user |
| 4 | ens_name | `string` | ENS name of the contact |
| 5 | `array[string]` | Array of `system_tags` for the user, this can currently be: `":contact/added", ":contact/blocked", ":contact/request-received"`||
### SyncInstallationPublicChat
The node uses `SyncInstallationPublicChat` message to synchronize in a best-effort the public chats to other devices.
```protobuf
message SyncInstallationPublicChat {
uint64 clock = 1;
string id = 2;
}
```
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | clock value of the chat |
| 2 | id | `string` | id of the chat synced |
### PairInstallation
The node uses `PairInstallation` messages to propagate information about a device to its paired devices.
```protobuf
message PairInstallation {
uint64 clock = 1;
string installation_id = 2;
string device_type = 3;
string name = 4;
}
```
Payload
| Field | Name | Type | Description |
| ----- | ---- | ---- | ---- |
| 1 | clock | `uint64` | clock value of the chat |
| 2| installation_id | `string` | A randomly generated id that identifies this device |
| 3 | device_type | `string` | The OS of the device `ios`,`android` or `desktop` |
| 4 | name | `string` | The self-assigned name of the device |
### MembershipUpdateMessage and MembershipUpdateEvent
`MembershipUpdateEvent` is a message used to propagate information about group membership changes in a group chat.
The details are in the [Group chats specs](/status/deprecated/group-chat.md).
## Upgradability
There are two ways to upgrade the protocol without breaking compatibility:
* A node always supports accretion
* A node does not support deletion of existing fields or messages, which might break compatibility
## Security Considerations
## Changelog
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
* Added language to include Waku in all relevant places
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
[Status Whitepaper](https://status.im/whitepaper.pdf)
[protobuf record](https://developers.google.com/protocol-buffers/)
[Protobuf](https://developers.google.com/protocol-buffers)
[Status user account](/status/deprecated/account.md)
[ens name or the 3-word pseudonym corresponding to the key](deprecated/account/#contact-verification)
[WHISPER-USAGE](/status/deprecated/whisper-usage.md)
[WAKU-USAGE](/status/deprecated/waku-usage.md)
[Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps)
[Group chats specs](/status/deprecated/group-chat.md)
[May 22, 2020 change commit](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)

View File

@@ -0,0 +1,753 @@
---
title: PUSH-NOTIFICATION-SERVER
name: Push notification server
status: deprecated
description: Status provides a set of Push notification services that can be used to achieve this functionality.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Andrea Maria Piana <andreap@status.im>
---
## Reason
Push notifications for iOS devices and some Android devices can only be implemented by relying on [APN service](https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html#//apple_ref/doc/uid/TP40008194-CH8-SW1) for iOS or [Firebase](https://firebase.google.com/).
This is useful for Android devices that do not support foreground services
or that often kill the foreground service.
iOS only allows certain kind of applications to keep a connection open when in the
background, VoIP for example, which current status client does not qualify for.
Applications on iOS can also request execution time when they are in the [background](https://developer.apple.com/documentation/uikit/app_and_environment/scenes/preparing_your_ui_to_run_in_the_background/updating_your_app_with_background_app_refresh)
but it has a limited set of use cases, for example it won't schedule any time
if the application was force quit,
and generally is not responsive enough to implement a push notification system.
Therefore Status provides a set of Push notification services
that can be used to achieve this functionality.
Because this can't be safely implemented in a privacy preserving manner,
clients MUST be given an option to opt-in to receiving and sending push notifications.
They are disabled by default.
## Requirements
The party releasing the app MUST possess a certificate for the Apple Push Notification service
and its has to run a [gorush](https://github.com/appleboy/gorush) publicly accessible server for sending the actual notification.
The party releasing the app, Status in this case, needs to run its own [gorush](https://github.com/appleboy/gorush)
## Components
### Gorush instance
A [gorush](https://github.com/appleboy/gorush) instance MUST be publicly available,
this will be used only by push notification servers.
### Push notification server
A push notification server used by clients to register for receiving and sending push notifications.
### Registering client
A Status client that wants to receive push notifications
### Sending client
A Status client that wants to send push notifications
## Registering with the push notification service
A client MAY register with one or more Push Notification services of their choice.
A `PNR message` (Push Notification Registration) MUST be sent to the [partitioned topic](status/deprecated/waku-usage/#partitioned-topic)
for the public key of the node, encrypted with this key.
The message MUST be wrapped in a [`ApplicationMetadataMessage`](status/deprecated/payload/#payload-wrapper) with type set to `PUSH_NOTIFICATION_REGISTRATION`.
The marshaled protobuf payload MUST also be encrypted with AES-GCM
using the DiffieHellman key generated from the client and server identity.
This is done in order to ensure that the extracted key from the signature will be
considered invalid if it can't decrypt the payload.
The content of the message MUST contain the following [protobuf record](https://developers.google.com/protocol-buffers/):
```protobuf
message PushNotificationRegistration {
enum TokenType {
UNKNOWN_TOKEN_TYPE = 0;
APN_TOKEN = 1;
FIREBASE_TOKEN = 2;
}
TokenType token_type = 1;
string device_token = 2;
string installation_id = 3;
string access_token = 4;
bool enabled = 5;
uint64 version = 6;
repeated bytes allowed_key_list = 7;
repeated bytes blocked_chat_list = 8;
bool unregister = 9;
bytes grant = 10;
bool allow_from_contacts_only = 11;
string apn_topic = 12;
bool block_mentions = 13;
repeated bytes allowed_mentions_chat_list = 14;
}
```
A push notification server will handle the message according to the following rules:
- it MUST extract the public key of the sender from the signature and verify that
the payload can be decrypted successfully.
- it MUST verify that `token_type` is supported
- it MUST verify that `device_token` is non empty
- it MUST verify that `installation_id` is non empty
- it MUST verify that `version` is non-zero and greater than the currently stored version for the public key and installation id of the sender, if any
- it MUST verify that `grant` is non empty and according to the [specs](#server-grant)
- it MUST verify that `access_token` is a valid [`uuid`](https://tools.ietf.org/html/rfc4122)
- it MUST verify that `apn_topic` is set if `token_type` is `APN_TOKEN`
If the message can't be decrypted, the message MUST be discarded.
If `token_type` is not supported, a response MUST be sent with `error` set to
`UNSUPPORTED_TOKEN_TYPE`.
If `token`,`installation_id`,`device_tokens`,`version` are empty, a response MUST
be sent with `error` set to `MALFORMED_MESSAGE`.
If the `version` is equal or less than the currently stored version, a response MUST
be sent with `error` set to `VERSION_MISMATCH`.
If any other error occurs the `error` should be set to `INTERNAL_ERROR`.
If the response is successful `success` MUST be set to `true` otherwise a response MUST be sent with `success` set to `false`.
`request_id` should be set to the `SHAKE-256` of the encrypted payload.
The response MUST be sent on the [partitioned topic](status/deprecated/waku-usage.md#partitioned-topic) of the sender
and MUST not be encrypted using the [secure transport](status/deprecated/secure-transport) to facilitate the usage of ephemeral keys.
The payload of the response is:
```protobuf
message PushNotificationRegistrationResponse {
bool success = 1;
ErrorType error = 2;
bytes request_id = 3;
enum ErrorType {
UNKNOWN_ERROR_TYPE = 0;
MALFORMED_MESSAGE = 1;
VERSION_MISMATCH = 2;
UNSUPPORTED_TOKEN_TYPE = 3;
INTERNAL_ERROR = 4;
}
}
```
The message MUST be wrapped in a [`ApplicationMetadataMessage`](status/deprecated/payloads.md#payload-wrapper) with type set to `PUSH_NOTIFICATION_REGISTRATION_RESPONSE`.
A client SHOULD listen for a response sent on the [partitioned topic](status/deprecated/waku-usage/#partitioned-topic)
that the key used to register.
If `success` is `true` the client has registered successfully.
If `success` is `false`:
- If `MALFORMED_MESSAGE` is returned, the request SHOULD NOT be retried without ensuring that it is correctly formed.
- If `INTERNAL_ERROR` is returned, the request MAY be retried, but the client MUST backoff exponentially
A client MAY register with multiple Push Notification Servers in order to increase availability.
A client SHOULD make sure that all the notification services they registered with have the same information about their tokens.
If no response is returned the request SHOULD be considered failed and MAY be retried with the same server or a different one, but clients MUST exponentially backoff after each trial.
If the request is successful the token SHOULD be [advertised](#advertising-a-push-notification-server) as described below
### Query topic
On successful registration the server MUST be listening to the topic derived from:
```protobuf
0XHexEncode(Shake256(CompressedClientPublicKey))
```
Using the topic derivation algorithm described [here](status/deprecated/waku-usage/#public-chats)
and listen for client queries.
### Server grant
A push notification server needs to demonstrate to a client that it was authorized
by the client to send them push notifications. This is done by building
a grant which is specific to a given client-server pair.
The grant is built as follow:
```protobuf
Signature(Keccak256(CompressedPublicKeyOfClient . CompressedPublicKeyOfServer . AccessToken), PrivateKeyOfClient)
```
When receiving a grant the server MUST be validate that the signature matches the registering client.
## Re-registering with the push notification server
A client SHOULD re-register with the node if the APN or FIREBASE token changes.
When re-registering a client SHOULD ensure that it has the most up-to-date
`PushNotificationRegistration` and increment `version` if necessary.
Once re-registered, a client SHOULD advertise the changes.
## Changing options
This is handled in exactly the same way as re-registering above.
## Unregistering from push notifications
To unregister a client MUST send a `PushNotificationRegistration` request as described
above with `unregister` set to `true`, or removing
their device information.
The server MUST remove all data about this user if `unregistering` is `true`,
apart from the `hash` of the public key and the `version` of the last options,
in order to make sure that old messages are not processed.
A client MAY unregister from a server on explicit logout if multiple chat keys
are used on a single device.
## Advertising a push notification server
Each user registered with one or more push notification servers SHOULD
advertise periodically the push notification services that they have registered with for each device they own.
```protobuf
message PushNotificationQueryInfo {
string access_token = 1;
string installation_id = 2;
bytes public_key = 3;
repeated bytes allowed_user_list = 4;
bytes grant = 5;
uint64 version = 6;
bytes server_public_key = 7;
}
message ContactCodeAdvertisement {
repeated PushNotificationQueryInfo push_notification_info = 1;
}
```
The message MUST be wrapped in a [`ApplicationMetadataMessage`](status/deprecated/payloads/#payload-wrapper) with type set to `PUSH_NOTIFICATION_QUERY_INFO`.
If no filtering is done based on public keys,
the access token SHOULD be included in the advertisement.
Otherwise it SHOULD be left empty.
This SHOULD be advertised on the [contact code topic](/status/deprecated/waku-usage.md#contact-code-topic)
and SHOULD be coupled with normal contact-code advertisement.
Every time a user register or re-register with a push notification service, their
contact-code SHOULD be re-advertised.
Multiple servers MAY be advertised for the same `installation_id` for redundancy reasons.
## Discovering a push notification server
To discover a push notification service for a given user, their [contact code topic](status/deprecated/waku-usage/#contact-code-topic)
SHOULD be listened to.
A mailserver can be queried for the specific topic to retrieve the most up-to-date
contact code.
## Querying the push notification server
If a token is not present in the latest advertisement for a user, the server
SHOULD be queried directly.
To query a server a message:
```protobuf
message PushNotificationQuery {
repeated bytes public_keys = 1;
}
```
The message MUST be wrapped in a [`ApplicationMetadataMessage`](status/deprecated/payloads/#payload-wrapper) with type set to `PUSH_NOTIFICATION_QUERY`.
MUST be sent to the server on the topic derived from the hashed public key of the
key we are querying, as [described above](#query-topic).
An ephemeral key SHOULD be used and SHOULD NOT be encrypted using the [secure transport](status/deprecated/secure-transport.md).
If the server has information about the client a response MUST be sent:
```protobuf
message PushNotificationQueryInfo {
string access_token = 1;
string installation_id = 2;
bytes public_key = 3;
repeated bytes allowed_user_list = 4;
bytes grant = 5;
uint64 version = 6;
bytes server_public_key = 7;
}
message PushNotificationQueryResponse {
repeated PushNotificationQueryInfo info = 1;
bytes message_id = 2;
bool success = 3;
}
```
A `PushNotificationQueryResponse` message MUST be wrapped in a [`ApplicationMetadataMessage`](status/deprecated/payloads.md#payload-wrapper) with type set to `PUSH_NOTIFICATION_QUERY_RESPONSE`.
Otherwise a response MUST NOT be sent.
If `allowed_key_list` is not set `access_token` MUST be set and `allowed_key_list` MUST NOT
be set.
If `allowed_key_list` is set `allowed_key_list` MUST be set and `access_token` MUST NOT be set.
If `access_token` is returned, the `access_token` SHOULD be used to send push notifications.
If `allowed_key_list` are returned, the client SHOULD decrypt each
token by generating an `AES-GCM` symmetric key from the DiffieHellman between the
target client and itself
If AES decryption succeeds it will return a valid [`uuid`](https://tools.ietf.org/html/rfc4122) which is what is used for access_token.
The token SHOULD be used to send push notifications.
The response MUST be sent on the [partitioned topic](status/deprecated/waku-usage/#partitioned-topic) of the sender
and MUST not be encrypted using the [secure transport](status/deprecated/secure-transport.md) to facilitate
the usage of ephemeral keys.
On receiving a response a client MUST verify `grant` to ensure that the server
has been authorized to send push notification to a given client.
## Sending a push notification
When sending a push notification, only the `installation_id` for the devices targeted
by the message SHOULD be used.
If a message is for all the user devices, all the `installation_id` known to the client MAY be used.
The number of devices MAY be capped in order to reduce resource consumption.
At least 3 devices SHOULD be targeted, ordered by last activity.
For any device that a token is available, or that a token is successfully queried,
a push notification message SHOULD be sent to the corresponding push notification server.
```protobuf
message PushNotification {
string access_token = 1;
string chat_id = 2;
bytes public_key = 3;
string installation_id = 4;
bytes message = 5;
PushNotificationType type = 6;
enum PushNotificationType {
UNKNOWN_PUSH_NOTIFICATION_TYPE = 0;
MESSAGE = 1;
MENTION = 2;
}
bytes author = 7;
}
message PushNotificationRequest {
repeated PushNotification requests = 1;
bytes message_id = 2;
}
```
A `PushNotificationRequest` message MUST be wrapped in a [`ApplicationMetadataMessage`](/status/deprecated/payloads.md#payload-wrapper) with type set to `PUSH_NOTIFICATION_REQUEST`.
Where `message` is the encrypted payload of the message and `chat_id` is the
`SHAKE-256` of the `chat_id`.
`message_id` is the id of the message
`author` is the `SHAKE-256` of the public key of the sender.
If multiple server are available for a given push notification, only one notification
MUST be sent.
If no response is received
a client SHOULD wait at least 3 seconds, after which the request MAY be retried against a different server
This message SHOULD be sent using an ephemeral key.
On receiving the message, the push notification server MUST validate the access token.
If the access token is valid, a notification MUST be sent to the gorush instance with the
following data:
```protobuf
{
"notifications": [
{
"tokens": ["token_a", "token_b"],
"platform": 1,
"message": "You have a new message",
"data": {
"chat_id": chat_id,
"message": message,
"installation_ids": [installation_id_1, installation_id_2]
}
}
]
}
```
Where platform is `1` for IOS and `2` for Firebase, according to the [gorush documentation](https://github.com/appleboy/gorush)
A server MUST return a response message:
```protobuf
message PushNotificationReport {
bool success = 1;
ErrorType error = 2;
enum ErrorType {
UNKNOWN_ERROR_TYPE = 0;
WRONG_TOKEN = 1;
INTERNAL_ERROR = 2;
NOT_REGISTERED = 3;
}
bytes public_key = 3;
string installation_id = 4;
}
message PushNotificationResponse {
bytes message_id = 1;
repeated PushNotificationReport reports = 2;
}
```
A `PushNotificationResponse` message MUST be wrapped in a [`ApplicationMetadataMessage`](/status/deprecated/payloads.md#payload-wrapper) with type set to `PUSH_NOTIFICATION_RESPONSE`.
Where `message_id` is the `message_id` sent by the client.
The response MUST be sent on the [partitioned topic](/status/deprecated/waku-usage.md#partitioned-topic) of the sender
and MUST not be encrypted using the [secure transport](/status/deprecated/secure-transport.md) to facilitate
the usage of ephemeral keys.
If the request is accepted `success` MUST be set to `true`.
Otherwise `success` MUST be set to `false`.
If `error` is `BAD_TOKEN` the client MAY query again the server for the token and
retry the request.
If `error` is `INTERNAL_ERROR` the client MAY retry the request.
## Flow
### Registration process
- A client will generate a notification token through `APN` or `Firebase`.
- The client will [register](#registering-with-the-push-notification-service) with one or more push notification server of their choosing.
- The server should process the response and respond according to the success of the operation
- If the request is not successful it might be retried, and adjusted according to the response. A different server can be also used.
- Once the request is successful the client should [advertise](#advertising-a-push-notification-server) the new coordinates
### Sending a notification
- A client should prepare a message and extract the targeted installation-ids
- It should retrieve the most up to date information for a given user, either by
querying a push notification server, a mailserver if not listening already to the given topic, or checking
the database locally
- It should then [send](#sending-a-push-notification) a push notification according
to the rules described
- The server should then send a request to the gorush server including all the required
information
### Receiving a push notification
- On receiving the notification, a client can open the right account by checking the
`installation_id` included. The `chat_id` MAY be used to open the chat if present.
- `message` can be decrypted and presented to the user. Otherwise messages can be pulled from the mailserver if the `message_id` is no already present.
## Protobuf description
### PushNotificationRegistration
`token_type`: the type of token. Currently supported is `APN_TOKEN` for Apple Push
`device_token`: the actual push notification token sent by `Firebase` or `APN`
and `FIREBASE_TOKEN` for firebase.
`installation_id`: the [`installation_id`](/status/deprecated/account.md) of the device
`access_token`: the access token that will be given to clients to send push notifications
`enabled`: whether the device wants to be sent push notifications
`version`: a monotonically increasing number identifying the current `PushNotificationRegistration`. Any time anything is changed in the record it MUST be increased by the client, otherwise the request will not be accepted.
`allowed_key_list`: a list of `access_token` encrypted with the AES key generated
by DiffieHellman between the publisher and the allowed
contact.
`blocked_chat_list`: a list of `SHA2-256` hashes of chat ids.
Any chat id in this list will not trigger a notification.
`unregister`: whether the account should be unregistered
`grant`: the grant for this specific server
`allow_from_contacts_only`: whether the client only wants push notifications from contacts
`apn_topic`: the APN topic for the push notification
`block_mentions`: whether the client does not want to be notified on mentions
`allowed_mentions_chat_list`: a list of `SHA2-256` hashes of chat ids where we want to receive mentions
#### Data disclosed
- Type of device owned by a given user
- The `FIREBASE` or `APN` push notification token
- Hash of the chat_id a user is not interested in for notifications
- The times a push notification record has been modified by the user
- The number of contacts a client has, in case `allowed_key_list` is set
### PushNotificationRegistrationResponse
`success`: whether the registration was successful
`error`: the error type, if any
`request_id`: the `SHAKE-256` hash of the `signature` of the request
`preferences`: the server stored preferences in case of an error
### ContactCodeAdvertisement
`push_notification_info`: the information for each device advertised
Data disclosed
- The chat key of the sender
### PushNotificationQuery
`public_keys`: the `SHAKE-256` of the public keys the client is interested in
Data disclosed
- The hash of the public keys the client is interested in
### PushNotificationQueryInfo
`access_token`: the access token used to send a push notification
`installation_id`: the `installation_id` of the device associated with the `access_token`
`public_key`: the `SHAKE-256` of the public key associated with this `access_token` and `installation_id`
`allowed_key_list`: a list of encrypted access tokens to be returned
to the client in case there's any filtering on public keys in place.
`grant`: the grant used to register with this server.
`version`: the version of the registration on the server.
`server_public_key`: the compressed public key of the server.
### PushNotificationQueryResponse
`info`: a list of `PushNotificationQueryInfo`.
`message_id`: the message id of the `PushNotificationQueryInfo` the server is replying to.
`success`: whether the query was successful.
### PushNotification
`access_token`: the access token used to send a push notification.
`chat_id`: the `SHAKE-256` of the `chat_id`.
`public_key`: the `SHAKE-256` of the compressed public key of the receiving client.
`installation_id`: the installation id of the receiving client.
`message`: the encrypted message that is being notified on.
`type`: the type of the push notification, either `MESSAGE` or `MENTION`
`author`: the `SHAKE-256` of the public key of the sender
Data disclosed
- The `SHAKE-256` of the `chat_id` the notification is to be sent for
- The cypher text of the message
- The `SHAKE-256` of the public key of the sender
- The type of notification
### PushNotificationRequest
`requests`: a list of `PushNotification`
`message_id`: the [status message id](/status/deprecated/payloads.md)
Data disclosed
- The status message id for which the notification is for
### PushNotificationResponse
`message_id`: the `message_id` being notified on.
`reports`: a list of `PushNotificationReport`
### PushNotificationReport
`success`: whether the push notification was successful.
`error`: the type of the error in case of failure.
`public_key`: the public key of the user being notified.
`installation_id`: the installation id of the user being notified.
## Anonymous mode of operations
An anonymous mode of operations MAY be provided by the client, where the
responsibility of propagating information about the user is left to the client,
in order to preserve privacy.
A client in anonymous mode can register with the server using a key different
from their chat key.
This will hide their real chat key.
This public key is effectively a secret and SHOULD only be disclosed to clients that you the user wants to be notified by.
A client MAY advertise the access token on the contact-code topic of the key generated.
A client MAY share their public key through [contact updates](/status/deprecated/payloads.md#contact-update)
A client receiving a push notification public key SHOULD listen to the contact code
topic of the push notification public key for updates.
The method described above effectively does not share the identity of the sender
nor the receiver to the server, but MAY result in missing push notifications as
the propagation of the secret is left to the client.
This can be mitigated by [device syncing](/status/deprecated/payloads.md), but not completely
addressed.
## Security considerations
If no anonymous mode is used, when registering with a push notification service a client discloses:
- The chat key
- The devices that will receive notifications
A client MAY disclose:
- The hash of the chat_ids they want to filter out
When running in anonymous mode, the client's chat key is not disclosed.
When querying a push notification server a client will disclose:
- That it is interested in sending push notification to another client,
but the querying client's chat key is not disclosed
When sending a push notification a client discloses:
- The `SHAKE-256` of the chat id
[//]: This section can be removed, for now leaving it here in order to help with the
review process. Point can be integrated, suggestion welcome.
## FAQ
### Why having ACL done at the server side and not the client?
We looked into silent notification for
[IOS](https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/pushing_background_updates_to_your_app) (android has no equivalent)
but can't be used as it's expected to receive maximum 2/3 per hour, so not our use case. There
are also issue when the user force quit the app.
### Why using an access token?
The access token is used to decouple the requesting information from the user from
actually sending the push notification.
Some ACL is necessary otherwise it would be too easy to spam users (it's still fairly
trivial, but with this method you could allow only contacts to send you push notifications).
Therefore your identity must be revealed to the server either when sending or querying.
By using an access token we increase deniability, as the server would know
who requested the token but not necessarily who sent a push notification.
Correlation between the two can be trivial in some cases.
This also allows a mode of use as we had before, where the server does not propagate
info at all, and it's left to the user to propagate the token, through contact requests
for example.
### Why advertise with the bundle?
Advertising with the bundle allows us to piggy-back on an already implemented behavior
and save some bandwidth in cases where is not filtering by public keys
### What's the bandwidth impact for this?
Generally speaking, for each 1-to-1 message and group chat message you will sending
1 and `number of participants` push notifications. This can be optimized if
multiple users are using the same push notification server. Queries have also
a bandwidth impact but they are made only when actually needed
### What's the information disclosed?
The data disclosed with each message sent by the client is above, but for a summary:
When you register with a push notification service you may disclose:
1) Your chat key
2) Which devices you have
3) The hash of the chat_ids you want to filter out
4) The hash of the public keys you are interested/not interested in
When you query a notification service you may disclose:
1) Your chat key
2) The fact that you are interested in sending push notification to a given user
Effectively this is fairly revealing if the user has a whitelist implemented.
Therefore sending notification should be optional.
### What prevents a user from generating a random key and getting an access token and spamming?
Nothing really, that's the same as the status app as a whole. the only mechanism that prevents
this is using a white-list as described above,
but that implies disclosing your true identity to the push notification server.
### Why not 0-knowledge proofs/quantum computing
We start simple, we can iterate
### How to handle backward/forward compatibility
Most of the request have a target, so protocol negotiation can happen. We cannot negotiated
the advertisement as that's effectively a broadcast, but those info should not change and we can
always accrete the message.
### Why ack_key?
That's necessary to avoid duplicated push notifications and allow for the retry
in case the notification is not successful.
Deduplication of the push notification is done on the client side, to reduce a bit
of centralization and also in order not to have to modify gorush.
### Can I run my own node?
Sure, the methods allow that
### Can I register with multiple nodes for redundancy
Yep
### What does my node disclose?
Your node will disclose the IP address is running from, as it makes an HTTP post to
gorush. A waku adapter could be used, but please not now.
### Does this have high-reliability requirements?
The gorush server yes, no way around it.
The rest, kind of, at least one node having your token needs to be up for you to receive notifications.
But you can register with multiple servers (desktop, status, etc) if that's a concern.
### Can someone else (i.e not status) run this?
Push notification servers can be run by anyone. Gorush can be run by anyone I take,
but we are in charge of the certificate, so they would not be able to notify status-clients.
## Changelog
### Version 0.1
[Released](https://github.com/status-im/specs/commit/)
- Initial version
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [APN Service](https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html#//apple_ref/doc/uid/TP40008194-CH8-SW1)
- [Background Execution on iOS](https://developer.apple.com/documentation/uikit/app_and_environment/scenes/preparing_your_ui_to_run_in_the_background/updating_your_app_with_background_app_refresh)
- [Firebase](https://firebase.google.com/)
- [Gorush](https://github.com/appleboy/gorush)
- [UUID Specification](https://tools.ietf.org/html/rfc4122)
- [Secure Transport](/status/deprecated/secure-transport.md)
- [Silent Notifications on iOS](https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/pushing_background_updates_to_your_app)
- [Waku Usage](/status/deprecated/waku-usage.md)
- [ENS Contract](https://github.com/ensdomains/ens)
- [Payloads](/status/deprecated/payloads.md)

View File

@@ -0,0 +1,583 @@
---
title: SECURE-TRANSPORT
name: Secure Transport
status: deprecated
description: This document describes how Status provides a secure channel between two peers, providing confidentiality, integrity, authenticity, and forward secrecy.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Andrea Maria Piana <andreap@status.im>
- Corey Petty <corey@status.im>
- Dean Eigenmann <dean@status.im>
- Oskar Thorén <oskar@status.im>
- Pedro Pombeiro <pedro@status.im>
---
## Abstract
This document describes how Status provides a secure channel between two peers,
and thus provide confidentiality, integrity, authenticity and forward secrecy.
It is transport-agnostic and works over asynchronous networks.
It builds on the [X3DH](https://signal.org/docs/specifications/x3dh/) and [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/) specifications, with some adaptations to operate in a decentralized environment.
## Introduction
This document describes how nodes establish a secure channel,
and how various conversational security properties are achieved.
### Definitions
- **Perfect Forward Secrecy** is a feature of specific key-agreement protocols
which provide assurances that session keys will not be compromised even if the private keys of the participants are compromised.
Specifically, past messages cannot be decrypted by a third-party who manages to get a hold of a private key.
- **Secret channel** describes a communication channel where Double Ratchet algorithm is in use.
### Design Requirements
- **Confidentiality**: The adversary should not be able to learn what data is being exchanged between two Status clients.
- **Authenticity**: The adversary should not be able to cause either endpoint of a Status 1:1 chat
to accept data from any third party as though it came from the other endpoint.
- **Forward Secrecy**: The adversary should not be able to learn
what data was exchanged between two Status clients if, at some later time,
the adversary compromises one or both of the endpoint devices.
- **Integrity**: The adversary should not be able to cause either endpoint of a Status 1:1 chat
to accept data that has been tampered with.
All of these properties are ensured by the use of [Signal's Double Ratchet](https://signal.org/docs/specifications/doubleratchet/)
### Conventions
Types used in this specification are defined using [Protobuf](https://developers.google.com/protocol-buffers/).
### Transport Layer
[Whisper](status/deprecated/whisper-usage) and [Waku](status/deprecated/waku-usage) serves as the transport layers for the Status chat protocol.
### User flow for 1-to-1 communications
#### Account generation
See [Account specification](status/deprecated/account)
#### Account recovery
If Alice later recovers her account, the Double Ratchet state information will not be available,
so she is no longer able to decrypt any messages received from existing contacts.
If an incoming message (on the same Whisper/Waku topic) fails to decrypt,
the node replies a message with the current bundle, so that the node notifies the other end of the new device.
Subsequent communications will use this new bundle.
## Messaging
All 1:1 and group chat messaging in Status is subject to end-to-end encryption
to provide users with a strong degree of privacy and security.
Public chat messages are publicly readable by anyone since there's no permission model
for who is participating in a public chat.
The rest of this document is purely about 1:1 and private group chat.
Private group chat largely reduces to 1:1 chat, since there's a secure channel between each pair-wise participant.
### End-to-end encryption
End-to-end encryption (E2EE) takes place between two clients.
The main cryptographic protocol is a [Status implementation](https://github.com/status-im/doubleratchet/) of the Double Ratchet protocol,
which is in turn derived from the [Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html), using a different ratchet.
The transport protocol subsequently encrypt the message payload - Whisper/Waku (see section [Transport Layer](#transport-layer)) -, using symmetric key encryption.
Furthermore, Status uses the concept of prekeys (through the use of [X3DH](https://signal.org/docs/specifications/x3dh/))
to allow the protocol to operate in an asynchronous environment.
It is not necessary for two parties to be online at the same time to initiate an encrypted conversation.
Status uses the following cryptographic primitives:
- Whisper/Waku
- AES-256-GCM
- ECIES
- ECDSA
- KECCAK-256
- X3DH
- Elliptic curve Diffie-Hellman key exchange (secp256k1)
- KECCAK-256
- ECDSA
- ECIES
- Double Ratchet
- HMAC-SHA-256 as MAC
- Elliptic curve Diffie-Hellman key exchange (Curve25519)
- AES-256-CTR with HMAC-SHA-256 and IV derived alongside an encryption key
The node achieves key derivation using HKDF.
### Prekeys
Every client initially generates some key material which is stored locally:
- Identity keypair based on secp256k1 - `IK`
- A signed prekey based on secp256k1 - `SPK`
- A prekey signature - `Sig(IK, Encode(SPK))`
More details can be found in the `X3DH Prekey bundle creation` section of [2/ACCOUNT](/status/deprecated/account.md#x3dh-prekey-bundles).
Prekey bundles can be extracted from any user's messages,
or found via searching for their specific topic, `{IK}-contact-code`.
TODO: See below on bundle retrieval, this seems like enhancement and parameter for recommendation
### Bundle retrieval
<!-- TODO: Potentially move this completely over to [Trust Establishment](./status-account-spec.md) -->
X3DH works by having client apps create and make available a bundle of prekeys (the X3DH bundle)
that can later be requested by other interlocutors when they wish to start a conversation with a given user.
In the X3DH specification, nodes typically use a shared server
to store bundles and allow other users to download them upon request.
Given Status' goal of decentralization,
Status chat clients cannot rely on the same type of infrastructure
and must achieve the same result using other means.
By growing order of convenience and security, the considered approaches are:
- contact codes;
- public and one-to-one chats;
- QR codes;
- ENS record;
- Decentralized permanent storage (e.g. Swarm, IPFS).
- Whisper/Waku
<!-- TODO: Comment, it isn't clear what we actually _do_. It seems as if this is exploring the problem space. From a protocol point of view, it might make sense to describe the interface, and then have a recommendation section later on that specifies what we do. See e.g. Signal's specs where they specify specifics later on. -->
Currently, only public and one-to-one message exchanges and Whisper/Waku is used to exchange bundles.
Since bundles stored in QR codes or ENS records cannot be updated to delete already used keys,
the approach taken is to rotate more frequently the bundle (once every 24 hours),
which will be propagated by the app through the channel available.
### 1:1 chat contact request
There are two phases in the initial negotiation of a 1:1 chat:
1. **Identity verification** (e.g., face-to-face contact exchange through QR code, Identicon matching).
A QR code serves two purposes simultaneously - identity verification and initial bundle retrieval;
1. **Asynchronous initial key exchange**, using X3DH.
For more information on account generation and trust establishment, see [2/ACCOUNT](/status/deprecated/account.md)
#### Initial key exchange flow (X3DH)
[Section 3 of the X3DH protocol](https://signal.org/docs/specifications/x3dh/#sending-the-initial-message) describes the initial key exchange flow, with some additional context:
- The users' identity keys `IK_A` and `IK_B` correspond to their respective Status chat public keys;
- Since it is not possible to guarantee that a prekey will be used only once in a decentralized world,
the one-time prekey `OPK_B` is not used in this scenario;
- Nodes do not send Bundles to a centralized server, but instead served in a decentralized way as described in [bundle retrieval](#bundle-retrieval).
Alice retrieves Bob's prekey bundle, however it is not specific to Alice. It contains:
([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L12))
``` protobuf
// X3DH prekey bundle
message Bundle {
bytes identity = 1;
map<string,SignedPreKey> signed_pre_keys = 2;
bytes signature = 4;
int64 timestamp = 5;
}
```golang
- `identity`: Identity key `IK_B`
- `signed_pre_keys`: Signed prekey `SPK_B` for each device, indexed by `installation-id`
- `signature`: Prekey signature <i>Sig(`IK_B`, Encode(`SPK_B`))</i>
- `timestamp`: When the bundle was created locally
([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L5))
``` protobuf
message SignedPreKey {
bytes signed_pre_key = 1;
uint32 version = 2;
}
```
The `signature` is generated by sorting `installation-id` in lexicographical order, and concatenating the `signed-pre-key` and `version`:
`installation-id-1signed-pre-key1version1installation-id2signed-pre-key2-version-2`
#### Double Ratchet
Having established the initial shared secret `SK` through X3DH, it can be used to seed a Double Ratchet exchange between Alice and Bob.
Please refer to the [Double Ratchet spec](https://signal.org/docs/specifications/doubleratchet/) for more details.
The initial message sent by Alice to Bob is sent as a top-level `ProtocolMessage` ([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L65))
containing a map of `DirectMessageProtocol` indexed by `installation-id` ([protobuf](https://github.com/status-im/status-go/blob/1ac9dd974415c3f6dee95145b6644aeadf02f02c/services/shhext/chat/encryption.proto#L56)):
``` protobuf
message ProtocolMessage {
string installation_id = 2;
repeated Bundle bundles = 3;
// One to one message, encrypted, indexed by installation_id
map<string,DirectMessageProtocol> direct_message = 101;
// Public chats, not encrypted
bytes public_message = 102;
}
```
- `bundles`: a sequence of bundles
- `installation_id`: the installation id of the sender
- `direct_message` is a map of `DirectMessageProtocol` indexed by `installation-id`
- `public_message`: unencrypted public chat message.
``` protobuf
message DirectMessageProtocol {
X3DHHeader X3DH_header = 1;
DRHeader DR_header = 2;
DHHeader DH_header = 101;
// Encrypted payload
bytes payload = 3;
}
```
```protobuf
- `X3DH_header`: the `X3DHHeader` field in `DirectMessageProtocol` contains:
([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L47))
``` protobuf
message X3DHHeader {
bytes key = 1;
bytes id = 4;
}
```
- `key`: Alice's ephemeral key `EK_A`;
- `id`: Identifier stating which of Bob's prekeys Alice used, in this case Bob's bundle signed prekey.
Alice's identity key `IK_A` is sent at the transport layer level (Whisper/Waku);
- `DR_header`: Double ratchet header ([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L31)). Used when Bob's public bundle is available:
``` protobuf
message DRHeader {
bytes key = 1;
uint32 n = 2;
uint32 pn = 3;
bytes id = 4;
}
```
- `key`: Alice's current ratchet public key (as mentioned in [DR spec section 2.2](https://signal.org/docs/specifications/doubleratchet/#symmetric-key-ratchet));
- `n`: number of the message in the sending chain;
- `pn`: length of the previous sending chain;
- `id`: Bob's bundle ID.
- `DH_header`: Diffie-Helman header (used when Bob's bundle is not available):
([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L42))
``` protobuf
message DHHeader {
bytes key = 1;
}
```
- `key`: Alice's compressed ephemeral public key.
- `payload`:
- if a bundle is available, contains payload encrypted with the Double Ratchet algorithm;
- otherwise, payload encrypted with output key of DH exchange (no Perfect Forward Secrecy).
```
<!-- TODO: A lot of links to status-go, seems likely these should be updated to status-protocol-go -->
## Security Considerations
The same considerations apply as in [section 4 of the X3DH spec](https://signal.org/docs/specifications/x3dh/#security-considerations) and [section 6 of the Double Ratchet spec](https://signal.org/docs/specifications/doubleratchet/#security-considerations), with some additions detailed below.
<!-- TODO: Add any additional context here not covered in the X3DH and DR specs -->
<!--
TODO: description here
### --- Security and Privacy Features
#### Confidentiality (YES)
> Only the intended recipients are able to read a message. Specifically, the message must not be readable by a server operator that is not a conversation participant
- Yes.
- There's a layer of encryption at Whisper as well as above with Double Ratchet
- Relay nodes and Mailservers can only read a topic of a Whisper message, and nothing within the payload.
#### Integrity (YES)
> No honest party will accept a message that has been modified in transit.
- Yes.
- Assuming a user validates (TODO: Check this assumption) every message they are able to decrypt and validate its signature from the sender, then it is not able to be altered in transit.
* [igorm] i'm really not sure about it, Whisper provides a signature, but I'm not sure we check it anywhere (simple grepping didn't give anything)
* [andrea] Whisper checks the signature and a public key is derived from it, we check the public key is a meaningful public key. The pk itself is not in the content of the message for public chats/1-to-1 so potentially you could send a message from a random account without having access to the private key, but that would not be much of a deal, as you might just as easily create a random account)
#### Authentication (YES)
> Each participant in the conversation receives proof of possession of a known long-term secret from all other participants that they believe to be participating in the conversation. In addition, each participant is able to verify that a message was sent from the claimed source
- 1:1 --- one-to-one messages are encrypted with the recipient's public key, and digitally signed by the sender's. In order to provide Perfect Forward Secrecy, we build on the X3DH and Double Ratchet specifications from Open Whisper Systems, with some adaptations to operate in a decentralized environment.
- group --- group chat is pairwise
- public --- A user subscribes to a public channel topic and the decryption key is derived from the topic name
**TODO:** Need to verify that this is actually the case
**TODO:** Fill in explicit details here
#### Participant Consistency (YES?)
> At any point when a message is accepted by an honest party, all honest parties are guaranteed to have the same view of the participant list
- **TODO:** Need details here
#### Destination Validation (YES?)
> When a message is accepted by an honest party, they can verify that they were included in the set of intended recipients for the message.
- Users are aware of the topic that a message was sent to, and that they have the ability to decrypt it.
-
#### Forward Secrecy (PARTIAL)
> Compromising all key material does not enable decryption of previously encrypted data
- After first back and forth between two contacts with PFS enabled, yes.
#### Backward Secrecy (YES)
> Compromising all key material does not enable decryption of succeeding encrypted data
- PFS requires both backward and forwards secrecy
[Andrea: This is not true, (Perfect) Forward Secrecy does not imply backward secrecy (which is also called post-compromise security, as signal calls it, or future secrecy, it's not well defined). Technically this is a NO , double ratchet offers good Backward secrecy, but not perfect. Effectively if all the key material is compromised, any future message received will be also compromised (due to the hash ratchet), until a DH ratchet step is completed (i.e. the compromised party generate a new random key and ratchet)]
#### Anonymity Preserving (PARTIAL)
> Any anonymity features provided by the underlying transport privacy architecture are not undermined (e.g., if the transport privacy system provides anonymity, the conversation security level does not de-anonymize users by linking key identifiers).
- by default, yes
- ENS Naming system attaches an identifier to a given public key
#### Speaker Consistency (PARTIAL)
> All participants agree on the sequence of messages sent by each participant. A protocol might perform consistency checks on blocks of messages during the protocol, or after every message is sent.
- We use Lamport timestamps for ordering of events.
- In addition to this, we use local timestamps to attempt a more intuitive ordering. [Andrea: currently this was introduced as a regression during performance optimization and might result in out-of-order messages if sent across day boundaries, so I consider it a bug and not part of the specs (it does not make the order more intuitive, quite the opposite as it might result in causally related messages being out-of-order, but helps dividing the messages in days)]
- Fundamentally, there's no single source of truth, nor consensus process for global ordering [Andrea: Global ordering does not need a consensus process i.e. if you order messages alphabetically, and you break ties consistently, you have global ordering, as all the participants will see the same ordering (as opposed to say order by the time the message was received locally), of course is not useful, you want to have causal + global to be meaningful]
TODO: Understand how this is different from Global Transcript
[Andrea: This is basically Global transcript for a single participants, we offer global transcript]
#### Causality Preserving (PARTIAL)
> Implementations can avoid displaying a message before messages that causally precede it
- Not yet, but in pipeline (data sync layer)
[Andrea: Messages are already causally ordered, we don't display messages that are causally related out-of-order, that's already granted by lamport timestamps]
TODO: Verify if this can be done already by looking at Lamport clock difference
#### Global Transcript (PARTIAL)
> All participants see all messages in the same order
- See directly above
[Andrea: messages are globally (total) ordered, so all participants see the same ordering]
#### Message Unlinkability (NO)
> If a judge is convinced that a participant authored one message in the conversation, this does not provide evidence that they authored other messages
- Currently, the Status software signs every messages sent with the user's public key, thus making it unable to provide unlinkability.
- This is not necessary though, and could be built in to have an option to not sign.
- Side note: moot account allows for this but is a function of the anonymity set that uses it. The more people that use this account the stronger the unlinkability.
#### Message Repudiation (NO)
> Given a conversation transcript and all cryptographic keys, there is no evidence that a given message was authored by any particular user
- All messages are digitally signed by their sender.
- The underlying transport, Whisper/Waku, does allow for unsigned messages, but we don't use it.
#### Participant Repudiation (NO)
> Given a conversation transcript and all cryptographic key material for all but one accused (honest) participant, there is no evidence that the honest participant was in a conversation with any of the other participants.
### --- Group related features
#### Computational Equality (YES)
> All chat participants share an equal computational load
- One a message is sent, all participants in a group chat perform the same steps to retrieve and decrypt it.
- If proof of work is actually used at the Whisper layer (basically turned off in Status) then the sender would have to do additional computational work to send messages.
#### Trust Equality (PARTIAL)
> No participant is more trusted or takes on more responsibility than any other
- 1:1 chats and public chats are equal
- group chats have admins (on purpose)
- Private Group chats have Administrators and Members. Upon construction, the creator is made an admin. These groups have the following privileges:
- Admins:
- Add group members
- Promote group members to admin
- Change group name
- Members:
- Accept invitation to group
- Leave group
- Non-Members:
- Invited by admins show up as "invited" in group; this leaks contact information
- Invited people don't opt-in to being invited
TODO: Group chat dynamics should have a documented state diagram
TODO: create issues for identity leak of invited members as well as current members of a group showing up who have not accepted yet [Andrea: that's an interesting point, didn't think of that. Currently we have this behavior for 2 reasons, backward compatibility with previous releases, which had no concept of joining, and also because we rely on other peers to propagate group info, so we don't have a single-message point of failure (the invitation), the first can be addressed easily, the second is trickier, without giving up the propagation mechanism (if we choose to give this up, then it's trivial)]
#### Subgroup Messaging (NO)
> Messages can be sent to a subset of participants without forming a new conversation
- This would require a new topic and either a new public chat or a new group chat
[Andrea: This is a YES, as messages are pairwise encrypted, and client-side fanout, so anyone could potentially send a message only to a subset of the group]
#### Contractible Membership (PARTIAL)
> After the conversation begins, participants can leave without restarting the protocol
- For 1:1, there is no way to ignore or block a user from sending you a message. This is currently in the pipeline.
- For public chats, Yes. A member simply stops subscribing to a specific topic and will no longer receive messages.
- For group chats: this assumes pairwise encryption OR key is renegotiated
- This only currently works on the identity level, and not the device level. A ghost device will have access to anything other devices have.
[Andrea: For group chats, that's possible as using pairwise encryption, also with group chats (which use device-to-device encryption), ghost devices is a bit more complicated, in general, they don't have access to the messages you send, i.e. If I send a message from device A1 to the group chat and there is a ghost device A2, it will not be able to decrypt the content, but will see that a message has been sent (as only paired devices are kept in sync, and those are explicitly approved by the user). Messages that you receive are different, so a ghost device (A2) will potentially be able to decrypt the message, but A1 can detect the ghost device (in most cases, it's complicated :), the pfs docs describe multi-device support), for one-to-one ghost devices are undetectable]
#### Expandable Membership (PARTIAL)
> After the conversation begins, participants can join without restarting the protocol.
- 1:1: no, only 1:1
- private group: yes, since it is pair-wise, each person in the group just creates a pair with the new member
- public: yes, as members of a public chat are only subscribing to a topic and receiving anyone sending messages to it.
### --- Usability and Adoption
#### Out-of-Order Resilient (PARTIAL)
> If a message is delayed in transit, but eventually arrives, its contents are accessible upon arrival
- Due to asynchronous forward secrecy and no additional services, private keys might be rotated
[Andrea: That's correct, in some cases if the message is delayed for too long, or really out-of-order, the specific message key might have been deleted, as we only keep the last 3000 message keys]
[Igor: TTL of a Whisper message can expire, so any node-in-transit will drop it. Also, I believe we ignore messages with skewed timestamps]
#### Dropped Message Resilient (PARTIAL)
> Messages can be decrypted without receipt of all previous messages. This is desirable for asynchronous and unreliable network services
- Public chats: yes, users are able to decrypt any message received at any time.
- 1-to-1/group chat also, this is a YES in my opinion
#### Asynchronous (PARTIAL)
> Messages can be sent securely to disconnected recipients and received upon their next connection
- The semantics around message reliability are currently poor
* [Igor: messages are stored on mailservers for way longer than TTL (30 days), but that requires Status infrastructure]
- There's a TTL in Whisper and mailserver can deliver messages after the fact
TODO: this requires more detail
#### Multi-Device Support (YES)
> A user can participate in the conversation using multiple devices at once. Each device must be able to send and receive messages. Ideally, all devices have identical views of the conversation. The devices might use a synchronized long-term key or distinct keys.
- Yes
- There is currently work being done to improve the syncing process between a user's devices.
#### No Additional Service (NO)
> The protocol does not require any infrastructure other than the protocol participants. Specifically, the protocol must not require additional servers for relaying messages or storing any kind of key material.
- The protocol requires Whisper/Waku relay servers and mailservers currently.
- The larger the number of Whisper/Waku relay servers, the better the transport security but there might be potential scaling problems.
- Mailservers act to provide asynchronicity so users can retrieve messages after coming back from an offline period.
-->
## Session management
A node identifies a peer by two pieces of data:
1) An `installation-id` which is generated upon creating a new account in the `Status` application
2) Their identity Whisper/Waku key
### Initialization
A node initializes a new session once a successful X3DH exchange has taken place. Subsequent messages will use the established session until re-keying is necessary.
### Concurrent sessions
If a node creates two sessions concurrently between two peers, the one with the symmetric key first in byte order SHOULD be used, this marks that the other has expired.
### Re-keying
On receiving a bundle from a given peer with a higher version, the old bundle SHOULD be marked as expired and a new session SHOULD be established on the next message sent.
### Multi-device support
Multi-device support is quite challenging as there is not a central place
where information on which and how many devices (identified by their respective `installation-id`) belongs to a whisper-identity / waku-identity.
Furthermore, account recovery always needs to be taken into consideration,
where a user wipes clean the whole device and the nodes loses all the information about any previous sessions.
Taking these considerations into account, the way the network propagates multi-device information using x3dh bundles,
which will contain information about paired devices as well as information about the sending device.
This means that every time a new device is paired, the bundle needs to be updated and propagated with the new information,
the user has the responsibility to make sure the pairing is successful.
The method is loosely based on [Sesame](https://signal.org/docs/specifications/sesame/).
### Pairing
When a user adds a new account in the `Status` application, a new `installation-id` will be generated.
The device should be paired as soon as possible if other devices are present.
Once paired the contacts will be notified of the new device and it will be included in further communications.
If a bundle received from the `IK` is different to the `installation-id`,
the device will be shown to the user and will have to be manually approved, to a maximum of 3.
Once that is done any message sent by one device will also be sent to any other enabled device.
Once a user enables a new device, a new bundle will be generated which will include pairing information.
The bundle will be propagated to contacts through the usual channels.
Removal of paired devices is a manual step that needs to be applied on each device,
and consist simply in disabling the device, at which point pairing information will not be propagated anymore.
### Sending messages to a paired group
When sending a message, the peer will send a message to other `installation-id` that they have seen.
The node caps the number of devices to 3, ordered by last activity.
The node sends messages using pairwise encryption, including their own devices.
Account recovery
Account recovery is no different from adding a new device, and it is handled in exactly the same way.
### Partitioned devices
In some cases (i.e. account recovery when no other pairing device is available, device not paired),
it is possible that a device will receive a message that is not targeted to its own `installation-id`.
In this case an empty message containing bundle information is sent back,
which will notify the receiving end of including this device in any further communication.
## Changelog
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
- Added language to include Waku in all relevant places
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [X3DH](https://signal.org/docs/specifications/x3dh/)
- [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/)
- [Protobuf](https://developers.google.com/protocol-buffers/)
- [Whisper](/status/deprecated/whisper-usage.md)
- [Waku](/status/deprecated/waku-usage.md)
- [Account specification](/status/deprecated/account.md)
- [Status implementation](https://github.com/status-im/doubleratchet/)
- [Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html)
- [X3DH](https://signal.org/docs/specifications/x3dh/)
- [ACCOUNT](/status/deprecated/account.md)
- [Sesame](https://signal.org/docs/specifications/sesame/)
- [May 22, 2020 commit change](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)

View File

@@ -0,0 +1,138 @@
---
title: WAKU-MAILSERVER
name: Waku Mailserver
status: deprecated
description: Waku Mailserver is a specification that allows messages to be stored permanently and to allow the stored messages to be delivered to requesting client nodes, regardless if the messages are not available in the network due to the message TTL expiring.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Babik <adam@status.im>
- Oskar Thorén <oskar@status.im>
- Samuel Hawksby-Robinson <samuel@status.im>
---
## Abstract
Being mostly offline is an intrinsic property of mobile clients.
They need to save network transfer and battery consumption to avoid spending too much money or constant charging.
Waku protocol, on the other hand, is an online protocol.
Messages are available in the Waku network only for short period of time calculate in seconds.
Waku Mailserver is a specification that allows messages to be stored permanently
and allows the stored messages to be delivered to requesting client nodes,
regardless if the messages are not available in the network due to the message TTL expiring.
## `Mailserver`
From the network perspective, a `Mailserver` is just like any other Waku node.
The only difference is that a `Mailserver` has the capability of archiving messages
and delivering them to its peers on-demand.
It is important to notice that a `Mailserver` will only handle requests from its direct peers
and exchanged packets between a `Mailserver` and a peer are p2p messages.
### Archiving messages
A node which wants to provide `Mailserver` functionality MUST store envelopes from
incoming message packets (Waku packet-code `0x01`). The envelopes can be stored in any
format, however they MUST be serialized and deserialized to the Waku envelope format.
A `Mailserver` SHOULD store envelopes for all topics to be generally useful for any peer,
however for specific use cases it MAY store envelopes for a subset of topics.
### Requesting messages
In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing `Mailserver` functionality.
This packet requires one argument which MUST be a Waku envelope.
In the Waku envelope's payload section, there MUST be RLP-encoded information about the details of the request:
```golang
[ Lower, Upper, Bloom, Limit, Cursor ]
```
`Lower`: 4-byte wide unsigned integer (UNIX time in seconds; oldest requested envelope's creation time)
`Upper`: 4-byte wide unsigned integer (UNIX time in seconds; newest requested envelope's creation time)
`Bloom`: 64-byte wide array of Waku topics encoded in a bloom filter to filter envelopes
`Limit`: 4-byte wide unsigned integer limiting the number of returned envelopes
`Cursor`: an array of a cursor returned from the previous request (optional)
The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit`
so that the requester can send another request using the obtained `Cursor` value.
What exactly is in the `Cursor` is up to the implementation.
The requester SHOULD NOT use a `Cursor` obtained from one `Mailserver` in a request to another `Mailserver` because the format or the result MAY be different.
The envelope MUST be encrypted with a symmetric key agreed between the requester and the `Mailserver`.
### Receiving historic messages
Historic messages MUST be sent to a peer as a packet with a P2P Message code (`0x7f`) followed by an array of Waku envelopes.
In order to receive historic messages from a `Mailserver`, a node MUST trust the selected `Mailserver`,
that is allowed to send packets with the P2P Message code. By default, the node discards such packets.
Received envelopes MUST be passed through the Waku envelope pipelines
so that they are picked up by registered filters and passed to subscribers.
For a requester, to know that all messages have been sent by a `Mailserver`,
it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by the following parameters:
```golang
[ RequestID, LastEnvelopeHash, Cursor ]
```
* `RequestID`: 32-byte wide array with a Keccak-256 hash of the envelope containing the original request
* `LastEnvelopeHash`: 32-byte wide array with a Keccak-256 hash of the last sent envelope for the request
* `Cursor`: an array of a cursor returned from the previous request (optional)
If `Cursor` is not empty, it means that not all messages were sent due to the set `Limit` in the request.
One or more consecutive requests MAY be sent with `Cursor` field filled in order to receive the rest of the messages.
## Security considerations
### Confidentiality
The node encrypts all Waku envelopes. A `Mailserver` node can not inspect their contents.
### Altruistic and centralized operator risk
In order to be useful, a `Mailserver` SHOULD be online most of time.
That means users either have to be a bit tech-savvy to run their own node,
or rely on someone else to run it for them.
Currently, one of Status's legal entities provides `Mailservers` in an altruistic manner,
but this is suboptimal from a decentralization, continuance and risk point of view.
Coming up with a better system for this is ongoing research.
A Status client SHOULD allow the `Mailserver` selection to be customizable.
### Privacy concerns
In order to use a `Mailserver`, a given node needs to connect to it directly,
i.e. add the `Mailserver` as its peer and mark it as trusted.
This means that the `Mailserver` is able to send direct p2p messages to the node instead of broadcasting them.
Effectively, it will have access to the bloom filter of topics that the user is interested in,
when it is online as well as many metadata like IP address.
### Denial-of-service
Since a `Mailserver` is delivering expired envelopes and has a direct TCP connection with the recipient,
the recipient is vulnerable to DoS attacks from a malicious `Mailserver` node.
## Changelog
### Version 0.1
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
* Created document
* Forked from [whisper-mailserver](/status/deprecated/whisper-mailserver.md)
* Change to keep `Mailserver` term consistent
* Replaced Whisper references with Waku
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
* [May 22, 2020 change commit](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)

View File

@@ -0,0 +1,392 @@
---
title: WAKU-USAGE
name: Waku Usage
status: deprecated
description: Status uses Waku to provide privacy-preserving routing and messaging on top of devP2P.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Babik <adam@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskar@status.im>
- Samuel Hawksby-Robinson <samuel@status.im>
---
## Abstract
Status uses [Waku](/waku/standards/legacy/6/waku1.md) to provide privacy-preserving routing
and messaging on top of devP2P.
Waku uses topics to partition its messages,
and these are leveraged for all chat capabilities.
In the case of public chats, the channel name maps directly to its Waku topic.
This allows anyone to listen on a single channel.
Additionally, since anyone can receive Waku envelopes,
it relies on the ability to decrypt messages to decide who is the correct recipient.
Status nodes do not rely upon this property,
and implement another secure transport layer on top of Whisper.
## Reason
Provide routing, metadata protection, topic-based multicasting and basic
encryption properties to support asynchronous chat.
## Terminology
* *Waku node*: an Ethereum node with Waku V1 enabled
* *Waku network*: a group of Waku nodes connected together through the internet connection and forming a graph
* *Message*: a decrypted Waku message
* *Offline message*: an archived envelope
* *Envelope*: an encrypted message with metadata like topic and Time-To-Live
## Waku packets
| Packet Name | Code | References |
| -------------------- | ---: | --- |
| Status | 0 | [Status](status), [WAKU-1](/waku/standards/legacy/6/waku1.md#status) |
| Messages | 1 | [WAKU-1](/waku/standards/legacy/6/waku1.md#messages) |
| Batch Ack | 11 | Undocumented. Marked for Deprecation |
| Message Response | 12 | [WAKU-1](/waku/standards/legacy/6/waku1.md#batch-ack-and-message-response) |
| Status Update | 22 | [WAKU-1](/waku/standards/legacy/6/waku1.md#status-update) |
| P2P Request Complete | 125 | [4/WAKU-MAILSERVER](/status/deprecated/waku-mailserver.md) |
| P2P Request | 126 | [4/WAKU-MAILSERVER](/status/deprecated/waku-mailserver.md), [WAKU-1](/waku/standards/legacy/6/waku1.md#p2p-request) |
| P2P Messages | 127 | [4/WAKU-MAILSERVER](/status/deprecated/waku-mailserver.md), [WAKU-1](/waku/standards/legacy/6/waku1.md#p2p-request-complete) |
## Waku node configuration
A Waku node must be properly configured to receive messages from Status clients.
Nodes use Waku's Proof Of Work algorithm to deter denial of service and various spam/flood attacks against the Whisper network.
The sender of a message must perform some work which in this case means processing time.
Because Status' main client is a mobile client, this easily leads to battery draining and poor performance of the app itself.
Hence, all clients MUST use the following Whisper node settings:
* proof-of-work requirement not larger than `0.002` for payloads less than 50,000 bytes
* proof-of-work requirement not larger than `0.000002` for payloads greater than or equal to 50,000 bytes
* time-to-live not lower than `10` (in seconds)
## Status
Handshake is a RLP-encoded packet sent to a newly connected peer. It MUST start with a Status Code (`0x00`) and follow up with items:
```golang
[
[ pow-requirement-key pow-requirement ]
[ bloom-filter-key bloom-filter ]
[ light-node-key light-node ]
[ confirmations-enabled-key confirmations-enabled ]
[ rate-limits-key rate-limits ]
[ topic-interest-key topic-interest ]
]
```
| Option Name | Key | Type | Description | References |
| ----------------------- | ------ | -------- | ----------- | --- |
| `pow-requirement` | `0x00` | `uint64` | minimum PoW accepted by the peer | [WAKU-1#pow-requirement](/waku/standards/legacy/6/waku1.md#pow-requirement-field) |
| `bloom-filter` | `0x01` | `[]byte` | bloom filter of Waku topic accepted by the peer | [WAKU-1#bloom-filter](/waku/standards/legacy/6/waku1.md#bloom-filter-field) |
| `light-node` | `0x02` | `bool` | when true, the peer won't forward envelopes through the Messages packet. | [WAKU-1#light-node](/waku/standards/legacy/6/waku1.md#light-node) |
| `confirmations-enabled` | `0x03` | `bool` | when true, the peer will send message confirmations | [WAKU-1#confirmations-enabled-field](/waku/standards/legacy/6/waku1.md#confirmations-enabled-field) |
| `rate-limits` | `0x04` | | See [Rate limiting](/waku/standards/legacy/6/waku1.md#rate-limits-field) | [WAKU-1#rate-limits](/waku/standards/legacy/6/waku1.md#rate-limits-field) |
| `topic-interest` | `0x05` | `[10000][4]byte` | Topic interest is used to share a node's interest in envelopes with specific topics. It does this in a more bandwidth considerate way, at the expense of some metadata protection. Peers MUST only send envelopes with specified topics. | [WAKU-1#topic-interest](/waku/standards/legacy/6/waku1.md#topic-interest-field), [the theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability) |
<!-- TODO Add `light-node` and `confirmations-enabled` links when https://github.com/vacp2p/specs/pull/128 is merged -->
## Rate limiting
In order to provide an optional very basic Denial-of-Service attack protection, each node SHOULD define its own rate limits.
The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
Each node MAY decide to whitelist, i.e. do not rate limit, selected IPs or peer IDs.
If a peer exceeds node's rate limits, the connection between them MAY be dropped.
Each node SHOULD broadcast its rate limits to its peers using `rate limits` in `status-options` via packet code `0x00` or `0x22`. The rate limits is RLP-encoded information:
```golang
[ IP limits, PeerID limits, Topic limits ]
```
`IP limits`: 4-byte wide unsigned integer
`PeerID limits`: 4-byte wide unsigned integer
`Topic limits`: 4-byte wide unsigned integer
The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD respect rate limits advertised by its peers. The number of packets SHOULD be throttled in order not to exceed peer's rate limits.
If the limit gets exceeded, the connection MAY be dropped by the peer.
## Keys management
The protocol requires a key (symmetric or asymmetric) for the following actions:
* signing & verifying messages (asymmetric key)
* encrypting & decrypting messages (asymmetric or symmetric key).
As nodes require asymmetric keys and symmetric keys to process incoming messages,
they must be available all the time and are stored in memory.
Keys management for PFS is described in [5/SECURE-TRANSPORT](/status/deprecated/secure-transport.md).
The Status protocols uses a few particular Waku topics to achieve its goals.
### Contact code topic
Nodes use the contact code topic to facilitate the discovery of X3DH bundles so that the first message can be PFS-encrypted.
Each user publishes periodically to this topic. If user A wants to contact user B, she SHOULD look for their bundle on this contact code topic.
Contact code topic MUST be created following the algorithm below:
```golang
contactCode := "0x" + hexEncode(activePublicKey) + "-contact-code"
var hash []byte = keccak256(contactCode)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Partitioned topic
Waku is broadcast-based protocol. In theory, everyone could communicate using a single topic but that would be extremely inefficient.
Opposite would be using a unique topic for each conversation, however,
this brings privacy concerns because it would be much easier to detect whether and when two parties have an active conversation.
Nodes use partitioned topics to broadcast private messages efficiently.
By selecting a number of topic, it is possible to balance efficiency and privacy.
Currently, nodes set the number of partitioned topics to `5000`. They MUST be generated following the algorithm below:
```golang
var partitionsNum *big.Int = big.NewInt(5000)
var partition *big.Int = big.NewInt(0).Mod(publicKey.X, partitionsNum)
partitionTopic := "contact-discovery-" + strconv.FormatInt(partition.Int64(), 10)
var hash []byte = keccak256(partitionTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Public chats
A public chat MUST use a topic derived from a public chat name following the algorithm below:
```golang
var hash []byte
hash = keccak256(name)
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
<!-- NOTE: commented out as it is currently not used. In code for potential future use. - C.P. Oct 8, 2019
### Personal discovery topic
Personal discovery topic is used to ???
A client MUST implement it following the algorithm below:
```golang
personalDiscoveryTopic := "contact-discovery-" + hexEncode(publicKey)
var hash []byte = keccak256(personalDiscoveryTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
Each Status Client SHOULD listen to this topic in order to receive ??? -->
### Group chat topic
Group chats does not have a dedicated topic.
All group chat messages (including membership updates) are sent as one-to-one messages to multiple recipients.
### Negotiated topic
When a client sends a one to one message to another client, it MUST listen to their negotiated topic.
This is computed by generating a diffie-hellman key exchange between two members
and taking the first four bytes of the `SHA3-256` of the key generated.
```golang
sharedKey, err := ecies.ImportECDSA(myPrivateKey).GenerateShared(
ecies.ImportECDSAPublic(theirPublicKey),
16,
16,
)
hexEncodedKey := hex.EncodeToString(sharedKey)
var hash []byte = keccak256(hexEncodedKey)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
A client SHOULD send to the negotiated topic only if it has received a message from all the devices included in the conversation.
### Flow
To exchange messages with client `B`, a client `A` SHOULD:
* Listen to client's `B` Contact Code Topic to retrieve their bundle information, including a list of active devices
* Send a message on client's `B` partitioned topic
* Listen to the Negotiated Topic between `A` & `B`
* Once client `A` receives a message from `B`, the Negotiated Topic SHOULD be used
## Message encryption
Even though, the protocol specifies an encryption layer that encrypts messages before passing them to the transport layer,
Waku protocol requires each Waku message to be encrypted anyway.
The node encrypts public and group messages using symmetric encryption, and creates the key from a channel name string.
The implementation is available in [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) JSON-RPC method of go-ethereum Whisper implementation.
The node encrypts one-to-one messages using asymmetric encryption.
## Message confirmations
Sending a message is a complex process where many things can go wrong.
Message confirmations tell a node that a message originating from it has been seen by its direct peers.
A node MAY send a message confirmation for any batch of messages received in a packet Messages Code (`0x01`).
A node sends a message confirmation using Batch Acknowledge packet (`0x0b`) or Message Response packet (`0x0c`).
The Batch Acknowledge packet is followed by a keccak256 hash of the envelopes batch data (raw bytes).
The Message Response packet is more complex and is followed by a Versioned Message Response:
```golang
[ Version, Response]
```
`Version`: a version of the Message Response, equal to `1`,
`Response`: `[ Hash, Errors ]` where `Hash` is a keccak256 hash of the envelopes batch data (raw bytes)
for which the confirmation is sent and `Errors` is a list of envelope errors when processing the batch.
A single error contains `[ Hash, Code, Description ]` where `Hash` is a hash of the processed envelope,
`Code` is an error code and `Description` is a descriptive error message.
The supported codes:
`1`: means time sync error which happens when an envelope is too old
or created in the future (the root cause is no time sync between nodes).
The drawback of sending message confirmations is that it increases the noise in the network because for each sent message,
one or more peers broadcast a corresponding confirmation. To limit that, both Batch Acknowledge packet (`0x0b`)
and Message Response packet (`0x0c`) are not broadcast to peers of the peers, i.e. they do not follow epidemic spread.
In the current Status network setup, only `Mailservers` support message confirmations.
A client posting a message to the network and after receiving a confirmation can be sure that the message got processed by the `Mailserver`.
If additionally, sending a message is limited to non-`Mailserver` peers,
it also guarantees that the message got broadcast through the network and it reached the selected `Mailserver`.
## Waku V1 extensions
### Request historic messages
Sends a request for historic messages to a `Mailserver`.
The `Mailserver` node MUST be a direct peer and MUST be marked as trusted (using `waku_markTrustedPeer`).
The request does not wait for the response.
It merely sends a peer-to-peer message to the `Mailserver` and it's up to `Mailserver` to process it and start sending historic messages.
The drawback of this approach is that it is impossible to tell which historic messages are the result of which request.
It's recommended to return messages from newest to oldest.
To move further back in time, use `cursor` and `limit`.
#### wakuext_requestMessages
**Parameters**:
* Object - The message request object:
* `mailServerPeer` - `String`: `Mailserver`'s enode address.
* `from` - `Number` (optional): Lower bound of time range as unix timestamp, default is 24 hours back from now.
* `to` - `Number` (optional): Upper bound of time range as unix timestamp, default is now.
* `limit` - `Number` (optional): Limit the number of messages sent back, default is no limit.
* `cursor` - `String` (optional): Used for paginated requests.
* `topics` - `Array`: hex-encoded message topics.
* `symKeyID` - `String`: an ID of a symmetric key used to authenticate with the `Mailserver`, derived from the `Mailserver` password.
**Returns**:
`Boolean` - returns `true` if the request was sent.
The above `topics` is then converted into a bloom filter and then and sent to the `Mailserver`.
<!-- TODO: Clarify actual request with bloom filter to mailserver -->
## Changelog
### Version 0.1
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
* Created document
* Forked from [3-whisper-usage](3-whisper-usage.md)
* Change to keep `Mailserver` term consistent
* Replaced Whisper references with Waku
* Added [Status options](#status) section
* Updated [Waku packets](#waku-packets) section to match Waku
* Added that `Batch Ack` is marked for deprecation
* Changed `shh_generateSymKeyFromPassword` to `waku_generateSymKeyFromPassword`
* [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/waku/api.go#L172-L175)
* [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/eth-node/bridge/geth/public_waku_api.go#L33-L36)
* Changed `shh_markTrustedPeer` to `waku_markTrustedPeer`
* [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/waku/api.go#L100-L108)
* Changed `shhext_requestMessages` to `wakuext_requestMessages`
* [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/services/wakuext/api.go#L76-L139)
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
* [Waku](waku)
* [WAKU1](/waku/standards/legacy/6/waku1.md)
* [WAKU-MAILSERVER](/status/deprecated/waku-mailserver.md)
* [The theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability)
* [SECURE-TRANSPORT](/status/deprecated/secure-transport.md)
* [May 22, 2020 commit](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
* [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword)
* [Key Change #1](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/waku/api.go#L172-L175)
* [Key Change #2](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/eth-node/bridge/geth/public_waku_api.go#L33-L36)
* [Key Change #3](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/waku/api.go#L100-L108)
* [Key Change #4](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/services/wakuext/api.go#L76-L139)

View File

@@ -0,0 +1,148 @@
---
title: WHISPER-MAILSERVER
name: Whisper mailserver
status: deprecated
description: Whisper Mailserver is a Whisper extension that allows to store messages permanently and deliver them to the clients even though they are already not available in the network and expired.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Babik <adam@status.im>
- Oskar Thorén <oskar@status.im>
---
## Abstract
Being mostly offline is an intrinsic property of mobile clients.
They need to save network transfer and battery consumption
to avoid spending too much money or constant charging.
Whisper protocol, on the other hand, is an online protocol.
Messages are available in the Whisper network only for short period of time calculate in seconds.
Whisper `Mailserver` is a Whisper extension that allows to store messages permanently
and deliver them to the clients even though they are already not available in the network and expired.
## `Mailserver`
From the network perspective, `Mailserver` is just like any other Whisper node.
The only difference is that it has a capability of archiving messages and delivering them to its peers on-demand.
It is important to notice that `Mailserver` will only handle requests from its direct peers
and exchanged packets between `Mailserver` and a peer are p2p messages.
### Archiving messages
A node which wants to provide `Mailserver` functionality MUST store envelopes
from incoming message packets (Whisper packet-code `0x01`).
The envelopes can be stored in any format,
however they MUST be serialized and deserialized to the Whisper envelope format.
A `Mailserver` SHOULD store envelopes for all topics to be generally useful for any peer,
however for specific use cases it MAY store envelopes for a subset of topics.
### Requesting messages
In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing `Mailserver` functionality.
This packet requires one argument which MUST be a Whisper envelope.
In the Whisper envelope's payload section, there MUST be RLP-encoded information about the details of the request:
```golang
[ Lower, Upper, Bloom, Limit, Cursor ]
```
`Lower`: 4-byte wide unsigned integer (UNIX time in seconds; oldest requested envelope's creation time)
`Upper`: 4-byte wide unsigned integer (UNIX time in seconds; newest requested envelope's creation time)
`Bloom`: 64-byte wide array of Whisper topics encoded in a bloom filter to filter envelopes
`Limit`: 4-byte wide unsigned integer limiting the number of returned envelopes
`Cursor`: an array of a cursor returned from the previous request (optional)
The `Cursor` field SHOULD be filled in
if a number of envelopes between `Lower` and `Upper` is greater than `Limit`
so that the requester can send another request using the obtained `Cursor` value.
What exactly is in the `Cursor` is up to the implementation.
The requester SHOULD NOT use a `Cursor` obtained from one `Mailserver` in a request to another `Mailserver`
because the format or the result MAY be different.
The envelope MUST be encrypted with a symmetric key agreed between the requester and `Mailserver`.
### Receiving historic messages
Historic messages MUST be sent to a peer as a packet with a P2P Message code (`0x7f`)
followed by an array of Whisper envelopes.
It is incompatible with the original Whisper spec (EIP-627) because it allows only a single envelope,
however, an array of envelopes is much more performant.
In order to stay compatible with EIP-627, a peer receiving historic message MUST handle both cases.
In order to receive historic messages from a `Mailserver`, a node MUST trust the selected `Mailserver`,
that is allowed to send packets with the P2P Message code. By default, the node discards such packets.
Received envelopes MUST be passed through the Whisper envelope pipelines
so that they are picked up by registered filters and passed to subscribers.
For a requester, to know that all messages have been sent by `Mailserver`,
it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by the following parameters:
```golang
[ RequestID, LastEnvelopeHash, Cursor ]
```
`RequestID`: 32-byte wide array with a Keccak-256 hash of the envelope containing the original request
`LastEnvelopeHash`: 32-byte wide array with a Keccak-256 hash of the last sent envelope for the request
`Cursor`: an array of a cursor returned from the previous request (optional)
If `Cursor` is not empty, it means that not all messages were sent due to the set `Limit` in the request.
One or more consecutive requests MAY be sent with `Cursor` field filled in order to receive the rest of the messages.
## Security considerations
### Confidentiality
The node encrypts all Whisper envelopes. A `Mailserver` node can not inspect their contents.
### Altruistic and centralized operator risk
In order to be useful, a `Mailserver` SHOULD be online most of the time. That means
users either have to be a bit tech-savvy to run their own node, or rely on someone
else to run it for them.
Currently, one of Status's legal entities provides `Mailservers` in an altruistic manner, but this is
suboptimal from a decentralization, continuance and risk point of view. Coming
up with a better system for this is ongoing research.
A Status client SHOULD allow the `Mailserver` selection to be customizable.
### Privacy concerns
In order to use a `Mailserver`, a given node needs to connect to it directly,
i.e. add the `Mailserver` as its peer and mark it as trusted.
This means that the `Mailserver` is able to send direct p2p messages to the node instead of broadcasting them.
Effectively, it will have access to the bloom filter of topics
that the user is interested in,
when it is online as well as many metadata like IP address.
### Denial-of-service
Since a `Mailserver` is delivering expired envelopes and has a direct TCP connection with the recipient,
the recipient is vulnerable to DoS attacks from a malicious `Mailserver` node.
## Changelog
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
- Change to keep `Mailserver` term consistent
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [Whisper](https://eips.ethereum.org/EIPS/eip-627)
- [EIP-627](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-627.md)
- [SECURE-TRANSPORT](/status/deprecated/secure-transport.md)
- [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword)
- [Whisper v6](https://eips.ethereum.org/EIPS/eip-627)
- [Waku V0](/waku/deprecated/5/waku0.md)
- [Waku V1](/waku/standards/legacy/6/waku1.md)
- [May 22, 2020 change commit](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)

View File

@@ -0,0 +1,401 @@
---
title: WHISPER-USAGE
name: Whisper Usage
status: deprecated
description: Status uses Whisper to provide privacy-preserving routing and messaging on top of devP2P.
editor: Filip Dimitrijevic <filip@status.im>
contributors:
- Adam Babik <adam@status.im>
- Andrea Piana <andreap@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskar@status.im>
---
## Abstract
Status uses [Whisper](https://eips.ethereum.org/EIPS/eip-627) to provide
privacy-preserving routing and messaging on top of devP2P.
Whisper uses topics to partition its messages,
and these are leveraged for all chat capabilities.
In the case of public chats, the channel name maps directly to its Whisper topic.
This allows anyone to listen on a single channel.
Additionally, since anyone can receive Whisper envelopes,
it relies on the ability to decrypt messages to decide who is the correct recipient.
Status nodes do not rely upon this property,
and implement another secure transport layer on top of Whisper.
Finally, using an extension of Whisper provides the ability to do offline messaging.
## Reason
Provide routing, metadata protection, topic-based multicasting and basic
encryption properties to support asynchronous chat.
## Terminology
* *Whisper node*: an Ethereum node with Whisper V6 enabled (in the case of go-ethereum, it's `--shh` option)
* *Whisper network*: a group of Whisper nodes connected together through the internet connection and forming a graph
* *Message*: a decrypted Whisper message
* *Offline message*: an archived envelope
* *Envelope*: an encrypted message with metadata like topic and Time-To-Live
## Whisper packets
| Packet Name | Code | EIP-627 | References |
| --- | --: | --- | --- |
| Status | 0 | ✔ | [Handshake](#handshake) |
| Messages | 1 | ✔ | [EIP-627](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-627.md) |
| PoW Requirement | 2 | ✔ | [EIP-627](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-627.md) |
| Bloom Filter | 3 | ✔ | [EIP-627](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-627.md) |
| Batch Ack | 11 | 𝘅 | Undocumented |
| Message Response | 12 | 𝘅 | Undocumented |
| P2P Sync Request | 123 | 𝘅 | Undocumented |
| P2P Sync Response | 124 | 𝘅 | Undocumented |
| P2P Request Complete | 125 | 𝘅 | [4/WHISPER-MAILSERVER](/status/deprecated/whisper-mailserver.md) |
| P2P Request | 126 | ✔ | [4/WHISPER-MAILSERVER](/status/deprecated/whisper-mailserver.md) |
| P2P Messages | 127 | ✔/𝘅 (EIP-627 supports only single envelope in a packet) | [4/WHISPER-MAILSERVER](/status/deprecated/whisper-mailserver.md) |
## Whisper node configuration
A Whisper node must be properly configured to receive messages from Status clients.
Nodes use Whisper's Proof Of Work algorithm to deter denial of service
and various spam/flood attacks against the Whisper network.
The sender of a message must perform some work which in this case means processing time.
Because Status' main client is a mobile client, this easily leads to battery draining and poor performance of the app itself.
Hence, all clients MUST use the following Whisper node settings:
* proof-of-work requirement not larger than `0.002`
* time-to-live not lower than `10` (in seconds)
## Handshake
Handshake is a RLP-encoded packet sent to a newly connected peer. It MUST start with a Status Code (`0x00`) and follow up with items:
```golang
[ protocolVersion, PoW, bloom, isLightNode, confirmationsEnabled, rateLimits ]
```
`protocolVersion`: version of the Whisper protocol
`PoW`: minimum PoW accepted by the peer
`bloom`: bloom filter of Whisper topic accepted by the peer
`isLightNode`: when true, the peer won't forward messages
`confirmationsEnabled`: when true, the peer will send message confirmations
`rateLimits`: is `[ RateLimitIP, RateLimitPeerID, RateLimitTopic ]` where each values is an integer with a number of accepted packets per second per IP, Peer ID, and Topic respectively
`bloom, isLightNode, confirmationsEnabled, and rateLimits` are all optional arguments in the handshake. However, if an optional field is specified, all optional fields preceding it MUST also be specified in order to be unambiguous.
## Rate limiting
In order to provide an optional very basic Denial-of-Service attack protection, each node SHOULD define its own rate limits.
The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
Each node MAY decide to whitelist, i.e. do not rate limit, selected IPs or peer IDs.
If a peer exceeds node's rate limits, the connection between them MAY be dropped.
Each node SHOULD broadcast its rate limits to its peers using rate limits packet code (`0x14`). The rate limits is RLP-encoded information:
```golang
[ IP limits, PeerID limits, Topic limits ]
```
`IP limits`: 4-byte wide unsigned integer
`PeerID limits`: 4-byte wide unsigned integer
`Topic limits`: 4-byte wide unsigned integer
The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD respect rate limits advertised by its peers.
The number of packets SHOULD be throttled in order not to exceed peer's rate limits.
If the limit gets exceeded, the connection MAY be dropped by the peer.
## Keys management
The protocol requires a key (symmetric or asymmetric) for the following actions:
* signing & verifying messages (asymmetric key)
* encrypting & decrypting messages (asymmetric or symmetric key).
As nodes require asymmetric keys and symmetric keys to process incoming messages,
they must be available all the time and are stored in memory.
Keys management for PFS is described in [5/SECURE-TRANSPORT](/status/deprecated/whisper-mailserver.md).
The Status protocols uses a few particular Whisper topics to achieve its goals.
### Contact code topic
Nodes use the contact code topic to facilitate the discovery of X3DH bundles so that the first message can be PFS-encrypted.
Each user publishes periodically to this topic.
If user A wants to contact user B, she SHOULD look for their bundle on this contact code topic.
Contact code topic MUST be created following the algorithm below:
```golang
contactCode := "0x" + hexEncode(activePublicKey) + "-contact-code"
var hash []byte = keccak256(contactCode)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Partitioned topic
Whisper is broadcast-based protocol.
In theory, everyone could communicate using a single topic but that would be extremely inefficient.
Opposite would be using a unique topic for each conversation,
however, this brings privacy concerns because it would be much easier to detect whether
and when two parties have an active conversation.
Nodes use partitioned topics to broadcast private messages efficiently.
By selecting a number of topic, it is possible to balance efficiency and privacy.
Currently, nodes set the number of partitioned topics to `5000`.
They MUST be generated following the algorithm below:
```golang
var partitionsNum *big.Int = big.NewInt(5000)
var partition *big.Int = big.NewInt(0).Mod(publicKey.X, partitionsNum)
partitionTopic := "contact-discovery-" + strconv.FormatInt(partition.Int64(), 10)
var hash []byte = keccak256(partitionTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Public chats
A public chat MUST use a topic derived from a public chat name following the algorithm below:
```golang
var hash []byte
hash = keccak256(name)
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
<!-- NOTE: commented out as it is currently not used. In code for potential future use. - C.P. Oct 8, 2019
### Personal discovery topic
Personal discovery topic is used to ???
A client MUST implement it following the algorithm below:
```golang
personalDiscoveryTopic := "contact-discovery-" + hexEncode(publicKey)
var hash []byte = keccak256(personalDiscoveryTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
Each Status Client SHOULD listen to this topic in order to receive ??? -->
<!-- NOTE: commented out as it is no longer valid as of V1. - C.P. Oct 8, 2019
### Generic discovery topic
Generic discovery topic is a legacy topic used to handle all one-to-one chats. The newer implementation should rely on [Partitioned Topic](#partitioned-topic) and [Personal discovery topic](#personal-discovery-topic).
Generic discovery topic MUST be created following [Public chats](#public-chats) topic algorithm using string `contact-discovery` as a name. -->
### Group chat topic
Group chats does not have a dedicated topic.
All group chat messages (including membership updates) are sent as one-to-one messages to multiple recipients.
### Negotiated topic
When a client sends a one to one message to another client, it MUST listen to their negotiated topic.
This is computed by generating a diffie-hellman key exchange between two members
and taking the first four bytes of the `SHA3-256` of the key generated.
```golang
sharedKey, err := ecies.ImportECDSA(myPrivateKey).GenerateShared(
ecies.ImportECDSAPublic(theirPublicKey),
16,
16,
)
hexEncodedKey := hex.EncodeToString(sharedKey)
var hash []byte = keccak256(hexEncodedKey)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
A client SHOULD send to the negotiated topic only if it has received a message from all the devices included in the conversation.
### Flow
To exchange messages with client `B`, a client `A` SHOULD:
* Listen to client's `B` Contact Code Topic to retrieve their bundle information, including a list of active devices
* Send a message on client's `B` partitioned topic
* Listen to the Negotiated Topic between `A` & `B`
* Once client `A` receives a message from `B`, the Negotiated Topic SHOULD be used
## Message encryption
Even though, the protocol specifies an encryption layer that encrypts messages before passing them to the transport layer,
Whisper protocol requires each Whisper message to be encrypted anyway.
The node encrypts public and group messages using symmetric encryption, and creates the key from a channel name string.
The implementation is available in [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) JSON-RPC method of go-ethereum Whisper implementation.
The node encrypts one-to-one messages using asymmetric encryption.
## Message confirmations
Sending a message is a complex process where many things can go wrong.
Message confirmations tell a node that a message originating from it has been seen by its direct peers.
A node MAY send a message confirmation for any batch of messages received in a packet Messages Code (`0x01`).
A node sends a message confirmation using Batch Acknowledge packet (`0x0b`) or Message Response packet (`0x0c`).
The Batch Acknowledge packet is followed by a keccak256 hash of the envelopes batch data (raw bytes).
The Message Response packet is more complex and is followed by a Versioned Message Response:
```golang
[ Version, Response]
```
`Version`: a version of the Message Response, equal to `1`,
`Response`: `[ Hash, Errors ]` where `Hash` is a keccak256 hash of the envelopes batch data (raw bytes)
for which the confirmation is sent and `Errors` is a list of envelope errors when processing the batch.
A single error contains `[ Hash, Code, Description ]` where `Hash` is a hash of the processed envelope,
`Code` is an error code and `Description` is a descriptive error message.
The supported codes:
`1`: means time sync error which happens when an envelope is too old
or created in the future (the root cause is no time sync between nodes).
The drawback of sending message confirmations is that it increases the noise in the network because for each sent message,
one or more peers broadcast a corresponding confirmation.
To limit that, both Batch Acknowledge packet (`0x0b`) and Message Response packet (`0x0c`) are not broadcast to peers of the peers,
i.e. they do not follow epidemic spread.
In the current Status network setup, only `Mailservers` support message confirmations.
A client posting a message to the network and after receiving a confirmation can be sure that the message got processed by the `Mailserver`.
If additionally, sending a message is limited to non-`Mailserver` peers,
it also guarantees that the message got broadcast through the network and it reached the selected `Mailserver`.
## Whisper / Waku bridging
In order to maintain compatibility between Whisper and Waku nodes,
a Status network that implements both Whisper and Waku messaging protocols
MUST have at least one node that is capable of discovering peers and implements
[Whisper v6](https://eips.ethereum.org/EIPS/eip-627),
[Waku V0](/waku/deprecated/5/waku0.md) and
[Waku V1](/waku/standards/legacy/6/waku1.md) specifications.
Additionally, any Status network that implements both Whisper and Waku messaging protocols
MUST implement bridging capabilities as detailed in
[Waku V1#Bridging](/waku/standards/legacy/6/waku1.md#waku-whisper-bridging).
## Whisper V6 extensions
### Request historic messages
Sends a request for historic messages to a `Mailserver`.
The `Mailserver` node MUST be a direct peer and MUST be marked as trusted (using `shh_markTrustedPeer`).
The request does not wait for the response.
It merely sends a peer-to-peer message to the `Mailserver`
and it's up to `Mailserver` to process it and start sending historic messages.
The drawback of this approach is that it is impossible to tell
which historic messages are the result of which request.
It's recommended to return messages from newest to oldest.
To move further back in time, use `cursor` and `limit`.
#### shhext_requestMessages
**Parameters**:
1. Object - The message request object:
* `mailServerPeer` - `String`: `Mailserver`'s enode address.
* `from` - `Number` (optional): Lower bound of time range as unix timestamp, default is 24 hours back from now.
* `to` - `Number` (optional): Upper bound of time range as unix timestamp, default is now.
* `limit` - `Number` (optional): Limit the number of messages sent back, default is no limit.
* `cursor` - `String` (optional): Used for paginated requests.
* `topics` - `Array`: hex-encoded message topics.
* `symKeyID` - `String`: an ID of a symmetric key used to authenticate with the `Mailserver`, derived from Mailserver password.
**Returns**:
`Boolean` - returns `true` if the request was sent.
The above `topics` is then converted into a bloom filter and then and sent to the `Mailserver`.
<!-- TODO: Clarify actual request with bloom filter to mailserver -->
## Changelog
### Version 0.3
Released [May 22, 2020](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)
* Added Whisper / Waku Bridging section
* Change to keep `Mailserver` term consistent
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
* [Whisper](https://eips.ethereum.org/EIPS/eip-627)
* [WHISPER-MAILSERVER](/status/deprecated/whisper-mailserver.md)
* [SECURE-TRANSPORT](/status/deprecated/secure-transport.md)
* [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword)
* [Whisper v6](https://eips.ethereum.org/EIPS/eip-627)
* [Waku V0](/waku/deprecated/5/waku0.md)
* [Waku V1](/waku/standards/legacy/6/waku1.md)
* [May 22, 2020 change commit](https://github.com/status-im/specs/commit/664dd1c9df6ad409e4c007fefc8c8945b8d324e8)

View File

@@ -1,6 +1,5 @@
---
title: 57/STATUS-Simple-Scaling
title: STATUS-SIMPLE-SCALING
name: Status Simple Scaling
status: raw
category: Informational
@@ -13,48 +12,68 @@ contributors:
## Abstract
This document describes how to scale [56/STATUS-COMMUNITIES](../56/communities.md) as well as [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
This document describes how to scale
[56/STATUS-COMMUNITIES](../56/communities.md) as well as [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
using Waku v2 protocol and components.
It also adds a few new aspects, where more sophisticated components are not yet researched and evaluated.
It also adds a few new aspects,
where more sophisticated components are not yet researched and evaluated.
> *Note:* (Parts of) this RFC will be deprecated in the future as we continue research to scale specific components
in a way that aligns better with our principles of decentralization and protecting anonymity.
This document informs about scaling at the current stage of research and shows it is practically possible.
> *Note:* (Parts of) this RFC will be deprecated in the future
as we continue research to scale specific components
in a way that aligns better with our principles of decentralization and
protecting anonymity.
This document informs about scaling at the current stage of research and
shows it is practically possible.
Practical feasibility is also a core goal for us.
We believe in incremental improvement, i.e. having a working decentralized scaling solution with trade-offs is better than a fully centralized solution.
We believe in incremental improvement, i.e.
having a working decentralized scaling solution with trade-offs
is better than a fully centralized solution.
## Background and Motivation
[56/STATUS-COMMUNITIES](../56/communities.md) as well as [55/STATUS-1TO1-CHAT](../55/1to1-chat.md) use Waku v2 protocols.
Both use Waku content topics (see [23/WAKU2-TOPICS](../../waku/informational/23/topics.md)) for content based filtering.
[56/STATUS-COMMUNITIES](../56/communities.md) as well as
[55/STATUS-1TO1-CHAT](../55/1to1-chat.md) use Waku v2 protocols.
Both use Waku content topics
(see [23/WAKU2-TOPICS](../../waku/informational/23/topics.md))
for content based filtering.
Waku v2 currently has scaling limitations in two dimensions:
1) Messages that are part of a specific content topic have to be disseminated in a single mesh network (i.e. pubsub topic).
1) Messages that are part of a specific content topic
have to be disseminated in a single mesh network (i.e. pubsub topic).
This limits scaling the number of messages disseminated in a specific content topic,
and by extension, the number of active nodes that are part of this content topic.
2) Scaling a large set of content topics requires distributing these over several mesh networks (which this document refers to as pubsub topic shards).
2) Scaling a large set of content topics requires distributing these over several
mesh networks (which this document refers to as pubsub topic shards).
This document focuses on the second scaling dimension.
With the scaling solutions discussed in this document,
each content topics can have a large set of active users, but still has to fit in a single pubsub mesh.
each content topics can have a large set of active users,
but still has to fit in a single pubsub mesh.
> *Note:* While it is possible to use the same content topic name on several shards,
each node that is interested in this content topic has to be subscribed to all respective shards, which does not scale.
Splitting content topics in a more sophisticated and efficient way will be part of a future document.
each node that is interested in this content topic
has to be subscribed to all respective shards, which does not scale.
Splitting content topics in a more sophisticated and
efficient way will be part of a future document.
## Relay Shards
Sharding the [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md) network is an integral part of scaling the Status app.
Sharding the [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md)
network is an integral part of scaling the Status app.
[51/WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md) specifies shards clusters, which are sets of `1024` shards (separate pubsub mesh networks).
[WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)
specifies shards clusters, which are sets of `1024` shards
(separate pubsub mesh networks).
Content topics specified by application protocols can be distributed over these shards.
The Status app protocols are assigned to shard cluster `16`,
as defined in [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/waku-RFC/informational/relay-static-shard-alloc.md).
as defined in [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/master/informational/relay-static-shard-alloc.md).
[WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md) specifies three sharding methods.
This document uses *static sharding*, which leaves the distribution of content topics to application protocols,
[WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)
specifies three sharding methods.
This document uses *static sharding*,
which leaves the distribution of content topics to application protocols,
but takes care of shard discovery.
The 1024 shards within the main Status shard cluster are allocated as follows.
@@ -69,7 +88,7 @@ The 1024 shards within the main Status shard cluster are allocated as follows.
| 768 - 895 | 1:1 chat |
| 896 - 1023 | media and control msgs |
Shard indices are mapped to pubsub topic names as follows (specified in [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md)).
Shard indices are mapped to pubsub topic names as follows (specified in [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)).
`/waku/2/rs/<cluster_id>/<shard_number>`
@@ -77,24 +96,27 @@ an example for the shard with index `18` in the Status shard cluster:
`/waku/2/rs/16/18`
In other words, the mesh network with the pubsub topic name `/waku/2/rs/16/18` carries messages associated with shard `18` in the Status shard cluster.
In other words, the mesh network with the pubsub topic name `/waku/2/rs/16/18`,
carries messages associated with shard `18` in the Status shard cluster.
#### Implementation Suggestion
The Waku implementation should offer an interface that allows Status nodes to subscribe to Status specific content topics like
The Waku implementation should offer an interface that
allows Status nodes to subscribe to Status specific content topics like
```
```yaml
subscribe("/status/xyz", 16, 18)
```
The shard cluster index `16` can be kept in the Status app configuration,
so that Status nodes can simply use
```
```yaml
subscribe("/status/xyz", 18)
```
which means: connect to the `"status/xyz"` content topic on shard `18` within the Status shard cluster.
which means: connect to the `"status/xyz"` content topic on shard `18`
within the Status shard cluster.
### Status Communities
@@ -132,34 +154,40 @@ message CommunityDescription {
}
```
> *Note*: Currently, Status app has allocated shared cluster `16` in [52/WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/waku-RFC/informational/relay-static-shard-alloc.md).
> *Note*: Currently, Status app has allocated shared cluster `16` in [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/master/informational/relay-static-shard-alloc.md).
Status app could allocate more shard clusters, for instance to establish a test net.
We could add the shard cluster index to the community description as well.
The recommendation for now is to keep it as a configuration option of the Status app.
> *Note*: Once this RFC moves forward, the new community description protobuf fields should be mentioned in [56/STATUS-COMMUNITIES](../56/communities.md).
The recommendation for now,
is to keep it as a configuration option of the Status app.
> *Note*: Once this RFC moves forward,
the new community description protobuf fields should be mentioned in [56/STATUS-COMMUNITIES](../56/communities.md).
Status communities can be mapped to shards in two ways: static, and owner-based.
#### Static Mapping
With static mapping, communities are assigned a specific shard index within the Status shard cluster.
This mapping is similar in nature to the shard cluster allocation in [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/waku-RFC/informational/relay-static-shard-alloc.md).
With static mapping,
communities are assigned a specific shard index within the Status shard cluster.
This mapping is similar in nature to the shard cluster allocation in [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/master/informational/relay-static-shard-alloc.md).
Shard indices allocated in that way are in the range `16 - 127`.
The Status CC community uses index `16` (not to confuse with shard cluster index `16`, which is the Status shard cluster).
The Status CC community uses index `16`
(not to confuse with shard cluster index `16`, which is the Status shard cluster).
#### Owner Mapping
> *Note*: This way of mapping will be specified post-MVP.
Community owners can choose to map their communities to any shard within the index range `128 - 767`.
Community owners can choose to map their communities to any shard within
the index range `128 - 767`.
### 1:1 Chat
[55/STATUS-1TO1-CHAT](../55/1to1-chat.md) uses partitioned topics to map 1:1 chats to a set of 5000 content topics.
This document extends this mapping to 8192 content topics that are, in turn, mapped to 128 shards in the index range of `768 - 895`.
[55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
uses partitioned topics to map 1:1 chats to a set of 5000 content topics.
This document extends this mapping to 8192 content topics that are, in turn,
mapped to 128 shards in the index range of `768 - 895`.
```
```js
contentPartitionsNum = 8192
contentPartition = mod(publicKey, contentPartitionsNum)
partitionContentTopic = "contact-discovery-" + contentPartition
@@ -175,74 +203,94 @@ shardIndex = 768 + mod(publicKey, shardNum)
As described in [30/ADAPTIVE-NODES](../../waku/informational/30/adaptive-nodes.md),
Waku supports a continuum of node types with respect to available resources.
Infrastructure nodes are powerful nodes that have a high bandwidth connection and a high up-time.
Infrastructure nodes are powerful nodes that have a high bandwidth connection and
a high up-time.
This document, which informs about simple ways of scaling Status over Waku,
assumes the presence of a set of such infrastructure nodes in each shard.
Infrastructure nodes are especially important for providing connectivity in the roll-out phase.
Infrastructure nodes are especially important for
providing connectivity in the roll-out phase.
Infrastructure nodes are not limited to Status fleets, or nodes run by community owners.
Infrastructure nodes are not limited to Status fleets, or
nodes run by community owners.
Anybody can run infrastructure nodes.
### Statically-Mapped Communities
Infrastructure nodes are provided by the community owner, or by members of the respective community.
Infrastructure nodes are provided by the community owner,
or by members of the respective community.
### Owner-Mapped Communities
Infrastructure nodes are part of a subset of the shards in the range `128 - 767`.
Recommendations on choosing this subset will be added in a future version of this document.
Recommendations on choosing this subset will be added
in a future version of this document.
Status fleet nodes make up a part of these infrastructure nodes.
### 1:1 chat
Infrastructure nodes are part of a subset of the shards in the range `768 - 985` (similar to owner-mapped communities).
Recommendations on choosing this subset will be added in a future version of this document.
Infrastructure nodes are part of a subset of the shards in the range `768 - 985`
(similar to owner-mapped communities).
Recommendations on choosing this subset will be added
in a future version of this document.
Desktop clients can choose to only use filter and lightpush.
> *Note*: Discussion: I'd suggest to set this as the default for the MVP.
The load on infrastructure nodes would not be higher, because they have to receive and relay each message anyways.
The load on infrastructure nodes would not be higher, because
they have to receive and relay each message anyways.
This comes as a trade-off to anonymity and decentralization,
but can significantly improve scaling.
We still have k-anonymity because several chat pairs are mapped into one content topic.
We could improve on this in the future, and research the applicability of PIR (private information retrieval) techniques in the future.
We still have k-anonymity because
several chat pairs are mapped into one content topic.
We could improve on this in the future, and research the applicability of PIR
(private information retrieval) techniques in the future.
## Infrastructure Shards
Waku messages are typically relayed in larger mesh networks comprised of nodes with varying resource profiles (see [30/ADAPTIVE-NODES](../../waku/informational/30/adaptive-nodes.md).
To maximise scaling, relaying of specific message types can be dedicated to shards where only infrastructure nodes with very strong resource profiles relay messages.
Waku messages are typically relayed in larger mesh networks
comprised of nodes with varying resource profiles
(see [30/ADAPTIVE-NODES](../../waku/informational/30/adaptive-nodes.md)).
To maximise scaling, relaying of specific message types can be dedicated to shards
where only infrastructure nodes with very strong resource profiles relay messages.
This comes as a trade-off to decentralization.
## Control Message Shards
To get the maximum scaling for select large communities for the Status scaling MVP,
specific control messages that cause significant load (at a high user number) SHOULD be moved to a separate control message shard.
specific control messages that cause significant load
(at a high user number) SHOULD be moved to a separate control message shard.
These control messages comprise:
* community description
* membership update
* membership update
* backup
* community request to join response
* sync profile picture
The relay functionality of control messages shards SHOULD be provided by infrastructure nodes.
The relay functionality of control messages shards SHOULD
be provided by infrastructure nodes.
Desktop clients should use light protocols as the default for control message shards.
Strong Desktop clients MAY opt in to support the relay network.
Each large community (in the index range of `16 - 127`) can get its dedicated control message shard (in the index range `896 - 1023`) if deemed necessary.
Each large community (in the index range of `16 - 127`)
can get its dedicated control message shard
(in the index range `896 - 1023`) if deemed necessary.
The Status CC community uses shard `896` as its control message shard.
This comes with trade-offs to decentralization and anonymity (see *Security Considerations* section).
This comes with trade-offs to decentralization and anonymity
(see *Security Considerations* section).
## Media Shards
Similar to control messages, media-heavy communities should use separate media shards (in the index range `896 - 1023`) for disseminating messages with large media data.
Similar to control messages, media-heavy communities should use separate media shards
(in the index range `896 - 1023`) for disseminating messages with large media data.
The Status CC community uses shard `897` as its media shard.
## Infrastructure-focused Community
Large communities MAY choose to mainly rely on infrastructure nodes for *all* message transfers (not limited to control, and media messages).
Large communities MAY choose to mainly rely on infrastructure nodes
for *all* message transfers (not limited to control, and media messages).
Desktop clients of such communities should use light protocols as the default.
Strong Desktop clients MAY opt in to support the relay network.
@@ -254,24 +302,29 @@ Light protocols may be used to save bandwidth,
at the (global) cost of not contributing to the network.
Using light protocols is RECOMMENDED for resource restricted nodes,
e.g. browsers,
and devices that (temporarily) have a low bandwidth connection or a connection with usage-based billing.
and devices that (temporarily)
have a low bandwidth connection or a connection with usage-based billing.
Light protocols comprise
* [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md) for sending messages
* [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md) for requesting messages with specific attributes
* [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/peer-exchange/peer-exchange.md) for discovering peers
* [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md)
for requesting messages with specific attributes
* [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md)
for discovering peers
## Waku Archive
Archive nodes are Waku nodes that offer the Waku archive service via the Waku store protocol ([13/WAKU2-STORE](../../waku/standards/core/13/store.md)).
Archive nodes are Waku nodes that offer the Waku archive service via
the Waku store protocol ([13/WAKU2-STORE](../../waku/standards/core/13/store.md)).
They are part of a set of shards and store all messages disseminated in these shards.
Nodes can request history messages via the [13/WAKU2-STORE](../../waku/standards/core/13/store.md).
The store service is not limited to a Status fleet.
Anybody can run a Waku Archive node in the Status shards.
> *Note*: There is no specification for discovering archive nodes associated with specific shards yet.
> *Note*: There is no specification for discovering archive nodes
associated with specific shards yet.
Nodes expect archive nodes to store all messages, regardless of shard association.
The recommendation for the allocation of archive nodes to shards is similar to the
@@ -280,35 +333,46 @@ In fact, the archive service can be offered by infrastructure nodes.
## Discovery
Shard discovery is covered by [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md).
This allows the Status app to abstract from the discovery process and simply address shards by their index.
Shard discovery is covered by [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md).
This allows the Status app to abstract from the discovery process and
simply address shards by their index.
### Libp2p Rendezvous and Circuit-Relay
To make nodes behind restrictive NATs discoverable,
this document suggests using [libp2p rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md).
Nodes can check whether they are behind a restrictive NAT using the [libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md).
Nodes can check whether they are behind a restrictive NAT using the
[libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md).
> *Note:* The following will move into [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md), or [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md):
Nodes behind restrictive NATs SHOULD not announce their publicly unreachable address via [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md) discovery.
> *Note:* The following will move into [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md),
or [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md):
Nodes behind restrictive NATs SHOULD not announce their publicly unreachable address
via [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md) discovery.
It is RECOMMENDED that nodes that are part of the relay network also act as rendezvous points.
This includes accepting register queries from peers, as well as answering rendezvous discover queries.
It is RECOMMENDED that nodes that are part of the relay network also
act as rendezvous points.
This includes accepting register queries from peers,
as well as answering rendezvous discover queries.
Nodes MAY opt-out of the rendezvous functionality.
To allow nodes to initiate connections to peers behind restrictive NATs (after discovery via rendezvous),
To allow nodes to initiate connections to peers behind restrictive NATs
(after discovery via rendezvous),
it is RECOMMENDED that nodes that are part of the Waku relay network also offer
[libp2p circuit relay](https://github.com/libp2p/specs/blob/6634ca7abb2f955645243d48d1cd2fd02a8e8880/relay/circuit-v2.md) functionality.
[libp2p circuit relay](https://github.com/libp2p/specs/blob/6634ca7abb2f955645243d48d1cd2fd02a8e8880/relay/circuit-v2.md)
functionality.
To minimize the load on circuit-relay nodes, nodes SHOULD
1) make use of the [limiting](https://github.com/libp2p/specs/blob/6634ca7abb2f955645243d48d1cd2fd02a8e8880/relay/circuit-v2.md#reservation)
functionality offered by the libp2p circuit relay protocols, and
2) use [DCUtR](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md) to upgrade to a direct connection.
2) use [DCUtR](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md)
to upgrade to a direct connection.
Nodes that do not announce themselves at all and only plan to use light protocols,
MAY use rendezvous discovery instead of or along-side [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/peer-exchange/peer-exchange.md).
For these nodes, rendezvous and [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/peer-exchange/peer-exchange.md) offer the same functionality,
MAY use rendezvous discovery instead of or along-side [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md).
For these nodes, rendezvous and
[WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md)
offer the same functionality,
but return node sets sampled in different ways.
Using both can help increasing connectivity.
@@ -321,119 +385,187 @@ Such nodes SHOULD, however, not register at circuit relays.
Registering a namespace via [lib-p2p rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md#interaction)
is done via a register query:
```
```rs
REGISTER{my-app, {QmA, AddrA}}
```
The app name, `my-app` contains the encoding of a single shard in string form:
```
```rs
"rs/"| to_string(<2-byte shard cluster index>) | "/" | to_string(<2-byte shard index>)
```
The string conversion SHOULD remove leading zeros.
> *Note:* Since the [ns](https://github.com/libp2p/specs/blob/master/rendezvous/README.md#protobuf) field is of type string,
a more efficient byte encoding is not utilized.
> *Note:* Since the [ns](https://github.com/libp2p/specs/blob/master/rendezvous/README.md#protobuf)
field is of type string, a more efficient byte encoding is not utilized.
Registering shard 2 in the Status shard cluster (with shard cluster index 16, see [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/waku-RFC/informational/relay-static-shard-alloc.md),
Registering shard 2 in the Status shard cluster (with shard cluster index 16,
see [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/master/informational/relay-static-shard-alloc.md),
the register query would look like
```
```rs
REGISTER{"rs/16/2", {QmA, AddrA}}
```
Participation in further shards is registered with further queries; one register query per shard.
Participation in further shards is registered with further queries;
one register query per shard.
A discovery query for nodes that are part of this shard would look like
```
```rs
DISCOVER{ns: "rs/16/2"}
```
## DoS Protection
Hereunder we describe the "opt-in message signing for DoS prevention" solution, designed *ad hoc* for Status MVP.
Hereunder we describe the "opt-in message signing for DoS prevention" solution,
designed *ad hoc* for Status MVP.
Since publishing messages to pubsub topics has no limits, anyone can publish messages at a very high rate and DoS the network.
This would elevate the bandwidth consumption of all nodes subscribed to said pubsub topic, making it prohibitive (in terms of bandwidth) to be subscribed to it.
In order to scale, we need some mechanism to prevent this from happening, otherwise all scaling efforts will be in vain.
Since RLN is not ready yet, hereunder we describe a simpler approach designed *ad hoc* for Status use case, feasible to implement for the MVP and that validates some of the ideas that will evolve to solutions such as RLN.
Since publishing messages to pubsub topics has no limits,
anyone can publish messages at a very high rate and DoS the network.
This would elevate the bandwidth consumption of all nodes subscribed
to said pubsub topic, making it prohibitive (in terms of bandwidth)
to be subscribed to it.
In order to scale, we need some mechanism to prevent this from happening,
otherwise all scaling efforts will be in vain.
Since RLN is not ready yet,
hereunder we describe a simpler approach designed *ad hoc* for Status use case,
feasible to implement for the MVP and
that validates some of the ideas that will evolve to solutions such as RLN.
With this approach, certain pubsub topics can be optionally configured to only accept messages signed with a given key, that only trusted entities know.
This key can be pre-shared among a set of participants, that are trusted to make fair usage of the network, publishing messages at a reasonable rate/size.
Note that this key can be shared/reused among multiple participants, and only one key is whitelisted per pubsub topic.
This is an opt-in solution that operators can choose to deploy in their shards (i.e. pubsub topics), but it's not enforced in the default one.
Operators can freely choose how they want to generate, and distribute the public keys. It's also their responsibility to handle the private key, sharing it with only trusted parties and keeping proper custody of it.
With this approach, certain pubsub topics can be optionally configured
to only accept messages signed with a given key,
that only trusted entities know.
This key can be pre-shared among a set of participants,
that are trusted to make fair usage of the network,
publishing messages at a reasonable rate/size.
Note that this key can be shared/reused among multiple participants, and
only one key is whitelisted per pubsub topic.
This is an opt-in solution that operators can choose to deploy in their shards
(i.e. pubsub topics), but it's not enforced in the default one.
Operators can freely choose how they want to generate, and
distribute the public keys.
It's also their responsibility to handle the private key,
sharing it with only trusted parties and keeping proper custody of it.
The following concepts are introduced:
* `private-key-topic`: A private key of 32 bytes, that allows the holder to sign messages and it's mapped to a `protected-pubsub-topic`.
* `app-message-hash`: Application `WakuMessage` hash, calculated as `sha256(concat(pubsubTopic, payload, contentTopic, timestamp, ephemeral))` with all elements in bytes.
* `message-signature`: ECDSA signature of `application-message-hash` using a given `private-key-topic`, 64 bytes.
* `private-key-topic`: A private key of 32 bytes,
that allows the holder to sign messages and it's mapped to a `protected-pubsub-topic`.
* `app-message-hash`: Application `WakuMessage` hash,
calculated as `sha256(concat(pubsubTopic, payload, contentTopic, timestamp, ephemeral))`
with all elements in bytes.
* `message-signature`: ECDSA signature of `application-message-hash`
using a given `private-key-topic`, 64 bytes.
* `public-key-topic`: The equivalent public key of `private-key-topic`.
* `protected-pubsub-topic`: Pubsub topic that only accepts messages that were signed with `private-key-topic`, where `verify(message-signature, app-message-hash, public-key-topic)` is only correct if the `message-signature` was produced by `private-key-topic`. See ECDSA signature verification algorithm.
* `protected-pubsub-topic`: Pubsub topic that only accepts messages
that were signed with `private-key-topic`,
where `verify(message-signature, app-message-hash, public-key-topic)`
is only correct if the `message-signature` was produced by `private-key-topic`.
See ECDSA signature verification algorithm.
This solution introduces two roles:
* Publisher: A node that knows the `private-key-topic` associated to `public-key-topic`, that can publish messages with a valid `message-signature` that are accepted and relayed by the nodes implementing this feature.
* Relayer: A node that knows the `public-key-topic`, which can be used to verify if the messages were signed with the equivalent `private-key-topic`. It allows distinguishing valid from invalid messages which protect the node against DoS attacks, assuming that the users of the key send messages of a reasonable size and rate. Note that a node can validate messages and relay them or not without knowing the private key.
* Publisher: A node that knows the `private-key-topic` associated to `public-key-topic`,
that can publish messages with a valid `message-signature` that are accepted and
relayed by the nodes implementing this feature.
* Relayer: A node that knows the `public-key-topic`,
which can be used to verify if the messages were signed with the equivalent `private-key-topic`.
It allows distinguishing valid from invalid messages
which protect the node against DoS attacks,
assuming that the users of the key send messages of a reasonable size and rate.
Note that a node can validate messages and
relay them or not without knowing the private key.
### Design requirements (publisher)
A publisher that wants to send messages that are relayed in the network for a given `protected-pubsub-topic` shall:
* be able to sign messages with the `private-key-topic` configured for that topic, producing a ECDSA signature of 64 bytes using deterministic signing complying with RFC 6979.
* include the signature of the `app-message-hash` (`message-signature`) that wishes to send in the `WakuMessage` `meta` field.
A publisher that wants to send messages
that are relayed in the network for a given `protected-pubsub-topic` shall:
The `app-message-hash` of the message shall be calculated as the `sha256` hash of the following fields of the message:
* be able to sign messages with the `private-key-topic` configured for that topic,
producing a ECDSA signature of 64 bytes using
deterministic signing complying with RFC 6979.
* include the signature of the `app-message-hash` (`message-signature`)
that wishes to send in the `WakuMessage` `meta` field.
```
The `app-message-hash` of the message shall be calculated as the `sha256` hash
of the following fields of the message:
```rs
sha256(concat(pubsubTopic, payload, contentTopic, timestamp, ephemeral))
```
Where fields are serialized into bytes using little-endian. Note that `ephemeral` is a boolean that is serialized to `0` if `false` and `1` if `true`.
Where fields are serialized into bytes using little-endian.
Note that `ephemeral` is a boolean that is serialized to `0` if `false` and
`1` if `true`.
### Design requirements (relay)
Requirements for the relay are listed below:
* A valid `protected-pubsub-topic` shall be configured with a `public-key-topic`, (derived from a `private-key-topic`). Note that the relay does not need to know the private key.
For simplicity, there is just one key per topic. Since this approach has clear privacy implications, this configuration is not part of the waku protocol, but of the application.
* A valid `protected-pubsub-topic` shall be configured with a `public-key-topic`,
(derived from a `private-key-topic`).
Note that the relay does not need to know the private key.
For simplicity, there is just one key per topic.
Since this approach has clear privacy implications,
this configuration is not part of the waku protocol, but of the application.
Requirements on the gossipsub validator:
* Relay nodes should use the existing gossipsub validators that allow to `Accept` or `Reject` messages, according to the following criteria:
* Relay nodes should use the existing gossipsub validators that allow to `Accept`
or `Reject` messages, according to the following criteria:
* If `timestamp` is not set (equals to 0) then `Reject` the message.
* If the `timestamp` is `abs(current_timestamp-timestamp) > MessageWindowInSec` then `Reject` the message.
* If the `timestamp` is `abs(current_timestamp-timestamp) > MessageWindowInSec`
then `Reject` the message.
* If `meta` is empty, `Reject` the message.
* If `meta` exists but its size is different than 64 bytes, `Reject` the message.
* If `meta` does not successfully verifies according to the ECDSA signature verification algorithm using `public-key-topic` and `app-message-hash`, then `Reject` the message.
* If `meta` does not successfully verifies according to the ECDSA signature
verification algorithm using `public-key-topic` and `app-message-hash`,
then `Reject` the message.
* If and only if all above conditions are met then `Accept` the message.
Other requirements:
* The node shall keep metrics on the messages validation output, `Accept` or `Reject`.
* (Optional). To further strengthen DoS protection, gossipsub [scoring](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators) can be used to trigger disconnections from peers sending multiple invalid messages. See `P4` penalty.
This protects each peer from DoS, since this score is used to trigger disconnections from nodes attempting to DoS them.
* The node shall keep metrics on the messages validation output,
`Accept` or `Reject`.
* (Optional). To further strengthen DoS protection,
gossipsub [scoring](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators)
can be used to trigger disconnections from peers sending multiple invalid messages.
See `P4` penalty.
This protects each peer from DoS,
since this score is used to trigger disconnections from nodes attempting to DoS them.
### Required changes
This solution is designed to be backward compatible so that nodes validating messages can coexist in the same topic with other nodes that don't perform validation. But note that only nodes that perform message validation will be protected against DoS. Nodes wishing to opt-in this DoS protection feature shall:
* Generate a `private-key-topic` and distribute it to a curated list of users, that are trusted to send messages at a reasonable rate.
* Redeploy the nodes, adding a new configuration where a `protected-pubsub-topic` is configured with a `public-key-topic`, used to verify the messages being relayed.
This solution is designed to be backward compatible so
that nodes validating messages can coexist in the same topic
with other nodes that don't perform validation.
But note that only nodes that perform message validation
will be protected against DoS.
Nodes wishing to opt-in this DoS protection feature shall:
* Generate a `private-key-topic` and distribute it to a curated list of users,
that are trusted to send messages at a reasonable rate.
* Redeploy the nodes, adding a new configuration
where a `protected-pubsub-topic` is configured with a `public-key-topic`,
used to verify the messages being relayed.
### Test vectors
Relay nodes complying with this specification shall accept the following message in the configured pubsub topic.
Relay nodes complying with this specification
shall accept the following message in the configured pubsub topic.
Given the following key pair:
```
```js
private-key-topic = 5526a8990317c9b7b58d07843d270f9cd1d9aaee129294c1c478abf7261dd9e6
public-key-topic = 049c5fac802da41e07e6cdf51c3b9a6351ad5e65921527f2df5b7d59fd9b56ab02bab736cdcfc37f25095e78127500da371947217a8cd5186ab890ea866211c3f6
```
And the following message to send:
```
```js
protected-pubsub-topic = pubsub-topic
contentTopic = content-topic
payload = 1A12E077D0E89F9CAC11FBBB6A676C86120B5AD3E248B1F180E98F15EE43D2DFCF62F00C92737B2FF6F59B3ABA02773314B991C41DC19ADB0AD8C17C8E26757B
@@ -443,19 +575,22 @@ ephemeral = true
The message hash and meta (aka signature) are calculated as follows.
```
```js
app-message-hash = 662F8C20A335F170BD60ABC1F02AD66F0C6A6EE285DA2A53C95259E7937C0AE9
message.meta = 127FA211B2514F0E974A055392946DC1A14052182A6ABEFB8A6CD7C51DA1BF2E40595D28EF1A9488797C297EED3AAC45430005FB3A7F037BDD9FC4BD99F59E63
```
Using `message.meta`, the relay node shall calculate the `app-message-hash` of the received message using `public-key-topic`, and with the values above, the signature should be verified, making the node `Accept` the message and relaying it to other nodes in the network.
Using `message.meta`, the relay node shall calculate the `app-message-hash`
of the received message using `public-key-topic`,
and with the values above, the signature should be verified,
making the node `Accept` the message and relaying it to other nodes in the network.
## Owner-Mapped Communities
## Owner Mapped Communities
Basic idea:
Tokenized load.
### 1:1 Chat
### 1 to 1 Chat
An idea we plan to explore in the future:
Map 1:1 chats to community shards, if both A and B are part of the respective community.
@@ -466,7 +601,8 @@ It could be rate-limited with RLN.
This document makes several trade-offs to privacy and anonymity.
Todo: elaborate.
See [WAKU2-ADVERSARIAL-MODELS](https://github.com/waku-org/specs/blob/waku-RFC/informational/adversarial-models.md) for information on Waku Anonymity.
See [WAKU2-ADVERSARIAL-MODELS](https://github.com/waku-org/specs/blob/master/informational/adversarial-models.md)
for information on Waku Anonymity.
## Copyright
@@ -475,15 +611,15 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
## References
* [56/STATUS-COMMUNITIES](../56/communities.md)
* [55/STATUS-1TO1-CHAT](.../55/1to1-chat.md)
* [23/WAKU2-TOPICS](../../waku/informational/23/)
* [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
* [23/WAKU2-TOPICS](../../waku/informational/23/topics.md)
* [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md)
* [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md)
* [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/waku-RFC/informational/relay-static-shard-alloc.md)
* [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)
* [WAKU2-RELAY-STATIC-SHARD-ALLOC](https://github.com/waku-org/specs/blob/master/informational/relay-static-shard-alloc.md)
* [30/ADAPTIVE-NODES](../../waku/informational/30/adaptive-nodes.md)
* [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md)
* [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md)
* [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/peer-exchange/peer-exchange.md)
* [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md)
* [13/WAKU2-STORE](../../waku/standards/core/13/store.md)
* [libp2p rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md)
* [libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md)
@@ -493,8 +629,9 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
* [DCUtR](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md)
* [scoring](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators)
* [Circuit Relay](https://docs.libp2p.io/concepts/nat/circuit-relay/)
* [WAKU2-ADVERSARIAL-MODELS](https://github.com/waku-org/specs/blob/waku-RFC/informational/adversarial-models.md)
* [WAKU2-ADVERSARIAL-MODELS](https://github.com/waku-org/specs/blob/master/informational/adversarial-models.md)
## Informative
* [Circuit Relay](https://docs.libp2p.io/concepts/nat/circuit-relay/)
* [WAKU2-ENR](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/enr.md)
* [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md)

View File

@@ -0,0 +1,415 @@
---
title: STATUS-PROTOCOLS
name: Status Protocol Stack
status: raw
category: Standards Track
description: Specifies the Status application protocol stack.
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Jimmy Debe <jimmy@status.im>
- Aaryamann Challani <p1ge0nh8er@proton.me>
---
## Abstract
This specification describes the Status Application protocol stack.
It focuses on elements and features in the protocol stack for all application-level functions:
- functional scope (also _broadcast audience_)
- content topic
- ephemerality
- end-to-end reliability layer
- encryption layer
- transport layer (Waku)
It also introduces strategies to restrict resource usage, distribute large messages, etc.
Application-level functions are out of scope and specified separately. See:
- [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
- [56/STATUS-COMMUNITIES](../56/communities.md)
## Status protocol stack
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
See the simplified diagram of the Status application protocol stack:
| |
|---|
| Status application layer |
| End-to-end reliability layer |
| Encryption layer |
| Transport layer (Waku) |
| |
## Status application layer
Application level functions are defined in the _application_ layer.
Status currently defines functionality to support three main application features:
- Status Communities, as specified in [56/STATUS-COMMUNITIES](../56/communities.md)
- Status 1:1 Chat, as specified in [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
- Status Private Group Chat, as specified in a subsection of [55/STATUS-1TO1-CHAT](../55/1to1-chat.md#negotiation-of-a-11-chat-amongst-multiple-participants-group-chat)
<!-- TODO: list functions not related to main app features, such as user sync, backup, push notifications, etc. -->
Each application-level function, regardless which feature set it supports, has the following properties:
1. Functional scope
2. Content topic
3. Ephemerality
### Functional Scope
Each Status app-level message MUST define a functional scope.
The functional scope MUST define the _minimum_ scope of the audience that should _participate_ in the app function the message is related to.
In other words, it determines the minimum subset of Status app participants
that should have access to messages related to that function.
Note that the functional scope is distinct from the number of participants that is _addressed_ by a specific message.
For example, a participant will address a 1:1 chat to only one other participant.
However, since all users of the Status app MUST be able to participate in 1:1 chats,
the functional scope of messages enabling 1:1 chats MUST be a global scope.
Similarly, since private group chats can be set up between any subset of Status app users,
the functional scope for messages related to private group chats MUST be global.
Along the same principle, messages that originate within communities are of global interest
for all users who have an interest in the Status Communities feature.
Such messages MUST have a global functional scope,
that can be accessed by any app users interested in communities.
A different group of messages are addressed only to the participant that generated those messages itself.
These _self-addressed_ messages MUST have a local functional scope.
If we further make a distinction between "control" and "content" messages,
we can distinguish five distinct functional scopes.
All Status messages MUST have one of these functional scopes:
#### Global general scope
1. _Global control_: messages enabling the basic functioning of the app to control general features that all app users should be able to participate in. Examples include Contact Requests, global Status Updates, Group Chat Invites, etc.
2. _Global content_: messages carrying user-generated content for global functions. Examples include 1:1 chat messages, images shared over private group chats, etc.
#### Global community scope
1. _Global community control_: messages enabling the basic functioning of the app to control features related to communities. Examples include Community Invites, Community Membership Updates, community Status Updates, etc.
2. _Global community content_: messages carrying user-generated content for members of any community.
> **Note:** a previous iteration of the Status Communities feature defined separate community-wide scopes for each community.
However, this model was deprecated and all communities now operate on a global, shared scope.
This implies that different communities will share shards on the routing layer.
#### Local scope
1. _Local_: messages related to functions that are only relevant to a single user. Also known as _self-addressed messages_. Examples include messages used to exchange information between app installations, such as User Backup and Sync messages.
Note that the functional scope is a logical property of Status messages.
It SHOULD however inform the underlying [transport layer sharding](#pubsub-topics-and-sharding) and [transport layer subscriptions](#subscribing).
In general a Status client SHOULD subscribe to participate in:
- all global functions
- global community functions if it is interested in this feature, and
- its own local functions.
### Content topics
Each Status app-level message MUST define a content topic that links messages in related app-level functions and sub-functions together.
This MUST be based on the filter use cases for [transport layer subscriptions](#subscribing)
and [retrieving historical messages](#retrieving-historical-messages).
A content topic SHOULD be identical across all messages that are always part of the same filter use case (or always form part of the same content-filtered query criteria).
In other words, the number of content topics defined in the app SHOULD match the number of filter use cases.
For the sake of illustration, consider the following common content topic and filter use cases:
- if all messages belonging to the same 1:1 chat are always filtered together, they SHOULD use the same content topic (see [55/STATUS-1TO1-CHAT](../55/1to1-chat.md))
- if all messages belonging to the same Community are always filtered together, they SHOULD use the same content topic (see [56/STATUS-COMMUNITIES](../56/communities.md)).
The app-level content topic MUST be populated in the `content_topic` field in the encapsulating Waku message (see [Waku messages](#waku-messages)).
### Ephemerality
Each Status app-level message MUST define its _ephemerality_.
Ephemerality is a boolean value, set to `true` if a message is considered ephemeral.
Ephemeral messages are messages emitted by the app that are transient in nature.
They only have temporary "real-time" value
and SHOULD NOT be stored and retrievable from historical message stores and sync caches.
Similarly, ephemeral message delivery is best-effort in nature and SHOULD NOT be considered in message reliability mechanisms (see [End-to-end reliability layer](#end-to-end-reliability-layer)).
An example of ephemeral messages would be periodic status update messages, indicating a particular user's online status.
Since only a user's current online status is of value, there is no need to store historical status update messages.
Since status updates are periodic, there is no strong need for end-to-end reliability as subsequent updates are always to follow.
App-level messages that are considered ephemeral, MUST set the `ephemeral` field in the encapsulating Waku message to `true` (see [Waku messages](#waku-messages))
## End-to-end reliability layer
The end-to-end reliability layer contains the functions related to one of the two end-to-end reliability schemes defined for Status app messages:
1. Minimum Viable protocol for Data Synchronisation, or MVDS (see [STATUS-MVDS-USAGE](./status-mvds.md))
2. Scalable distributed log reliability (spec and a punchier name TBD, see the [original forum post announcement](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/16))
Ephemeral messages SHOULD omit this layer.
Non-ephemeral 1:1 chat messages SHOULD make use of MVDS to achieve reliable data synchronisation between the two parties involved in the communication.
Non-ephemeral private group chat messages build on a set of 1:1 chat links
and consequently SHOULD also make use of MVDS to achieve reliable data synchronisation between all parties involved in the communication.
Non-ephemeral 1:1 and private group chat messages MAY make use of of [scalable distributed log reliability](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/16) in future.
Since MVDS does not scale for large number of participants in the communication,
non-ephemeral community messages MUST use scalable distributed log reliability as defined in this [original forum post announcement](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/16).
The app MUST use a single channel ID per community.
## Encryption layer
The encryption layer wraps the Status App and Reliability layers in an encrypted payload.
<!-- TODO: This section is TBD. We may want to design a way for Communities to use de-MLS in a separate spec and generally simplify Status encryption. -->
## Waku transport layer
The Waku transport layer contains the functions allowing Status protocols to use [10/WAKU2](../../waku/standards/core/10/waku2.md) infrastructure as transport.
### Waku messages
Each Status application message MUST be transformed to a [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) with the following structure:
```protobuf
syntax = "proto3";
message WakuMessage {
bytes payload = 1;
string content_topic = 2;
optional uint32 version = 3;
optional sint64 timestamp = 10;
optional bytes meta = 11;
optional bool ephemeral = 31;
}
```
- `payload` MUST be set to the full encrypted payload received from the higher layers
- `version` MUST be set to `1`
- `ephemeral` MUST be set to `true` if the app-level message is ephemeral
- `content_topic` MUST be set to the app-level content topic
- `timestamp` MUST be set to the current Unix epoch timestamp (in nanosecond precision)
### Pubsub topics and sharding
All Waku messages are published to pubsub topics as defined in [23/WAKU2-TOPICS](../../waku/informational/23/topics.md).
Since pubsub topics define a routing layer for messages,
they can be used to shard traffic.
The pubsub topic used for publishing a message depends on the app-level [functional scope](#functional-scope).
#### Self-addressed messages
The application MUST define at least one distinct pubsub topic for self-addressed messages.
The application MAY define a set of more than one pubsub topic for self-addressed messages to allow traffic sharding for scalability.
#### Global messages
The application MUST define at least one distinct pubsub topic for global control messages and global content messages.
The application MAY defined a set of more than one pubsub topic for global messages to allow traffic sharding for scalability.
It is RECOMMENDED that separate pubsub topics be used for global control messages and global content messages.
#### Community messages
The application SHOULD define at least one distinct pubsub topic for global community control messages and global community content messages.
The application MAY define a set of more than one pubsub topic for global community messages to allow traffic sharding for scalability.
It is RECOMMENDED that separate pubsub topics be used for global community control messages and global community content messages.
#### Large messages
The application MAY define separate pubsub topics for large messages.
These pubsub topics for large messages MAY be distinct for each functional scope.
### Resource usage
The application SHOULD use a range of Waku protocols to interact with the Waku transport layer.
The specific set of Waku protocols used depend on desired functionality and resource usage profile for the specific client.
Resources can be restricted in terms of bandwidth and computing resources.
Waku protocols that are more appropriate for resource-restricted environments are often termed "light protocols".
Waku protocols that consume more resources, but simultaneously contribute more to Waku infrastructure, are often termed "full protocols".
The terms "full" and "light" is just a useful abstraction than a strict binary, though,
and Status clients can operate along a continuum of resource usage profiles,
each using the combination of "full" and "light" protocols most appropriate to match its environment and motivations.
To simplify interaction with the selection of "full" and "light" protocols,
Status clients MUST define a "full mode" and "light mode"
to allow users to select whether their client would prefer "full protocols" or "light protocols" by default.
Status Desktop clients are assumed to have more resources available and SHOULD use full mode by default.
Status Mobile clients are assumed to operate with more resource restrictions and SHOULD use light mode by default.
For the purposes of the rest of this document,
clients in full mode will be referred to as "full clients" and
clients in light mode will be referred to as "light clients".
### Discovery
The application MUST make use of at least one discovery method to discover and connect to Waku peers
useful for the user functions specific to that instance of the application.
The specific Waku discovery protocol used for discovery depends on the use case and resource-availability of the client.
1. [EIP-1459: DNS-based discovery](https://eips.ethereum.org/EIPS/eip-1459) is useful for initial connection to bootstrap peers.
2. [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md) allows decentralized discovery of Waku peers.
3. [34/WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/315264c202e0973476e2f1e2d0b01bea4fe1ad31/standards/core/peer-exchange.md) allows requesting peers from a service node
and is appropriate for resource-restricted discovery.
All clients SHOULD use DNS-based discovery on startup
to discover a set of bootstrap peers for initial connection.
Full clients SHOULD use [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md) for continuous ambient peer discovery.
Light clients SHOULD use [34/WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/315264c202e0973476e2f1e2d0b01bea4fe1ad31/standards/core/peer-exchange.md) to discover a set of service peers
used by that instance of the application.
### Subscribing
The application MUST subscribe to receive the traffic necessary for minimal app operation
and to enable the user functions specific to that instance of the application.
The specific Waku protocol used for subscription depends on the resource-availability of the client:
1. Filter client protocol, as specified in [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md), allows subscribing for traffic with content topic granularity and is appropriate for resource-restricted subscriptions.
2. Relay protocol, as specified in [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md), allows subscribing to traffic only with pubsub topic granularity and therefore is more resource-intensive. Relay subscription also allows the application instance to contribute to the overall routing infrastructure, which adds to its overall higher resource usage but benefits the ecosystem.
Full clients SHOULD use relay protocol as preferred method to subscribe to pubsub topics matching the scopes:
1. Global control
2. Global content
3. Global community control, if the client has activated the Status Communities feature
4. Global community content, if the client has activated the Status Communities feature
Light clients SHOULD use filter protocol to subscribe only to the content topics relevant to the user.
#### Self-addressed messages
Status clients (full or light) MUST NOT subscribe to topics for messages with self-addressed scopes.
See [Self-addressed messages](#self-addressed-messages-4).
#### Large messages
Status clients (full or light) SHOULD NOT subscribe to topics set aside for large messages.
See [Large messages](#large-messages-4).
### Publishing
The application MUST publish user and app generated messages via the Waku transport layer.
The specific Waku protocol used for publishing depends on the resource-availability of the client:
1. Lightpush protocol, as specified in [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md) allows publishing to a pubsub topic via an intermediate "full node" and is more appropriate for resource-restricted publishing.
2. Relay protocol, as specified in [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md), allows publishing directly into the relay routing network and is therefore more resource-intensive.
Full clients SHOULD use relay protocol to publish to pubsub topics matching the scopes:
1. Global control
2. Global content
3. Global community control, if the client has activated the Status Communities feature
4. Global community content, if the client has activated the Status Communities feature
Light clients SHOULD use lightpush protocol to publish control and content messages.
#### Self-addressed messages
Status clients (full or light) MUST use lightpush protocol to publish self-addressed messages.
See [Self-addressed messages](#self-addressed-messages-4).
#### Large messages
Status clients (full or light) SHOULD use lightpush protocols to publish to pubsub topics set aside for large messages.
See [Large messages](#large-messages-4).
### Retrieving historical messages
Status clients SHOULD use the store query protocol, as specified in [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md), to retrieve historical messages relevant to the client from store service nodes in the network.
Status clients SHOULD use [content filtered queries](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md#content-filtered-queries) with `include_data` set to `true`,
to retrieve the full contents of historical messages that the client may have missed during offline periods,
or to populate the local message database when the client starts up for the first time.
#### Store queries for reliability
Status clients MAY use periodic content filtered queries with `include_data` set to `false`,
to retrieve only the message hashes of past messages on content topics relevant to the client.
This can be used to compare the hashes available in the local message database with the hashes in the query response
in order to identify possible missing messages.
Once the Status client has identified a set of missing message hashes
it SHOULD use [message hash lookup queries](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md#message-hash-lookup-queries) with `include_data` set to `true`
to retrieve the full contents of the missing messages based on the hash.
Status clients MAY use [presence queries](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md#presence-queries)
to determine if one or more message hashes known to the client is present in the store service node.
Clients MAY use this method to determine if a message that originated from the client
has been successfully stored.
#### Self-addressed messages
Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve self-addressed messages relevant to that client.
See [Self-addressed messages](#self-addressed-messages-4).
#### Large messages
Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve large messages relevant to that client.
See [Large messages](#large-messages-4).
### Providing services
Status clients MAY provide service-side protocols to other clients.
Full clients SHOULD mount
the filter service protocol (see [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md))
and lightpush service protocol (see [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md))
in order to provide light subscription and publishing services to other clients
for each pubsub topic to which they have a relay subscription.
Full clients SHOULD mount
the peer exchange service protocol (see [34/WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/315264c202e0973476e2f1e2d0b01bea4fe1ad31/standards/core/peer-exchange.md))
to provide light discovery services to other clients.
Status clients MAY mount the store query protocol as service node (see [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md))
to store historical messages and
provide store services to other clients
for each pubsub topic to which they have a relay subscription
### Self-addressed messages
Messages with a _local_ functional scope (see [Functional scope](#functional-scope)),
also known as _self-addressed_ messages,
MUST be published to a distinct pubsub topic or a distinct _set_ of pubsub topics
used exclusively for messages with local scope (see [Pubsub topics and sharding](#pubsub-topics-and-sharding)).
Status clients (full or light) MUST use lightpush protocol to publish self-addressed messages (see [Publishing](#publishing)).
Status clients (full or light) MUST NOT subscribe to topics for messages with self-addressed scopes (see [Subscribing](#subscribing)).
Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve self-addressed messages relevant to that client (see [Retrieving historical messages](#retrieving-historical-messages)).
### Large messages
The application MAY define separate pubsub topics for large messages.
These pubsub topics for large messages MAY be distinct for each functional scope (see [Pubsub topics and sharding](#pubsub-topics-and-sharding)).
Status clients (full or light) SHOULD use lightpush protocols to publish to pubsub topics set aside for large messages (see [Publishing](#publishing)).
Status clients (full or light) SHOULD NOT subscribe to topics set aside for large messages (see [Subscribing](#subscribing)).
Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve large messages relevant to that client (see [Retrieving historical messages](#retrieving-historical-messages)).
#### Chunking
The Status application MAY use a chunking mechanism to break down large payloads
into smaller segments for individual Waku transport.
The definition of a large message is up to the application.
However, the maximum size for a [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) payload is 150KB.
Status application payloads that exceed this size MUST be chunked into smaller pieces
and MUST be considered a "large message".
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)
2. [56/STATUS-COMMUNITIES](../56/communities.md)
3. [10/WAKU2](../../waku/standards/core/10/waku2.md)
4. [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md)
5. [12/WAKU2-FILTER](../../waku/standards/core/12/filter.md)
6. [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md)
7. [23/WAKU2-TOPICS](../../waku/informational/23/topics.md)
8. [19/WAKU2-LIGHTPUSH](../../waku/standards/core/19/lightpush.md)
9. [Scalable distributed log reliability](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/16)
10. [STATUS-MVDS-USAGE](./status-mvds.md)
11. [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md)

View File

@@ -10,20 +10,26 @@ contributors:
## Abstract
This document lists the types of messages that are using [MVDS](/vac/2/mvds.md) in the Status application.
This document lists the types of messages that are using [MVDS](/vac/2/mvds.md)
in the Status application.
## Background
Status app uses MVDS to ensure messages going through Waku are acknolwedged by the recipient. This is to ensure that the messages are not missed by any interested parties.
Status app uses MVDS to ensure messages going through Waku
are acknolwedged by the recipient.
This is to ensure that the messages are not missed by any interested parties.
## Message types
Various Message Types contain distinct information defined by the app to facilitate convenient serialization and deserialization.
Various Message Types contain distinct information defined by the app
to facilitate convenient serialization and deserialization.
E2E reliability is a feature that ensures messages are delivered to the recipient. This is initially achieved by using MVDS in Status.
E2E reliability is a feature that ensures messages are delivered to the recipient.
This is initially achieved by using MVDS in Status.
Chat Type specifies the category of chat that a message belongs to. It can be OneToOne (aka Direct Message), GroupChat, or CommunityChat. These are the three main types of chats in Status.
Chat Type specifies the category of chat that a message belongs to.
It can be OneToOne (aka Direct Message), GroupChat, or CommunityChat.
These are the three main types of chats in Status.
| Message Type | Use MVDS | Need e2e reliability | Chat Type |
|----------------------------------------------------------------------------|-------------------------------------|----------------------|-------------------------|
@@ -114,9 +120,6 @@ Chat Type specifies the category of chat that a message belongs to. It can be On
| ApplicationMetadataMessage_COMMUNITY_SHARED_ADDRESSES_REQUEST | No | No | CommunityChat |
| ApplicationMetadataMessage_COMMUNITY_SHARED_ADDRESSES_RESPONSE | No | No | CommunityChat |
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

View File

@@ -1,204 +0,0 @@
---
title: STATUS-WAKU2-USAGE
name: Status Waku2 Usage
status: raw
category: Best Current Practice
description: Defines how the Status application uses the Waku protocols.
editor: Aaryamann Challani <p1ge0nh8er@proton.me>
contributors:
- Jimmy Debe <jimmy@status.im>
---
## Abstract
Status is a chat application which has several features, including, but not limited to -
- Private 1:1 chats, described by [55/STATUS-1TO1-CHAT](/spec/55)
- Large scale group chats, described by [56/STATUS-COMMUNITIES](/spec/56)
This specification describes how a Status implementation will make use of the underlying infrastructure,
Waku, which is described in [10/WAKU2](/spec/10).
## Background
The Status application aspires to achieve censorship resistance and incorporates specific privacy features,
leveraging the comprehensive set of protocols offered by Waku to enhance these attributes.
Waku protocols provide secure communication capabilities over decentralized networks.
Once integrated, an application will benefit from privacy-preserving,
censorship resistance and spam protected communcation.
Since Status uses a large set of Waku protocols,
it is imperative to describe how each are used.
## Terminology
| Name | Description |
| --------------- | --------- |
| `RELAY`| This refers to the Waku Relay protocol, described in [11/WAKU2-RELAY](/spec/11) |
| `FILTER` | This refers to the Waku Filter protocol, described in [12/WAKU2-FILTER](/spec/12) |
| `STORE` | This refers to the Waku Store protocol, described in [13/WAKU2-STORE](/spec/13) |
| `MESSAGE` | This refers to the Waku Message format, described in [14/WAKU2-MESSAGE](/spec/14) |
| `LIGHTPUSH` | This refers to the Waku Lightpush protocol, described in [19/WAKU2-LIGHTPUSH](/spec/19) |
| Discovery | This refers to a peer discovery method used by a Waku node. |
| `Pubsub Topic` / `Content Topic` | This refers to the routing of messages within the Waku network, described in [23/WAKU2-TOPICS](/spec/23/) |
### Waku Node:
Software that is configured with a set of Waku protocols.
A Status client comprises of a Waku node that is a `RELAY` node or a non-relay node.
### Light Client:
A Status client that operates within resource constrained environments is a node configured as light client.
Light clients do not run a `RELAY`.
Instead, Status light clients,
can request services from other `RELAY` node that provide `LIGHTPUSH` service.
## Protocol Usage
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”,
“NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
The following is a list of Waku Protocols used by a Status application.
### 1. `RELAY`
The `RELAY` MUST NOT be used by Status light clients.
The `RELAY` is used to broadcast messages between Status clients.
All Status messages are transformed into [14/WAKU2-MESSAGE](/spec/14), which are sent over the wire.
All Status message types are described in [62/STATUS-PAYLOAD](/spec/62).
Status Clients MUST transform the following object into a `MESSAGE` as described below -
```go
type StatusMessage struct {
SymKey[] []byte // [optional] The symmetric key used to encrypt the message
PublicKey []byte // [optional] The public key to use for asymmetric encryption
Sig string // [optional] The private key used to sign the message
PubsubTopic string // The Pubsub topic to publish the message to
ContentTopic string // The Content topic to publish the message to
Payload []byte // A serialized representation of a Status message to be sent
Padding []byte // Padding that must be applied to the Payload
TargetPeer string // [optional] The target recipient of the message
Ephemeral bool // If the message is not to be stored, this is set to `true`
}
```
1. A user MUST only provide either a Symmetric key OR an Asymmetric keypair to encrypt the message.
If both are received, the implementation MUST throw an error.
2. `WakuMessage.Payload` MUST be set to `StatusMessage.Payload`
3. `WakuMessage.Key` MUST be set to `StatusMessage.SymKey`
4. `WakuMessage.Version` MUST be set to `1`
5. `WakuMessage.Ephemeral` MUST be set to `StatusMessage.Ephemeral`
6. `WakuMessage.ContentTopic` MUST be set to `StatusMessage.ContentTopic`
7. `WakuMessage.Timestamp` MUST be set to the current Unix epoch timestamp (in nanosecond precision)
### 2. `STORE`
This protocol MUST remain optional according to the user's preferences,
it MAY be enabled on Light clients as well.
Messages received via [11/WAKU2-RELAY](/spec/11), are stored in a database.
When Waku node running this protocol is service node,
it MUST provide the complete list of network messages.
Status clients SHOULD request historical messages from this service node.
The messages that have the `WakuMessage.Ephemeral` flag set to true will not be stored.
The Status client MAY provide a method to prune the database of older records to save storage.
### 3. `FILTER`
This protocol SHOULD be enabled on Light clients.
This protocol SHOULD be used to filter messages based on a given criteria, such as the `Content Topic` of a `MESSAGE`.
This allows a reduction in bandwidth consumption by the Status client.
#### Content filtering protocol identifers:
The `filter-subcribe` SHOULD be implemented on `RELAY` nodes to provide `FILTER` services.
`filter-subscribe`:
/vac/waku/filter-subscribe/2.0.0-beta1
The `filter-push` SHOULD be implemented on light clients to receive messages.
`filter-push`:
/vac/waku/filter-push/2.0.0-beta1
Status clients SHOULD apply a filter for all the `Content Topic` they are interested in,
such as `Content Topic` derived from -
1. 1:1 chats with other users, described in [55/STATUS-1TO1-CHAT](/spec/55)
2. Group chats
3. Community Channels, described in [56/STATUS-COMMUNITIES](/spec/56)
### 4. `LIGHTPUSH`
The `LIGHTPUSH` protocol MUST be enabled on Status light clients.
A Status `RELAY` node MAY implement `LIGHTPUSH` to support light clients.
Peers will be able to publish messages,
without running a full-fledged [11/WAKU2-RELAY](/spec/11) protocol.
When a Status client is publishing a message,
it MUST check if Light mode is enabled,
and if so, it MUST publish the message via this protocol.
### 5. Discovery
A discovery method MUST be supported by Light clients and Full clients
Status clients SHOULD make use of the following peer discovery methods that are provided by Waku,
such as -
1. [EIP-1459: DNS-Based Discovery](https://eips.ethereum.org/EIPS/eip-1459)
2. [33/WAKU2-DISCV5](/spec/33):
A node discovery protocol to create decentralized network of interconnected Waku nodes.
3. [34/WAKU2-PEER-EXCHANGE](/spec/34):
A peer discovery protocol for resource restricted devices.
Status clients MAY use any combination of the above peer discovery methods,
which is suited best for their implementation.
## Security/Privacy Considerations
This specification inherits the security and privacy considerations from the following specifications -
1. [10/WAKU2](/spec/10)
2. [11/WAKU2-RELAY](/spec/11)
3. [12/WAKU2-FILTER](/spec/12)
4. [13/WAKU2-STORE](/spec/13)
5. [14/WAKU2-MESSAGE](/spec/14)
6. [23/WAKU2-TOPICS](/spec/23)
7. [19/WAKU2-LIGHTPUSH](/spec/19)
8. [55/STATUS-1TO1-CHAT](/spec/55)
9. [56/STATUS-COMMUNITIES](/spec/56)
10. [62/STATUS-PAYLOAD](/spec/62)
11. [EIP-1459: DNS-Based Discovery](https://eips.ethereum.org/EIPS/eip-1459)
12. [33/WAKU2-DISCV5](/spec/33)
13. [34/WAKU2-PEER-EXCHANGE](/spec/34)
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [55/STATUS-1TO1-CHAT](/spec/55)
2. [56/STATUS-COMMUNITIES](/spec/56)
3. [10/WAKU2](/spec/10)
4. [11/WAKU2-RELAY](/spec/11)
5. [12/WAKU2-FILTER](/spec/12)
6. [13/WAKU2-STORE](/spec/13)
7. [14/WAKU2-MESSAGE](/spec/14)
8. [23/WAKU2-TOPICS](/spec/23)
9. [19/WAKU2-LIGHTPUSH](/spec/19)
10. [64/WAKU2-NETWORK](/spec/64)
11. [62/STATUS-PAYLOAD](/spec/62)
12. [EIP-1459: DNS-Based Discovery](https://eips.ethereum.org/EIPS/eip-1459)
13. [33/WAKU2-DISCV5](/spec/33)
14. [34/WAKU2-PEER-EXCHANGE](/spec/34)

157
status/raw/url-data.md Normal file
View File

@@ -0,0 +1,157 @@
---
title: STATUS-URL-DATA
name: Status URL Data
status: raw
category: Standards Track
tags:
editor: Felicio Mununga <felicio@status.im>
contributors:
- Aaryamann Challani <aaryamann@status.im>
---
## Abstract
This document specifies serialization, compression, and
encoding techniques used to transmit data within URLs in the context of Status protocols.
## Motivation
When sharing URLs,
link previews often expose metadata to the websites behind those links.
To reduce reliance on external servers for providing appropriate link previews,
this specification proposes a standard method for encoding data within URLs.
## Terminology
- Community: Refer to [STATUS-COMMUNITIES](../56/communities.md)
- Channel: Refer to terminology in [STATUS-COMMUNITIES](../56/communities.md)
- User: Refer to terminology in [STATUS-COMMUNITIES](../56/communities.md)
- Shard Refer to terminology in [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)
## Wire Format
```protobuf
syntax = "proto3";
message Community {
// Display name of the community
string display_name = 1;
// Description of the community
string description = 2;
// Number of members in the community
uint32 members_count = 3;
// Color of the community title
string color = 4;
// List of tag indices
repeated uint32 tag_indices = 5;
}
message Channel {
// Display name of the channel
string display_name = 1;
// Description of the channel
string description = 2;
// Emoji of the channel
string emoji = 3;
// Color of the channel title
string color = 4;
// Community the channel belongs to
Community community = 5;
// UUID of the channel
string uuid = 6;
}
message User {
// Display name of the user
string display_name = 1;
// Description of the user
string description = 2;
// Color of the user title
string color = 3;
}
message URLData {
// Community, Channel, or User
bytes content = 1;
uint32 shard_cluster = 2;
uint32 shard_index = 3;
}
```
## Implementation
The above wire format describes the data encoded in the URL.
The data MUST be serialized, compressed, and encoded using the following standards:
Encoding
- [Base64url](https://datatracker.ietf.org/doc/html/rfc4648)
### Compression
- [Brotli](https://datatracker.ietf.org/doc/html/rfc7932)
### Serialization
- [Protocol buffers version 3](https://protobuf.dev/reference/protobuf/proto3-spec/)
### Implementation Pseudocode
Encoding
Encoding the URL MUST be done in the following order:
```protobuf
raw_data = {User | Channel | Community}
serialized_data = protobuf_serialize(raw_data)
compressed_data = brotli_compress(serialized_data)
encoded_url_data = base64url_encode(compressed_data)
```
The `encoded_url_data` is then used to generate a signature using the private key.
#### Decoding
Decoding the URL MUST be done in the following order:
```protobuf
url_data = base64url_decode(encoded_url_data)
decompressed_data = brotli_decompress(url_data)
deserialized_data = protobuf_deserialize(decompressed_data)
raw_data = deserialized_data.content
```
The `raw_data` is then used to construct the appropriate data structure
(User, Channel, or Community).
### Example
- See <https://github.com/status-im/status-web/pull/345/files>
<!-- # (Further Optional Sections) -->
## Discussions
- See <https://github.com/status-im/status-web/issues/327>
## Proof of concept
- See <https://github.com/felicio/status-web/blob/825262c4f07a68501478116c7382862607a5544e/packages/status-js/src/utils/encode-url-data.compare.test.ts#L4>
<!-- # Security Considerations -->
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [Proposal Google Sheet](https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing)
2. [Base64url](https://datatracker.ietf.org/doc/html/rfc4648)
3. [Brotli](https://datatracker.ietf.org/doc/html/rfc7932)
4. [Protocol buffers version 3](https://protobuf.dev/reference/protobuf/proto3-spec/)
5. [STATUS-URL-SCHEME](./url-scheme.md)
<!-- ## informative
A list of additional references. -->

View File

@@ -10,7 +10,8 @@ contributors:
## Abstract
This document describes URL scheme for previewing and deep linking content as well as for triggering actions.
This document describes URL scheme for previewing and
deep linking content as well as for triggering actions.
## Background / Rationale / Motivation
@@ -53,7 +54,9 @@ A standard track RFC in `stable` status MUST feature this section.
A standard track RFC in `raw` or `draft` status SHOULD feature this section.
Informational RFCs (in any state) may feature this section.
If there are none, this section MUST explicitly state that fact.
This section MAY contain additional relevant information, e.g. an explanation as to why there are no security consideration for the respective document. -->
This section MAY contain additional relevant information,
e.g. an explanation as to why there are no security consideration
for the respective document. -->
## Discussions

1
testfile.txt Normal file
View File

@@ -0,0 +1 @@
"test"

View File

@@ -15,36 +15,48 @@ contributors:
- Jimmy Debe <jimmy@status.im>
---
This document describes a consensus-oriented specification system (COSS) for building interoperable technical specifications.
COSS is based on a lightweight editorial process that seeks to engage the widest possible range of interested parties and move rapidly to consensus through working code.
This document describes a consensus-oriented specification system (COSS)
for building interoperable technical specifications.
COSS is based on a lightweight editorial process that
seeks to engage the widest possible range of interested parties and
move rapidly to consensus through working code.
This specification is based on [Unprotocols 2/COSS](https://github.com/unprotocols/rfc/blob/master/2/README.md), used by the [ZeromMQ](https://rfc.zeromq.org/) project.
This specification is based on [Unprotocols 2/COSS](https://github.com/unprotocols/rfc/blob/master/2/README.md),
used by the [ZeromMQ](https://rfc.zeromq.org/) project.
It is equivalent except for some areas:
- recommending the use of a permissive licenses, such as CC0 (with the exception of this document);
- recommending the use of a permissive licenses,
such as CC0 (with the exception of this document);
- miscellaneous metadata, editor, and format/link updates;
- more inheritance from the [IETF Standards Process](https://www.rfc-editor.org/rfc/rfc2026.txt),
e.g. using RFC categories: Standards Track, Informational, and Best Common Practice;
- standards track specifications SHOULD follow a specific structure that both streamlines editing,
and helps implementers to quickly comprehend the specification
- standards track specifications SHOULD
follow a specific structure that both streamlines editing,
and helps implementers to quickly comprehend the specification
- specifications MUST feature a header providing specific meta information
- raw specifications will not be assigned numbers
- section explaining the [IFT](https://free.technology/) Request For Comments specification process managed by the Vac service department
- section explaining the [IFT](https://free.technology/)
Request For Comments specification process managed by the Vac service department
## License
Copyright (c) 2008-24 the Editor and Contributors.
This Specification is free software;
you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation;
you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation;
either version 3 of the License, or (at your option) any later version.
This specification is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
This specification is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program;
if not, see http://www.gnu.org/licenses.
You should have received a copy of the GNU General Public License
along with this program;
if not, see [gnu.org](http://www.gnu.org/licenses).
## Change Process
@@ -52,27 +64,33 @@ This document is governed by the [1/COSS](./coss.md) (COSS).
## Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
"SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
[RFC 2119](http://tools.ietf.org/html/rfc2119).
## Goals
The primary goal of COSS is to facilitate the process of writing, proving, and improving new technical specifications.
A "technical specification" defines a protocol, a process, an API, a use of language, a methodology,
or any other aspect of a technical environment that can usefully be documented for the purposes of technical or social interoperability.
The primary goal of COSS is to facilitate the process of writing, proving, and
improving new technical specifications.
A "technical specification" defines a protocol, a process, an API, a use of language,
a methodology, or any other aspect of a technical environment that
can usefully be documented for the purposes of technical or social interoperability.
COSS is intended to above all be economical and rapid, so that it is useful to small teams with little time to spend on more formal processes.
COSS is intended to above all be economical and rapid,
so that it is useful to small teams with little time to spend on more formal processes.
Principles:
* We aim for rough consensus and running code; [inspired by the IETF Tao](https://www.ietf.org/about/participate/tao/).
* Specifications are small pieces, made by small teams.
* Specifications should have a clearly responsible editor.
* The process should be visible, objective, and accessible to anyone.
* The process should clearly separate experiments from solutions.
* The process should allow deprecation of old specifications.
- We aim for rough consensus and running code; [inspired by the IETF Tao](https://www.ietf.org/about/participate/tao/).
- Specifications are small pieces, made by small teams.
- Specifications should have a clearly responsible editor.
- The process should be visible, objective, and accessible to anyone.
- The process should clearly separate experiments from solutions.
- The process should allow deprecation of old specifications.
Specifications should take minutes to explain, hours to design, days to write, weeks to prove, months to become mature, and years to replace.
Specifications should take minutes to explain, hours to design, days to write,
weeks to prove, months to become mature, and years to replace.
Specifications have no special status except that accorded by the community.
## Architecture
@@ -80,47 +98,54 @@ Specifications have no special status except that accorded by the community.
COSS is designed around fast, easy to use communications tools.
Primarily, COSS uses a wiki model for editing and publishing specifications texts.
* The *domain* is the conservancy for a set of specifications.
* The *domain* is implemented as an Internet domain.
* Each specification is a document together with references and attached resources.
* A *sub-domain* is a initiative under a specific domain.
- The *domain* is the conservancy for a set of specifications.
- The *domain* is implemented as an Internet domain.
- Each specification is a document together with references and attached resources.
- A *sub-domain* is a initiative under a specific domain.
Individuals can become members of the *domain* by completing the necessary legal clearance.
The copyright, patent, and trademark policies of the domain must be clarified in an Intellectual Property policy that applies to the domain.
Individuals can become members of the *domain*
by completing the necessary legal clearance.
The copyright, patent, and trademark policies of the domain must be clarified
in an Intellectual Property policy that applies to the domain.
Specifications exist as multiple pages, one page per version,
(discussed below in "Branching and Merging"),
which should be assigned URIs that MAY include an number identifier.
Thus, we refer to new specifications by specifying its domain, its sub-domain and short name.
Thus, we refer to new specifications by specifying its domain,
its sub-domain and short name.
The syntax for a new specification reference is:
<domain>/<sub-domain>/<shortname>
For example, this specification should be **rfc.vac.dev/vac/COSS**,
For example, this specification should be **rfc.vac.dev/vac/COSS**,
if the status were **raw**.
A number will be assigned to the specification when obtaining **draft** status.
A number will be assigned to the specification when obtaining **draft** status.
New versions of the same specification will be assigned a new number.
The syntax for a specification reference is:
<domain>/<sub-domain>/<number>/<shortname>
For example, this specification is **rfc.vac.dev/vac/1/COSS**.
The short form **1/COSS** may be used when referring to the specification from other specifications in the same domain.
The short form **1/COSS** may be used when referring to the specification
from other specifications in the same domain.
Specifications (excluding raw specifications) carries a different number including branches.
Specifications (excluding raw specifications)
carries a different number including branches.
## COSS Lifecycle
Every specification has an independent lifecycle that documents clearly its current status.
Every specification has an independent lifecycle that
documents clearly its current status.
For a specification to receive a lifecycle status,
a new specification SHOULD be presented by the team of the sub-domain.
After discussion amongst the contributors has reached a rough consensus,
as described in [RFC7282](https://www.rfc-editor.org/rfc/rfc7282.html),
the specification MAY begin the process to upgrade it's status.
A specification has five possible states that reflect its maturity and contractual weight:
A specification has five possible states that reflect its maturity and
contractual weight:
![Lifecycle diagram](./images/lifecycle.png)
@@ -128,12 +153,13 @@ A specification has five possible states that reflect its maturity and contractu
All new specifications are **raw** specifications.
Changes to raw specifications can be unilateral and arbitrary.
A sub-domain MAY use the **raw** status for new specifications that live under their domain.
A sub-domain MAY use the **raw** status for new specifications
that live under their domain.
Raw specifications have no contractual weight.
### Draft Specifications
When raw specifications can be demonstrated,
When raw specifications can be demonstrated,
they become **draft** specifications and are assigned numbers.
Changes to draft specifications should be done in consultation with users.
Draft specifications are contracts between the editors and implementers.
@@ -141,72 +167,89 @@ Draft specifications are contracts between the editors and implementers.
### Stable Specifications
When draft specifications are used by third parties, they become **stable** specifications.
Changes to stable specifications should be restricted to cosmetic ones, errata and clarifications.
Changes to stable specifications should be restricted to cosmetic ones,
errata and clarifications.
Stable specifications are contracts between editors, implementers, and end-users.
### Deprecated Specifications
When stable specifications are replaced by newer draft specifications, they become **deprecated** specifications.
Deprecated specifications should not be changed except to indicate their replacements, if any.
When stable specifications are replaced by newer draft specifications,
they become **deprecated** specifications.
Deprecated specifications should not be changed except
to indicate their replacements, if any.
Deprecated specifications are contracts between editors, implementers and end-users.
### Retired Specifications
When deprecated specifications are no longer used in products, they become **retired** specifications.
When deprecated specifications are no longer used in products,
they become **retired** specifications.
Retired specifications are part of the historical record.
They should not be changed except to indicate their replacements, if any.
Retired specifications have no contractual weight.
### Deleted Specifications
Deleted specifications are those that have not reached maturity (stable) and were discarded.
Deleted specifications are those that have not reached maturity (stable) and
were discarded.
They should not be used and are only kept for their historical value.
Only Raw and Draft specifications can be deleted.
## Editorial control
A specification MUST have a single responsible editor,
the only person who SHALL change the status of the specification through the lifecycle stages.
the only person who SHALL change the status of the specification
through the lifecycle stages.
A specification MAY also have additional contributors who contribute changes to it.
It is RECOMMENDED to use a process similar to [C4 process](https://github.com/unprotocols/rfc/blob/master/1/README.md)
to maximize the scale and diversity of contributions.
Unlike the original C4 process however, it is RECOMMENDED to use CC0 as a more permissive license alternative.
Unlike the original C4 process however,
it is RECOMMENDED to use CC0 as a more permissive license alternative.
We SHOULD NOT use GPL or GPL-like license.
One exception is this specification, as this was the original license for this specification.
The editor is responsible for accurately maintaining the state of specifications,
for retiring different versions that may live in other places and
The editor is responsible for accurately maintaining the state of specifications,
for retiring different versions that may live in other places and
for handling all comments on the specification.
## Branching and Merging
Any member of the domain MAY branch a specification at any point.
This is done by copying the existing text, and creating a new specification with the same name and content, but a new number.
This is done by copying the existing text, and
creating a new specification with the same name and content, but a new number.
Since **raw** specifications are not assigned a number,
branching by any member of a sub-domain MAY differentiate specifications based on date, contributors, or
branching by any member of a sub-domain MAY differentiate specifications
based on date, contributors, or
version number within the document.
The ability to branch a specification is necessary in these circumstances:
* To change the responsible editor for a specification, with or without the cooperation of the current responsible editor.
* To rejuvenate a specification that is stable but needs functional changes.
This is the proper way to make a new version of a specification that is in stable or deprecated status.
* To resolve disputes between different technical opinions.
- To change the responsible editor for a specification,
with or without the cooperation of the current responsible editor.
- To rejuvenate a specification that is stable but needs functional changes.
This is the proper way to make a new version of a specification
that is in stable or deprecated status.
- To resolve disputes between different technical opinions.
The responsible editor of a branched specification is the person who makes the branch.
Branches, including added contributions, are derived works and thus licensed under the same terms as the original specification.
This means that contributors are guaranteed the right to merge changes made in branches back into their original specifications.
Branches, including added contributions, are derived works and
thus licensed under the same terms as the original specification.
This means that contributors are guaranteed the right to merge changes made in branches
back into their original specifications.
Technically speaking, a branch is a *different* specification, even if it carries the same name.
Technically speaking, a branch is a *different* specification,
even if it carries the same name.
Branches have no special status except that accorded by the community.
## Conflict resolution
COSS resolves natural conflicts between teams and vendors by allowing anyone to define a new specification.
There is no editorial control process except that practised by the editor of a new specification.
The administrators of a domain (moderators) may choose to interfere in editorial conflicts,
COSS resolves natural conflicts between teams and
vendors by allowing anyone to define a new specification.
There is no editorial control process except
that practised by the editor of a new specification.
The administrators of a domain (moderators)
may choose to interfere in editorial conflicts,
and may suspend or ban individuals for behaviour they consider inappropriate.
## Specification Structure
@@ -214,7 +257,8 @@ and may suspend or ban individuals for behaviour they consider inappropriate.
### Meta Information
Specifications MUST contain the following metadata.
It is RECOMMENDED that specification metadata is specified as a YAML header (where possible).
It is RECOMMENDED that specification metadata is specified as a YAML header
(where possible).
This will enable programmatic access to specification metadata.
| Key | Value | Type | Example |
@@ -225,55 +269,60 @@ This will enable programmatic access to specification metadata.
| **category** | category | string | Best Current Practice |
| **tags** | 0 or several tags | list | waku-application, waku-core-protocol |
| **editor** | editor name/email | string | Oskar Thoren <oskarth@titanproxy.com> |
| **contributors** | contributors | list | - Pieter Hintjens <ph@imatix.com><br> - André Rebentisch <andre@openstandards.de><br> - Alberto Barrionuevo <abarrio@opentia.es><br> - Chris Puttick <chris.puttick@thehumanjourney.net><br> - Yurii Rashkovskii <yrashk@gmail.com> |
| **contributors** | contributors | list | - Pieter Hintjens <ph@imatix.com> - André Rebentisch <andre@openstandards.de> - Alberto Barrionuevo <abarrio@opentia.es> - Chris Puttick <chris.puttick@thehumanjourney.net> - Yurii Rashkovskii <yrashk@gmail.com> |
### IFT/Vac RFC Process
> [!Note]
This section is introduced to allow contributors to understand the IFT
This section is introduced to allow contributors to understand the IFT
(Institute of Free Technology) Vac RFC specification process.
Other organizations may make changes to this section according to their needs.
Vac is a department under the IFT organization that provides RFC (Request For Comments) specification services.
Vac is a department under the IFT organization that provides RFC (Request For Comments)
specification services.
This service works to help facilitate the RFC process, assuring standards are followed.
Contributors within the service SHOULD assist a *sub-domain* in creating a new specification,
editing a specification, and promoting the status of a specification along with other tasks.
Contributors within the service SHOULD assist a *sub-domain* in creating a new specification,
editing a specification, and
promoting the status of a specification along with other tasks.
Once a specification reaches some level of maturity by rough consensus,
the specification SHOULD enter the [Vac RFC](rfc.vac.dev) process.
Similar to the IETF working group adoption described in [RFC6174](https://www.rfc-editor.org/rfc/rfc6174.html),
the specification SHOULD enter the [Vac RFC](https://rfc.vac.dev/) process.
Similar to the IETF working group adoption described in [RFC6174](https://www.rfc-editor.org/rfc/rfc6174.html),
the Vac RFC process SHOULD facilitate all updates to the specification.
Specifications are introduced by projects,
Specifications are introduced by projects,
under a specific *domain*, with the intention of becoming technically mature documents.
The IFT domain currently houses the following projects:
- [Status](status.app)
- [Status](https://status.app/)
- [Waku](https://waku.org/)
- [Codex](https://codex.storage/)
- [Nimbus](https://nimbus.team/)
- [Nomos](https://nomos.tech/)
When a specification is promoted to *draft* status,
the number that is assigned MAY be incremental
When a specification is promoted to *draft* status,
the number that is assigned MAY be incremental
or by the *sub-domain* and the Vac RFC process.
Standards track specifications MUST be based on the [Vac RFC template](../template.md) before obtaining a new status.
Standards track specifications MUST be based on the
[Vac RFC template](../template.md) before obtaining a new status.
All changes, comments, and contributions SHOULD be documented.
## Conventions
Where possible editors and contributors are encouraged to:
* Refer to and build on existing work when possible, especially IETF specifications.
* Contribute to existing specifications rather than reinvent their own.
* Use collaborative branching and merging as a tool for experimentation.
* Use Semantic Line Breaks: https://sembr.org/.
- Refer to and build on existing work when possible, especially IETF specifications.
- Contribute to existing specifications rather than reinvent their own.
- Use collaborative branching and merging as a tool for experimentation.
- Use Semantic Line Breaks: [sembr](https://sembr.org/).
## Appendix A. Color Coding
It is RECOMMENDED to use color coding to indicate specification's status. Color coded specifications SHOULD use the following color scheme:
It is RECOMMENDED to use color coding to indicate specification's status.
Color coded specifications SHOULD use the following color scheme:
* ![raw](https://raw.githubusercontent.com/unprotocols/rfc/master/2/raw.svg)
* ![draft](https://raw.githubusercontent.com/unprotocols/rfc/master/2/draft.svg)
* ![stable](https://raw.githubusercontent.com/unprotocols/rfc/master/2/stable.svg)
* ![deprecated](https://raw.githubusercontent.com/unprotocols/rfc/master/2/deprecated.svg)
* ![retired](https://raw.githubusercontent.com/unprotocols/rfc/master/2/retired.svg)
* ![deleted](https://raw.githubusercontent.com/unprotocols/rfc/master/2/deleted.svg)
- ![raw](https://raw.githubusercontent.com/unprotocols/rfc/master/2/raw.svg)
- ![draft](https://raw.githubusercontent.com/unprotocols/rfc/master/2/draft.svg)
- ![stable](https://raw.githubusercontent.com/unprotocols/rfc/master/2/stable.svg)
- ![deprecated](https://raw.githubusercontent.com/unprotocols/rfc/master/2/deprecated.svg)
- ![retired](https://raw.githubusercontent.com/unprotocols/rfc/master/2/retired.svg)
- ![deleted](https://raw.githubusercontent.com/unprotocols/rfc/master/2/deleted.svg)

View File

@@ -9,9 +9,14 @@ contributors:
- Oskar Thorén <oskarth@titanproxy.com>
---
In this specification, we describe a minimum viable protocol for data synchronization inspired by the Bramble Synchronization Protocol[^1]. This protocol is designed to ensure reliable messaging between peers across an unreliable peer-to-peer (P2P) network where they may be unreachable or unresponsive.
In this specification, we describe a minimum viable protocol for
data synchronization inspired by the Bramble Synchronization Protocol[^1].
This protocol is designed to ensure reliable messaging
between peers across an unreliable peer-to-peer (P2P) network where
they may be unreachable or unresponsive.
We present a reference implementation[^2] including a simulation to demonstrate its performance.
We present a reference implementation[^2]
including a simulation to demonstrate its performance.
## Definitions
@@ -25,7 +30,11 @@ We present a reference implementation[^2] including a simulation to demonstrate
### Secure Transport
This specification does not define anything related to the transport of packets. It is assumed that this is abstracted in such a way that any secure transport protocol could be easily implemented. Likewise, properties such as confidentiality, integrity, authenticity and forward secrecy are assumed to be provided by a layer below.
This specification does not define anything related to the transport of packets.
It is assumed that this is abstracted in such a way that
any secure transport protocol could be easily implemented.
Likewise, properties such as confidentiality, integrity, authenticity and
forward secrecy are assumed to be provided by a layer below.
### Payloads
@@ -50,22 +59,29 @@ message Message {
}
```
*The payload field numbers are kept more "unique" to ensure no overlap with other protocol buffers.*
*The payload field numbers are kept more "unique" to*
*ensure no overlap with other protocol buffers.*
Each payload contains the following fields:
- **Acks:** This field contains a list (can be empty) of `message identifiers` informing the recipient that sender holds a specific message.
- **Offers:** This field contains a list (can be empty) of `message identifiers` that the sender would like to give to the recipient.
- **Requests:** This field contains a list (can be empty) of `message identifiers` that the sender would like to receive from the recipient.
- **Acks:** This field contains a list (can be empty)
of `message identifiers` informing the recipient that sender holds a specific message.
- **Offers:** This field contains a list (can be empty)
of `message identifiers` that the sender would like to give to the recipient.
- **Requests:** This field contains a list (can be empty)
of `message identifiers` that the sender would like to receive from the recipient.
- **Messages:** This field contains a list of messages (can be empty).
**Message Identifiers:** Each `message` has a message identifier calculated by hashing the `group_id`, `timestamp` and `body` fields as follows:
**Message Identifiers:** Each `message` has a message identifier calculated by
hashing the `group_id`, `timestamp` and `body` fields as follows:
```
```js
HASH("MESSAGE_ID", group_id, timestamp, body);
```
**Group Identifiers:** Each `message` is assigned into a **group** using the `group_id` field, groups are independent synchronization contexts between peers.
**Group Identifiers:** Each `message` is assigned into a **group**
using the `group_id` field,
groups are independent synchronization contexts between peers.
The current `HASH` function used is `sha256`.
@@ -73,50 +89,68 @@ The current `HASH` function used is `sha256`.
### State
We refer to `state` as set of records for the types `OFFER`, `REQUEST` and `MESSAGE` that every node SHOULD store per peer. `state` MUST NOT contain `ACK` records as we do not retransmit those periodically. The following information is stored for records:
We refer to `state` as set of records for the types `OFFER`, `REQUEST` and
`MESSAGE` that every node SHOULD store per peer.
`state` MUST NOT contain `ACK` records as we do not retransmit those periodically.
The following information is stored for records:
- **Type** - Either `OFFER`, `REQUEST` or `MESSAGE`
- **Send Count** - The amount of times a record has been sent to a peer.
- **Send Epoch** - The next epoch at which a record can be sent to a peer.
- **Type** - Either `OFFER`, `REQUEST` or `MESSAGE`
- **Send Count** - The amount of times a record has been sent to a peer.
- **Send Epoch** - The next epoch at which a record can be sent to a peer.
### Flow
A maximum of one payload SHOULD be sent to peers per epoch, this payload contains all `ACK`, `OFFER`, `REQUEST` and `MESSAGE` records for the specific peer. Payloads are created every epoch, containing reactions to previously received records by peers or new records being sent out by nodes.
A maximum of one payload SHOULD be sent to peers per epoch,
this payload contains all `ACK`, `OFFER`, `REQUEST` and
`MESSAGE` records for the specific peer.
Payloads are created every epoch,
containing reactions to previously received records by peers or
new records being sent out by nodes.
Nodes MAY have two modes with which they can send records: `BATCH` and `INTERACTIVE` mode. The following rules dictate how nodes construct payloads every epoch for any given peer for both modes.
Nodes MAY have two modes with which they can send records:
`BATCH` and `INTERACTIVE` mode.
The following rules dictate how nodes construct payloads
every epoch for any given peer for both modes.
> ***NOTE:** A node may send messages both in interactive and in batch mode.*
#### Interactive Mode
- A node initially offers a `MESSAGE` when attempting to send it to a peer. This means an `OFFER` is added to the next payload and state for the given peer.
- When a node receives an `OFFER`, a `REQUEST` is added to the next payload and state for the given peer.
- When a node receives a `REQUEST` for a previously sent `OFFER`, the `OFFER` is removed from the state and the corresponding `MESSAGE` is added to the next payload and state for the given peer.
- When a node receives a `MESSAGE`, the `REQUEST` is removed from the state and an `ACK` is added to the next payload for the given peer.
- When a node receives an `ACK`, the `MESSAGE` is removed from the state for the given peer.
- All records that require retransmission are added to the payload, given `Send Epoch` has been reached.
- A node initially offers a `MESSAGE` when attempting to send it to a peer.
This means an `OFFER` is added to the next payload and state for the given peer.
- When a node receives an `OFFER`, a `REQUEST` is added to the next payload and
state for the given peer.
- When a node receives a `REQUEST` for a previously sent `OFFER`,
the `OFFER` is removed from the state and
the corresponding `MESSAGE` is added to the next payload and
state for the given peer.
- When a node receives a `MESSAGE`, the `REQUEST` is removed from the state and
an `ACK` is added to the next payload for the given peer.
- When a node receives an `ACK`,
the `MESSAGE` is removed from the state for the given peer.
- All records that require retransmission are added to the payload,
given `Send Epoch` has been reached.
<p align="center">
<img src="./images/interactive.png" />
<br />
Figure 1: Delivery without retransmissions in interactive mode.
</p>
![notification](./images/interactive.png)
Figure 1: Delivery without retransmissions in interactive mode.
#### Batch Mode
1. When a node sends a `MESSAGE`, it is added to the next payload and the state for the given peer.
2. When a node receives a `MESSAGE`, an `ACK` is added to the next payload for the corresponding peer.
3. When a node receives an `ACK`, the `MESSAGE` is removed from the state for the given peer.
4. All records that require retransmission are added to the payload, given `Send Epoch` has been reached.
1. When a node sends a `MESSAGE`,
it is added to the next payload and the state for the given peer.
2. When a node receives a `MESSAGE`,
an `ACK` is added to the next payload for the corresponding peer.
3. When a node receives an `ACK`,
the `MESSAGE` is removed from the state for the given peer.
4. All records that require retransmission are added to the payload,
given `Send Epoch` has been reached.
<!-- diagram -->
<p align="center">
<img src="./images/batch.png" />
<br />
Figure 2: Delivery without retransmissions in batch mode.
</p>
![notification](./images/batch.png)
Figure 2: Delivery without retransmissions in batch mode.
> ***NOTE:** Batch mode is higher bandwidth whereas interactive mode is higher latency.*
@@ -124,21 +158,28 @@ Nodes MAY have two modes with which they can send records: `BATCH` and `INTERACT
### Retransmission
The record of the type `Type` SHOULD be retransmitted every time `Send Epoch` is smaller than or equal to the current epoch.
The record of the type `Type` SHOULD be retransmitted
every time `Send Epoch` is smaller than or equal to the current epoch.
`Send Epoch` and `Send Count` MUST be increased every time a record is retransmitted. Although no function is defined on how to increase `Send Epoch`, it SHOULD be exponentially increased until reaching an upper bound where it then goes back to a lower epoch in order to prevent a record's `Send Epoch`'s from becoming too large.
`Send Epoch` and `Send Count` MUST be increased every time a record is retransmitted.
Although no function is defined on how to increase `Send Epoch`,
it SHOULD be exponentially increased until reaching an upper bound
where it then goes back to a lower epoch in order to
prevent a record's `Send Epoch`'s from becoming too large.
> ***NOTE:** We do not retransmission `ACK`s as we do not know when they have arrived, therefore we simply resend them every time we receive a `MESSAGE`.*
> ***NOTE:** We do not retransmission `ACK`s as we do not know when they have arrived,
therefore we simply resend them every time we receive a `MESSAGE`.*
## Formal Specification
MVDS has been formally specified using TLA+: <https://github.com/vacp2p/formalities/tree/master/MVDS>.
## Acknowledgments
- Preston van Loon
- Greg Markou
- Rene Nayman
- Jacek Sieka
- Preston van Loon
- Greg Markou
- Rene Nayman
- Jacek Sieka
## Copyright

View File

@@ -7,44 +7,56 @@ editor: Hanno Cornelius <hanno@status.im>
contributors:
---
`25/LIBP2P-DNS-DISCOVERY` specifies a scheme to implement [`libp2p`](https://libp2p.io/) peer discovery via DNS for Waku v2.
The generalised purpose is to retrieve an arbitrarily long, authenticated, updateable list of [`libp2p` peers](https://docs.libp2p.io/concepts/peer-id/) to bootstrap connection to a `libp2p` network.
Since [`10/WAKU2`](../../waku/standards/core/10/waku2.md) currently specifies use of [`libp2p` peer identities](https://docs.libp2p.io/concepts/peer-id/),
this method is suitable for a new Waku v2 node to discover other Waku v2 nodes to connect to.
`25/LIBP2P-DNS-DISCOVERY` specifies a scheme to implement [`libp2p`](https://libp2p.io/)
peer discovery via DNS for Waku v2.
The generalised purpose is to retrieve an arbitrarily long, authenticated,
updateable list of [`libp2p` peers](https://docs.libp2p.io/concepts/peer-id/)
to bootstrap connection to a `libp2p` network.
Since [`10/WAKU2`](../../waku/standards/core/10/waku2.md)
currently specifies use of [`libp2p` peer identities](https://docs.libp2p.io/concepts/peer-id/),
this method is suitable for a new Waku v2 node
to discover other Waku v2 nodes to connect to.
This specification is largely based on [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459),
with the only deviation being the type of address being encoded (`multiaddr` vs `enr`).
Also see [this earlier explainer](https://vac.dev/dns-based-discovery) for more background on the suitability of DNS based discovery for Waku v2.
Also see [this earlier explainer](https://vac.dev/dns-based-discovery)
for more background on the suitability of DNS based discovery for Waku v2.
# List encoding
## List encoding
The peer list MUST be encoded as a [Merkle tree](https://www.wikiwand.com/en/Merkle_tree).
EIP-1459 specifies [the URL scheme](https://eips.ethereum.org/EIPS/eip-1459#specification) to refer to such a DNS node list.
EIP-1459 specifies [the URL scheme](https://eips.ethereum.org/EIPS/eip-1459#specification)
to refer to such a DNS node list.
This specification uses the same approach, but with a `matree` scheme:
```
```yaml
matree://<key>@<fqdn>
```
where
- `matree` is the selected `multiaddr` Merkle tree scheme
- `<fqdn>` is the fully qualified domain name on which the list can be found
- `<key>` is the base32 encoding of the compressed 32-byte binary public key that signed the list.
- `<key>` is the base32 encoding of the compressed 32-byte binary public key
that signed the list.
The example URL from EIP-1459, adapted to the above scheme becomes:
```
```yaml
matree://AM5FCQLWIZX2QFPNJAP7VUERCCRNGRHWZG3YYHIUV7BVDQ5FDPRT2@peers.example.org
```
Each entry within the Merkle tree MUST be contained within a [DNS TXT record](https://www.rfc-editor.org/rfc/rfc1035.txt)
and stored in a subdomain (except for the base URL `matree` entry).
The content of any TXT record MUST be small enough to fit into the 512-byte limit imposed on UDP DNS packets,
The content of any TXT record
MUST be small enough to fit into the 512-byte limit imposed on UDP DNS packets,
which limits the number of hashes that can be contained within a branch entry.
The subdomain name for each entry is the base32 encoding of the abbreviated keccak256 hash of its text content.
See [this example](https://eips.ethereum.org/EIPS/eip-1459#dns-record-structure) of a fully populated tree for more information.
The subdomain name for each entry
is the base32 encoding of the abbreviated keccak256 hash of its text content.
See [this example](https://eips.ethereum.org/EIPS/eip-1459#dns-record-structure)
of a fully populated tree for more information.
# Entry types
## Entry types
The following entry types are derived from [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459)
and adapted for use with `multiaddrs`:
@@ -53,11 +65,12 @@ and adapted for use with `multiaddrs`:
The tree root entry MUST use the following format:
```
```yaml
matree-root:v1 m=<ma-root> l=<link-root> seq=<sequence number> sig=<signature>
```
where
- `ma-root` and `link-root` refer to the root hashes of subtrees
containing `multiaddrs` and links to other subtrees, respectively
- `sequence-number` is the tree's update sequence number.
@@ -71,11 +84,12 @@ encoded as URL-safe base64
Branch entries MUST take the format:
```
```yaml
matree-branch:<h₁>,<h₂>,...,<hₙ>
```
where
- `<h₁>,<h₂>,...,<hₙ>` are the hashes of other subtree entries
## Leaf entries
@@ -87,7 +101,7 @@ There are two types of leaf entries:
For the subtree pointed to by `link-root`,
leaf entries MUST take the format:
```
```yaml
matree://<key>@<fqdn>
```
@@ -98,37 +112,42 @@ which links to a different list located in another domain.
For the subtree pointed to by `ma-root`,
leaf entries MUST take the format:
```
```yaml
ma:<multiaddr>
```
which contains the `multiaddr` of a `libp2p` peer.
# Client protocol
## Client protocol
A client MUST adhere to the [client protocol](https://eips.ethereum.org/EIPS/eip-1459#client-protocol) as specified in EIP-1459,
A client MUST adhere to the [client protocol](https://eips.ethereum.org/EIPS/eip-1459#client-protocol)
as specified in EIP-1459,
and adapted for usage with `multiaddr` entry types below:
To find nodes at a given DNS name a client MUST perform the following steps:
1. Resolve the TXT record of the DNS name and check whether it contains a valid `matree-root:v1` entry.
1. Resolve the TXT record of the DNS name and
check whether it contains a valid `matree-root:v1` entry.
2. Verify the signature on the root against the known public key
and check whether the sequence number is larger than or equal to any previous number seen for that name.
and check whether the sequence number is larger than or
equal to any previous number seen for that name.
3. Resolve the TXT record of a hash subdomain indicated in the record
and verify that the content matches the hash.
4. If the resolved entry is of type:
- `matree-branch`: parse the list of hashes and continue resolving them (step 3).
- `ma`: import the `multiaddr` and add it to a local list of discovered nodes.
# Copyright
- `matree-branch`: parse the list of hashes and continue resolving them (step 3).
- `ma`: import the `multiaddr` and add it to a local list of discovered nodes.
## Copyright
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
# References
## References
1. [`10/WAKU2`](../../waku/standards/core/10/waku2.md)
1. [EIP-1459: Client Protocol](https://eips.ethereum.org/EIPS/eip-1459#client-protocol)
1. [EIP-1459: Node Discovery via DNS ](https://eips.ethereum.org/EIPS/eip-1459)
1. [EIP-1459: Node Discovery via DNS](https://eips.ethereum.org/EIPS/eip-1459)
1. [`libp2p`](https://libp2p.io/)
1. [`libp2p` peer identity](https://docs.libp2p.io/concepts/peer-id/)
1. [Merkle trees](https://www.wikiwand.com/en/Merkle_tree)

View File

@@ -8,7 +8,8 @@ contributors:
- Dean Eigenmann <dean@status.im>
---
A remote log is a replication of a local log. This means a node can read data that originally came from a node that is offline.
A remote log is a replication of a local log.
This means a node can read data that originally came from a node that is offline.
This specification is complemented by a proof of concept implementation[^1].
@@ -112,7 +113,12 @@ message RemoteLog {
<!-- TODO: Better name for Pair, Mapping? -->
<!-- TODO: Consider making more useful in conjunction with metadata field. It makes sense to explicitly list what sequence a message is <local hash, remote hash, data, seqid> this way I can easily sync a messages prior or after a specific number. To enable this to be dynamic it might make sense to add page info so that I am aware which page I can find seqid on -->
<!-- TODO: Consider making more useful in conjunction with metadata field.
It makes sense to explicitly list what sequence a message is <local hash,
remote hash, data, seqid> this way I can easily sync a messages prior or
after a specific number.
To enable this to be dynamic it might make sense to add page info so
that I am aware which page I can find seqid on -->
## Synchronization
@@ -122,12 +128,13 @@ There are four fundamental roles:
1. Alice
2. Bob
2. Name system (NS)
3. Content-addressed storage (CAS)
3. Name system (NS)
4. Content-addressed storage (CAS)
The *remote log* protobuf is what is stored in the name system.
"Bob" can represent anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS.
"Bob" can represent anything from 0 to N participants. Unlike Alice,
Bob only needs read-only access to NS and CAS.
<!-- TODO: Document random node as remote log -->
<!-- TODO: Document how to find initial remote log (e.g. per sync contexts -->
@@ -136,11 +143,7 @@ The *remote log* protobuf is what is stored in the name system.
<!-- diagram -->
<p align="center">
<img src="./images/remote-log.png" />
<br />
Figure 1: Remote log data synchronization.
</p>
![notification](./images/remote-log.png)
<!-- Document the flow wrt operations -->
@@ -157,7 +160,7 @@ modes:
**Data format:**
```
```yaml
| H1_3 | H2_3 |
| H1_2 | H2_2 |
| H1_1 | H2_1 |
@@ -177,7 +180,7 @@ A remote log MAY also choose to embed the wire payloads that corresponds to the
native hash. This bypasses the need for a dedicated CAS and additional
round-trips, with a trade-off in bandwidth usage.
```
```yaml
| H1_3 | | C_3 |
| H1_2 | | C_2 |
| H1_1 | | C_1 |
@@ -198,13 +201,16 @@ log. The latter is useful for things like backups on durable storage.
The pointer to the 'next page' is another remote log entry, at a previous point
in time.
<!-- TODO: Determine requirement re overlapping, adjacent, and/or missing entries -->
<!-- TODO: Determine requirement re overlapping, adjacent,
and/or missing entries -->
<!-- TODO: Document message ordering append only requirements -->
### Interaction with MVDS
[vac.mvds.Message](../2/mvds.md/#payloads) payloads are the only payloads that MUST be uploaded. Other messages types MAY be uploaded, depending on the implementation.
[vac.mvds.Message](../2/mvds.md/#payloads) payloads are the only payloads
that MUST be uploaded.
Other messages types MAY be uploaded, depending on the implementation.
## Acknowledgments

View File

@@ -15,32 +15,43 @@ contributors:
## Abstract
The following specification covers the RLN construct as well as some auxiliary libraries useful for interacting with it.
Rate limiting nullifier (RLN) is a construct based on zero-knowledge proofs that provides an anonymous rate-limited signaling/messaging framework suitable for decentralized (and centralized) environments.
The following specification covers the RLN construct
as well as some auxiliary libraries useful for interacting with it.
Rate limiting nullifier (RLN) is a construct based on zero-knowledge proofs that
provides an anonymous rate-limited signaling/messaging framework
suitable for decentralized (and centralized) environments.
Anonymity refers to the unlinkability of messages to their owner.
## Motivation
RLN guarantees a messaging rate is enforced cryptographically while preserving the anonymity of the message owners.
A wide range of applications can benefit from RLN and provide desirable security features.
For example,
an e-voting system can integrate RLN to contain the voting rate while protecting the voters-vote unlinkability.
Another use case is to protect an anonymous messaging system against DDoS and
RLN guarantees a messaging rate is enforced cryptographically
while preserving the anonymity of the message owners.
A wide range of applications can benefit from RLN and
provide desirable security features.
For example,
an e-voting system can integrate RLN to contain the voting rate while
protecting the voters-vote unlinkability.
Another use case is to protect an anonymous messaging system against DDoS and
spam attacks by constraining messaging rate of users.
This latter use case is explained in [17/WAKU2-RLN-RELAY RFC](../../waku/standards/core/17/rln-relay.md).
## Wire Format Specification
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
“SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
### Flow
The users participate in the protocol by first registering to an application-defined group referred by the _membership group_.
The users participate in the protocol by
first registering to an application-defined group referred by the _membership group_.
Registration to the group is mandatory for signaling in the application.
After registration, group members can generate a zero-knowledge proof of membership for their signals and
can participate in the application.
Usually, the membership requires a financial or
social stake which is beneficial for the prevention of inclusion of Sybils within the _membership group_.
Group members are allowed to send one signal per external nullifier
After registration, group members can generate a zero-knowledge proof of membership
for their signals and can participate in the application.
Usually, the membership requires a financial or
social stake which is beneficial for the prevention
of inclusion of Sybils within the _membership group_.
Group members are allowed to send one signal per external nullifier
(an identifier that groups signals and can be thought of as a voting booth).
If a user generates more signals than allowed,
the user risks being slashed - by revealing his membership secret credentials.
@@ -54,20 +65,22 @@ Generally the flow can be described by the following steps:
### Registration
Depending on the application requirements, the registration can be implemented in different ways, for example:
Depending on the application requirements,
the registration can be implemented in different ways, for example:
- centralized registrations, by using a central server
- decentralized registrations, by using a smart contract
The users' identity commitments
The users' identity commitments
(explained in section [User Identity](#user-identity)) are stored in a Merkle tree,
and the users can obtain a Merkle proof proving that they are part of the group.
Also depending on the application requirements,
usually a financial or social stake is introduced.
An example for financial stake is:
An example for financial stake is:
For each registration a certain amount of ETH is required.
An example for social stake is using [Interep](https://interep.link/) as a registry -
An example for social stake is using [Interep](https://interep.link/) as a registry,
users need to prove that they have a highly reputable social media account.
#### Implementation notes
@@ -85,38 +98,45 @@ The user's identity is composed of:
```
For registration, the user MUST submit their `identity_commitment`
For registration, the user MUST submit their `identity_commitment`
(along with any additional registration requirements) to the registry.
Upon registration, they SHOULD receive `leaf_index` value which represents their position in the Merkle tree.
Upon registration, they SHOULD receive `leaf_index` value
which represents their position in the Merkle tree.
Receiving a `leaf_index` is not a hard requirement and is application specific.
The other way around is the users calculating the `leaf_index` themselves upon successful registration.
The other way around is
the users calculating the `leaf_index` themselves upon successful registration.
### Signaling
After registration,
the users can participate in the application by sending signals to the other participants in a decentralised manner or
the users can participate in the application by
sending signals to the other participants in a decentralised manner or
to a centralised server.
Along with their signal,
they MUST generate a zero-knowledge proof by using the circuit with the specification described above.
they MUST generate a zero-knowledge proof by
using the circuit with the specification described above.
For generating a proof,
the users need to obtain the required parameters or compute them themselves,
depending on the application implementation and client libraries supported by the application.
For example,
depending on the application implementation and
client libraries supported by the application.
For example,
the users MAY store the membership Merkle tree on their end and
generate a Merkle proof whenever they want to generate a signal.
#### Implementation notes
#### Implementation Notes
##### Signal hash
The signal hash can be generated by hashing the raw signal (or content) using the `keccak256` hash function.
The signal hash can be generated by hashing the raw signal (or content)
using the `keccak256` hash function.
##### External nullifier
The external nullifier MUST be computed as the Poseidon hash of the current epoch
(e.g. a value equal to or derived from the current UNIX timestamp divided by the epoch length) and
the RLN identifier.
The external nullifier MUST be computed as the Poseidon hash of the current epoch
(e.g. a value equal to or
derived from the current UNIX timestamp divided by the epoch length)
and the RLN identifier.
```js
@@ -124,7 +144,7 @@ external_nullifier = poseidonHash([epoch, rln_identifier]);
```
##### Obtaining Merkle proof
##### Obtaining Merkle proof
The Merkle proof SHOULD be obtained locally or from a trusted third party.
By using the [incremental Merkle tree algorithm](https://github.com/appliedzkp/incrementalquintree/blob/master/ts/IncrementalQuinTree.ts),
@@ -141,11 +161,12 @@ The proof (`Merkle_proof`) is composed of the following fields:
```
1. **root** - The root of membership group Merkle tree at the time of publishing the message
2. **indices** - The index fields of the leafs in the Merkle tree - used by the Merkle tree algorithm for verification
3. **path_elements** - Auxiliary data structure used for storing the path to the leaf -
used by the Merkle proof algorithm for verificaton
1. **root** - The root of membership group Merkle tree
at the time of publishing the message
2. **indices** - The index fields of the leafs in the Merkle tree -
used by the Merkle tree algorithm for verification
3. **path_elements** - Auxiliary data structure used for storing the path
to the leaf - used by the Merkle proof algorithm for verificaton
##### Generating proof
@@ -167,9 +188,11 @@ the user MUST submit the following fields to the circuit:
##### Calculating output
The proof output is calculated locally,
in order for the required fields for proof verification to be sent along with the proof.
in order for the required fields for proof verification
to be sent along with the proof.
The proof output is composed of the `y` share of the secret equation and the `internal_nullifier`.
The `internal_nullifier` represents a unique fingerprint of a user for a given `epoch` and app.
The `internal_nullifier` represents a unique fingerprint of a user
for a given `epoch` and app.
The following fields are needed for proof output calculation:
```js
@@ -235,15 +258,16 @@ the slashing will be implemented only on the server.
Otherwise if the application is distributed,
the slashing will be implemented on each user's client.
#### Implementation notes
#### Notes from Implementation
Each user of the protocol
Each user of the protocol
(server or otherwise) MUST store metadata for each message received by each user,
for the given `epoch`.
The data can be deleted when the `epoch` passes.
Storing metadata is REQUIRED, so that if a user sends more than one unique signal per `epoch`,
Storing metadata is REQUIRED,
so that if a user sends more than one unique signal per `epoch`,
they can be slashed and removed from the protocol.
The metadata stored contains the `x`, `y` shares and
The metadata stored contains the `x`, `y` shares and
the `internal_nullifier` for the user for each message.
If enough such shares are present, the user's secret can be retreived.
@@ -265,27 +289,32 @@ One way of storing received metadata (`messaging_metadata`) is the following for
##### Verification
The output message verification consists of the following steps:
- `external_nullifier` correctness
- non-duplicate message check
- `zk_proof` zero-knowledge proof verification
- spam verification
**1. `external_nullifier` correctness**
Upon received `output_message`, first the `epoch` and `rln_identifier` fields are checked,
Upon received `output_message`,
first the `epoch` and `rln_identifier` fields are checked,
to ensure that the message matches the current `external_nullifier`.
If the `external_nullifier` is correct the verification continues, otherwise, the message is discarded.
If the `external_nullifier` is correct the verification continues, otherwise,
the message is discarded.
**2. non-duplicate message check**
The received message is checked to ensure it is not duplicate.
The duplicate message check is performed by verifying that the `x` and `y`
The duplicate message check is performed by verifying that the `x` and `y`
fields do not exist in the `messaging_metadata` object.
If the `x` and `y` fields exist in the `x_shares` and `y_shares` array for the `external_nullifier` and
If the `x` and `y` fields exist in the `x_shares` and
`y_shares` array for the `external_nullifier` and
the `internal_nullifier` the message can be considered as a duplicate.
Duplicate messages are discarded.
**3. `zk_proof` verification**
The `zk_proof` SHOULD be verified by providing the `zk_proof` field to the circuit verifier along with the `public_signal`:
The `zk_proof` SHOULD be verified by providing the `zk_proof` field
to the circuit verifier along with the `public_signal`:
```js
@@ -303,35 +332,42 @@ If the proof verification is correct,
the verification continues, otherwise the message is discarded.
**4. Double signaling verification**
After the proof is verified the `x`, and `y` fields are added to the `x_shares` and `y_shares`
arrays of the `messaging_metadata` `external_nullifier` and `internal_nullifier` object.
If the length of the arrays is equal to the signaling threshold (`limit`), the user can be slashed.
After the proof is verified the `x`, and
`y` fields are added to the `x_shares` and `y_shares`
arrays of the `messaging_metadata` `external_nullifier` and
`internal_nullifier` object.
If the length of the arrays is equal to the signaling threshold (`limit`),
the user can be slashed.
##### Slashing
After the verification,
the user SHOULD be slashed if two different shares are present to reconstruct their `identity_secret_hash` from `x_shares` and `y_shares` fields, for their `internal_nullifier`.
After the verification,
the user SHOULD be slashed if two different shares are present
to reconstruct their `identity_secret_hash` from `x_shares` and
`y_shares` fields, for their `internal_nullifier`.
The secret can be retreived by the properties of the Shamir's secret sharing scheme.
In particular the secret (`a_0`) can be retrieved by computing [Lagrange polynomials](https://en.wikipedia.org/wiki/Lagrange_polynomial).
After the secret is retreived,
the user's `identity_commitment` SHOULD be generated from the secret and
it can be used for removing the user from the membership Merkle tree
the user's `identity_commitment` SHOULD be generated from the secret and
it can be used for removing the user from the membership Merkle tree
(zeroing out the leaf that contains the user's `identity_commitment`).
Additionally, depending on the application the `identity_secret_hash` MAY be used for taking the user's provided stake.
Additionally, depending on the application the `identity_secret_hash`
MAY be used for taking the user's provided stake.
### Technical overview
The main RLN construct is implemented using a [ZK-SNARK](https://z.cash/technology/zksnarks/) circuit.
However, it is helpful to describe the other necessary outside components for interaction with the circuit,
The main RLN construct is implemented using a
[ZK-SNARK](https://z.cash/technology/zksnarks/) circuit.
However, it is helpful to describe
the other necessary outside components for interaction with the circuit,
which together with the ZK-SNARK circuit enable the above mentioned features.
#### Terminology
| Term | Description |
|---------------------------|-------------------------------------------------------------------------------------|
| **ZK-SNARK** | https://z.cash/technology/zksnarks/ |
| **ZK-SNARK** | [zksnarks](https://z.cash/technology/zksnarks/) |
| **Stake** | Financial or social stake required for registering in the RLN applications. Common stake examples are: locking cryptocurrency (financial), linking reputable social identity. |
| **Identity secret** | An array of two unique random components (identity nullifier and identity trapdoor), which must be kept private by the user. Secret hash and identity commitment are derived from this array. |
| **Identity nullifier** | Random 32 byte value used as component for identity secret generation. |
@@ -344,7 +380,6 @@ which together with the ZK-SNARK circuit enable the above mentioned features.
| **RLN membership tree** | Merkle tree data structure, filled with identity commitments of the users. Serves as a data structure that ensures user registrations. |
| **Merkle proof** | Proof that a user is member of the RLN membership tree. |
#### RLN Zero-Knowledge Circuit specific terms
| Term | Description |
@@ -358,14 +393,19 @@ which together with the ZK-SNARK circuit enable the above mentioned features.
#### Zero-Knowledge Circuits specification
Anonymous signaling with a controlled rate limit is enabled by proving that the user is part of a group which has high barriers to entry (form of stake) and
Anonymous signaling with a controlled rate limit
is enabled by proving that the user is part of a group
which has high barriers to entry (form of stake) and
enabling secret reveal if more than 1 unique signal is produced per external nullifier.
The membership part is implemented using membership [Merkle trees](https://en.wikipedia.org/wiki/Merkle_tree) and Merkle proofs,
The membership part is implemented using
membership [Merkle trees](https://en.wikipedia.org/wiki/Merkle_tree) and Merkle proofs,
while the secret reveal part is enabled by using the Shamir's Secret Sharing scheme.
Essentially the protocol requires the users to generate zero-knowledge proof to be able to send signals and
Essentially the protocol requires the users to generate zero-knowledge proof
to be able to send signals and
participate in the application.
The zero knowledge proof proves that the user is member of a group,
but also enforces the user to share part of their secret for each signal in an external nullifier.
but also enforces the user to share part of their secret
for each signal in an external nullifier.
The external nullifier is usually represented by timestamp or a time interval.
It can also be thought of as a voting booth in voting applications.
@@ -378,24 +418,29 @@ using the [circomlib](https://docs.circom.io/) library.
##### Circuit parameters
**Public Inputs**
###### Public Inputs
- `x`
- `external_nullifier`
**Private Inputs**
* `identity_secret_hash`
* `path_elements` - rln membership proof component
* `identity_path_index` - rln membership proof component
###### Private Inputs
- `identity_secret_hash`
- `path_elements` - rln membership proof component
- `identity_path_index` - rln membership proof component
###### Outputs
**Outputs**
- `y`
- `root` - the rln membership tree root
- `internal_nullifier`
##### Hash function
Canonical [Poseidon hash implementation](https://eprint.iacr.org/2019/458.pdf) is used,
as implemented in the [circomlib library](https://github.com/iden3/circomlib/blob/master/circuits/poseidon.circom), according to the Poseidon paper.
Canonical [Poseidon hash implementation](https://eprint.iacr.org/2019/458.pdf)
is used,
as implemented in the [circomlib library](https://github.com/iden3/circomlib/blob/master/circuits/poseidon.circom),
according to the Poseidon paper.
This Poseidon hash version (canonical implementation) uses the following parameters:
| Hash inputs | `t` | `RF` | `RP`|
@@ -408,23 +453,25 @@ This Poseidon hash version (canonical implementation) uses the following paramet
|6 | 7 | 8 | 63|
|7 | 8 | 8 | 64|
|8 | 9 | 8 | 63|
##### Membership implementation
For a valid signal, a user's `identity_commitment`
For a valid signal, a user's `identity_commitment`
(more on identity commitments below) must exist in identity membership tree.
Membership is proven by providing a membership proof (witness).
The fields from the membership proof REQUIRED for the verification are:
`path_elements` and `identity_path_index`.
[IncrementalQuinTree](https://github.com/appliedzkp/incrementalquintree) algorithm is used for constructing the Membership Merkle tree.
[IncrementalQuinTree](https://github.com/appliedzkp/incrementalquintree)
algorithm is used for constructing the Membership Merkle tree.
The circuits are reused from this repository.
You can find out more details about the IncrementalQuinTree algorithm [here](https://ethresear.ch/t/gas-and-circuit-constraint-benchmarks-of-binary-and-quinary-incremental-Merkle-trees-using-the-poseidon-hash-function/7446).
#### Slashing and Shamir's Secret Sharing
Slashing is enabled by using polynomials and [Shamir's Secret sharing](https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing).
In order to produce a valid proof, `identity_secret_hash` as a private input to the circuit.
In order to produce a valid proof,
`identity_secret_hash` as a private input to the circuit.
Then a secret equation is created in the form of:
```js
@@ -440,29 +487,38 @@ in order for their proof to be verified.
`x` is the hashed signal, while the `y` is the circuit output.
With more than one pair of unique shares, anyone can derive `a_0`, i.e. the `identity_secret_hash`.
The hash of a signal will be the evaluation point `x`.
In this way,
a member who sends more than one unique signal per `external_nullifier` risks their identity secret being revealed.
In this way,
a member who sends more than one unique signal per `external_nullifier`
risks their identity secret being revealed.
Note that shares used in different epochs and
Note that shares used in different epochs and
different RLN apps cannot be used to derive the `identity_secret_hash`.
Thanks to the `external_nullifier` definition, also shares computed from same secret within same epoch but in different RLN apps cannot be used to derive the identity secret hash.
Thanks to the `external_nullifier` definition,
also shares computed from same secret within same epoch but
in different RLN apps cannot be used to derive the identity secret hash.
The `rln_identifier` is a random value from a finite field, unique per RLN app,
and is used for additional cross-application security -
to protect the user secrets being compromised if they use the same credentials accross different RLN apps.
and is used for additional cross-application security -
to protect the user secrets being compromised if they use
the same credentials accross different RLN apps.
If `rln_identifier` is not present,
the user uses the same credentials and
the user uses the same credentials and
sends a different message for two different RLN apps using the same `external_nullifier`,
then their user signals can be grouped by the `internal_nullifier` which could lead the user's secret revealed.
This is because two separate signals under the same `internal_nullifier` can be treated as rate limiting violation.
then their user signals can be grouped by the `internal_nullifier`
which could lead the user's secret revealed.
This is because two separate signals under the same `internal_nullifier`
can be treated as rate limiting violation.
With adding the `rln_identifier` field we obscure the `internal_nullifier`,
so this kind of attack can be hardened because we don't have the same `internal_nullifier` anymore.
so this kind of attack can be hardened because
we don't have the same `internal_nullifier` anymore.
#### Identity credentials generation
In order to be able to generate valid proofs, the users MUST be part of the identity membership Merkle tree.
They are part of the identity membership Merkle tree if their `identity_commitment` is placed in a leaf in the tree.
In order to be able to generate valid proofs,
the users MUST be part of the identity membership Merkle tree.
They are part of the identity membership Merkle tree if
their `identity_commitment` is placed in a leaf in the tree.
The identity credentials of a user are composed of:
@@ -483,11 +539,13 @@ identity_secret = [identity_nullifier, identity_trapdoor];
```
The same secret SHOULD NOT be used accross different protocols,
because revealing the secret at one protocol could break privacy for the user in the other protocols.
because revealing the secret at one protocol
could break privacy for the user in the other protocols.
##### `identity_secret_hash`
The `identity_secret_hash` is generated by obtaining a Poseidon hash of the `identity_secret` array:
The `identity_secret_hash` is generated by obtaining a Poseidon hash
of the `identity_secret` array:
```js
@@ -507,47 +565,60 @@ identity_commitment = poseidonHash([identity_secret_hash]);
### Appendix A: Security Considerations
RLN is an experimental and still un-audited technology.
RLN is an experimental and still un-audited technology.
This means that the circuits have not been yet audited.
Another consideration is the security of the underlying primitives.
zk-SNARKS require a trusted setup for generating a prover and verifier keys.
The standard for this is to use trusted [Multi-Party Computation (MPC)](https://en.wikipedia.org/wiki/Secure_multi-party_computation) ceremony,
which requires two phases.
The standard for this is to use trusted
[Multi-Party Computation (MPC)](https://en.wikipedia.org/wiki/Secure_multi-party_computation)
ceremony, which requires two phases.
Trusted MPC ceremony has not yet been performed for the RLN circuits.
#### SSS Security Assumptions
Shamir-Secret Sharing requires polynomial coefficients to be independent of each other.
However, `a_1` depends on `a_0` through the Poseidon hash algorithm.
Due to the design of Poseidon,
it is possible to [attack](https://github.com/Rate-Limiting-Nullifier/rln-circuits/pull/7#issuecomment-1416085627) the protocol.
It was decided *not* to change the circuits design, since at the moment the attack is infeasible.
Therefore, implementers must be aware that the current version provides approximately 160-bit security and not 254.
Shamir-Secret Sharing requires polynomial coefficients
to be independent of each other.
However, `a_1` depends on `a_0` through the Poseidon hash algorithm.
Due to the design of Poseidon,
it is possible to
[attack](https://github.com/Rate-Limiting-Nullifier/rln-circuits/pull/7#issuecomment-1416085627)
the protocol.
It was decided _not_ to change the circuits design,
since at the moment the attack is infeasible.
Therefore, implementers must be aware that the current version
provides approximately 160-bit security and not 254.
Possible improvements:
* [change the circuit](https://github.com/Rate-Limiting-Nullifier/rln-circuits/pull/7#issuecomment-1416085627) to make coefficients independent;
* switch to other hash function (Keccak, SHA);
- [change the circuit](https://github.com/Rate-Limiting-Nullifier/rln-circuits/pull/7#issuecomment-1416085627)
to make coefficients independent;
- switch to other hash function (Keccak, SHA);
### Appendix B: Identity Scheme Choice
The hashing scheme used is based on the design decisions which also include the Semaphore circuits.
The hashing scheme used is based on the design decisions
which also include the Semaphore circuits.
Our goal was to ensure compatibility of the secrets for apps that use Semaphore and
RLN circuits while also not compromising on security because of using the same secrets.
For example, let's say there is a voting app that uses Semaphore,
and also a chat app that uses RLN.
The UX would be better if the users would not need to care about complicated identity management
(secrets and commitments) they use for each app,
The UX would be better if
the users would not need to care about complicated identity management
(secrets and commitments) they use for each app,
and it would be much better if they could use a single id commitment for this.
Also in some cases these kind of dependency is required -
RLN chat app using Interep as a registry (instead of using financial stake).
One potential concern about this interoperability is a slashed user on the RLN app side
having their security compromised on the semaphore side apps as well.
I.e obtaining the user's secret,
One potential concern about this interoperability is a slashed user
on the RLN app side having their security compromised
on the semaphore side apps as well.
i.e. obtaining the user's secret,
anyone would be able to generate valid semaphore proofs as the slashed user.
We don't want that,
and we should keep user's app specific security threats in the domain of that app alone.
We don't want that,
and we should keep user's app specific security threats
in the domain of that app alone.
To achieve the above interoperability UX while preventing the shared app security model
To achieve the above interoperability UX
while preventing the shared app security model
(i.e slashing user on an RLN app having impact on Semaphore apps),
we had to do the follow in regard the identity secret and identity commitment:
@@ -569,16 +640,16 @@ Secret components for generting RLN proof:
- `identity_secret_hash`
When a user is slashed on the RLN app side, their `identity_secret_hash` is revealed.
However, a semaphore proof can't be generated because
However, a semaphore proof can't be generated because
we do not know the user's `identity_nullifier` and `identity_trapdoor`.
With this design we achieve:
`identity_commitment` (Semaphore) == `identity_commitment` (RLN)
secret (semaphore) != secret (RLN).
This is the only option we had for the scheme in order to satisfy the properties described above.
This is the only option we had for the scheme
in order to satisfy the properties described above.
Also, for RLN we do a single secret component input for the circuit.
Thus we need to hash the secret array (two components) to a secret hash,
@@ -586,17 +657,20 @@ and we use that as a secret component input.
### Appendix C: Auxiliary Tooling
There are few additional tools implemented for easier integrations and usage of the RLN protocol.
There are few additional tools implemented for easier integrations and
usage of the RLN protocol.
[`zerokit`](https://github.com/vacp2p/zerokit) is a set of Zero Knowledge modules,
[`zerokit`](https://github.com/vacp2p/zerokit) is a set of Zero Knowledge modules,
written in Rust and designed to be used in many different environments.
Among different modules, it supports `Semaphore` and `RLN`.
[`zk-kit`](https://github.com/appliedzkp/zk-kit) is a typescript library which exposes APIs for identity credentials generation,
[`zk-kit`](https://github.com/appliedzkp/zk-kit)
is a typescript library which exposes APIs for identity credentials generation,
as well as proof generation.
It supports various protocols (`Semaphore`, `RLN`).
[`zk-keeper`](https://github.com/akinovak/zk-keeper) is a browser plugin which allows for safe credential storing and
[`zk-keeper`](https://github.com/akinovak/zk-keeper)
is a browser plugin which allows for safe credential storing and
proof generation.
You can think of MetaMask for zero-knowledge proofs.
It uses `zk-kit` under the hood.
@@ -690,7 +764,8 @@ assert!(verified);
```
For more details please visit the [`zerokit`](https://github.com/vacp2p/zerokit) library.
For more details please visit the
[`zerokit`](https://github.com/vacp2p/zerokit) library.
## Copyright
@@ -719,7 +794,8 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
- [rust](https://www.rust-lang.org/)
### Informative
- [1] https://medium.com/privacy-scaling-explorations/rate-limiting-nullifier-a-spam-protection-mechanism-for-anonymous-environments-bbe4006a57d
- [2] https://research.nccgroup.com/2020/06/24/security-considerations-of-zk-snark-parameter-multi-party-computation/
- [3] https://github.com/Rate-Limiting-Nullifier/rln-circuits/
- [4] https://rate-limiting-nullifier.github.io/rln-docs/
- [1] [privacy-scaling-explorations](https://medium.com/privacy-scaling-explorations/rate-limiting-nullifier-a-spam-protection-mechanism-for-anonymous-environments-bbe4006a57d)
- [2] [security-considerations-of-zk-snark-parameter-multi-party-computation](https://research.nccgroup.com/2020/06/24/)security-considerations-of-zk-snark-parameter-multi-party-computation/
- [3] [rln-circuits](https://github.com/Rate-Limiting-Nullifier/rln-circuits/)
- [4] [rln docs](https://rate-limiting-nullifier.github.io/rln-docs/)

View File

@@ -10,15 +10,22 @@ contributors:
- Oskar Thorén <oskarth@titanproxy.com>
---
In this specification, we describe a method to construct message history that will aid the consistency guarantees of [2/MVDS](../2/mvds.md). Additionally, we explain how data sync can be used for more lightweight messages that do not require full synchronization.
In this specification, we describe a method to construct message history that
will aid the consistency guarantees of [2/MVDS](../2/mvds.md).
Additionally,
we explain how data sync can be used for more lightweight messages that
do not require full synchronization.
## Motivation
In order for more efficient synchronization of conversational messages, information should be provided allowing a node to more effectively synchronize the dependencies for any given message.
In order for more efficient synchronization of conversational messages,
information should be provided allowing a node to more effectively synchronize
the dependencies for any given message.
## Format
We introduce the metadata message which is used to convey information about a message and how it SHOULD be handled.
We introduce the metadata message which is used to convey information about a message
and how it SHOULD be handled.
```protobuf
package vac.mvds;
@@ -29,7 +36,8 @@ message Metadata {
}
```
Nodes MAY transmit a `Metadata` message by extending the MVDS [message](../2/mvds.md/#payloads) with a `metadata` field.
Nodes MAY transmit a `Metadata` message by extending the MVDS [message](../2/mvds.md/#payloads)
with a `metadata` field.
```diff
message Message {
@@ -44,37 +52,53 @@ message Message {
| Name | Description |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `parents` | list of parent [`message identifier`s](../2/mvds.md/#payloads) for the specific message. |
| `parents` | list of parent [`message identifier`s](../2/mvds.md/#payloads) for the specific message. |
| `ephemeral` | indicates whether a message is ephemeral or not. |
## Usage
### `parents`
This field contains a list of parent [`message identifier`s](../2/mvds.md/#payloads) for the specific message. It MUST NOT contain any messages as parent whose `ack` flag was set to `false`. This establishes a directed acyclic graph (DAG)[^2] of persistent messages.
This field contains a list of parent [`message identifier`s](../2/mvds.md/#payloads)
for the specific message.
It MUST NOT contain any messages as parent whose `ack` flag was set to `false`.
This establishes a directed acyclic graph (DAG)[^2] of persistent messages.
Nodes MAY buffer messages until dependencies are satisfied for causal consistency[^3], they MAY also pass the messages straight away for eventual consistency[^4].
Nodes MAY buffer messages until dependencies are satisfied for causal consistency[^3],
they MAY also pass the messages straight away for eventual consistency[^4].
A parent is any message before a new message that a node is aware of that has no children.
A parent is any message before a new message that
a node is aware of that has no children.
The number of parents for a given message is bound by [0, N], where N is the number of nodes participating in the conversation, therefore the space requirements for the `parents` field is O(N).
The number of parents for a given message is bound by [0, N],
where N is the number of nodes participating in the conversation,
therefore the space requirements for the `parents` field is O(N).
If a message has no parents it is considered a root. There can be multiple roots, which might be disconnected, giving rise to multiple DAGs.
If a message has no parents it is considered a root.
There can be multiple roots, which might be disconnected,
giving rise to multiple DAGs.
### `ephemeral`
When the `ephemeral` flag is set to `false`, a node MUST send an acknowledgment when they have received and processed a message. If it is set to `true`, it SHOULD NOT send any acknowledgment. The flag is `false` by default.
When the `ephemeral` flag is set to `false`,
a node MUST send an acknowledgment when they have received and processed a message.
If it is set to `true`, it SHOULD NOT send any acknowledgment.
The flag is `false` by default.
Nodes MAY decide to not persist ephemeral messages, however they MUST NOT be shared as part of the message history.
Nodes MAY decide to not persist ephemeral messages,
however they MUST NOT be shared as part of the message history.
Nodes SHOULD send ephemeral messages in batch mode. As their delivery is not needed to be guaranteed.
Nodes SHOULD send ephemeral messages in batch mode.
As their delivery is not needed to be guaranteed.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## Footnotes
[^1]: [2/MVDS](../2/mvds.md)
[^2]: <https://en.wikipedia.org/wiki/Directed_acyclic_graph>
[^3]: Jepsen. [Causal Consistency](https://jepsen.io/consistency/models/causal). Jepsen, LLC.
[^4]: <https://en.wikipedia.org/wiki/Eventual_consistency>
1: [2/MVDS](../2/mvds.md)
2: [directed_acyclic_graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
3: Jepsen. [Causal Consistency](https://jepsen.io/consistency/models/causal)
Jepsen, LLC.
4: <https://en.wikipedia.org/wiki/Eventual_consistency>

View File

@@ -1,9 +1,9 @@
# Vac RFCs
Vac builds public good protocols for the decentralise web.
Vac builds public good protocols for the decentralised web.
Vac acts as a custodian for the protocols that live in the RFC-Index repository.
With the goal of widespread adoption,
Vac will make sure the protocols adhere to the set of principles,
including but not limited to liberty, security, privacy, decentralisation, and inclusivity.
Vac will make sure the protocols adhere to a set of principles,
including but not limited to liberty, security, privacy, decentralisation and inclusivity.
To learn more, visit [Vac Research](https://vac.dev/)

File diff suppressed because it is too large Load Diff

View File

@@ -1,13 +1,19 @@
---
title: ETH-SECPM
name: Secure channel setup using Ethereum accounts
status: raw
status: deleted
category: Standards Track
tags:
editor: Ramses Fernandez <ramses@status.im>
contributors:
---
## NOTE
The content of this specification has been split between
[ETH-DEMLS](vac/raw/eth-demls.md) and [NOISE-X3DH-RATCHET](vac/raw/noise-x3dh-ratchet.md)
RFCs.
## Motivation
The need for secure communications has become paramount.
@@ -416,7 +422,7 @@ Credentials MUST follow the specifications of section 5.3 of
Below follows the flow diagram for the generation of credentials.
Users MUST generate key pairs by themselves.
![figure1](./images/eth-secpm_credential.png)
![figure1](/vac/raw/images/eth-secpm_credential.png)
### Message framing
@@ -759,10 +765,10 @@ CredentialType credential_types<V>;
The flow diagram shows the procedure to fetch key material from other
users:
![figure2](./images/eth-secpm_fetching.png)
![figure2](/vac/raw/images/eth-secpm_fetching.png)
Below follows the flow diagram for the creation of a group:
![figure3](./images/eth-secpm_creation.png)
![figure3](/vac/raw/images/eth-secpm_creation.png)
### Group evolution
@@ -837,15 +843,15 @@ The client MUST apply the proposals in the list in the order described
in Section 12.3 of [RFC9420](https://datatracker.ietf.org/docrfc9420/).
Below follows the flow diagram for the addition of a member to a group:
![figure4](./images/eth-secpm_add.png)
![figure4](/vac/raw/images/eth-secpm_add.png)
The diagram below shows the procedure to remove a group member:
![figure5](./images/eth-secpm_remove.png)
![figure5](/vac/raw/images/eth-secpm_remove.png)
The flow diagram below shows an update procedure:
![figure6](./images/eth-secpm_update.png)
![figure6](/vac/raw/images/eth-secpm_update.png)
### Commit messages
@@ -1287,7 +1293,7 @@ and checks that it corresponds to an address contained in the ACL.
7. Off-chain - Alice sends a welcome message to Bob.
8. Off-chain - Alice SHOULD broadcasts a message announcing the
addition of Bob to other users of the group.
![figure7](./images/eth-secpm_onchain-register-1.png)
![figure7](/vac/raw/images/eth-secpm_onchain-register-1.png)
#### Alice does not know Bobs Ethereum address
@@ -1310,7 +1316,7 @@ contract.
8. Off-chain - Alice SHOULD broadcasts a message announcing the
addition of Bob to other users of the group.
![figure8](./images/eth-secpm_onchain-register-2.png)
![figure8](/vac/raw/images/eth-secpm_onchain-register-2.png)
### Considerations regarding smart contracts
@@ -1330,7 +1336,7 @@ off-chain message.
- The creator of the contract MUST update the ACL, and send
messages to the group for key update.
![figure9](./images/eth-secpm_onchain-update.png)
![figure9](/vac/raw/images/eth-secpm_onchain-update.png)
> It is important to note that both
user removal and updates of any kind

1034
vac/raw/eth-mls-onchain.md Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -18,21 +18,31 @@ Tor Push adds sender identity protection to gossipsub.
**Protocol identifier**: /meshsub/1.1.0
Note: Gossipsub Tor Push does not have a dedicated protocol identifier.
It uses the same identifier as gossipsub and works with all [pubsub](https://github.com/libp2p/specs/tree/master/pubsub) based protocols.
This allows nodes that are oblivious to Tor Push to process messages received via Tor Push.
It uses the same identifier as gossipsub and
works with all [pubsub](https://github.com/libp2p/specs/tree/master/pubsub)
based protocols.
This allows nodes that are oblivious to Tor Push to process messages received via
Tor Push.
## Background
Without extensions, [libp2p gossipsub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md)
does not protect sender identities.
A possible design of an anonymity extension to gossipsub is pushing messages through an anonymization network before they enter the gossipsub network.
A possible design of an anonymity extension to gossipsub
is pushing messages through an anonymization network
before they enter the gossipsub network.
[Tor](https://www.torproject.org/) is currently the largest anonymization network.
It is well researched and works reliably.
Basing our solution on Tor both inherits existing security research, as well as allows for a quick deployment.
Basing our solution on Tor both inherits existing security research,
as well as allows for a quick deployment.
Using the anonymization network approach, even the first gossipsub node that relays a given message cannot link the message to its sender (within a relatively strong adversarial model).
Taking the low bandwidth overhead and the low latency overhead into consideration, Tor offers very good anonymity properties.
Using the anonymization network approach,
even the first gossipsub node that relays a given message
cannot link the message to its sender
(within a relatively strong adversarial model).
Taking the low bandwidth overhead and the low latency overhead into consideration,
Tor offers very good anonymity properties.
## Functional Operation
@@ -44,17 +54,20 @@ because Tor Push uses the same Protocol ID as gossipsub.
Messages are sent over Tor via [SOCKS5](https://www.rfc-editor.org/rfc/rfc1928).
Tor Push uses a dedicated libp2p context to prevent information leakage.
To significantly increase resilience and mitigate circuit failures,
Tor Push establishes several connections, each to a different randomly selected gossipsub node.
Tor Push establishes several connections,
each to a different randomly selected gossipsub node.
## Specification
This section specifies the format of Tor Push messages, as well as how Tor Push messages are received and sent, respectively.
This section specifies the format of Tor Push messages,
as well as how Tor Push messages are received and sent, respectively.
### Wire Format
The wire format of a Tor Push message corresponds verbatim to a typical [libp2p pubsub message](https://github.com/libp2p/specs/tree/master/pubsub#the-message).
The wire format of a Tor Push message corresponds verbatim to a typical
[libp2p pubsub message](https://github.com/libp2p/specs/tree/master/pubsub#the-message).
```
```protobuf
message Message {
optional string from = 1;
optional bytes data = 2;
@@ -67,12 +80,15 @@ message Message {
### Receiving Tor Push Messages
Any node supporting a protocol with ID `/meshsub/1.1.0` (e.g. gossipsub), can receive Tor Push messages.
Receiving nodes are oblivious to Tor Push and will process incoming messages according to the respective `meshsub/1.1.0` specification.
Any node supporting a protocol with ID `/meshsub/1.1.0` (e.g. gossipsub),
can receive Tor Push messages.
Receiving nodes are oblivious to Tor Push and
will process incoming messages according to the respective `meshsub/1.1.0` specification.
### Sending Tor Push Messages
In the following, we refer to nodes sending Tor Push messages as Tp-nodes (Tor Push nodes).
In the following, we refer to nodes sending Tor Push messages as Tp-nodes
(Tor Push nodes).
Tp-nodes MUST setup a separate libp2p context, i.e. [libp2p switch](https://docs.libp2p.io/concepts/multiplex/switch/),
which MUST NOT be used for any purpose other than Tor Push.
@@ -81,29 +97,38 @@ The Tp-context MUST NOT share any data, e.g. peer lists, with the default contex
Tp-peers are peers a Tp-node plans to send Tp-messages to.
Tp-peers MUST support `/meshsub/1.1.0`.
For retrieving Tp-peers, Tp-nodes SHOULD use an ambient peer discovery method that retrieves a random peer sample (from the set of all peers), e.g. [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md).
For retrieving Tp-peers,
Tp-nodes SHOULD use an ambient peer discovery method
that retrieves a random peer sample (from the set of all peers),
e.g. [33/WAKU2-DISCV5](../../waku/standards/core/33/discv5.md).
Tp-nodes MUST establish a connection as described in sub-section [Tor Push Connection Establishment](#connection-establishment) to at least one Tp-peer.
To significantly increase resilience, Tp-nodes SHOULD establish Tp-connections to `D` peers,
Tp-nodes MUST establish a connection as described in sub-section
[Tor Push Connection Establishment](#connection-establishment) to at least one Tp-peer.
To significantly increase resilience,
Tp-nodes SHOULD establish Tp-connections to `D` peers,
where `D` is the [desired gossipsub out-degree](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md#parameters),
with a default value of `8`.
Each Tp-message MUST be sent via the Tp-context over at least one Tp-connection.
To increase resilience, Tp-messages SHOULD be sent via the Tp-context over all available Tp-connections.
To increase resilience,
Tp-messages SHOULD be sent via the Tp-context over all available Tp-connections.
Control messages of any kind, e.g. gossipsub graft, MUST NOT be sent via Tor Push.
#### Connection Establishment
Tp-nodes establish a `/meshsub/1.1.0` connection to tp-peers via [SOCKS5](https://www.rfc-editor.org/rfc/rfc1928) over [Tor](https://www.torproject.org/).
Tp-nodes establish a `/meshsub/1.1.0` connection to tp-peers via
[SOCKS5](https://www.rfc-editor.org/rfc/rfc1928) over [Tor](https://www.torproject.org/).
Establishing connections, which in turn establishes the respective Tor circuits, can be done ahead of time.
Establishing connections, which in turn establishes the respective Tor circuits,
can be done ahead of time.
#### Epochs
Tor Push introduces epochs.
The default epoch duration is 10 minutes.
(We might adjust this default value based on experiments and evaluation in future versions of this document.
(We might adjust this default value based on experiments and
evaluation in future versions of this document.
It seems a good trade-off between traceablity and circuit building overhead.)
For each epoch, the Tp-context SHOULD be refreshed, which includes
@@ -112,7 +137,9 @@ For each epoch, the Tp-context SHOULD be refreshed, which includes
* Tp-peer list
* connections to Tp-peers
Both Tp-peer selection for the next epoch and establishing connections to the newly selected peers SHOULD be done during the current epoch
Both Tp-peer selection for the next epoch and
establishing connections to the newly selected peers
SHOULD be done during the current epoch
and be completed before the new epoch starts.
This avoids adding latency to message transmission.
@@ -120,42 +147,54 @@ This avoids adding latency to message transmission.
### Fingerprinting Attacks
Protocols that feature distinct patterns are prone to fingerprinting attacks when using them over Tor Push.
Protocols that feature distinct patterns are prone to fingerprinting attacks
when using them over Tor Push.
Both malicious guards and exit nodes could detect these patterns
and link the sender and receiver, respectively, to transmitted traffic.
As a mitigation, such protocols can introduce dummy messages and/or padding to hide patterns.
As a mitigation, such protocols can introduce dummy messages and/or
padding to hide patterns.
### DoS
#### General DoS against Tor
Using untargeted DoS to prevent Tor Push messages from entering the gossipsub network would cost vast resources,
because Tor Push transmits messages over several circuits and the Tor network is well established.
Using untargeted DoS to prevent Tor Push messages
from entering the gossipsub network would cost vast resources,
because Tor Push transmits messages over several circuits and
the Tor network is well established.
#### Targeting the Guard
Denying the service of a specific guard node blocks Tp-nodes using the respective guard.
Denying the service of a specific guard node
blocks Tp-nodes using the respective guard.
Tor guard selection will replace this guard [TODO elaborate].
Still, messages might be delayed during this window which might be critical to certain applications.
Still, messages might be delayed during this window
which might be critical to certain applications.
#### Targeting the Gossipsub Network
Without sophisticated rate limiting (for example using [17/WAKU2-RLN-RELAY](../../waku/standards/core/17/rln-relay.md)),
attackers can spam the gossipsub network.
It is not enough to just block peers that send too many messages,
because these messages might actually come from a Tor exit node that many honest Tp-nodes use.
Without Tor Push, protocols on top of gossipsub could block peers if they exceed a certain message rate.
because these messages might actually come from a Tor exit node
that many honest Tp-nodes use.
Without Tor Push,
protocols on top of gossipsub could block peers
if they exceed a certain message rate.
With Tor Push, this would allow the reputation-based DoS attack described in
[Bitcoin over Tor isn't a Good Idea](https://ieeexplore.ieee.org/abstract/document/7163022).
#### Peer Discovery
The discovery mechanism could be abused to link requesting nodes to their Tor connections to discovered nodes.
The discovery mechanism could be abused to link requesting nodes
to their Tor connections to discovered nodes.
An attacker that controls both the node that responds to a discovery query,
and the node whos ENR the response contains,
can link the requester to a Tor connection that is expected to be opened to the node represented by the returned ENR soon after.
can link the requester to a Tor connection
that is expected to be opened to the node represented by the returned ENR soon after.
Further, the discovery mechanism (e.g. discv5) could be abused to distribute disproportionately many malicious nodes.
Further, the discovery mechanism (e.g. discv5)
could be abused to distribute disproportionately many malicious nodes.
For instance if p% of the nodes in the network are malicious,
an attacker could manipulate the discovery to return malicious nodes with 2p% probability.
The discovery mechanism needs to be resilient against this attack.
@@ -163,8 +202,11 @@ The discovery mechanism needs to be resilient against this attack.
### Roll-out Phase
During the roll-out phase of Tor Push, during which only a few nodes use Tor Push,
attackers can narrow down the senders of Tor messages to the set of gossipsub nodes that do not originate messages.
Nodes who want anonymity guarantees even during the roll-out phase can use separate network interfaces for their default context and Tp-context, respectively.
attackers can narrow down the senders of Tor messages
to the set of gossipsub nodes that do not originate messages.
Nodes who want anonymity guarantees even during the roll-out phase
can use separate network interfaces for their default context and
Tp-context, respectively.
For the best protection, these contexts should run on separate physical machines.
## Copyright

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 35 KiB

897
vac/raw/mix.md Normal file
View File

@@ -0,0 +1,897 @@
---
title: LIBP2P-MIX
name: Libp2p Mix Protocol
status: raw
category: Standards Track
tags:
editor: Akshaya Mani <akshaya@status.im>
contributors: Daniel Kaiser <danielkaiser@status.im>
---
## Abstract
The Mix Protocol defines a decentralized anonymous message routing layer for
libp2p networks.
It enables sender anonymity by routing each message through a decentralized mix
overlay network
composed of participating libp2p nodes, known as mix nodes. Each message is
routed independently
in a stateless manner, allowing other libp2p protocols to selectively anonymize
messages without
modifying their core protocol behavior.
## 1. Introduction
The Mix Protocol is a custom libp2p protocol that defines a message-layer
routing abstraction
designed to provide sender anonymity in peer-to-peer systems built on the libp2p
stack.
It addresses the absence of native anonymity primitives in libp2p by offering a
modular,
content-agnostic protocol that other libp2p protocols can invoke when anonymity
is required.
This document describes the design, behavior, and integration of the Mix
Protocol within the
libp2p architecture. Rather than replacing or modifying existing libp2p
protocols, the Mix Protocol
complements them by operating independently of connection state and protocol
negotiation.
It is intended to be used as an optional anonymity layer that can be selectively
applied on a
per-message basis.
Integration with other libp2p protocols is handled through external interface
components&mdash;the Mix Entry
and Exit layers&mdash;which mediate between these protocols and the Mix Protocol
instances.
These components allow applications to defer anonymity concerns to the Mix layer
without altering
their native semantics or transport assumptions.
The rest of this document describes the motivation for the protocol, defines
relevant terminology,
presents the protocol architecture, and explains how the Mix Protocol
interoperates with the broader
libp2p protocol ecosystem.
## 2. Terminology
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
“SHOULD NOT”, “RECOMMENDED”,
“MAY”, and “OPTIONAL” in this document are to be interpreted as described in
[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
The following terms are used throughout this specification:
- **Origin Protocol**
A libp2p protocol (_e.g.,_ Ping, GossipSub) that generates and receives the
actual message payload.
The origin protocol MUST decide on a per-message basis whether to route the
message through the Mix Protocol
or not.
- **Mix Node**
A libp2p node that supports the Mix Protocol and participates in the mix
network.
A mix node initiates anonymous routing when invoked with a message.
It also receives and processes Sphinx packets when selected as a hop in a mix
path.
- **Mix Path**
A non-repeating sequence of mix nodes through which a Sphinx packet is routed
across the mix network.
- **Mixify**
A per-message flag set by the origin protocol to indicate that a message should
be routed using
the Mix Protocol or not.
Only messages with mixify set are forwarded to the Mix Entry Layer.
Other messages SHOULD be routed using the origin protocols default behavior.
The phrases 'messages to be mixified', 'to mixify a message' and related
variants are used
informally throughout this document to refer to messages that either have the
`mixify` flag set
or are selected to have it set.
- **Mix Entry Layer**
A component that receives messages to be _mixified_ from an origin protocol and
forwards them to the
local Mix Protocol instance.
The Entry Layer is external to the Mix Protocol.
- **Mix Exit Layer**
A component that receives decrypted messages from a Mix Protocol instance and
delivers them
to the appropriate origin protocol instance at the destination.
Like the Entry Layer, it is external to the Mix Protocol.
- **Mixnet or Mix Network**
A decentralized overlay network formed by all nodes that support the Mix
Protocol.
It operates independently of libp2ps protocol-level routing and origin protocol
behavior.
- **Sphinx Packet**
A cryptographic packet format used by the Mix Protocol to encapsulate messages.
It uses layered encryption to hide routing information and protect message
contents as packets are forwarded hop-by-hop.
Sphinx packets are fixed-size and indistinguishable from one another, providing
unlinkability and metadata protection.
## 3. Motivation and Background
libp2p enables modular peer-to-peer applications, but it lacks built-in support
for sender anonymity.
Most protocols expose persistent peer identifiers, transport metadata, or
traffic patterns that
can be exploited to deanonymize users through passive observation or
correlation.
While libp2p supports NAT traversal mechanisms such as Circuit Relay, these
focus on connectivity
rather than anonymity. Relays may learn peer identities during stream setup and
can observe traffic
timing and volume, offering no protection against metadata analysis.
libp2p also supports a Tor transport for network-level anonymity, tunneling
traffic through long-lived,
encrypted circuits. However, Tor relies on session persistence and is ill-suited
for protocols
requiring per-message unlinkability.
The Mix Protocol addresses this gap with a decentralized message routing layer
based on classical
mix network principles. It applies layered encryption and per-hop delays to
obscure both routing paths
and timing correlations. Each message is routed independently, providing
resistance to traffic analysis
and protection against metadata leakage
By decoupling anonymity from connection state and transport negotiation, the Mix
Protocol offers
a modular privacy abstraction that existing libp2p protocols can adopt without
altering their
core behavior.
To better illustrate the differences in design goals and threat models, the
following subsection contrasts
the Mix Protocol with Tor, a widely known anonymity system.
### 3.1 Comparison with Tor
The Mix Protocol differs fundamentally from Tor in several ways:
- **Unlinkability**: In the Mix Protocol, there is no direct connection between
source and destination.
Each message is routed independently, eliminating correlation through persistent
circuits.
- **Delay-based mixing**: Mix nodes introduce randomized delays (e.g., from an
exponential distribution)
before forwarding messages, making timing correlation significantly harder.
- **High-latency focus**: Tor prioritizes low-latency communication for
interactive web traffic,
whereas the Mix Protocol is designed for scenarios where higher latency is
acceptable
in exchange for stronger anonymity.
- **Message-based design**: Each message in the Mix Protocol is self-contained
and independently routed.
No sessions or state are maintained between messages.
- **Resistance to endpoint attacks**: The Mix Protocol is less susceptible to
certain endpoint-level attacks,
such as traffic volume correlation or targeted probing, since messages are
delayed, reordered, and unlinkable at each hop.
To understand the underlying anonymity properties of the Mix Protocol, we next
describe the core components of a mix network.
## 4. Mixing Strategy and Packet Format
The Mix Protocol relies on two core design elements to achieve sender
unlinkability and metadata
protection: a mixing strategy and a a cryptographically secure mix packet
format.
### 4.1 Mixing Strategy
A mixing strategy defines how mix nodes delay and reorder incoming packets to
resist timing
correlation and input-output linkage. Two commonly used approaches are
batch-based mixing and
continuous-time mixing.
In batching-based mixing, each mix node collects incoming packets over a fixed
or adaptive
interval, shuffles them, and forwards them in a batch. While this provides some
unlinkability,
it introduces high latency, requires synchronized flushing rounds, and may
result in bursty
output traffic. Anonymity is bounded by the batch size, and performance may
degrade under variable
message rates.
The Mix Protocol instead uses continuous-time mixing, where each mix node
applies a randomized
delay to every incoming packet, typically drawn from an exponential
distribution. This enables
theoretically unbounded anonymity sets, since any packet may, with non-zero
probability,
be delayed arbitrarily long. In practice, the distribution is truncated once the
probability
of delay falls below a negligible threshold. Continuous-time mixing also offers
improved
bandwidth utilization and smoother output traffic compared to batching-based
approaches.
To make continuous-time mixing tunable and predictable, the sender MUST select
the mean delay
for each hop and encode it into the Sphinx packet header. This allows top-level
applications
to balance latency and anonymity according to their requirements.
### 4.2 Mix Packet Format
A mix packet format defines how messages are encapsulated and routed through a
mix network.
It must ensure unlinkability between incoming and outgoing packets, prevent
metadata leakage
(e.g., path length, hop position, or payload size), and support uniform
processing by mix nodes
regardless of direction or content.
The Mix Protocol uses [Sphinx
packets](https://cypherpunks.ca/~iang/pubs/Sphinx_Oakland09.pdf)
to meet these goals.
Each message is encrypted in layers corresponding to the selected mix path. As a
packet traverses
the network, each mix node removes one encryption layer to obtain the next hop
and delay,
while the remaining payload remains encrypted and indistinguishable.
Sphinx packets are fixed in size and bit-wise unlinkable. This ensures that they
appear identical
on the wire regardless of payload, direction, or route length, reducing
opportunities for correlation
based on packet size or format. Even mix nodes learn only the immediate routing
information
and the delay to be applied. They do not learn their position in the path or the
total number of hops.
The packet format is resistant to tagging and replay attacks and is compact and
efficient to
process. Sphinx packets also include per-hop integrity checks and enforces a
maximum path length.
Together with a constant-size header and payload, this provides bounded
protection
against
endless routing and malformed packet propagation.
It also supports anonymous and indistinguishable reply messages through
[Single-Use Reply Blocks
(SURBs)](https://cypherpunks.ca/~iang/pubs/Sphinx_Oakland09.pdf),
although reply support is not implemented yet.
A complete specification of the Sphinx packet structure and fields is provided
in [Section 6].
## 5. Protocol Overview
The Mix Protocol defines a decentralized, message-based routing layer that
provides sender anonymity
within the libp2p framework.
It is agnostic to message content and semantics. Each message is treated as an
opaque payload,
wrapped into a [Sphinx
packet](https://cypherpunks.ca/~iang/pubs/Sphinx_Oakland09.pdf) and routed
independently through a randomly selected mix path. Along the path, each mix
node removes one layer
of encryption, adds a randomized delay, and forwards the packet to the next hop.
This combination of
layered encryption and per-hop delay provides resistance to traffic analysis and
enables message-level
unlinkability.
Unlike typical custom libp2p protocols, the Mix protocol is stateless&mdash;it
does not establish
persistent streams, negotiate protocols, or maintain sessions. Each message is
self-contained
and routed independently.
The Mix Protocol sits above the transport layer and below the protocol layer in
the libp2p stack.
It provides a modular anonymity layer that other libp2p protocols MAY invoke
selectively on a
per-message basis.
Integration with other libp2p protocols is handled through external components
that mediate
between the origin protocol and the Mix Protocol instances. This enables
selective anonymous routing
without modifying protocol semantics or internal behavior.
The following subsections describe how the Mix Protocol integrates with origin
protocols via
the Mix Entry and Exit layers, how per-message anonymity is controlled through
the `mixify` flag,
the rationale for defining Mix as a protocol rather than a transport, and the
end-to-end message
interaction flow.
### 5.1 Integration with Origin Protocols
libp2p protocols that wish to anonymize messages MUST do so by integrating with
the Mix Protocol
via the Mix Entry and Exit layers.
- The **Mix Entry Layer** receives messages to be _mixified_ from an origin
protocol and forwards them
to the local Mix Protocol instance.
- The **Mix Exit Layer** receives the final decrypted message from a Mix
Protocol instance and
forwards it to the appropriate origin protocol instance at the destination over
a client-only connection.
This integration is external to the Mix Protocol and is not handled by mix nodes
themselves.
### 5.2 Mixify Option
Some origin protocols may require selective anonymity, choosing to anonymize
_only_ certain messages
based on their content, context, or destination. For example, a protocol may
only anonymize messages
containing sensitive metadata while delivering others directly to optimize
performance.
To support this, origin protocols MAY implement a per-message `mixify` flag that
indicates whether a message should be routed using the Mix Protocol.
- If the flag is set, the message MUST be handed off to the Mix Entry Layer for
anonymous routing.
- If the flag is not set, the message SHOULD be routed using the origin
protocols default mechanism.
This design enables protocols to invoke the Mix Protocol only for selected
messages, providing fine-grained control over privacy and performance
trade-offs.
### 5.3 Why a Protocol, Not a Transport
The Mix Protocol is specified as a custom libp2p protocol rather than a
transport to support
selective anonymity while remaining compatible with libp2ps architecture.
As noted in [Section 5.2](#52-mixify-option), origin protocols may anonymize
only specific messages
based on content or context. Supporting such selective behavior requires
invoking Mix on a per-message basis.
libp2p transports, however, are negotiated per peer connection and apply
globally to all messages
exchanged between two peers. Enabling selective anonymity at the transport layer
would
therefore require
changes to libp2ps core transport semantics.
Defining Mix as a protocol avoids these constraints and offers several benefits:
- Supports selective invocation on a per-message basis.
- Works atop existing secure transports (_e.g.,_ QUIC, TLS) without requiring
changes to the transport stack.
- Preserves a stateless, content-agnostic model focused on anonymous message
routing.
- Integrates seamlessly with origin protocols via the Mix Entry and Exit layers.
This design preserves the modularity of the libp2p stack and allows Mix to be
adopted without altering existing transport or protocol behavior.
### 5.4 Protocol Interaction Flow
A typical end-to-end Mix Protocol flow consists of the following three
conceptual phases.
Only the second phase&mdash;the anonymous routing performed by mix
nodes&mdash;is part of the core
Mix Protocol. The entry-side and exit-side integration steps are handled
externally by the Mix Entry
and Exit layers.
1. **Entry-side Integration (Mix Entry Layer):**
- The origin protocol generates a message and sets the `mixify` flag.
- The message is passed to the Mix Entry Layer, which invokes the local Mix
Protocol instance with
the message, destination, and origin protocol codec as input.
2. **Anonymous Routing (Core Mix Protocol):**
- The Mix Protocol instance wraps the message in a Sphinx packet and selects a
random mix path.
- Each mix node along the path:
- Processes the Sphinx packet by removing one encryption layer.
- Applies a delay and forwards the packet to the next hop.
- The final node in the path (exit node) decrypts the final layer, extracting
the original plaintext message, destination, and origin protocol codec.
3. **Exit-side Integration (Mix Exit Layer):**
- The Mix Exit Layer receives the plaintext message, destination, and origin
protocol codec.
- It routes the message to the destination origin protocol instance using a
client-only connection.
The destination node does not need to support the Mix Protocol to receive or
respond
to anonymous messages.
The behavior described above represents the core Mix Protocol. In addition, the
protocol
supports a set of pluggable components that extend its functionality. These
components cover
areas such as node discovery, delay strategy, spam resistance, cover traffic
generation,
and incentivization. Some are REQUIRED for interoperability; others are OPTIONAL
or deployment-specific.
The next section describes each component.
### 5.5 Stream Management and Multiplexing
Each Mix Protocol message is routed independently, and forwarding it to the next
hop requires
opening a new libp2p stream using the Mix Protocol. This applies to both the
initial Sphinx packet
transmission and each hop along the mix path.
In high-throughput environments (_e.g._, messaging systems with continuous
anonymous traffic),
mix nodes may frequently communicate with a subset of mix nodes. Opening a new
stream for each
Sphinx packet in such scenarios can incur performance costs, as each stream
setup requires a
multistream handshake for protocol negotiation.
While libp2p supports multiplexing multiple streams over a single transport
connection using
stream muxers such as mplex and yamux, it does not natively support reusing a
stream over multiple
message transmissions. However, stream reuse may be desirable in the mixnet
setting to reduce overhead
and avoid hitting per protocol stream limits between peers.
The lifecycle of streams, including their reuse, eviction, or pooling strategy,
is outside the
scope of this specification. It SHOULD be handled by the libp2p host, connection
manager, or
transport stack.
Mix Protocol implementations MUST NOT assume persistent stream availability and
SHOULD gracefully
fall back to opening a new stream when reuse is not possible.
## 6. Pluggable Components
Pluggable components define functionality that extends or configures the
behavior of the Mix Protocol
beyond its core message routing logic. Each component in this section falls into
one of two categories:
- Required for interoperability and path construction (_e.g.,_ discovery, delay
strategy).
- Optional or deployment-specific (_e.g.,_ spam protection, cover traffic,
incentivization).
The following subsections describe the role and expected behavior of each.
### 6.1 Discovery
The Mix Protocol does not mandate a specific peer discovery mechanism. However,
nodes participating in
the mixnet MUST be discoverable so that other nodes can construct routing paths
that include them.
To enable this, regardless of the discovery mechanism used, each mix node MUST
make the following
information available to peers:
- Indicate Mix Protocol support (_e.g.,_ using a `mix` field or bit).
- Its X25519 public key for Sphinx encryption.
- One or more routable libp2p multiaddresses that identify the mix nodes own
network endpoints.
To support sender anonymity at scale, discovery mechanism SHOULD support
_unbiased random sampling_
from the set of live mix nodes. This enables diverse path construction and
reduces exposure to
adversarial routing bias.
While no existing mechanism provides unbiased sampling by default,
[Wakus ambient discovery](https://rfc.vac.dev/waku/standards/core/33/discv5/)
&mdash; an extension
over [Discv5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)
&mdash; demonstrates
an approximate solution. It combines topic-based capability advertisement with
periodic
peer sampling. A similar strategy could potentially be adapted for the Mix
Protocol.
A more robust solution would involve integrating capability-aware discovery
directly into the
libp2p stack, such as through extensions to `libp2p-kaddht`. This would enable
direct lookup of
mix nodes based on protocol support and eliminate reliance on external
mechanisms such as Discv5.
Such an enhancement remains exploratory and is outside the scope of this
specification.
Regardless of the mechanism, the goal is to ensure mix nodes are discoverable
and that path selection
is resistant to bias and node churn.
### 6.2 Delay Strategy
The Mix Protocol uses per-hop delay as a core mechanism for achieving timing
unlinkability.
For each hop in the mix path, the sender MUST specify a mean delay value, which
is embedded in
the Sphinx packet header. The mix node at each hop uses this value to sample a
randomized delay
before forwarding the packet.
By default, delays are sampled from an exponential distribution. This supports
continuous-time mixing,
produces smooth output traffic, and enables tunable trade-offs between latency
and anonymity.
Importantly, it allows for unbounded anonymity sets: each packet may, with
non-zero probability,
be delayed arbitrarily long.
The delay strategy is considered pluggable, and other distributions MAY be used
to match
application-specific anonymity or performance requirements. However, any delay
strategy
MUST ensure that:
- Delays are sampled independently at each hop.
- Delay sampling introduces sufficient variability to obscure timing correlation
between packet
arrival and forwarding across multiple hops.
Strategies that produce deterministic or tightly clustered output delays are NOT
RECOMMENDED,
as they increase the risk of timing correlation. Delay strategies SHOULD
introduce enough uncertainty
to prevent adversaries from linking packet arrival and departure times, even
when monitoring
multiple hops concurrently.
### 6.3 Spam Protection
The Mix Protocol supports optional spam protection mechanisms to defend
recipients against
abusive or unsolicited traffic. These mechanisms are applied at the exit node,
which is the
final node in the mix path before the message is delivered to its destination
via the respective
libp2p protocol.
Exit nodes that enforce spam protection MUST validate the attached proof before
forwarding
the message. If validation fails, the message MUST be discarded.
Common strategies include Proof of Work (PoW), Verifiable Delay Functions
(VDFs), and Rate-limiting Nullifiers (RLNs).
The sender is responsible for appending the appropriate spam protection data
(e.g., nonce, timestamp)
to the message payload. The format and verification logic depend on the selected
method.
An example using PoW is included in Appendix A.
Note: The spam protection mechanisms described above are intended to protect the
destination application
or protocol from message abuse or flooding. They do not provide protection
against denial-of-service (DoS) or
resource exhaustion attacks targeting the mixnet itself (_e.g.,_ flooding mix
nodes with traffic,
inducing processing overhead, or targeting bandwidth).
Protections against attacks targeting the mixnet itself are not defined in this
specification
but are critical to the long-term robustness of the system. Future versions of
the protocol may
define mechanisms to rate-limit clients, enforce admission control, or
incorporate incentives and
accountability to defend the mixnet itself from abuse.
### 6.4 Cover Traffic
Cover traffic is an optional mechanism used to improve privacy by making the
presence or absence
of actual messages indistinguishable to observers. It helps achieve
_unobservability_ where
a passive adversary cannot determine whether a node is sending real messages or
not.
In the Mix Protocol, cover traffic is limited to _loop messages_ &mdash; dummy
Sphinx packets
that follow a valid mix path and return to the originating node. These messages
carry no application
payload but are indistinguishable from real messages in structure, size, and
routing behavior.
Cover traffic MAY be generated by either mix nodes or senders. The strategy for
generating
such traffic &mdash; such as timing and frequency &mdash; is pluggable and not
specified
in this document.
Implementations that support cover traffic SHOULD generate loop messages at
randomized intervals.
This helps mask actual sending behavior and increases the effective anonymity
set. Timing
strategies such as Poisson processes or exponential delays are commonly used,
but the choice is
left to the implementation.
In addition to
enhancing privacy, loop messages can be used to assess network liveness or path
reliability
without requiring explicit acknowledgments.
### 6.5 Incentivization
The Mix Protocol supports a simple tit-for-tat model to discourage free-riding
and promote
mix node participation. In this model, nodes that wish to send anonymous
messages using the
Mix Protocol MUST also operate a mix node. This requirement ensures that
participants contribute to
the anonymity set they benefit from, fostering a minimal form of fairness and
reciprocity.
This tit-for-tat model is intentionally lightweight and decentralized. It deters
passive use
of the mixnet by requiring each user to contribute bandwidth and processing
capacity. However, it
does not guarantee the quality of service provided by participating nodes. For
example, it
does not prevent nodes from running low-quality or misbehaving mix instances,
nor does it
deter participation by compromised or transient peers.
The Mix Protocol does not mandate any form of payment, token exchange, or
accounting. More
sophisticated economic models &mdash; such as stake-based participation,
credentialed relay networks,
or zero-knowledge proof-of-contribution systems &mdash; MAY be layered on top of
the protocol or
enforced via external coordination.
Additionally, network operators or application-layer policies MAY require nodes
to maintain
minimum uptime, prove their participation, or adhere to service-level
guarantees.
While the Mix Protocol defines a minimum participation requirement, additional
incentivization
extensions are considered pluggable and experimental in this version of the
specification.
No specific mechanism is standardized.
## 7. Core Mix Protocol Responsibilities
This section defines the core routing behavior of the Mix Protocol, which all
conforming
implementations MUST support.
The Mix Protocol defines the logic for anonymously routing messages through the
decentralized
mix network formed by participating libp2p nodes. Each mix node MUST implement
support for:
- initiating anonymous routing when invoked with a message.
- receiving and processing Sphinx packets when selected as a hop in a mix path.
These roles and their required behaviors are defined in the following
subsections.
### 7.1 Protocol Identifier
The Mix Protocol is identified by the protocol string `"/mix/1.0.0"`.
All Mix Protocol interactions occur over libp2p streams negotiated using this
identifier.
Each Sphinx packet transmission&mdash;whether initiated locally or forwarded as
part of a
mix path&mdash;involves opening a new libp2p stream to the next hop.
Implementations MAY optimize
performance by reusing streams where appropriate; see [Section
5.5](#55-stream-management-and-multiplexing)
for more details on stream management.
### 7.2 Initiation
A mix node initiates anonymous routing only when it is explicitly invoked with a
message
to be routed. As specified in [Section 5.2](#52-mixify-option), the decision to
anonymize a
message is made by the origin protocol. When anonymization is required, the
origin protocol instance
forwards the message to the Mix Entry Layer, which then passes the message to
the local
Mix Protocol instance for routing.
To perform message initiation, a mix node MUST:
- Select a random mix path.
- Assign a delay value for each hop and encode it into the Sphinx packet header.
- Wrap message in a Sphinx packet by applying layered encryption in reverse
order of nodes
in the selected mix path.
- Forward the resulting packet to the first mix node in the mix path using the
Mix Protocol.
The Mix Protocol does not interpret message content or origin protocol context.
Each invocation is
stateless, and the implementation MUST NOT retain routing metadata or
per-message state
after the packet is forwarded.
### 7.3 Sphinx Packet Receiving and Processing
A mix node that receives a Sphinx packet is oblivious to its position in the
path. The
first hop is indistinguishable from other intermediary hops in terms of
processing and behavior.
After decrypting one layer of the Sphinx packet, the node MUST inspect the
routing information.
If this layer indicates that the next hop is the final destination, the packet
MUST be processed
as an exit. Otherwise, it MUST be processed as an intermediary.
#### 7.3.1 Intermediary Processing
To process a Sphinx packet as an intermediary, a mix node MUST:
- Extract the next hop address and associated delay from the decrypted packet.
- Wait for the specified delay.
- Forward the updated packet to the next hop using the Mix Protocol.
A mix node performing intermediary processing MUST treat each packet as
stateless and self-contained.
#### 7.3.2 Exit Processing
To process a Sphinx packet as an exit, a mix node MUST:
- Extract the plaintext message from the final decrypted packet.
- Validate any attached spam protection proof.
- Discard the message if spam protection validation fails.
- Forward the valid message to the Mix Exit Layer for delivery to the
destination origin protocol instance.
The node MUST NOT retain decrypted content after forwarding.
## 8. Sphinx Packet Format
The Mix Protocol uses the Sphinx packet format to enable unlinkable, multi-hop
message routing
with per-hop confidentiality and integrity. Each message transmitted through the
mix network is
encapsulated in a Sphinx packet constructed by the initiating mix node. The
packet is encrypted in
layers such that each hop in the mix path can decrypt exactly one layer and
obtain the next-hop
routing information and delay value, without learning the complete path or the
message origin.
Sphinx packets are self-contained and indistinguishable on the wire, providing
strong metadata
protection. Mix nodes forward packets without retaining state or requiring
knowledge of the
source or destination beyond their immediate routing target.
To ensure uniformity, each Sphinx packet consists of a fixed-length header and a
payload
that is padded to a fixed maximum size. Although the original message payload
may vary in length,
padding ensures that all packets are identical in size on the wire. This ensures
unlinkability
and protects against correlation attacks based on message size.
If a message exceeds the maximum supported payload size, it MUST be fragmented
before being passed
to the Mix Protocol. Fragmentation and reassembly are the responsibility of the
origin protocol
or the top-level application. The Mix Protocol handles only messages that do not
require
fragmentation.
The structure, encoding, and size constraints of the Sphinx packet are detailed
in the following
subsections.
### 8.1 Packet Structure Overview
Each Sphinx packet consists of three fixed-length header fields &mdash; $α$,
$β$, and $γ$ &mdash;
followed by a fixed-length encrypted payload $δ$. Together, these components
enable per-hop message
processing with strong confidentiality and integrity guarantees in a stateless
and unlinkable manner.
- **$α$ (Alpha)**: An ephemeral public value. Each mix node uses its private key
and $α$ to
derive a shared session key for that hop. This session key is used to decrypt
and process
one layer of the packet.
- **$β$ (Beta)**: The nested encrypted routing information. It encodes the next
hop address, the forwarding delay,
integrity check $γ$ for the next hop, and the $β$ for subsequent hops.
- **$γ$ (Gamma)**: A message authentication code computed over $β$ using the
session key derived
from $α$. It ensures header integrity at each hop.
- **$δ$ (Delta)**: The encrypted payload. It consists of the message padded to a
fixed maximum length and
encrypted in layers corresponding to each hop in the mix path.
At each hop, the mix node derives the session key from $α$, verifies the header
integrity
using $γ$, decrypts one layer of $β$ to extract the next hop and delay, and
decrypts one layer
of $δ$. It then constructs a new packet with updated values of $α$, $β$, $γ$,
and $δ$, and
forwards it to the next hop.
All Sphinx packets are fixed in size and indistinguishable on the wire. This
uniform format,
combined with layered encryption and per-hop integrity protection, ensures
unlinkability,
tamper resistance, and robustness against correlation attacks.
The structure and semantics of these fields, the cryptographic primitives used,
and the construction
and processing steps are defined in the following subsections.
### 8.2 Cryptographic Primitives
This section defines the cryptographic primitives used in Sphinx packet
construction and processing.
- **Security Parameter**: All cryptographic operations target a minimum of
$\kappa = 128$ bits of
security, balancing performance with resistance to modern attacks.
- **Elliptic Curve Group $\mathbb{G}$**:
- **Curve**: Curve25519
- **Purpose**: Used for deriving DiffieHellman-style shared key at each hop
using $α$.
- **Representation**: Small 32-byte group elements, efficient for both
encryption and key exchange.
- **Key Derivation Function (KDF)**:
- **Purpose**: To derive encryption keys, IVs, and MAC key from the shared
session key at each hop.
- **Construction**: SHA-256 hash with output truncated to 128 bits.
- **Key Derivation**: The KDF key separation labels (_e.g.,_ `"aes_key"`,
`"mac_key"`)
are fixed strings and MUST be agreed upon across implementations.
- **Symmetric Encryption**: AES-128 in Counter Mode (AES-CTR)
- **Purpose**: To encrypt $β$ and $δ$ for each hop.
- **Keys and IVs**: Each derived from the session key for the hop using the KDF.
- **Message Authentication Code (MAC)**:
- **Construction**: HMAC-SHA-256 with output truncated to 128 bits.
- **Purpose**: To compute $γ$ for each hop.
- **Key**: Derived using KDF from the session key for the hop.
These primitives are used consistently throughout packet construction and
decryption, as described in the following sections.

View File

@@ -0,0 +1,345 @@
---
title: NOISE-X3DH-DOUBLE-RATCHET
name: Secure 1-to-1 channel setup using X3DH and the double ratchet
status: raw
category: Standards Track
tags:
editor: Ramses Fernandez <ramses@status.im>
contributors:
---
## Motivation
The need for secure communications has become paramount.
This specification outlines a protocol describing a
secure 1-to-1 comunication channel between 2 users. The
main components are the X3DH key establishment mechanism,
combined with the double ratchet. The aim of this
combination of schemes is providing a protocol with both
forward secrecy and post-compromise security.
## Theory
The specification is based on the noise protocol framework.
It corresponds to the double ratchet scheme combined with
the X3DH algorithm, which will be used to initialize the former.
We chose to express the protocol in noise to be be able to use
the noise streamlined implementation and proving features.
The X3DH algorithm provides both authentication and forward
secrecy, as stated in the
[X3DH specification](https://signal.org/docs/specifications/x3dh/).
This protocol will consist of several stages:
1. Key setting for X3DH: this step will produce
prekey bundles for Bob which will be fed into X3DH.
It will also allow Alice to generate the keys required
to run the X3DH algorithm correctly.
2. Execution of X3DH: This step will output
a common secret key `SK` together with an additional
data vector `AD`. Both will be used in the double
ratchet algorithm initialization.
3. Execution of the double ratchet algorithm
for forward secure, authenticated communications,
using the common secret key `SK`, obtained from X3DH, as a root key.
The protocol assumes the following requirements:
- Alice knows Bobs Ethereum address.
- Bob is willing to participate in the protocol,
and publishes his public key.
- Bobs ownership of his public key is verifiable,
- Alice wants to send message M to Bob.
- An eavesdropper cannot read Ms content
even if she is storing it or relaying it.
## Syntax
### Cryptographic suite
The following cryptographic functions MUST be used:
- `X488` as Diffie-Hellman function `DH`.
- `SHA256` as KDF.
- `AES256-GCM` as AEAD algorithm.
- `SHA512` as hash function.
- `XEd448` for digital signatures.
### X3DH initialization
This scheme MUST work on the curve curve448.
The X3DH algorithm corresponds to the IX pattern in Noise.
Bob and Alice MUST define personal key pairs
`(ik_B, IK_B)` and `(ik_A, IK_A)` respectively where:
- The key `ik` must be kept secret,
- and the key `IK` is public.
Bob MUST generate new keys using
`(ik_B, IK_B) = GENERATE_KEYPAIR(curve = curve448)`.
Bob MUST also generate a public key pair
`(spk_B, SPK_B) = GENERATE_KEYPAIR(curve = curve448)`.
`SPK` is a public key generated and stored at medium-term.
Both signed prekey and the certificate MUST
undergo periodic replacement.
After replacing the key,
Bob keeps the old private key of `SPK`
for some interval, dependant on the implementation.
This allows Bob to decrypt delayed messages.
Bob MUST sign `SPK` for authentication:
`SigSPK = XEd448(ik, Encode(SPK))`
A final step requires the definition of
`prekey_bundle = (IK, SPK, SigSPK, OPK_i)`
One-time keys `OPK` MUST be generated as
`(opk_B, OPK_B) = GENERATE_KEYPAIR(curve = curve448)`.
Before sending an initial message to Bob,
Alice MUST generate an AD: `AD = Encode(IK_A) || Encode(IK_B)`.
Alice MUST generate ephemeral key pairs
`(ek, EK) = GENERATE_KEYPAIR(curve = curve448)`.
The function `Encode()` transforms a
curve448 public key into a byte sequence.
This is specified in the [RFC 7748](http://www.ietf.org/rfc/rfc7748.txt)
on elliptic curves for security.
One MUST consider `q = 2^446 - 13818066809895115352007386748515426880336692474882178609894547503885`
for digital signatures with `(XEd448_sign, XEd448_verify)`:
```text
XEd448_sign((ik, IK), message):
Z = randbytes(64)
r = SHA512(2^456 - 2 || ik || message || Z )
R = (r * convert_mont(5)) % q
h = SHA512(R || IK || M)
s = (r + h * ik) % q
return (R || s)
```
```text
XEd448_verify(u, message, (R || s)):
if (R.y >= 2^448) or (s >= 2^446): return FALSE
h = (SHA512(R || 156326 || message)) % q
R_check = s * convert_mont(5) - h * 156326
if R == R_check: return TRUE
return FALSE
```
```text
convert_mont(u):
u_masked = u % mod 2^448
inv = ((1 - u_masked)^(2^448 - 2^224 - 3)) % (2^448 - 2^224 - 1)
P.y = ((1 + u_masked) * inv)) % (2^448 - 2^224 - 1)
P.s = 0
return P
```
### Use of X3DH
This specification combines the double ratchet
with X3DH using the following data as initialization for the former:
- The `SK` output from X3DH becomes the `SK`
input of the double ratchet. See section 3.3 of
[Signal Specification](https://signal.org/docs/specifications/doubleratchet/)
for a detailed description.
- The `AD` output from X3DH becomes the `AD`
input of the double ratchet. See sections 3.4 and 3.5 of
[Signal Specification](https://signal.org/docs/specifications/doubleratchet/)
for a detailed description.
- Bobs signed prekey `SigSPKB` from X3DH is used as Bobs
initial ratchet public key of the double ratchet.
X3DH has three phases:
1. Bob publishes his identity key and prekeys to a server,
a network, or dedicated smart contract.
2. Alice fetches a prekey bundle from the server,
and uses it to send an initial message to Bob.
3. Bob receives and processes Alice's initial message.
Alice MUST perform the following computations:
```text
dh1 = DH(IK_A, SPK_B, curve = curve448)
dh2 = DH(EK_A, IK_B, curve = curve448)
dh3 = DH(EK_A, SPK_B)
SK = KDF(dh1 || dh2 || dh3)
```
Alice MUST send to Bob a message containing:
- `IK_A, EK_A`.
- An identifier to Bob's prekeys used.
- A message encrypted with AES256-GCM using `AD` and `SK`.
Upon reception of the initial message, Bob MUST:
1. Perform the same computations above with the `DH()` function.
2. Derive `SK` and construct `AD`.
3. Decrypt the initial message encrypted with `AES256-GCM`.
4. If decryption fails, abort the protocol.
### Initialization of the double datchet
In this stage Bob and Alice have generated key pairs
and agreed a shared secret `SK` using X3DH.
Alice calls `RatchetInitAlice()` defined below:
```text
RatchetInitAlice(SK, IK_B):
state.DHs = GENERATE_KEYPAIR(curve = curve448)
state.DHr = IK_B
state.RK, state.CKs = HKDF(SK, DH(state.DHs, state.DHr))
state.CKr = None
state.Ns, state.Nr, state.PN = 0
state.MKSKIPPED = {}
```
The HKDF function MUST be the proposal by
[Krawczyk and Eronen](http://www.ietf.org/rfc/rfc5869.txt).
In this proposal `chaining_key` and `input_key_material`
MUST be replaced with `SK` and the output of `DH` respectively.
Similarly, Bob calls the function `RatchetInitBob()` defined below:
```text
RatchetInitBob(SK, (ik_B,IK_B)):
state.DHs = (ik_B, IK_B)
state.Dhr = None
state.RK = SK
state.CKs, state.CKr = None
state.Ns, state.Nr, state.PN = 0
state.MKSKIPPED = {}
```
### Encryption
This function performs the symmetric key ratchet.
```text
RatchetEncrypt(state, plaintext, AD):
state.CKs, mk = HMAC-SHA256(state.CKs)
header = HEADER(state.DHs, state.PN, state.Ns)
state.Ns = state.Ns + 1
return header, AES256-GCM_Enc(mk, plaintext, AD || header)
```
The `HEADER` function creates a new message header
containing the public key from the key pair output of the `DH`function.
It outputs the previous chain length `pn`,
and the message number `n`.
The returned header object contains ratchet public key
`dh` and integers `pn` and `n`.
### Decryption
The function `RatchetDecrypt()` decrypts incoming messages:
```text
RatchetDecrypt(state, header, ciphertext, AD):
plaintext = TrySkippedMessageKeys(state, header, ciphertext, AD)
if plaintext != None:
return plaintext
if header.dh != state.DHr:
SkipMessageKeys(state, header.pn)
DHRatchet(state, header)
SkipMessageKeys(state, header.n)
state.CKr, mk = HMAC-SHA256(state.CKr)
state.Nr = state.Nr + 1
return AES256-GCM_Dec(mk, ciphertext, AD || header)
```
Auxiliary functions follow:
```text
DHRatchet(state, header):
state.PN = state.Ns
state.Ns = state.Nr = 0
state.DHr = header.dh
state.RK, state.CKr = HKDF(state.RK, DH(state.DHs, state.DHr))
state.DHs = GENERATE_KEYPAIR(curve = curve448)
state.RK, state.CKs = HKDF(state.RK, DH(state.DHs, state.DHr))
```
```text
SkipMessageKeys(state, until):
if state.NR + MAX_SKIP < until:
raise Error
if state.CKr != none:
while state.Nr < until:
state.CKr, mk = HMAC-SHA256(state.CKr)
state.MKSKIPPED[state.DHr, state.Nr] = mk
state.Nr = state.Nr + 1
```
```text
TrySkippedMessageKey(state, header, ciphertext, AD):
if (header.dh, header.n) in state.MKSKIPPED:
mk = state.MKSKIPPED[header.dh, header.n]
delete state.MKSKIPPED[header.dh, header.n]
return AES256-GCM_Dec(mk, ciphertext, AD || header)
else: return None
```
## Information retrieval
### Static data
Some data, such as the key pairs `(ik, IK)` for Alice and Bob,
MAY NOT be regenerated after a period of time.
Therefore the prekey bundle MAY be stored in long-term
storage solutions, such as a dedicated smart contract
which outputs such a key pair when receiving an Ethereum wallet
address.
Storing static data is done using a dedicated
smart contract `PublicKeyStorage` which associates
the Ethereum wallet address of a user with his public key.
This mapping is done by `PublicKeyStorage`
using a `publicKeys` function, or a `setPublicKey` function.
This mapping is done if the user passed an authorization process.
A user who wants to retrieve a public key associated
with a specific wallet address calls a function `getPublicKey`.
The user provides the wallet address as the only
input parameter for `getPublicKey`.
The function outputs the associated public key
from the smart contract.
### Ephemeral data
Storing ephemeral data on Ethereum MAY be done using
a combination of on-chain and off-chain solutions.
This approach provides an efficient solution to
the problem of storing updatable data in Ethereum.
1. Ethereum stores a reference or a hash
that points to the off-chain data.
2. Off-chain solutions can include systems like IPFS,
traditional cloud storage solutions, or
decentralized storage networks such as a
[Swarm](https://www.ethswarm.org).
In any case, the user stores the associated
IPFS hash, URL or reference in Ethereum.
The fact of a user not updating the ephemeral information
can be understood as Bob not willing to participate in any
communication.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [The Double Ratchet Algorithm](https://signal.org/docs/specifications/doubleratchet/)
- [The X3DH Key Agreement Protocol](https://signal.org/docs/specifications/x3dh/)

View File

@@ -10,54 +10,76 @@ contributors:
## Abstract
This spec integrates [Interep](https://interep.link) into the [RLN](../32/rln-v1.md) spec.
Interep is a group management protocol that allows for the creation of groups of users and the management of their membership.
This spec integrates [Interep](https://interep.link)
into the [RLN](../32/rln-v1.md) spec.
Interep is a group management protocol
that allows for the creation of groups of users and
the management of their membership.
It is used to manage the membership of the RLN group.
Interep ties in web2 identities with reputation, and sorts the users into groups based on their reputation score.
Interep ties in web2 identities with reputation, and
sorts the users into groups based on their reputation score.
For example, a GitHub user with over 100 followers is considered to have "gold" reputation.
Interep uses [Semaphore](https://semaphore.appliedzkp.org/) under the hood to allow anonymous signaling of membership in a group.
Therefore, a user with a "gold" reputation can prove the existence of their membership without revealing their identity.
Interep uses [Semaphore](https://semaphore.appliedzkp.org/)
under the hood to allow anonymous signaling of membership in a group.
Therefore, a user with a "gold" reputation can prove the existence
of their membership without revealing their identity.
RLN is used for spam prevention, and Interep is used for group management.
By using Interep with RLN, we allow users to join RLN membership groups without the need for on-chain financial stake.
By using Interep with RLN,
we allow users to join RLN membership groups
without the need for on-chain financial stake.
## Motivation
To have Sybil-Resistant group management, there are [implementations](https://github.com/vacp2p/rln-contract) of RLN which make use of financial stake on-chain.
To have Sybil-Resistant group management,
there are [implementations](https://github.com/vacp2p/rln-contract)
of RLN which make use of financial stake on-chain.
However, this is not ideal because it reduces the barrier of entry for honest participants.
In this case, honest participants will most likely have a web2 identity accessible to them, which can be used for joining an Interep reputation group.
By modifying the RLN spec to use Interep, we can have Sybil-Resistant group management without the need for on-chain financial stake.
In this case,
honest participants will most likely have a web2 identity accessible to them,
which can be used for joining an Interep reputation group.
By modifying the RLN spec to use Interep,
we can have Sybil-Resistant group management
without the need for on-chain financial stake.
Since RLN and Interep both use Semaphore-style credentials, it is possible to use the same set of credentials for both.
Since RLN and Interep both use Semaphore-style credentials,
it is possible to use the same set of credentials for both.
## Functional Operation
Using Interep with RLN involves the following steps -
1. Generate Semaphore credentials
1. Generate Semaphore credentials
2. Verify reputation and join Interep group
3. Join RLN membership group via interaction with Smart Contract, by passing a proof of membership to the Interep group
3. Join RLN membership group via interaction with Smart Contract,
by passing a proof of membership to the Interep group
### 1. Generate Semaphore credentials
Semaphore credentials are generated in a standard way, depicted in the [Semaphore documentation](https://semaphore.appliedzkp.org/docs/guides/identities#create-deterministic-identities).
Semaphore credentials are generated in a standard way,
depicted in the [Semaphore documentation](https://semaphore.appliedzkp.org/docs/guides/identities#create-deterministic-identities).
### 2. Verify reputation and join Interep group
Using the Interep app deployed on [Goerli](https://goerli.interep.link/), the user can check their reputation tier and join the corresponding group.
Using the Interep app deployed on [Goerli](https://goerli.interep.link/),
the user can check their reputation tier and join the corresponding group.
This results in a transaction to the Interep contract, which adds them to the group.
### 3. Join RLN membership group
Instead of sending funds to the RLN contract to join the membership group, the user can send a proof of membership to the Interep group.
This proof is generated by the user, and is verified by the contract.
The contract ensures that the user is a member of the Interep group, and then adds them to the RLN membership group.
Instead of sending funds to the RLN contract to join the membership group,
the user can send a proof of membership to the Interep group.
This proof is generated by the user, and
is verified by the contract.
The contract ensures that the user is a member of the Interep group, and
then adds them to the RLN membership group.
Following is the modified signature of the register function in the RLN contract -
Following is the modified signature of the register function
in the RLN contract -
```solidity
/// @param groupId: Id of the group.
@@ -83,19 +105,29 @@ Messages are verified the same way as in the [RLN spec](../32/rln-v1.md/#verific
## Slashing
The slashing mechanism is the same as in the [RLN spec](../32/rln-v1.md/#slashing).
It is important to note that the slashing may not have the intended effect on the user, since the only consequence is that they cannot send messages.
This is due to the fact that the user can send a identity commitment in the registration to the RLN contract, which is different than the one used in the Interep group.
It is important to note that the slashing
may not have the intended effect on the user,
since the only consequence is that they cannot send messages.
This is due to the fact that the user
can send a identity commitment in the registration to the RLN contract,
which is different than the one used in the Interep group.
## Proof of Concept
A proof of concept is available at [vacp2p/rln-interp-contract](https://github.com/vacp2p/rln-interep-contract) which integrates Interep with RLN.
A proof of concept is available at
[vacp2p/rln-interp-contract](https://github.com/vacp2p/rln-interep-contract)
which integrates Interep with RLN.
## Security Considerations
1. As mentioned in [Slashing](#slashing), the slashing mechanism may not have the intended effect on the user.
1. As mentioned in [Slashing](#slashing),
the slashing mechanism may not have the intended effect on the user.
2. This spec inherits the security considerations of the [RLN spec](../32/rln-v1.md/#security-considerations).
3. This spec inherits the security considerations of [Interep](https://docs.interep.link/).
4. A user may make multiple registrations using the same Interep proofs but different identity commitments. The way to mitigate this is to check if the nullifier hash has been detected previously in proof verification.
4. A user may make multiple registrations using the same Interep proofs but
different identity commitments.
The way to mitigate this is to check if the nullifier hash has been detected
previously in proof verification.
## References

View File

@@ -9,42 +9,57 @@ contributors:
## Abstract
This specification describes the usage of stealth commitments to add prospective users to a network-governed [32/RLN-V1](./32/rln-v1.md) membership set.
This specification describes the usage of stealth commitments
to add prospective users to a network-governed
[32/RLN-V1](./32/rln-v1.md) membership set.
## Motivation
When [32/RLN-V1](./32/rln-v1.md) is enforced in [10/Waku2](../waku/standards/core/10/waku2.md),
When [32/RLN-V1](./32/rln-v1.md) is enforced in [10/Waku2](../waku/standards/core/10/waku2.md),
all users are required to register to a membership set.
The membership set will store user identities allowing the secure interaction within an application.
Forcing a user to do an on-chain transaction to join a membership set is an onboarding friction,
The membership set will store user identities
allowing the secure interaction within an application.
Forcing a user to do an on-chain transaction
to join a membership set is an onboarding friction,
and some projects may be opposed to this method.
To improve the user experience,
stealth commitments can be used by a counterparty to register identities on the user's behalf,
stealth commitments can be used by a counterparty
to register identities on the user's behalf,
while maintaining the user's anonymity.
This document specifies a privacy-preserving mechanism,
allowing a counterparty to utilize [32/RLN-V1](./32/rln-v1.md) to register an `identityCommitment` on-chain.
Counterparties will be able to register members to a RLN membership set without exposing the user's private keys.
allowing a counterparty to utilize [32/RLN-V1](./32/rln-v1.md)
to register an `identityCommitment` on-chain.
Counterparties will be able to register members
to a RLN membership set without exposing the user's private keys.
## Background
The [32/RLN-V1](./32/rln-v1.md) protocol,
consists of a smart contract that stores a `idenitityCommitment` in a membership set.
The [32/RLN-V1](./32/rln-v1.md) protocol,
consists of a smart contract that stores a `idenitityCommitment`
in a membership set.
In order for a user to join the membership set,
the user is required to make a transaction on the blockchain.
A set of public keys is used to compute a stealth commitment for a user,
as described in [ERC-5564](https://eips.ethereum.org/EIPS/eip-5564).
This specification is an implementation of the [ERC-5564](https://eips.ethereum.org/EIPS/eip-5564) scheme,
This specification is an implementation of the
[ERC-5564](https://eips.ethereum.org/EIPS/eip-5564) scheme,
tailored to the curve that is used in the [32/RLN-V1](./32/rln-v1.md) protocol.
This can be used in a couple of ways in applications:
1. Applications can add users to the [32/RLN-V1](./32/rln-v1.md) membership set in a batch.
2. Users of the application can register other users to the [32/RLN-V1](./32/rln-v1.md) membership set.
This is useful when the prospective user does not have access to funds on the network that [32/RLN-V1](./32/rln-v1.md) is deployed on.
1. Applications can add users
to the [32/RLN-V1](./32/rln-v1.md) membership set in a batch.
2. Users of the application
can register other users to the [32/RLN-V1](./32/rln-v1.md) membership set.
This is useful when the prospective user does not have access to funds
on the network that [32/RLN-V1](./32/rln-v1.md) is deployed on.
## Wire Format Specification
The two parties, the requester and the receiver, MUST exchange the following information:
The two parties, the requester and the receiver,
MUST exchange the following information:
```protobuf
@@ -55,14 +70,16 @@ message Request {
// The viewing public key of the requester
bytes viewing_public_key = 2;
}
```
### Generate Stealth Commitment
The application or user SHOULD generate a `stealth_commitment` after a request to do so is received.
The application or user SHOULD generate a `stealth_commitment`
after a request to do so is received.
This commitment MAY be inserted into the corresponding application membership set.
Once the membership set is updated, the receiver SHOULD exchange the following as a response to the request:
Once the membership set is updated,
the receiver SHOULD exchange the following as a response to the request:
```protobuf
@@ -81,18 +98,22 @@ message Response {
```
The receiver MUST generate an `ephemeral_public_key`, `view_tag` and `stealth_commitment`.
This will be used to check the stealth commitment used to register to the membership set,
The receiver MUST generate an `ephemeral_public_key`,
`view_tag` and `stealth_commitment`.
This will be used to check the stealth commitment
used to register to the membership set,
and the user MUST be able to check ownership with their `viewing_public_key`.
## Implementation Suggestions
An implementation of the Stealth Address scheme is available in the [erc-5564-bn254](https://github.com/rymnc/erc-5564-bn254) repository,
An implementation of the Stealth Address scheme is available in the
[erc-5564-bn254](https://github.com/rymnc/erc-5564-bn254) repository,
which also includes a test to generate a stealth commitment for a given user.
## Security/Privacy Considerations
This specification inherits the security and privacy considerations of the [Stealth Address](https://eips.ethereum.org/EIPS/eip-5564) scheme.
This specification inherits the security and privacy considerations of the
[Stealth Address](https://eips.ethereum.org/EIPS/eip-5564) scheme.
## Copyright

View File

@@ -9,17 +9,26 @@ contributors:
## Abstract
The protocol specified in this document is an improvement of [32/RLN-V1](../32/rln-v1.md), being more general construct, that allows to set various limits for an epoch (it's 1 message per epoch in [32/RLN-V1](../32/rln-v1.md)) while remaining almost as simple as it predecessor.
Moreover, it allows to set different rate-limits for different RLN app users based on some public data, e.g. stake or reputation.
The protocol specified in this document is an improvement of [32/RLN-V1](../32/rln-v1.md),
being more general construct, that allows to set various limits for an epoch
(it's 1 message per epoch in [32/RLN-V1](../32/rln-v1.md))
while remaining almost as simple as it predecessor.
Moreover, it allows to set different rate-limits
for different RLN app users based on some public data,
e.g. stake or reputation.
## Motivation
The main goal of this RFC is to generalize [32/RLN-V1](../32/rln-v1.md) and expand its applications.
The main goal of this RFC is to generalize [32/RLN-V1](../32/rln-v1.md) and
expand its applications.
There are two different subprotocols based on this protocol:
* RLN-Same - RLN with the same rate-limit for all users;
* RLN-Diff - RLN that allows to set different rate-limits for different users.
It is important to note that by using a large epoch limit value, users will be able to remain anonymous, because their `internal_nullifiers` will not be repeated until they exceed the limit.
It is important to note that by using a large epoch limit value,
users will be able to remain anonymous,
because their `internal_nullifiers` will not be repeated until they exceed the limit.
## Flow
@@ -29,11 +38,13 @@ As in [32/RLN-V1](../32/rln-v1.md), the general flow can be described by three s
2. Signaling
3. Verification and slashing
The two sub-protocols have different flows, and hence are defined separately.
The two sub-protocols have different flows, and
hence are defined separately.
### Important note
All terms and parameters used remain the same as in [32/RLN-V1](../32/rln-v1.md), more details [here](../32/rln-v1.md/#technical-overview)
All terms and parameters used remain the same as in [32/RLN-V1](../32/rln-v1.md),
more details [here](../32/rln-v1.md/#technical-overview)
## RLN-Same flow
@@ -41,13 +52,11 @@ All terms and parameters used remain the same as in [32/RLN-V1](../32/rln-v1.md)
The registration process in the RLN-Same subprotocol does not differ from [32/RLN-V1](../32/rln-v1.md).
### Signalling
#### Proof generation
Signalling
For proof generation, the user needs to submit the following fields to the circuit:
```
```js
{
identity_secret: identity_secret_hash,
path_elements: Merkle_proof.path_elements,
@@ -59,11 +68,11 @@ For proof generation, the user needs to submit the following fields to the circu
}
```
#### Calculating output
Calculating output
The following fields are needed for proof output calculation:
```
```js
{
identity_secret_hash: bigint,
external_nullifier: bigint,
@@ -74,7 +83,7 @@ The following fields are needed for proof output calculation:
The output `[y, internal_nullifier]` is calculated in the following way:
```
```js
a_0 = identity_secret_hash
a_1 = poseidonHash([a0, external_nullifier, message_id])
@@ -85,29 +94,36 @@ internal_nullifier = poseidonHash([a_1])
## RLN-Diff flow
### Registration
Registration
**id_commitment** in [32/RLN-V1](../32/rln-v1.md) is equal to `poseidonHash(identity_secret)`.
The goal of RLN-Diff is to set different rate-limits for different users.
It follows that **id_commitment** must somehow depend on the `user_message_limit` parameter, where 0 <= `user_message_limit` <= `message_limit`.
**id_commitment** in [32/RLN-V1](../32/rln-v1.md) is equal to `poseidonHash(identity_secret)`.
The goal of RLN-Diff is to set different rate-limits for different users.
It follows that **id_commitment** must somehow depend
on the `user_message_limit` parameter,
where 0 <= `user_message_limit` <= `message_limit`.
There are few ways to do that:
1. Sending `identity_secret_hash` = `poseidonHash(identity_secret, userMessageLimit)` and zk proof that `user_message_limit` is valid (is in the right range).
This approach requires zkSNARK verification, which is an expensive operation on the blockchain.
2. Sending the same `identity_secret_hash` as in [32/RLN-V1](../32/rln-v1.md) (`poseidonHash(identity_secret)`) and a user_message_limit publicly to a server or smart-contract where `rate_commitment` = `poseidonHash(identity_secret_hash, userMessageLimit)` is calculated.
The leaves in the membership Merkle tree would be the rate_commitments of the users.
This approach requires additional hashing in the Circuit, but it eliminates the need for zk proof verification for the registration.
Both methods are correct, and the choice of the method is left to the implementer.
1. Sending `identity_secret_hash` = `poseidonHash(identity_secret, userMessageLimit)`
and zk proof that `user_message_limit` is valid (is in the right range).
This approach requires zkSNARK verification,
which is an expensive operation on the blockchain.
2. Sending the same `identity_secret_hash` as in [32/RLN-V1](../32/rln-v1.md)
(`poseidonHash(identity_secret)`) and a user_message_limit publicly to a server
or smart-contract where
`rate_commitment` = `poseidonHash(identity_secret_hash, userMessageLimit)` is calculated.
The leaves in the membership Merkle tree would be the rate_commitments of the users.
This approach requires additional hashing in the Circuit, but
it eliminates the need for zk proof verification for the registration.
Both methods are correct, and the choice of the method is left to the implementer.
It is recommended to use second method for the reasons already described.
The following flow description will also be based on the second method.
### Signalling
#### Proof generation
Signalling
For proof generation, the user need to submit the following fields to the circuit:
```
```js
{
identity_secret: identity_secret_hash,
path_elements: Merkle_proof.path_elements,
@@ -119,76 +135,92 @@ For proof generation, the user need to submit the following fields to the circui
}
```
#### Calculating output
Calculating output
The Output is calculated in the same way as the RLN-Same sub-protocol.
### Verification and slashing
Verification and slashing in both subprotocols remain the same as in [32/RLN-V1](../32/rln-v1.md).
The only difference that may arise is the `message_limit` check in RLN-Same, since it is now a public input of the Circuit.
The only difference that may arise is the `message_limit` check in RLN-Same,
since it is now a public input of the Circuit.
### ZK Circuits specification
The design of the [32/RLN-V1](../32/rln-v1.md) circuits is different from the circuits of this protocol.
The design of the [32/RLN-V1](../32/rln-v1.md) circuits
is different from the circuits of this protocol.
RLN-v2 requires additional algebraic constraints.
The membership proof and Shamir's Secret Sharing constraints remain unchanged.
The ZK Circuit is implemented using a [Groth-16 ZK-SNARK](https://eprint.iacr.org/2016/260.pdf),
using the [circomlib](https://docs.circom.io/) library.
using the [circomlib](https://docs.circom.io/) library.
Both schemes contain compile-time constants/system parameters:
* DEPTH - depth of membership Merkle tree
* LIMIT_BIT_SIZE - bit size of `limit` numbers, e.g. for the 16 - maximum `limit` number is 65535.
The main difference of the protocol is that instead of a new polynomial (a new value `a_1`) for a new epoch, a new polynomial is generated for each message.
The user assigns an identifier to each message; the main requirement is that this identifier be in the range from 1 to `limit`.
* DEPTH - depth of membership Merkle tree
* LIMIT_BIT_SIZE - bit size of `limit` numbers,
e.g. for the 16 - maximum `limit` number is 65535.
The main difference of the protocol is that instead of a new polynomial
(a new value `a_1`) for a new epoch, a new polynomial is generated for each message.
The user assigns an identifier to each message;
the main requirement is that this identifier be in the range from 1 to `limit`.
This is proven using range constraints.
### RLN-Same circuit
#### Circuit parameters
**Public Inputs**
- `x`
- `external_nullifier`
- `message_limit` - limit per epoch
Public Inputs
**Private Inputs**
- `identity_secret_hash`
- `path_elements`
- `identity_path_index`
- `message_id`
* `x`
* `external_nullifier`
* `message_limit` - limit per epoch
**Outputs**
- `y`
- `root`
- `internal_nullifier`
Private Inputs
* `identity_secret_hash`
* `path_elements`
* `identity_path_index`
* `message_id`
Outputs
* `y`
* `root`
* `internal_nullifier`
### RLN-Diff circuit
In the RLN-Diff scheme, instead of the public parameter `message_limit`, a parameter is used that is set for each user during registration (`user_message_limit`); the `message_id` value is compared to it in the same way as it is compared to `message_limit` in the case of RLN-Same.
In the RLN-Diff scheme, instead of the public parameter `message_limit`,
a parameter is used that is set for each user during registration (`user_message_limit`);
the `message_id` value is compared to it in the same way
as it is compared to `message_limit` in the case of RLN-Same.
#### Circuit parameters
Circuit parameters
**Public Inputs**
- `x`
- `external_nullifier`
Public Inputs
**Private Inputs**
- `identity_secret_hash`
- `path_elements`
- `identity_path_index`
- `message_id`
- `user_message_limit`
* `x`
* `external_nullifier`
**Outputs**
- `y`
- `root`
- `internal_nullifier`
Private Inputs
* `identity_secret_hash`
* `path_elements`
* `identity_path_index`
* `message_id`
* `user_message_limit`
Outputs
* `y`
* `root`
* `internal_nullifier`
## Appendix A: Security considerations
Although there are changes in the circuits, this spec inherits all the security considerations of [32/RLN-V1](../32/rln-v1.md).
Although there are changes in the circuits,
this spec inherits all the security considerations of [32/RLN-V1](../32/rln-v1.md).
## Copyright
@@ -196,6 +228,6 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
## References
- [1](https://zkresear.ch/t/rate-limit-nullifier-v2-circuits/102)
- [2](https://github.com/Rate-Limiting-Nullifier/rln-circuits-v2)
- [3](../32/rln-v1.md/#technical-overview)
* [1](https://zkresear.ch/t/rate-limit-nullifier-v2-circuits/102)
* [2](https://github.com/Rate-Limiting-Nullifier/rln-circuits-v2)
* [3](../32/rln-v1.md/#technical-overview)

276
vac/raw/sds.md Normal file
View File

@@ -0,0 +1,276 @@
---
title: SDS
name: Scalable Data Sync protocol for distributed logs
status: raw
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Akhil Peddireddy <akhil@status.im>
---
## Abstract
This specification introduces the Scalable Data Sync (SDS) protocol
to achieve end-to-end reliability
when consolidating distributed logs in a decentralized manner.
The protocol is designed for a peer-to-peer (p2p) topology
where an append-only log is maintained by each member of a group of nodes
who may individually append new entries to their local log at any time and
is interested in merging new entries from other nodes in real-time or close to real-time
while maintaining a consistent order.
The outcome of the log consolidation procedure is
that all nodes in the group eventually reflect in their own logs
the same entries in the same order.
The protocol aims to scale to very large groups.
## Motivation
A common application that fits this model is a p2p group chat (or group communication),
where the participants act as log nodes
and the group conversation is modelled as the consolidated logs
maintained on each node.
The problem of end-to-end reliability can then be stated as
ensuring that all participants eventually see the same sequence of messages
in the same causal order,
despite the challenges of network latency, message loss,
and scalability present in any communications transport layer.
The rest of this document will assume the terminology of a group communication:
log nodes being the _participants_ in the group chat
and the logged entries being the _messages_ exchanged between participants.
## Design Assumptions
We make the following simplifying assumptions for a proposed reliability protocol:
* **Broadcast routing:**
Messages are broadcast disseminated by the underlying transport.
The selected transport takes care of routing messages
to all participants of the communication.
* **Store nodes:**
There are high-availability caches (a.k.a. Store nodes)
from which missed messages can be retrieved.
These caches maintain the full history of all messages that have been broadcast.
This is an optional element in the protocol design,
but improves scalability by reducing direct interactions between participants.
* **Message ID:**
Each message has a globally unique, immutable ID (or hash).
Messages can be requested from the high-availability caches or
other participants using the corresponding message ID.
## Wire protocol
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
“SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
### Message
Messages MUST adhere to the following meta structure:
```protobuf
syntax = "proto3";
message HistoryEntry {
string message_id = 1; // Unique identifier of the SDS message, as defined in `Message`
optional bytes retrieval_hint = 2; // Optional information to help remote parties retrieve this SDS message; For example, A Waku deterministic message hash or routing payload hash
}
message Message {
// 1 Reserved for sender/participant id
string message_id = 2; // Unique identifier of the message
string channel_id = 3; // Identifier of the channel to which the message belongs
optional int32 lamport_timestamp = 10; // Logical timestamp for causal ordering in channel
repeated HistoryEntry causal_history = 11; // List of preceding message IDs that this message causally depends on. Generally 2 or 3 message IDs are included.
optional bytes bloom_filter = 12; // Bloom filter representing received message IDs in channel
optional bytes content = 20; // Actual content of the message
}
```
Each message MUST include its globally unique identifier in the `message_id` field,
likely based on a message hash.
The `channel_id` field MUST be set to the identifier of the channel of group communication
that is being synchronized.
For simple group communications without individual channels,
the `channel_id` SHOULD be set to `0`.
The `lamport_timestamp`, `causal_history` and
`bloom_filter` fields MUST be set according to the [protocol steps](#protocol-steps)
set out below.
These fields MAY be left unset in the case of [ephemeral messages](#ephemeral-messages).
The message `content` MAY be left empty for [periodic sync messages](#periodic-sync-message),
otherwise it MUST contain the application-level content
### Participant state
Each participant MUST maintain:
* A Lamport timestamp for each channel of communication,
initialized to current epoch time in nanosecond resolution.
* A bloom filter for received message IDs per channel.
The bloom filter SHOULD be rolled over and
recomputed once it reaches a predefined capacity of message IDs.
Furthermore,
it SHOULD be designed to minimize false positives through an optimal selection of
size and hash functions.
* A buffer for unacknowledged outgoing messages
* A buffer for incoming messages with unmet causal dependencies
* A local log (or history) for each channel,
containing all message IDs in the communication channel,
ordered by Lamport timestamp.
Messages in the unacknowledged outgoing buffer can be in one of three states:
1. **Unacknowledged** - there has been no acknowledgement of message receipt
by any participant in the channel
2. **Possibly acknowledged** - there has been ambiguous indication that the message
has been _possibly_ received by at least one participant in the channel
3. **Acknowledged** - there has been sufficient indication that the message
has been received by at least some of the participants in the channel.
This state will also remove the message from the outgoing buffer.
### Protocol Steps
For each channel of communication,
participants MUST follow these protocol steps to populate and interpret
the `lamport_timestamp`, `causal_history` and `bloom_filter` fields.
#### Send Message
Before broadcasting a message:
* the participant MUST increase its local Lamport timestamp by `1` and
include this in the `lamport_timestamp` field.
* the participant MUST determine the preceding few message IDs in the local history
and include these in an ordered list in the `causal_history` field.
The number of message IDs to include in the `causal_history` depends on the application.
We recommend a causal history of two message IDs.
* the participant MAY include a `retrieval_hint` in the `HistoryEntry`
for each message ID in the `causal_history` field.
This is an application-specific field to facilitate retrieval of messages,
e.g. from high-availability caches.
* the participant MUST include the current `bloom_filter`
state in the broadcast message.
After broadcasting a message,
the message MUST be added to the participants buffer
of unacknowledged outgoing messages.
#### Receive Message
Upon receiving a message,
* the participant MUST [review the ACK status](#review-ack-status) of messages
in its unacknowledged outgoing buffer
using the received message's causal history and bloom filter.
* if the message has a populated `content` field,
the participant MUST include the received message ID in its local bloom filter.
* the participant MUST verify that all causal dependencies are met
for the received message.
Dependencies are met if the message IDs in the `causal_history` of the received message
appear in the local history of the receiving participant.
If all dependencies are met and the message has a populated `content` field,
the participant MUST [deliver the message](#deliver-message).
If dependencies are unmet,
the participant MUST add the message to the incoming buffer of messages
with unmet causal dependencies.
#### Deliver Message
Triggered by the [Receive Message](#receive-message) procedure.
If the received messages Lamport timestamp is greater than the participant's
local Lamport timestamp,
the participant MUST update its local Lamport timestamp to match the received message.
The participant MUST insert the message ID into its local log,
based on Lamport timestamp.
If one or more message IDs with the same Lamport timestamp already exists,
the participant MUST follow the [Resolve Conflicts](#resolve-conflicts) procedure.
#### Resolve Conflicts
Triggered by the [Deliver Message](#deliver-message) procedure.
The participant MUST order messages with the same Lamport timestamp
in ascending order of message ID.
If the message ID is implemented as a hash of the message,
this means the message with the lowest hash would precede
other messages with the same Lamport timestamp in the local log.
#### Review ACK Status
Triggered by the [Receive Message](#receive-message) procedure.
For each message in the unacknowledged outgoing buffer,
based on the received `bloom_filter` and `causal_history`:
* the participant MUST mark all messages in the received `causal_history` as **acknowledged**.
* the participant MUST mark all messages included in the `bloom_filter`
as **possibly acknowledged**.
If a message appears as **possibly acknowledged** in multiple received bloom filters,
the participant MAY mark it as acknowledged based on probabilistic grounds,
taking into account the bloom filter size and hash number.
#### Periodic Incoming Buffer Sweep
The participant MUST periodically check causal dependencies for each message
in the incoming buffer.
For each message in the incoming buffer:
* the participant MAY attempt to retrieve missing dependencies from the Store node
(high-availability cache) or other peers.
It MAY use the application-specific `retrieval_hint` in the `HistoryEntry` to facilitate retrieval.
* if all dependencies of a message are met,
the participant MUST proceed to [deliver the message](#deliver-message).
If a message's causal dependencies have failed to be met
after a predetermined amount of time,
the participant MAY mark them as **irretrievably lost**.
#### Periodic Outgoing Buffer Sweep
The participant MUST rebroadcast **unacknowledged** outgoing messages
after a set period.
The participant SHOULD use distinct resend periods for **unacknowledged** and
**possibly acknowledged** messages,
prioritizing **unacknowledged** messages.
#### Periodic Sync Message
For each channel of communication,
participants SHOULD periodically send sync messages to maintain state.
These sync messages:
* MUST be sent with empty content
* MUST include an incremented Lamport timestamp
* MUST include causal history and bloom filter according to regular message rules
* MUST NOT be added to the unacknowledged outgoing buffer
* MUST NOT be included in causal histories of subsequent messages
* MUST NOT be included in bloom filters
* MUST NOT be added to the local log
Since sync messages are not persisted,
they MAY have non-unique message IDs without impacting the protocol.
To avoid network activity bursts in large groups,
a participant MAY choose to only send periodic sync messages
if no other messages have been broadcast in the channel after a random backoff period.
Participants MUST process the causal history and bloom filter of these sync messages
following the same steps as regular messages,
but MUST NOT persist the sync messages themselves.
#### Ephemeral Messages
Participants MAY choose to send short-lived messages for which no synchronization
or reliability is required.
These messages are termed _ephemeral_.
Ephemeral messages SHOULD be sent with `lamport_timestamp`, `causal_history`, and
`bloom_filter` unset.
Ephemeral messages SHOULD NOT be added to the unacknowledged outgoing buffer
after broadcast.
Upon reception,
ephemeral messages SHOULD be delivered immediately without buffering for causal dependencies
or including in the local log.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

View File

@@ -1,15 +1,15 @@
---
slug: XX
title: XX/(WAKU2|LOGOS|CODEX|*)-TEMPLATE
name: (Waku v2 | Logos | Codex) RFC Template
status: (raw|draft|stable)
category: (Standards Track|Informational|Best Current Practice)
title: TEMPLATE
name: RFC Template
status: raw/draft/stable/deprecated
category: Standards Track/Informational/Best Current Practice
tags: an optional list of tags, not standard
editor: Daniel Kaiser <danielkaiser@status.im>
contributors:
---
# (Info, remove this section)
## (Info, remove this section)
This section contains meta info about writing RFCs.
This section (including its subsections) MUST be removed.
@@ -23,60 +23,69 @@ The `tags` metadata SHOULD contain a list of tags if applicable.
Currently identified tags comprise
* `waku/core-protocol` for Waku protocol definitions (e.g. store, relay, light push),
* `waku/application` for applications built on top of Waku protocol (e.g. eth-dm, toy-chat),
* `waku/application` for applications built on top of Waku protocol
(e.g. eth-dm, toy-chat),
## Abstract
# Abstract
## Background / Rationale / Motivation
This section serves as an introduction providing background information and
a motivation/rationale for why the specified protocol is useful.
# Background / Rationale / Motivation
This section serves as an introduction providing background information and a motivation/rationale for why the specified protocol is useful.
# Theory / Semantics
## Theory / Semantics
A standard track RFC in `stable` status MUST feature this section.
A standard track RFC in `raw` or `draft` status SHOULD feature this section.
This section SHOULD explain in detail how the proposed protocol works.
It may touch on the wire format where necessary for the explanation.
This section MAY also specify endpoint behaviour when receiving specific messages, e.g. the behaviour of certain caches etc.
This section MAY also specify endpoint behaviour when receiving specific messages,
e.g. the behaviour of certain caches etc.
# Wire Format Specification / Syntax
## Wire Format Specification / Syntax
A standard track RFC in `stable` status MUST feature this section.
A standard track RFC in `raw` or `draft` status SHOULD feature this section.
This section SHOULD not contain explanations of semantics and focus on concisely defining the wire format.
This section SHOULD not contain explanations of semantics and
focus on concisely defining the wire format.
Implementations MUST adhere to these exact formats to interoperate with other implementations.
It is fine, if parts of the previous section that touch on the wire format are repeated.
The purpose of this section is having a concise definition of what an implementation sends and accepts.
Parts that are not specified here are considered implementation details. Implementors are free to decide on how to implement these details.
An optional *implementation suggestions* section may provide suggestions on how to approach implementation details, and, if available, point to existing implementations for reference.
The purpose of this section is having a concise definition
of what an implementation sends and accepts.
Parts that are not specified here are considered implementation details.
Implementors are free to decide on how to implement these details.
An optional *implementation suggestions* section may provide suggestions
on how to approach implementation details, and,
if available, point to existing implementations for reference.
# Implementation Suggestions (optional)
## Implementation Suggestions (optional)
## (Further Optional Sections)
# (Further Optional Sections)
# Security/Privacy Considerations
## Security/Privacy Considerations
A standard track RFC in `stable` status MUST feature this section.
A standard track RFC in `raw` or `draft` status SHOULD feature this section.
Informational RFCs (in any state) may feature this section.
If there are none, this section MUST explicitly state that fact.
This section MAY contain additional relevant information, e.g. an explanation as to why there are no security consideration for the respective document.
This section MAY contain additional relevant information,
e.g. an explanation as to why there are no security consideration
for the respective document.
# Copyright
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
# References
## References
References MAY be subdivided into normative and informative.
## normative
A list of references that MUST be read to fully understand and/or implement this protocol.
A list of references that MUST be read to fully understand and/or
implement this protocol.
See [RFC3967 Section 1.1](https://datatracker.ietf.org/doc/html/rfc3967#section-1.1).
## informative
A list of additional references.

View File

@@ -1,5 +1,7 @@
# Waku RFCs
Waku builds a family of privacy-preserving, censorship-resistant communication protocols for web3 applications.
Waku builds a family of privacy-preserving,
censorship-resistant communication protocols for web3 applications.
Contributors can visit [Waku RFCs](https://github.com/waku-org/specs) for new Waku specifications under discussion.
Contributors can visit [Waku RFCs](https://github.com/waku-org/specs)
for new Waku specifications under discussion.

View File

@@ -9,13 +9,17 @@ editor: Hanno Cornelius <hanno@status.im>
## Introduction
This specification describes the JSON-RPC API that Waku v2 nodes MAY adhere to. Refer to the [Waku v2 specification](../10/waku2.md) for more information on Waku v2.
This specification describes the JSON-RPC API that Waku v2 nodes MAY adhere to.
Refer to the [Waku v2 specification](../10/waku2.md)
for more information on Waku v2.
## Wire Protocol
### Transport
Nodes SHOULD expose an accessible [JSON-RPC](https://www.jsonrpc.org/specification) API. The JSON-RPC version SHOULD be `2.0`. Below is an example request:
Nodes SHOULD expose an accessible
[JSON-RPC](https://www.jsonrpc.org/specification) API.
The JSON-RPC version SHOULD be `2.0`. Below is an example request:
```json
{
@@ -37,7 +41,10 @@ Nodes SHOULD expose an accessible [JSON-RPC](https://www.jsonrpc.org/specificati
### Types
In this specification, the primitive types `Boolean`, `String`, `Number` and `Null`, as well as the structured types `Array` and `Object`, are to be interpreted according to the [JSON-RPC specification](https://www.jsonrpc.org/specification#conventions). It also adopts the same capitalisation conventions.
In this specification, the primitive types `Boolean`, `String`,
`Number` and `Null`, as well as the structured types `Array` and `Object`,
are to be interpreted according to the [JSON-RPC specification](https://www.jsonrpc.org/specification#conventions).
It also adopts the same capitalisation conventions.
The following structured types are defined for use throughout the document:
@@ -57,23 +64,27 @@ Refer to [`Waku Message` specification](../14/message.md) for more information.
## Method naming
The JSON-RPC methods in this document are designed to be mappable to HTTP REST endpoints. Method names follow the pattern `<method_type>_waku_<protocol_version>_<api>_<api_version>_<resource>`
The JSON-RPC methods in this document are designed to be mappable to HTTP REST endpoints.
Method names follow the pattern `<method_type>_waku_<protocol_version>_<api>_<api_version>_<resource>`
- `<method_type>`: prefix of the HTTP method type that most closely matches the JSON-RPC function. Supported `method_type` values are `get`, `post`, `put`, `delete` or `patch`.
- `<method_type>`:
prefix of the HTTP method type that most closely matches the JSON-RPC function.
Supported `method_type` values are `get`, `post`, `put`, `delete` or `patch`.
- `<protocol_version>`: Waku version. Currently **v2**.
- `<api>`: one of the listed APIs below, e.g. `store`, `debug`, or `relay`.
- `<api_version>`: API definition version. Currently **v1** for all APIs.
- `<resource>`: the resource or resource path being addressed
The method `post_waku_v2_relay_v1_message`, for example, would map to the HTTP REST endpoint `POST /waku/v2/relay/v1/message`.
The method `post_waku_v2_relay_v1_message`, for example,
would map to the HTTP REST endpoint `POST /waku/v2/relay/v1/message`.
## Debug API
### Types
Types
The following structured types are defined for use on the Debug API:
#### WakuInfo
### WakuInfo
`WakuInfo` is an `Object` containing the following fields:
@@ -82,9 +93,7 @@ The following structured types are defined for use on the Debug API:
| `listenAddresses` | `Array`[`String`] | mandatory | Listening addresses of the node |
| `enrUri` | `String` | optional | ENR URI of the node |
#### WakuInfo
### `get_waku_v2_debug_v1_info`
`get_waku_v2_debug_v1_info`
The `get_waku_v2_debug_v1_info` method retrieves information about a Waku v2 node
@@ -96,94 +105,120 @@ none
- [**`WakuInfo`**](#wakuinfo) - information about a Waku v2 node
### `get_waku_v2_debug_v1_version`
The `get_waku_v2_debug_v1_version` method retrieves the version of a Waku v2 node as a string.
The `get_waku_v2_debug_v1_version` method retrieves the version of a Waku v2 node
as a string.
The version SHOULD follow [semantic versioning](https://semver.org/).
In case the node's current build is based on a git commit between semantic versions,
the retrieved version string MAY contain the git commit hash alone or in combination with the latest semantic version.
the retrieved version string MAY contain the git commit hash alone or
in combination with the latest semantic version.
#### Parameters
Parameters
none
#### Response
Response
- **`string`** - represents the version of a Waku v2 node
## Relay API
Refer to the [Waku Relay specification](../11/relay.md) for more information on the relaying of messages.
Refer to the [Waku Relay specification](../11/relay.md)
for more information on the relaying of messages.
### `post_waku_v2_relay_v1_message`
`post_waku_v2_relay_v1_message`
The `post_waku_v2_relay_v1_message` method publishes a message to be relayed on a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor)
The `post_waku_v2_relay_v1_message` method publishes a message to be relayed on a
[PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor)
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `topic` | `String` | mandatory | The [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) being published on |
| `message` | [`WakuMessage`](#wakumessage) | mandatory | The `message` being relayed |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `post_waku_v2_relay_v1_subscriptions`
`post_waku_v2_relay_v1_subscriptions`
The `post_waku_v2_relay_v1_subscriptions` method subscribes a node to an array of [PubSub `topics`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
The `post_waku_v2_relay_v1_subscriptions` method subscribes a node to an array of
[PubSub `topics`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `topics` | `Array`[`String`] | mandatory | The [PubSub `topics`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) being subscribed to |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `delete_waku_v2_relay_v1_subscriptions`
`delete_waku_v2_relay_v1_subscriptions`
The `delete_waku_v2_relay_v1_subscriptions` method unsubscribes a node from an array of [PubSub `topics`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
The `delete_waku_v2_relay_v1_subscriptions` method unsubscribes a node from an array
of [PubSub `topics`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `topics` | `Array`[`String`] | mandatory | The [PubSub `topics`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) being unsubscribed from |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `get_waku_v2_relay_v1_messages`
`get_waku_v2_relay_v1_messages`
The `get_waku_v2_relay_v1_messages` method returns a list of messages that were received on a subscribed [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) after the last time this method was called. The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object) if no subscription exists for the polled `topic`. If no message has yet been received on the polled `topic`, the server SHOULD return an empty list. This method can be used to poll a `topic` for new messages.
The `get_waku_v2_relay_v1_messages` method returns a list of messages
that were received on a subscribed
[PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor)
after the last time this method was called.
The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object)
if no subscription exists for the polled `topic`.
If no message has yet been received on the polled `topic`,
the server SHOULD return an empty list.
This method can be used to poll a `topic` for new messages.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `topic` | `String` | mandatory | The [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) to poll for the latest messages |
#### Response
Response
- **`Array`[[`WakuMessage`](#wakumessage)]** - the latest `messages` on the polled `topic` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Array`[[`WakuMessage`](#wakumessage)]** -
the latest `messages` on the polled `topic` or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
## Relay Private API
The Private API provides functionality to encrypt/decrypt `WakuMessage` payloads using either symmetric or asymmetric cryptography. This allows backwards compatibility with [Waku v1 nodes](../../legacy/6/waku1.md).
It is the API client's responsibility to keep track of the keys used for encrypted communication. Since keys must be cached by the client and provided to the node to encrypt/decrypt payloads, a Private API SHOULD NOT be exposed on non-local or untrusted nodes.
The Private API provides functionality to encrypt/decrypt `WakuMessage` payloads
using either symmetric or asymmetric cryptography.
This allows backwards compatibility with [Waku v1 nodes](../../legacy/6/waku1.md).
It is the API client's responsibility to keep track of the keys
used for encrypted communication.
Since keys must be cached by the client and
provided to the node to encrypt/decrypt payloads,
a Private API SHOULD NOT be exposed on non-local or untrusted nodes.
### Types
Types
The following structured types are defined for use on the Private API:
#### KeyPair
### KeyPair
`KeyPair` is an `Object` containing the following fields:
@@ -194,33 +229,40 @@ The following structured types are defined for use on the Private API:
### `get_waku_v2_private_v1_symmetric_key`
Generates and returns a symmetric key that can be used for message encryption and decryption.
Generates and returns a symmetric key that can be used for message encryption and
decryption.
#### Parameters
Parameters
none
#### Response
Response
- **`String`** - A new symmetric key as hex encoded data string
### `get_waku_v2_private_v1_asymmetric_keypair`
Generates and returns a public/private key pair that can be used for asymmetric message encryption and decryption.
Generates and returns a public/private key pair
that can be used for asymmetric message encryption and decryption.
#### Parameters
Parameters
none
#### Response
Response
- **[`KeyPair`](#keypair)** - A new public/private key pair as hex encoded data strings
### `post_waku_v2_private_v1_symmetric_message`
`post_waku_v2_private_v1_symmetric_message`
The `post_waku_v2_private_v1_symmetric_message` method publishes a message to be relayed on a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
The `post_waku_v2_private_v1_symmetric_message` method publishes a message
to be relayed on a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
Before being relayed, the message payload is encrypted using the supplied symmetric key. The client MUST provide a symmetric key.
Before being relayed,
the message payload is encrypted using the supplied symmetric key.
The client MUST provide a symmetric key.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
@@ -228,17 +270,22 @@ Before being relayed, the message payload is encrypted using the supplied symmet
| `message` | [`WakuMessage`](#wakumessage) | mandatory | The (unencrypted) `message` being relayed |
| `symkey` | `String` | mandatory | The hex encoded symmetric key to use for payload encryption. This field MUST be included if symmetric key cryptography is selected |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `post_waku_v2_private_v1_asymmetric_message`
`post_waku_v2_private_v1_asymmetric_message`
The `post_waku_v2_private_v1_asymmetric_message` method publishes a message to be relayed on a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
The `post_waku_v2_private_v1_asymmetric_message` method publishes a message
to be relayed on a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
Before being relayed, the message payload is encrypted using the supplied public key. The client MUST provide a public key.
Before being relayed,
the message payload is encrypted using the supplied public key.
The client MUST provide a public key.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
@@ -246,66 +293,88 @@ Before being relayed, the message payload is encrypted using the supplied public
| `message` | [`WakuMessage`](#wakumessage) | mandatory | The (unencrypted) `message` being relayed |
| `publicKey` | `String` | mandatory | The hex encoded public key to use for payload encryption. This field MUST be included if asymmetric key cryptography is selected |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `get_waku_v2_private_v1_symmetric_messages`
The `get_waku_v2_private_v1_symmetric_messages` method decrypts and returns a list of messages that were received on a subscribed [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) after the last time this method was called. The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object) if no subscription exists for the polled `topic`. If no message has yet been received on the polled `topic`, the server SHOULD return an empty list. This method can be used to poll a `topic` for new messages.
The `get_waku_v2_private_v1_symmetric_messages` method decrypts and
returns a list of messages that were received on a subscribed
[PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor)
after the last time this method was called.
The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object)
if no subscription exists for the polled `topic`.
If no message has yet been received on the polled `topic`,
the server SHOULD return an empty list.
This method can be used to poll a `topic` for new messages.
Before returning the messages, the server decrypts the message payloads using the supplied symmetric key. The client MUST provide a symmetric key.
Before returning the messages,
the server decrypts the message payloads using the supplied symmetric key.
The client MUST provide a symmetric key.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `topic` | `String` | mandatory | The [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) to poll for the latest messages |
| `symkey` | `String` | mandatory | The hex encoded symmetric key to use for payload decryption. This field MUST be included if symmetric key cryptography is selected |
#### Response
Response
- **`Array`[[`WakuMessage`](#wakumessage)]** - the latest `messages` on the polled `topic` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Array`[[`WakuMessage`](#wakumessage)]** -
the latest `messages` on the polled `topic` or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `get_waku_v2_private_v1_asymmetric_messages`
`get_waku_v2_private_v1_asymmetric_messages`
The `get_waku_v2_private_v1_asymmetric_messages` method decrypts and returns a list of messages that were received on a subscribed [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) after the last time this method was called. The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object) if no subscription exists for the polled `topic`. If no message has yet been received on the polled `topic`, the server SHOULD return an empty list. This method can be used to poll a `topic` for new messages.
The `get_waku_v2_private_v1_asymmetric_messages` method decrypts and
returns a list of messages that were received on a subscribed [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor)
after the last time this method was called.
The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object)
if no subscription exists for the polled `topic`.
If no message has yet been received on the polled `topic`,
the server SHOULD return an empty list.
This method can be used to poll a `topic` for new messages.
Before returning the messages, the server decrypts the message payloads using the supplied private key. The client MUST provide a private key.
Before returning the messages,
the server decrypts the message payloads using the supplied private key.
The client MUST provide a private key.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `topic` | `String` | mandatory | The [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) to poll for the latest messages |
| `privateKey` | `String` | mandatory | The hex encoded private key to use for payload decryption. This field MUST be included if asymmetric key cryptography is selected |
#### Response
- **`Array`[[`WakuMessage`](#wakumessage)]** - the latest `messages` on the polled `topic` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
Response
- **`Array`[[`WakuMessage`](#wakumessage)]** -
the latest `messages` on the polled `topic` or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
## Store API
Refer to the [Waku Store specification](../13/store.md) for more information on message history retrieval.
### Types
Refer to the [Waku Store specification](../13/store.md)
for more information on message history retrieval.
The following structured types are defined for use on the Store API:
#### StoreResponse
### StoreResponse
`StoreResponse` is an `Object` containing the following fields:
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `messages` | `Array`[[`WakuMessage`](#wakumessage)] | mandatory | Array of retrieved historical messages |
| `pagingOptions` | [`PagingOptions`](#pagingOptions) | [conditional](#get_waku_v2_store_v1_messages) | Paging information from which to resume further historical queries |
| `pagingOptions` | [`PagingOptions`](#pagingoptions) | [conditional](#get_waku_v2_store_v1_messages) | Paging information from which to resume further historical queries |
#### PagingOptions
`PagingOptions` is an `Object` containing the following fields:
`pagingOptions` is an `Object` containing the following fields:
| Field | Type | Inclusion | Description |
| ----: |:-----------------:| :---: |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -322,7 +391,7 @@ The following structured types are defined for use on the Store API:
| `digest` | `String` | mandatory | A hash for the message at this [`Index`](#index) |
| `receivedTime` | `Number` | mandatory | UNIX timestamp in nanoseconds at which the message at this [`Index`](#index) was received |
#### ContentFilter
ContentFilter
`ContentFilter` is an `Object` containing the following fields:
@@ -330,11 +399,19 @@ The following structured types are defined for use on the Store API:
| ----: | :---: | :---: |----------- |
| `contentTopic` | `String` | mandatory | The content topic of a [`WakuMessage`](#wakumessage) |
### `get_waku_v2_store_v1_messages`
`get_waku_v2_store_v1_messages`
The `get_waku_v2_store_v1_messages` method retrieves historical messages on specific content topics. This method MAY be called with [`PagingOptions`](#pagingoptions), to retrieve historical messages on a per-page basis. If the request included [`PagingOptions`](#pagingoptions), the node MUST return messages on a per-page basis and include [`PagingOptions`](#pagingoptions) in the response. These [`PagingOptions`](#pagingoptions) MUST contain a `cursor` pointing to the [`Index`](#index) from which a new page can be requested.
The `get_waku_v2_store_v1_messages` method retrieves historical messages
on specific content topics.
This method MAY be called with [`PagingOptions`](#pagingoptions),
to retrieve historical messages on a per-page basis.
If the request included [`PagingOptions`](#pagingoptions),
the node MUST return messages on a per-page basis and
include [`PagingOptions`](#pagingoptions) in the response.
These [`PagingOptions`](#pagingoptions) MUST contain a `cursor` pointing
to the [`Index`](#index) from which a new page can be requested.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
@@ -344,19 +421,21 @@ The `get_waku_v2_store_v1_messages` method retrieves historical messages on spec
| `endTime` | `Number` | optional | The inclusive upper bound on the [`timestamp`](../14/message.md/#message-attributes) of queried [`WakuMessage`s](#wakumessage). This field holds the Unix epoch time in nanoseconds as a 64-bits integer value. |
| `pagingOptions` | [`PagingOptions`](#pagingoptions) | optional | Pagination information |
#### Response
Response
- [**`StoreResponse`**](#storeresponse) - the response to a `query` for historical messages.
- [**`StoreResponse`**](#storeresponse) -
the response to a `query` for historical messages.
## Filter API
Refer to the [Waku Filter specification](../12/filter.md) for more information on content filtering.
Refer to the [Waku Filter specification](../12/filter.md)
for more information on content filtering.
### Types
Types
The following structured types are defined for use on the Filter API:
#### ContentFilter
### ContentFilter
`ContentFilter` is an `Object` containing the following fields:
@@ -366,57 +445,73 @@ The following structured types are defined for use on the Filter API:
### `post_waku_v2_filter_v1_subscription`
The `post_waku_v2_filter_v1_subscription` method creates a subscription in a [light node](../12/filter.md/#rationale) for messages that matches a content filter and, optionally, a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
The `post_waku_v2_filter_v1_subscription` method creates a subscription in a
[light node](../12/filter.md/#rationale) for messages that matches a content filter
and, optionally, a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `contentFilters` | `Array`[[`ContentFilter`](#contentfilter)] | mandatory | Array of content filters being subscribed to |
| `topic` | `String` | optional | Message topic |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** - `true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `delete_waku_v2_filter_v1_subscription`
`delete_waku_v2_filter_v1_subscription`
The `delete_waku_v2_filter_v1_subscription` method removes subscriptions in a [light node](../12/filter.md/#rationale) matching a content filter and, optionally, a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
The `delete_waku_v2_filter_v1_subscription` method removes subscriptions
in a [light node](../12/filter.md/#rationale) matching a content filter and,
optionally, a [PubSub `topic`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor).
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `contentFilters` | `Array`[[`ContentFilter`](#contentfilter)] | mandatory | Array of content filters being unsubscribed from |
| `topic` | `String` | optional | Message topic |
#### Response
Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
### `get_waku_v2_filter_v1_messages`
The `get_waku_v2_filter_v1_messages` method returns a list of messages that were received on a subscribed content `topic` after the last time this method was called. The server MUST respond with an [error](https://www.jsonrpc.org/specification#error_object) if no subscription exists for the polled content `topic`. If no message has yet been received on the polled content `topic`, the server SHOULD respond with an empty list. This method can be used to poll a content `topic` for new messages.
The `get_waku_v2_filter_v1_messages` method returns a list of messages
that were received on a subscribed content `topic`
after the last time this method was called.
The server MUST respond with an
[error](https://www.jsonrpc.org/specification#error_object)
if no subscription exists for the polled content `topic`.
If no message has yet been received on the polled content `topic`,
the server SHOULD respond with an empty list.
This method can be used to poll a content `topic` for new messages.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `contentTopic` | `String` | mandatory | The content topic to poll for the latest messages |
#### Response
Response
- **`Array`[[`WakuMessage`](#wakumessage)]** - the latest `messages` on the polled content `topic` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **`Array`[[`WakuMessage`](#wakumessage)]** -
the latest `messages` on the polled content `topic` or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
## Admin API
The Admin API provides privileged accesses to the internal operations of a Waku v2 node.
### Types
The Admin API provides privileged accesses
to the internal operations of a Waku v2 node.
The following structured types are defined for use on the Admin API:
#### WakuPeer
### WakuPeer
`WakuPeer` is an `Object` containing the following fields:
@@ -428,13 +523,19 @@ The following structured types are defined for use on the Admin API:
### `get_waku_v2_admin_v1_peers`
The `get_waku_v2_admin_v1_peers` method returns an array of peers registered on this node. Since a Waku v2 node may open either continuous or ad hoc connections, depending on the negotiated protocol, these peers may have different connected states. The same peer MAY appear twice in the returned array, if it is registered for more than one protocol.
The `get_waku_v2_admin_v1_peers` method returns an array of peers
registered on this node.
Since a Waku v2 node may open either continuous or ad hoc connections,
depending on the negotiated protocol,
these peers may have different connected states.
The same peer MAY appear twice in the returned array,
if it is registered for more than one protocol.
#### Parameters
Parameters
none
- none
#### Response
Response
- **`Array`[[`WakuPeer`](#wakupeer)]** - Array of peers registered on this node
@@ -442,30 +543,36 @@ none
The `post_waku_v2_admin_v1_peers` method connects a node to a list of peers.
#### Parameters
Parameters
| Field | Type | Inclusion | Description |
| ----: | :---: | :---: |----------- |
| `peers` | `Array`[`String`] | mandatory | Array of peer `multiaddrs` to connect to. Each `multiaddr` must contain the [location and identity addresses](https://docs.libp2p.io/concepts/addressing/) of a peer. |
#### Response
- **`Bool`** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
Response
- **`Bool`** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
## Example usage
### Store API
Store API
#### `get_waku_v2_store_v1_messages`
### `get_waku_v2_store_v1_messages`
This method is part of the `store` API and the specific resources to retrieve are (historical) `messages`. The protocol (`waku`) is on `v2`, whereas the Store API definition is on `v1`.
This method is part of the `store` API and
the specific resources to retrieve are (historical) `messages`.
The protocol (`waku`) is on `v2`, whereas the Store API definition is on `v1`.
1. `get` *all* the historical messages for content topic **"/waku/2/default-content/proto"**; no paging required
1.`get` *all* the historical messages for content topic
**"/waku/2/default-content/proto"**; no paging required
#### Request
```curl -d '{"jsonrpc":"2.0","id":"id","method":"get_waku_v2_store_v1_messages", "params":["", [{"contentTopic":"/waku/2/default-content/proto"}]]}' --header "Content-Type: application/json" http://localhost:8545```
```curl
curl -d '{"jsonrpc":"2.0","id":"id","method":"get_waku_v2_store_v1_messages", "params":["", [{"contentTopic":"/waku/2/default-content/proto"}]]}' --header "Content-Type: application/json" http://localhost:8545
```
```jsonrpc
{
@@ -481,7 +588,7 @@ This method is part of the `store` API and the specific resources to retrieve ar
}
```
#### Response
Response
```jsonrpc
{
@@ -513,11 +620,16 @@ This method is part of the `store` API and the specific resources to retrieve ar
---
2. `get` a single page of historical messages for content topic **"/waku/2/default-content/proto"**; 2 messages per page, backward direction. Since this is the initial query, no `cursor` is provided, so paging will be performed from the end of the list.
2.`get` a single page of historical messages for content topic **"/waku/2/default-content/proto"**;
2 messages per page, backward direction.
Since this is the initial query, no `cursor` is provided,
so paging will be performed from the end of the list.
#### Request
Request
```curl -d '{"jsonrpc":"2.0","id":"id","method":"get_waku_v2_store_v1_messages", "params":[ "", [{"contentTopic":"/waku/2/default-content/proto"}],{"pageSize":2,"forward":false}]}' --header "Content-Type: application/json" http://localhost:8545```
```curl
curl -d '{"jsonrpc":"2.0","id":"id","method":"get_waku_v2_store_v1_messages", "params":[ "", [{"contentTopic":"/waku/2/default-content/proto"}],{"pageSize":2,"forward":false}]}' --header "Content-Type: application/json" http://localhost:8545
```
```jsonrpc
{
@@ -537,7 +649,7 @@ This method is part of the `store` API and the specific resources to retrieve ar
}
```
#### Response
Response
```jsonrpc
{
@@ -571,11 +683,14 @@ This method is part of the `store` API and the specific resources to retrieve ar
---
3. `get` the next page of historical messages for content topic **"/waku/2/default-content/proto"**, using the cursor received above; 2 messages per page, backward direction.
3.`get` the next page of historical messages for content topic **"/waku/2/default-content/proto"**,
using the cursor received above; 2 messages per page, backward direction.
#### Request
Request
```curl -d '{"jsonrpc":"2.0","id":"id","method":"get_waku_v2_store_v1_messages", "params":[ "", [{"contentTopic":"/waku/2/default-content/proto"}],{"pageSize":2,"cursor":{"digest":"abcdef","receivedTime":1605887187000000000},"forward":false}]}' --header "Content-Type: application/json" http://localhost:8545```
```curl
curl -d '{"jsonrpc":"2.0","id":"id","method":"get_waku_v2_store_v1_messages", "params":[ "", [{"contentTopic":"/waku/2/default-content/proto"}],{"pageSize":2,"cursor":{"digest":"abcdef","receivedTime":1605887187000000000},"forward":false}]}' --header "Content-Type: application/json" http://localhost:8545
```
```jsonrpc
{
@@ -599,7 +714,7 @@ This method is part of the `store` API and the specific resources to retrieve ar
}
```
#### Response
Response
```jsonrpc
{

View File

@@ -9,21 +9,49 @@ contributor: Ebube Ud <ebube@status.im>
## Abstract
This specification outlines how we do accounting and settlement based on the provision and usage of resources, most immediately bandwidth usage and/or storing and retrieving of Waku message. This enables nodes to cooperate and efficiently share resources, and in the case of unequal nodes to settle the difference through a relaxed payment mechanism in the form of sending cheques.
This specification outlines how we do accounting and settlement based on the provision
and usage of resources, most immediately bandwidth usage and/or
storing and retrieving of Waku message.
This enables nodes to cooperate and efficiently share resources,
and in the case of unequal nodes to settle the difference
through a relaxed payment mechanism in the form of sending cheques.
**Protocol identifier***: `/vac/waku/swap/2.0.0-beta1`
## Motivation
The Waku network makes up a service network, and some nodes provide a useful service to other nodes. We want to account for that, and when imbalances arise, settle this. The core of this approach has some theoretical backing in game theory, and variants of it have practically been proven to work in systems such as Bittorrent. The specific model use was developed by the Swarm project (previously part of Ethereum), and we re-use contracts that were written for this purpose.
The Waku network makes up a service network, and
some nodes provide a useful service to other nodes.
We want to account for that, and when imbalances arise, settle this.
The core of this approach has some theoretical backing in game theory, and
variants of it have practically been proven to work in systems such as Bittorrent.
The specific model use was developed by the Swarm project
(previously part of Ethereum), and
we re-use contracts that were written for this purpose.
By using a delayed payment mechanism in the form of cheques, a barter-like mechanism can arise, and nodes can decide on their own policy as opposed to be strictly tied to a specific payment scheme. Additionally, this delayed settlement eases requirements on the underlying network in terms of transaction speed or costs.
By using a delayed payment mechanism in the form of cheques,
a barter-like mechanism can arise, and
nodes can decide on their own policy
as opposed to be strictly tied to a specific payment scheme.
Additionally, this delayed settlement eases requirements
on the underlying network in terms of transaction speed or costs.
Theoretically, nodes providing and using resources over a long, indefinite, period of time can be seen as an iterated form of [Prisoner's Dilemma (PD)](https://en.wikipedia.org/wiki/Prisoner%27s_dilemma). Specifically, and more intuitively, since we have a cost and benefit profile for each provision/usage (of Waku Message's, e.g.), and the pricing can be set such that mutual cooperation is incentivized, this can be analyzed as a form of donations game.
Theoretically, nodes providing and using resources over a long,
indefinite, period of time can be seen as an iterated form of
[Prisoner's Dilemma (PD)](https://en.wikipedia.org/wiki/Prisoner%27s_dilemma).
Specifically, and more intuitively,
since we have a cost and benefit profile for each provision/usage
(of Waku Message's, e.g.), and
the pricing can be set such that mutual cooperation is incentivized,
this can be analyzed as a form of donations game.
## Game Theory - Iterated prisoner's dilemma / donation game
What follows is a sketch of what the game looks like between two nodes. We can look at it as a special case of iterated prisoner's dilemma called a [Donation game](https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Special_case:_donation_game) where each node can cooperate with some benefit `b` at a personal cost `c`, where `b>c`.
What follows is a sketch of what the game looks like between two nodes.
We can look at it as a special case of iterated prisoner's dilemma called a
[Donation game](https://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Special_case:_donation_game)
where each node can cooperate with some benefit `b` at a personal cost `c`,
where `b>c`.
From A's point of view:
@@ -32,7 +60,11 @@ A/B | Cooperate | Defect
Cooperate | b-c | -c
Defect | b | 0
What this means is that if A and B cooperates, A gets some benefit `b` minus a cost `c`. If A cooperates and B defects she only gets the cost, and if she defects and B cooperates A only gets the benefit. If both defect they get neither benefit nor cost.
What this means is that if A and B cooperates,
A gets some benefit `b` minus a cost `c`.
If A cooperates and B defects she only gets the cost,
and if she defects and B cooperates A only gets the benefit.
If both defect they get neither benefit nor cost.
The generalized form of PD is:
@@ -50,18 +82,32 @@ And the following holds:
In our case, this means `b>b-c>0>-c` and `2(b-c)> b-c` which is trivially true.
As this is an iterated game with no clear finishing point in most circumstances, a tit-for-tat strategy is simple, elegant and functional. To be more theoretically precise, this also requires reasonable assumptions on error rate and discount parameter. This captures notions such as "does the perceived action reflect the intended action" and "how much do you value future (uncertain) actions compared to previous actions". See [Axelrod - Evolution of Cooperation (book)](https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation) for more details. In specific circumstances, nodes can choose slightly different policies if there's a strong need for it. A policy is simply how a node chooses to act given a set of circumstances.
As this is an iterated game with no clear finishing point in most circumstances,
a tit-for-tat strategy is simple, elegant and functional.
To be more theoretically precise,
this also requires reasonable assumptions on error rate and discount parameter.
This captures notions such as
"does the perceived action reflect the intended action" and
"how much do you value future (uncertain) actions compared to previous actions".
See [Axelrod - Evolution of Cooperation (book)](https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation)
for more details.
In specific circumstances,
nodes can choose slightly different policies if there's a strong need for it.
A policy is simply how a node chooses to act given a set of circumstances.
A tit-for-tat strategy basically means:
- cooperate first (perform service/beneficial action to other node)
- defect when node stops cooperating (disconnect and similar actions), i.e. when it stops performing according to set parameters re settlement
- defect when node stops cooperating (disconnect and similar actions),
i.e. when it stops performing according to set parameters re settlement
- resume cooperation if other node does so
This can be complemented with node selection mechanisms.
## SWAP protocol overview
We use SWAP for accounting and settlement in conjunction with other request/reply protocols in Waku v2,
We use SWAP for accounting and
settlement in conjunction with other request/reply protocols in Waku v2,
where accounting is done in a pairwise manner.
It is an acronym with several possible meanings (as defined in the Book
of Swarm), for example:
@@ -71,39 +117,63 @@ of Swarm), for example:
- send waiver as payment
- start without a penny
This approach is based on communicating payment thresholds and sending cheques as indications of later payments.
This approach is based on communicating payment thresholds and
sending cheques as indications of later payments.
Communicating payment thresholds MAY be done out-of-band or as part of the handshake.
Sending cheques is done once payment threshold is hit.
See [Book of Swarm](https://web.archive.org/web/20210126130038/https://gateway.ethswarm.org/bzz/latest.bookofswarm.eth) section 3.2. on Peer-to-peer accounting etc., for more context and details.
See [Book of Swarm](https://web.archive.org/web/20210126130038/https://gateway.ethswarm.org/bzz/latest.bookofswarm.eth)
section 3.2. on Peer-to-peer accounting etc., for more context and details.
### Accounting
Nodes perform their own accounting for each relevant peer based on some "volume"/bandwidth metric. For now we take this to mean the number of `WakuMessage`s exchanged.
Nodes perform their own accounting for each relevant peer
based on some "volume"/bandwidth metric.
For now we take this to mean the number of `WakuMessage`s exchanged.
Additionally, a price is attached to each unit. Currently, this is simply a "karma counter" and equal to 1 per message.
Additionally, a price is attached to each unit.
Currently, this is simply a "karma counter" and equal to 1 per message.
Each accounting balance SHOULD be w.r.t. to a given protocol it is accounting for.
NOTE: This may later be complemented with other metrics, either as part of SWAP or more likely outside of it. For example, online time can be communicated and attested to as a form of enhanced quality of service to inform peer selection.
NOTE: This may later be complemented with other metrics,
either as part of SWAP or more likely outside of it.
For example, online time can be communicated and
attested to as a form of enhanced quality of service to inform peer selection.
### Flow
Assuming we have two store nodes, one operating mostly as a client (A) and another as server (B).
Assuming we have two store nodes,
one operating mostly as a client (A) and another as server (B).
1. Node A performs a handshake with B node. B node responds and both nodes communicate their payment threshold.
2. Node A and B creates an accounting entry for the other peer, keep track of peer and current balance.
3. Node A issues a `HistoryRequest`, and B responds with a `HistoryResponse`. Based on the number of WakuMessages in the response, both nodes update their accounting records.
4. When payment threshold is reached, Node A sends over a cheque to reach a neutral balance. Settlement of this is currently out of scope, but would occur through a SWAP contract (to be specified). (mock and hard phase).
1. Node A performs a handshake with B node.
B node responds and both nodes communicate their payment threshold.
2. Node A and B creates an accounting entry for the other peer,
keep track of peer and current balance.
3. Node A issues a `HistoryRequest`, and B responds with a `HistoryResponse`.
Based on the number of WakuMessages in the response,
both nodes update their accounting records.
4. When payment threshold is reached,
Node A sends over a cheque to reach a neutral balance.
Settlement of this is currently out of scope,
but would occur through a SWAP contract (to be specified).
(mock and hard phase).
5. If disconnect threshold is reached, Node B disconnects Node A (mock and hard phase).
Note that not all of these steps are mandatory in initial stages, see below for more details. For example, the payment threshold MAY initially be set out of bounds, and policy is only activated in the mock and hard phase.
Note that not all of these steps are mandatory in initial stages,
see below for more details.
For example, the payment threshold MAY initially be set out of bounds,
and policy is only activated in the mock and hard phase.
### Protobufs
We use protobuf to specify the handshake and signature. This current protobuf is a work in progress. This is needed for mock and hard phase.
We use protobuf to specify the handshake and signature.
This current protobuf is a work in progress.
This is needed for mock and hard phase.
A handshake gives initial information about payment thresholds and possibly other information. A cheque is best thought of as a promise to pay at a later date.
A handshake gives initial information about payment thresholds and
possibly other information.
A cheque is best thought of as a promise to pay at a later date.
```protobuf
@@ -126,11 +196,13 @@ message Cheque {
## Incremental integration and roll-out
To incrementally integrate this into Waku v2, we have divided up the roll-out into three phases:
To incrementally integrate this into Waku v2,
we have divided up the roll-out into three phases:
- Soft - accounting only
- Mock - send mock cheques and take word for it
- Hard Test - blockchain integration and deployed to public testnet (Goerli, Optimism testnet or similar)
- Hard Test - blockchain integration and deployed to public testnet
(Goerli, Optimism testnet or similar)
- Hard Main - deployed to a public mainnet
An implementation MAY support any of these phases.
@@ -140,7 +212,8 @@ An implementation MAY support any of these phases.
In the soft phase only accounting is performed, without consequence for the
peers. No disconnect or sending of cheques is performed at this tage.
SWAP protocol is performed in conjunction with another request-reply protocol to account for its usage.
SWAP protocol is performed in conjunction with another request-reply protocol
to account for its usage.
It SHOULD be done for [13/WAKU2-STORE](../../core/13/store.md)
and it MAY be done for other request/reply protocols.
@@ -151,10 +224,12 @@ and SHOULD indicate when a peer is out of bounds (either of its thresholds met).
In the mock phase, we send mock cheques and send cheques/disconnect peers as appropriate.
- If a node reaches a disconnect threshold, which MUST be outside the payment threshold, it SHOULD disconnect the other peer.
- If a node reaches a disconnect threshold,
which MUST be outside the payment threshold, it SHOULD disconnect the other peer.
- If a node is within payment balance, the other node SHOULD stay connected to it.
- If a node receives a valid Cheque it SHOULD update its internal accounting records.
- If any node behaves badly, the other node is free to disconnect and pick another node.
- If any node behaves badly, the other node is free to disconnect and
pick another node.
- Peer rating is out of scope of this specification.
### Hard phase

View File

@@ -11,11 +11,25 @@ contributors:
- Kim De Mey <kimdemey@status.im>
---
This specification describes the format of Waku messages within the ÐΞVp2p Wire Protocol. This spec substitutes [EIP-627](https://eips.ethereum.org/EIPS/eip-627). Waku is a fork of the original Whisper protocol that enables better usability for resource restricted devices, such as mostly-offline bandwidth-constrained smartphones. It does this through (a) light node support, (b) historic messages (with a mailserver) (c) expressing topic interest for better bandwidth usage and (d) basic rate limiting.
This specification describes the format of Waku messages within the ÐΞVp2p Wire Protocol.
This spec substitutes [EIP-627](https://eips.ethereum.org/EIPS/eip-627).
Waku is a fork of the original Whisper protocol that enables better usability
for resource restricted devices,
such as mostly-offline bandwidth-constrained smartphones.
It does this through (a) light node support,
(b) historic messages (with a mailserver)
(c) expressing topic interest for better bandwidth usage and
(d) basic rate limiting.
## Motivation
Waku was created to incrementally improve in areas that Whisper is lacking in, with special attention to resource restricted devices. We specify the standard for Waku messages in order to ensure forward compatibility of different Waku clients, backwards compatibility with Whisper clients, as well as to allow multiple implementations of Waku and its capabilities. We also modify the language to be more unambiguous, concise and consistent.
Waku was created to incrementally improve in areas that Whisper is lacking in,
with special attention to resource restricted devices.
We specify the standard for Waku messages
in order to ensure forward compatibility of different Waku clients,
backwards compatibility with Whisper clients,
as well as to allow multiple implementations of Waku and its capabilities.
We also modify the language to be more unambiguous, concise and consistent.
## Definitions
@@ -29,23 +43,47 @@ Waku was created to incrementally improve in areas that Whisper is lacking in, w
### Use of DevP2P
For nodes to communicate, they MUST implement devp2p and run RLPx. They MUST have some way of connecting to other nodes. Node discovery is largely out of scope for this spec, but see the appendix for some suggestions on how to do this.
For nodes to communicate, they MUST implement devp2p and run RLPx.
They MUST have some way of connecting to other nodes.
Node discovery is largely out of scope for this spec,
but see the appendix for some suggestions on how to do this.
### Gossip based routing
In Whisper, messages are gossiped between peers. Whisper is a form of rumor-mongering protocol that works by flooding to its connected peers based on some factors. Messages are eligible for retransmission until their TTL expires. A node SHOULD relay messages to all connected nodes if an envelope matches their PoW and bloom filter settings. If a node works in light mode, it MAY choose not to forward envelopes. A node MUST NOT send expired envelopes, unless the envelopes are sent as a [mailserver](./mailserver.md) response. A node SHOULD NOT send a message to a peer that it has already sent before.
In Whisper, messages are gossiped between peers.
Whisper is a form of rumor-mongering protocol
that works by flooding to its connected peers based on some factors.
Messages are eligible for retransmission until their TTL expires.
A node SHOULD relay messages to all connected nodes
if an envelope matches their PoW and bloom filter settings.
If a node works in light mode, it MAY choose not to forward envelopes.
A node MUST NOT send expired envelopes,
unless the envelopes are sent as a [mailserver](./mailserver.md) response.
A node SHOULD NOT send a message to a peer that it has already sent before.
## Wire Specification
### Use of RLPx transport protocol
All Waku messages are sent as devp2p RLPx transport protocol, version 5[^1] packets. These packets MUST be RLP-encoded arrays of data containing two objects: packet code followed by another object (whose type depends on the packet code). See [informal RLP spec](https://github.com/ethereum/wiki/wiki/RLP) and the [Ethereum Yellow Paper, appendix B](https://ethereum.github.io/yellowpaper/paper.pdf) for more details on RLP.
All Waku messages are sent as devp2p RLPx transport protocol,
version 5[^1] packets.
These packets MUST be RLP-encoded arrays of data containing two objects:
packet code followed by another object (whose type depends on the packet code).
See [informal RLP spec](https://github.com/ethereum/wiki/wiki/RLP) and
the [Ethereum Yellow Paper, appendix B](https://ethereum.github.io/yellowpaper/paper.pdf)
for more details on RLP.
Waku is a RLPx subprotocol called `waku` with version `0`. The version number corresponds to the major version in the header spec. Minor versions should not break compatibility of `waku`, this would result in a new major. (Some exceptions to this apply in the Draft stage of where client implementation is rapidly change).
Waku is a RLPx subprotocol called `waku` with version `0`.
The version number corresponds to the major version in the header spec.
Minor versions should not break compatibility of `waku`,
this would result in a new major.
(Some exceptions to this apply in the Draft stage
of where client implementation is rapidly change).
### ABNF specification
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234) we have the following format:
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234)
we have the following format:
```abnf
; Packet codes 0 - 127 are reserved for Waku protocol
@@ -129,21 +167,23 @@ p2p-message = 1*waku-envelope
packet-format = "[" packet-code packet-format "]"
required-packet = 0 status /
1 messages /
22 status-update /
1 messages /
22 status-update /
optional-packet = 126 p2p-request / 127 p2p-message
packet = "[" required-packet [ optional-packet ] "]"
```
All primitive types are RLP encoded. Note that, per RLP specification, integers are encoded starting from `0x00`.
All primitive types are RLP encoded. Note that, per RLP specification,
integers are encoded starting from `0x00`.
### Packet Codes
The message codes reserved for Waku protocol: 0 - 127.
Messages with unknown codes MUST be ignored without generating any error, for forward compatibility of future versions.
Messages with unknown codes MUST be ignored without generating any error,
for forward compatibility of future versions.
The Waku sub-protocol MUST support the following packet codes:
@@ -170,17 +210,25 @@ The Status message serves as a Waku handshake and peers MUST exchange this
message upon connection. It MUST be sent after the RLPx handshake and prior to
any other Waku messages.
A Waku node MUST await the Status message from a peer before engaging in other Waku protocol activity with that peer.
When a node does not receive the Status message from a peer, before a configurable timeout, it SHOULD disconnect from that peer.
A Waku node MUST await the Status message from a peer
before engaging in other Waku protocol activity with that peer.
When a node does not receive the Status message from a peer,
before a configurable timeout, it SHOULD disconnect from that peer.
Upon retrieval of the Status message, the node SHOULD validate the message
received and validated the Status message. Note that its peer might not be in
the same state.
When a node is receiving other Waku messages from a peer before a Status
message is received, the node MUST ignore these messages and SHOULD disconnect from that peer. Status messages received after the handshake is completed MUST also be ignored.
message is received,
the node MUST ignore these messages and SHOULD disconnect from that peer.
Status messages received after the handshake is completed MUST also be ignored.
The status message MUST contain an association list containing various options. All options within this association list are OPTIONAL, ordering of the key-value pairs is not guaranteed and therefore MUST NOT be relied on. Unknown keys in the association list SHOULD be ignored.
The status message MUST contain an association list containing various options.
All options within this association list are OPTIONAL,
ordering of the key-value pairs is not guaranteed and
therefore MUST NOT be relied on.
Unknown keys in the association list SHOULD be ignored.
#### Messages
@@ -188,87 +236,130 @@ This packet is used for sending the standard Waku envelopes.
#### Status Update
The Status Update message is used to communicate an update of the settings of the node.
The Status Update message is used to communicate an update
of the settings of the node.
The format is the same as the Status message, all fields are optional.
If none of the options are specified the message MUST be ignored and considered a noop.
Fields that are omitted are considered unchanged, fields that haven't changed SHOULD not
be transmitted.
If none of the options are specified the message MUST be ignored and
considered a noop.
Fields that are omitted are considered unchanged,
fields that haven't changed SHOULD not be transmitted.
**PoW Requirement update**
##### PoW Requirement update
When PoW is updated, peers MUST NOT deliver the sender envelopes with PoW lower than specified in this message.
When PoW is updated, peers MUST NOT deliver the sender envelopes
with PoW lower than specified in this message.
PoW is defined as average number of iterations, required to find the current BestBit (the number of leading zero bits in the hash), divided by message size and TTL:
PoW is defined as average number of iterations,
required to find the current BestBit
(the number of leading zero bits in the hash), divided by message size and TTL:
PoW = (2**BestBit) / (size * TTL)
> PoW = (2**BestBit) / (size * TTL)
PoW calculation:
fn short_rlp(envelope) = rlp of envelope, excluding env_nonce field.
fn pow_hash(envelope, env_nonce) = sha3(short_rlp(envelope) ++ env_nonce)
fn pow(pow_hash, size, ttl) = 2**leading_zeros(pow_hash) / (size * ttl)
```rust
fn short_rlp(envelope) = rlp of envelope, excluding env_nonce field.
fn pow_hash(envelope, env_nonce) = sha3(short_rlp(envelope) ++ env_nonce)
fn pow(pow_hash, size, ttl) = 2**leading_zeros(pow_hash) / (size * ttl)
```
where size is the size of the RLP-encoded envelope, excluding `env_nonce` field (size of `short_rlp(envelope)`).
where size is the size of the RLP-encoded envelope,
excluding `env_nonce` field (size of `short_rlp(envelope)`).
**Bloom filter update**
##### Bloom filter update
The bloom filter is used to identify a number of topics to a peer without compromising (too much) privacy over precisely what topics are of interest. Precise control over the information content (and thus efficiency of the filter) may be maintained through the addition of bits.
The bloom filter is used to identify a number of topics
to a peer without compromising (too much)
privacy over precisely what topics are of interest.
Precise control over the information content (and thus efficiency of the filter)
may be maintained through the addition of bits.
Blooms are formed by the bitwise OR operation on a number of bloomed topics. The bloom function takes the topic and projects them onto a 512-bit slice. At most, three bits are marked for each bloomed topic.
Blooms are formed by the bitwise OR operation on a number of bloomed topics.
The bloom function takes the topic and projects them onto a 512-bit slice.
At most, three bits are marked for each bloomed topic.
The projection function is defined as a mapping from a 4-byte slice S to a 512-bit slice D; for ease of explanation, S will dereference to bytes, whereas D will dereference to bits.
The projection function is defined as a mapping from a 4-byte slice S
to a 512-bit slice D; for ease of explanation, S will dereference to bytes,
whereas D will dereference to bits.
LET D[*] = 0
FOREACH i IN { 0, 1, 2 } DO
LET n = S[i]
IF S[3] & (2 ** i) THEN n += 256
D[n] = 1
END FOR
```python
LET D[*] = 0
FOREACH i IN { 0, 1, 2 } DO
LET n = S[i]
IF S[3] & (2 ** i) THEN n += 256
D[n] = 1
END FOR
```
A full bloom filter (all the bits set to 1) means that the node is to be considered a `Full Node` and it will accept any topic.
A full bloom filter (all the bits set to 1)
means that the node is to be considered a `Full Node` and it will accept any topic.
If both Topic Interest and bloom filter are specified, Topic Interest always takes precedence and bloom filter MUST be ignored.
If both Topic Interest and bloom filter are specified,
Topic Interest always takes precedence and bloom filter MUST be ignored.
If only bloom filter is specified, the current Topic Interest MUST be discarded and only the updated bloom filter MUST be used when forwarding or posting envelopes.
If only bloom filter is specified, the current Topic Interest MUST be discarded and
only the updated bloom filter MUST be used when forwarding or posting envelopes.
A bloom filter with all bits set to 0 signals that the node is not currently interested in receiving any envelope.
A bloom filter with all bits set to 0 signals
that the node is not currently interested in receiving any envelope.
**Topic Interest update**
##### Topic Interest update
This packet is used by Waku nodes for sharing their interest in messages with specific topics. It does this in a more bandwidth considerate way, at the expense of some metadata protection. Peers MUST only send envelopes with specified topics.
This packet is used by Waku nodes for sharing their interest
in messages with specific topics.
It does this in a more bandwidth considerate way,
at the expense of some metadata protection.
Peers MUST only send envelopes with specified topics.
It is currently bounded to a maximum of 10000 topics.
If you are interested in more topics than that,
this is currently underspecified and likely requires updating it.
The constant is subject to change.
It is currently bounded to a maximum of 10000 topics. If you are interested in more topics than that, this is currently underspecified and likely requires updating it. The constant is subject to change.
If only Topic Interest is specified,
the current bloom filter MUST be discarded and
only the updated Topic Interest MUST be used when forwarding or posting envelopes.
If only Topic Interest is specified, the current bloom filter MUST be discarded and only the updated Topic Interest MUST be used when forwarding or posting envelopes.
An empty array signals that the node
is not currently interested in receiving any envelope.
An empty array signals that the node is not currently interested in receiving any envelope.
**Rate Limits update**
##### Rate Limits update
This packet is used for informing other nodes of their self defined rate limits.
In order to provide basic Denial-of-Service attack protection, each node SHOULD define its own rate limits. The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
In order to provide basic Denial-of-Service attack protection,
each node SHOULD define its own rate limits.
The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
Each node MAY decide to whitelist, i.e. do not rate limit, selected IPs or peer IDs.
If a peer exceeds node's rate limits, the connection between them MAY be dropped.
Each node SHOULD broadcast its rate limits to its peers using the rate limits packet. The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD broadcast its rate limits to its peers using the rate limits packet.
The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD respect rate limits advertised by its peers. The number of packets SHOULD be throttled in order not to exceed peer's rate limits. If the limit gets exceeded, the connection MAY be dropped by the peer.
Each node SHOULD respect rate limits advertised by its peers.
The number of packets SHOULD be throttled in order not to exceed peer's rate limits.
If the limit gets exceeded, the connection MAY be dropped by the peer.
**Message Confirmations update**
##### Message Confirmations update
Message confirmations tell a node that a message originating from it has been received by its peers, allowing a node to know whether a message has or has not been received.
Message confirmations tell a node that a message originating
from it has been received by its peers,
allowing a node to know whether a message has or has not been received.
A node MAY send a message confirmation for any batch of messages received with a packet Messages Code.
A node MAY send a message confirmation for any batch of messages
received with a packet Messages Code.
A message confirmation is sent using Batch Acknowledge packet or Message Response packet. The Batch Acknowledge packet is followed by a keccak256 hash of the envelopes batch data.
A message confirmation is sent using Batch Acknowledge packet or
Message Response packet.
The Batch Acknowledge packet is followed by a keccak256 hash
of the envelopes batch data.
The current `version` of the message response is `1`.
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234) we have the following format:
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234)
we have the following format:
```abnf
; a version of the Message Response
@@ -294,159 +385,268 @@ confirmation = "[" version response "]"
```
The supported codes:
`1`: means time sync error which happens when an envelope is too old or created in the future (the root cause is no time sync between nodes).
The drawback of sending message confirmations is that it increases the noise in the network because for each sent message, a corresponding confirmation is broadcast by one or more peers.
`1`: means time sync error which happens when an envelope is too old or
created in the future (the root cause is no time sync between nodes).
The drawback of sending message confirmations
is that it increases the noise in the network because for each sent message,
a corresponding confirmation is broadcast by one or more peers.
#### P2P Request
This packet is used for sending Dapp-level peer-to-peer requests, e.g. Waku Mail Client requesting old messages from the [Waku Mail Server](./mailserver.md).
This packet is used for sending Dapp-level peer-to-peer requests,
e.g. Waku Mail Client requesting old messages from the [Waku Mail Server](./mailserver.md).
#### P2P Message
This packet is used for sending the peer-to-peer messages, which are not supposed to be forwarded any further. E.g. it might be used by the Waku Mail Server for delivery of old (expired) messages, which is otherwise not allowed.
This packet is used for sending the peer-to-peer messages,
which are not supposed to be forwarded any further.
E.g. it might be used by the Waku Mail Server for delivery of old
(expired) messages, which is otherwise not allowed.
### Payload Encryption
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme with SECP-256k1 public key.
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme
with SECP-256k1 public key.
Symmetric encryption uses AES GCM algorithm with random 96-bit nonce.
### Packet code Rationale
Packet codes `0x00` and `0x01` are already used in all Waku / Whisper versions. Packet code `0x02` and `0x03` were previously used in Whisper but are deprecated as of Waku v0.4
Packet codes `0x00` and `0x01` are already used in all Waku / Whisper versions.
Packet code `0x02` and `0x03` were previously used in Whisper but
are deprecated as of Waku v0.4
Packet code `0x22` is used to dynamically change the settings of a node.
Packet codes `0x7E` and `0x7F` may be used to implement Waku Mail Server and Client. Without P2P messages it would be impossible to deliver the old messages, since they will be recognized as expired, and the peer will be disconnected for violating the Whisper protocol. They might be useful for other purposes when it is not possible to spend time on PoW, e.g. if a stock exchange will want to provide live feed about the latest trades.
Packet codes `0x7E` and `0x7F` may be used to implement Waku Mail Server and Client.
Without P2P messages it would be impossible to deliver the old messages,
since they will be recognized as expired,
and the peer will be disconnected for violating the Whisper protocol.
They might be useful for other purposes
when it is not possible to spend time on PoW,
e.g. if a stock exchange will want to provide live feed about the latest trades.
## Additional capabilities
Waku supports multiple capabilities. These include light node, rate limiting and bridging of traffic. Here we list these capabilities, how they are identified, what properties they have and what invariants they must maintain.
Waku supports multiple capabilities.
These include light node, rate limiting and bridging of traffic.
Here we list these capabilities, how they are identified,
what properties they have and what invariants they must maintain.
Additionally there is the capability of a mailserver which is documented in its on [specification](mailserver.md).
Additionally there is the capability of a mailserver
which is documented in its on [specification](mailserver.md).
### Light node
The rationale for light nodes is to allow for interaction with waku on resource restricted devices as bandwidth can often be an issue.
The rationale for light nodes is to allow for interaction with waku
on resource restricted devices as bandwidth can often be an issue.
Light nodes MUST NOT forward any incoming messages, they MUST only send their own messages. When light nodes happen to connect to each other, they SHOULD disconnect. As this would result in messages being dropped between the two.
Light nodes MUST NOT forward any incoming messages,
they MUST only send their own messages.
When light nodes happen to connect to each other,
they SHOULD disconnect.
As this would result in messages being dropped between the two.
Light nodes are identified by the `light_node` value in the status message.
### Accounting for resources (experimental)
Nodes MAY implement accounting, keeping track of resource usage. It is heavily inspired by Swarm's [SWAP protocol](https://www.bokconsulting.com.au/wp-content/uploads/2016/09/tron-fischer-sw3.pdf), and works by doing pairwise accounting for resources.
Nodes MAY implement accounting, keeping track of resource usage.
It is heavily inspired by
Swarm's [SWAP protocol](https://www.bokconsulting.com.au/wp-content/uploads/2016/09/tron-fischer-sw3.pdf),
and works by doing pairwise accounting for resources.
Each node keeps track of resource usage with all other nodes. Whenever an envelope is received from a node that is expected (fits bloom filter or topic interest, is legal, etc) this is tracked.
Each node keeps track of resource usage with all other nodes.
Whenever an envelope is received from a node that is expected
(fits bloom filter or topic interest, is legal, etc) this is tracked.
Every epoch (say, every minute or every time an event happens) statistics SHOULD be aggregated and saved by the client:
Every epoch (say, every minute or every time an event happens)
statistics SHOULD be aggregated and saved by the client:
| peer | sent | received |
|-------|------|----------|
| peer1 | 0 | 123 |
| peer2 | 10 | 40 |
In later versions this will be amended by nodes communication thresholds, settlements and disconnect logic.
In later versions this will be amended by nodes communication thresholds,
settlements and disconnect logic.
## Upgradability and Compatibility
### General principles and policy
These are policies that guide how we make decisions when it comes to upgradability, compatibility, and extensibility:
These are policies that guide how we make decisions when it comes to upgradability,
compatibility, and extensibility:
1. Waku aims to be compatible with previous and future versions.
2. In cases where we want to break this compatibility, we do so gracefully and as a single decision point.
2. In cases where we want to break this compatibility, we do so gracefully and
as a single decision point.
3. To achieve this,
we employ the following two general strategies:
3. To achieve this, we employ the following two general strategies:
- a) Accretion (including protocol negotiation) over changing data
- b) When we want to change things, we give it a new name (for example, a version number).
- b) When we want to change things, we give it a new name
(for example, a version number).
Examples:
- We enable bridging between `shh/6` and `waku/0` until such a time as when we are ready to gracefully drop support for `shh/6` (1, 2, 3).
- When we add parameter fields, we (currently) do so by accreting them in a list, so old clients can ignore new fields (dynamic list) and new clients can use new capabilities (1, 3).
- To better support (2) and (3) in the future, we will likely release a new version that gives better support for open, growable maps (association lists or native map type) (3)
- When we we want to provide a new set of messages that have different requirements, we do so under a new protocol version and employ protocol versioning. This is a form of accretion at a level above - it ensures a client can support both protocols at once and drop support for legacy versions gracefully. (1,2,3)
- We enable bridging between `shh/6` and
`waku/0` until such a time as when we are ready to gracefully drop support
for `shh/6` (1, 2, 3).
- When we add parameter fields, we (currently) do so by accreting them in a list,
so old clients can ignore new fields (dynamic list)
and new clients can use new capabilities (1, 3).
- To better support (2) and (3) in the future,
we will likely release a new version that gives better support for open,
growable maps (association lists or native map type) (3)
- When we we want to provide a new set of messages that have different requirements,
we do so under a new protocol version and employ protocol versioning.
This is a form of accretion at a level above -
it ensures a client can support both protocols at once and
drop support for legacy versions gracefully. (1,2,3)
### Backwards Compatibility
Waku is a different subprotocol from Whisper so it isn't directly compatible. However, the data format is the same, so compatibility can be achieved by the use of a bridging mode as described below. Any client which does not implement certain packet codes should gracefully ignore the packets with those codes. This will ensure the forward compatibility.
Waku is a different subprotocol from Whisper so it isn't directly compatible.
However, the data format is the same,
so compatibility can be achieved by the use of a bridging mode as described below.
Any client which does not implement certain packet codes
should gracefully ignore the packets with those codes.
This will ensure the forward compatibility.
### Waku-Whisper bridging
`waku/0` and `shh/6` are different DevP2P subprotocols, however they share the same data format making their envelopes compatible. This means we can bridge the protocols naively, this works as follows.
`waku/0` and `shh/6` are different DevP2P subprotocols,
however they share the same data format making their envelopes compatible.
This means we can bridge the protocols naively, this works as follows.
**Roles:**
- Waku client A, only Waku capability
- Whisper client B, only Whisper capability
- WakuWhisper bridge C, both Waku and Whisper capability
**Flow:**
1. A posts message; B posts message.
2. C picks up message from A and B and relays them both to Waku and Whisper.
3. A receives message on Waku; B on Whisper.
**Note**: This flow means if another bridge C1 is active, we might get duplicate relaying for a message between C1 and C2. I.e. Whisper(<>Waku<>Whisper)<>Waku, A-C1-C2-B. Theoretically this bridging chain can get as long as TTL permits.
**Note**: This flow means if another bridge C1 is active,
we might get duplicate relaying for a message between C1 and C2.
I.e. Whisper(<>Waku<>Whisper)<>Waku, A-C1-C2-B.
Theoretically this bridging chain can get as long as TTL permits.
### Forward Compatibility
It is desirable to have a strategy for maintaining forward compatibility between `waku/0` and future version of waku. Here we outline some concerns and strategy for this.
It is desirable to have a strategy for maintaining forward compatibility
between `waku/0` and future version of waku.
Here we outline some concerns and strategy for this.
- **Connecting to nodes with multiple versions:** The way this SHOULD be accomplished in the future is by negotiating the versions of subprotocols, within the `hello` message nodes transmit their capabilities along with a version. As suggested in [EIP-8](https://eips.ethereum.org/EIPS/eip-8), if a node connects that has a higher version number for a specific capability, the node with a lower number SHOULD assume backwards compatibility. The node with the higher version will decide if compatibility can be assured between versions, if this is not the case it MUST disconnect.
- **Adding new packet codes:** New packet codes can be added easily due to the available packet codes. Unknown packet codes SHOULD be ignored. Upgrades that add new packet codes SHOULD implement some fallback mechanism if no response was received for nodes that do not yet understand this packet.
- **Adding new options in `status-options`:** New options can be added to the `status-options` association list in the `status` and `status-update` packet as options are OPTIONAL and unknown option keys SHOULD be ignored. A node SHOULD NOT disconnect from a peer when receiving `status-options` with unknown option keys.
- **Connecting to nodes with multiple versions:**
The way this SHOULD be accomplished in the future
is by negotiating the versions of subprotocols,
within the `hello` message nodes transmit their capabilities along with a version.
As suggested in [EIP-8](https://eips.ethereum.org/EIPS/eip-8),
if a node connects that has a higher version number for a specific capability,
the node with a lower number SHOULD assume backwards compatibility.
The node with the higher version
will decide if compatibility can be assured between versions,
if this is not the case it MUST disconnect.
- **Adding new packet codes:**
New packet codes can be added easily due to the available packet codes.
Unknown packet codes SHOULD be ignored.
Upgrades that add new packet codes SHOULD implement some fallback mechanism
if no response was received for nodes that do not yet understand this packet.
- **Adding new options in `status-options`:**
New options can be added to the `status-options` association list
in the `status` and `status-update` packet as options are OPTIONAL and
unknown option keys SHOULD be ignored.
A node SHOULD NOT disconnect from a peer
when receiving `status-options` with unknown option keys.
## Appendix A: Security considerations
There are several security considerations to take into account when running Waku. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used. The security considerations for extra capabilities such as [mailservers](./mailserver.md#security-considerations) can be found in their respective specifications.
There are several security considerations to take into account when running Waku.
Chief among them are: scalability, DDoS-resistance and privacy.
These also vary depending on what capabilities are used.
The security considerations for extra capabilities such as [mailservers](./mailserver.md#security-considerations)
can be found in their respective specifications.
### Scalability and UX
**Bandwidth usage:**
In version 0 of Waku, bandwidth usage is likely to be an issue. For more investigation into this, see the theoretical scaling model described [here](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
In version 0 of Waku, bandwidth usage is likely to be an issue.
For more investigation into this,
see the theoretical scaling model described [here](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
**Gossip-based routing:**
Use of gossip-based routing doesn't necessarily scale. It means each node can see a message multiple times, and having too many light nodes can cause propagation probability that is too low. See [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/) for more and a possible Kademlia based alternative.
Use of gossip-based routing doesn't necessarily scale.
It means each node can see a message multiple times,
and having too many light nodes can cause propagation probability that is too low.
See [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/)
for more and a possible Kademlia based alternative.
**Lack of incentives:**
Waku currently lacks incentives to run nodes, which means node operators are more likely to create centralized choke points.
Waku currently lacks incentives to run nodes,
which means node operators are more likely to create centralized choke points.
### Privacy
**Light node privacy:**
The main privacy concern with light nodes is that directly connected peers will know that a message originates from them (as it are the only ones it sends). This means nodes can make assumptions about what messages (topics) their peers are interested in.
The main privacy concern with light nodes
is that directly connected peers will know that a message originates from them
(as it are the only ones it sends).
This means nodes can make assumptions about what messages (topics)
their peers are interested in.
**Bloom filter privacy:**
By having a bloom filter where only the topics you are interested in are set, you reveal which messages you are interested in. This is a fundamental tradeoff between bandwidth usage and privacy, though the tradeoff space is likely suboptimal in terms of the [Anonymity](https://eprint.iacr.org/2017/954.pdf) [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
By having a bloom filter where only the topics you are interested in are set,
you reveal which messages you are interested in.
This is a fundamental tradeoff between bandwidth usage and privacy,
though the tradeoff space is likely suboptimal in terms of the
[Anonymity](https://eprint.iacr.org/2017/954.pdf) [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
**Privacy guarantees not rigorous:**
Privacy for Whisper / Waku haven't been studied rigorously for various threat models like global passive adversary, local active attacker, etc. This is unlike e.g. Tor and mixnets.
Privacy for Whisper / Waku haven't been studied rigorously for various threat models
like global passive adversary, local active attacker, etc.
This is unlike e.g. Tor and mixnets.
**Topic hygiene:**
Similar to bloom filter privacy, if you use a very specific topic you reveal more information. See scalability model linked above.
Similar to bloom filter privacy,
if you use a very specific topic you reveal more information.
See scalability model linked above.
### Spam resistance
**PoW bad for heterogeneous devices:**
Proof of work is a poor spam prevention mechanism. A mobile device can only have a very low PoW in order not to use too much CPU / burn up its phone battery. This means someone can spin up a powerful node and overwhelm the network.
Proof of work is a poor spam prevention mechanism.
A mobile device can only have a very low PoW
in order not to use too much CPU / burn up its phone battery.
This means someone can spin up a powerful node and overwhelm the network.
### Censorship resistance
**Devp2p TCP port blockable:**
By default Devp2p runs on port `30303`, which is not commonly used for any other service. This means it is easy to censor, e.g. airport WiFi. This can be mitigated somewhat by running on e.g. port `80` or `443`, but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
By default Devp2p runs on port `30303`,
which is not commonly used for any other service.
This means it is easy to censor, e.g. airport WiFi.
This can be mitigated somewhat by running on e.g. port `80` or `443`,
but there are still outstanding issues.
See libp2p and Tor's Pluggable Transport for how this can be improved.
## Appendix B: Implementation Notes
@@ -461,17 +661,22 @@ By default Devp2p runs on port `30303`, which is not commonly used for any other
Notes useful for implementing Waku mode.
1. Avoid duplicate envelopes
- Avoid duplicate envelopes:
To avoid duplicate envelopes, only connect to one Waku node. Benign duplicate envelopes is an intrinsic property of Whisper which often leads to a N factor increase in traffic, where N is the number of peers you are connected to.
To avoid duplicate envelopes, only connect to one Waku node.
Benign duplicate envelopes is an intrinsic property of Whisper
which often leads to a N factor increase in traffic,
where N is the number of peers you are connected to.
2. Topic specific recommendations
- Topic specific recommendations -
Consider partition topics based on some usage, to avoid too much traffic on a single topic.
Consider partition topics based on some usage,
to avoid too much traffic on a single topic.
### Node discovery
Resource restricted devices SHOULD use [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459) to discover nodes.
Resource restricted devices SHOULD use
[EIP-1459](https://eips.ethereum.org/EIPS/eip-1459) to discover nodes.
Known static nodes MAY also be used.
@@ -487,16 +692,21 @@ Released [April 21,2020](https://github.com/vacp2p/specs/commit/9e650995f2417984
Released [March 17,2020](https://github.com/vacp2p/specs/commit/7b9dc562bc50c6bb844ac575cb221ec9cda2530a)
- Clarify the preferred way of handling unknown keys in the `status-options` association list.
- Correct spec/implementation mismatch: Change RLP keys to be the their int values in order to reflect production behavior
- Clarify the preferred way of handling unknown keys
in the `status-options` association list.
- Correct spec/implementation mismatch:
Change RLP keys to be the their int values in order to reflect production behavior
### Version 0.4
Released [February 21, 2020](https://github.com/vacp2p/specs/commit/17bd066e317bbe33af07146b721d73f24de47e88).
- Simplify implementation matrix with latest state
- Introduces a new required packet code Status Code (`0x22`) for communicating option changes
- Deprecates the following packet codes: PoW Requirement (`0x02`), Bloom Filter (`0x03`), Rate limits (`0x20`), Topic interest (`0x21`) - all superseded by the new Status Code (`0x22`)
- Introduces a new required packet code Status Code (`0x22`)
for communicating option changes
- Deprecates the following packet codes:
PoW Requirement (`0x02`), Bloom Filter (`0x03`), Rate limits (`0x20`),
Topic interest (`0x21`) - all superseded by the new Status Code (`0x22`)
- Increased `topic-interest` capacity from 1000 to 10000
### Version 0.3
@@ -506,7 +716,8 @@ Released [February 13, 2020](https://github.com/vacp2p/specs/commit/73138d6ba954
- Recommend DNS based node discovery over other Discovery methods.
- Mark spec as Draft mode in terms of its lifecycle.
- Simplify Changelog and misc formatting.
- Handshake/Status message not compatible with shh/6 nodes; specifying options as association list.
- Handshake/Status message not compatible with shh/6 nodes;
specifying options as association list.
- Include topic-interest in Status handshake.
- Upgradability policy.
- `topic-interest` packet code.
@@ -522,7 +733,8 @@ Released [December 10, 2019](https://github.com/vacp2p/specs/blob/waku-0.2.0/wak
- More details on handshake modifications.
- Accounting for resources mode (experimental)
- Appendix with security considerations: scalability and UX, privacy, and spam resistance.
- Appendix with implementation notes and implementation matrix across various clients with breakdown per capability.
- Appendix with implementation notes and
implementation matrix across various clients with breakdown per capability.
- More details on handshake and parameters.
- Describe rate limits in more detail.
- More details on mailserver and mail client API.
@@ -549,7 +761,6 @@ confirmations-enabled and rate-limits
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## Footnotes
[^1]: Felix Lange et al. [The RLPx Transport Protocol](https://github.com/ethereum/devp2p/blob/master/rlpx.md). Ethereum.

View File

@@ -1,6 +1,7 @@
# Deprecated RFCs
# Deprecated RFCs
Deprecated specifications are no longer used in Waku products.
This subdirectory is for achrive purpose and
This subdirectory is for achrive purpose and
should not be used in production ready implementations.
Visit [Waku RFCs](https://github.com/waku-org/specs) for new Waku specifications under discussion.
Visit [Waku RFCs](https://github.com/waku-org/specs)
for new Waku specifications under discussion.

View File

@@ -19,12 +19,14 @@ This protocol is mainly used to:
Currently, all main Waku v2 implementations support the toy chat protocol:
[nim-waku](https://github.com/status-im/nim-waku/blob/master/examples/v2/chat2.nim),
js-waku ([NodeJS](https://github.com/status-im/js-waku/tree/main/examples/cli-chat) and [web](https://github.com/status-im/js-waku/tree/main/examples/web-chat))
js-waku ([NodeJS](https://github.com/status-im/js-waku/tree/main/examples/cli-chat)
and [web](https://github.com/status-im/js-waku/tree/main/examples/web-chat))
and [go-waku](https://github.com/status-im/go-waku/tree/master/examples/chat2).
Note that this is completely separate from the protocol the Status app is using for its chat functionality.
Note that this is completely separate from the protocol the Status app
is using for its chat functionality.
# Design
## Design
The chat protocol enables sending and receiving messages in a chat room.
There is currently only one chat room, which is tied to the content topic.
@@ -32,7 +34,7 @@ The messages SHOULD NOT be encrypted.
The `contentTopic` MUST be set to `/toy-chat/2/huilong/proto`.
# Payloads
## Payloads
```protobuf
syntax = "proto3";
@@ -48,6 +50,6 @@ message Chat2Message {
- `nick`: The nickname of the user sending the message,
- `payload`: The text of the messages, UTF-8 encoded.
# Copyright
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

View File

@@ -8,20 +8,21 @@ editor: Oskar Thoren <oskarth@titanproxy.com>
contributors:
- Hanno Cornelius <hanno@status.im>
- Daniel Kaiser <danielkaiser@status.im>
- Filip Dimitrijevic <filip@status.im>
---
This document outlines recommended usage of topic names in Waku v2.
In [10/WAKU2 spec](../../standards/core/10/waku2.md) there are two types of topics:
In [10/WAKU2 spec](/waku/standards/core/10/waku2.md) there are two types of topics:
- pubsub topics, used for routing
- Pubsub topics, used for routing
- Content topics, used for content-based filtering
## Pubsub Topics
Pubsub topics are used for routing of messages (see [11/WAKU2-RELAY](../../standards/core/11/relay.md)),
and can be named implicitly by Waku sharding (see [RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md)).
This document comprises recommendations for explicitly naming pubsub topics (e.g. when choosing *named sharding* as specified in [RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md)).
Pubsub topics are used for routing of messages (see [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)),
and can be named implicitly by Waku sharding (see [RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)).
This document comprises recommendations for explicitly naming pubsub topics
(e.g. when choosing *named sharding* as specified in [RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)).
### Pubsub Topic Format
@@ -29,26 +30,32 @@ Pubsub topics SHOULD follow the following structure:
`/waku/2/{topic-name}`
This namespaced structure makes compatibility, discoverability, and automatic handling of new topics easier.
This namespaced structure makes compatibility, discoverability,
and automatic handling of new topics easier.
The first two parts indicate
The first two parts indicate:
1) it relates to the Waku protocol domain, and
2) the version is 2.
If applicable, it is RECOMMENDED to structure `{topic-name}` in a hierarchical way as well.
If applicable, it is RECOMMENDED to structure `{topic-name}`
in a hierarchical way as well.
> *Note*: In previous versions of this document, the structure was `/waku/2/{topic-name}/{encoding}`.
The now deprecated `/{encoding}` was always set to `/proto`,
which indicated that the [data field](../../standards/core/11/RELAY.md/#protobuf-definition) in pubsub is serialized/encoded as protobuf.
which indicated that the [data field](/waku/standards/core/11/relay.md#protobuf-definition)
in pubsub is serialized/encoded as protobuf.
The inspiration for this format was taken from
[Ethereum 2 P2P spec](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#topics-and-messages).
However, because the payload of messages transmitted over [11/WAKU2-RELAY](../../standards/core/11/relay.md) must be a [14/WAKU2-MESSAGE](../../standards/core/14/message.md),
However, because the payload of messages transmitted over [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
must be a [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md),
which specifies the wire format as protobuf,`/proto` is the only valid encoding.
This makes the `/proto` indication obsolete.
The encoding of the `payload` field of a Waku Message is indicated by the `/{encoding}` part of the content topic name.
The encoding of the `payload` field of a WakuMessage
is indicated by the `/{encoding}` part of the content topic name.
Specifying an encoding is only significant for the actual payload/data field.
Waku preserves this option by allowing to specify an encoding for the WakuMessage payload field as part of the content topic name.
Waku preserves this option by allowing to specify an encoding
for the WakuMessage payload field as part of the content topic name.
### Default PubSub Topic
@@ -56,25 +63,27 @@ The Waku v2 default pubsub topic is:
`/waku/2/default-waku/proto`
The `{topic name}` part is `default-waku/proto`, which indicates it is default topic for exchanging WakuMessages;
The `{topic name}` part is `default-waku/proto`,
which indicates it is default topic for exchanging WakuMessages;
`/proto` remains for backwards compatibility.
### Application Specific Names
Larger apps can segregate their pubsub meshes using topics named like:
```
```text
/waku/2/status/
/waku/2/walletconnect/
```
This indicates that these networks carry WakuMessages, but for different domains completely.
This indicates that these networks carry WakuMessages,
but for different domains completely.
### Named Topic Sharding Example
The following is an example of named sharding, as specified in [RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md).
The following is an example of named sharding, as specified in [RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md).
```
```text
waku/2/waku-9_shard-0/
...
waku/2/waku-9_shard-9/
@@ -86,16 +95,19 @@ This indicates explicitly that the network traffic has been partitioned into 10
The other type of topic that exists in Waku v2 is a content topic.
This is used for content based filtering.
See [14/WAKU2-MESSAGE spec](../../standards/core/14/message.md) for where this is specified.
See [14/WAKU2-MESSAGE spec](/waku/standards/core/14/message.md)
for where this is specified.
Note that this doesn't impact routing of messages between relaying nodes,
but it does impact how request/reply protocols such as
[12/WAKU2-FILTER](../../standards/core/12/filter.md) and [13/WAKU2-STORE](../../standards/core/13/store.md) are used.
but it does impact using request/reply protocols such as
[12/WAKU2-FILTER](/waku/standards/core/12/filter.md) and
[13/WAKU2-STORE](/waku/standards/core/13/store.md).
This is especially useful for nodes that have limited bandwidth,
and only want to pull down messages that match this given content topic.
Since all messages are relayed using the relay protocol regardless of content topic,
you MAY use any content topic you wish without impacting how messages are relayed.
you MAY use any content topic you wish
without impacting how messages are relayed.
### Content Topic Format
@@ -110,20 +122,44 @@ As an example, here's the content topic used for an upcoming testnet:
### Content Topic Naming Recommendations
Application names should be unique to avoid conflicting issues with other protocols.
Applications should specify their version (if applicable) in the version field.
Application names SHOULD be unique to avoid conflicting issues with other protocols.
Application version (if applicable) SHOULD be specified in the version field.
The `{content-topic-name}` portion of the content topic is up to the application,
and depends on the problem domain.
It can be hierarchical, for instance to separate content, or to indicate different bandwidth and privacy guarantees.
The encoding field indicates the serialization/encoding scheme for the [WakuMessage payload](../../standards/core/14/message.md/#payloads) field.
It can be hierarchical, for instance to separate content, or
to indicate different bandwidth and privacy guarantees.
The encoding field indicates the serialization/encoding scheme
for the [WakuMessage payload](/waku/standards/core/14/message.md#payloads) field.
### Content Topic usage guidelines
Applications SHOULD be mindful while designing/using content topics
so that a bloat of content-topics does not happen.
A content-topic bloat causes performance degradation in Store
and Filter protocols while trying to retrieve messages.
Store queries have been noticed to be considerably slow
(e.g doubling of response-time when content-topic count is increased from 10 to 100)
when a lot of content-topics are involved in a single query.
Similarly, a number of filter subscriptions increase,
which increases complexity on client side to maintain
and manage these subscriptions.
Applications SHOULD analyze the query/filter criteria for fetching messages from the network
and select/design content topics to match such filter criteria.
e.g: even though applications may want to segregate messages into different sets
based on some application logic,
if those sets of messages are always fetched/queried together from the network,
then all those messages SHOULD use a single content-topic.
## Differences with Waku v1
In [5/WAKU1](../../deprecated/5/waku0.md) there is no actual routing.
In [5/WAKU1](/waku/deprecated/5/waku0.md) there is no actual routing.
All messages are sent to all other nodes.
This means that we are implicitly using the same pubsub topic that would be something like:
This means that we are implicitly using the same pubsub topic
that would be something like:
```
```text
/waku/1/default-waku/rlp
```
@@ -131,23 +167,25 @@ Topics in Waku v1 correspond to Content Topics in Waku v2.
### Bridging Waku v1 and Waku v2
To bridge Waku v1 and Waku v2 we have a [15/WAKU-BRIDGE](../../standards/core/15/bridge.md).
To bridge Waku v1 and Waku v2 we have a [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md).
For mapping Waku v1 topics to Waku v2 content topics,
the following structure for the content topic SHOULD be used:
```
```text
/waku/1/<4bytes-waku-v1-topic>/rfc26
```
The `<4bytes-waku-v1-topic>` SHOULD be the lowercase hex representation of the 4-byte Waku v1 topic.
The `<4bytes-waku-v1-topic>` SHOULD be the lowercase hex representation
of the 4-byte Waku v1 topic.
A `0x` prefix SHOULD be used.
`/rfc26` indicates that the bridged content is encoded according to RFC [26/WAKU2-PAYLOAD](../../standards/application/26/payload.md).
See [15/WAKU-BRIDGE](../../standards/core/15/bridge.md) for a description of the bridged fields.
`/rfc26` indicates that the bridged content is encoded according to RFC [26/WAKU2-PAYLOAD](/waku/standards/application/26/payload.md).
See [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md)
for a description of the bridged fields.
This creates a direct mapping between the two protocols.
For example:
```
```text
/waku/1/0x007f80ff/rfc26
```
@@ -158,13 +196,13 @@ Copyright and related rights waived via
## References
* [10/WAKU2 spec](../../standards/core/10/waku2.md)
* [11/WAKU2-RELAY](../../standards/core/11/relay.md)
* [RELAY-SHARDING](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/relay-sharding.md)
* [Ethereum 2 P2P spec](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#topics-and-messages)
* [14/WAKU2-MESSAGE](../../standards/core/14/message.md)
* [12/WAKU2-FILTER](../../standards/core/12/filter.md)
* [13/WAKU2-STORE](../../standards/core/13/store.md)
* [6/WAKU1](../../deprecated/5/waku0.md)
* [15/WAKU-BRIDGE](../../standards/core/15/bridge.md)
* [26/WAKU-PAYLOAD](../../standards/application/26/payload.md)
- [10/WAKU2 spec](/waku/standards/core/10/waku2.md)
- [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
- [RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)
- [Ethereum 2 P2P spec](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#topics-and-messages)
- [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)
- [12/WAKU2-FILTER](/waku/standards/core/12/filter.md)
- [13/WAKU2-STORE](/waku/standards/core/13/store.md)
- [6/WAKU1](/waku/deprecated/5/waku0.md)
- [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md)
- [26/WAKU-PAYLOAD](/waku/standards/application/26/payload.md)

View File

@@ -5,26 +5,35 @@ name: Waku v2 Client Peer Management Recommendations
status: draft
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Filip Dimitrijevic <filip@status.im>
---
`27/WAKU2-PEERS` describes a recommended minimal set of peer storage and peer management features to be implemented by Waku v2 clients.
`27/WAKU2-PEERS` describes a recommended minimal set of peer storage and
peer management features to be implemented by Waku v2 clients.
In this context, peer _storage_ refers to a client's ability to keep track of discovered or statically-configured peers and their metadata.
In this context, peer _storage_ refers to a client's ability to keep track of discovered
or statically-configured peers and their metadata.
It also deals with matters of peer _persistence_,
or the ability to store peer data on disk to resume state after a client restart.
Peer _management_ is a closely related concept and refers to the set of actions a client MAY choose to perform based on its knowledge of its connected peers,
e.g. triggering reconnects/disconnects, keeping certain connections alive, etc.
Peer _management_ is a closely related concept and
refers to the set of actions a client MAY choose to perform
based on its knowledge of its connected peers,
e.g. triggering reconnects/disconnects,
keeping certain connections alive, etc.
## Peer store
The peer store SHOULD be an in-memory data structure where information about discovered or configured peers are stored.
It SHOULD be considered the main source of truth for peer-related information in a Waku v2 client.
The peer store SHOULD be an in-memory data structure
where information about discovered or configured peers are stored.
It SHOULD be considered the main source of truth
for peer-related information in a Waku v2 client.
Clients MAY choose to persist this store on-disk.
### Tracked peer metadata
It is RECOMMENDED that a Waku v2 client tracks at least the following information about each of its peers in a peer store:
It is RECOMMENDED that a Waku v2 client tracks at least the following information
about each of its peers in a peer store:
| Metadata | Description |
| --- | --- |
@@ -36,16 +45,19 @@ It is RECOMMENDED that a Waku v2 client tracks at least the following informatio
### Peer connectivity
A Waku v2 client SHOULD track _at least_ the following connectivity states for each of its peers:
- **`NotConnected`**: The peer has been discovered or configured on this client,
A Waku v2 client SHOULD track _at least_ the following connectivity states
for each of its peers:
- **`NotConnected`**: The peer has been discovered or configured on this client,
but no attempt has yet been made to connect to this peer.
This is the default state for a new peer.
- **`CannotConnect`**: The client attempted to connect to this peer, but failed.
- **`CanConnect`**: The client was recently connected to this peer and disconnected gracefully.
- **`Connected`**: The client is actively connected to this peer.
- **`CannotConnect`**: The client attempted to connect to this peer, but failed.
- **`CanConnect`**: The client was recently connected to this peer and
disconnected gracefully.
- **`Connected`**: The client is actively connected to this peer.
This list does not preclude clients from tracking more advanced connectivity metadata,
such as a peer's blacklist status (see [`18/WAKU2-SWAP`](../../standards/application/18/swap.md)).
such as a peer's blacklist status (see [`18/WAKU2-SWAP`](/waku/deprecated/18/swap.md)).
### Persistence
@@ -55,30 +67,40 @@ Peer persistence MAY be used to resume peer connections after a client restart.
## Peer management
Waku v2 clients will have different requirements when it comes to managing the peers tracked in the [**peer store**](#peer-store).
Waku v2 clients will have different requirements
when it comes to managing the peers tracked in the [**peer store**](#peer-store).
It is RECOMMENDED that clients support:
- [automatic reconnection](#reconnecting-peers) to peers under certain conditions
- [connection keep-alive](#connection-keep-alive)
### Reconnecting peers
A Waku v2 client MAY choose to reconnect to previously connected, managed peers under certain conditions.
A Waku v2 client MAY choose to reconnect to previously connected,
managed peers under certain conditions.
Such conditions include, but are not limited to:
- Reconnecting to all `relay`-capable peers after a client restart. This will require [persistent peer storage](#persistence).
- Reconnecting to all `relay`-capable peers after a client restart.
This will require [persistent peer storage](#persistence).
If a client chooses to automatically reconnect to previous peers,
it MUST respect the [backing off period](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange) specified for GossipSub v1.1 before attempting to reconnect.
it MUST respect the
[backing off period](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange)
specified for GossipSub v1.1 before attempting to reconnect.
This requires keeping track of the [last time each peer was disconnected](#tracked-peer-metadata).
### Connection keep-alive
A Waku v2 client MAY choose to implement a keep-alive mechanism to certain peers.
If a client chooses to implement keep-alive on a connection,
it SHOULD do so by sending periodic [libp2p pings](https://docs.libp2p.io/concepts/protocols/#ping) as per `10/WAKU2` [client recommendations](../../standards/core/10/waku2.md/#recommendations-for-clients).
The recommended period between pings SHOULD be _at most_ 50% of the shortest idle connection timeout for the specific client and transport.
it SHOULD do so by sending periodic [libp2p pings](https://docs.libp2p.io/concepts/fundamentals/protocols/#ping)
as per `10/WAKU2` [client recommendations](/waku/standards/core/10/waku2.md#recommendations-for-clients).
The recommended period between pings SHOULD be _at most_ 50%
of the shortest idle connection timeout for the specific client and transport.
For example, idle TCP connections often times out after 10 to 15 minutes.
> **Implementation note:** the `nim-waku` client currently implements a keep-alive mechanism every `5 minutes`,
> **Implementation note:**
the `nim-waku` client currently implements a keep-alive mechanism every `5 minutes`,
in response to a TCP connection timeout of `10 minutes`.
## Copyright
@@ -91,9 +113,9 @@ Copyright and related rights waived via
- [`Peer ID`](https://docs.libp2p.io/concepts/peer-id/)
- [`multiaddrs`](https://docs.libp2p.io/concepts/addressing/)
- [`protocol IDs`](https://docs.libp2p.io/concepts/protocols/#protocol-ids)
- [`11/WAKU2-RELAY`](../../standards/core/11/relay.md)
- [`13/WAKU2-STORE`](../../standards/core/13/store.md)
- [`18/WAKU2-SWAP`](../../standards/application/18/swap.md)
- [backing off period](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange)
- [libp2p pings](https://docs.libp2p.io/concepts/protocols/#ping)
- [`10/WAKU2` client recommendations](../../standards/core/10/waku2.md/#recommendations-for-clients)
- [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md)
- [`13/WAKU2-STORE`](/waku/standards/core/13/store.md)
- [`18/WAKU2-SWAP`](/waku/deprecated/18/swap.md)
- [backing off period](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md/#prune-backoff-and-peer-exchange)
- [libp2p pings](https://docs.libp2p.io/concepts/fundamentals/protocols/#ping)
- [`10/WAKU2` client recommendations](/waku/standards/core/10/waku2.md#recommendations-for-clients)

View File

@@ -5,13 +5,15 @@ name: Waku v2 Client Parameter Configuration Recommendations
status: draft
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Filip Dimitrijevic <filip@status.im>
---
`29/WAKU2-CONFIG` describes the RECOMMENDED values to assign to configurable parameters for Waku v2 clients.
`29/WAKU2-CONFIG` describes the RECOMMENDED values
to assign to configurable parameters for Waku v2 clients.
Since Waku v2 is built on [libp2p](https://github.com/libp2p/specs),
most of the parameters and reasonable defaults are derived from there.
Waku v2 relay messaging is specified in [`11/WAKU2-RELAY`](../../standards/core/11/relay.md),
Waku v2 relay messaging is specified in [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md),
a minor extension of the [libp2p GossipSub protocol](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md).
GossipSub behaviour is controlled by a series of adjustable parameters.
Waku v2 clients SHOULD configure these parameters to the recommended values below.
@@ -25,7 +27,7 @@ We repeat them here with RECOMMMENDED values for `11/WAKU2-RELAY` implementation
|----------------------|-------------------------------------------------------|-------------------|
| `D` | The desired outbound degree of the network | 6 |
| `D_low` | Lower bound for outbound degree | 4 |
| `D_high` | Upper bound for outbound degree | 8 |
| `D_high` | Upper bound for outbound degree | 8 |
| `D_lazy` | (Optional) the outbound degree for gossip emission | `D` |
| `heartbeat_interval` | Time between heartbeats | 1 second |
| `fanout_ttl` | Time-to-live for each topic's fanout state | 60 seconds |
@@ -35,8 +37,10 @@ We repeat them here with RECOMMMENDED values for `11/WAKU2-RELAY` implementation
## GossipSub v1.1 parameters
GossipSub v1.1 extended GossipSub v1.0 and introduced [several new parameters](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#overview-of-new-parameters).
We repeat the global parameters here with RECOMMMENDED values for `11/WAKU2-RELAY` implementations.
GossipSub v1.1 extended GossipSub v1.0 and
introduced [several new parameters](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#overview-of-new-parameters).
We repeat the global parameters here
with RECOMMMENDED values for `11/WAKU2-RELAY` implementations.
| Parameter | Description | RECOMMENDED value |
|----------------|------------------------------------------------------------------------|-------------------|
@@ -46,12 +50,15 @@ We repeat the global parameters here with RECOMMMENDED values for `11/WAKU2-RELA
| `D_score` | Number of peers to retain by score when pruning from oversubscription | `D_low` |
| `D_out` | Number of outbound connections to keep in the mesh. | `D_low` - 1 |
`11/WAKU2-RELAY` clients SHOULD implement a peer scoring mechanism with the parameter constraints as [specified by libp2p](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#overview-of-new-parameters).
`11/WAKU2-RELAY` clients SHOULD implement a peer scoring mechanism
with the parameter constraints as
[specified by libp2p](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#overview-of-new-parameters).
## Other configuration
The following behavioural parameters are not specified by `libp2p`,
but nevertheless describes constraints that `11/WAKU2-RELAY` clients MAY choose to implement.
but nevertheless describes constraints that `11/WAKU2-RELAY` clients
MAY choose to implement.
| Parameter | Description | RECOMMENDED value |
|--------------------|---------------------------------------------------------------------------|-------------------|
@@ -68,7 +75,7 @@ Copyright and related rights waived via
## References
- [libp2p](https://github.com/libp2p/specs)
- [11/WAKU2-RELAY](../../standards/core/11/relay.md)
- [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
- [libp2p GossipSub protocol](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md)
- [corresponding libp2p specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md#parameters)
- [several new parameters](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#overview-of-new-parameters)

View File

@@ -5,13 +5,16 @@ name: Adaptive nodes
status: draft
editor: Oskar Thorén <oskarth@titanproxy.com>
contributors:
- Filip Dimitrijevic <filip@status.im>
---
This is an informational spec that show cases the concept of adaptive nodes.
## Node types - a continuum
We can look at node types as a continuum, from more restricted to less restricted, fewer resources to more resources.
We can look at node types as a continuum,
from more restricted to less restricted,
fewer resources to more resources.
![Node types - a continuum](./images/adaptive_node_continuum2.png)
@@ -29,11 +32,13 @@ Some examples:
- Desktop: download, leave in background, contribute somewhat
- Cluster: expensive, upkeep, but can contribute a lot
These are also illustrative, so a node in a browser in certain environment might contribute similarly to Desktop.
These are also illustrative,
so a node in a browser in certain environment might contribute similarly to Desktop.
### Adaptive nodes
We call these nodes *adaptive nodes* to highlights different modes of contributing, such as:
We call these nodes *adaptive nodes* to highlights different modes of contributing,
such as:
- Only leeching from the network
- Relaying messages for one or more topics
@@ -55,11 +60,18 @@ Each node can choose which protocols to support, depending on its resources and
![Protocol selection](./images/adaptive_node_protocol_selection2.png)
In the case of protocols like [11/WAKU2-RELAY](../../standards/core/11/relay.md) etc (12, 13, 19, 21) these correspond to Libp2p protocols.
Protocols like [11/WAKU2-RELAY](/waku/standards/core/11/relay.md),
as well as [12], [13], [19], and [21], correspond to libp2p protocols.
However, other protocols like 16/WAKU2-RPC (local HTTP JSON-RPC), 25/LIBP2P-DNS-DISCOVERY, Discovery v5 (DevP2P) or interfacing with distributed storage, are running on different network stacks.
However, other protocols like 16/WAKU2-RPC
(local HTTP JSON-RPC), 25/LIBP2P-DNS-DISCOVERY,
Discovery v5 (DevP2P) or interfacing with distributed storage,
are running on different network stacks.
This is in addition to protocols that specify payloads, such as 14/WAKU2-MESSAGE, 26/WAKU2-PAYLOAD, or application specific ones. As well as specs that act more as recommendations, such as 23/WAKU2-TOPICS or 27/WAKU2-PEERS.
This is in addition to protocols that specify payloads, such as 14/WAKU2-MESSAGE,
26/WAKU2-PAYLOAD, or application specific ones.
As well as specs that act more as recommendations,
such as 23/WAKU2-TOPICS or 27/WAKU2-PEERS.
## Waku network visualization
@@ -67,36 +79,47 @@ We can better visualize the network with some illustrative examples.
### Topology and topics
The first one shows an example topology with different PubSub topics for the relay protocol.
This illustration shows an example topology with different PubSub topics
for the relay protocol.
![Waku Network visualization](./images/adaptive_node_network_topology_protocols2.png)
### Legend
This illustration shows an example of content topics a node is interested in.
![Waku Network visualization legend](./images/adaptive_node_network_topology_protocols_legend.png)
The dotted box shows what content topics (application-specific) a node is interested in.
The dotted box shows what content topics (application-specific)
a node is interested in.
A node that is purely providing a service to the network might not care.
In this example, we see support for toy chat, a topic in Waku v1 (Status chat), WalletConnect, and SuperRare community.
In this example, we see support for toy chat,
a topic in Waku v1 (Status chat), WalletConnect, and SuperRare community.
### Auxiliary network
This is a separate component with its own topology.
Behavior and interaction with other protocols specified in Vac RFCs, e.g. 25/LIBP2P-DNS-DISCOVERY, 15/WAKU-BRIDGE, etc.
Behavior and interaction with other protocols specified in Vac RFCs,
e.g. [25/LIBP2P-DNS-DISCOVERY](/vac/25/libp2p-dns-discovery.md)
and [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md).
### Node Cross Section
This one shows a cross-section of nodes in different dimensions and shows how the connections look different for different protocols.
This one shows a cross-section of nodes in different dimensions and
shows how the connections look different for different protocols.
![Node Cross Section](./images/adaptive_node_cross_section2.png)
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [11/WAKU2-RELAY](../../standards/core/11/relay.md)
- [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
- [25/LIBP2P-DNS-DISCOVERY](/vac/25/libp2p-dns-discovery.md)
- [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md)

View File

@@ -10,25 +10,38 @@ contributors:
**Content Topics**:
- Public Key Broadcast: `/eth-pm/1/public-key/proto`,
- Private Message: `/eth-pm/1/private-message/proto`.
- Public Key Broadcast: `/eth-pm/1/public-key/proto`
- Private Message: `/eth-pm/1/private-message/proto`
## Abstract
This specification explains the Toy Ethereum Private Message protocol
which enables a peer to send an encrypted message to another peer
using the Waku v2 network, and the peer's Ethereum address.
The main purpose of this specification is to demonstrate how Waku v2 can be used for encrypted messaging purposes,
using Ethereum accounts for identity.
This protocol caters for Web3 wallets restrictions, allowing it to be implemented only using standard Web3 API.
In the current state, the protocol has privacy and features [limitations](#limitations), has not been audited
and hence is not fit for production usage.
We hope this can be an inspiration for developers wishing to build on top of Waku v2.
over the Waku network using the peer's Ethereum address.
## Goal
Alice wants to send an encrypted message to Bob, where only Bob can decrypt the message.
Alice wants to send an encrypted message to Bob,
where only Bob can decrypt the message.
Alice only knows Bob's Ethereum Address.
The goal of this specification
is to demonstrate how Waku can be used for encrypted messaging purposes,
using Ethereum accounts for identity.
This protocol caters to Web3 wallet restrictions,
allowing it to be implemented using standard Web3 API.
In the current state,
Toy Ethereum Private Message, ETH-PM, has privacy and features [limitations](#limitations),
has not been audited and hence is not fit for production usage.
We hope this can be an inspiration for developers
wishing to build on top of Waku.
## Design Requirements
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
“SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
## Variables
Here are the variables used in the protocol and their definition:
@@ -38,18 +51,18 @@ Here are the variables used in the protocol and their definition:
- `B'` is Bob's Encryption Public Key, for which `b'` is the private key.
- `M` is the private message that Alice sends to Bob.
## Design Requirements
The proposed protocol MUST adhere to the following design requirements:
1. Alice knows Bob's Ethereum address,
2. Bob is willing to participate to Eth-PM, and publishes `B'`,
3. Bob's ownership of `B'` MUST be verifiable,
4. Alice wants to send message `M` to Bob,
5. Bob SHOULD be able to get `M` using [10/WAKU2 spec](../../core/10/waku2.md),
6. Participants only have access to their Ethereum Wallet via the Web3 API,
7. Carole MUST NOT be able to read `M`'s content even if she is storing it or relaying it,
8. [Waku Message Version 1](../26/payload.md) Asymmetric Encryption is used for encryption purposes.
1. Alice knows Bob's Ethereum address
2. Bob is willing to participate to Eth-PM, and publishes `B'`
3. Bob's ownership of `B'` MUST be verifiable
4. Alice wants to send message `M` to Bob
5. Bob SHOULD be able to get `M` using [10/WAKU2](waku/standards/core/10/waku2.md)
6. Participants only have access to their Ethereum Wallet via the Web3 API
7. Carole MUST NOT be able to read `M`'s content,
even if she is storing it or relaying it
8. [Waku Message Version 1](waku/standards/application/26/payload.md) Asymmetric Encryption
is used for encryption purposes.
## Limitations
@@ -58,18 +71,22 @@ meaning that there is no programmatic way for Bob to reply to Alice
or verify her identity.
Private messages are sent on the same content topic for all users.
As the recipient data is encrypted, all participants must decrypt all messages which can lead to scalability issues.
As the recipient data is encrypted,
all participants must decrypt all messages which can lead to scalability issues.
This protocol does not guarantee Perfect Forward Secrecy nor Future Secrecy:
If Bob's private key is compromised, past and future messages could be decrypted.
A solution combining regular [X3DH](https://www.signal.org/docs/specifications/x3dh/)
bundle broadcast with [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/) encryption would remove these limitations;
See the [Status secure transport spec](https://specs.status.im/spec/5) for an example of a protocol that achieves this in a peer-to-peer setting.
bundle broadcast with [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/)
encryption would remove these limitations;
See the [Status secure transport specification](status/deprecated/secure-transport.md)
for an example of a protocol that achieves this in a peer-to-peer setting.
Bob MUST decide to participate in the protocol before Alice can send him a message.
This is discussed in more in details in [Consideration for a non-interactive/uncoordinated protocol](#consideration-for-a-non-interactiveuncoordinated-protocol)
This is discussed in more detail in
[Consideration for a non-interactive/uncoordinated protocol](#consideration-for-a-non-interactiveuncoordinated-protocol)
## The protocol
## The Protocol
### Generate Encryption KeyPair
@@ -82,22 +99,25 @@ Then Bob can compute the corresponding SECP-256k1 Public Key, `B'`.
For Alice to encrypt messages for Bob,
Bob SHOULD broadcast his Encryption Public Key `B'`.
To prove that the Encryption Public Key `B'` is indeed owned by the owner of Bob's Ethereum Account `B`,
To prove that the Encryption Public Key `B'`
is indeed owned by the owner of Bob's Ethereum Account `B`,
Bob MUST sign `B'` using `B`.
### Sign Encryption Public Key
To prove ownership of the Encryption Public Key,
Bob must sign it using [EIP-712](https://eips.ethereum.org/EIPS/eip-712) v3,
meaning calling `eth_signTypedData_v3` on his Wallet's API.
meaning calling `eth_signTypedData_v3` on his wallet's API.
Note: While v4 also exists,
it is not available on all wallets and the features brought by v4 is not needed for the current use case.
Note: While v4 also exists, it is not available on all wallets and
the features brought by v4 is not needed for the current use case.
The `TypedData` to be passed to `eth_signTypedData_v3` MUST be as follows, where:
- `encryptionPublicKey` is Bob's Encryption Public Key, `B'`, in hex format, **without** `0x` prefix.
- `bobAddress` is Bob's Ethereum address, corresponding to `B`, in hex format, **with** `0x` prefix.
- `encryptionPublicKey` is Bob's Encryption Public Key, `B'`,
in hex format, **without** `0x` prefix.
- `bobAddress` is Bob's Ethereum address, corresponding to `B`,
in hex format, **with** `0x` prefix.
```js
const typedData = {
@@ -143,8 +163,9 @@ message PublicKeyMessage {
}
```
This MUST be wrapped in a Waku Message version 0, with the Public Key Broadcast content topic.
Finally, Bob SHOULD publish the message on Waku v2.
This MUST be wrapped in a [14/WAKU-MESSAGE](/waku/standards/core/14/message.md) version 0,
with the Public Key Broadcast content topic.
Finally, Bob SHOULD publish the message on Waku.
## Consideration for a non-interactive/uncoordinated protocol
@@ -155,71 +176,93 @@ it is not enough in itself to deduce Bob's Public Key.
This is why the protocol dictates that Bob MUST send his Public Key first,
and Alice MUST receive it before she can send him a message.
Moreover, nim-waku, the reference implementation of [13/WAKU2-STORE](../../core/13/store.md),
Moreover, nwaku, the reference implementation of [13/WAKU2-STORE](/waku/standards/core/13/store.md),
stores messages for a maximum period of 30 days.
This means that Bob would need to broadcast his public key at least every 30 days to be reachable.
This means that Bob would need to broadcast his public key
at least every 30 days to be reachable.
Below we are reviewing possible solutions to mitigate this "sign up" step.
### Retrieve the public key from the blockchain
If Bob has signed at least one transaction with his account then his Public Key can be extracted from the transaction's ECDSA signature.
The challenge with this method is that standard Web3 Wallet API does not allow Alice to specifically retrieve all/any transaction sent by Bob.
If Bob has signed at least one transaction with his account
then his Public Key can be extracted from the transaction's ECDSA signature.
The challenge with this method is that standard Web3 Wallet API
does not allow Alice to specifically retrieve all/any transaction sent by Bob.
Alice would instead need to use the `eth.getBlock` API to retrieve Ethereum blocks one by one.
For each block, she would need to check the `from` value of each transaction until she finds a transaction sent by Bob.
Alice would instead need to use the `eth.getBlock` API
to retrieve Ethereum blocks one by one.
For each block, she would need to check the `from` value of each transaction
until she finds a transaction sent by Bob.
This process is resource intensive and can be slow when using services such as Infura due to rate limits in place,
This process is resource intensive and
can be slow when using services such as Infura due to rate limits in place,
which makes it inappropriate for a browser or mobile phone environment.
An alternative would be to either run a backend that can connect directly to an Ethereum node,
An alternative would be to either run a backend
that can connect directly to an Ethereum node,
use a centralized blockchain explorer
or use a decentralized indexing service such as [The Graph](https://thegraph.com/).
Note that these would resolve a UX issue only if a sender wants to proceed with _air drops_.
Note that these would resolve a UX issue
only if a sender wants to proceed with _air drops_.
Indeed, if Bob does not publish his Public Key in the first place
then it can be an indication that he simply does not participate in this protocol and hence will not receive messages.
then it MAY be an indication that he does not participate in this protocol
and hence will not receive messages.
However, these solutions would be helpful if the sender wants to proceed with an _air drop_ of messages:
Send messages over Waku for users to retrieve later, once they decide to participate in this protocol.
However, these solutions would be helpful
if the sender wants to proceed with an _air drop_ of messages:
Send messages over Waku for users to retrieve later,
once they decide to participate in this protocol.
Bob may not want to participate first but may decide to participate at a later stage
and would like to access previous messages.
This could make sense in an NFT offer scenario:
Users send offers to any NFT owner,
NFT owner may decide at some point to participate in the protocol and retrieve previous offers.
NFT owner may decide at some point to participate in the protocol and
retrieve previous offers.
### Publishing the public in long term storage
Another improvement would be for Bob not having to re-publish his public key every 30 days or less.
Similarly to above, if Bob stops publishing his public key then it may be an indication that he does not participate in the protocol anymore.
Another improvement would be for Bob not having to re-publish his public key
every 30 days or less.
Similarly to above,
if Bob stops publishing his public key
then it MAY be an indication that he does not participate in the protocol anymore.
In any case, the protocol could be modified to store the Public Key in a more permanent storage, such as a dedicated smart contract on the blockchain.
In any case,
the protocol could be modified to store the Public Key in a more permanent storage,
such as a dedicated smart contract on the blockchain.
## Send Private Message
Alice MAY monitor the Waku v2 to collect Ethereum Address and Encryption Public Key tuples.
Alice SHOULD verify that the `signature`s of `PublicKeyMessage`s she receives are valid as per EIP-712.
Alice MAY monitor the Waku network to collect Ethereum Address and
Encryption Public Key tuples.
Alice SHOULD verify that the `signature`s of `PublicKeyMessage`s she receives
are valid as per EIP-712.
She SHOULD drop any message without a signature or with an invalid signature.
Using Bob's Encryption Public Key, retrieved via [10/WAKU2 spec](../../core/10/waku2.md), Alice MAY now send an encrypted message to Bob.
Using Bob's Encryption Public Key,
retrieved via [10/WAKU2](/waku/standards/core/10/waku2.md),
Alice MAY now send an encrypted message to Bob.
If she wishes to do so, Alice MUST encrypt her message `M` using Bob's Encryption Public Key `B'`,
as per [26/WAKU-PAYLOAD Asymmetric Encryption specs](../26/payload.md/#asymmetric).
If she wishes to do so,
Alice MUST encrypt her message `M` using Bob's Encryption Public Key `B'`,
as per [26/WAKU-PAYLOAD Asymmetric Encryption specs](waku/standards/application/26/payload.md/#asymmetric).
Alice SHOULD now publish this message on the Private Message content topic.
# Copyright
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [10/WAKU2 spec](../../core/10/waku2.md)
- [Waku Message Version 1](../26/payload.md)
- [10/WAKU2](/waku/standards/core/10/waku2.md)
- [Waku Message Version 1](waku/standards/application/26/payload.md)
- [X3DH](https://www.signal.org/docs/specifications/x3dh/)
- [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/)
- [Status secure transport spec](https://specs.status.im/spec/5)
- [Status secure transport specification](status/deprecated/secure-transport.md)
- [EIP-712](https://eips.ethereum.org/EIPS/eip-712)
- [13/WAKU2-STORE](../../core/13/store.md)
- [13/WAKU2-STORE](/waku/standards/core/13/store.md)
- [The Graph](https://thegraph.com/)

View File

@@ -7,23 +7,45 @@ editor: Sanaz Taheri <sanaz@status.im>
contributors:
---
The reliability of [13/WAKU2-STORE](../../core/13/store.md) protocol heavily relies on the fact that full nodes i.e., those who persist messages have high availability and uptime and do not miss any messages.
If a node goes offline, then it will risk missing all the messages transmitted in the network during that time.
In this specification, we provide a method that makes the store protocol resilient in presence of faulty nodes.
Relying on this method, nodes that have been offline for a time window will be able to fix the gap in their message history when getting back online.
Moreover, nodes with lower availability and uptime can leverage this method to reliably provide the store protocol services as a full node.
The reliability of [13/WAKU2-STORE](../../core/13/store.md)
protocol heavily relies on the fact that full nodes i.e.,
those who persist messages have high availability and
uptime and do not miss any messages.
If a node goes offline,
then it will risk missing all the messages transmitted
in the network during that time.
In this specification,
we provide a method that makes the store protocol resilient
in presence of faulty nodes.
Relying on this method,
nodes that have been offline for a time window will be able to fix the gap
in their message history when getting back online.
Moreover, nodes with lower availability and
uptime can leverage this method to reliably provide the store protocol services
as a full node.
## Method description
As the first step towards making the [13/WAKU2-STORE](../../core/13/store.md) protocol fault-tolerant, we introduce a new type of time-based query through which nodes fetch message history from each other based on their desired time window.
This method operates based on the assumption that the querying node knows some other nodes in the store protocol which have been online for that targeted time window.
## Method description
As the first step
towards making the [13/WAKU2-STORE](../../core/13/store.md) protocol fault-tolerant,
we introduce a new type of time-based query through which nodes fetch message history
from each other based on their desired time window.
This method operates based on the assumption that the querying node
knows some other nodes in the store protocol
which have been online for that targeted time window.
## Security Consideration
The main security consideration to take into account while using this method is that a querying node has to reveal its offline time to the queried node.
This will gradually result in the extraction of the node's activity pattern which can lead to inference attacks.
The main security consideration to take into account
while using this method is that a querying node
has to reveal its offline time to the queried node.
This will gradually result in the extraction of the node's activity pattern
which can lead to inference attacks.
## Wire Specification
We extend the [HistoryQuery](../../core/13/store.md/#payloads) protobuf message with two fields of `start_time` and `end_time` to signify the time range to be queried.
We extend the [HistoryQuery](../../core/13/store.md/#payloads) protobuf message
with two fields of `start_time` and `end_time` to signify the time range to be queried.
### Payloads
@@ -44,24 +66,43 @@ message HistoryQuery {
### HistoryQuery
RPC call to query historical messages.
- `start_time`: this field MAY be filled out to signify the starting point of the queried time window.
This field holds the Unix epoch time in nanoseconds.
The `messages` field of the corresponding [`HistoryResponse`](../../core/13/store.md/#HistoryResponse) MUST contain historical waku messages whose [`timestamp`](../../core/14/message.md/#Payloads) is larger than or equal to the `start_time`.
- `end_time` this field MAY be filled out to signify the ending point of the queried time window.
This field holds the Unix epoch time in nanoseconds.
The `messages` field of the corresponding [`HistoryResponse`](../../core/13/store.md/#HistoryResponse) MUST contain historical waku messages whose [`timestamp`](../../core/14/message.md/#Payloads) is less than or equal to the `end_time`.
A time-based query is considered valid if its `end_time` is larger than or equal to the `start_time`.
Queries that do not adhere to this condition will not get through e.g. an open-end time query in which the `start_time` is given but no `end_time` is supplied is not valid.
If both `start_time` and `end_time` are omitted then no time-window filter takes place.
- `start_time`:
this field MAY be filled out to signify the starting point of the queried time window.
This field holds the Unix epoch time in nanoseconds.
The `messages` field of the corresponding
[`HistoryResponse`](../../core/13/store.md/#HistoryResponse)
MUST contain historical waku messages whose
[`timestamp`](../../core/14/message.md/#Payloads)
is larger than or equal to the `start_time`.
- `end_time`:
this field MAY be filled out to signify the ending point of the queried time window.
This field holds the Unix epoch time in nanoseconds.
The `messages` field of the corresponding
[`HistoryResponse`](../../core/13/store.md/#HistoryResponse)
MUST contain historical waku messages whose
[`timestamp`](../../core/14/message.md/#Payloads) is less than or equal to the `end_time`.
A time-based query is considered valid if
its `end_time` is larger than or equal to the `start_time`.
Queries that do not adhere to this condition will not get through e.g.
an open-end time query in which the `start_time` is given but
no `end_time` is supplied is not valid.
If both `start_time` and
`end_time` are omitted then no time-window filter takes place.
In order to account for nodes asynchrony, and
assuming that nodes may be out of sync for at most 20 seconds
(i.e., 20000000000 nanoseconds),
the querying nodes SHOULD add an offset of 20 seconds to their offline time window.
That is if the original window is [`l`,`r`]
then the history query SHOULD be made for `[start_time: l - 20s, end_time: r + 20s]`.
In order to account for nodes asynchrony, and assuming that nodes may be out of sync for at most 20 seconds (i.e., 20000000000 nanoseconds), the querying nodes SHOULD add an offset of 20 seconds to their offline time window.
That is if the original window is [`l`,`r`] then the history query SHOULD be made for `[start_time: l - 20s, end_time: r + 20s]`.
Note that `HistoryQuery` preserves `AND` operation among the queried attributes.
As such, The `messages` field of the corresponding [`HistoryResponse`](../../core/13/store.md/#HistoryResponse) MUST contain historical waku messages that satisfy the indicated `pubsubtopic` AND `contentFilters` AND the time range [`start_time`, `end_time`].
Note that `HistoryQuery` preserves `AND` operation among the queried attributes.
As such, the `messages` field of the corresponding
[`HistoryResponse`](../../core/13/store.md/#HistoryResponse)
MUST contain historical waku messages that satisfy the indicated `pubsubtopic` AND
`contentFilters` AND the time range [`start_time`, `end_time`].
## Copyright

View File

@@ -5,25 +5,68 @@ name: Waku Message Payload Encryption
status: draft
editor: Oskar Thoren <oskarth@titanproxy.com>
contributors:
- Oskar Thoren <oskarth@titanproxy.com>
---
This specification describes how Waku provides confidentiality, authenticity, and integrity, as well as some form of unlinkability.
Specifically, it describes how encryption, decryption and signing works in [6/WAKU1](../../legacy/6/waku1.md) and in [10/WAKU2 spec](../../core/10/waku2.md) with [14/WAKU-MESSAGE version 1](../../core/14/message.md/#version1).
## Abstract
This specification effectively replaces [7/WAKU-DATA](../../legacy/7/data.md) as well as [6/WAKU1 Payload encryption](../../legacy/6/waku1.md/#payload-encryption) but written in a way that is agnostic and self-contained for Waku v1 and Waku v2.
This specification describes how Waku provides confidentiality, authenticity, and
integrity, as well as some form of unlinkability.
Specifically, it describes how encryption, decryption and
signing works in [6/WAKU1](waku/standards/legacy/6/waku1.md) and
in [10/WAKU2](waku/standards/core/10/waku2.md) with [14/WAKU-MESSAGE](waku/standards/core/14/message.md/#version1).
Large sections of the specification originate from [EIP-627: Whisper spec](https://eips.ethereum.org/EIPS/eip-627) as well from [RLPx Transport Protocol spec (ECIES encryption)](https://github.com/ethereum/devp2p/blob/master/rlpx.md#ecies-encryption) with some modifications.
This specification effectively replaces [7/WAKU-DATA](waku/standards/legacy/7/data.md)
as well as [6/WAKU1 Payload encryption](waku/standards/legacy/6/waku1.md/#payload-encryption)
but written in a way that is agnostic and self-contained for [6/WAKU1](waku/standards/legacy/6/waku1.md) and [10/WAKU2](waku/standards/core/10/waku2.md).
Large sections of the specification originate from
[EIP-627: Whisper spec](https://eips.ethereum.org/EIPS/eip-627) as well from
[RLPx Transport Protocol spec (ECIES encryption)](https://github.com/ethereum/devp2p/blob/master/rlpx.md#ecies-encryption)
with some modifications.
## Specification
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
For [6/WAKU1](waku/standards/legacy/6/waku1.md),
the `data` field is used in the [waku envelope](waku/standards/legacy/6/waku1.md#abnf-specification)
and the field MAY contain the encrypted payload.
For [10/WAKU2](waku/standards/core/10/waku2.md),
the `payload` field is used in `WakuMessage`
and MAY contain the encrypted payload.
The fields that are concatenated and
encrypted as part of the `data` (Waku legacy) or
`payload` (Waku) field are:
- `flags`
- `payload-length`
- `payload`
- `padding`
- `signature`
## Design requirements
- *Confidentiality*: The adversary should not be able to learn what data is being sent from one Waku node to one or several other Waku nodes.
- *Authenticity*: The adversary should not be able to cause Waku endpoint to accept data from any third party as though it came from the other endpoint.
- *Integrity*: The adversary should not be able to cause a Waku endpoint to accept data that has been tampered with.
- *Confidentiality*:
The adversary SHOULD NOT be able to learn what data is being sent from one Waku node
to one or several other Waku nodes.
- *Authenticity*:
The adversary SHOULD NOT be able to cause Waku endpoint
to accept data from any third party as though it came from the other endpoint.
- *Integrity*:
The adversary SHOULD NOT be able to cause a Waku endpoint to
accept data that has been tampered with.
Notable, *forward secrecy* is not provided for at this layer.
If this property is desired, a more fully featured secure communication protocol can be used on top, such as [Status 5/SECURE-TRANSPORT](https://specs.status.im/spec/5).
If this property is desired,
a more fully featured secure communication protocol can be used on top.
It also provides some form of *unlinkability* since:
- only participants who are able to decrypt a message can see its signature
- payload are padded to a fixed length
@@ -35,27 +78,16 @@ It also provides some form of *unlinkability* since:
- KECCAK-256
ECIES is using the following cryptosystem:
- Curve: secp256k1
- KDF: NIST SP 800-56 Concatenation Key Derivation Function, with SHA-256 option
- MAC: HMAC with SHA-256
- AES: AES-128-CTR
## Specification
For [6/WAKU1](../../legacy/6/waku1.md), the `data` field is used in the `waku envelope`, and the field MAY contain the encrypted payload.
For [10/WAKU2 spec](../../core/10/waku2.md), the `payload` field is used in `WakuMessage` and MAY contain the encrypted payload.
The fields that are concatenated and encrypted as part of the `data` (Waku v1) / `payload` (Waku v2) field are:
- flags
- payload-length
- payload
- padding
- signature
### ABNF
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234) we have the following format:
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234)
we have the following format:
```abnf
; 1 byte; first two bits contain the size of payload-length field,
@@ -76,36 +108,50 @@ signature = 65OCTET
data = flags payload-length payload padding [signature]
; This field is called payload in Waku v2
; This field is called payload in Waku
payload = data
```
### Signature
Those unable to decrypt the payload/data are also unable to access the signature.
The signature, if provided, is the ECDSA signature of the Keccak-256 hash of the unencrypted data using the secret key of the originator identity.
The signature is serialized as the concatenation of the `r`, `s` and `v` parameters of the SECP-256k1 ECDSA signature, in that order.
The signature, if provided,
SHOULD be the ECDSA signature of the Keccak-256 hash of the unencrypted data
using the secret key of the originator identity.
The signature is serialized as the concatenation of the `r`, `s` and `v` parameters
of the SECP-256k1 ECDSA signature, in that order.
`r` and `s` MUST be big-endian encoded, fixed-width 256-bit unsigned.
`v` MUST be an 8-bit big-endian encoded, non-normalized and should be either 27 or 28.
`v` MUST be an 8-bit big-endian encoded,
non-normalized and should be either 27 or 28.
See [Ethereum "Yellow paper": Appendix F Signing transactions](https://ethereum.github.io/yellowpaper/paper.pdf) for more information on signature generation, parameters and public key recovery.
See [Ethereum "Yellow paper": Appendix F Signing transactions](https://ethereum.github.io/yellowpaper/paper.pdf)
for more information on signature generation, parameters and public key recovery.
### Encryption
#### Symmetric
Symmetric encryption uses AES-256-GCM for [authenticated encryption](https://en.wikipedia.org/wiki/Authenticated_encryption).
The output of encryption is of the form (`ciphertext`, `tag`, `iv`) where `ciphertext` is the encrypted message, `tag` is a 16 byte message authentication tag and `iv` is a 12 byte initialization vector (nonce).
The message authentication `tag` and initialization vector `iv` field MUST be appended to the resulting `ciphertext`, in that order.
Note that previous specifications and some implementations might refer to `iv` as `nonce` or `salt`.
Symmetric encryption uses AES-256-GCM for
[authenticated encryption](https://en.wikipedia.org/wiki/Authenticated_encryption).
The output of encryption is of the form (`ciphertext`, `tag`, `iv`)
where `ciphertext` is the encrypted message,
`tag` is a 16 byte message authentication tag and
`iv` is a 12 byte initialization vector (nonce).
The message authentication `tag` and
initialization vector `iv` field MUST be appended to the resulting `ciphertext`,
in that order.
Note that previous specifications and
some implementations might refer to `iv` as `nonce` or `salt`.
#### Asymmetric
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme (ECIES) with SECP-256k1 public key.
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme
(ECIES) with SECP-256k1 public key.
#### ECIES
This section originates from the [RLPx Transport Protocol spec](https://github.com/ethereum/devp2p/blob/master/rlpx.md#ecies-encryption) spec with minor modifications.
This section originates from the [RLPx Transport Protocol spec](https://github.com/ethereum/devp2p/blob/master/rlpx.md#ecies-encryption)
specification with minor modifications.
The cryptosystem used is:
@@ -116,24 +162,39 @@ The cryptosystem used is:
Special notation used: `X || Y` denotes concatenation of `X` and `Y`.
Alice wants to send an encrypted message that can be decrypted by Bob's static private key `kB`. Alice knows about Bobs static public key `KB`.
Alice wants to send an encrypted message that can be decrypted by
Bob's static private key `kB`.
Alice knows about Bobs static public key `KB`.
To encrypt the message `m`, Alice generates a random number `r` and corresponding elliptic curve public key `R = r * G` and computes the shared secret `S = Px` where `(Px, Py) = r * KB`.
She derives key material for encryption and authentication as `kE || kM = KDF(S, 32)` as well as a random initialization vector `iv`.
Alice sends the encrypted message `R || iv || c || d` where `c = AES(kE, iv , m)` and `d = MAC(sha256(kM), iv || c)` to Bob.
To encrypt the message `m`, Alice generates a random number `r` and
corresponding elliptic curve public key `R = r * G` and
computes the shared secret `S = Px` where `(Px, Py) = r * KB`.
She derives key material for encryption and
authentication as `kE || kM = KDF(S, 32)`
as well as a random initialization vector `iv`.
Alice sends the encrypted message `R || iv || c || d` where `c = AES(kE, iv , m)`
and `d = MAC(sha256(kM), iv || c)` to Bob.
For Bob to decrypt the message `R || iv || c || d`, he derives the shared secret `S = Px` where `(Px, Py) = kB * R` as well as the encryption and authentication keys `kE || kM = KDF(S, 32)`.
Bob verifies the authenticity of the message by checking whether `d == MAC(sha256(kM), iv || c)` then obtains the plaintext as `m = AES(kE, iv || c)`.
For Bob to decrypt the message `R || iv || c || d`,
he derives the shared secret `S = Px` where `(Px, Py) = kB * R`
as well as the encryption and authentication keys `kE || kM = KDF(S, 32)`.
Bob verifies the authenticity of the message
by checking whether `d == MAC(sha256(kM), iv || c)`
then obtains the plaintext as `m = AES(kE, iv || c)`.
### Padding
The padding field is used to align data size, since data size alone might reveal important metainformation.
The `padding` field is used to align data size,
since data size alone might reveal important metainformation.
Padding can be arbitrary size.
However, it is recommended that the size of Data Field (excluding the IV and tag) before encryption (i.e. plain text) SHOULD be a multiple of 256 bytes.
However, it is recommended that the size of `data` field
(excluding the `iv` and `tag`) before encryption (i.e. plain text)
SHOULD be a multiple of 256 bytes.
### Decoding a message
In order to decode a message, a node SHOULD try to apply both symmetric and asymmetric decryption operations.
In order to decode a message, a node SHOULD try to apply both symmetric and
asymmetric decryption operations.
This is because the type of encryption is not included in the message.
## Copyright
@@ -142,13 +203,13 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
## References
1. [6/WAKU1](../../legacy/6/waku1.md)
2. [10/WAKU2 spec](../../core/10/waku2.md)
3. [14/WAKU-MESSAGE version 1](../../core/14/message.md/#version1)
4. [7/WAKU-DATA](../../legacy/7/data.md)
1. [6/WAKU1](waku/standards/legacy/6/waku1.md)
2. [10/WAKU2 spec](waku/standards/core/10/waku2.md)
3. [14/WAKU-MESSAGE version 1](waku/standards/core/14/message.md/#version1)
4. [7/WAKU-DATA](waku/standards/legacy/7/data.md)
5. [EIP-627: Whisper spec](https://eips.ethereum.org/EIPS/eip-627)
6. [RLPx Transport Protocol spec (ECIES encryption)](https://github.com/ethereum/devp2p/blob/master/rlpx.md#ecies-encryption)
7. [Status 5/SECURE-TRANSPORT](https://specs.status.im/spec/5)
7. [Status 5/SECURE-TRANSPORT](status/deprecated/secure-transport.md)
8. [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234)
9. [Ethereum "Yellow paper": Appendix F Signing transactions](https://ethereum.github.io/yellowpaper/paper.pdf)
10. [authenticated encryption](https://en.wikipedia.org/wiki/Authenticated_encryption)

View File

@@ -7,84 +7,112 @@ category: Standards Track
tags: waku-application
editor: Aaryamann Challani <p1ge0nh8er@proton.me>
contributors:
- Andrea Piana <andreap@status.im>
- Pedro Pombeiro <pedro@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskarth@titanproxy.com>
- Dean Eigenmann <dean@status.im>
- Andrea Piana <andreap@status.im>
- Pedro Pombeiro <pedro@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskarth@titanproxy.com>
- Dean Eigenmann <dean@status.im>
- Filip Dimitrijevic <filip@status.im>
---
## Abstract
This document describes a method that can be used to provide a secure channel between two peers, and thus provide confidentiality, integrity, authenticity and forward secrecy.
This document describes a method that can be used to provide a secure channel
between two peers, and thus provide confidentiality, integrity,
authenticity and forward secrecy.
It is transport-agnostic and works over asynchronous networks.
It builds on the [X3DH](https://signal.org/docs/specifications/x3dh/) and [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/) specifications, with some adaptations to operate in a decentralized environment.
It builds on the [X3DH](https://signal.org/docs/specifications/x3dh/)
and [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/) specifications,
with some adaptations to operate in a decentralized environment.
## Motivation
Nodes on a network may want to communicate with each other in a secure manner, without other nodes network being able to read their messages.
Nodes on a network may want to communicate with each other in a secure manner,
without other nodes network being able to read their messages.
## Specification
### Definitions
- **Perfect Forward Secrecy** is a feature of specific key-agreement protocols which provide assurances that session keys will not be compromised even if the private keys of the participants are compromised.
Specifically, past messages cannot be decrypted by a third-party who manages to get a hold of a private key.
- **Secret channel** describes a communication channel where a Double Ratchet algorithm is in use.
- **Perfect Forward Secrecy** is a feature of specific key-agreement protocols
which provide assurances that session keys will not be compromised
even if the private keys of the participants are compromised.
Specifically, past messages cannot be decrypted by a third-party
who manages to obtain those private key.
- **Secret channel** describes a communication channel
where a Double Ratchet algorithm is in use.
### Design Requirements
- **Confidentiality**: The adversary should not be able to learn what data is being exchanged between two Status clients.
- **Authenticity**: The adversary should not be able to cause either endpoint to accept data from any third party as though it came from the other endpoint.
- **Forward Secrecy**: The adversary should not be able to learn what data was exchanged between two clients if, at some later time, the adversary compromises one or both of the endpoints.
- **Integrity**: The adversary should not be able to cause either endpoint to accept data that has been tampered with.
- **Confidentiality**:
The adversary should not be able to learn what data is being exchanged
between two Status clients.
- **Authenticity**:
The adversary should not be able to cause either endpoint
to accept data from any third party as though it came from the other endpoint.
- **Forward Secrecy**:
The adversary should not be able to learn what data was exchanged
between two clients if, at some later time,
the adversary compromises one or both of the endpoints.
- **Integrity**:
The adversary should not be able to cause either endpoint
to accept data that has been tampered with.
All of these properties are ensured by the use of [Signal's Double Ratchet](https://signal.org/docs/specifications/doubleratchet/)
### Conventions
Types used in this specification are defined using the [Protobuf](https://developers.google.com/protocol-buffers/) wire format.
## Specification
Types used in this specification are defined using the
[Protobuf](https://developers.google.com/protocol-buffers/) wire format.
### End-to-End Encryption
End-to-end encryption (E2EE) takes place between two clients.
The main cryptographic protocol is a Double Ratchet protocol, which is derived from the [Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html), using a different ratchet.
[The Waku v2 protocol](../../core/10/waku2.md) subsequently encrypts the message payload, using symmetric key encryption.
Furthermore, the concept of prekeys (through the use of [X3DH](https://signal.org/docs/specifications/x3dh/)) is used to allow the protocol to operate in an asynchronous environment.
It is not necessary for two parties to be online at the same time to initiate an encrypted conversation.
End-to-end encryption (E2EE) takes place between two clients.
The main cryptographic protocol is a Double Ratchet protocol,
which is derived from the
[Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html),
using a different ratchet.
[The Waku v2 protocol](/waku/standards/core/10/waku2.md)
subsequently encrypts the message payload, using symmetric key encryption.
Furthermore, the concept of prekeys
(through the use of [X3DH](https://signal.org/docs/specifications/x3dh/))
is used to allow the protocol to operate in an asynchronous environment.
It is not necessary for two parties to be online at the same time
to initiate an encrypted conversation.
### Cryptographic Protocols
This protocol uses the following cryptographic primitives:
- X3DH
- Elliptic curve Diffie-Hellman key exchange (secp256k1)
- KECCAK-256
- ECDSA
- ECIES
- Elliptic curve Diffie-Hellman key exchange (secp256k1)
- KECCAK-256
- ECDSA
- ECIES
- Double Ratchet
- HMAC-SHA-256 as MAC
- Elliptic curve Diffie-Hellman key exchange (Curve25519)
- AES-256-CTR with HMAC-SHA-256 and IV derived alongside an encryption key
- HMAC-SHA-256 as MAC
- Elliptic curve Diffie-Hellman key exchange (Curve25519)
- AES-256-CTR with HMAC-SHA-256 and IV derived alongside an encryption key
The node achieves key derivation using [HKDF](https://www.rfc-editor.org/rfc/rfc5869).
### Pre-keys
Every client SHOULD initially generate some key material which is stored locally:
- Identity keypair based on secp256k1 - `IK`
- A signed prekey based on secp256k1 - `SPK`
- A prekey signature - `Sig(IK, Encode(SPK))`
More details can be found in the `X3DH Prekey bundle creation` section of [2/ACCOUNT](https://specs.status.im/spec/2#x3dh-prekey-bundles).
Prekey bundles MAY be extracted from any peer's messages, or found via searching for their specific topic, `{IK}-contact-code`.
Prekey bundles MAY be extracted from any peer's messages,
or found via searching for their specific topic, `{IK}-contact-code`.
The following methods can be used to retrieve prekey bundles from a peer's messages:
- contact codes;
- public and one-to-one chats;
- QR codes;
@@ -94,28 +122,41 @@ The following methods can be used to retrieve prekey bundles from a peer's messa
Waku SHOULD be used for retrieving prekey bundles.
Since bundles stored in QR codes or ENS records cannot be updated to delete already used keys, the bundle MAY be rotated every 24 hours, and distributed via Waku.
Since bundles stored in QR codes or
ENS records cannot be updated to delete already used keys,
the bundle MAY be rotated every 24 hours, and distributed via Waku.
### Flow
The key exchange can be summarized as follows:
1. Initial key exchange: Two parties, Alice and Bob, exchange their prekey bundles, and derive a shared secret.
1. Initial key exchange: Two parties, Alice and Bob, exchange their prekey bundles,
and derive a shared secret.
2. Double Ratchet: The two parties use the shared secret to derive a new encryption key for each message they send.
2. Double Ratchet:
The two parties use the shared secret to derive a new encryption key
for each message they send.
3. Chain key update: The two parties update their chain keys. The chain key is used to derive new encryption keys for future messages.
3. Chain key update: The two parties update their chain keys.
The chain key is used to derive new encryption keys for future messages.
4. Message key derivation: The two parties derive a new message key from their chain key, and use it to encrypt a message.
4. Message key derivation:
The two parties derive a new message key from their chain key, and
use it to encrypt a message.
#### 1. Initial key exchange flow (X3DH)
[Section 3 of the X3DH protocol](https://signal.org/docs/specifications/x3dh/#sending-the-initial-message) describes the initial key exchange flow, with some additional context:
- The peers' identity keys `IK_A` and `IK_B` correspond to their public keys;
- Since it is not possible to guarantee that a prekey will be used only once in a decentralized world, the one-time prekey `OPK_B` is not used in this scenario;
- Nodes SHOULD not send Bundles to a centralized server, but instead provide them in a decentralized way as described in the [Pre-keys section](#pre-keys).
[Section 3 of the X3DH protocol](https://signal.org/docs/specifications/x3dh/#sending-the-initial-message)
describes the initial key exchange flow, with some additional context:
Alice retrieves Bob's prekey bundle, however it is not specific to Alice. It contains:
- The peers' identity keys `IK_A` and `IK_B` correspond to their public keys;
- Since it is not possible to guarantee that a prekey will be used only once
in a decentralized world, the one-time prekey `OPK_B` is not used in this scenario;
- Nodes SHOULD not send Bundles to a centralized server,
but instead provide them in a decentralized way as described in the [Pre-keys section](#pre-keys).
Alice retrieves Bob's prekey bundle, however it is not specific to Alice.
It contains:
([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L12))
@@ -144,17 +185,23 @@ message SignedPreKey {
}
```
The `signature` is generated by sorting `installation-id` in lexicographical order, and concatenating the `signed-pre-key` and `version`:
The `signature` is generated by sorting `installation-id` in lexicographical order,
and concatenating the `signed-pre-key` and `version`:
`installation-id-1signed-pre-key1version1installation-id2signed-pre-key2-version-2`
#### 2. Double Ratchet
Having established the initial shared secret `SK` through X3DH, it SHOULD be used to seed a Double Ratchet exchange between Alice and Bob.
Having established the initial shared secret `SK` through X3DH,
it SHOULD be used to seed a Double Ratchet exchange between Alice and Bob.
Refer to the [Double Ratchet spec](https://signal.org/docs/specifications/doubleratchet/) for more details.
Refer to the [Double Ratchet spec](https://signal.org/docs/specifications/doubleratchet/)
for more details.
The initial message sent by Alice to Bob is sent as a top-level `ProtocolMessage` ([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L65)) containing a map of `DirectMessageProtocol` indexed by `installation-id` ([reference wire format](https://github.com/status-im/status-go/blob/1ac9dd974415c3f6dee95145b6644aeadf02f02c/services/shhext/chat/encryption.proto#L56)):
The initial message sent by Alice to Bob is sent as a top-level `ProtocolMessage`
([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L65))
containing a map of `DirectMessageProtocol` indexed by `installation-id`
([reference wire format](https://github.com/status-im/status-go/blob/1ac9dd974415c3f6dee95145b6644aeadf02f02c/services/shhext/chat/encryption.proto#L56)):
``` protobuf
message ProtocolMessage {
@@ -180,67 +227,84 @@ message EncryptedMessageProtocol {
bytes payload = 3;
}
```
Where:
- `X3DH_header`: the `X3DHHeader` field in `DirectMessageProtocol` contains:
([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L47))
``` protobuf
message X3DHHeader {
// Alice's ephemeral key `EK_A`
bytes key = 1;
// Bob's bundle signed prekey
bytes id = 4;
}
```
- `DR_header`: Double ratchet header ([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L31)). Used when Bob's public bundle is available:
``` protobuf
message DRHeader {
// Alice's current ratchet public key (as mentioned in [DR spec section 2.2](https://signal.org/docs/specifications/doubleratchet/#symmetric-key-ratchet))
bytes key = 1;
// number of the message in the sending chain
uint32 n = 2;
// length of the previous sending chain
uint32 pn = 3;
// Bob's bundle ID
bytes id = 4;
}
```
```protobuf
message X3DHHeader {
// Alice's ephemeral key `EK_A`
bytes key = 1;
// Bob's bundle signed prekey
bytes id = 4;
}
```
- `DR_header`: Double ratchet header ([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L31)).
Used when Bob's public bundle is available:
``` protobuf
message DRHeader {
// Alice's current ratchet public key
bytes key = 1;
// number of the message in the sending chain
uint32 n = 2;
// length of the previous sending chain
uint32 pn = 3;
// Bob's bundle ID
bytes id = 4;
}
```
Alice's current ratchet public key (above) is mentioned in
[DR spec section 2.2](https://signal.org/docs/specifications/doubleratchet/#symmetric-key-ratchet)
- `DH_header`: Diffie-Hellman header (used when Bob's bundle is not available):
([reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L42))
``` protobuf
message DHHeader {
// Alice's compressed ephemeral public key.
bytes key = 1;
}
```
``` protobuf
message DHHeader {
// Alice's compressed ephemeral public key.
bytes key = 1;
}
```
#### 3. Chain key update
The chain key MUST be updated according to the `DR_Header` received in the `EncryptedMessageProtocol` message, described in [2.Double Ratchet](#2-double-ratchet).
The chain key MUST be updated according to the `DR_Header`
received in the `EncryptedMessageProtocol` message,
described in [2.Double Ratchet](#2-double-ratchet).
#### 4. Message key derivation
The message key MUST be derived from a single ratchet step in the symmetric-key ratchet as described in [Symmetric key ratchet](https://signal.org/docs/specifications/doubleratchet/#symmetric-key-ratchet)
The message key MUST be derived from a single ratchet step in the symmetric-key ratchet
as described in [Symmetric key ratchet](https://signal.org/docs/specifications/doubleratchet/#symmetric-key-ratchet)
The message key MUST be used to encrypt the next message to be sent.
## Security Considerations
1. Inherits the security considerations of [X3DH](https://signal.org/docs/specifications/x3dh/#security-considerations) and [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/#security-considerations).
1. Inherits the security considerations of [X3DH](https://signal.org/docs/specifications/x3dh/#security-considerations)
and [Double Ratchet](https://signal.org/docs/specifications/doubleratchet/#security-considerations).
2. Inherits the security considerations of the [Waku v2 protocol](../../core/10/waku2.md).
2. Inherits the security considerations of the [Waku v2 protocol](/waku/standards/core/10/waku2.md).
3. The protocol is designed to be used in a decentralized manner, however, it is possible to use a centralized server to serve prekey bundles. In this case, the server is trusted.
3. The protocol is designed to be used in a decentralized manner, however,
it is possible to use a centralized server to serve prekey bundles.
In this case, the server is trusted.
## Privacy Considerations
1. This protocol does not provide message unlinkability. It is possible to link messages signed by the same keypair.
1. This protocol does not provide message unlinkability.
It is possible to link messages signed by the same keypair.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
@@ -249,10 +313,8 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
- [Signal's Double Ratchet](https://signal.org/docs/specifications/doubleratchet/)
- [Protobuf](https://developers.google.com/protocol-buffers/)
- [Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html)
- [The Waku v2 protocol](../../core/10/waku2.md)
- [The Waku v2 protocol](/waku/standards/core/10/waku2.md)
- [HKDF](https://www.rfc-editor.org/rfc/rfc5869)
- [2/ACCOUNT](https://specs.status.im/spec/2#x3dh-prekey-bundles)
- [reference wire format](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L12)
- [Symmetric key ratchet](https://signal.org/docs/specifications/doubleratchet/#symmetric-key-ratchet)
-
-

View File

@@ -7,39 +7,48 @@ category: Standards Track
tags: waku-application
editor: Aaryamann Challani <p1ge0nh8er@proton.me>
contributors:
- Andrea Piana <andreap@status.im>
- Pedro Pombeiro <pedro@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskarth@titanproxy.com>
- Dean Eigenmann <dean@status.im>
- Andrea Piana <andreap@status.im>
- Pedro Pombeiro <pedro@status.im>
- Corey Petty <corey@status.im>
- Oskar Thorén <oskarth@titanproxy.com>
- Dean Eigenmann <dean@status.im>
- Filip Dimitrijevic <filip@status.im>
---
## Abstract
This document specifies how to manage sessions based on an X3DH key exchange.
This includes how to establish new sessions, how to re-establish them, how to maintain them, and how to close them.
This includes how to establish new sessions,
how to re-establish them, how to maintain them, and how to close them.
[53/WAKU2-X3DH](../53/x3dh.md) specifies the Waku `X3DH` protocol for end-to-end encryption.
[53/WAKU2-X3DH](/waku/standards/application/53/x3dh.md) specifies the Waku `X3DH` protocol
for end-to-end encryption.
Once two peers complete an X3DH handshake, they SHOULD establish an X3DH session.
## Session Establishment
A node identifies a peer by their `installation-id` which MAY be interpreted as a device identifier.
A node identifies a peer by their `installation-id`
which MAY be interpreted as a device identifier.
### Discovery of pre-key bundles
The node's pre-key bundle MUST be broadcast on a content topic derived from the node's public key, so that the first message may be PFS-encrypted.
Each peer MUST publish their pre-key bundle periodically to this topic, otherwise they risk not being able to perform key-exchanges with other peers.
Each peer MAY publish to this topic when their metadata changes, so that the other peer can update their local record.
The node's pre-key bundle MUST be broadcast on a content topic
derived from the node's public key, so that the first message may be PFS-encrypted.
Each peer MUST publish their pre-key bundle periodically to this topic,
otherwise they risk not being able to perform key-exchanges with other peers.
Each peer MAY publish to this topic when their metadata changes,
so that the other peer can update their local record.
If peer A wants to send a message to peer B, it MUST derive the topic from peer B's public key, which has been shared out of band.
Partitioned topics have been used to balance privacy and efficiency of broadcasting pre-key bundles.
If peer A wants to send a message to peer B,
it MUST derive the topic from peer B's public key, which has been shared out of band.
Partitioned topics have been used to balance privacy and
efficiency of broadcasting pre-key bundles.
The number of partitions that MUST be used is 5000.
The topic MUST be derived as follows:
```
```js
var partitionsNum *big.Int = big.NewInt(5000)
var partition *big.Int = big.NewInt(0).Mod(peerBPublicKey, partitionsNum)
@@ -59,22 +68,24 @@ for i = 0; i < topicLen; i++ {
```
### Initialization
A node initializes a new session once a successful X3DH exchange has taken place.
A node initializes a new session once a successful X3DH exchange has taken place.
Subsequent messages will use the established session until re-keying is necessary.
### Negotiated topic to be used for the session
After the peers have performed the initial key exchange, they MUST derive a topic from their shared secret to send messages on.
To obtain this value, take the first four bytes of the keccak256 hash of the shared secret encoded in hexadecimal format.
After the peers have performed the initial key exchange,
they MUST derive a topic from their shared secret to send messages on.
To obtain this value, take the first four bytes of the keccak256 hash
of the shared secret encoded in hexadecimal format.
```
```js
sharedKey, err := ecies.ImportECDSA(myPrivateKey).GenerateShared(
ecies.ImportECDSAPublic(theirPublicKey),
16,
16,
ecies.ImportECDSAPublic(theirPublicKey),
16,
16,
)
hexEncodedKey := hex.EncodeToString(sharedKey)
var hash []byte = keccak256(hexEncodedKey)
@@ -90,75 +101,103 @@ for i = 0; i < topicLen; i++ {
}
```
To summarize, following is the process for peer B to establish a session with peer A:
1. Listen to peer B's Contact Code Topic to retrieve their bundle information, including a list of active devices
To summarize,
following is the process for peer B to establish a session with peer A:
1. Listen to peer B's Contact Code Topic to retrieve their bundle information,
including a list of active devices
2. Peer A sends their pre-key bundle on peer B's partitioned topic
3. Peer A and peer B perform the key-exchange using the shared pre-key bundles
3. The negotiated topic is derived from the shared secret
4. Peers A & B exchange messages on the negotiated topic
4. The negotiated topic is derived from the shared secret
5. Peers A & B exchange messages on the negotiated topic
### Concurrent sessions
If a node creates two sessions concurrently between two peers, the one with the symmetric key first in byte order SHOULD be used, this marks that the other has expired.
If a node creates two sessions concurrently between two peers,
the one with the symmetric key first in byte order SHOULD be used,
this marks that the other has expired.
### Re-keying
On receiving a bundle from a given peer with a higher version, the old bundle SHOULD be marked as expired and a new session SHOULD be established on the next message sent.
On receiving a bundle from a given peer with a higher version,
the old bundle SHOULD be marked as expired and
a new session SHOULD be established on the next message sent.
### Multi-device support
Multi-device support is quite challenging as there is not a central place where information on which and how many devices (identified by their respective `installation-id`) a peer has, is stored.
Multi-device support is quite challenging
as there is not a central place where information on which and how many devices
(identified by their respective `installation-id`) a peer has, is stored.
Furthermore, account recovery always needs to be taken into consideration, where a user wipes clean the whole device and the node loses all the information about any previous sessions.
Taking these considerations into account, the way the network propagates multi-device information using X3DH bundles, which will contain information about paired devices as well as information about the sending device.
This means that every time a new device is paired, the bundle needs to be updated and propagated with the new information, the user has the responsibility to make sure the pairing is successful.
Furthermore, account recovery always needs to be taken into consideration,
where a user wipes clean the whole device and
the node loses all the information about any previous sessions.
Taking these considerations into account,
the way the network propagates multi-device information using X3DH bundles,
which will contain information about paired devices
as well as information about the sending device.
This means that every time a new device is paired,
the bundle needs to be updated and propagated with the new information,
the user has the responsibility to make sure the pairing is successful.
The method is loosely based on [Signal's Sesame Algorithm](https://signal.org/docs/specifications/sesame/).
### Pairing
A new `installation-id` MUST be generated on a per-device basis.
The device should be paired as soon as possible if other devices are present.
A new `installation-id` MUST be generated on a per-device basis.
The device should be paired as soon as possible if other devices are present.
If a bundle is received, which has the same `IK` as the keypair present on the device, the devices MAY be paired.
Once a user enables a new device, a new bundle MUST be generated which includes pairing information.
If a bundle is received, which has the same `IK` as the keypair present on the device,
the devices MAY be paired.
Once a user enables a new device,
a new bundle MUST be generated which includes pairing information.
The bundle MUST be propagated to contacts through the usual channels.
Removal of paired devices is a manual step that needs to be applied on each device, and consist simply in disabling the device, at which point pairing information will not be propagated anymore.
Removal of paired devices is a manual step that needs to be applied on each device,
and consist simply in disabling the device,
at which point pairing information will not be propagated anymore.
### Sending messages to a paired group
When sending a message, the peer SHOULD send a message to other `installation-id` that they have seen.
The node caps the number of devices to `n`, ordered by last activity.
When sending a message,
the peer SHOULD send a message to other `installation-id` that they have seen.
The node caps the number of devices to `n`, ordered by last activity.
The node sends messages using pairwise encryption, including their own devices.
Where `n` is the maximum number of devices that can be paired.
### Account recovery
Account recovery is the same as adding a new device, and it MUST be handled the same way.
Account recovery is the same as adding a new device,
and it MUST be handled the same way.
### Partitioned devices
In some cases (i.e. account recovery when no other pairing device is available, device not paired), it is possible that a device will receive a message that is not targeted to its own `installation-id`.
In this case an empty message containing bundle information MUST be sent back, which will notify the receiving end not to include the device in any further communication.
In some cases
(i.e. account recovery when no other pairing device is available, device not paired),
it is possible that a device will receive a message
that is not targeted to its own `installation-id`.
In this case an empty message containing bundle information MUST be sent back,
which will notify the receiving end not to include the device in any further communication.
## Security Considerations
1. Inherits all security considerations from [53/WAKU2-X3DH](../53/x3dh.md).
1. Inherits all security considerations from [53/WAKU2-X3DH](/waku/standards/application/53/x3dh.md).
### Recommendations
1. The value of `n` SHOULD be configured by the app-protocol.
- The default value SHOULD be 3, since a larger number of devices will result in a larger bundle size, which may not be desirable in a peer-to-peer network.
- The default value SHOULD be 3,
since a larger number of devices will result in a larger bundle size,
which may not be desirable in a peer-to-peer network.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [53/WAKU2-X3DH](../53/x3dh.md)
2. [Signal's Sesame Algorithm](https://signal.org/docs/specifications/sesame/)
- [53/WAKU2-X3DH](/waku/standards/application/53/x3dh.md)
- [Signal's Sesame Algorithm](https://signal.org/docs/specifications/sesame/)

View File

@@ -14,388 +14,497 @@ contributors:
## Abstract
Waku v2 is family of modular peer-to-peer protocols for secure communication.
The protocols are designed to be secure, privacy-preserving, censorship-resistant and being able to run in resource restricted environments.
At a high level, it implements Pub/Sub over [libp2p](https://github.com/libp2p/specs) and adds a set of capabilities to it.
Waku is a family of modular peer-to-peer protocols for secure communication.
The protocols are designed to be secure, privacy-preserving, censorship-resistant
and being able to run in resource-restricted environments.
At a high level, it implements Pub/Sub over [libp2p](https://github.com/libp2p/specs)
and adds a set of capabilities to it.
These capabilities are things such as:
(i) retrieving historical messages for mostly-offline devices
(ii) adaptive nodes, allowing for heterogeneous nodes to contribute to the network
(iii) preserving bandwidth usage for resource-restriced devices
This makes Waku ideal for running a p2p protocol on mobile and in similarly restricted environments.
This makes Waku ideal for running a p2p protocol on mobile devices and
other similar restricted environments.
Historically, it has its roots in [6/WAKU1](../../legacy/6/waku1.md),
which stems from [Whisper](https://eips.ethereum.org/EIPS/eip-627), originally part of the Ethereum stack.
However, Waku v2 acts more as a thin wrapper for PubSub and has a different API.
It is implemented in an iterative manner where initial focus is on porting essential functionality to libp2p.
Historically, it has its roots in [6/WAKU1](/waku/standards/legacy/6/waku1.md),
which stems from [Whisper](https://eips.ethereum.org/EIPS/eip-627),
originally part of the Ethereum stack.
However, Waku acts more as a thin wrapper for Pub/Sub and has a different API.
It is implemented in an iterative manner where initial focus
is on porting essential functionality to libp2p.
See [rough road map (2020)](https://vac.dev/waku-v2-plan) for more historical context.
## Motivation and goals
## Motivation and Goals
Waku as a family of protocols is designed to have a set of properties that are useful for many applications:
Waku, as a family of protocols, is designed to have a set of properties
that are useful for many applications:
1. **Useful for generalized messaging.**
1.**Useful for generalized messaging.**
Many applications require some form of messaging protocol to communicate between different subsystems or different nodes.
This messaging can be human-to-human or machine-to-machine or a mix.
Many applications require some form of messaging protocol to communicate
between different subsystems or different nodes.
This messaging can be human-to-human, machine-to-machine or a mix.
Waku is designed to work for all these scenarios.
2. **Peer-to-peer.**
2.**Peer-to-peer.**
Applications sometimes have requirements that make them suitable for peer-to-peer solutions:
- Censorship-resistant with no single point of failure
Applications sometimes have requirements that make them suitable
for peer-to-peer solutions:
- Censorship-resistant with no single point of failure
- Adaptive and scalable network
- Shared infrastructure
3. **Runs anywhere.**
3.**Runs anywhere.**
Applications often run in restricted environments, where resources or the environment is restricted in some fashion.
Applications often run in restricted environments,
where resources or the environment is restricted in some fashion.
For example:
- Limited bandwidth, CPU, memory, disk, battery, etc
- Limited bandwidth, CPU, memory, disk, battery, etc.
- Not being publicly connectable
- Only being intermittently connected; mostly-offline
4. **Privacy-preserving.**
4.**Privacy-preserving.**
Applications often have a desire for some privacy guarantees, such as:
- Pseudonymity and not being tied to any personally identifiable information (PII)
- Metadata protection in transit
- Various forms of unlinkability, etc
5. **Modular design.**
- Pseudonymity and not being tied to any personally identifiable information (PII)
- Metadata protection in transit
- Various forms of unlinkability, etc.
Applications often have different trade-offs when it comes to what properties they and their users value.
Waku is designed in a modular fashion where an application protocol or node can choose what protocols they run.
5.**Modular design.**
Applications often have different trade-offs when it comes to what properties they
and their users value.
Waku is designed in a modular fashion where an application protocol or
node can choose what protocols they run.
We call this concept *adaptive nodes*.
For example:
- Resource usage vs metadata protection
- Providing useful services to the network vs mostly using it
- Stronger guarantees for spam protection vs economic registration cost
For more on the concept of adaptive nodes and what this means in practice,
please see the [30/ADAPTIVE-NODES](../../../informational/30/adaptive-nodes.md) spec.
please see the [30/ADAPTIVE-NODES](/waku/informational/30/adaptive-nodes.md) spec.
## Network interaction domains
## Specification
While Waku is best thought of as a single cohesive thing, there are three network interaction domains:
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
### Network Interaction Domains
While Waku is best thought of as a single cohesive thing,
there are three network interaction domains:
(a) gossip domain
(b) discovery domain
(c) req/resp domain
(c) request/response domain
### Protocols and identifiers
#### Protocols and Identifiers
Since Waku v2 is built on top of libp2p, many protocols have a libp2p protocol identifier.
The current main [protocol identifiers](https://docs.libp2p.io/concepts/protocols/) are:
Since Waku is built on top of libp2p, many protocols have a libp2p protocol identifier.
The current main [protocol identifiers](https://docs.libp2p.io/concepts/protocols/)
are:
1. `/vac/waku/relay/2.0.0`
2. `/vac/waku/store/2.0.0-beta4`
2. `/vac/waku/store-query/3.0.0`
3. `/vac/waku/filter/2.0.0-beta1`
4. `/vac/waku/lightpush/2.0.0-beta1`
This is in addition to protocols that specify messages, payloads, and recommended usages.
Since these aren't negotiated libp2p protocols, they are referred to by their RFC ID.
This is in addition to protocols that specify messages, payloads, and
recommended usages.
Since these aren't negotiated libp2p protocols,
they are referred to by their RFC ID.
For example:
- [14/WAKU2-MESSAGE](../14/message.md) and [26/WAKU-PAYLOAD](../../application/26/payload.md) for message payloads
- [23/WAKU2-TOPICS](../../../informational/23/topics.md) and [27/WAKU2-PEERS](../../../informational/27/peers.md) for recommendations around usage
- [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) and
[26/WAKU-PAYLOAD](/waku/standards/application/26/payload.md) for message payloads
- [23/WAKU2-TOPICS](/waku/informational/23/topics.md) and
[27/WAKU2-PEERS](/waku/informational/27/peers.md) for recommendations around usage
There are also more experimental libp2p protocols such as:
1. `/vac/waku/swap/2.0.0-beta1`
2. `/vac/waku/waku-rln-relay/2.0.0-alpha1`
1. `/vac/waku/waku-rln-relay/2.0.0-alpha1`
2. `/vac/waku/peer-exchange/2.0.0-alpha1`
These protocols and their semantics are elaborated on in their own specs.
The semantics of these protocols are referred to by RFC ID [17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md) and [34/WAKU2-PEER-EXCHANGE](/waku/standards/core/34/peer-exchange.md).
#### Use of libp2p and Protobuf
### Use of libp2p and protobuf
Unless otherwise specified, all protocols are implemented over libp2p and use Protobuf by default.
Unless otherwise specified,
all protocols are implemented over libp2p and use Protobuf by default.
Since messages are exchanged over a [bi-directional binary stream](https://docs.libp2p.io/concepts/protocols/),
as a convention, libp2p protocols prefix binary message payloads with the length of the message in bytes.
as a convention,
libp2p protocols prefix binary message payloads with
the length of the message in bytes.
This length integer is encoded as a [protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints).
### Gossip domain
#### Gossip Domain
Waku is using gossiping to disseminate messages throughout the network.
**Protocol identifier**: `/vac/waku/relay/2.0.0`
See [11/WAKU2-RELAY](../11/relay.md) spec for more details.
See [11/WAKU2-RELAY](/waku/standards/core/11/relay.md) specification for more details.
For an experimental privacy-preserving economic spam protection mechanism, see [17/WAKU2-RLN-RELAY](../17/rln-relay.md).
For an experimental privacy-preserving economic spam protection mechanism,
see [17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md).
See [23/WAKU2-TOPICS](../../../informational/23/topics.md) for more information about recommended topic usage.
See [23/WAKU2-TOPICS](/waku/informational/23/topics.md)
for more information about the recommended topic usage.
### Direct use of libp2p protocols
#### Direct use of libp2p protocols
In addition to `/vac/waku/*` protocols, Waku v2 MAY directly use the following libp2p protocols:
In addition to `/vac/waku/*` protocols,
Waku MAY directly use the following libp2p protocols:
* [libp2p ping protocol](https://docs.libp2p.io/concepts/protocols/#ping) with protocol id
- [libp2p ping protocol](https://docs.libp2p.io/concepts/protocols/#ping)
with protocol id
```
```text
/ipfs/ping/1.0.0
```
for liveness checks between peers, or to keep peer-to-peer connections alive.
for liveness checks between peers, or
to keep peer-to-peer connections alive.
* [libp2p identity and identity/push](https://docs.libp2p.io/concepts/protocols/#identify) with protocol IDs
- [libp2p identity and identity/push](https://docs.libp2p.io/concepts/protocols/#identify)
with protocol IDs
```
```text
/ipfs/id/1.0.0
```
and
```
```text
/ipfs/id/push/1.0.0
```
respectively, as basic means for capability discovery.
These protocols are anyway used by the libp2p connection establishment layer Waku v2 is built on.
We plan to introduce a new Vac capability discovery protocol with better anonymity properties and more functionality.
These protocols are anyway used by the libp2p connection
establishment layer Waku is built on.
We plan to introduce a new Vac capability discovery protocol
with better anonymity properties and more functionality.
# Transports
Waku v2 is built in top of libp2p, and like libp2p it strives to be transport agnostic.
We define a set of recommended transports in order to achieve a baseline of interoperability between clients.
#### Transports
Waku is built in top of libp2p, and like libp2p it strives to be transport agnostic.
We define a set of recommended transports in order to achieve a baseline of
interoperability between clients.
This section describes these recommended transports.
Waku client implementations SHOULD support the TCP transport.
Where TCP is supported it MUST be enabled for both dialing and listening,
even if other transports are available.
Where TCP is supported it MUST be enabled for both dialing and listening, even if other transports are available.
Waku nodes running in environments that do not allow the use of TCP directly,
MAY use other transports.
Waku v2 nodes where the environment do not allow to use TCP directly, MAY use other transports.
A Waku node SHOULD support secure websockets for bidirectional communication streams,
for example in a web browser context.
A Waku v2 node SHOULD support secure websockets for bidirectional communication streams, for example in a web browser context.
A node MAY support unsecure websockets if required by the application or
running environment.
A node MAY support unsecure websockets if required by the application or running environment.
### Discovery Domain
#### Discovery Methods
### Discovery domain
#### Discovery methods
Waku v2 can retrieve a list of nodes to connect to using DNS-based discovery as per [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459).
Waku can retrieve a list of nodes to connect to using DNS-based discovery
as per [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459).
While this is a useful way of bootstrapping connection to a set of peers,
it MAY be used in conjunction with an [ambient peer discovery](https://docs.libp2p.io/concepts/publish-subscribe/#discovery) procedure to find still other nodes to connect to,
it MAY be used in conjunction with an [ambient peer discovery](https://docs.libp2p.io/concepts/publish-subscribe/#discovery)
procedure to find other nodes to connect to,
such as [Node Discovery v5](https://github.com/ethereum/devp2p/blob/8fd5f7e1c1ec496a9d8dc1640a8548b8a8b5986b/discv5/discv5.md).
More ambient peer discovery methods are being tested for Waku v2,
and will be specified for wider adoption.
It is possible to bypass the discovery domain by specifying static nodes.
#### Use of ENR
[31/WAKU2-ENR](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/enr.md) describes the usage of [EIP-778 ENR (Ethereum Node Records)](https://eips.ethereum.org/EIPS/eip-778) for Waku v2 discovery purposes.
It introduces two new ENR fields, `multiaddrs` and `waku2`, that a Waku v2 node MAY use for discovery purposes.
These fields MUST be used under certain conditions, as set out in the spec.
Both EIP-1459 DNS-based discovery and Node Discovery v5 operates on ENR,
and it's reasonable to expect even wider utility for ENR in Waku v2 networks in future.
[WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md)
describes the usage of [EIP-778 ENR (Ethereum Node Records)](https://eips.ethereum.org/EIPS/eip-778)
for Waku discovery purposes.
It introduces two new ENR fields, `multiaddrs` and
`waku2`, that a Waku node MAY use for discovery purposes.
These fields MUST be used under certain conditions, as set out in the specification.
Both EIP-1459 DNS-based discovery and Node Discovery v5 operate on ENR,
and it's reasonable to expect even wider utility for ENR in Waku networks in the future.
### Request/Reply domain
### Request/Response Domain
In addition to the Gossip domain,
Waku provides a set of Request/Reply protocols.
Waku provides a set of request/response protocols.
They are primarily used in order to get Waku to run in resource restricted environments,
such as low bandwidth or being mostly offline.
#### Historical message support
#### Historical Message Support
**Protocol identifier***: `/vac/waku/store/2.0.0-beta4`
**Protocol identifier***: `/vac/waku/store-query/3.0.0`
This is used to fetch historical messages for mostly offline devices.
See [13/WAKU2-STORE spec](../13/store.md) spec for more details.
See [13/WAKU2-STORE spec](/waku/standards/core/13/store.md) specification for more details.
There is also an experimental fault-tolerant addition to the store protocol that relaxes the high availability requirement.
See [21/WAKU2-FT-STORE](../../application/21/fault-tolerant-store.md)
There is also an experimental fault-tolerant addition to the store protocol
that relaxes the high availability requirement.
See [21/WAKU2-FAULT-TOLERANT-STORE](/waku/standards/application/21/fault-tolerant-store.md)
#### Content filtering
#### Content Filtering
**Protocol identifier***: `/vac/waku/filter/2.0.0-beta1`
This is used to make fetching of a subset of messages more bandwidth preserving.
See [12/WAKU2-FILTER](../12/filter.md) spec for more details.
This is used to preserve more bandwidth when fetching a subset of messages.
See [12/WAKU2-FILTER](/waku/standards/core/12/filter.md) specification for more details.
#### Light push
#### LightPush
**Protocol identifier***: `/vac/waku/lightpush/2.0.0-beta1`
This is used for nodes with short connection windows and limited bandwidth to publish messages into the Waku network.
See [19/WAKU2-LIGHTPUSH](../19/lightpush.md) spec for more details.
This is used for nodes with short connection windows and
limited bandwidth to publish messages into the Waku network.
See [19/WAKU2-LIGHTPUSH](/waku/standards/core/19/lightpush.md) specification for more details.
#### Other protocols
#### Other Protocols
The above is a non-exhaustive list,
and due to the modular design of Waku there may be other protocols here that provide a useful service to the Waku network.
and due to the modular design of Waku,
there may be other protocols here that provide a useful service to the Waku network.
### Overview of protocol interaction
### Overview of Protocol Interaction
See the sequence diagram below for an overview of how different protocols interact.
![Overview of how protocols interact in Waku v2.](./images/overview.png)
![Overview of how protocols interact in Waku.](./images/overview.png)
0. We have six nodes, A-F.
The protocols initially mounted are indicated as such.
The PubSub topics `pubtopic1` and `pubtopic2` is used for routing and indicates that it is subscribed to messages on that topic for relay, see [11/WAKU2-RELAY](../11/relay.md) for details.
Ditto for [13/WAKU2-STORE](../13/store.md) where it indicates that these messages are persisted on that node.
The PubSub topics `pubtopic1` and
`pubtopic2` is used for routing and
indicates that it is subscribed to messages on that topic for relay,
see [11/WAKU2-RELAY](/waku/standards/core/11/relay.md) for details.
Ditto for [13/WAKU2-STORE](/waku/standards/core/13/store.md)
where it indicates that these messages are persisted on that node.
1. Node A creates a WakuMessage `msg1` with a ContentTopic `contentTopic1`.
See [14/WAKU2-MESSAGE](../14/message.md) for more details.
If WakuMessage version is set to 1, we use the [6/WAKU1](../../legacy/6/waku1.md) compatible `data` field with encryption.
See [7/WAKU-DATA](../../legacy/7/data.md) for more details.
See [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) for more details.
If WakuMessage version is set to 1,
we use the [6/WAKU1](/waku/standards/legacy/6/waku1.md) compatible `data` field with encryption.
See [7/WAKU-DATA](/waku/standards/legacy/7/data.md) for more details.
2. Node F requests to get messages filtered by PubSub topic `pubtopic1` and ContentTopic `contentTopic1`.
Node D subscribes F to this filter and will in the future forward messages that match that filter.
See [12/WAKU2-FILTER](../12/filter.md) for more details.
2. Node F requests to get messages filtered by PubSub topic `pubtopic1` and
ContentTopic `contentTopic1`.
Node D subscribes F to this filter and
will in the future forward messages that match that filter.
See [12/WAKU2-FILTER](/waku/standards/core/12/filter.md) for more details.
3. Node A publishes `msg1` on `pubtopic1` and subscribes to that relay topic pick it up.
It then gets relayed further from B to D, but not C since it doesn't subscribe to that topic.
See [11/WAKU2-RELAY](../11/relay.md).
3. Node A publishes `msg1` on `pubtopic1` and
subscribes to that relay topic.
It then gets relayed further from B to D, but
not C since it doesn't subscribe to that topic.
See [11/WAKU2-RELAY](/waku/standards/core/11/relay.md).
4. Node D saves `msg1` for possible later retrieval by other nodes.
See [13/WAKU2-STORE](../13/store.md).
See [13/WAKU2-STORE](/waku/standards/core/13/store.md).
5. Node D also pushes `msg1` to F, as it has previously subscribed F to this filter.
See [12/WAKU2-FILTER](../12/filter.md).
5. Node D also pushes `msg1` to F,
as it has previously subscribed F to this filter.
See [12/WAKU2-FILTER](/waku/standards/core/12/filter.md).
6. At a later time, Node E comes online.
It then requests messages matching `pubtopic1` and `contentTopic1` from Node D.
Node D responds with messages meeting this (and possibly other) criteria. See [13/WAKU2-STORE](../13/store.md).
It then requests messages matching `pubtopic1` and
`contentTopic1` from Node D.
Node D responds with messages meeting this (and possibly other) criteria.
See [13/WAKU2-STORE](/waku/standards/core/13/store.md).
## Appendix A: Upgradability and Compatibility
### Compatibility with Waku v1
### Compatibility with Waku Legacy
Waku v1 and Waku v2 are different protocols all together.
They use a different transport protocol underneath; Waku v1 is devp2p RLPx based while Waku v2 uses libp2p.
[6/WAKU1](/waku/standards/legacy/6/waku1.md) and Waku are different protocols all together.
They use a different transport protocol underneath;
[6/WAKU1](/waku/standards/legacy/6/waku1.md) is devp2p RLPx based while Waku uses libp2p.
The protocols themselves also differ as does their data format.
Compatibility can be achieved only by using a bridge that not only talks both devp2p RLPx and libp2p, but that also transfers (partially) the content of a packet from one version to the other.
Compatibility can be achieved only by using a bridge
that not only talks both devp2p RLPx and libp2p,
but that also transfers (partially) the content of a packet from one version
to the other.
See [15/WAKU-BRIDGE](../15/bridge.md) for details on a bidirectional bridge mode.
See [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md) for details on a bidirectional bridge mode.
# Appendix B: Security
## Appendix B: Security
Each protocol layer of Waku v2 provides a distinct service and is associated with a separate set of security features and concerns.
Therefore, the overall security of Waku v2 depends on how the different layers are utilized.
In this section, we overview the security properties of Waku v2 protocols against a static adversarial model which is described below.
Note that a more detailed security analysis of each Waku protocol is supplied in its respective specification as well.
Each protocol layer of Waku provides a distinct service and
is associated with a separate set of security features and concerns.
Therefore, the overall security of Waku
depends on how the different layers are utilized.
In this section,
we overview the security properties of Waku protocols
against a static adversarial model which is described below.
Note that a more detailed security analysis of each Waku protocol
is supplied in its respective specification as well.
## Primary Adversarial Model
### Primary Adversarial Model
In the primary adversarial model, we consider adversary as a passive entity that attempts to collect information from others to conduct an attack,
In the primary adversarial model,
we consider adversary as a passive entity that attempts to collect information
from others to conduct an attack,
but it does so without violating protocol definitions and instructions.
The following are **not** considered as part of the adversarial model:
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs of peers
(unless the adversary is one end of the communication).
Specifically, the communication channels are assumed to be secure.
The following are **not** considered as part of the adversarial model:
## Security Features
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links
between arbitrary pairs of peers
(unless the adversary is one end of the communication).
Specifically, the communication channels are assumed to be secure.
### Pseudonymity
### Security Features
Waku v2 by default guarantees pseudonymity for all of the protocol layers since parties do not have to disclose their true identity
#### Pseudonymity
Waku by default guarantees pseudonymity for all of the protocol layers
since parties do not have to disclose their true identity
and instead they utilize libp2p `PeerID` as their identifiers.
While pseudonymity is an appealing security feature, it does not guarantee full anonymity since the actions taken under the same pseudonym
i.e., `PeerID` can be linked together and potentially result in the re-identification of the true actor.
While pseudonymity is an appealing security feature,
it does not guarantee full anonymity since the actions taken under the same pseudonym
i.e., `PeerID` can be linked together and
potentially result in the re-identification of the true actor.
### Anonymity / Unlinkability
#### Anonymity / Unlinkability
At a high level, anonymity is the inability of an adversary in linking an actor to its data/performed action (the actor and action are context-dependent).
To be precise about linkability, we use the term Personally Identifiable Information (PII) to refer to any piece of data that could potentially be used to uniquely identify a party.
For example, the signature verification key, and the hash of one's static IP address are unique for each user and hence count as PII.
Notice that users' actions can be traced through their PIIs (e.g., signatures) and hence result in their re-identification risk.
As such, we seek anonymity by avoiding linkability between actions and the actors / actors' PII. Concerning anonymity, Waku v2 provides the following features:
At a high level,
anonymity is the inability of an adversary in linking an actor
to its data/performed action (the actor and action are context-dependent).
To be precise about linkability,
we use the term Personally Identifiable Information (PII)
to refer to any piece of data that could potentially
be used to uniquely identify a party.
For example, the signature verification key, and
the hash of one's static IP address are unique for each user and
hence count as PII.
Notice that users' actions can be traced through their PIIs
(e.g., signatures) and hence result in their re-identification risk.
As such, we seek anonymity by avoiding linkability between actions and
the actors / actors' PII. Concerning anonymity, Waku provides the following features:
**Publisher-Message Unlinkability**:
This feature signifies the unlinkability of a publisher to its published messages in the 11/WAKU2-RELAY protocol.
The [Publisher-Message Unlinkability](../11/relay.md/#security-analysis) is enforced through the `StrictNoSign` policy due to which the data fields of pubsub messages that count as PII for the publisher must be left unspecified.
This feature signifies the unlinkability of a publisher
to its published messages in the 11/WAKU2-RELAY protocol.
The [Publisher-Message Unlinkability](/waku/standards/core/11/relay.md/#security-analysis)
is enforced through the `StrictNoSign` policy due to which the data fields
of pubsub messages that count as PII for the publisher must be left unspecified.
**Subscriber-Topic Unlinkability**:
This feature stands for the unlinkability of the subscriber to its subscribed topics in the 11/WAKU2-RELAY protocol.
The [Subscriber-Topic Unlinkability](../11/relay.md/#security-analysis) is achieved through the utilization of a single PubSub topic.
As such, subscribers are not re-identifiable from their subscribed topic IDs as the entire network is linked to the same topic ID.
This level of unlinkability / anonymity is known as [k-anonymity](https://www.privitar.com/blog/k-anonymity-an-introduction/) where k is proportional to the system size (number of subscribers).
Note that there is no hard limit on the number of the pubsub topics, however, the use of one topic is recommended for the sake of anonymity.
This feature stands for the unlinkability of the subscriber
to its subscribed topics in the 11/WAKU2-RELAY protocol.
The [Subscriber-Topic Unlinkability](/waku/standards/core/11/relay.md/#security-analysis)
is achieved through the utilization of a single PubSub topic.
As such, subscribers are not re-identifiable from their subscribed topic IDs
as the entire network is linked to the same topic ID.
This level of unlinkability / anonymity is known as [k-anonymity](https://www.privitar.com/blog/k-anonymity-an-introduction/)
where k is proportional to the system size (number of subscribers).
Note that there is no hard limit on the number of the pubsub topics, however,
the use of one topic is recommended for the sake of anonymity.
### Spam protection
#### Spam protection
This property indicates that no adversary can flood the system (i.e., publishing a large number of messages in a short amount of time), either accidentally or deliberately, with any kind of message i.e. even if the message content is valid or useful.
Spam protection is partly provided in `11/WAKU2-RELAY` through the [scoring mechanism](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#spam-protection-measures) provided for by GossipSub v1.1.
At a high level, peers utilize a scoring function to locally score the behavior of their connections and remove peers with a low score.
This property indicates that no adversary can flood the system
(i.e., publishing a large number of messages in a short amount of time),
either accidentally or deliberately, with any kind of message
i.e. even if the message content is valid or useful.
Spam protection is partly provided in `11/WAKU2-RELAY`
through the [scoring mechanism](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#spam-protection-measures)
provided for by GossipSub v1.1.
At a high level,
peers utilize a scoring function to locally score the behavior
of their connections and remove peers with a low score.
### Data confidentiality, Integrity, and Authenticity
#### Data confidentiality, Integrity, and Authenticity
Confidentiality can be addressed through data encryption whereas integrity and authenticity are achievable through digital signatures.
These features are provided for in [14/WAKU2-MESSAGE (version 1)](../14/message.md/#version-1)` through payload encryption as well as encrypted signatures.
Confidentiality can be addressed through data encryption whereas integrity and
authenticity are achievable through digital signatures.
These features are provided for in [14/WAKU2-MESSAGE (version 1)](/waku/standards/core/14/message.md/#version-1)`
through payload encryption as well as encrypted signatures.
## Security Considerations
### Security Considerations
**Lack of anonymity/unlinkability in the protocols involving direct connections including `13/WAKU2-STORE` and `12/WAKU2-FILTER` protocols**:
The anonymity/unlinkability is not guaranteed in the protocols like `13/WAKU2-STORE` and `12/WAKU2-FILTER` where peers need to have direct connections to benefit from the designated service.
This is because during the direct connections peers utilize `PeerID` to identify each other,
therefore the service obtained in the protocol is linkable to the beneficiary's `PeerID` (which counts as PII).
For `13/WAKU2-STORE`, the queried node would be able to link the querying node's `PeerID` to its queried topics.
Likewise, in the `12/WAKU2-FILTER`, a full node can link the light node's `PeerID`s to its content filter.
Lack of anonymity/unlinkability in the protocols involving direct connections
including `13/WAKU2-STORE` and `12/WAKU2-FILTER` protocols:
<!-- TODO: to inspect the nim-libp2p codebase and figure out the exact use of PeerIDs in direct communication, it might be the case that the requester does not have to disclose its PeerID-->
The anonymity/unlinkability is not guaranteed in the protocols like `13/WAKU2-STORE`
and `12/WAKU2-FILTER` where peers need to have direct connections
to benefit from the designated service.
This is because during the direct connections peers utilize `PeerID`
to identify each other,
therefore the service obtained in the protocol is linkable
to the beneficiary's `PeerID` (which counts as PII).
For `13/WAKU2-STORE`,
the queried node would be able to link the querying node's `PeerID`
to its queried topics.
Likewise, in the `12/WAKU2-FILTER`,
a full node can link the light node's `PeerID`s to its content filter.
<!--TODO: might be good to add a figure visualizing the Waku protocol stack and the security features of each layer-->
<!-- TODO: to inspect the nim-libp2p codebase and
figure out the exact use of PeerIDs in direct communication,
it might be the case that the requester does not have to disclose its PeerID-->
<!--TODO: might be good to add a figure visualizing the Waku protocol stack and
the security features of each layer-->
## Appendix C: Implementation Notes
### Implementation Matrix
There are multiple implementations of Waku v2 and its protocols:
There are multiple implementations of Waku and its protocols:
- [nim-waku (Nim)](https://github.com/status-im/nim-waku/)
- [go-waku (Go)](https://github.com/status-im/go-waku/)
- [js-waku (NodeJS and Browser)](https://github.com/status-im/js-waku/)
Below you can find an overview of the specs that they implement as they relate to Waku v2.
This includes Waku v1 specs, as they are used for bridging between the two networks.
Below you can find an overview of the specifications that they implement
as they relate to Waku.
This includes Waku legacy specifications, as they are used for bridging between the two networks.
| Spec | nim-waku (Nim) | go-waku (Go) | js-waku (Node JS) | js-waku (Browser JS) |
| ---- | -------------- | ------------ | ----------------- | -------------------- |
|[6/WAKU1](../../legacy/6/waku1.md)|✔|||
|[7/WAKU-DATA](../../legacy/7/data.md)|✔|✔||
|[8/WAKU-MAIL](../../legacy/8/mail.md)|✔|||
|[9/WAKU-RPC](../../legacy/9/rpc.md)|✔|||
|[10/WAKU2](../10/waku2.md)|✔|🚧|🚧|🚧|
|[11/WAKU2-RELAY](../11/relay.md)|✔|✔|✔|✔|
|[12/WAKU2-FILTER](../12/filter.md)|✔|✔||
|[13/WAKU2-STORE](../13/store.md)|✔|✔|✔\*|✔\*|
|[14/WAKU2-MESSAGE](../14/message.md))|✔|✔|✔|✔|
|[15/WAKU2-BRIDGE](../15/bridge.md)|✔|||
|[16/WAKU2-RPC](../16/rpc.md)|✔|||
|[17/WAKU2-RLN-RELAY](../17/rln-relay.md)|🚧|||
|[18/WAKU2-SWAP](../../application/18/swap.md)|🚧|||
|[19/WAKU2-LIGHTPUSH](../19/lightpush.md)|✔|✔|✔\**|✔\**|
|[21/WAKU2-FAULT-TOLERANT-STORE](../../application/21/fault-tolerant-store.md)|✔|✔||
|[6/WAKU1](/waku/standards/legacy/6/waku1.md)|✔||||
|[7/WAKU-DATA](/waku/standards/legacy/7/data.md)|✔|✔|||
|[8/WAKU-MAIL](/waku/standards/legacy/8/mail.md)|✔||||
|[9/WAKU-RPC](/waku/standards/legacy/9/rpc.md)|✔||||
|[10/WAKU2](/waku/standards/core/10/waku2.md)|✔|🚧|🚧||
|[11/WAKU2-RELAY](/waku/standards/core/11/relay.md)|✔|✔|✔|✔|
|[12/WAKU2-FILTER](/waku/standards/core/12/filter.md)|✔|✔|||
|[13/WAKU2-STORE](/waku/standards/core/13/store.md)|✔|✔|✔\*|✔\*|
|[14/WAKU2-MESSAGE](/waku/standards/core/14/message.md))|✔|✔|✔|✔|
|[15/WAKU2-BRIDGE](/waku/standards/core/15/bridge.md)|✔||||
|[16/WAKU2-RPC](/waku/deprecated/16/rpc.md)|✔||||
|[17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md)|🚧||||
|[18/WAKU2-SWAP](/waku/standards/application/18/swap.md)|🚧||||
|[19/WAKU2-LIGHTPUSH](/waku/standards/core/19/lightpush.md)|✔|✔|✔\**|✔\**|
|[21/WAKU2-FAULT-TOLERANT-STORE](/waku/standards/application/21/fault-tolerant-store.md)|✔|✔|||
*js-waku implements [13/WAKU2-STORE](../13/store.md) as a querying node only.
**js-waku only implements [19/WAKU2-LIGHTPUSH](../19/lightpush.md) requests.
*js-waku implements [13/WAKU2-STORE](/waku/standards/core/13/store.md) as a querying node only.
**js-waku only implements [19/WAKU2-LIGHTPUSH](/waku/standards/core/19/lightpush.md) requests.
### Recommendations for Clients
### Recommendations for clients
To implement a minimal Waku client,
we recommend implementing the following subset in the following order:
To implement a minimal Waku v2 client, we recommend implementing the following subset in the following order:
- [10/WAKU2](/waku/standards/core/10/waku2.md) - this specification
- [11/WAKU2-RELAY](/waku/standards/core/11/relay.md) - for basic operation
- [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) - version 0 (unencrypted)
- [13/WAKU2-STORE](/waku/standards/core/13/store.md) - for historical messaging (query mode only)
- [10/WAKU2](../10/waku2.md) - this spec
- [11/WAKU2-RELAY](../11/relay.md) - for basic operation
- [14/WAKU2-MESSAGE](../14/message.md) - version 0 (unencrypted)
- [13/WAKU2-STORE](../13/store.md) - for historical messaging (query mode only)
To get compatibility with Waku Legacy:
To get compatibility with Waku v1:
- [7/WAKU-DATA](../../legacy/7/data.md)
- [14/WAKU2-MESSAGE](../14/message.md) - version 1 (encrypted with `7/WAKU-DATA`)
- [7/WAKU-DATA](/waku/standards/legacy/7/data.md)
- [14/WAKU2-MESSAGE](/waku/standards/14/message.md) - version 1 (encrypted with `7/WAKU-DATA`)
For an interoperable keep-alive mechanism:
@@ -404,23 +513,33 @@ with periodic pings to connected peers
## Appendix D: Future work
The following features are currently experimental and under research and initial implementation:
The following features are currently experimental,
under research and initial implementations:
**Economic Spam resistance**:
We aim to enable an incentivized spam protection technique to enhance `11/WAKU2-RELAY` by using rate limiting nullifiers.
More details on this can be found in [17/WAKU2-RLN-RELAY](../17/rln-relay.md).
In this advanced method, peers are limited to a certain rate of messaging per epoch and an immediate financial penalty is enforced for spammers who break this rate.
**Economic Spam Resistance**:
We aim to enable an incentivized spam protection technique
to enhance `11/WAKU2-RELAY` by using rate limiting nullifiers.
More details on this can be found in [17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md).
In this advanced method,
peers are limited to a certain rate of messaging per epoch and
an immediate financial penalty is enforced for spammers who break this rate.
**Prevention of Denial of Service (DoS) and Node Incentivization**:
Denial of service signifies the case where an adversarial node exhausts another node's service capacity (e.g., by making a large number of requests) and makes it unavailable to the rest of the system.
DoS attack is to be mitigated through the accounting model as described in [18/WAKU2-SWAP](../../application/18/swap.md).
Denial of service signifies the case where an adversarial node
exhausts another node's service capacity (e.g., by making a large number of requests)
and makes it unavailable to the rest of the system.
DoS attack is to be mitigated through the accounting model as described in [18/WAKU2-SWAP](/waku/deprecated/18/swap.md).
In a nutshell, peers have to pay for the service they obtain from each other.
In addition to incentivizing the service provider, accounting also makes DoS attacks costly for malicious peers.
The accounting model can be used in `13/WAKU2-STORE` and `12/WAKU2-FILTER` to protect against DoS attacks.
Additionally, this gives node operators who provide a useful service to the network an incentive to perform that service.
See [18/WAKU2-SWAP](../../application/18/swap.md) for more details on this piece of work.
In addition to incentivizing the service provider,
accounting also makes DoS attacks costly for malicious peers.
The accounting model can be used in `13/WAKU2-STORE` and
`12/WAKU2-FILTER` to protect against DoS attacks.
Additionally, this gives node operators who provide a useful service to the network
an incentive to perform that service.
See [18/WAKU2-SWAP](/waku/deprecated/18/swap.md)
for more details on this piece of work.
## Copyright
@@ -430,31 +549,31 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
1. [libp2p specs](https://github.com/libp2p/specs)
2. [6/WAKU1](../../legacy/6/waku1.md)
2. [6/WAKU1](/waku/standards/legacy/6/waku1.md)
3. [Whisper spec (EIP627)](https://eips.ethereum.org/EIPS/eip-627)
4. [Waku v2 plan](https://vac.dev/waku-v2-plan)
5. [30/ADAPTIVE-NODES](../../../informational/30/adaptive-nodes.md)
5. [30/ADAPTIVE-NODES](/waku/informational/30/adaptive-nodes.md)
6. [Protocol Identifiers](https://docs.libp2p.io/concepts/protocols/)
7. [14/WAKU2-MESSAGE](../14/message.md)
7. [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)
8. [26/WAKU-PAYLOAD](../../application/26/payload.md)
8. [26/WAKU-PAYLOAD](/waku/standards/application/26/payload.md)
9. [23/WAKU2-TOPICS](../../../informational/23/topics.md)
9. [23/WAKU2-TOPICS](/waku/informational/23/topics.md)
10. [27/WAKU2-PEERS](../../../informational/27/peers.md)
10. [27/WAKU2-PEERS](/waku/informational/27/peers.md)
11. [bi-directional binary stream](https://docs.libp2p.io/concepts/protocols/)
11. [bi-directional binary stream](https://docs.libp2p.io/concepts/protocols/)
12. [Protobuf varint encoding](https://developers.google.com/protocol-buffers/docs/encoding#varints)
13. [11/WAKU2-RELAY spec](../11/relay.md)
13. [11/WAKU2-RELAY spec](/waku/standards/core/11/relay.md)
14. [17/WAKU2-RLN-RELAY](../17/rln-relay.md)
14. [17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md)
15. [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459)
@@ -462,21 +581,21 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
17. [Node Discovery v5](https://github.com/ethereum/devp2p/blob/8fd5f7e1c1ec496a9d8dc1640a8548b8a8b5986b/discv5/discv5.md)
18. [31/WAKU2-ENR](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/enr.md)
18. [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md)
19. [EIP-778 ENR (Ethereum Node Records)](https://eips.ethereum.org/EIPS/eip-778)
20. [13/WAKU2-STORE spec](../13/store.md)
20. [13/WAKU2-STORE spec](/waku/standards/core/13/store.md)
21. [21/WAKU2-FT-STORE](../../application/21/ft-store.md)
21. [21/WAKU2-FT-STORE](/waku/standards/application/21/ft-store.md)
22. [12/WAKU2-FILTER](../12/filter.md)
22. [12/WAKU2-FILTER](/waku/standards/core/12/filter.md)
23. [19/WAKU2-LIGHTPUSH](../19/lightpush.md)
23. [19/WAKU2-LIGHTPUSH](/waku/standards/core/19/lightpush.md)
24. [7/WAKU-DATA](../../legacy/7/data.md)
24. [7/WAKU-DATA](/waku/standards/legacy/7/data.md)
25. [15/WAKU-BRIDGE](../15/bridge.md)
25. [15/WAKU-BRIDGE](/waku/standards/core/15/bridge.md)
26. [k-anonymity](https://www.privitar.com/blog/k-anonymity-an-introduction/)
@@ -488,13 +607,12 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
30. [js-waku (NodeJS and Browser)](https://github.com/status-im/js-waku/)
31. [8/WAKU-MAIL](../../legacy/8/mail.md)
31. [8/WAKU-MAIL](/waku/standards/legacy/8/mail.md)
32. [9/WAKU-RPC](../../legacy/9/rpc.md)
32. [9/WAKU-RPC](/waku/standards/legacy/9/rpc.md)
33. [16/WAKU2-RPC](../16/rpc.md)
33. [16/WAKU2-RPC](waku/deprecated/16/rpc.md)
34. [18/WAKU2-SWAP spec](../../application/18/swap.md)
34. [18/WAKU2-SWAP spec](waku/deprecated/18/swap.md)
35. [21/WAKU2-FAULT-TOLERANT-STORE](../../application/21/fault-tolerant-store.md)

View File

@@ -10,53 +10,84 @@ contributors:
- Sanaz Taheri <sanaz@status.im>
---
`11/WAKU2-RELAY` specifies a [Publish/Subscribe approach](https://docs.libp2p.io/concepts/publish-subscribe/) to peer-to-peer messaging with a strong focus on privacy, censorship-resistance, security and scalability.
Its current implementation is a minor extension of the [libp2p GossipSub protocol](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md) and prescribes gossip-based dissemination.
As such the scope is limited to defining a separate [`protocol id`](https://github.com/libp2p/specs/blob/master/connections/README.md#protocol-negotiation) for `11/WAKU2-RELAY`, establishing privacy and security requirements, and defining how the underlying GossipSub is to be interpreted and implemented within the Waku and cryptoeconomic domain.
`11/WAKU2-RELAY` specifies a [Publish/Subscribe approach](https://docs.libp2p.io/concepts/publish-subscribe/)
to peer-to-peer messaging with a strong focus on privacy,
censorship-resistance, security and scalability.
Its current implementation is a minor extension of the
[libp2p GossipSub protocol](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md)
and prescribes gossip-based dissemination.
As such the scope is limited to defining a separate
[`protocol id`](https://github.com/libp2p/specs/blob/master/connections/README.md#protocol-negotiation)
for `11/WAKU2-RELAY`, establishing privacy and security requirements,
and defining how the underlying GossipSub is to be interpreted and
implemented within the Waku and cryptoeconomic domain.
`11/WAKU2-RELAY` should not be confused with [libp2p circuit relay](https://github.com/libp2p/specs/tree/master/relay).
**Protocol identifier**: `/vac/waku/relay/2.0.0`
## Security Requirements
The `11/WAKU2-RELAY` protocol is designed to provide the following security properties under a static [Adversarial Model](#adversarial-model).
Note that data confidentiality, integrity, and authenticity are currently considered out of scope for `11/WAKU2-RELAY` and must be handled by higher layer protocols such as [`14/WAKU2-MESSAGE`](../14/message.md).
The `11/WAKU2-RELAY` protocol is designed to provide the following security properties
under a static [Adversarial Model](#adversarial-model).
Note that data confidentiality, integrity, and
authenticity are currently considered out of scope for `11/WAKU2-RELAY` and
must be handled by higher layer protocols such as [`14/WAKU2-MESSAGE`](../14/message.md).
<!-- May add the definition of the unsupported feature:
Confidentiality indicates that an adversary should not be able to learn the data carried by the `WakuRelay` protocol.
Integrity indicates that the data transferred by the `WakuRelay` protocol can not be tampered with by an adversarial entity without being detected.
Authenticity no adversary can forge data on behalf of a targeted publisher and make it accepted by other subscribers as if the origin is the target. -->
Confidentiality indicates that an adversary
should not be able to learn the data carried by the `WakuRelay` protocol.
Integrity indicates that the data transferred by the `WakuRelay` protocol
can not be tampered with by an adversarial entity without being detected.
Authenticity no adversary can forge data on behalf of a targeted publisher and
make it accepted by other subscribers as if the origin is the target. -->
- **Publisher-Message Unlinkability**:
This property indicates that no adversarial entity can link a published `Message` to its publisher.
This feature also implies the unlinkability of the publisher to its published topic ID as the `Message` embodies the topic IDs.
This property indicates that no adversarial entity can link a published `Message`
to its publisher.
This feature also implies the unlinkability of the publisher
to its published topic ID as the `Message` embodies the topic IDs.
- **Subscriber-Topic Unlinkability**:
This feature stands for the inability of any adversarial entity from linking a subscriber to its subscribed topic IDs.
This feature stands for the inability of any adversarial entity
from linking a subscriber to its subscribed topic IDs.
<!-- TODO: more requirements can be added, but that needs further and deeper investigation-->
<!-- TODO: more requirements can be added,
but that needs further and deeper investigation-->
### Terminology
_Personally identifiable information_ (PII) refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key, and the hash of one's static IP address are unique for each user and hence count as PII.
_Personally identifiable information_ (PII)
refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key,
and the hash of one's static IP address are unique for each user and
hence count as PII.
## Adversarial Model
- Any entity running the `11/WAKU2-RELAY` protocol is considered an adversary.
This includes publishers, subscribers, and all the peers' direct connections.
Furthermore, we consider the adversary as a passive entity that attempts to collect information from others to conduct an attack but it does so without violating protocol definitions and instructions.
For example, under the passive adversarial model, no malicious subscriber hides the messages it receives from other subscribers as it is against the description of `11/WAKU2-RELAY`.
However, a malicious subscriber may learn which topics are subscribed to by which peers.
- The following are **not** considered as part of the adversarial model:
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs of peers (unless the adversary is one end of the communication).
- Any entity running the `11/WAKU2-RELAY` protocol is considered an adversary.
This includes publishers, subscribers, and all the peers' direct connections.
Furthermore,
we consider the adversary as a passive entity that attempts to collect information
from others to conduct an attack but
it does so without violating protocol definitions and instructions.
For example, under the passive adversarial model,
no malicious subscriber hides the messages it receives from other subscribers
as it is against the description of `11/WAKU2-RELAY`.
However,
a malicious subscriber may learn which topics are subscribed to by which peers.
- The following are **not** considered as part of the adversarial model:
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs
of peers (unless the adversary is one end of the communication).
In other words, the communication channels are assumed to be secure.
## Wire Specification
The [PubSub interface specification](https://github.com/libp2p/specs/blob/master/pubsub/README.md) defines the protobuf RPC messages exchanged between peers participating in a GossipSub network.
We republish these messages here for ease of reference and define how `11/WAKU2-RELAY` uses and interprets each field.
The [PubSub interface specification](https://github.com/libp2p/specs/blob/master/pubsub/README.md)
defines the protobuf RPC messages
exchanged between peers participating in a GossipSub network.
We republish these messages here for ease of reference and
define how `11/WAKU2-RELAY` uses and interprets each field.
### Protobuf definitions
@@ -86,10 +117,11 @@ message RPC {
```
> **_NOTE:_**
The various [control messages](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md#control-messages) defined for GossipSub are used as specified there.
The various [control messages](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md#control-messages)
defined for GossipSub are used as specified there.
> **_NOTE:_**
The [`TopicDescriptor`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor) is not currently used by `11/WAKU2-RELAY`.
The [`TopicDescriptor`](https://github.com/libp2p/specs/blob/master/pubsub/README.md#the-topic-descriptor)
is not currently used by `11/WAKU2-RELAY`.
### Message fields
@@ -103,19 +135,25 @@ See [`14/WAKU2-MESSAGE`](../14/message.md) for more details.
- The `seqno` field MUST NOT be used, following the [`StrictNoSign` signature policy](#signature-policy).
- The `topicIDs` field MUST contain the content-topics that a message is being published on.
- The `topicIDs` field MUST contain the content-topics
that a message is being published on.
- The `signature` field MUST NOT be used, following the [`StrictNoSign` signature policy](#signature-policy).
- The `signature` field MUST NOT be used,
following the [`StrictNoSign` signature policy](#signature-policy).
- The `key` field MUST NOT be used, following the [`StrictNoSign` signature policy](#signature-solicy).
- The `key` field MUST NOT be used,
following the [`StrictNoSign` signature policy](#signature-policy).
### SubOpts fields
The `SubOpts` protobuf defines the format in which subscription options are relayed between peers.
A `11/WAKU2-RELAY` node MAY decide to subscribe or unsubscribe from topics by sending updates using `SubOpts`.
The `SubOpts` protobuf defines the format
in which subscription options are relayed between peers.
A `11/WAKU2-RELAY` node MAY decide to subscribe or
unsubscribe from topics by sending updates using `SubOpts`.
The following usage requirements apply:
- The `subscribe` field MUST contain a boolean, where `true` indicates subscribe and `false` indicates unsubscribe to a topic.
- The `subscribe` field MUST contain a boolean,
where `true` indicates subscribe and `false` indicates unsubscribe to a topic.
- The `topicid` field MUST contain the pubsub topic.
@@ -124,44 +162,93 @@ The following usage requirements apply:
### Signature Policy
The [`StrictNoSign` option](https://github.com/libp2p/specs/blob/master/pubsub/README.md#signature-policy-options) MUST be used, to ensure that messages are built without the `signature`, `key`, `from` and `seqno` fields.
Note that this does not merely imply that these fields be empty, but that they MUST be _absent_ from the marshalled message.
The [`StrictNoSign` option](https://github.com/libp2p/specs/blob/master/pubsub/README.md#signature-policy-options)
MUST be used, to ensure that messages are built without the `signature`,
`key`, `from` and `seqno` fields.
Note that this does not merely imply that these fields be empty, but
that they MUST be _absent_ from the marshalled message.
## Security Analysis
<!-- TODO: realized that the prime security objective of the `WakuRelay` protocol is to provide peers unlinkability as such this feature is prioritized over other features e.g., unlinkability is preferred over authenticity and integrity. It might be good to motivate unlinkability and its impact on the relay protocol or other protocols invoking relay protocol.-->
<!-- TODO: realized that the prime security objective of the `WakuRelay`
protocol is to provide peers unlinkability
as such this feature is prioritized over other features
e.g., unlinkability is preferred over authenticity and integrity.
It might be good to motivate unlinkability and
its impact on the relay protocol or other protocols invoking relay protocol.-->
- **Publisher-Message Unlinkability**:
To address publisher-message unlinkability, one should remove any PII from the published message.
As such, `11/WAKU2-RELAY` follows the `StrictNoSign` policy as described in [libp2p PubSub specs](https://github.com/libp2p/specs/tree/master/pubsub#message-signing).
As the result of the `StrictNoSign` policy, `Message`s should be built without the `from`, `signature` and `key` fields since each of these three fields individually counts as PII for the author of the message (one can link the creation of the message with libp2p peerId and thus indirectly with the IP address of the publisher).
Note that removing identifiable information from messages cannot lead to perfect unlinkability.
The direct connections of a publisher might be able to figure out which `Message`s belong to that publisher by analyzing its traffic.
The possibility of such inference may get higher when the `data` field is also not encrypted by the upper-level protocols. <!-- TODO: more investigation on traffic analysis attacks and their success probability-->
To address publisher-message unlinkability,
one should remove any PII from the published message.
As such, `11/WAKU2-RELAY` follows the `StrictNoSign` policy as described in
[libp2p PubSub specs](https://github.com/libp2p/specs/tree/master/pubsub#message-signing).
As the result of the `StrictNoSign` policy,
`Message`s should be built without the `from`,
`signature` and `key` fields since each of these three fields individually
counts as PII for the author of the message
(one can link the creation of the message with libp2p peerId and
thus indirectly with the IP address of the publisher).
Note that removing identifiable information from messages
cannot lead to perfect unlinkability.
The direct connections of a publisher
might be able to figure out which `Message`s belong to that publisher
by analyzing its traffic.
The possibility of such inference may get higher
when the `data` field is also not encrypted by the upper-level protocols.
<!-- TODO: more investigation on traffic analysis attacks and their success probability-->
- **Subscriber-Topic Unlinkability:**
To preserve subscriber-topic unlinkability, it is recommended by [`10/WAKU2`](../10/waku2.md) to use a single PubSub topic in the `11/WAKU2-RELAY` protocol.
This allows an immediate subscriber-topic unlinkability where subscribers are not re-identifiable from their subscribed topic IDs as the entire network is linked to the same topic ID.
This level of unlinkability / anonymity is known as [k-anonymity](https://www.privitar.com/blog/k-anonymity-an-introduction/) where k is proportional to the system size (number of participants of Waku relay protocol).
To preserve subscriber-topic unlinkability,
it is recommended by [`10/WAKU2`](../10/waku2.md) to use a single PubSub topic
in the `11/WAKU2-RELAY` protocol.
This allows an immediate subscriber-topic unlinkability
where subscribers are not re-identifiable from their subscribed topic IDs
as the entire network is linked to the same topic ID.
This level of unlinkability / anonymity
is known as [k-anonymity](https://www.privitar.com/blog/k-anonymity-an-introduction/)
where k is proportional to the system size
(number of participants of Waku relay protocol).
However, note that `11/WAKU2-RELAY` supports the use of more than one topic.
In case that more than one topic id is utilized, preserving unlinkability is the responsibility of the upper-level protocols which MAY adopt [partitioned topics technique](https://specs.status.im/spec/10#partitioned-topic) to achieve K-anonymity for the subscribed peers.
In case that more than one topic id is utilized,
preserving unlinkability is the responsibility of the upper-level protocols
which MAY adopt
[partitioned topics technique](https://specs.status.im/spec/10#partitioned-topic)
to achieve K-anonymity for the subscribed peers.
## Future work
- **Economic spam resistance**:
In the spam-protected `11/WAKU2-RELAY` protocol, no adversary can flood the system with spam messages (i.e., publishing a large number of messages in a short amount of time).
In the spam-protected `11/WAKU2-RELAY` protocol,
no adversary can flood the system with spam messages
(i.e., publishing a large number of messages in a short amount of time).
Spam protection is partly provided by GossipSub v1.1 through [scoring mechanism](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#spam-protection-measures).
At a high level, peers utilize a scoring function to locally score the behavior of their connections and remove peers with a low score.
`11/WAKU2-RELAY` aims at enabling an advanced spam protection mechanism with economic disincentives by utilizing Rate Limiting Nullifiers.
In a nutshell, peers must conform to a certain message publishing rate per a system-defined epoch, otherwise, they get financially penalized for exceeding the rate.
More details on this new technique can be found in [`17/WAKU2-RLN-RELAY`](../17/rln-relay.md).
<!-- TODO havn't checked if all the measures in libp2p GossipSub v1.1 are taken in the nim-libp2p as well, may need to audit the code -->
At a high level,
peers utilize a scoring function to locally score the behavior of their connections
and remove peers with a low score.
`11/WAKU2-RELAY` aims at enabling an advanced spam protection mechanism
with economic disincentives by utilizing Rate Limiting Nullifiers.
In a nutshell,
peers must conform to a certain message publishing rate per a system-defined epoch,
otherwise, they get financially penalized for exceeding the rate.
More details on this new technique can be found in [`17/WAKU2-RLN-RELAY`](../17/rln-relay.md).
<!-- TODO havn't checked if all the measures in libp2p GossipSub v1.1
are taken in the nim-libp2p as well, may need to audit the code -->
- Providing **Unlinkability**, **Integrity** and **Authenticity** simultaneously:
Integrity and authenticity are typically addressed through digital signatures and Message Authentication Code (MAC) schemes, however, the usage of digital signatures (where each signature is bound to a particular peer) contradicts with the unlinkability requirement (messages signed under a certain signature key are verifiable by a verification key that is bound to a particular publisher).
As such, integrity and authenticity are missing features in `11/WAKU2-RELAY` in the interest of unlinkability.
In future work, advanced signature schemes like group signatures can be utilized to enable authenticity, integrity, and unlinkability simultaneously.
In a group signature scheme, a member of a group can anonymously sign a message on behalf of the group as such the true signer is indistinguishable from other group members. <!-- TODO: shall I add a reference for group signatures?-->
Integrity and authenticity are typically addressed through digital signatures and
Message Authentication Code (MAC) schemes, however,
the usage of digital signatures (where each signature is bound to a particular peer)
contradicts with the unlinkability requirement
(messages signed under a certain signature key are verifiable by a verification key
that is bound to a particular publisher).
As such, integrity and authenticity are missing features in `11/WAKU2-RELAY`
in the interest of unlinkability.
In future work, advanced signature schemes like group signatures
can be utilized to enable authenticity, integrity, and unlinkability simultaneously.
In a group signature scheme, a member of a group can anonymously sign a message
on behalf of the group as such the true signer
is indistinguishable from other group members.
<!-- TODO: shall I add a reference for group signatures?-->
## Copyright

View File

@@ -13,66 +13,67 @@ contributors:
- Ebube Ud <ebube@status.im>
---
previous versions: [00](./previous-versions00)
---
`WakuFilter` is a protocol that enables subscribing to messages that a peer receives. This is a more lightweight version of `WakuRelay` specifically designed for bandwidth restricted devices. This is due to the fact that light nodes subscribe to full-nodes and only receive the messages they desire.
## Content filtering
previous versions: [00](/waku/standards/core/12/previous-versions/00/filter.md)
**Protocol identifiers**:
- _filter-subscribe_: `/vac/waku/filter-subscribe/2.0.0-beta1`
- _filter-push_: `/vac/waku/filter-push/2.0.0-beta1`
Content filtering is a way to do [message-based
filtering](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_filtering).
Currently the only content filter being applied is on `contentTopic`. This
corresponds to topics in Waku v1.
---
## Rationale
## Abstract
Unlike the `store` protocol for historical messages, this protocol allows for
native lower latency scenarios such as instant messaging. It is thus
complementary to it.
This specification describes the `12/WAKU2-FILTER` protocol,
which enables a client to subscribe to a subset of real-time messages from a Waku peer.
This is a more lightweight version of [11/WAKU2-RELAY](/waku/standards/core/11/relay.md),
useful for bandwidth restricted devices.
This is often used by nodes with lower resource limits to subscribe to full Relay nodes and
only receive the subset of messages they desire,
based on content topic interest.
Strictly speaking, it is not just doing basic request response, but performs
sender push based on receiver intent. While this can be seen as a form of light
pub/sub, it is only used between two nodes in a direct fashion. Unlike the
Gossip domain, this is meant for light nodes which put a premium on bandwidth.
## Motivation
Unlike the [13/WAKU2-STORE](/waku/standards/core/13/store.md) protocol
for historical messages, this protocol allows for native lower latency scenarios,
such as instant messaging.
It is thus complementary to it.
Strictly speaking, it is not just doing basic request-response, but
performs sender push based on receiver intent.
While this can be seen as a form of light publish/subscribe,
it is only used between two nodes in a direct fashion. Unlike the
Gossip domain, this is suitable for light nodes which put a premium on bandwidth.
No gossiping takes place.
It is worth noting that a light node could get by with only using the `store`
protocol to query for a recent time window, provided it is acceptable to do
frequent polling.
It is worth noting that a light node could get by with only using the
[13/WAKU2-STORE](/waku/standards/core/13/store.md) protocol to
query for a recent time window, provided it is acceptable to do frequent polling.
## Semantics
## Design Requirements
The key words “MUST”, “MUST NOT”, “REQUIRED”,
“SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
The effectiveness and reliability of the content filtering service enabled by `WakuFilter` protocol rely on the *high availability* of the full nodes as the service providers. To this end, full nodes must feature *high uptime* (to persistently listen and capture the network messages) as well as *high Bandwidth* (to provide timely message delivery to the light nodes).
### Content filtering
## Security Consideration
Note that while using `WakuFilter` allows light nodes to save bandwidth, it comes with a privacy cost in the sense that they need to disclose their liking topics to the full nodes to retrieve the relevant messages. Currently, anonymous subscription is not supported by the `WakuFilter`, however, potential solutions in this regard are sketched below in [Future Work](#future-work) section.
Content filtering is a way to do
[message-based filtering](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_filtering).
Currently the only content filter being applied is on `contentTopic`.
### Terminology
The term Personally identifiable information (PII) refers to any piece of data that can be used to uniquely identify a user. For example, the signature verification key, and the hash of one's static IP address are unique for each user and hence count as PII.
## Adversarial Model
Any node running the `WakuFilter` protocol i.e., both the subscriber node and the queried node are considered as an adversary. Furthermore, we consider the adversary as a passive entity that attempts to collect information from other nodes to conduct an attack but it does so without violating protocol definitions and instructions. For example, under the passive adversarial model, no malicious node intentionally hides the messages matching to one's subscribed content filter as it is against the description of the `WakuFilter` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the nodes and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs of nodes (unless the adversary is one end of the communication). In specific, the communication channels are assumed to be secure.
The term Personally identifiable information (PII)
refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key, and
the hash of one's static IP address are unique for each user and hence count as PII.
### Protobuf
```protobuf
syntax = "proto3";
// 12/WAKU2-FILTER rfc: https://rfc.vac.dev/spec/12/
package waku.filter.v2;
// Protocol identifier: /vac/waku/filter-subscribe/2.0.0-beta1
message FilterSubscribeRequest {
enum FilterSubscribeType {
@@ -106,13 +107,15 @@ message MessagePush {
### Filter-Subscribe
A filter service node MUST support the _filter-subscribe_ protocol
to allow filter clients to subscribe, modify, refresh and unsubscribe a desired set of filter criteria.
The combination of different filter criteria for a specific filter client node is termed a "subscription".
A filter client is interested in receiving messages matching the filter criteria in its registered subscriptions.
to allow filter clients to subscribe, modify, refresh and
unsubscribe a desired set of filter criteria.
The combination of different filter criteria
for a specific filter client node is termed a "subscription".
A filter client is interested in receiving messages matching the filter criteria
in its registered subscriptions.
Since a filter service node is consuming resources to provide this service,
it MAY account for usage and adapt its service provision to certain clients.
An incentive mechanism is currently planned but underspecified.
#### Filter Subscribe Request
@@ -123,92 +126,118 @@ Each request MUST include a `filter_subscribe_type`, indicating the type of requ
#### Filter Subscribe Response
In return to any `FilterSubscribeRequest`,
a filter service node SHOULD respond with a `FilterSubscribeResponse` with a `requestId` matching that of the request.
This response MUST contain a `status_code` indicating if the request was successful or not.
When responding to a `FilterSubscribeRequest`,
a filter service node SHOULD send a `FilterSubscribeResponse`
with a `requestId` matching that of the request.
This response MUST contain a `status_code` indicating if the request was successful
or not.
Successful status codes are in the `2xx` range.
Client nodes SHOULD consider all other status codes as error codes and assume that the requested operation had failed.
In addition, the filter service node MAY choose to provide a more detailed status description in the `status_desc` field.
Client nodes SHOULD consider all other status codes as error codes and
assume that the requested operation had failed.
In addition,
the filter service node MAY choose to provide a more detailed status description
in the `status_desc` field.
#### Filter matching
In the description of each request type below,
the term "filter criteria" refers to the combination of `pubsub_topic` and a set of `content_topics`.
The request MAY include filter criteria, conditional to the selected `filter_subscribe_type`.
the term "filter criteria" refers to the combination of `pubsub_topic` and
a set of `content_topics`.
The request MAY include filter criteria,
conditional to the selected `filter_subscribe_type`.
If the request contains filter criteria,
it MUST contain a `pubsub_topic`
and the `content_topics` set MUST NOT be empty.
A `WakuMessage` matches filter criteria when its `content_topic` is in the `content_topics` set
A [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) matches filter criteria
when its `content_topic` is in the `content_topics` set
and it was published on a matching `pubsub_topic`.
#### Filter Subscribe Types
The following filter subscribe types are defined:
The filter-subscribe types are defined as follows:
##### SUBSCRIBER_PING
A filter client that sends a `FilterSubscribeRequest` with `filter_subscribe_type` set to `SUBSCRIBER_PING`
requests that the service node SHOULD indicate if it has any active subscriptions for this client.
A filter client that sends a `FilterSubscribeRequest` with
`filter_subscribe_type` set to `SUBSCRIBER_PING`,
requests that the filter service node SHOULD indicate if it has any active subscriptions
for this client.
The filter client SHOULD exclude any filter criteria from the request.
The filter service node SHOULD respond with a success code if it has any active subscriptions for this client
or an error code if not.
The filter service node SHOULD respond with a success `status_code`
if it has any active subscriptions for this client
or an error `status_code` if not.
The filter service node SHOULD ignore any filter criteria in the request.
##### SUBSCRIBE
A filter client that sends a `FilterSubscribeRequest` with `filter_subscribe_type` set to `SUBSCRIBE`
requests that the service node SHOULD push messages matching this filter to the client.
A filter client that sends a `FilterSubscribeRequest` with
`filter_subscribe_type` set to `SUBSCRIBE`
requests that the filter service node SHOULD push messages
matching this filter to the client.
The filter client MUST include the desired filter criteria in the request.
A client MAY use this request type to _modify_ an existing subscription
by providing _additional_ filter criteria in a new request.
A client MAY use this request type to _refresh_ an existing subscription
by providing _the same_ filter criteria in a new request.
The filter service node SHOULD respond with a success code if it successfully honored this request
or an error code if not.
The filter service node SHOULD respond with an error code and discard the request
if the subscribe request does not contain valid filter criteria,
The filter service node SHOULD respond with a success `status_code`
if it successfully honored this request
or an error `status_code` if not.
The filter service node SHOULD respond with an error `status_code` and
discard the request if the `FilterSubscribeRequest`
does not contain valid filter criteria,
i.e. both a `pubsub_topic` _and_ a non-empty `content_topics` set.
##### UNSUBSCRIBE
A filter client that sends a `FilterSubscribeRequest` with `filter_subscribe_type` set to `UNSUBSCRIBE`
requests that the service node SHOULD _stop_ pushing messages matching this filter to the client.
The filter client MUST include the filter criteria it desires to unsubscribe from in the request.
A filter client that sends a `FilterSubscribeRequest` with
`filter_subscribe_type` set to `UNSUBSCRIBE`
requests that the service node SHOULD _stop_ pushing messages
matching this filter to the client.
The filter client MUST include the filter criteria
it desires to unsubscribe from in the request.
A client MAY use this request type to _modify_ an existing subscription
by providing _a subset of_ the original filter criteria to unsubscribe from in a new request.
The filter service node SHOULD respond with a success code if it successfully honored this request
or an error code if not.
The filter service node SHOULD respond with an error code and discard the request
if the unsubscribe request does not contain valid filter criteria,
by providing _a subset of_ the original filter criteria
to unsubscribe from in a new request.
The filter service node SHOULD respond with a success `status_code`
if it successfully honored this request
or an error `status_code` if not.
The filter service node SHOULD respond with an error `status_code` and
discard the request if the unsubscribe request does not contain valid filter criteria,
i.e. both a `pubsub_topic` _and_ a non-empty `content_topics` set.
##### UNSUBSCRIBE_ALL
A filter client that sends a `FilterSubscribeRequest` with `filter_subscribe_type` set to `UNSUBSCRIBE_ALL`
requests that the service node SHOULD _stop_ pushing messages matching _any_ filter to the client.
A filter client that sends a `FilterSubscribeRequest` with
`filter_subscribe_type` set to `UNSUBSCRIBE_ALL`
requests that the service node SHOULD _stop_ pushing messages
matching _any_ filter to the client.
The filter client SHOULD exclude any filter criteria from the request.
The filter service node SHOULD remove any existing subscriptions for this client.
It SHOULD respond with a success code if it successfully honored this request
or an error code if not.
It SHOULD respond with a success `status_code` if it successfully honored this request
or an error `status_code` if not.
### Filter-Push
A filter client node MUST support the _filter-push_ protocol
to allow filter service nodes to push messages matching registered subscriptions to this client.
to allow filter service nodes to push messages
matching registered subscriptions to this client.
A filter service node SHOULD push all messages
matching the filter criteria in a registered subscription
to the subscribed filter client.
These [`WakuMessage`s](../14/message.md) are likely to come from [`11/WAKU2-RELAY`](../11/relay.md),
These [`WakuMessage`s](/waku/standards/core/14/message.md)
are likely to come from [`11/WAKU2-RELAY`](/waku/standards/core/11/relay.md),
but there MAY be other sources or protocols where this comes from.
This is up to the consumer of the protocol.
If a message push fails,
the filter service node MAY consider the client node to be unreachable.
If a specific filter client node is not reachable from the service node for a period of time,
the filter service node MAY choose to stop pushing messages to the client and remove its subscription.
If a specific filter client node is not reachable from the service node
for a period of time,
the filter service node MAY choose to stop pushing messages to the client and
remove its subscription.
This period is up to the service node implementation.
We consider `1 minute` to be a reasonable default.
It is RECOMMENDED to set `1 minute` as a reasonable default.
#### Message Push
@@ -224,29 +253,69 @@ A filter client SHOULD verify that each `MessagePush` it receives
originated from a service node where the client has an active subscription
and that it matches filter criteria belonging to that subscription.
---
## Future Work
### Adversarial Model
Any node running the `WakuFilter` protocol
i.e., both the subscriber node and
the queried node are considered as an adversary.
Furthermore, we consider the adversary as a passive entity
that attempts to collect information from other nodes to conduct an attack but
it does so without violating protocol definitions and instructions.
For example, under the passive adversarial model,
no malicious node intentionally hides the messages
matching to one's subscribed content filter
as it is against the description of the `WakuFilter` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the nodes and their connections.
- An adversary that can eavesdrop on communication links
between arbitrary pairs of nodes (unless the adversary is one end of the communication).
In specific, the communication channels are assumed to be secure.
### Security Considerations
Note that while using `WakuFilter` allows light nodes to save bandwidth,
it comes with a privacy cost in the sense that they need to
disclose their liking topics to the full nodes to retrieve the relevant messages.
Currently, anonymous subscription is not supported by the `WakuFilter`, however,
potential solutions in this regard are discussed below.
#### Future Work
<!-- Alternative title: Filter-subscriber unlinkability -->
**Anonymous filter subscription**: This feature guarantees that nodes can anonymously subscribe for a message filter (i.e., without revealing their exact content filter). As such, no adversary in the `WakuFilter` protocol would be able to link nodes to their subscribed content filers. The current version of the `WakuFilter` protocol does not provide anonymity as the subscribing node has a direct connection to the full node and explicitly submits its content filter to be notified about the matching messages. However, one can consider preserving anonymity through one of the following ways:
- By hiding the source of the subscription i.e., anonymous communication. That is the subscribing node shall hide all its PII in its filter request e.g., its IP address. This can happen by the utilization of a proxy server or by using Tor<!-- TODO: if nodes have to disclose their PeerIDs (e.g., for authentication purposes) when connecting to other nodes in the WakuFilter protocol, then Tor does not preserve anonymity since it only helps in hiding the IP. So, the PeerId usage in switches must be investigated further. Depending on how PeerId is used, one may be able to link between a subscriber and its content filter despite hiding the IP address-->.
Note that the current structure of filter requests i.e., `FilterRPC` does not embody any piece of PII, otherwise, such data fields must be treated carefully to achieve anonymity.
- By deploying secure 2-party computations in which the subscribing node obtains the messages matching a content filter whereas the full node learns nothing about the content filter as well as the messages pushed to the subscribing node. Examples of such 2PC protocols are [Oblivious Transfers](https://link.springer.com/referenceworkentry/10.1007%2F978-1-4419-5906-5_9#:~:text=Oblivious%20transfer%20(OT)%20is%20a,information%20the%20receiver%20actually%20obtains.) and one-way Private Set Intersections (PSI).
**Anonymous filter subscription**:
This feature guarantees that nodes can anonymously subscribe for a message filter
(i.e., without revealing their exact content filter).
As such, no adversary in the `WakuFilter` protocol
would be able to link nodes to their subscribed content filers.
The current version of the `WakuFilter` protocol does not provide anonymity
as the subscribing node has a direct connection to the full node and
explicitly submits its content filter to be notified about the matching messages.
However, one can consider preserving anonymity through one of the following ways:
## Changelog
- By hiding the source of the subscription i.e., anonymous communication.
That is the subscribing node shall hide all its PII in its filter request
e.g., its IP address.
This can happen by the utilization of a proxy server or by using Tor
<!-- TODO: if nodes have to disclose their PeerIDs
(e.g., for authentication purposes)
when connecting to other nodes in the WakuFilter protocol,
then Tor does not preserve anonymity since it only helps in hiding the IP.
So, the PeerId usage in switches must be investigated further.
Depending on how PeerId is used,
one may be able to link between a subscriber and
its content filter despite hiding the IP address-->.
Note that the current structure of filter requests
i.e., `FilterRPC` does not embody any piece of PII, otherwise,
such data fields must be treated carefully to achieve anonymity.
### Next
- Added initial threat model and security analysis.
### 2.0.0-beta2
Initial draft version. Released [2020-10-28](https://github.com/vacp2p/specs/commit/5ceeb88cee7b918bb58f38e7c4de5d581ff31e68)
- Fix: Ensure contentFilter is a repeated field, on implementation
- Change: Add ability to unsubscribe from filters. Make `subscribe` an explicit boolean indication. Edit protobuf field order to be consistent with libp2p.
### 2.0.0-beta1
Initial draft version. Released [2020-10-05](https://github.com/vacp2p/specs/commit/31857c7434fa17efc00e3cd648d90448797d107b)
- By deploying secure 2-party computations in which
the subscribing node obtains the messages matching a content filter
whereas the full node learns nothing about the content filter as well as
the messages pushed to the subscribing node.
Examples of such 2PC protocols are
[Oblivious Transfers](https://link.springer.com/referenceworkentry/10.1007%2F978-1-4419-5906-5_9#:~:text=Oblivious%20transfer%20(OT)%20is%20a,information%20the%20receiver%20actually%20obtains.)
and one-way Private Set Intersections (PSI).
## Copyright
@@ -255,13 +324,14 @@ Copyright and related rights waived via
## References
- [message-based
filtering](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_filtering)
- [`WakuMessage`s](../14/message.md)
- [`11/WAKU2-RELAY`](../11/relay.md)
- [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
- [message-based filtering](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_filtering)
- [13/WAKU2-STORE](/waku/standards/core/13/store.md)
- [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)
- [Oblivious Transfers](https://link.springer.com/referenceworkentry/10.1007%2F978-1-4419-5906-5_9#:~:text=Oblivious%20transfer%20(OT)%20is%20a,information%20the%20receiver%20actually%20obtains)
- previous versions: [00](./previous-versions00)
- 12/WAKU2-FILTER previous version: [00](waku/standards/core/12/previous-versions/00/filter.md)
### Informative
1. [Message Filtering (Wikipedia)](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_filtering)
2. [Libp2p PubSub spec - topic validation](https://github.com/libp2p/specs/tree/master/pubsub#topic-validation)

View File

@@ -4,6 +4,7 @@ title: 12/WAKU2-FILTER
name: Waku v2 Filter
status: draft
tags: waku-core
version: v00
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Dean Eigenmann <dean@status.im>
@@ -11,10 +12,13 @@ contributors:
- Sanaz Taheri <sanaz@status.im>
- Ebube Ud <ebube@status.im>
---
version: 00
---
`WakuFilter` is a protocol that enables subscribing to messages that a peer receives. This is a more lightweight version of `WakuRelay` specifically designed for bandwidth restricted devices. This is due to the fact that light nodes subscribe to full-nodes and only receive the messages they desire.
`WakuFilter` is a protocol that enables subscribing to messages that a peer receives.
This is a more lightweight version of `WakuRelay`
specifically designed for bandwidth restricted devices.
This is due to the fact that light nodes subscribe to full-nodes and
only receive the messages they desire.
## Content filtering
@@ -41,24 +45,49 @@ It is worth noting that a light node could get by with only using the `store`
protocol to query for a recent time window, provided it is acceptable to do
frequent polling.
## Design Requirements
The effectiveness and reliability of the content filtering service enabled by `WakuFilter` protocol rely on the *high availability* of the full nodes as the service providers. To this end, full nodes must feature *high uptime* (to persistently listen and capture the network messages) as well as *high Bandwidth* (to provide timely message delivery to the light nodes).
The effectiveness and reliability of the content filtering service
enabled by `WakuFilter` protocol rely on the *high availability* of the full nodes
as the service providers.
To this end, full nodes must feature *high uptime*
(to persistently listen and capture the network messages)
as well as *high Bandwidth* (to provide timely message delivery to the light nodes).
## Security Consideration
Note that while using `WakuFilter` allows light nodes to save bandwidth, it comes with a privacy cost in the sense that they need to disclose their liking topics to the full nodes to retrieve the relevant messages. Currently, anonymous subscription is not supported by the `WakuFilter`, however, potential solutions in this regard are sketched below in [Future Work](#future-work) section.
Note that while using `WakuFilter` allows light nodes to save bandwidth,
it comes with a privacy cost in the sense that they need to disclose their liking
topics to the full nodes to retrieve the relevant messages.
Currently, anonymous subscription is not supported by the `WakuFilter`, however,
potential solutions in this regard are sketched below in [Future Work](#future-work)
section.
### Terminology
The term Personally identifiable information (PII) refers to any piece of data that can be used to uniquely identify a user. For example, the signature verification key, and the hash of one's static IP address are unique for each user and hence count as PII.
The term Personally identifiable information (PII)
refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key, and
the hash of one's static IP address are unique for each user and hence count as PII.
## Adversarial Model
Any node running the `WakuFilter` protocol i.e., both the subscriber node and the queried node are considered as an adversary. Furthermore, we consider the adversary as a passive entity that attempts to collect information from other nodes to conduct an attack but it does so without violating protocol definitions and instructions. For example, under the passive adversarial model, no malicious node intentionally hides the messages matching to one's subscribed content filter as it is against the description of the `WakuFilter` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the nodes and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs of nodes (unless the adversary is one end of the communication). In specific, the communication channels are assumed to be secure.
Any node running the `WakuFilter` protocol i.e.,
both the subscriber node and the queried node are considered as an adversary.
Furthermore, we consider the adversary as a passive entity
that attempts to collect information from other nodes to conduct an attack but
it does so without violating protocol definitions and instructions.
For example, under the passive adversarial model,
no malicious node intentionally hides the messages matching
to one's subscribed content filter as it is against the description
of the `WakuFilter` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the nodes and their connections.
- An adversary that can eavesdrop on communication links
between arbitrary pairs of nodes (unless the adversary is one end of the communication).
In specific, the communication channels are assumed to be secure.
### Protobuf
@@ -86,13 +115,14 @@ message FilterRPC {
#### FilterRPC
A node MUST send all Filter messages (`FilterRequest`, `MessagePush`) wrapped inside a
`FilterRPC` this allows the node handler to determine how to handle a message as the Waku
Filter protocol is not a request response based protocol but instead a push based system.
A node MUST send all Filter messages (`FilterRequest`, `MessagePush`)
wrapped inside a `FilterRPC` this allows the node handler
to determine how to handle a message as the Waku Filter protocol
is not a request response based protocol but instead a push based system.
The `requestId` MUST be a uniquely generated string. When a `MessagePush` is sent
the `requestId` MUST match the `requestId` of the subscribing `FilterRequest` whose filters
matched the message causing it to be pushed.
the `requestId` MUST match the `requestId` of the subscribing `FilterRequest`
whose filters matched the message causing it to be pushed.
#### FilterRequest
@@ -100,7 +130,7 @@ A `FilterRequest` contains an optional topic, zero or more content filters and
a boolean signifying whether to subscribe or unsubscribe to the given filters.
True signifies 'subscribe' and false signifies 'unsubscribe'.
A node that sends the RPC with a filter request and `subscribe` set to 'true'
A node that sends the RPC with a filter request and `subscribe` set to 'true'
requests that the filter node SHOULD notify the light requesting node of messages
matching this filter.
@@ -123,7 +153,8 @@ mechanism is currently planned but underspecified.
#### MessagePush
A filter node that has received a filter request SHOULD push all messages that
match this filter to a light node. These [`WakuMessage`'s](../14/message.md) are likely to come from the
match this filter to a light node. These [`WakuMessage`'s](../14/message.md)
are likely to come from the
`relay` protocol and be kept at the Node, but there MAY be other sources or
protocols where this comes from. This is up to the consumer of the protocol.
@@ -135,25 +166,54 @@ period of time (e.g. a TTL), then the filter node MAY choose to not push these
messages to the node. This period is up to the consumer of the protocol and node
implementation, though a reasonable default is one minute.
---
# Future Work
---
## Future Work
<!-- Alternative title: Filter-subscriber unlinkability -->
**Anonymous filter subscription**: This feature guarantees that nodes can anonymously subscribe for a message filter (i.e., without revealing their exact content filter). As such, no adversary in the `WakuFilter` protocol would be able to link nodes to their subscribed content filers. The current version of the `WakuFilter` protocol does not provide anonymity as the subscribing node has a direct connection to the full node and explicitly submits its content filter to be notified about the matching messages. However, one can consider preserving anonymity through one of the following ways:
- By hiding the source of the subscription i.e., anonymous communication. That is the subscribing node shall hide all its PII in its filter request e.g., its IP address. This can happen by the utilization of a proxy server or by using Tor<!-- TODO: if nodes have to disclose their PeerIDs (e.g., for authentication purposes) when connecting to other nodes in the WakuFilter protocol, then Tor does not preserve anonymity since it only helps in hiding the IP. So, the PeerId usage in switches must be investigated further. Depending on how PeerId is used, one may be able to link between a subscriber and its content filter despite hiding the IP address-->.
Note that the current structure of filter requests i.e., `FilterRPC` does not embody any piece of PII, otherwise, such data fields must be treated carefully to achieve anonymity.
- By deploying secure 2-party computations in which the subscribing node obtains the messages matching a content filter whereas the full node learns nothing about the content filter as well as the messages pushed to the subscribing node. Examples of such 2PC protocols are [Oblivious Transfers](https://link.springer.com/referenceworkentry/10.1007%2F978-1-4419-5906-5_9#:~:text=Oblivious%20transfer%20(OT)%20is%20a,information%20the%20receiver%20actually%20obtains.) and one-way Private Set Intersections (PSI).
**Anonymous filter subscription**:
This feature guarantees that nodes can anonymously subscribe for a message filter
(i.e., without revealing their exact content filter).
As such, no adversary in the `WakuFilter` protocol would be able to link nodes
to their subscribed content filers.
The current version of the `WakuFilter` protocol does not provide anonymity
as the subscribing node has a direct connection to the full node and
explicitly submits its content filter to be notified about the matching messages.
However, one can consider preserving anonymity through one of the following ways:
- By hiding the source of the subscription i.e., anonymous communication.
That is the subscribing node shall hide all its PII in its filter request e.g.,
its IP address.
This can happen by the utilization of a proxy server or by using Tor
<!-- TODO: if nodes have to disclose their PeerIDs (e.g., for authentication purposes)
when connecting to other nodes in the WakuFilter protocol,
then Tor does not preserve anonymity since it only helps in hiding the IP.
So, the PeerId usage in switches must be investigated further.
Depending on how PeerId is used, one may be able to link between a subscriber and
its content filter despite hiding the IP address-->.
Note that the current structure of filter requests i.e.,
`FilterRPC` does not embody any piece of PII, otherwise,
such data fields must be treated carefully to achieve anonymity.
- By deploying secure 2-party computations in which the subscribing node obtains
the messages matching a content filter whereas the full node learns nothing
about the content filter as well as the messages pushed to the subscribing node.
Examples of such 2PC protocols are [Oblivious Transfers](https://link.springer.com/referenceworkentry/10.1007%2F978-1-4419-5906-5_9#:~:text=Oblivious%20transfer%20(OT)%20is%20a,information%20the%20receiver%20actually%20obtains.)
and one-way Private Set Intersections (PSI).
## Changelog
### Next
- Added initial threat model and security analysis.
### 2.0.0-beta2
Initial draft version. Released [2020-10-28](https://github.com/vacp2p/specs/commit/5ceeb88cee7b918bb58f38e7c4de5d581ff31e68)
- Fix: Ensure contentFilter is a repeated field, on implementation
- Change: Add ability to unsubscribe from filters. Make `subscribe` an explicit boolean indication. Edit protobuf field order to be consistent with libp2p.
- Change: Add ability to unsubscribe from filters.
Make `subscribe` an explicit boolean indication.
Edit protobuf field order to be consistent with libp2p.
### 2.0.0-beta1

View File

@@ -0,0 +1,359 @@
---
slug: 13
title: 13/WAKU2-STORE
name: Waku v2 Store
status: draft
tags: waku-core
version: 00
editor: Simon-Pierre Vivier <simvivier@status.im>
contributors:
- Dean Eigenmann <dean@status.im>
- Oskar Thorén <oskarth@titanproxy.com>
- Aaryamann Challani <p1ge0nh8er@proton.me>
- Sanaz Taheri <sanaz@status.im>
- Hanno Cornelius <hanno@status.im>
---
## Abstract
This specification explains the `13/WAKU2-STORE` protocol
which enables querying of messages received through the relay protocol and
stored by other nodes.
It also supports pagination for more efficient querying of historical messages.
**Protocol identifier***: `/vac/waku/store/2.0.0-beta4`
## Terminology
The term PII, Personally Identifiable Information,
refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key, and
the hash of one's static IP address are unique for each user and hence count as PII.
## Design Requirements
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt).
Nodes willing to provide the storage service using `13/WAKU2-STORE` protocol,
SHOULD provide a complete and full view of message history.
As such, they are required to be *highly available* and
specifically have a *high uptime* to consistently receive and store network messages.
The high uptime requirement makes sure that no message is missed out
hence a complete and intact view of the message history
is delivered to the querying nodes.
Nevertheless, in case storage provider nodes cannot afford high availability,
the querying nodes may retrieve the historical messages from multiple sources
to achieve a full and intact view of the past.
The concept of `ephemeral` messages introduced in
[`14/WAKU2-MESSAGE`](../14/message.md) affects `13/WAKU2-STORE` as well.
Nodes running `13/WAKU2-STORE` SHOULD support `ephemeral` messages as specified in
[14/WAKU2-MESSAGE](../14/message.md).
Nodes running `13/WAKU2-STORE` SHOULD NOT store messages
with the `ephemeral` flag set to `true`.
## Adversarial Model
Any peer running the `13/WAKU2-STORE` protocol, i.e.
both the querying node and the queried node, are considered as an adversary.
Furthermore,
we currently consider the adversary as a passive entity
that attempts to collect information from other peers to conduct an attack but
it does so without violating protocol definitions and instructions.
As we evolve the protocol,
further adversarial models will be considered.
For example, under the passive adversarial model,
no malicious node hides or
lies about the history of messages
as it is against the description of the `13/WAKU2-STORE` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links
between arbitrary pairs of peers (unless the adversary is one end of the communication).
In specific, the communication channels are assumed to be secure.
## Wire Specification
Peers communicate with each other using a request / response API.
The messages sent are Protobuf RPC messages which are implemented using
[protocol buffers v3](https://developers.google.com/protocol-buffers/).
The following are the specifications of the Protobuf messages.
### Payloads
```protobuf
syntax = "proto3";
message Index {
bytes digest = 1;
sint64 receiverTime = 2;
sint64 senderTime = 3;
string pubsubTopic = 4;
}
message PagingInfo {
uint64 pageSize = 1;
Index cursor = 2;
enum Direction {
BACKWARD = 0;
FORWARD = 1;
}
Direction direction = 3;
}
message ContentFilter {
string contentTopic = 1;
}
message HistoryQuery {
// the first field is reserved for future use
string pubsubtopic = 2;
repeated ContentFilter contentFilters = 3;
PagingInfo pagingInfo = 4;
}
message HistoryResponse {
// the first field is reserved for future use
repeated WakuMessage messages = 2;
PagingInfo pagingInfo = 3;
enum Error {
NONE = 0;
INVALID_CURSOR = 1;
}
Error error = 4;
}
message HistoryRPC {
string request_id = 1;
HistoryQuery query = 2;
HistoryResponse response = 3;
}
```
#### Index
To perform pagination,
each `WakuMessage` stored at a node running the `13/WAKU2-STORE` protocol
is associated with a unique `Index` that encapsulates the following parts.
- `digest`: a sequence of bytes representing the SHA256 hash of a `WakuMessage`.
The hash is computed over the concatenation of `contentTopic`
and `payload` fields of a `WakuMessage` (see [14/WAKU2-MESSAGE](../14/message.md)).
- `receiverTime`: the UNIX time in nanoseconds
at which the `WakuMessage` is received by the receiving node.
- `senderTime`: the UNIX time in nanoseconds
at which the `WakuMessage` is generated by its sender.
- `pubsubTopic`: the pubsub topic on which the `WakuMessage` is received.
#### PagingInfo
`PagingInfo` holds the information required for pagination.
It consists of the following components.
- `pageSize`: A positive integer indicating the number of queried `WakuMessage`s
in a `HistoryQuery`
(or retrieved `WakuMessage`s in a `HistoryResponse`).
- `cursor`: holds the `Index` of a `WakuMessage`.
- `direction`: indicates the direction of paging
which can be either `FORWARD` or `BACKWARD`.
#### ContentFilter
`ContentFilter` carries the information required for filtering historical messages.
- `contentTopic` represents the content topic of the queried historical `WakuMessage`.
This field maps to the `contentTopic` field of the [14/WAKU2-MESSAGE](../14/message.md).
#### HistoryQuery
RPC call to query historical messages.
- The `pubsubTopic` field MUST indicate the pubsub topic
of the historical messages to be retrieved.
This field denotes the pubsub topic on which `WakuMessage`s are published.
This field maps to `topicIDs` field of `Message` in [`11/WAKU2-RELAY`](../11/relay.md).
Leaving this field empty means no filter on the pubsub topic
of message history is requested.
This field SHOULD be left empty in order to retrieve the historical `WakuMessage`
regardless of the pubsub topics on which they are published.
- The `contentFilters` field MUST indicate the list of content filters
based on which the historical messages are to be retrieved.
Leaving this field empty means no filter on the content topic
of message history is required.
This field SHOULD be left empty in order
to retrieve historical `WakuMessage` regardless of their content topics.
- `PagingInfo` holds the information required for pagination.
Its `pageSize` field indicates the number of `WakuMessage`s
to be included in the corresponding `HistoryResponse`.
It is RECOMMENDED that the queried node defines a maximum page size internally.
If the querying node leaves the `pageSize` unspecified,
or if the `pageSize` exceeds the maximum page size,
the queried node SHOULD auto-paginate the `HistoryResponse`
to no more than the configured maximum page size.
This allows mitigation of long response time for `HistoryQuery`.
In the forward pagination request,
the `messages` field of the `HistoryResponse` SHALL contain, at maximum,
the `pageSize` amount of `WakuMessage` whose `Index`
values are larger than the given `cursor`
(and vise versa for the backward pagination).
Note that the `cursor` of a `HistoryQuery` MAY be empty
(e.g., for the initial query), as such, and
depending on whether the `direction` is `BACKWARD` or
`FORWARD` the last or the first `pageSize` `WakuMessage` SHALL be returned,
respectively.
#### Sorting Messages
The queried node MUST sort the `WakuMessage` based on their `Index`,
where the `senderTime` constitutes the most significant part and
the `digest` comes next, and
then perform pagination on the sorted result.
As such, the retrieved page contains an ordered list of `WakuMessage`
from the oldest messages to the most recent one.
Alternatively, the `receiverTime` (instead of `senderTime`)
MAY be used to sort messages during the paging process.
However, it is RECOMMENDED the use of the `senderTime`
for sorting as it is invariant and
consistent across all the nodes.
This has the benefit of `cursor` reusability i.e.,
a `cursor` obtained from one node can be consistently used
to query from another node.
However, this `cursor` reusability does not hold when the `receiverTime` is utilized
as the receiver time is affected by the network delay and
nodes' clock asynchrony.
#### HistoryResponse
RPC call to respond to a HistoryQuery call.
- The `messages` field MUST contain the messages found,
these are [14/WAKU2-MESSAGE](../14/message.md) types.
- `PagingInfo` holds the paging information based
on which the querying node can resume its further history queries.
The `pageSize` indicates the number of returned Waku messages
(i.e., the number of messages included in the `messages` field of `HistoryResponse`).
The `direction` is the same direction as in the corresponding `HistoryQuery`.
In the forward pagination, the `cursor` holds the `Index` of the last message
in the `HistoryResponse` `messages` (and the first message in the backward paging).
Regardless of the paging direction,
the retrieved `messages` are always sorted in ascending order
based on their timestamp as explained in the [sorting messages](#sorting-messages)section,
that is, from the oldest to the most recent.
The requester SHALL embed the returned `cursor` inside its next `HistoryQuery`
to retrieve the next page of the [14/WAKU2-MESSAGE](../14/message.md).
The `cursor` obtained from one node SHOULD NOT be used in a request to another node
because the result may be different.
- The `error` field contains information about any error that has occurred
while processing the corresponding `HistoryQuery`.
`NONE` stands for no error.
This is also the default value.
`INVALID_CURSOR` means that the `cursor` field of `HistoryQuery`
does not match with the `Index` of any of the `WakuMessage`
persisted by the queried node.
## Security Consideration
The main security consideration to take into account
while using this protocol is that a querying node
have to reveal their content filters of interest to the queried node,
hence potentially compromising their privacy.
## Future Work
- **Anonymous query**: This feature guarantees that nodes
can anonymously query historical messages from other nodes i.e.,
without disclosing the exact topics of [14/WAKU2-MESSAGE](../14/message.md)
they are interested in.
As such, no adversary in the `13/WAKU2-STORE` protocol
would be able to learn which peer is interested in which content filters i.e.,
content topics of [14/WAKU2-MESSAGE](../14/message.md).
The current version of the `13/WAKU2-STORE` protocol does not provide anonymity
for historical queries,
as the querying node needs to directly connect to another node
in the `13/WAKU2-STORE` protocol and
explicitly disclose the content filters of its interest
to retrieve the corresponding messages.
However, one can consider preserving anonymity through one of the following ways:
- By hiding the source of the request i.e., anonymous communication.
That is the querying node shall hide all its PII in its history request
e.g., its IP address.
This can happen by the utilization of a proxy server or by using Tor.
Note that the current structure of historical requests
does not embody any piece of PII, otherwise,
such data fields must be treated carefully to achieve query anonymity.
<!-- TODO: if nodes have to disclose their PeerIDs
(e.g., for authentication purposes) when connecting to other nodes
in the store protocol,
then Tor does not preserve anonymity since it only helps in hiding the IP.
So, the PeerId usage in switches must be investigated further.
Depending on how PeerId is used, one may be able to link between a querying node
and its queried topics despite hiding the IP address-->
- By deploying secure 2-party computations in which the querying node
obtains the historical messages of a certain topic,
the queried node learns nothing about the query.
Examples of such 2PC protocols are secure one-way Private Set Intersections (PSI).
<!-- TODO: add a reference for PSIs? -->
<!-- TODO: more techniques to be included -->
<!-- TODO: Censorship resistant:
this is about a node that hides the historical messages from other nodes.
This attack is not included in the specs
since it does not fit the passive adversarial model
(the attacker needs to deviate from the store protocol).-->
- **Robust and verifiable timestamps**:
Messages timestamp is a way to show that the message existed
prior to some point in time.
However, the lack of timestamp verifiability can create room for a range of attacks,
including injecting messages with invalid timestamps pointing to the far future.
To better understand the attack,
consider a store node whose current clock shows `2021-01-01 00:00:30`
(and assume all the other nodes have a synchronized clocks +-20seconds).
The store node already has a list of messages,
`(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20)`,
that are sorted based on their timestamp.
An attacker sends a message with an arbitrary large timestamp e.g.,
10 hours ahead of the correct clock `(m',2021-01-01 10:00:30)`.
The store node places `m'` at the end of the list,
```text
(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20),(m',2021-01-01 10:00:30).
```
Now another message arrives with a valid timestamp e.g.,
`(m11, 2021-01-01 00:00:45)`.
However, since its timestamp precedes the malicious message `m'`,
it gets placed before `m'` in the list i.e.,
```text
(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20), (m11, 2021-01-01 00:00:45), (m',2021-01-01 10:00:30).
```
In fact, for the next 10 hours,
`m'` will always be considered as the most recent message and
served as the last message to the querying nodes irrespective
of how many other messages arrive afterward.
A robust and verifiable timestamp allows the receiver of a message
to verify that a message has been generated prior to the claimed timestamp.
One solution is the use of [open timestamps](https://opentimestamps.org/) e.g.,
block height in Blockchain-based timestamps.
That is, messages contain the most recent block height
perceived by their senders at the time of message generation.
This proves accuracy within a range of minutes (e.g., in Bitcoin blockchain) or
seconds (e.g., in Ethereum 2.0) from the time of origination.
## Copyright
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [14/WAKU2-MESSAGE](../14/message.md)
2. [protocol buffers v3](https://developers.google.com/protocol-buffers/)
3. [11/WAKU2-RELAY](../11/relay.md)
4. [Open timestamps](https://opentimestamps.org/)

View File

@@ -1,253 +1,435 @@
---
slug: 13
title: 13/WAKU2-STORE
name: Waku v2 Store
name: Waku Store Query
status: draft
tags: waku-core
editor: Simon-Pierre Vivier <simvivier@status.im>
version: 01
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Dean Eigenmann <dean@status.im>
- Oskar Thorén <oskarth@titanproxy.com>
- Aaryamann Challani <p1ge0nh8er@proton.me>
- Sanaz Taheri <sanaz@status.im>
- Hanno Cornelius <hanno@status.im>
---
Previous version: [00](/waku/standards/core/13/previous-versions/00/store.md)
## Abstract
This specification explains the `13/WAKU2-STORE` protocol which enables querying of messages received through the relay protocol and
stored by other nodes.
It also supports pagination for more efficient querying of historical messages.
**Protocol identifier***: `/vac/waku/store/2.0.0-beta4`
This specification explains the `WAKU2-STORE` protocol,
which enables querying of [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)s.
## Terminology
The term PII, Personally Identifiable Information,
refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key, and
**Protocol identifier***: `/vac/waku/store-query/3.0.0`
### Terminology
The term PII, Personally Identifiable Information,
refers to any piece of data that can be used to uniquely identify a user.
For example, the signature verification key, and
the hash of one's static IP address are unique for each user and hence count as PII.
## Design Requirements
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”,
“RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt).
Nodes willing to provide the storage service using `13/WAKU2-STORE` protocol,
SHOULD provide a complete and full view of message history.
As such, they are required to be *highly available* and
specifically have a *high uptime* to consistently receive and store network messages.
The high uptime requirement makes sure that no message is missed out hence a complete and
intact view of the message history is delivered to the querying nodes.
Nevertheless, in case storage provider nodes cannot afford high availability,
the querying nodes may retrieve the historical messages from multiple sources to achieve a full and intact view of the past.
The concept of `ephemeral` messages introduced in [`14/WAKU2-MESSAGE`](../14/message.md) affects `13/WAKU2-STORE` as well.
Nodes running `13/WAKU2-STORE` SHOULD support `ephemeral` messages as specified in [14/WAKU2-MESSAGE](../14/message.md).
Nodes running `13/WAKU2-STORE` SHOULD NOT store messages with the `ephemeral` flag set to `true`.
## Adversarial Model
Any peer running the `13/WAKU2-STORE` protocol, i.e.
both the querying node and the queried node, are considered as an adversary.
Furthermore,
we currently consider the adversary as a passive entity that attempts to collect information from other peers to conduct an attack but
it does so without violating protocol definitions and instructions.
As we evolve the protocol,
further adversarial models will be considered.
For example, under the passive adversarial model,
no malicious node hides or
lies about the history of messages as it is against the description of the `13/WAKU2-STORE` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs of peers (unless the adversary is one end of the communication).
In specific, the communication channels are assumed to be secure.
## Wire Specification
Peers communicate with each other using a request / response API.
The messages sent are Protobuf RPC messages which are implemented using [protocol buffers v3](https://developers.google.com/protocol-buffers/).
The following are the specifications of the Protobuf messages.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt).
### Design Requirements
The concept of `ephemeral` messages introduced in [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) affects `WAKU2-STORE` as well.
Nodes running `WAKU2-STORE` SHOULD support `ephemeral` messages as specified in [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md).
Nodes running `WAKU2-STORE` SHOULD NOT store messages with the `ephemeral` flag set to `true`.
### Payloads
```protobuf
syntax = "proto3";
message Index {
bytes digest = 1;
sint64 receiverTime = 2;
sint64 senderTime = 3;
string pubsubTopic = 4;
// Protocol identifier: /vac/waku/store-query/3.0.0
package waku.store.v3;
import "waku/message/v1/message.proto";
message WakuMessageKeyValue {
optional bytes message_hash = 1; // Globally unique key for a Waku Message
// Full message content and associated pubsub_topic as value
optional waku.message.v1.WakuMessage message = 2;
optional string pubsub_topic = 3;
}
message PagingInfo {
uint64 pageSize = 1;
Index cursor = 2;
enum Direction {
BACKWARD = 0;
FORWARD = 1;
}
Direction direction = 3;
}
message ContentFilter {
string contentTopic = 1;
}
message HistoryQuery {
// the first field is reserved for future use
string pubsubtopic = 2;
repeated ContentFilter contentFilters = 3;
PagingInfo pagingInfo = 4;
}
message HistoryResponse {
// the first field is reserved for future use
repeated WakuMessage messages = 2;
PagingInfo pagingInfo = 3;
enum Error {
NONE = 0;
INVALID_CURSOR = 1;
}
Error error = 4;
}
message HistoryRPC {
message StoreQueryRequest {
string request_id = 1;
HistoryQuery query = 2;
HistoryResponse response = 3;
bool include_data = 2; // Response should include full message content
// Filter criteria for content-filtered queries
optional string pubsub_topic = 10;
repeated string content_topics = 11;
optional sint64 time_start = 12;
optional sint64 time_end = 13;
// List of key criteria for lookup queries
repeated bytes message_hashes = 20; // Message hashes (keys) to lookup
// Pagination info. 50 Reserved
optional bytes pagination_cursor = 51; // Message hash (key) from where to start query (exclusive)
bool pagination_forward = 52;
optional uint64 pagination_limit = 53;
}
message StoreQueryResponse {
string request_id = 1;
optional uint32 status_code = 10;
optional string status_desc = 11;
repeated WakuMessageKeyValue messages = 20;
optional bytes pagination_cursor = 51;
}
```
#### Index
### General Store Query Concepts
To perform pagination,
each `WakuMessage` stored at a node running the `13/WAKU2-STORE` protocol is associated with a unique `Index` that encapsulates the following parts.
- `digest`: a sequence of bytes representing the SHA256 hash of a `WakuMessage`.
The hash is computed over the concatenation of `contentTopic` and `payload` fields of a `WakuMessage` (see [14/WAKU2-MESSAGE](../14/message.md)).
- `receiverTime`: the UNIX time in nanoseconds at which the `WakuMessage` is received by the receiving node.
- `senderTime`: the UNIX time in nanoseconds at which the `WakuMessage` is generated by its sender.
- `pubsubTopic`: the pubsub topic on which the `WakuMessage` is received.
#### Waku Message Key-Value Pairs
#### PagingInfo
The store query protocol operates as a query protocol for a key-value store of historical messages,
with each entry having a [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)
and associated `pubsub_topic` as the value,
and [deterministic message hash](/waku/standards/core/14/message.md#deterministic-message-hashing) as the key.
The store can be queried to return either a set of keys or a set of key-value pairs.
`PagingInfo` holds the information required for pagination. It consists of the following components.
- `pageSize`: A positive integer indicating the number of queried `WakuMessage`s in a `HistoryQuery`
(or retrieved `WakuMessage`s in a `HistoryResponse`).
- `cursor`: holds the `Index` of a `WakuMessage`.
- `direction`: indicates the direction of paging which can be either `FORWARD` or `BACKWARD`.
Within the store query protocol,
the [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) keys and
values MUST be represented in a `WakuMessageKeyValue` message.
This message MUST contain the deterministic `message_hash` as the key.
It MAY contain the full [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) and
associated pubsub topic as the value in the `message` and
`pubsub_topic` fields, depending on the use case as set out below.
#### ContentFilter
`ContentFilter` carries the information required for filtering historical messages.
- `contentTopic` represents the content topic of the queried historical `WakuMessage`.
This field maps to the `contentTopic` field of the [14/WAKU2-MESSAGE](../14/message.md).
#### HistoryQuery
If the message contains a value entry in addition to the key,
both the `message` and `pubsub_topic` fields MUST be populated.
The message MUST NOT have either `message` or `pubsub_topic` populated with the other unset.
Both fields MUST either be set or unset.
RPC call to query historical messages.
#### Waku Message Store Eligibility
- The `pubsubTopic` field MUST indicate the pubsub topic of the historical messages to be retrieved.
This field denotes the pubsub topic on which `WakuMessage`s are published.
This field maps to `topicIDs` field of `Message` in [`11/WAKU2-RELAY`](../11/relay.md).
Leaving this field empty means no filter on the pubsub topic of message history is requested.
This field SHOULD be left empty in order to retrieve the historical `WakuMessage` regardless of the pubsub topics on which they are published.
- The `contentFilters` field MUST indicate the list of content filters based on which the historical messages are to be retrieved.
Leaving this field empty means no filter on the content topic of message history is required.
This field SHOULD be left empty in order to retrieve historical `WakuMessage` regardless of their content topics.
- `PagingInfo` holds the information required for pagination.
Its `pageSize` field indicates the number of `WakuMessage`s to be included in the corresponding `HistoryResponse`.
It is RECOMMENDED that the queried node defines a maximum page size internally.
If the querying node leaves the `pageSize` unspecified,
or if the `pageSize` exceeds the maximum page size,
the queried node SHOULD auto-paginate the `HistoryResponse` to no more than the configured maximum page size.
This allows mitigation of long response time for `HistoryQuery`.
In the forward pagination request,
the `messages` field of the `HistoryResponse` SHALL contain, at maximum,
the `pageSize` amount of `WakuMessage` whose `Index` values are larger than the given `cursor`
(and vise versa for the backward pagination).
Note that the `cursor` of a `HistoryQuery` MAY be empty (e.g., for the initial query), as such, and
depending on whether the `direction` is `BACKWARD` or `FORWARD` the last or the first `pageSize` `WakuMessage` SHALL be returned, respectively.
In order for a message to be eligible for storage:
#### Sorting Messages
The queried node MUST sort the `WakuMessage` based on their `Index`,
where the `senderTime` constitutes the most significant part and the `digest` comes next, and
then perform pagination on the sorted result.
As such, the retrieved page contains an ordered list of `WakuMessage` from the oldest messages to the most recent one.
Alternatively, the `receiverTime` (instead of `senderTime` ) MAY be used to sort messages during the paging process.
However, it is RECOMMENDED the use of the `senderTime` for sorting as it is invariant and
consistent across all the nodes.
This has the benefit of `cursor` reusability i.e.,
a `cursor` obtained from one node can be consistently used to query from another node.
However, this `cursor` reusability does not hold when the `receiverTime` is utilized as the receiver time is affected by the network delay and
nodes' clock asynchrony.
- it MUST be a _valid_ [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md).
- the `timestamp` field MUST be populated with the Unix epoch time,
at which the message was generated in nanoseconds.
If at the time of storage the `timestamp` deviates by more than 20 seconds
either into the past or the future when compared to the store nodes internal clock,
the store node MAY reject the message.
- the `ephemeral` field MUST be set to `false`.
#### HistoryResponse
#### Waku message sorting
RPC call to respond to a HistoryQuery call.
- The `messages` field MUST contain the messages found,
these are [14/WAKU2-MESSAGE](../14/message.md) types.
- `PagingInfo` holds the paging information based on which the querying node can resume its further history queries.
The `pageSize` indicates the number of returned Waku messages (i.e., the number of messages included in the `messages` field of `HistoryResponse`).
The `direction` is the same direction as in the corresponding `HistoryQuery`.
In the forward pagination, the `cursor` holds the `Index` of the last message in the `HistoryResponse` `messages` (and the first message in the backward paging).
Regardless of the paging direction, the retrieved `messages` are always sorted in ascending order based on their timestamp as explained in the [sorting messages](#sorting-messages) section, that is, from the oldest to the most recent.
The requester SHALL embed the returned `cursor` inside its next `HistoryQuery` to retrieve the next page of the [14/WAKU2-MESSAGE](../14/message.md).
The `cursor` obtained from one node SHOULD NOT be used in a request to another node because the result may be different.
- The `error` field contains information about any error that has occurred while processing the corresponding `HistoryQuery`.
`NONE` stands for no error.
This is also the default value.
`INVALID_CURSOR` means that the `cursor` field of `HistoryQuery` does not match with the `Index` of any of the `WakuMessage` persisted by the queried node.
The key-value entries in the store MUST be time-sorted by the [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) `timestamp` attribute.
Where two or more key-value entries have identical `timestamp` values,
the entries MUST be further sorted by the natural order of their message hash keys.
Within the context of traversing over key-value entries in the store,
_"forward"_ indicates traversing the entries in ascending order,
whereas _"backward"_ indicates traversing the entries in descending order.
## Security Consideration
#### Pagination
The main security consideration to take into account while using this protocol is that a querying node have to reveal their content filters of interest to the queried node, hence potentially compromising their privacy.
If a large number of entries in the store service node match the query criteria provided in a `StoreQueryRequest`,
the client MAY make use of pagination
in a chain of store query request and response transactions
to retrieve the full response in smaller batches termed _"pages"_.
Pagination can be performed either in [a _forward_ or _backward_ direction](#waku-message-sorting).
## Future Work
A store query client MAY indicate the maximum number of matching entries it wants in the `StoreQueryResponse`,
by setting the page size limit in the `pagination_limit` field.
Note that a store service node MAY enforce its own limit
if the `pagination_limit` is unset
or larger than the service node's internal page size limit.
A `StoreQueryResponse` with a populated `pagination_cursor` indicates that more stored entries match the query than included in the response.
A `StoreQueryResponse` without a populated `pagination_cursor` indicates that
there are no more matching entries in the store.
The client MAY request the next page of entries from the store service node
by populating a subsequent `StoreQueryRequest` with the `pagination_cursor`
received in the `StoreQueryResponse`.
All other fields and query criteria MUST be the same as in the preceding `StoreQueryRequest`.
A `StoreQueryRequest` without a populated `pagination_cursor` indicates that
the client wants to retrieve the "first page" of the stored entries matching the query.
### Store Query Request
A client node MUST send all historical message queries within a `StoreQueryRequest` message.
This request MUST contain a `request_id`.
The `request_id` MUST be a uniquely generated string.
If the store query client requires the store service node to include [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) values in the query response,
it MUST set `include_data` to `true`.
If the store query client requires the store service node to return only message hash keys in the query response,
it SHOULD set `include_data` to `false`.
By default, therefore, the store service node assumes `include_data` to be `false`.
A store query client MAY include query filter criteria in the `StoreQueryRequest`.
There are two types of filter use cases:
1. Content filtered queries and
2. Message hash lookup queries
#### Content filtered queries
A store query client MAY request the store service node to filter historical entries by a content filter.
Such a client MAY create a filter on content topic, on time range or on both.
To filter on content topic,
the client MUST populate _both_ the `pubsub_topic` _and_ `content_topics` field.
The client MUST NOT populate either `pubsub_topic` or
`content_topics` and leave the other unset.
Both fields MUST either be set or unset.
A mixed content topic filter with just one of either `pubsub_topic` or
`content_topics` set, SHOULD be regarded as an invalid request.
To filter on time range, the client MUST set `time_start`, `time_end` or both.
Each `time_` field should contain a Unix epoch timestamp in nanoseconds.
An unset `time_start` SHOULD be interpreted as "from the oldest stored entry".
An unset `time_end` SHOULD be interpreted as "up to the youngest stored entry".
If any of the content filter fields are set,
namely `pubsub_topic`, `content_topic`, `time_start`, or `time_end`,
the client MUST NOT set the `message_hashes` field.
#### Message hash lookup queries
A store query client MAY request the store service node to filter historical entries by one or
more matching message hash keys.
This type of query acts as a "lookup" against a message hash key or
set of keys already known to the client.
In order to perform a lookup query,
the store query client MUST populate the `message_hashes` field with the list of message hash keys it wants to lookup in the store service node.
If the `message_hashes` field is set,
the client MUST NOT set any of the content filter fields,
namely `pubsub_topic`, `content_topic`, `time_start`, or `time_end`.
#### Presence queries
A presence query is a special type of lookup query that allows a client to check for the presence of one or
more messages in the store service node,
without retrieving the full contents (values) of the messages.
This can, for example, be used as part of a reliability mechanism,
whereby store query clients verify that previously published messages have been successfully stored.
In order to perform a presence query,
the store query client MUST populate the `message_hashes` field in the `StoreQueryRequest` with the list of message hashes
for which it wants to verify presence in the store service node.
The `include_data` property MUST be set to `false`.
The client SHOULD interpret every `message_hash` returned in the `messages` field of the `StoreQueryResponse` as present in the store.
The client SHOULD assume that all other message hashes included in the original `StoreQueryRequest` but
not in the `StoreQueryResponse` is not present in the store.
#### Pagination info
The store query client MAY include a message hash as `pagination_cursor`,
to indicate at which key-value entry a store service node SHOULD start the query.
The `pagination_cursor` is treated as exclusive
and the corresponding entry will not be included in subsequent store query responses.
For forward queries,
only messages following (see [sorting](#waku-message-sorting)) the one indexed at `pagination_cursor`
will be returned.
For backward queries,
only messages preceding (see [sorting](#waku-message-sorting)) the one indexed at `pagination_cursor`
will be returned.
If the store query client requires the store service node to perform a forward query,
it MUST set `pagination_forward` to `true`.
If the store query client requires the store service node to perform a backward query,
it SHOULD set `pagination_forward` to `false`.
By default, therefore, the store service node assumes pagination to be backward.
A store query client MAY indicate the maximum number of matching entries it wants in the `StoreQueryResponse`,
by setting the page size limit in the `pagination_limit` field.
Note that a store service node MAY enforce its own limit
if the `pagination_limit` is unset
or larger than the service node's internal page size limit.
See [pagination](#pagination) for more on how the pagination info is used in store transactions.
### Store Query Response
In response to any `StoreQueryRequest`,
a store service node SHOULD respond with a `StoreQueryResponse` with a `requestId` matching that of the request.
This response MUST contain a `status_code` indicating if the request was successful or not.
Successful status codes are in the `2xx` range.
A client node SHOULD consider all other status codes as error codes and
assume that the requested operation had failed.
In addition,
the store service node MAY choose to provide a more detailed status description in the `status_desc` field.
#### Filter matching
For [content filtered queries](#content-filtered-queries),
an entry in the store service node matches the filter criteria in a `StoreQueryRequest` if each of the following conditions are met:
- its `content_topic` is in the request `content_topics` set
and it was published on a matching `pubsub_topic` OR the request `content_topics` and
`pubsub_topic` fields are unset
- its `timestamp` is _larger or equal_ than the request `start_time` OR the request `start_time` is unset
- its `timestamp` is _smaller_ than the request `end_time` OR the request `end_time` is unset
Note that for content filtered queries, `start_time` is treated as _inclusive_ and
`end_time` is treated as _exclusive_.
For [message hash lookup queries](#message-hash-lookup-queries),
an entry in the store service node matches the filter criteria if its `message_hash` is in the request `message_hashes` set.
The store service node SHOULD respond with an error code and
discard the request if the store query request contains both content filter criteria
and message hashes.
#### Populating response messages
The store service node SHOULD populate the `messages` field in the response
only with entries matching the filter criteria provided in the corresponding request.
Regardless of whether the response is to a _forward_ or _backward_ query,
the `messages` field in the response MUST be ordered in a forward direction
according to the [message sorting rules](#waku-message-sorting).
If the corresponding `StoreQueryRequest` has `include_data` set to true,
the service node SHOULD populate both the `message_hash` and
`message` for each entry in the response.
In all other cases,
the store service node SHOULD populate only the `message_hash` field for each entry in the response.
#### Paginating the response
The response SHOULD NOT contain more `messages` than the `pagination_limit` provided in the corresponding `StoreQueryRequest`.
It is RECOMMENDED that the store node defines its own maximum page size internally.
If the `pagination_limit` in the request is unset,
or exceeds this internal maximum page size,
the store service node SHOULD ignore the `pagination_limit` field and
apply its own internal maximum page size.
In response to a _forward_ `StoreQueryRequest`:
- if the `pagination_cursor` is set,
the store service node SHOULD populate the `messages` field
with matching entries following the `pagination_cursor` (exclusive).
- if the `pagination_cursor` is unset,
the store service node SHOULD populate the `messages` field
with matching entries from the first entry in the store.
- if there are still more matching entries in the store
after the maximum page size is reached while populating the response,
the store service node SHOULD populate the `pagination_cursor` in the `StoreQueryResponse`
with the message hash key of the _last_ entry _included_ in the response.
In response to a _backward_ `StoreQueryRequest`:
- if the `pagination_cursor` is set,
the store service node SHOULD populate the `messages` field
with matching entries preceding the `pagination_cursor` (exclusive).
- if the `pagination_cursor` is unset,
the store service node SHOULD populate the `messages` field
with matching entries from the last entry in the store.
- if there are still more matching entries in the store
after the maximum page size is reached while populating the response,
the store service node SHOULD populate the `pagination_cursor` in the `StoreQueryResponse`
with the message hash key of the _first_ entry _included_ in the response.
### Security Consideration
The main security consideration while using this protocol is that a querying node has to reveal its content filters of interest to the queried node,
hence potentially compromising their privacy.
#### Adversarial Model
Any peer running the `WAKU2-STORE` protocol, i.e.
both the querying node and the queried node, are considered as an adversary.
Furthermore,
we currently consider the adversary as a passive entity that attempts to collect information from other peers to conduct an attack but
it does so without violating protocol definitions and instructions.
As we evolve the protocol,
further adversarial models will be considered.
For example, under the passive adversarial model,
no malicious node hides or
lies about the history of messages as it is against the description of the `WAKU2-STORE` protocol.
The following are not considered as part of the adversarial model:
- An adversary with a global view of all the peers and their connections.
- An adversary that can eavesdrop on communication links between arbitrary pairs of peers (unless the adversary is one end of the communication).
Specifically, the communication channels are assumed to be secure.
### Future Work
- **Anonymous query**: This feature guarantees that nodes can anonymously query historical messages from other nodes i.e.,
without disclosing the exact topics of [14/WAKU2-MESSAGE](../14/message.md) they are interested in.
As such, no adversary in the `13/WAKU2-STORE` protocol would be able to learn which peer is interested in which content filters i.e.,
content topics of [14/WAKU2-MESSAGE](../14/message.md).
The current version of the `13/WAKU2-STORE` protocol does not provide anonymity for historical queries,
as the querying node needs to directly connect to another node in the `13/WAKU2-STORE` protocol and
explicitly disclose the content filters of its interest to retrieve the corresponding messages.
However, one can consider preserving anonymity through one of the following ways:
- By hiding the source of the request i.e., anonymous communication.
That is the querying node shall hide all its PII in its history request e.g., its IP address.
This can happen by the utilization of a proxy server or by using Tor.
Note that the current structure of historical requests does not embody any piece of PII, otherwise,
such data fields must be treated carefully to achieve query anonymity.
<!-- TODO: if nodes have to disclose their PeerIDs (e.g., for authentication purposes) when connecting to other nodes in the store protocol, then Tor does not preserve anonymity since it only helps in hiding the IP. So, the PeerId usage in switches must be investigated further. Depending on how PeerId is used, one may be able to link between a querying node and its queried topics despite hiding the IP address-->
- By deploying secure 2-party computations in which the querying node obtains the historical messages of a certain topic,
the queried node learns nothing about the query.
Examples of such 2PC protocols are secure one-way Private Set Intersections (PSI).
<!-- TODO: add a reference for PSIs? --> <!-- TODO: more techniques to be included -->
<!-- TODO: Censorship resistant: this is about a node that hides the historical messages from other nodes. This attack is not included in the specs since it does not fit the passive adversarial model (the attacker needs to deviate from the store protocol).-->
without disclosing the exact topics of [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) they are interested in.
As such, no adversary in the `WAKU2-STORE` protocol would be able to learn which peer is interested in which content filters i.e.,
content topics of [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md).
The current version of the `WAKU2-STORE` protocol does not provide anonymity for historical queries,
as the querying node needs to directly connect to another node in the `WAKU2-STORE` protocol and
explicitly disclose the content filters of its interest to retrieve the corresponding messages.
However, one can consider preserving anonymity through one of the following ways:
- **Robust and verifiable timestamps**: Messages timestamp is a way to show that the message existed prior to some point in time.
- By hiding the source of the request i.e., anonymous communication.
That is the querying node shall hide all its PII in its history request e.g.,
its IP address.
This can happen by the utilization of a proxy server or by using Tor.
Note that the current structure of historical requests does not embody any piece of PII, otherwise,
such data fields must be treated carefully to achieve query anonymity.
<!-- TODO: if nodes have to disclose their PeerIDs
(e.g., for authentication purposes) when connecting to other nodes in the store protocol,
then Tor does not preserve anonymity since it only helps in hiding the IP.
So, the PeerId usage in switches must be investigated further.
Depending on how PeerId is used, one may be able to link between a querying node
and its queried topics despite hiding the IP address-->
- By deploying secure 2-party computations
in which the querying node obtains the historical messages of a certain topic,
the queried node learns nothing about the query.
Examples of such 2PC protocols are secure one-way Private Set Intersections (PSI).
<!-- TODO: add a reference for PSIs? --> <!-- TODO: more techniques to be included -->
<!-- TODO: Censorship resistant:
this is about a node that hides the historical messages from other nodes.
This attack is not included in the specs since it does not fit the
passive adversarial model (the attacker needs to deviate from the store protocol).-->
- **Robust and verifiable timestamps**: Messages timestamp is a way to show that
the message existed prior to some point in time.
However, the lack of timestamp verifiability can create room for a range of attacks,
including injecting messages with invalid timestamps pointing to the far future.
including injecting messages with invalid timestamps pointing to the far future.
To better understand the attack,
consider a store node whose current clock shows `2021-01-01 00:00:30` (and assume all the other nodes have a synchronized clocks +-20seconds).
consider a store node whose current clock shows `2021-01-01 00:00:30`
(and assume all the other nodes have a synchronized clocks +-20seconds).
The store node already has a list of messages,
`(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20)`,
that are sorted based on their timestamp.
An attacker sends a message with an arbitrary large timestamp e.g.,
10 hours ahead of the correct clock `(m',2021-01-01 10:00:30)`.
10 hours ahead of the correct clock `(m',2021-01-01 10:00:30)`.
The store node places `m'` at the end of the list,
`(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20), (m',2021-01-01 10:00:30)`.
`(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20),
(m',2021-01-01 10:00:30)`.
Now another message arrives with a valid timestamp e.g.,
`(m11, 2021-01-01 00:00:45)`.
However, since its timestamp precedes the malicious message `m'`,
it gets placed before `m'` in the list i.e.,
`(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20), (m11, 2021-01-01 00:00:45), (m',2021-01-01 10:00:30)`.
`(m1,2021-01-01 00:00:00), (m2,2021-01-01 00:00:01), ..., (m10:2021-01-01 00:00:20),
(m11, 2021-01-01 00:00:45), (m',2021-01-01 10:00:30)`.
In fact, for the next 10 hours,
`m'` will always be considered as the most recent message and
served as the last message to the querying nodes irrespective of how many other messages arrive afterward.
served as the last message to the querying nodes irrespective of how many other
messages arrive afterward.
A robust and verifiable timestamp allows the receiver of a message to verify that a message has been generated prior to the claimed timestamp.
A robust and verifiable timestamp allows the receiver of a message to verify that
a message has been generated prior to the claimed timestamp.
One solution is the use of [open timestamps](https://opentimestamps.org/) e.g.,
block height in Blockchain-based timestamps.
That is, messages contain the most recent block height perceived by their senders at the time of message generation.
This proves accuracy within a range of minutes (e.g., in Bitcoin blockchain) or
seconds (e.g., in Ethereum 2.0) from the time of origination.
block height in Blockchain-based timestamps.
That is, messages contain the most recent block height perceived by their senders
at the time of message generation.
This proves accuracy within a range of minutes (e.g., in Bitcoin blockchain) or
seconds (e.g., in Ethereum 2.0) from the time of origination.
## Copyright
@@ -255,7 +437,7 @@ Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
1. [14/WAKU2-MESSAGE](../14/message.md)
1. [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)
2. [protocol buffers v3](https://developers.google.com/protocol-buffers/)
3. [11/WAKU2-RELAY](../11/relay.md)
4. [Open timestamps](https://opentimestamps.org/)
3. [Open timestamps](https://opentimestamps.org/)

View File

@@ -2,7 +2,7 @@
slug: 14
title: 14/WAKU2-MESSAGE
name: Waku v2 Message
status: draft
status: stable
category: Standards Track
tags: waku/core-protocol
editor: Hanno Cornelius <hanno@status.im>
@@ -16,12 +16,17 @@ contributors:
## Abstract
Waku v2 is a family of modular peer-to-peer protocols for secure communication.
These protocols are designed to be secure, privacy-preserving, and censorship-resistant and can run in resource-restricted environments.
At a high level, Waku v2 implements a Pub/Sub messaging pattern over libp2p and adds capabilities.
The present document specifies the Waku v2 message format, a way to encapsulate the messages sent with specific information security goals, and Whisper/Waku v1 backward compatibility.
[10/WAKU2](/waku/standards/core/10/waku2.md) is a family of modular peer-to-peer protocols
for secure communication.
These protocols are designed to be secure, privacy-preserving,
and censorship-resistant and can run in resource-restricted environments.
At a high level,
[10/WAKU2](/waku/standards/core/10/waku2.md) implements a publish/subscribe messaging pattern over libp2p and
adds capabilities.
The present document specifies the [10/WAKU2](/waku/standards/core/10/waku2.md) message format.
A way to encapsulate the messages sent with specific information security goals,
and Whisper/[6/WAKU1](/waku/standards/legacy/6/waku1.md) backward compatibility.
## Motivation
@@ -29,48 +34,65 @@ When sending messages over Waku, there are multiple requirements:
- One may have a separate encryption layer as part of the application.
- One may want to provide efficient routing for resource-restricted devices.
- One may want to provide compatibility with [Waku v1 envelopes](../../legacy/6/waku1.md).
- One may want to provide compatibility with [6/WAKU1](/waku/standards/legacy/6/waku1.md) envelopes.
- One may want encrypted payloads by default.
- One may want to provide unlinkability to get metadata protection.
This specification attempts to provide for these various requirements.
## Semantics
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
“SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
### Waku Message
A Waku message is constituted by the combination of data payload and attributes that, for example, a *publisher* sends to a *topic* and is eventually delivered to *subscribers*.
A `WakuMessage` is constituted by the combination of data payload and
attributes that, for example, a *publisher* sends to a *topic* and
is eventually delivered to *subscribers*.
Waku message attributes are key-value pairs of metadata associated with a message.
And the message data payload is the part of the transmitted Waku message that is the actual message information.
The data payload is also treated as a Waku message attribute for convenience.
The `WakuMessage` attributes are key-value pairs of metadata associated with a message.
The message data payload is the part of the transmitted `WakuMessage`
that is the actual message information.
The data payload is also treated as a `WakuMessage` attribute for convenience.
### Message Attributes
* The `payload` attribute MUST contain the message data payload to be sent.
- The `payload` attribute MUST contain the message data payload to be sent.
* The `content_topic` attribute MUST specify a string identifier that can be used for content-based filtering,
as described in [23/WAKU2-TOPICS](../../../informational/23/topics.md).
- The `content_topic` attribute MUST specify a string identifier
that can be used for content-based filtering,
as described in [23/WAKU2-TOPICS](/waku/informational/23/topics.md).
* The `meta` attribute, if present, contains an arbitrary application-specific variable-length byte array with a maximum length limit of 64 bytes.
This attribute can be utilized to convey supplementary details to various Waku protocols, thereby enabling customized processing based on its contents.
- The `meta` attribute, if present,
contains an arbitrary application-specific variable-length byte array
with a maximum length limit of 64 bytes.
This attribute can be utilized to convey supplementary details
to various [10/WAKU2](/waku/standards/core/10/waku2.md) protocols,
thereby enabling customized processing based on its contents.
* The `version` attribute, if present, contains a version number to discriminate different types of payload encryption.
- The `version` attribute, if present,
contains a version number to discriminate different types of payload encryption.
If omitted, the value SHOULD be interpreted as version 0.
* The `timestamp` attribute, if present, signifies the time at which the message was generated by its sender.
- The `timestamp` attribute, if present,
signifies the time at which the message was generated by its sender.
This attribute MAY contain the Unix epoch time in nanoseconds.
If the attribute is omitted, it SHOULD be interpreted as timestamp 0.
* The `ephemeral` attribute, if present, signifies the transient nature of the message.
For example, an ephemeral message SHOULD not be persisted by the Waku network.
- The `rate_limit_proof` attribute, if present,
contains a rate limit proof encoded as per [17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md).
- The `ephemeral` attribute, if present, signifies the transient nature of the message.
For example, an ephemeral message SHOULD not be persisted by other nodes on the same network.
If this attribute is set to `true`, the message SHOULD be interpreted as ephemeral.
If, instead, the attribute is omitted or set to `false`, the message SHOULD be interpreted as non-ephemeral.
If, instead, the attribute is omitted or set to `false`,
the message SHOULD be interpreted as non-ephemeral.
## Wire Format
The Waku message wire format is specified using [protocol buffers v3](https://developers.google.com/protocol-buffers/).
The `WakuMessage` wire format is specified using [protocol buffers v3](https://developers.google.com/protocol-buffers/).
```protobuf
syntax = "proto3";
@@ -81,64 +103,80 @@ message WakuMessage {
optional uint32 version = 3;
optional sint64 timestamp = 10;
optional bytes meta = 11;
optional bytes rate_limit_proof = 21;
optional bool ephemeral = 31;
}
```
An example proto file following this specification can be found [here (vacp2p/waku)](https://github.com/vacp2p/waku/blob/main/waku/message/v1/message.proto).
## Payload encryption
The Waku message payload MAY be encrypted.
The message `version` attribute indicates the schema used to encrypt the payload data.
The `WakuMessage` payload MAY be encrypted.
The message `version` attribute indicates
the schema used to encrypt the payload data.
- **Version 0:**
The payload SHOULD be interpreted as unencrypted; additionally, it CAN indicate that the message payload has been encrypted at the application layer.
The payload SHOULD be interpreted as unencrypted; additionally,
it CAN indicate that the message payload has been encrypted
at the application layer.
- **Version 1:**
The payload SHOULD be encrypted using Waku v1 payload encryption specified in [26/WAKU-PAYLOAD](../../application/26/payload.md).
The payload SHOULD be encrypted using [6/WAKU1](/waku/standards/legacy/6/waku1.md) payload encryption specified in [26/WAKU-PAYLOAD](/waku/standards/application/26/payload.md).
This provides asymmetric and symmetric encryption.
The key agreement is performed out of band.
And provides an encrypted signature and padding for some form of unlinkability.
- **Version 2:**
The payload SHOULD be encoded according to [35/WAKU2-NOISE]([/spec/35](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/noise.md)).
Waku Noise protocol provides symmetric encryption and asymmetric key exchange.
The payload SHOULD be encoded according to [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md).
The Waku Noise protocol provides symmetric encryption and asymmetric key exchange.
Any `version` value not included in this list is reserved for future specification.
And, in this case, the payload SHOULD be interpreted as unencrypted by the Waku layer.
## Whisper/[6/WAKU1](/waku/standards/legacy/6/waku1.md) envelope compatibility
## Whisper/Waku v1 envelope compatibility
Whisper/[6/WAKU1](/waku/standards/legacy/6/waku1.md) envelopes are compatible with Waku messages format.
Whisper/Waku v1 envelopes are compatible with Waku v2 messages format.
- Whisper/[6/WAKU1](/waku/standards/legacy/6/waku1.md) `topic` field
SHOULD be mapped to Waku message's `content_topic` attribute.
- Whisper/[6/WAKU1](/waku/standards/legacy/6/waku1.md) `data` field SHOULD be mapped to Waku message's `payload` attribute.
* Whisper/Waku v1 `topic` field SHOULD be mapped to Waku v2 message's `content_topic` attribute.
* Whisper/Waku v1 `data` field SHOULD be mapped to Waku v2 message's `payload` attribute.
Waku v2 implements a pub/sub messaging pattern over libp2p.
This makes redundant some Whisper/Waku v1 envelope fields (e.g., `expiry`, `ttl`, `topic`, etc.), so they can be ignored.
[10/WAKU2](/waku/standards/core/10/waku2.md) implements a publish/subscribe messaging pattern over libp2p.
This makes some Whisper/[6/WAKU1](/waku/standards/legacy/6/waku1.md) envelope fields redundant
(e.g., `expiry`, `ttl`, `topic`, etc.), so they can be ignored.
## Deterministic message hashing
In Protocol Buffers v3, the deterministic serialization is not canonical across the different implementations and languages.
It is also unstable across different builds with schema changes due to unknown fields.
In Protocol Buffers v3,
the deterministic serialization is not canonical across the different implementations
and languages.
It is also unstable across different builds with schema changes due to unknown fields.
To overcome this interoperability limitation, a Waku v2 message's hash MUST be computed following this schema:
To overcome this interoperability limitation,
a [10/WAKU2](/waku/standards/core/10/waku2.md) message's hash MUST be computed following this schema:
```
```js
message_hash = sha256(concat(pubsub_topic, message.payload, message.content_topic, message.meta, message.timestamp))
```
If an optional attribute, such as `meta`, is absent, the concatenation of attributes SHOULD exclude it. This recommendation is made to ensure that the concatenation process proceeds smoothly when certain attributes are missing and to maintain backward compatibility.
If an optional attribute, such as `meta`, is absent,
the concatenation of attributes SHOULD exclude it.
This recommendation is made to ensure that the concatenation process proceeds smoothly
when certain attributes are missing and to maintain backward compatibility.
This hashing schema is deemed appropriate for use cases where a cross-implementation deterministic hash is needed, such as message deduplication and integrity validation. The collision probability offered by this hashing schema can be considered negligible. This is due to the deterministic concatenation order of the message attributes, coupled with using a SHA-2 (256-bit) hashing algorithm.
This hashing schema is deemed appropriate for use cases
where a cross-implementation deterministic hash is needed,
such as message deduplication and integrity validation.
The collision probability offered by this hashing schema can be considered negligible.
This is due to the deterministic concatenation order of the message attributes,
coupled with using a SHA-2 (256-bit) hashing algorithm.
### Test vectors
Waku message hash computation (`meta` size of 12 bytes):
```
The `WakuMessage` hash computation (`meta` size of 12 bytes):
```js
pubsub_topic = "/waku/2/default-waku/proto" (0x2f77616b752f322f64656661756c742d77616b752f70726f746f)
message.payload = 0x010203045445535405060708
message.content_topic = "/waku/2/default-content/proto" (0x2f77616b752f322f64656661756c742d636f6e74656e742f70726f746f)
@@ -148,9 +186,9 @@ message.timestamp = 0x175789bfa23f8400
message_hash = 0x64cce733fed134e83da02b02c6f689814872b1a0ac97ea56b76095c3c72bfe05
```
Waku message hash computation (`meta` size of 64 bytes):
The `WakuMessage` hash computation (`meta` size of 64 bytes):
```
```js
pubsub_topic = "/waku/2/default-waku/proto" (0x2f77616b752f322f64656661756c742d77616b752f70726f746f)
message.payload = 0x010203045445535405060708
message.content_topic = "/waku/2/default-content/proto" (0x2f77616b752f322f64656661756c742d636f6e74656e742f70726f746f)
@@ -160,8 +198,9 @@ message.timestamp = 0x175789bfa23f8400
message_hash = 0x7158b6498753313368b9af8f6e0a0a05104f68f972981da42a43bc53fb0c1b27
```
Waku message hash computation (`meta` attribute not present):
```
The `WakuMessage` hash computation (`meta` attribute not present):
```js
pubsub_topic = "/waku/2/default-waku/proto" (0x2f77616b752f322f64656661756c742d77616b752f70726f746f)
message.payload = 0x010203045445535405060708
message.content_topic = "/waku/2/default-content/proto" (0x2f77616b752f322f64656661756c742d636f6e74656e742f70726f746f)
@@ -171,8 +210,9 @@ message.timestamp = 0x175789bfa23f8400
message_hash = 0xa2554498b31f5bcdfcbf7fa58ad1c2d45f0254f3f8110a85588ec3cf10720fd8
```
Waku message hash computation (`payload` length 0):
```
The `WakuMessage` hash computation (`payload` length 0):
```js
pubsub_topic = "/waku/2/default-waku/proto" (0x2f77616b752f322f64656661756c742d77616b752f70726f746f)
message.payload = []
message.content_topic = "/waku/2/default-content/proto" (0x2f77616b752f322f64656661756c742d636f6e74656e742f70726f746f)
@@ -186,33 +226,48 @@ message_hash = 0x483ea950cb63f9b9d6926b262bb36194d3f40a0463ce8446228350bd44e96de
### Confidentiality, integrity, and authenticity
The level of confidentiality, integrity, and authenticity of the Waku message payload is discretionary.
Accordingly, the application layer shall utilize the encryption and signature schemes supported by Waku v2 to meet the application-specific privacy needs.
The level of confidentiality, integrity, and
authenticity of the `WakuMessage` payload is discretionary.
Accordingly, the application layer shall utilize the encryption and
signature schemes supported by [10/WAKU2](/waku/standards/core/10/waku2.md),
to meet the application-specific privacy needs.
### Reliability of the `timestamp` attribute
The Waku message `timestamp` attribute is set by the sender.
Therefore, because message timestamps arent independently verified, this attribute is prone to exploitation and misuse.
The message `timestamp` attribute is set by the sender.
Therefore, because message timestamps arent independently verified,
this attribute is prone to exploitation and misuse.
It should not solely be relied upon for operations such as message ordering.
For example, a malicious actor can arbitrarily set the `timestamp` of a Waku message to a high value so that it always shows up as the most recent message in a chat application.
Applications using Waku messages `timestamp` attribute are recommended to use additional methods for more robust message ordering.
An example of how to deal with message ordering against adversarial message timestamps can be found in the Status protocol, see [62/STATUS-PAYLOADS](../../../../status/62/payloads.md/#clock-vs-timestamp-and-message-ordering).
For example, a malicious actor can arbitrarily set the `timestamp` of a `WakuMessage`
to a high value so that it always shows up as the most recent message
in a chat application.
Applications using [10/WAKU2](/waku/standards/core/10/waku2.md) messages `timestamp` attribute
are RECOMMENDED to use additional methods for more robust message ordering.
An example of how to deal with message ordering against adversarial message timestamps
can be found in the Status protocol,
see [62/STATUS-PAYLOADS](/status/62/payloads.md/#clock-vs-timestamp-and-message-ordering).
### Reliability of the `ephemeral` attribute
The Waku message `ephemeral` attribute is set by the sender.
Since there is currently no incentive mechanism for network participants to behave correctly, this attribute is inherently insecure.
A malicious actor can tamper with the value of a Waku messages `ephemeral` attribute, and the receiver would not be able to verify the integrity of the message.
The message `ephemeral` attribute is set by the sender.
Since there is currently no incentive mechanism
for network participants to behave correctly,
this attribute is inherently insecure.
A malicious actor can tamper with the value of a Waku messages `ephemeral` attribute,
and the receiver would not be able to verify the integrity of the message.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [6/WAKU1](../../legacy/6/waku1.md)
- [Google Protocol buffers v3](https://developers.google.com/protocol-buffers/)
- [26/WAKU-PAYLOAD](../../application/26/payload.md)
- [35/WAKU2-NOISE]([/spec/35](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/noise.md))
- [62/STATUS-PAYLOADS](../../../../status/62/payloads.md/#clock-vs-timestamp-and-message-ordering)
- [10/WAKU2](/waku/standards/core/10/waku2.md)
- [6/WAKU1](/waku/standards/legacy/6/waku1.md)
- [23/WAKU2-TOPICS](/waku/informational/23/topics.md)
- [17/WAKU2-RLN-RELAY](/waku/standards/core/17/rln-relay.md)
- [64/WAKU2-NETWORK](/waku/standards/core/64/network.md)
- [protocol buffers v3](https://developers.google.com/protocol-buffers/)
- [26/WAKU-PAYLOAD](/waku/standards/application/26/payload.md)
- [WAKU2-NOISE](https://github.com/waku-org/specs/blob/master/standards/application/noise.md)
- [62/STATUS-PAYLOADS](/status/62/payloads.md/#clock-vs-timestamp-and-message-ordering)

View File

@@ -7,24 +7,38 @@ tags: waku-core
editor: Hanno Cornelius <hanno@status.im>
---
A bridge between Waku v1 and Waku v2.
## Abstract
## Bridge
This specification describes how [6/WAKU1](/waku/standards/legacy/6/waku1.md)
traffic can be used with [10/WAKU2](/waku/standards/core/10/waku2.md) networks.
## Wire Format
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
A bridge requires supporting both Waku versions:
* Waku v1 - using devp2p RLPx protocol
* Waku v2 - using libp2p protocols
* [6/WAKU1](/waku/standards/legacy/6/waku1.md) - using devp2p RLPx protocol
* [10/WAKU2](/waku/standards/core/10/waku2.md) - using libp2p protocols
Packets received on the Waku v1 network SHOULD be published just once on the
Waku v2 network. More specifically, the bridge SHOULD publish
this through the Waku Relay (PubSub domain).
## Publishing Packets
Publishing such packet will require the creation of a new `Message` with a
new `WakuMessage` as data field. The `data` and `topic` field from the Waku v1
`Envelope` MUST be copied to the `payload` and `contentTopic` fields of the
`WakuMessage`. Other fields such as nonce, expiry and ttl will be dropped as
they become obsolete in Waku v2.
Packets received on [6/WAKU1](/waku/standards/legacy/6/waku1.md) networks
SHOULD be published just once on [10/WAKU2](/waku/standards/core/10/waku2.md) networks.
More specifically, the bridge SHOULD publish
this through [11/WAKU2-RELAY](/waku/standards/core/11/relay.md) (PubSub domain).
When publishing such packet,
the creation of a new `Message` with a new `WakuMessage` as data field is REQUIRED.
The `data` and
`topic` field, from the [6/WAKU1](/waku/standards/legacy/6/waku1.md) `Envelope`,
MUST be copied to the `payload` and `content_topic` fields of the `WakuMessage`.
See [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md#wire-format)
for message format details.
Other fields such as `nonce`, `expiry` and
`ttl` will be dropped as they become obsolete in [10/WAKU2](/waku/standards/core/10/waku2.md).
Before this is done, the usual envelope verification still applies:
@@ -32,36 +46,54 @@ Before this is done, the usual envelope verification still applies:
* PoW verification
* Size verification
Bridging SHOULD occur through the `WakuRelay`, but it MAY also be done on other Waku
v2 protocols (e.g. `WakuFilter`). The latter is however not advised as it will
increase the complexity of the bridge and because of the
[Security Considerations](#security-considerations) explained further below.
Bridging SHOULD occur through the [11/WAKU2-RELAY](/waku/standards/core/11/relay.md),
but it MAY also be done on other [10/WAKU2](/waku/standards/core/10/waku2.md) protocols
(e.g. [12/WAKU2-FILTER](/waku/standards/core/12/filter.md)).
The latter is however not advised as it will
increase the complexity of the bridge and
because of the [Security Considerations](#security-considerations) explained further below.
Packets received on the Waku v2 network SHOULD be posted just once on the Waku
v1 network. The Waku v2 `WakuMessage` contains only the `payload` and
`contentTopic` fields. The bridge MUST create a new Waku v1 `Envelope` and
copy over the `payload` and `contentFilter` fields to the `data` and `topic`
fields. Next, before posting on the network, the bridge MUST set a new expiry
and ttl and do the PoW nonce calculation.
Packets received on [10/WAKU2](/waku/standards/core/10/waku2.md) networks,
SHOULD be posted just once on [6/WAKU1](/waku/standards/legacy/6/waku1.md) networks.
The [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md) contains only the `payload` and
`contentTopic` fields.
The bridge MUST create a new [6/WAKU1](/waku/standards/legacy/6/waku1.md) `Envelope` and
copy over the `payload` and `contentFilter`
fields to the `data` and `topic` fields.
Next, before posting on the network,
the bridge MUST set a new `expiry`, `ttl` and do the PoW `nonce` calculation.
### Security Considerations
As mentioned above, a bridge will be posting new Waku v1 envelopes, which
requires doing the PoW nonce calculation.
This could be a DoS attack vector, as the PoW calculation will make it more
expensive to post the message compared to the original publishing on the Waku v2
network. Low PoW setting will lower this problem, but it is likely that it is
still more expensive.
As mentioned above,
a bridge will be posting new [6/WAKU1](/waku/standards/legacy/6/waku1.md) envelopes,
which requires doing the PoW `nonce` calculation.
For this reason, bridges SHOULD probably be run independently of other nodes, so
that a bridge that gets overwhelmed does not disrupt regular Waku v2 to v2
This could be a DoS attack vector,
as the PoW calculation will make it more expensive to post the message
compared to the original publishing on [10/WAKU2](/waku/standards/core/10/waku2.md) networks.
Low PoW setting will lower this problem,
but it is likely that it is still more expensive.
For this reason, it is RECOMMENDED to run bridges independently of other nodes,
so that a bridge that gets overwhelmed does not disrupt regular Waku v2 to v2
traffic.
Bridging functionality SHOULD also be carefully implemented so that messages do
not bounce back and forth between the two networks. The bridge SHOULD properly
track messages with a seen filter so that no amplification can be achieved here.
not bounce back and forth between the [10/WAKU2](/waku/standards/core/10/waku2.md) and
[6/WAKU1](/waku/standards/legacy/6/waku1.md) networks.
The bridge SHOULD properly track messages with a seen filter,
so that no amplification occurs.
## Copyright
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
* [6/WAKU1](/waku/standards/legacy/6/waku1.md)
* [10/WAKU2](/waku/standards/core/10/waku2.md)
* [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
* [14/WAKU2-MESSAGE](/waku/standards/core/14/message.md)
* [12/WAKU2-FILTER](/waku/standards/core/12/filter.md)

View File

@@ -13,65 +13,87 @@ contributors:
---
## Abstract
This specification describes the `17/WAKU2-RLN-RELAY` protocol,
which is an extension of [`11/WAKU2-RELAY`](../11/relay.md) to provide spam protection using [Rate Limiting Nullifiers (RLN)](../../../../vac/32/rln-v1.md).
The security objective is to contain spam activity in the (64/WAKU-NETWORK)[] by enforcing a global messaging rate to all the peers.
Peers that violate the messaging rate are considered spammers and
This specification describes the `17/WAKU2-RLN-RELAY` protocol,
which is an extension of [`11/WAKU2-RELAY`](../11/relay.md)
to provide spam protection using [Rate Limiting Nullifiers (RLN)](../../../../vac/32/rln-v1.md).
The security objective is to contain spam activity in the [64/WAKU-NETWORK](../64/network.md)
by enforcing a global messaging rate to all the peers.
Peers that violate the messaging rate are considered spammers and
their message is considered spam.
Spammers are also financially punished and removed from the system.
Spammers are also financially punished and removed from the system.
## Motivation
In open and anonymous p2p messaging networks,
one big problem is spam resistance.
In open and anonymous p2p messaging networks,
one big problem is spam resistance.
Existing solutions, such as Whispers proof of work,
are computationally expensive hence not suitable for resource-limited nodes.
Other reputation-based approaches might not be desirable,
are computationally expensive hence not suitable for resource-limited nodes.
Other reputation-based approaches might not be desirable,
due to issues around arbitrary exclusion and privacy.
We augment the [`11/WAKU2-RELAY`](../11/relay.md) protocol with a novel construct of [RLN](../../../../vac/32/rln-v1.md) to enable an efficient economic spam prevention mechanism that can be run in resource-constrained environments.
We augment the [`11/WAKU2-RELAY`](../11/relay.md) protocol
with a novel construct of [RLN](../../../../vac/32/rln-v1.md)
to enable an efficient economic spam prevention mechanism
that can be run in resource-constrained environments.
## Specification
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
“SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
### Flow
The messaging rate is defined by the `period` which indicates how many messages can be sent in a given period.
We define an `epoch` as $\lceil$ `unix_time` / `period` $\rceil$.
For example, if `unix_time` is `1644810116` and we set `period` to `30`,
The messaging rate is defined by the `period`
which indicates how many messages can be sent in a given period.
We define an `epoch` as $\lceil$ `unix_time` / `period` $\rceil$.
For example, if `unix_time` is `1644810116` and we set `period` to `30`,
then `epoch` is $\lceil$ `(unix_time/period)` $\rceil$ `= 54827003`.
> **NOTE:** The `epoch` refers to the epoch in RLN and not Unix epoch.
This means a message can only be sent every period, where the `period` is up to the application.
This means a message can only be sent every period,
where the `period` is up to the application.
See section [Recommended System Parameters](#recommended-system-parameters) for the RECOMMENDED method to set a sensible `period` value depending on the application.
Peers subscribed to a spam-protected `pubsubTopic` are only allowed to send one message per `epoch`.
The higher-level layers adopting `17/WAKU2-RLN-RELAY` MAY choose to enforce the messaging rate for `WakuMessages` with a specific `contentTopic` published on a `pubsubTopic`.
See section [Recommended System Parameters](#recommended-system-parameters)
for the RECOMMENDED method to set a sensible `period` value depending on the application.
Peers subscribed to a spam-protected `pubsubTopic`
are only allowed to send one message per `epoch`.
The higher-level layers adopting `17/WAKU2-RLN-RELAY`
MAY choose to enforce the messaging rate for `WakuMessages`
with a specific `contentTopic` published on a `pubsubTopic`.
#### Setup and Registration
A `pubsubTopic` that is spam-protected requires subscribed peers to form a [RLN group](../../../../vac/32/rln-v1.md).
- Peers MUST be registered to the RLN group to be able to publish messages.
- Registration MAY be moderated through a smart contract deployed on the Ethereum blockchain.
- Registration MAY be moderated through a smart contract
deployed on the Ethereum blockchain.
Each peer has an [RLN key pair](../../../../vac/32/rln-v1.md) denoted by `sk`
and `pk`.
Each peer has an [RLN key pair](../../../../vac/32/rln-v1.md) denoted by `sk` and `pk`.
- The secret key `sk` is secret data and MUST be persisted securely by the peer.
- The state of the membership contract SHOULD contain a list of all registered members' public identity keys i.e.,
`pk`s.
- The state of the membership contract
SHOULD contain a list of all registered members' public identity keys i.e., `pk`s.
For registration, a peer MUST create a transaction to invoke the registration function on the contract,
which registers its `pk` in the RLN group.
- The transaction MUST transfer additional tokens to the contract to be staked.
For registration,
a peer MUST create a transaction to invoke the registration function on the contract,
which registers its `pk` in the RLN group.
- The transaction MUST transfer additional tokens to the contract to be staked.
This amount is denoted by `staked_fund` and is a system parameter.
The peer who has the secret key `sk` associated with a registered `pk` would be able to withdraw a portion `reward_portion` of the staked fund by providing valid proof.
The peer who has the secret key `sk` associated with a registered `pk`
would be able to withdraw a portion `reward_portion`
of the staked fund by providing valid proof.
`reward_portion` is also a system parameter.
> **NOTE:** Initially `sk` is only known to its owning peer however,
it may get exposed to other peers in case the owner attempts spamming the system i.e.,
sending more than one message per `epoch`.
it may get exposed to other peers in case the owner attempts spamming the system
i.e., sending more than one message per `epoch`.
An overview of registration is illustrated in Figure 1.
@@ -79,28 +101,33 @@ An overview of registration is illustrated in Figure 1.
#### Publishing
To publish at a given `epoch`,
the publishing peer proceeds based on the regular [`11/WAKU2-RELAY`](../11/relay.md) protocol.
However, to protect against spamming, each `WakuMessage`
(which is wrapped inside the `data` field of a PubSub message)
MUST carry a [`RateLimitProof`](##RateLimitProof) with the following fields.
Section [Payload](#payloads) covers the details about the type and encoding of these fields.
To publish at a given `epoch`, the publishing peer proceeds
based on the regular [`11/WAKU2-RELAY`](../11/relay.md) protocol.
However, to protect against spamming, each `WakuMessage`
(which is wrapped inside the `data` field of a PubSub message)
MUST carry a [`RateLimitProof`](#ratelimitproof) with the following fields.
Section [Payload](#payloads) covers the details about the type and
encoding of these fields.
- The `merkle_root` contains the root of the Merkle tree.
- The `epoch` represents the current epoch.
- The `nullifier` is an internal nullifier acting as a fingerprint that allows specifying whether two messages are published by the same peer during the same `epoch`.
- The `nullifier` is an internal nullifier acting as a fingerprint
that allows specifying whether two messages are published by the same peer
during the same `epoch`.
- The `nullifier` is a deterministic value derived from `sk` and
`epoch` therefore any two messages issued by the same peer
(i.e., using the same `sk`) for the same `epoch` are guaranteed to have identical `nullifier`s.
- The `share_x` and `share_y` can be seen as partial disclosure of peer's `sk` for the intended `epoch`.
(i.e., using the same `sk`)
for the same `epoch` are guaranteed to have identical `nullifier`s.
- The `share_x` and
`share_y` can be seen as partial disclosure of peer's `sk` for the intended `epoch`.
They are derived deterministically from peer's `sk` and
current `epoch` using [Shamir secret sharing scheme](../../../../vac/32/rln-v1.md).
current `epoch` using [Shamir secret sharing scheme](../../../../vac/32/rln-v1.md).
If a peer discloses more than one such pair (`share_x`, `share_y`) for the same `epoch`,
it would allow full disclosure of its `sk` and
If a peer discloses more than one such pair (`share_x`, `share_y`) for the same `epoch`,
it would allow full disclosure of its `sk` and
hence get access to its staked fund in the membership contract.
- The `proof` field is a zero-knowledge proof signifying that:
- The `proof` field is a zero-knowledge proof signifying that:
1. The message owner is the current member of the group i.e.,
the peer's identity commitment key, `pk`,
@@ -108,73 +135,108 @@ is part of the membership group Merkle tree with the root `merkle_root`.
2. `share_x` and `share_y` are correctly computed.
3. The `nullifier` is constructed correctly.
For more details about the proof generation check [RLN](../../../../vac/32/rln-v1.md)
The proof generation relies on the knowledge of two pieces of private information i.e., `sk` and `authPath`.
The `authPath` is a subset of Merkle tree nodes by which a peer can prove the inclusion of its `pk` in the group. <!-- TODO refer to RLN RFC for authPath def -->
The proof generation relies on the knowledge of two pieces of private information
i.e., `sk` and `authPath`.
The `authPath` is a subset of Merkle tree nodes
by which a peer can prove the inclusion of its `pk` in the group.
<!-- TODO refer to RLN RFC for authPath def -->
The proof generation also requires a set of public inputs which are:
the Merkle tree root `merkle_root`, the current `epoch`, and
the message for which the proof is going to be generated.
In `17/WAKU2-RLN-RELAY`, the message is the concatenation of `WakuMessage`'s `payload` filed and
its `contentTopic` i.e., `payload||contentTopic`.
the message for which the proof is going to be generated.
In `17/WAKU2-RLN-RELAY`,
the message is the concatenation of `WakuMessage`'s `payload` filed and
its `contentTopic` i.e., `payload||contentTopic`.
#### Group Synchronization
Proof generation relies on the knowledge of Merkle tree root `merkle_root` and `authPath` which both require access to the membership Merkle tree.
Getting access to the Merkle tree can be done in various ways:
Proof generation relies on the knowledge of Merkle tree root `merkle_root` and
`authPath` which both require access to the membership Merkle tree.
Getting access to the Merkle tree can be done in various ways:
1. Peers construct the tree locally.
This can be done by listening to the registration and
This can be done by listening to the registration and
deletion events emitted by the membership contract.
Peers MUST update the local Merkle tree on a per-block basis.
This is discussed further in the [Merkle Root Validation](#merkle-root-validation) section.
This is discussed further
in the [Merkle Root Validation](#merkle-root-validation) section.
2. For synchronizing the state of slashed `pk`s,
disseminate such information through a `pubsubTopic` to which all peers are subscribed.
disseminate such information through a `pubsubTopic` to which all peers are subscribed.
A deletion transaction SHOULD occur on the membership contract.
The benefit of an off-chain slashing is that it allows real-time removal of spammers as opposed to on-chain slashing in which peers get informed with a delay,
The benefit of an off-chain slashing
is that it allows real-time removal of spammers as opposed to on-chain slashing
in which peers get informed with a delay,
where the delay is due to mining the slashing transaction.
For the group synchronization,
one important security consideration is that peers MUST make sure they always use the most recent Merkle tree root in their proof generation.
The reason is that using an old root can allow inference about the index of the user's `pk` in the membership tree hence compromising user privacy and breaking message unlinkability.
For the group synchronization,
one important security consideration is that peers MUST make sure they always use
the most recent Merkle tree root in their proof generation.
The reason is that using an old root can allow inference
about the index of the user's `pk` in the membership tree
hence compromising user privacy and breaking message unlinkability.
#### Routing
Upon the receipt of a PubSub message via [`11/WAKU2-RELAY`](../11/relay.md) protocol,
the routing peer parses the `data` field as a `WakuMessage` and gets access to the `RateLimitProof` field.
Upon the receipt of a PubSub message via [`11/WAKU2-RELAY`](../11/relay.md) protocol,
the routing peer parses the `data` field as a `WakuMessage` and
gets access to the `RateLimitProof` field.
The peer then validates the `RateLimitProof` as explained next.
##### Epoch Validation
If the `epoch` attached to the `WakuMessage` is more than `max_epoch_gap`,
apart from the routing peer's current `epoch`,
then the `WakuMessage` MUST be discarded and considered invalid.
This is to prevent a newly registered peer from spamming the system by messaging for all the past epochs.
`max_epoch_gap` is a system parameter for which we provide some recommendations in section [Recommended System Parameters](#recommended-system-parameters).
This is to prevent a newly registered peer from spamming the system
by messaging for all the past epochs.
`max_epoch_gap` is a system parameter
for which we provide some recommendations in section [Recommended System Parameters](#recommended-system-parameters).
##### Merkle Root Validation
The routing peers MUST check whether the provided Merkle root in the `RateLimitProof` is valid.
It can do so by maintaining a local set of valid Merkle roots,
The routing peers MUST check whether the provided Merkle root
in the `RateLimitProof` is valid.
It can do so by maintaining a local set of valid Merkle roots,
which consist of `acceptable_root_window_size` past roots.
These roots refer to the final state of the Merkle tree after a whole block consisting of group changes is processed.
These roots refer to the final state of the Merkle tree
after a whole block consisting of group changes is processed.
The Merkle roots are updated on a per-block basis instead of a per-event basis.
This is done because if Merkle roots are updated on a per-event basis, some peers could send messages with a root that refers to a Merkle tree state that might get invalidated while the message is still propagating in the network, due to many registrations happening during this time frame.
By updating roots on a per-block basis instead, we will have only one root update per-block processed, regardless on how many registrations happened in a block, and peers will be able to successfully propagate messages in a time frame corresponding to roughly the size of the roots window times the block mining time.
This is done because if Merkle roots are updated on a per-event basis,
some peers could send messages with a root that refers to a Merkle tree state
that might get invalidated while the message is still propagating in the network,
due to many registrations happening during this time frame.
By updating roots on a per-block basis instead,
we will have only one root update per-block processed,
regardless on how many registrations happened in a block, and
peers will be able to successfully propagate messages in a time frame
corresponding to roughly the size of the roots window times the block mining time.
Atomic processing of the blocks are necessary so that even if the peer is unable to process one event, the previous roots remain valid, and can be used to generate valid RateLimitProof's.
Atomic processing of the blocks are necessary
so that even if the peer is unable to process one event,
the previous roots remain valid, and can be used to generate valid RateLimitProof's.
This also allows peers which are not well connected to the network to be able to send messages, accounting for network delay.
This network delay is related to the nature of asynchronous network conditions, which means that peers see membership changes asynchronously, and therefore may have differing local Merkle trees.
See [Recommended System Parameters](#recommended-system-parameters) on choosing an appropriate `acceptable_root_window_size`.
This also allows peers which are not well connected to the network
to be able to send messages, accounting for network delay.
This network delay is related to the nature of asynchronous network conditions,
which means that peers see membership changes asynchronously, and
therefore may have differing local Merkle trees.
See [Recommended System Parameters](#recommended-system-parameters)
on choosing an appropriate `acceptable_root_window_size`.
##### Proof Verification
The routing peers MUST check whether the zero-knowledge proof `proof` is valid.
It does so by running the zk verification algorithm as explained in [RLN](../../../../vac/32/rln-v1.md).
If `proof` is invalid then the message MUST be discarded.
It does so by running the zk verification algorithm as explained in [RLN](../../../../vac/32/rln-v1.md).
If `proof` is invalid then the message MUST be discarded.
##### Spam detection
To enable local spam detection and slashing,
routing peers MUST record the `nullifier`, `share_x`, and `share_y`
of incoming messages which are not discarded i.e., not found spam or with invalid proof or epoch.
To spot spam messages, the peer checks whether a message with an identical `nullifier` has already been relayed.
To enable local spam detection and slashing,
routing peers MUST record the `nullifier`, `share_x`, and `share_y`
of incoming messages which are not discarded i.e.,
not found spam or with invalid proof or epoch.
To spot spam messages, the peer checks whether a message
with an identical `nullifier` has already been relayed.
1. If such a message exists and its `share_x` and `share_y`
components are different from the incoming message, then slashing takes place.
@@ -183,9 +245,11 @@ of the new message and the `share'_x` and `share'_y`
of the old record to reconstruct the `sk` of the message owner.
The `sk` then MAY be used to delete the spammer from the group and
withdraw a portion `reward_portion` of its staked funds.
3. If the `share_x` and `share_y` fields of the previously relayed message are identical to the incoming message,
2. If the `share_x` and
`share_y` fields of the previously relayed message are identical
to the incoming message,
then the message is a duplicate and MUST be discarded.
4. If none is found, then the message gets relayed.
3. If none is found, then the message gets relayed.
An overview of the routing procedure and slashing is provided in Figure 2.
@@ -196,9 +260,10 @@ An overview of the routing procedure and slashing is provided in Figure 2.
### Payloads
Payloads are protobuf messages implemented using [protocol buffers v3](https://developers.google.com/protocol-buffers/).
Nodes MAY extend the [14/WAKU2-MESSAGE](../14/message.md) with a `rate_limit_proof` field to indicate that their message is not spam.
Nodes MAY extend the [14/WAKU2-MESSAGE](../14/message.md)
with a `rate_limit_proof` field to indicate that their message is not spam.
```diff
```diff
syntax = "proto3";
@@ -221,22 +286,27 @@ message WakuMessage {
}
```
#### WakuMessage
`rate_limit_proof` holds the information required to prove that the message owner has not exceeded the message rate limit.
`rate_limit_proof` holds the information required to prove that the message owner
has not exceeded the message rate limit.
#### RateLimitProof
Below is the description of the fields of `RateLimitProof` and their types.
| Parameter | Type | Description |
| Parameter | Type | Description |
| ----: | ----------- | ----------- |
| `proof` | array of 256 bytes uncompressed or 128 bytes compressed | the zkSNARK proof as explained in the [Publishing process](##Publishing) |
| `proof` | array of 256 bytes uncompressed or 128 bytes compressed | the zkSNARK proof as explained in the [Publishing process](#publishing) |
| `merkle_root` | array of 32 bytes in little-endian order | the root of membership group Merkle tree at the time of publishing the message |
| `share_x` and `share_y`| array of 32 bytes each | Shamir secret shares of the user's secret identity key `sk` . `share_x` is the Poseidon hash of the `WakuMessage`'s `payload` concatenated with its `contentTopic` . `share_y` is calculated using [Shamir secret sharing scheme](../../../../vac/32/rln-v1.md) |
| `nullifier` | array of 32 bytes | internal nullifier derived from `epoch` and peer's `sk` as explained in [RLN construct](../../../../vac/32/rln-v1.md)|
### Recommended System Parameters
The system parameters are summarized in the following table, and the RECOMMENDED values for a subset of them are presented next.
The system parameters are summarized in the following table,
and the RECOMMENDED values for a subset of them are presented next.
| Parameter | Description |
| ----: |----------- |
@@ -247,35 +317,55 @@ The system parameters are summarized in the following table, and the RECOMMENDED
| `acceptable_root_window_size` | The maximum number of past Merkle roots to store |
#### Epoch Length
A sensible value for the `period` depends on the application for which the spam protection is going to be used.
For example, while the `period` of `1` second i.e.,
messaging rate of `1` per second, might be acceptable for a chat application,
A sensible value for the `period` depends on the application
for which the spam protection is going to be used.
For example, while the `period` of `1` second i.e.,
messaging rate of `1` per second, might be acceptable for a chat application,
might be too low for communication among Ethereum network validators.
One should look at the desired throughput of the application to decide on a proper `period` value.
One should look at the desired throughput of the application
to decide on a proper `period` value.
#### Maximum Epoch Gap
We discussed in the [Routing](#routing) section that the gap between the epoch observed by the routing peer and
the one attached to the incoming message should not exceed a threshold denoted by `max_epoch_gap`.
We discussed in the [Routing](#routing) section that the gap between the epoch
observed by the routing peer and
the one attached to the incoming message
should not exceed a threshold denoted by `max_epoch_gap`.
The value of `max_epoch_gap` can be measured based on the following factors.
- Network transmission delay `Network_Delay`: the maximum time that it takes for a message to be fully disseminated in the GossipSub network.
- Clock asynchrony `Clock_Asynchrony`: The maximum difference between the Unix epoch clocks perceived by network peers which can be due to clock drifts.
- Network transmission delay `Network_Delay`:
the maximum time that it takes for a message to be fully disseminated
in the GossipSub network.
- Clock asynchrony `Clock_Asynchrony`:
The maximum difference between the Unix epoch clocks perceived
by network peers which can be due to clock drifts.
With a reasonable approximation of the preceding values, one can set `max_epoch_gap` as
`max_epoch_gap` $= \lceil \frac{\text{Network Delay} + \text{Clock Asynchrony}}{\text{Epoch Length}}\rceil$ where `period` is the length of the `epoch` in seconds.
`Network_Delay` and `Clock_Asynchrony` MUST have the same resolution as `period` .
By this formulation, `max_epoch_gap` indeed measures the maximum number of `epoch`s that can elapse since a message gets routed from its origin to all the other peers in the network.
With a reasonable approximation of the preceding values,
one can set `max_epoch_gap` as
`acceptable_root_window_size` depends upon the underlying chain's average blocktime, `block_time`
`max_epoch_gap`
$= \lceil \frac{\text{Network Delay} + \text{Clock Asynchrony}}{\text{Epoch Length}}\rceil$
where `period` is the length of the `epoch` in seconds.
`Network_Delay` and `Clock_Asynchrony` MUST have the same resolution as `period`.
By this formulation, `max_epoch_gap` indeed measures the maximum number of `epoch`s
that can elapse since a message gets routed from its origin
to all the other peers in the network.
`acceptable_root_window_size` depends upon the underlying chain's average blocktime,
`block_time`
The lower bound for the `acceptable_root_window_size` SHOULD be set as $acceptable_root_window_size=(Network_Delay)/block_time$
`Network_Delay` MUST have the same resolution as `block_time`.
By this formulation, `acceptable_root_window_size` will provide a lower bound of how many roots can be acceptable by a routing peer.
By this formulation,
`acceptable_root_window_size` will provide a lower bound
of how many roots can be acceptable by a routing peer.
The `acceptable_root_window_size` should indicate how many blocks may have been mined during the time it takes for a peer to receive a message.
This formula represents a lower bound of the number of acceptable roots.
The `acceptable_root_window_size` should indicate how many blocks may have been mined
during the time it takes for a peer to receive a message.
This formula represents a lower bound of the number of acceptable roots.
## Copyright
@@ -284,9 +374,10 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
## References
1. [`11/WAKU2-RELAY`](../11/relay.md)
2. [RLN](../../../../vac/32/rln-v1.md)
3. [14/WAKU2-MESSAGE](../14/message.md)
4. [RLN documentation](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view)
5. [Public inputs to the RLN circuit](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Public-Inputs)
6. [Shamir secret sharing scheme used in RLN](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Linear-Equation-amp-SSS)
7. [RLN internal nullifier](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Nullifiers)
2. [64/WAKU-NETWORK](../64/network.md)
3. [RLN](../../../../vac/32/rln-v1.md)
4. [14/WAKU2-MESSAGE](../14/message.md)
5. [RLN documentation](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view)
6. [Public inputs to the RLN circuit](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Public-Inputs)
7. [Shamir secret sharing scheme used in RLN](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Linear-Equation-amp-SSS)
8. [RLN internal nullifier](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Nullifiers)

View File

@@ -13,8 +13,11 @@ contributors:
## Motivation and Goals
Light nodes with short connection windows and limited bandwidth wish to publish messages into the Waku network.
Additionally, there is sometimes a need for confirmation that a message has been received "by the network"
Light nodes with short connection windows and
limited bandwidth wish to publish messages into the Waku network.
Additionally,
there is sometimes a need for confirmation
that a message has been received "by the network"
(here, at least one node).
`19/WAKU2-LIGHTPUSH` is a request/response protocol for this.
@@ -45,13 +48,18 @@ message PushRPC {
### Message Relaying
Nodes that respond to `PushRequests` MUST either
relay the encapsulated message via [11/WAKU2-RELAY](../11/relay.md) protocol on the specified `pubsub_topic`,
or forward the `PushRequest` via 19/LIGHTPUSH on a [WAKU2-DANDELION](https://github.com/waku-org/specs/blob/waku-RFC/standards/application/dandelion.md) stem.
If they are unable to do so for some reason, they SHOULD return an error code in `PushResponse`.
relay the encapsulated message via [11/WAKU2-RELAY](../11/relay.md) protocol
on the specified `pubsub_topic`,
or forward the `PushRequest` via 19/LIGHTPUSH on a [WAKU2-DANDELION](https://github.com/waku-org/specs/blob/master/standards/application/dandelion.md)
stem.
If they are unable to do so for some reason,
they SHOULD return an error code in `PushResponse`.
## Security Considerations
Since this can introduce an amplification factor, it is RECOMMENDED for the node relaying to the rest of the network to take extra precautions.
Since this can introduce an amplification factor,
it is RECOMMENDED for the node relaying to the rest of the network
to take extra precautions.
This can be done by rate limiting via [17/WAKU2-RLN-RELAY](../17/rln-relay.md).
Note that the above is currently not fully implemented.
@@ -63,5 +71,5 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
## References
* [11/WAKU2-RELAY](../11/relay.md)
* [WAKU2-DANDELION](https://github.com/waku-org/specs/blob/waku-RFC/standards/application/dandelion.md)
* [WAKU2-DANDELION](https://github.com/waku-org/specs/blob/master/standards/application/dandelion.md)
* [17/WAKU2-RLN-RELAY](../17/rln-relay.md)

View File

@@ -4,92 +4,131 @@ title: 33/WAKU2-DISCV5
name: Waku v2 Discv5 Ambient Peer Discovery
status: draft
editor: Daniel Kaiser <danielkaiser@status.im>
contributors:
contributors:
- Filip Dimitrijevic <filip@status.im>
---
## Abstract
`33/WAKU2-DISCV5` specifies a modified version of [Ethereum's Node Discovery Protocol v5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md) as a means for ambient node discovery.
[10/WAKU2](../10/waku2.md) uses the `33/WAKU2-DISCV5` ambient node discovery network for establishing a decentralized network of interconnected Waku2 nodes.
In its current version, the `33/WAKU2-DISCV5` discovery network is isolated from the Ethereum Discovery v5 network.
Isolation improves discovery efficiency, which is especially significant with a low number of Waku nodes compared to the total number of Ethereum nodes.
`33/WAKU2-DISCV5` specifies a modified version of
[Ethereum's Node Discovery Protocol v5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)
as a means for ambient node discovery.
[10/WAKU2](/waku/standards/core/10/waku2.md) uses the `33/WAKU2-DISCV5` ambient node discovery network
for establishing a decentralized network of interconnected Waku2 nodes.
In its current version,
the `33/WAKU2-DISCV5` discovery network
is isolated from the Ethereum Discovery v5 network.
Isolation improves discovery efficiency,
which is especially significant with a low number of Waku nodes
compared to the total number of Ethereum nodes.
## Disclaimer
This version of `33/WAKU2-DISCV5` has a focus on timely deployment of an efficient discovery method for [10/WAKU2](../10/waku2.md).
This version of `33/WAKU2-DISCV5` has a focus on timely deployment
of an efficient discovery method for [10/WAKU2](/waku/standards/core/10/waku2.md).
Establishing a separate discovery network is in line with this focus.
However, we are aware of potential resilience problems (see section on security considerations) and are [discussing](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8)
However, we are aware of potential resilience problems
(see section on security considerations) and
are [discussing](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8)
and researching hybrid approaches.
## Background and Rationale
[11/WAKU2-RELAY](../11/relay.md) assumes the existence of a network of Waku2 nodes.
For establishing and growing this network, new nodes trying to join the Waku2 network need a means of discovering nodes within the network.
[10/WAKU2](../10/waku2.md) supports the following discovery methods in order of increasing decentralization
[11/WAKU2-RELAY](/waku/standards/core/11/relay.md) assumes the existence of a network of Waku2 nodes.
For establishing and growing this network,
new nodes trying to join the Waku2 network
need a means of discovering nodes within the network.
[10/WAKU2](/waku/standards/core/10/waku2.md) supports the following discovery methods
in order of increasing decentralization
* hard coded bootstrap nodes
* [`DNS discovery`](https://rfc.vac.dev/spec/10/#discovery-domain) (based on [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459))
* `peer-exchange` (work in progress)
* [`DNS discovery`](/waku/standards/10/waku2.md/#discovery-domain) (based on [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459))
* [`34/WAKU2-PEER-EXCHANGE`](/waku/standards/core/34/peer-exchange.md)
* `33/WAKU2-DISCV5` (specified in this document)
The purpose of ambient node discovery within [10/WAKU2](../10/waku2.md) is discovering Waku2 nodes in a decentralized way.
The unique selling point of `33/WAKU2-DISCV5` is its holistic view of the network, which allows avoiding hotspots and allows merging the network after a split.
While the other methods provide either a fixed or local set of nodes, `33/WAKU2-DISCV5` can provide a random sample of Waku2 nodes.
Future iterations of this document will add the possibility of efficiently discovering Waku2 nodes that have certain capabilities, e.g. holding messages of a certain time frame during which the querying node was offline.
The purpose of ambient node discovery within [10/WAKU2](/waku/standards/core/10/waku2.md)
is discovering Waku2 nodes in a decentralized way.
The unique selling point of `33/WAKU2-DISCV5` is its holistic view of the network,
which allows avoiding hotspots and allows merging the network after a split.
While the other methods provide either a fixed or local set of nodes,
`33/WAKU2-DISCV5` can provide a random sample of Waku2 nodes.
Future iterations of this document will add the possibility
of efficiently discovering Waku2 nodes that have certain capabilities,
e.g. holding messages of a certain time frame
during which the querying node was offline.
### Separate Discovery Network
#### w.r.t. Waku2 Relay Network
`33/WAKU2-DISCV5` spans an overlay network separate from the [GossipSub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md) network [11/WAKU2-RELAY](../11/relay.md) builds on.
`33/WAKU2-DISCV5` spans an overlay network separate from the
[GossipSub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md)
network [11/WAKU2-RELAY](/waku/standards/core/11/relay.md) builds on.
Because it is a P2P network on its own, it also depends on bootstrap nodes.
Having a separate discovery network reduces load on the bootstrap nodes, because the actual work is done by randomly discovered nodes.
Having a separate discovery network reduces load on the bootstrap nodes,
because the actual work is done by randomly discovered nodes.
This also increases decentralization.
#### w.r.t. Ethereum Discovery v5
`33/WAKU2-DISCV5` spans a discovery network isolated from the Ethereum Discovery v5 network.
`33/WAKU2-DISCV5` spans a discovery network
isolated from the Ethereum Discovery v5 network.
Another simple solution would be taking part in the Ethereum Discovery network, and filtering Waku nodes based on whether they support [WAKU2-ENR](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/enr.md).
Another simple solution would be taking part in the Ethereum Discovery network,
and filtering Waku nodes based on whether they support [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md).
This solution is more resilient towards eclipse attacks.
However, this discovery method is very inefficient for small percentages of Waku nodes (see [estimation](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8)).
However, this discovery method is very inefficient
for small percentages of Waku nodes
(see [estimation](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8)).
It boils down to random walk discovery and does not offer a O(log(n)) hop bound.
The rarer the requested property (in this case Waku), the longer a random walk will take until finding an appropriate node, which leads to a needle-in-the-haystack problem.
Using a dedicated Waku2 discovery network, nodes can query this discovery network for a random set of nodes
and all (well-behaving) returned nodes can serve as bootstrap nodes for other Waku2 protocols.
The rarer the requested property (in this case Waku),
the longer a random walk will take until finding an appropriate node,
which leads to a needle-in-the-haystack problem.
Using a dedicated Waku2 discovery network,
nodes can query this discovery network for a random set of nodes
and all (well-behaving)
returned nodes can serve as bootstrap nodes for other Waku2 protocols.
A more sophisticated solution would be using [Discv5 topic discovery](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement).
However, in its current state it also has efficiency problems for small percentages of Waku nodes and is still in the design phase ([see here](https://github.com/ethereum/devp2p/issues/199)).
However, in its current state it also has efficiency problems for small percentages
of Waku nodes and is still in the design phase
([see here](https://github.com/ethereum/devp2p/issues/199)).
Currently, the Ethereum discv5 network is very efficient in finding other discv5 nodes,
but it is not so efficient for finding discv5 nodes that have a specific property or offer specific services, e.g. Waku.
Currently,
the Ethereum discv5 network is very efficient in finding other discv5 nodes,
but it is not so efficient for finding discv5 nodes
that have a specific property or
offer specific services, e.g. Waku.
As part of our [discv5 roadmap](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121), we consider two ideas for future versions of `33/WAKU2-DISCV5`
As part of our [discv5 roadmap](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121),
we consider two ideas for future versions of `33/WAKU2-DISCV5`
* [Discv5 topic discovery](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement) with adjustments (ideally upstream)
* a hybrid solution that uses both a separate discv5 network and a Waku-ENR-filtered Ethereum discv5 network
* [Discv5 topic discovery](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement)
with adjustments (ideally upstream)
* a hybrid solution that uses both a separate discv5 network and
a Waku-ENR-filtered Ethereum discv5 network
## Semantics
`33/WAKU2-DISCV5` fully inherits the [discv5 semantics](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md).
Before announcing their address via Waku2 discv5, nodes SHOULD check if this address is publicly reachable.
Nodes MAY use the [libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md) to perform that check.
Before announcing their address via Waku2 discv5,
nodes SHOULD check if this address is publicly reachable.
Nodes MAY use the [libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md)
to perform that check.
Nodes SHOULD only announce publicly reachable addresses via Waku2 discv5,
to avoid cluttering peer lists with nodes that are not reachable.
## Wire Format Specification
`33/WAKU2-DISCV5` inherits the [discv5 wire protocol](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-wire.md) except for the following differences
`33/WAKU2-DISCV5` inherits the [discv5 wire protocol](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-wire.md)
except for the following differences
## WAKU2-Specific `protocol-id`
Ethereum discv5:
<pre>
<code>
```text
header = static-header || authdata
static-header = protocol-id || version || flag || nonce || authdata-size
protocol-id = <b>"discv5"</b>
@@ -97,12 +136,12 @@ version = 0x0001
authdata-size = uint16 -- byte length of authdata
flag = uint8 -- packet type identifier
nonce = uint96 -- nonce of message
</code>
</pre>
```
`33/WAKU2-DISCV5`:
<pre>
```text
kcode>
header = static-header || authdata
static-header = protocol-id || version || flag || nonce || authdata-size
@@ -111,65 +150,76 @@ version = 0x0001
authdata-size = uint16 -- byte length of authdata
flag = uint8 -- packet type identifier
nonce = uint96 -- nonce of message
</code>
</pre>
```
## Suggestions for Implementations
Existing discv5 implementations
* can be augmented to make the `protocol-id` selectable using a compile-time flag as in [this feature branch](https://github.com/kaiserd/nim-eth/blob/add-selectable-protocol-id-static/eth/p2p/discoveryv5/encoding.nim#L34) of nim-eth/discv5.
* can be augmented to make the `protocol-id` selectable using a compile-time flag
as in [this feature branch](https://github.com/kaiserd/nim-eth/blob/add-selectable-protocol-id-static/eth/p2p/discoveryv5/encoding.nim#L34)
of nim-eth/discv5.
* can be forked followed by changing the `protocol-id` string as in [go-waku](https://github.com/status-im/go-waku/blob/master/waku/v2/discv5/discover.go#L135-L137).
## Security Considerations
### Sybil attack
Implementations should limit the number of bucket entries that have the same network parameters (IP address / port) to mitigate Sybil attacks.
Implementations should limit the number of bucket entries
that have the same network parameters (IP address / port) to mitigate Sybil attacks.
### Eclipse attack
Eclipse attacks aim to eclipse certain regions in a DHT.
Malicious nodes provide false routing information for certain target regions.
The larger the desired eclipsed region, the more resources (i.e. controlled nodes) the attacker needs.
The larger the desired eclipsed region,
the more resources (i.e. controlled nodes) the attacker needs.
This introduces an efficiency versus resilience tradeoff.
Discovery is more efficient if information about target objects (e.g. network parameters of nodes supporting Waku) are closer to a specific DHT address.
If nodes providing specific information are closer to each other, they cover a smaller range in the DHT and are easier to eclipse.
Discovery is more efficient if information about target objects
(e.g. network parameters of nodes supporting Waku) are closer to a specific DHT address.
If nodes providing specific information are closer to each other,
they cover a smaller range in the DHT and are easier to eclipse.
Sybil attacks greatly increase the power of eclipse attacks, because they significantly reduce resources necessary to mount a successful eclipse attack.
Sybil attacks greatly increase the power of eclipse attacks,
because they significantly reduce resources necessary
to mount a successful eclipse attack.
## Security Implications of a Separate Discovery Network
A dedicated Waku discovery network is more likely to be subject to successful eclipse attacks (and to DoS attacks in general).
A dedicated Waku discovery network is more likely to be subject
to successful eclipse attacks (and to DoS attacks in general).
This is because eclipsing in a smaller network requires less resources for the attacker.
DoS attacks render the whole network unusable if the percentage of attacker nodes is sufficient.
DoS attacks render the whole network unusable
if the percentage of attacker nodes is sufficient.
Using random walk discovery would mitigate eclipse attacks targeted at specific capabilities, e.g. Waku.
However, this is because eclipse attacks aim at the DHT overlay structure, which is not used by random walks.
Using random walk discovery would mitigate eclipse attacks
targeted at specific capabilities, e.g. Waku.
However, this is because eclipse attacks aim at the DHT overlay structure,
which is not used by random walks.
So, this mitigation would come at the cost of giving up overlay routing efficiency.
The efficiency loss is especially severe with a relatively small number of Waku nodes.
Properly protecting against eclipse attacks is challenging and raises research questions that we will address in future stages of our discv5 roadmap.
Properly protecting against eclipse attacks is challenging and
raises research questions that we will address in future stages of our discv5 roadmap.
## References
1. [10/WAKU2](../10/waku2.md)
1. [11/WAKU2-RELAY](../11/relay.md)
1. [`WAKU2-ENR`](https://github.com/waku-org/specs/blob/waku-RFC/standards/core/enr.md)
1. [Node Discovery Protocol v5 (`discv5`)](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)
1. [`discv5` semantics](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md).
1. [`discv5` wire protocol](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-wire.md)
1. [`discv5` topic discovery](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement)
1. [Waku DNS discovery](https://rfc.vac.dev/spec/10/#discovery-domain)
1. [libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md)
1. [`EIP-1459`](https://eips.ethereum.org/EIPS/eip-1459)
1. [`GossipSub`](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md)
1. [Waku discv5 roadmap discussion](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121)
1. [discovery efficiency estimation](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8)
1. [implementation: Nim](https://github.com/kaiserd/nim-eth/blob/add-selectable-protocol-id-static/eth/p2p/discoveryv5/encoding.nim)
1. [implementation: Go](https://github.com/status-im/go-waku/blob/master/waku/v2/discv5/discover.go)
1. [10/WAKU2](/waku/standards/core/10/waku2.md)
2. [`34/WAKU2-PEER-EXCHANGE`](/waku/standards/core/34/peer-exchange.md)
3. [11/WAKU2-RELAY](/waku/standards/core/11/relay.md)
4. [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md)
5. [Node Discovery Protocol v5 (`discv5`)](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md)
6. [`discv5` semantics](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md).
7. [`discv5` wire protocol](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-wire.md)
8. [`discv5` topic discovery](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement)
9. [libp2p AutoNAT protocol](https://github.com/libp2p/specs/blob/master/autonat/README.md)
10. [`EIP-1459`](https://eips.ethereum.org/EIPS/eip-1459)
11. [`GossipSub`](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md)
12. [Waku discv5 roadmap discussion](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121)
13. [discovery efficiency estimation](https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8)
14. [implementation: Nim](https://github.com/kaiserd/nim-eth/blob/add-selectable-protocol-id-static/eth/p2p/discoveryv5/encoding.nim)
15. [implementation: Go](https://github.com/status-im/go-waku/blob/master/waku/v2/discv5/discover.go)
## Copyright

View File

@@ -0,0 +1,118 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="279pt" height="156pt" viewBox="0 0 279 156" version="1.1">
<defs>
<g>
<symbol overflow="visible" id="glyph0-0">
<path style="stroke:none;" d="M 0.640625 2.265625 L 0.640625 -9.015625 L 7.03125 -9.015625 L 7.03125 2.265625 Z M 1.359375 1.546875 L 6.328125 1.546875 L 6.328125 -8.296875 L 1.359375 -8.296875 Z M 1.359375 1.546875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-1">
<path style="stroke:none;" d="M 5.265625 -5.921875 C 5.128906 -5.992188 4.984375 -6.046875 4.828125 -6.078125 C 4.679688 -6.117188 4.519531 -6.140625 4.34375 -6.140625 C 3.6875 -6.140625 3.179688 -5.925781 2.828125 -5.5 C 2.484375 -5.082031 2.3125 -4.476562 2.3125 -3.6875 L 2.3125 0 L 1.15625 0 L 1.15625 -7 L 2.3125 -7 L 2.3125 -5.90625 C 2.5625 -6.332031 2.878906 -6.648438 3.265625 -6.859375 C 3.648438 -7.066406 4.117188 -7.171875 4.671875 -7.171875 C 4.753906 -7.171875 4.84375 -7.164062 4.9375 -7.15625 C 5.03125 -7.144531 5.132812 -7.128906 5.25 -7.109375 Z M 5.265625 -5.921875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-2">
<path style="stroke:none;" d="M 7.1875 -3.78125 L 7.1875 -3.21875 L 1.90625 -3.21875 C 1.957031 -2.425781 2.195312 -1.820312 2.625 -1.40625 C 3.050781 -1 3.644531 -0.796875 4.40625 -0.796875 C 4.84375 -0.796875 5.269531 -0.847656 5.6875 -0.953125 C 6.101562 -1.066406 6.515625 -1.226562 6.921875 -1.4375 L 6.921875 -0.359375 C 6.515625 -0.179688 6.09375 -0.046875 5.65625 0.046875 C 5.21875 0.140625 4.78125 0.1875 4.34375 0.1875 C 3.21875 0.1875 2.328125 -0.132812 1.671875 -0.78125 C 1.023438 -1.4375 0.703125 -2.320312 0.703125 -3.4375 C 0.703125 -4.582031 1.007812 -5.488281 1.625 -6.15625 C 2.25 -6.832031 3.085938 -7.171875 4.140625 -7.171875 C 5.078125 -7.171875 5.816406 -6.863281 6.359375 -6.25 C 6.910156 -5.644531 7.1875 -4.820312 7.1875 -3.78125 Z M 6.046875 -4.125 C 6.035156 -4.75 5.859375 -5.25 5.515625 -5.625 C 5.171875 -6 4.71875 -6.1875 4.15625 -6.1875 C 3.507812 -6.1875 2.992188 -6.003906 2.609375 -5.640625 C 2.222656 -5.285156 2 -4.78125 1.9375 -4.125 Z M 6.046875 -4.125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-3">
<path style="stroke:none;" d="M 1.890625 -3.5 C 1.890625 -2.644531 2.0625 -1.976562 2.40625 -1.5 C 2.757812 -1.019531 3.238281 -0.78125 3.84375 -0.78125 C 4.457031 -0.78125 4.9375 -1.019531 5.28125 -1.5 C 5.632812 -1.976562 5.8125 -2.644531 5.8125 -3.5 C 5.8125 -4.34375 5.632812 -5.003906 5.28125 -5.484375 C 4.9375 -5.960938 4.457031 -6.203125 3.84375 -6.203125 C 3.238281 -6.203125 2.757812 -5.960938 2.40625 -5.484375 C 2.0625 -5.003906 1.890625 -4.34375 1.890625 -3.5 Z M 5.8125 -1.046875 C 5.570312 -0.628906 5.265625 -0.316406 4.890625 -0.109375 C 4.523438 0.0859375 4.082031 0.1875 3.5625 0.1875 C 2.71875 0.1875 2.03125 -0.148438 1.5 -0.828125 C 0.96875 -1.503906 0.703125 -2.394531 0.703125 -3.5 C 0.703125 -4.59375 0.96875 -5.476562 1.5 -6.15625 C 2.03125 -6.832031 2.71875 -7.171875 3.5625 -7.171875 C 4.082031 -7.171875 4.523438 -7.066406 4.890625 -6.859375 C 5.265625 -6.660156 5.570312 -6.351562 5.8125 -5.9375 L 5.8125 -7 L 6.953125 -7 L 6.953125 2.65625 L 5.8125 2.65625 Z M 5.8125 -1.046875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-4">
<path style="stroke:none;" d="M 1.09375 -2.765625 L 1.09375 -7 L 2.234375 -7 L 2.234375 -2.8125 C 2.234375 -2.144531 2.363281 -1.644531 2.625 -1.3125 C 2.882812 -0.976562 3.269531 -0.8125 3.78125 -0.8125 C 4.40625 -0.8125 4.894531 -1.007812 5.25 -1.40625 C 5.613281 -1.800781 5.796875 -2.34375 5.796875 -3.03125 L 5.796875 -7 L 6.953125 -7 L 6.953125 0 L 5.796875 0 L 5.796875 -1.078125 C 5.515625 -0.648438 5.191406 -0.332031 4.828125 -0.125 C 4.460938 0.0820312 4.035156 0.1875 3.546875 0.1875 C 2.742188 0.1875 2.132812 -0.0625 1.71875 -0.5625 C 1.300781 -1.0625 1.09375 -1.796875 1.09375 -2.765625 Z M 3.984375 -7.171875 Z M 3.984375 -7.171875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-5">
<path style="stroke:none;" d="M 5.671875 -6.796875 L 5.671875 -5.703125 C 5.347656 -5.867188 5.007812 -5.992188 4.65625 -6.078125 C 4.300781 -6.160156 3.9375 -6.203125 3.5625 -6.203125 C 3 -6.203125 2.570312 -6.113281 2.28125 -5.9375 C 2 -5.769531 1.859375 -5.507812 1.859375 -5.15625 C 1.859375 -4.882812 1.957031 -4.671875 2.15625 -4.515625 C 2.363281 -4.367188 2.773438 -4.226562 3.390625 -4.09375 L 3.78125 -4 C 4.601562 -3.832031 5.1875 -3.585938 5.53125 -3.265625 C 5.875 -2.941406 6.046875 -2.5 6.046875 -1.9375 C 6.046875 -1.28125 5.785156 -0.757812 5.265625 -0.375 C 4.753906 0 4.050781 0.1875 3.15625 0.1875 C 2.78125 0.1875 2.390625 0.148438 1.984375 0.078125 C 1.578125 0.00390625 1.144531 -0.101562 0.6875 -0.25 L 0.6875 -1.4375 C 1.113281 -1.21875 1.53125 -1.050781 1.9375 -0.9375 C 2.351562 -0.832031 2.765625 -0.78125 3.171875 -0.78125 C 3.710938 -0.78125 4.128906 -0.875 4.421875 -1.0625 C 4.710938 -1.25 4.859375 -1.507812 4.859375 -1.84375 C 4.859375 -2.15625 4.753906 -2.394531 4.546875 -2.5625 C 4.335938 -2.726562 3.875 -2.890625 3.15625 -3.046875 L 2.765625 -3.140625 C 2.046875 -3.285156 1.53125 -3.515625 1.21875 -3.828125 C 0.90625 -4.140625 0.75 -4.566406 0.75 -5.109375 C 0.75 -5.765625 0.976562 -6.269531 1.4375 -6.625 C 1.90625 -6.988281 2.570312 -7.171875 3.4375 -7.171875 C 3.851562 -7.171875 4.25 -7.140625 4.625 -7.078125 C 5 -7.015625 5.347656 -6.921875 5.671875 -6.796875 Z M 5.671875 -6.796875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-6">
<path style="stroke:none;" d="M 2.34375 -8.984375 L 2.34375 -7 L 4.71875 -7 L 4.71875 -6.109375 L 2.34375 -6.109375 L 2.34375 -2.3125 C 2.34375 -1.738281 2.421875 -1.367188 2.578125 -1.203125 C 2.734375 -1.046875 3.050781 -0.96875 3.53125 -0.96875 L 4.71875 -0.96875 L 4.71875 0 L 3.53125 0 C 2.644531 0 2.03125 -0.164062 1.6875 -0.5 C 1.351562 -0.832031 1.1875 -1.4375 1.1875 -2.3125 L 1.1875 -6.109375 L 0.34375 -6.109375 L 0.34375 -7 L 1.1875 -7 L 1.1875 -8.984375 Z M 2.34375 -8.984375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-7">
<path style="stroke:none;" d="M 2.3125 -1.046875 L 2.3125 2.65625 L 1.15625 2.65625 L 1.15625 -7 L 2.3125 -7 L 2.3125 -5.9375 C 2.5625 -6.351562 2.867188 -6.660156 3.234375 -6.859375 C 3.597656 -7.066406 4.039062 -7.171875 4.5625 -7.171875 C 5.40625 -7.171875 6.09375 -6.832031 6.625 -6.15625 C 7.15625 -5.476562 7.421875 -4.59375 7.421875 -3.5 C 7.421875 -2.394531 7.15625 -1.503906 6.625 -0.828125 C 6.09375 -0.148438 5.40625 0.1875 4.5625 0.1875 C 4.039062 0.1875 3.597656 0.0859375 3.234375 -0.109375 C 2.867188 -0.316406 2.5625 -0.628906 2.3125 -1.046875 Z M 6.234375 -3.5 C 6.234375 -4.34375 6.054688 -5.003906 5.703125 -5.484375 C 5.359375 -5.960938 4.882812 -6.203125 4.28125 -6.203125 C 3.664062 -6.203125 3.179688 -5.960938 2.828125 -5.484375 C 2.484375 -5.003906 2.3125 -4.34375 2.3125 -3.5 C 2.3125 -2.644531 2.484375 -1.976562 2.828125 -1.5 C 3.179688 -1.019531 3.664062 -0.78125 4.28125 -0.78125 C 4.882812 -0.78125 5.359375 -1.019531 5.703125 -1.5 C 6.054688 -1.976562 6.234375 -2.644531 6.234375 -3.5 Z M 6.234375 -3.5 "/>
</symbol>
<symbol overflow="visible" id="glyph0-8">
<path style="stroke:none;" d="M 3.921875 -6.1875 C 3.304688 -6.1875 2.816406 -5.945312 2.453125 -5.46875 C 2.097656 -4.988281 1.921875 -4.332031 1.921875 -3.5 C 1.921875 -2.65625 2.097656 -1.992188 2.453125 -1.515625 C 2.804688 -1.035156 3.296875 -0.796875 3.921875 -0.796875 C 4.535156 -0.796875 5.019531 -1.035156 5.375 -1.515625 C 5.726562 -2.003906 5.90625 -2.664062 5.90625 -3.5 C 5.90625 -4.320312 5.726562 -4.972656 5.375 -5.453125 C 5.019531 -5.941406 4.535156 -6.1875 3.921875 -6.1875 Z M 3.921875 -7.171875 C 4.921875 -7.171875 5.703125 -6.84375 6.265625 -6.1875 C 6.835938 -5.539062 7.125 -4.644531 7.125 -3.5 C 7.125 -2.351562 6.835938 -1.453125 6.265625 -0.796875 C 5.703125 -0.140625 4.921875 0.1875 3.921875 0.1875 C 2.910156 0.1875 2.117188 -0.140625 1.546875 -0.796875 C 0.984375 -1.453125 0.703125 -2.351562 0.703125 -3.5 C 0.703125 -4.644531 0.984375 -5.539062 1.546875 -6.1875 C 2.117188 -6.84375 2.910156 -7.171875 3.921875 -7.171875 Z M 3.921875 -7.171875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-9">
<path style="stroke:none;" d="M 7.015625 -4.21875 L 7.015625 0 L 5.875 0 L 5.875 -4.1875 C 5.875 -4.851562 5.742188 -5.347656 5.484375 -5.671875 C 5.222656 -6.003906 4.835938 -6.171875 4.328125 -6.171875 C 3.703125 -6.171875 3.207031 -5.972656 2.84375 -5.578125 C 2.488281 -5.179688 2.3125 -4.640625 2.3125 -3.953125 L 2.3125 0 L 1.15625 0 L 1.15625 -7 L 2.3125 -7 L 2.3125 -5.90625 C 2.59375 -6.332031 2.914062 -6.648438 3.28125 -6.859375 C 3.65625 -7.066406 4.085938 -7.171875 4.578125 -7.171875 C 5.378906 -7.171875 5.984375 -6.921875 6.390625 -6.421875 C 6.804688 -5.921875 7.015625 -5.1875 7.015625 -4.21875 Z M 7.015625 -4.21875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-10">
<path style="stroke:none;" d="M 5.8125 -5.9375 L 5.8125 -9.71875 L 6.953125 -9.71875 L 6.953125 0 L 5.8125 0 L 5.8125 -1.046875 C 5.570312 -0.628906 5.265625 -0.316406 4.890625 -0.109375 C 4.523438 0.0859375 4.082031 0.1875 3.5625 0.1875 C 2.71875 0.1875 2.03125 -0.148438 1.5 -0.828125 C 0.96875 -1.503906 0.703125 -2.394531 0.703125 -3.5 C 0.703125 -4.59375 0.96875 -5.476562 1.5 -6.15625 C 2.03125 -6.832031 2.71875 -7.171875 3.5625 -7.171875 C 4.082031 -7.171875 4.523438 -7.066406 4.890625 -6.859375 C 5.265625 -6.660156 5.570312 -6.351562 5.8125 -5.9375 Z M 1.890625 -3.5 C 1.890625 -2.644531 2.0625 -1.976562 2.40625 -1.5 C 2.757812 -1.019531 3.238281 -0.78125 3.84375 -0.78125 C 4.457031 -0.78125 4.9375 -1.019531 5.28125 -1.5 C 5.632812 -1.976562 5.8125 -2.644531 5.8125 -3.5 C 5.8125 -4.34375 5.632812 -5.003906 5.28125 -5.484375 C 4.9375 -5.960938 4.457031 -6.203125 3.84375 -6.203125 C 3.238281 -6.203125 2.757812 -5.960938 2.40625 -5.484375 C 2.0625 -5.003906 1.890625 -4.34375 1.890625 -3.5 Z M 1.890625 -3.5 "/>
</symbol>
<symbol overflow="visible" id="glyph0-11">
<path style="stroke:none;" d=""/>
</symbol>
<symbol overflow="visible" id="glyph0-12">
<path style="stroke:none;" d="M 1.25 -9.328125 L 2.515625 -9.328125 L 2.515625 -1.0625 L 7.0625 -1.0625 L 7.0625 0 L 1.25 0 Z M 1.25 -9.328125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-13">
<path style="stroke:none;" d="M 1.203125 -7 L 2.359375 -7 L 2.359375 0 L 1.203125 0 Z M 1.203125 -9.71875 L 2.359375 -9.71875 L 2.359375 -8.265625 L 1.203125 -8.265625 Z M 1.203125 -9.71875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-14">
<path style="stroke:none;" d="M 9.359375 -6.296875 L 2.921875 -4 L 9.359375 -1.71875 L 9.359375 -0.59375 L 1.359375 -3.5 L 1.359375 -4.53125 L 9.359375 -7.4375 Z M 9.359375 -6.296875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-15">
<path style="stroke:none;" d="M 1.25 -9.328125 L 7.15625 -9.328125 L 7.15625 -8.265625 L 2.515625 -8.265625 L 2.515625 -5.5 L 6.953125 -5.5 L 6.953125 -4.4375 L 2.515625 -4.4375 L 2.515625 -1.0625 L 7.265625 -1.0625 L 7.265625 0 L 1.25 0 Z M 1.25 -9.328125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-16">
<path style="stroke:none;" d="M 1.25 -9.328125 L 2.953125 -9.328125 L 7.09375 -1.53125 L 7.09375 -9.328125 L 8.3125 -9.328125 L 8.3125 0 L 6.609375 0 L 2.484375 -7.796875 L 2.484375 0 L 1.25 0 Z M 1.25 -9.328125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-17">
<path style="stroke:none;" d="M 5.6875 -4.375 C 5.957031 -4.28125 6.21875 -4.082031 6.46875 -3.78125 C 6.726562 -3.488281 6.984375 -3.078125 7.234375 -2.546875 L 8.515625 0 L 7.171875 0 L 5.96875 -2.390625 C 5.664062 -3.015625 5.367188 -3.425781 5.078125 -3.625 C 4.785156 -3.832031 4.390625 -3.9375 3.890625 -3.9375 L 2.515625 -3.9375 L 2.515625 0 L 1.25 0 L 1.25 -9.328125 L 4.109375 -9.328125 C 5.171875 -9.328125 5.960938 -9.101562 6.484375 -8.65625 C 7.015625 -8.207031 7.28125 -7.535156 7.28125 -6.640625 C 7.28125 -6.054688 7.140625 -5.566406 6.859375 -5.171875 C 6.585938 -4.785156 6.195312 -4.519531 5.6875 -4.375 Z M 2.515625 -8.296875 L 2.515625 -4.984375 L 4.109375 -4.984375 C 4.710938 -4.984375 5.171875 -5.125 5.484375 -5.40625 C 5.796875 -5.6875 5.953125 -6.097656 5.953125 -6.640625 C 5.953125 -7.179688 5.796875 -7.59375 5.484375 -7.875 C 5.171875 -8.15625 4.710938 -8.296875 4.109375 -8.296875 Z M 2.515625 -8.296875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-18">
<path style="stroke:none;" d="M 1.359375 -6.296875 L 1.359375 -7.4375 L 9.359375 -4.53125 L 9.359375 -3.5 L 1.359375 -0.59375 L 1.359375 -1.71875 L 7.796875 -4 Z M 1.359375 -6.296875 "/>
</symbol>
</g>
</defs>
<g id="surface3285">
<rect x="0" y="0" width="279" height="156" style="fill:rgb(100%,100%,100%);fill-opacity:1;stroke:none;"/>
<path style="fill-rule:evenodd;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 30.7 0 L 34.7 0 L 34.7 1.2 L 30.7 1.2 Z M 30.7 0 " transform="matrix(20,0,0,20,-612,2)"/>
<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
<use xlink:href="#glyph0-1" x="10" y="18.005968"/>
<use xlink:href="#glyph0-2" x="15" y="18.005968"/>
<use xlink:href="#glyph0-3" x="22.777778" y="18.005968"/>
<use xlink:href="#glyph0-4" x="30.833333" y="18.005968"/>
<use xlink:href="#glyph0-2" x="38.888889" y="18.005968"/>
<use xlink:href="#glyph0-5" x="46.666667" y="18.005968"/>
<use xlink:href="#glyph0-6" x="53.333333" y="18.005968"/>
<use xlink:href="#glyph0-2" x="58.333333" y="18.005968"/>
<use xlink:href="#glyph0-1" x="66.111111" y="18.005968"/>
</g>
<path style="fill-rule:evenodd;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 40.5 0 L 44.5 0 L 44.5 1.2 L 40.5 1.2 Z M 40.5 0 " transform="matrix(20,0,0,20,-612,2)"/>
<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
<use xlink:href="#glyph0-1" x="206" y="18.005968"/>
<use xlink:href="#glyph0-2" x="211" y="18.005968"/>
<use xlink:href="#glyph0-5" x="218.777778" y="18.005968"/>
<use xlink:href="#glyph0-7" x="225.444444" y="18.005968"/>
<use xlink:href="#glyph0-8" x="233.5" y="18.005968"/>
<use xlink:href="#glyph0-9" x="241.277778" y="18.005968"/>
<use xlink:href="#glyph0-10" x="249.333333" y="18.005968"/>
<use xlink:href="#glyph0-2" x="257.388889" y="18.005968"/>
<use xlink:href="#glyph0-1" x="265.166667" y="18.005968"/>
</g>
<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 32.7 1.2 L 32.7 7.6 " transform="matrix(20,0,0,20,-612,2)"/>
<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 42.5 1.2 L 42.5 7.6 " transform="matrix(20,0,0,20,-612,2)"/>
<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 32.7 2.4 L 42.016211 3.445703 " transform="matrix(20,0,0,20,-612,2)"/>
<path style="fill-rule:evenodd;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 42.388867 3.4875 L 41.864063 3.680273 L 42.016211 3.445703 L 41.919922 3.183398 Z M 42.388867 3.4875 " transform="matrix(20,0,0,20,-612,2)"/>
<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
<use xlink:href="#glyph0-11" x="96" y="106.005968"/>
<use xlink:href="#glyph0-12" x="100.166667" y="106.005968"/>
<use xlink:href="#glyph0-13" x="107.388889" y="106.005968"/>
<use xlink:href="#glyph0-5" x="111" y="106.005968"/>
<use xlink:href="#glyph0-6" x="117.666667" y="106.005968"/>
<use xlink:href="#glyph0-14" x="122.666667" y="106.005968"/>
<use xlink:href="#glyph0-15" x="133.5" y="106.005968"/>
<use xlink:href="#glyph0-16" x="141.555556" y="106.005968"/>
<use xlink:href="#glyph0-17" x="151" y="106.005968"/>
<use xlink:href="#glyph0-18" x="159.888889" y="106.005968"/>
</g>
<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
<use xlink:href="#glyph0-1" x="112" y="52.005968"/>
<use xlink:href="#glyph0-2" x="117" y="52.005968"/>
<use xlink:href="#glyph0-3" x="124.777778" y="52.005968"/>
<use xlink:href="#glyph0-4" x="132.833333" y="52.005968"/>
<use xlink:href="#glyph0-2" x="140.888889" y="52.005968"/>
<use xlink:href="#glyph0-5" x="148.666667" y="52.005968"/>
<use xlink:href="#glyph0-6" x="155.333333" y="52.005968"/>
</g>
<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 42.5 5.3 L 33.184375 6.250586 " transform="matrix(20,0,0,20,-612,2)"/>
<path style="fill-rule:evenodd;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 32.811133 6.288672 L 33.283203 5.989258 L 33.184375 6.250586 L 33.333984 6.486523 Z M 32.811133 6.288672 " transform="matrix(20,0,0,20,-612,2)"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 17 KiB

View File

@@ -0,0 +1,231 @@
---
slug: 34
title: 34/WAKU2-PEER-EXCHANGE
name: Waku2 Peer Exchange
status: draft
category: Standards Track
tags: waku/core-protocol
editor: Hanno Cornelius <hanno@status.im>
contributors:
- Daniel Kaiser <danielkaiser@status.im>
---
## Abstract
This document specifies a simple request-response peer exchange protocol.
Responders send information about a requested number of peers.
The main purpose of this protocol is providing resource restricted devices with peers.
***Protocol Identifier***
> /vac/waku/peer-exchange/2.0.0-alpha1
## Background and Motivation
It may not be feasible, on resource restricted devices,
to take part in distributed random sampling ambient peer discovery protocols,
such as [33/WAKU2-DISCV5](/waku/standards/core/33/discv5.md).
The Waku peer discovery protocol, specified in this document,
allows resource restricted devices to request a list of peers from a service node.
Network parameters necessary to connect to this service node COULD be learned
from a static bootstrapping method or
using [EIP-1459: Node Discovery via DNS](https://eips.ethereum.org/EIPS/eip-1459).
The advantage of using Waku peer exchange to discover new peers,
compared to relying on a static peer list or DNS discovery, is a more even load distribution.
If a lot of resource restricted nodes would use the same service nodes as relay
or store nodes, the load on these would be very high.
Heavily used static nodes also add a centralized element.
Downtime of such a node might significantly impact the network.
However, the resource efficiency of this protocol comes at an anonymity cost,
which is explained in the
[Security/Privacy Considerations](#security-considerations) section.
This protocol SHOULD only be used if [33/WAKU2-DISCV5](/waku/standards/core/33/discv5.md) is infeasible.
## Theory and Protocol Semantics
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
The peer exchange protocol, specified in this document,
is a simple request-response protocol.
As Figure 1 illustrates, the requesting node sends a request to a peer,
which acts as the responder.
The responder replies with a list of ENRs as specified in [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md).
The [multiaddresses](https://docs.libp2p.io/concepts/addressing/)
used to connect to the respective peers can be extracted from the ENRs.
![Figure 1: The responder provides a list of ENRs to the requester. These ENRs contain the information necessary for connecting to the respective peers.](./images/protocol.svg)
In order to protect its anonymity,
the responder MUST NOT provide peers from its actively used peer list
as this opens pathways to *Neighbourhood Surveillance* attacks, as described in the
[Security/Privacy Considerations Section](#security-considerations).
The responder SHOULD provide a set of peers
that has been retrieved using ambient peer discovery methods supporting random sampling,
e.g. [33/WAKU2-DISCV5](/waku/standards/core/33/discv5.md).
This both protects the responder's anonymity as well as helps distributing load.
To allow for fast responses, responders SHOULD retrieve peers unsolicited
(before receiving a query) and
maintain a queue of peers for the purpose of providing them in peer exchange responses.
To get the best anonymity properties with respect to response peer sets,
responders SHOULD use each of these peers only once.
To save bandwidth, and as a trade off to anonymity,
responders MAY maintain a larger cache of exchange peers and
randomly sample response sets from this local cache.
The size of the cache SHOULD be large enough to allow randomly sampling peer sets
that (on average) do not overlap too much.
The responder SHOULD periodically replace the oldest peers in the cache.
The RECOMMENDED options for the cache size are described in the
[Implementation Suggestions Section](#implementation-suggestions).
Requesters, in the context of the specified peer exchange protocol,
SHOULD be resource restricted devices.
While any node could technically act as a requester,
using the peer exchange protocol comes with two drawbacks:
* reducing [anonymity](#security-considerations)
* causing load on responder nodes
## Wire Format Specification
```protobuf
syntax = "proto3";
message PeerInfo {
bytes enr = 1;
}
message PeerExchangeQuery {
uint64 num_peers = 1;
}
message PeerExchangeResponse {
repeated PeerInfo peer_infos = 1;
}
message PeerExchangeRPC {
PeerExchangeQuery query = 1;
PeerExchangeResponse response = 2;
}
```
The `enr` field contains a Waku ENR as specified in [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md).
Requesters send a `PeerExchangeQuery` to a peer.
Responders SHOULD include a maximum of `num_peers` `PeerInfo` instances into a response.
Responders send a `PeerExchangeResponse` to requesters
containing a list of `PeerInfo` instances, which in turn hold an ENR.
## Implementation Suggestions
### Discovery Interface
Implementations can implement the libp2p discovery interface:
* [nim](https://github.com/status-im/nim-libp2p/issues/140)
* [javascript](https://github.com/libp2p/js-libp2p-interfaces/tree/master/packages/interface-peer-discovery).
### Exchange Peer Cache Size
The size of the (optional) exchange peer cache discussed in
[Theory and Protocol Semantics](#theory-and-protocol-semantics)
depends on the average number of requested peers,
which is expected to be the outbound degree of the underlying
[libp2p gossipsub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md)
mesh network.
The RECOMMENDED value for this outbound degree is 6 (see parameter `D` in [29/WAKU2-CONFIG](/waku/informational/29/config.md)).
It is RECOMMENDED for the cache to hold at least 10 times as many peers (60).
The RECCOMENDED cache size also depends on the number of requesters a responder
is expected to serve within a *refresh cycle*.
A refresh cycle is the time interval in which all peers in the cache
are expected to be replaced.
If the number of requests expected per refresh cycle exceeds 600
(10 times the above recommended 60),
it is RECOMMENDED to increase the cache size to at least a tenth of that number.
## Security Considerations
### Privacy and Anonymity
The peer exchange protocol specified in this document comes with anonymity and
security implications.
We differentiate these implications into the requester and responder side, respectively.
### Requester
With a simple peer exchange protocol,
the requester is inherently susceptible to both *neighbourhood surveillance* and
*controlled neighbourhood* attacks.
To mount a *neighbourhood surveillance* attack,
an attacker has to connect to the peers of the victim node.
The peer exchange protocol allows a malicious responder to easily get into this position.
The responder connects to a set of peers and
simply returns this set of peers to the requester.
The peer exchange protocol also makes it much easier to get into the position
required for the *controlled neighbourhood* attack:
A malicious responder provides controlled peers in the response peer list.
More on these attacks may be found in our [research log article](https://vac.dev/wakuv2-relay-anon).
As a weak mitigation the requester MAY ask several peers and
select a subset of the returned peers.
### Responder
Responders that answer with active mesh peers are more vulnerable
to a *neighbourhood surveillance* attack.
Responding with the set of active mesh peers allows a malicious requester to
get into the required position more easily.
It takes away the first hurdle of the *neighbourhood surveillance* attack:
The attacker knows which peers to try to connect to.
This increased vulnerability can be avoided by only responding
with randomly sampled sets of peers,
e.g. by requesting a random peer set via [33/WAKU2-DISCV5](/waku/standards/core/33/discv5.md).
(As stated in the [Theory and Protocol Semantics Section](#theory-and-protocol-semantics),
these peer sets SHOULD be retrieved unsolicitedly before
receiving requests to achieve faster response times.)
Responders are also susceptible to amplification DoS attacks.
Requesters send a simple message request which causes responders to
engage in ambient peer discovery to retrieve a new random peer set.
As a mitigation, responders MAY feature a `seen cache` for requests and
only answer once per time interval.
The exchange-peer cache discussed in [Theory and Protocol Semantics Section](#theory-and-protocol-semantics)
also provides mitigation.
Still, frequent queries can tigger the refresh cycle more often.
The `seen cache` MAY be used in conjunction to provide additional mitigation.
### Further Considerations
The response field contains ENRs as specified in [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md).
While ENRs contain signatures, they do not violate the [Waku relay no-sign policy](/waku/standards/core/11/relay.md#signature-policy),
because they are part of the discovery domain and
are not propagated in the relay domain.
However, there might still be some form of leakage:
ENRs could be used to track peers and facilitate linking attacks.
We will investigate this further in our Waku anonymity analysis.
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
* [33/WAKU2-DISCV5](/waku/standards/core/33/discv5.md)
* [EIP-1459: Node Discovery via DNS](https://eips.ethereum.org/EIPS/eip-1459)
* [WAKU2-ENR](https://github.com/waku-org/specs/blob/master/standards/core/enr.md)
* [multiaddress](https://docs.libp2p.io/concepts/addressing/)
* [libp2p discovery interface in nim](https://github.com/status-im/nim-libp2p/issues/140)
* [libp2p discovery interface in javascript](https://github.com/libp2p/js-libp2p-interfaces/tree/master/packages/interface-peer-discovery)
* [libp2p gossipsub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md)
* [29/WAKU2-CONFIG](/waku/informational/29/config.md)
* [Waku Relay Anonymity](https://vac.dev/wakuv2-relay-anon)
* [Waku relay no-sign policy](/waku/standards/core/11/relay.md#signature-policy)

File diff suppressed because it is too large Load Diff

View File

@@ -11,7 +11,8 @@ contributors:
## Abstract
This specification describes an opinionated deployment of [10/WAKU2](../10/waku2.md) protocols to form a coherent and
This specification describes an opinionated deployment of [10/WAKU2](../10/waku2.md)
protocols to form a coherent and
shared decentralized messaging network that is open-access,
useful for generalized messaging, privacy-preserving, scalable and
accessible even to resource-restricted devices.
@@ -24,35 +25,47 @@ All The Waku Network configuration parameters are listed [here](https://github.c
### Routing protocol
The Waku Network is built on the [17/WAKU2-RLN-RELAY](../17/rln-relay.md) routing protocol,
which in turn is an extension of [11/WAKU2-RELAY](../11/relay.md) with spam protection measures.
The Waku Network is built on the
[17/WAKU2-RLN-RELAY](../17/rln-relay.md) routing protocol,
which in turn is an extension of
[11/WAKU2-RELAY](../11/relay.md) with spam protection measures.
### Network shards
Traffic in the Waku Network is sharded into eight [17/WAKU2-RLN-RELAY](../17/rln-relay.md) pubsub topics.
Traffic in the Waku Network is sharded into eight
[17/WAKU2-RLN-RELAY](../17/rln-relay.md) pubsub topics.
Each pubsub topic is named according to the static shard naming format
defined in [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md)
with:
* `<cluster_id>` set to `1`
* `<shard_number>` occupying the range `0` to `7`.
In other words, the Waku Network is a [17/WAKU2-RLN-RELAY](../17/rln-relay.md) network
routed on the combination of the eight pubsub topics:
```
```text
/waku/2/rs/1/0
/waku/2/rs/1/1
...
/waku/2/rs/1/7
```
A node MUST use [WAKU-METADATA](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) protocol to identify the `<cluster_id>` that every
inbound/outbound peer that attempts to connect supports. In any of the following cases, the node MUST trigger a disconnection:
* [WAKU-METADATA](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) dial fails.
* [WAKU-METADATA](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) reports an empty `<cluster_id>`.
* [WAKU-METADATA](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) reports a `<cluster_id>` different than `1`.
A node MUST use [66/WAKU2-METADATA](../66/metadata.md)
protocol to identify the `<cluster_id>` that every
inbound/outbound peer that attempts to connect supports.
In any of the following cases, the node MUST trigger a disconnection:
* [66/WAKU2-METADATA](../66/metadata.md)
dial fails.
* [66/WAKU2-METADATA](../66/metadata.md)
reports an empty `<cluster_id>`.
* [66/WAKU2-METADATA](../66/metadata.md)
reports a `<cluster_id>` different than `1`.
## Roles
There are two distinct roles evident in the network, those of:
1) nodes, and
2) applications.
@@ -61,7 +74,7 @@ There are two distinct roles evident in the network, those of:
Nodes are the individual software units
using [10/WAKU2](../10/waku2.md) protocols to form a p2p messaging network.
Nodes, in turn, can participate in a shard as full relayers, i.e. _relay nodes_,
or by running a combination of protocols suitable for resource-restricted environments,
or by running a combination of protocols suitable for resource-restricted environments,
i.e. _non-relay nodes_.
Nodes can also provide various services to the network,
such as storing historical messages or protecting the network against spam.
@@ -77,92 +90,131 @@ but MUST be subscribed to at least one defined shard.
Each relay node SHOULD be subscribed to as many shards as it has resources to support.
If a relay node supports an encapsulating application,
it SHOULD be subscribed to all the shards servicing that application.
If resource restrictions prevent a relay node from servicing all shards used by the encapsulating application,
If resource restrictions prevent a relay node from servicing all shards
used by the encapsulating application,
it MAY choose to support some shards as a non-relay node.
#### Bootstrapping and discovery
Nodes MAY use any method to bootstrap connection to the network,
but it is RECOMMENDED that each node retrieves a list of bootstrap peers to connect to using [EIP-1459 DNS-based discovery](https://eips.ethereum.org/EIPS/eip-1459).
Relay nodes SHOULD use [33/WAKU2-DISCV5](../33/discv5.md) to continually discover other peers in the network.
but it is RECOMMENDED that each node retrieves a list of bootstrap peers to connect
to using [EIP-1459 DNS-based discovery](https://eips.ethereum.org/EIPS/eip-1459).
Relay nodes SHOULD use [33/WAKU2-DISCV5](../33/discv5.md) to continually discover
other peers in the network.
Each relay node MUST encode its supported shards into its discoverable ENR,
as described in [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md/#discovery).
The ENR MUST be updated if the set of supported shards change.
A node MAY choose to ignore discovered peers that do not support any of the shards in its own subscribed set.
A node MAY choose to ignore discovered peers that do not support any of the shards
in its own subscribed set.
#### Transports
Relay nodes MUST follow [10/WAKU2](../10/waku2.md) specifications with regards to supporting different transports.
If TCP transport is available, each relay node MUST support it as transport for both dialing and listening.
In addition, a relay node SHOULD support secure websockets for bidirectional communication streams,
Relay nodes MUST follow [10/WAKU2](../10/waku2.md) specifications with regards
to supporting different transports.
If TCP transport is available,
each relay node MUST support it as transport for both dialing and listening.
In addition,
a relay node SHOULD support secure websockets for bidirectional communication streams,
for example to allow connections from and to web browser-based clients.
A relay node MAY support unsecure websockets if required by the application or running environment.
A relay node MAY support unsecure websockets if required by the application or
running environment.
#### Default services
For each supported shard,
each relay node SHOULD enable and support the following protocols as a service node:
1. [12/WAKU2-FILTER](../12/filter.md) to allow resource-restricted peers to subscribe to messages matching a specific content filter.
2. [13/WAKU2-STORE](../13/store.md) to allow other peers to request historical messages from this node.
3. [19/WAKU2-LIGHTPUSH](../19/lightpush.md) to allow resource-restricted peers to request publishing a message to the network on their behalf.
4. [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md) to allow resource-restricted peers to discover more peers in a resource efficient way.
1. [12/WAKU2-FILTER](../12/filter.md) to allow resource-restricted peers to subscribe
to messages matching a specific content filter.
2. [13/WAKU2-STORE](../13/store.md) to allow other peers to request historical messages
from this node.
3. [19/WAKU2-LIGHTPUSH](../19/lightpush.md) to allow resource-restricted peers to
request publishing a message to the network on their behalf.
4. [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md)
to allow resource-restricted peers to discover more peers
in a resource efficient way.
#### Store service nodes
Each relay node SHOULD support [13/WAKU2-STORE](../13/store.md) as a store service node,
for each supported shard.
The store SHOULD be configured to retain at least `12` hours of messages per supported shard.
Store service nodes SHOULD only store messages with a valid [`rate_limit_proof`](#message-attributes) attribute.
Each relay node SHOULD support [13/WAKU2-STORE](../13/store.md)
as a store service node, for each supported shard.
The store SHOULD be configured to retain at least `12` hours of messages
per supported shard.
Store service nodes SHOULD only store messages
with a valid [`rate_limit_proof`](#message-attributes) attribute.
#### Non-relay nodes
Nodes MAY opt out of relay functionality on any network shard
and instead request services from relay nodes as clients
using any of the defined service protocols:
1. [12/WAKU2-FILTER](../12/filter.md) to subscribe to messages matching a specific content filter.
2. [13/WAKU2-STORE](../13/store.md) to request historical messages matching a specific content filter.
3. [19/WAKU2-LIGHTPUSH](../19/lightpush.md) to request publishing a message to the network.
4. [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md) to discover more peers in a resource efficient way.
1. [12/WAKU2-FILTER](../12/filter.md)
to subscribe to messages matching a specific content filter.
2. [13/WAKU2-STORE](../13/store.md)
to request historical messages matching a specific content filter.
3. [19/WAKU2-LIGHTPUSH](../19/lightpush.md)
to request publishing a message to the network.
4. [WAKU2-PEER-EXCHANGE](https://github.com/waku-org/specs/blob/master/standards/core/peer-exchange.md)
to discover more peers in a resource efficient way.
#### Store client nodes
Nodes MAY request historical messages from [13/WAKU2-STORE](../13/store.md) service nodes as store clients.
A store client SHOULD discard any messages retrieved from a store service node that do not contain a valid [`rate_limit_proof`](#message-attributes) attribute.
The client MAY consider service nodes returning messages without a valid [`rate_limit_proof`](#message-attributes) attribute as untrustworthy.
Nodes MAY request historical messages from [13/WAKU2-STORE](../13/store.md)
service nodes as store clients.
A store client SHOULD discard any messages retrieved from a store service node
that do not contain a valid [`rate_limit_proof`](#message-attributes) attribute.
The client MAY consider service nodes returning messages
without a valid [`rate_limit_proof`](#message-attributes) attribute as untrustworthy.
The mechanism by which this may happen is currently underdefined.
### Applications
Applications are the higher-layer projects or platforms that make use of the generalized messaging capability of the network.
In other words, an application defines a payload used in the various [10/WAKU2](../10/waku2.md) protocols.
Any participant in an application SHOULD make use of an underlying node in order to communicate on the network.
Applications are the higher-layer projects or
platforms that make use of the generalized messaging capability of the network.
In other words,
an application defines a payload used in the various [10/WAKU2](../10/waku2.md) protocols.
Any participant in an application SHOULD make use of an underlying node
in order to communicate on the network.
Applications SHOULD make use of an [autosharding](#autosharding) API
to allow the underlying node to automatically select the target shard on the Waku Network.
to allow the underlying node to automatically select the target shard
on the Waku Network.
See the section on [autosharding](#autosharding) for more.
## RLN rate-limiting
The [17/WAKU2-RLN-RELAY](../17/rln-relay.md) protocol uses [RLN-V2](https://github.com/vacp2p/rfc-index/blob/a5b24ac0a27da361312260f9da372a0e6e812212/vac/raw/rln-v2.md) proofs
to ensure that a pre-agreed rate limit of `x` messages every `y` seconds is not exceeded by any publisher.
The [17/WAKU2-RLN-RELAY](../17/rln-relay.md) protocol uses [RLN-V2](https://github.com/vacp2p/rfc-index/blob/a5b24ac0a27da361312260f9da372a0e6e812212/vac/raw/rln-v2.md)
proofs to ensure that a pre-agreed rate limit
of `x` messages every `y` seconds is not exceeded by any publisher.
While the network is under capacity,
individual relayers MAY choose to freely route messages without RLN proofs
up to a discretionary bandwidth limit,
after which messages without proofs MUST be discarded by relay nodes.
This bandwidth limit SHOULD be enforced using a [bandwidth validation mechanism](#free-bandwidth-exceeded) separate from a RLN rate-limiting.
This implies that quality of service and reliability is significantly lower for messages without proofs
This bandwidth limit SHOULD be enforced using a [bandwidth validation mechanism](#free-bandwidth-exceeded)
separate from a RLN rate-limiting.
This implies that quality of service and
reliability is significantly lower for messages without proofs
and at times of high network utilization these messages may not be relayed at all.
### RLN Parameters
The Waku Network uses the following RLN parameters:
* `rlnRelayUserMessageLimit=100`: Amount of messages that a membership is allowed to publish per epoch. Configurable between `0` and `MAX_MESSAGE_LIMIT`.
* `rlnRelayUserMessageLimit=100`:
Amount of messages that a membership is allowed to publish per epoch.
Configurable between `0` and `MAX_MESSAGE_LIMIT`.
* `rlnEpochSizeSec=600`: Size of the epoch in seconds.
* `rlnRelayChainId=11155111`: Network in which the RLN contract is deployed, aka Sepolia.
* `rlnRelayEthContractAddress=0xCB33Aa5B38d79E3D9Fa8B10afF38AA201399a7e3`: Network address where RLN memberships are stored.
* `staked_fund=0`: In other words, the Waku Network does not use RLN staking. Registering a membership just requires to pay gas.
* `MAX_MESSAGE_LIMIT=100`: Maximum amount of messages allowed per epoch for any membership. Enforced in the contract.
* `max_epoch_gap=20`: Maximum allowed gap in seconds into the past or future compared to the validator's clock.
* `rlnRelayChainId=11155111`: Network in which the RLN contract is deployed,
aka Sepolia.
* `rlnRelayEthContractAddress=0xCB33Aa5B38d79E3D9Fa8B10afF38AA201399a7e3`:
Network address where RLN memberships are stored.
* `staked_fund=0`: In other words, the Waku Network does not use RLN staking.
Registering a membership just requires to pay gas.
* `MAX_MESSAGE_LIMIT=100`:
Maximum amount of messages allowed per epoch for any membership.
Enforced in the contract.
* `max_epoch_gap=20`: Maximum allowed gap in seconds into the past or
future compared to the validator's clock.
Nodes MUST _reject_ messages not respecting any of these parameters.
Nodes SHOULD use Network Time Protocol (NTP) to synchronize their own clocks,
@@ -189,36 +241,53 @@ according to the rules discussed under [message validation](#message-validation)
### Message Attributes
- The mandatory `payload` attribute MUST contain the message data payload as crafted by the application.
- The mandatory `content_topic` attribute MUST specify a string identifier that can be used for content-based filtering.
* The mandatory `payload` attribute MUST contain the message data payload
as crafted by the application.
* The mandatory `content_topic` attribute MUST specify a string identifier
that can be used for content-based filtering.
This is also crafted by the application.
See [Autosharding](#autosharding) for more on the content topic format.
- The optional `meta` attribute MAY be omitted.
* The optional `meta` attribute MAY be omitted.
If present, will form part of the message uniqueness vector described in [14/WAKU2-MESSAGE](../14/message.md).
- The optional `version` attribute SHOULD be set to `0`. It MUST be interpreted as `0` if not present.
- The mandatory `timestamp` attribute MUST contain the Unix epoch time at which the message was generated by the application.
* The optional `version` attribute SHOULD be set to `0`.
It MUST be interpreted as `0` if not present.
* The mandatory `timestamp` attribute MUST contain the Unix epoch time
at which the message was generated by the application.
The value MUST be in nanoseconds.
It MAY contain a fudge factor of up to 1 seconds in either direction to improve resistance to timing attacks.
- The optional `ephemeral` attribute MUST be set to `true` if the message should not be persisted by the Waku Network.
- The optional `rate_limit_proof` attribute SHOULD be populated with the RLN proof as set out in [RLN Proofs](#rln-proofs).
It MAY contain a fudge factor of up to 1 seconds in either direction
to improve resistance to timing attacks.
* The optional `ephemeral` attribute MUST be set to `true`,
if the message should not be persisted by the Waku Network.
* The optional `rate_limit_proof` attribute SHOULD be populated with the RLN proof
as set out in [RLN Proofs](#rln-proofs).
Messages with this field unpopulated MAY be discarded from the network by relayers.
This field MUST be populated if the message should be persisted by the Waku Network.
### Message Size
Any [14/WAKU2-MESSAGE](../14/message.md) published to the network MUST NOT exceed an absolute maximum size of `150` kilobytes.
This limit applies to the entire message after protobuf serialization, including attributes.
It is RECOMMENDED not to exceed an average size of `4` kilobytes for [14/WAKU2-MESSAGE](../14/message.md) published to the network.
Any [14/WAKU2-MESSAGE](../14/message.md) published to the network
MUST NOT exceed an absolute maximum size of `150` kilobytes.
This limit applies to the entire message after protobuf serialization,
including attributes.
It is RECOMMENDED not to exceed an average size of `4` kilobytes
for [14/WAKU2-MESSAGE](../14/message.md) published to the network.
### Message Validation
Relay nodes MUST apply [gossipsub v1.1 validation](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators) to each relayed message and
SHOULD apply all of the rules set out in the section below to determine the validity of a message.
Relay nodes MUST apply [gossipsub v1.1 validation](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators)
to each relayed message and
SHOULD apply all of the rules set out in the section below
to determine the validity of a message.
Validation has one of three outcomes,
repeated here from the [gossipsub specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators) for ease of reference:
1. Accept - the message is considered valid and it MUST be delivered and forwarded to the network.
2. Reject - the message is considered invalid, MUST be rejected and SHOULD trigger a gossipsub scoring penalty against the transmitting peer.
3. Ignore - the message SHOULD NOT be delivered and forwarded to the network, but this MUST NOT trigger a gossipsub scoring penalty against the transmitting peer.
repeated here from the [gossipsub specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#extended-validators)
for ease of reference:
1. Accept - the message is considered valid and
it MUST be delivered and forwarded to the network.
2. Reject - the message is considered invalid, MUST be rejected and
SHOULD trigger a gossipsub scoring penalty against the transmitting peer.
3. Ignore - the message SHOULD NOT be delivered and forwarded to the network,
but this MUST NOT trigger a gossipsub scoring penalty against the transmitting peer.
The following validation rules are defined:
@@ -266,7 +335,8 @@ If a message contains an RLN proof
and the relay node detects double signaling
according to the verification process described in [RLN-V2](https://github.com/vacp2p/rfc-index/blob/a5b24ac0a27da361312260f9da372a0e6e812212/vac/raw/rln-v2.md),
the relay node MUST _reject_ the message
for violating the agreed rate limit of `rlnRelayUserMessageLimit` messages every `rlnEpochSizeSec` second.
for violating the agreed rate limit of `rlnRelayUserMessageLimit` messages
every `rlnEpochSizeSec` second.
This SHOULD trigger a penalty against the transmitting peer.
## Autosharding
@@ -281,7 +351,7 @@ Applications using autosharding MUST use content topics in the format
defined in [WAKU2-RELAY-SHARDING](https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md/#content-topics-format-for-autosharding)
and SHOULD use the short length format:
```
```text
/{application-name}/{version-of-the-application}/{content-topic-name}/{encoding}
```
@@ -300,7 +370,7 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
* [17/WAKU2-RLN-RELAY](../17/rln-relay.md)
* [11/WAKU2-RELAY](../11/relay.md)
* [WAKU2-RELAY-SHARDING](../../core/relay-sharding.md)
* [WAKU-METADATA](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md)
* [66/WAKU2-METADATA](../66/metadata.md)
* [EIP-1459 DNS-based discovery](https://eips.ethereum.org/EIPS/eip-1459)
* [33/WAKU2-DISCV5](../33/discv5.md)
* [12/WAKU2-FILTER](../12/filter.md)

View File

@@ -5,28 +5,34 @@ name: Waku Metadata Protocol
status: draft
editor: Alvaro Revuelta <alrevuelta@status.im>
contributors:
- Filip Dimitrijevic <filip@status.im>
---
## Abstract
This specification describes the metadata that can be associated with a [10/WAKU2](../10/waku2.md) node.
This specification describes the metadata
that can be associated with a [10/WAKU2](/waku/standards/core/10/waku2.md) node.
## Metadata Protocol
Waku specifies a req/resp protocol that provides information about the node's medatadata.
Such metadata is meant to be used by the node to decide if a peer is worth connecting or not.
The node that makes the request, includes its metadata so that the receiver is aware of it,
without requiring an extra interaction.
Waku specifies a req/resp protocol that provides information about the node's medatadata.
Such metadata is meant to be used by the node to decide if a peer is worth connecting
or not.
The node that makes the request,
includes its metadata so that the receiver is aware of it,
without requiring an extra interaction.
The parameters are the following:
* `clusterId`: Unique identifier of the cluster that the node is running in.
* `shards`: Shard indexes that the node is subscribed to.
***Protocol Identifier***
/vac/waku/metadata/1.0.0
> /vac/waku/metadata/1.0.0
### Request
```proto
```protobuf
message WakuMetadataRequest {
optional uint32 cluster_id = 1;
repeated uint32 shards = 2;
@@ -35,7 +41,7 @@ message WakuMetadataRequest {
### Response
```proto
```protobuf
message WakuMetadataResponse {
optional uint32 cluster_id = 1;
repeated uint32 shards = 2;
@@ -44,8 +50,9 @@ message WakuMetadataResponse {
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
Copyright and related rights waived via
[CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## References
- [10/WAKU2](../10/waku2.md)
* [10/WAKU2](/waku/standards/core/10/waku2.md)

View File

@@ -11,11 +11,23 @@ contributors:
- Kim De Mey <kimdemey@status.im>
---
This specification describes the format of Waku packets within the ÐΞVp2p Wire Protocol. This spec substitutes [EIP-627](https://eips.ethereum.org/EIPS/eip-627). Waku is a fork of the original Whisper protocol that enables better usability for resource restricted devices, such as mostly-offline bandwidth-constrained smartphones. It does this through (a) light node support, (b) historic envelopes (with a mailserver) (c) expressing topic interest for better bandwidth usage and (d) basic rate limiting.
This specification describes the format of Waku packets within the ÐΞVp2p Wire Protocol.
This spec substitutes [EIP-627](https://eips.ethereum.org/EIPS/eip-627).
Waku is a fork of the original Whisper protocol
that enables better usability for resource restricted devices,
such as mostly-offline bandwidth-constrained smartphones.
It does this through (a) light node support, (b) historic envelopes
(with a mailserver) (c) expressing topic interest for better bandwidth usage
and (d) basic rate limiting.
## Motivation
Waku was created to incrementally improve in areas that Whisper is lacking in, with special attention to resource restricted devices. We specify the standard for Waku packets in order to ensure forward compatibility of different Waku clients, backwards compatibility with Whisper clients, as well as to allow multiple implementations of Waku and its capabilities. We also modify the language to be more unambiguous, concise and consistent.
Waku was created to incrementally improve in areas that Whisper is lacking in,
with special attention to resource restricted devices.
We specify the standard for Waku packets in order to ensure forward compatibility
of different Waku clients, backwards compatibility with Whisper clients,
as well as to allow multiple implementations of Waku and its capabilities.
We also modify the language to be more unambiguous, concise and consistent.
## Definitions
@@ -30,34 +42,60 @@ Waku was created to incrementally improve in areas that Whisper is lacking in, w
### Use of DevP2P
For nodes to communicate, they MUST implement devp2p and run RLPx. They MUST have some way of connecting to other nodes. Node discovery is largely out of scope for this spec, but see the appendix for some suggestions on how to do this.
For nodes to communicate, they MUST implement devp2p and run RLPx.
They MUST have some way of connecting to other nodes.
Node discovery is largely out of scope for this spec,
but see the appendix for some suggestions on how to do this.
This protocol needs to advertise the `waku/1` [capability](https://ethereum.gitbooks.io/frontier-guide/devp2p.html).
### Gossip based routing
In Whisper, envelopes are gossiped between peers. Whisper is a form of rumor-mongering protocol that works by flooding to its connected peers based on some factors. Envelopes are eligible for retransmission until their TTL expires. A node SHOULD relay envelopes to all connected nodes if an envelope matches their PoW and bloom filter settings. If a node works in light mode, it MAY choose not to forward envelopes. A node MUST NOT send expired envelopes, unless the envelopes are sent as a [8/WAKU-MAIL](../8/mail.md) response. A node SHOULD NOT send an envelope to a peer that it has already sent before.
In Whisper, envelopes are gossiped between peers.
Whisper is a form of rumor-mongering protocol
that works by flooding to its connected peers based on some factors.
Envelopes are eligible for retransmission until their TTL expires.
A node SHOULD relay envelopes to all connected nodes
if an envelope matches their PoW and bloom filter settings.
If a node works in light mode, it MAY choose not to forward envelopes.
A node MUST NOT send expired envelopes,
unless the envelopes are sent as a [8/WAKU-MAIL](../8/mail.md) response.
A node SHOULD NOT send an envelope to a peer that it has already sent before.
### Maximum Packet Size
Nodes SHOULD limit the maximum size of both packets and envelopes. If a packet or envelope exceeds its limit, it MUST be dropped.
Nodes SHOULD limit the maximum size of both packets and envelopes.
If a packet or envelope exceeds its limit, it MUST be dropped.
- **RLPx Packet Size** - This size MUST be checked before a message is decoded.
- **Waku Envelope Size** - Each envelope contained in an RLPx packet MUST then separately be checked against the maximum envelope size.
- **Waku Envelope Size** - Each envelope contained in an RLPx packet
MUST then separately be checked against the maximum envelope size.
Clients MAY use their own maximum packet and envelope sizes. The default values are `1.5mb` for the RLPx Packet and `1mb` for a Waku envelope.
Clients MAY use their own maximum packet and envelope sizes.
The default values are `1.5mb` for the RLPx Packet and `1mb` for a Waku envelope.
## Wire Specification
### Use of RLPx transport protocol
All Waku packets are sent as devp2p RLPx transport protocol, version 5[^1] packets. These packets MUST be RLP-encoded arrays of data containing two objects: packet code followed by another object (whose type depends on the packet code). See [informal RLP spec](https://github.com/ethereum/wiki/wiki/RLP) and the [Ethereum Yellow Paper, appendix B](https://ethereum.github.io/yellowpaper/paper.pdf) for more details on RLP.
All Waku packets are sent as devp2p RLPx transport protocol, version 5[^1] packets.
These packets MUST be RLP-encoded arrays of data containing two objects:
packet code followed by another object (whose type depends on the packet code).
See [informal RLP spec](https://github.com/ethereum/wiki/wiki/RLP) and
the [Ethereum Yellow Paper, appendix B](https://ethereum.github.io/yellowpaper/paper.pdf)
for more details on RLP.
Waku is a RLPx subprotocol called `waku` with version `0`. The version number corresponds to the major version in the header spec. Minor versions should not break compatibility of `waku`, this would result in a new major. (Some exceptions to this apply in the Draft stage of where client implementation is rapidly change).
Waku is a RLPx subprotocol called `waku` with version `0`.
The version number corresponds to the major version in the header spec.
Minor versions should not break compatibility of `waku`,
this would result in a new major.
(Some exceptions to this apply in the Draft stage
of where client implementation is rapidly change).
### ABNF specification
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234) we have the following format:
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234)
we have the following format:
```abnf
; Packet codes 0 - 127 are reserved for Waku protocol
@@ -185,13 +223,15 @@ optional-packet = 11 batch-ack /
packet = "[" required-packet [ optional-packet ] "]"
```
All primitive types are RLP encoded. Note that, per RLP specification, integers are encoded starting from `0x00`.
All primitive types are RLP encoded. Note that, per RLP specification,
integers are encoded starting from `0x00`.
### Packet Codes
The packet codes reserved for Waku protocol: 0 - 127.
Packets with unknown codes MUST be ignored without generating any error, for forward compatibility of future versions.
Packets with unknown codes MUST be ignored without generating any error,
for forward compatibility of future versions.
The Waku sub-protocol MUST support the following packet codes:
@@ -219,17 +259,25 @@ The Status packet serves as a Waku handshake and peers MUST exchange this
packet upon connection. It MUST be sent after the RLPx handshake and prior to
any other Waku packets.
A Waku node MUST await the Status packet from a peer before engaging in other Waku protocol activity with that peer.
When a node does not receive the Status packet from a peer, before a configurable timeout, it SHOULD disconnect from that peer.
A Waku node MUST await the Status packet from a peer
before engaging in other Waku protocol activity with that peer.
When a node does not receive the Status packet from a peer,
before a configurable timeout, it SHOULD disconnect from that peer.
Upon retrieval of the Status packet, the node SHOULD validate the packet
received and validated the Status packet. Note that its peer might not be in
the same state.
When a node is receiving other Waku packets from a peer before a Status
packet is received, the node MUST ignore these packets and SHOULD disconnect from that peer. Status packets received after the handshake is completed MUST also be ignored.
packet is received, the node MUST ignore these packets and
SHOULD disconnect from that peer.
Status packets received after the handshake is completed MUST also be ignored.
The Status packet MUST contain an association list containing various options. All options within this association list are OPTIONAL, ordering of the key-value pairs is not guaranteed and therefore MUST NOT be relied on. Unknown keys in the association list SHOULD be ignored.
The Status packet MUST contain an association list containing various options.
All options within this association list are OPTIONAL,
ordering of the key-value pairs is not guaranteed and
therefore MUST NOT be relied on.
Unknown keys in the association list SHOULD be ignored.
#### Messages
@@ -237,75 +285,105 @@ This packet is used for sending the standard Waku envelopes.
#### Status Update
The Status Update packet is used to communicate an update of the settings of the node.
The Status Update packet is used to communicate an update
of the settings of the node.
The format is the same as the Status packet, all fields are optional.
If none of the options are specified the packet MUST be ignored and considered a noop.
Fields that are omitted are considered unchanged, fields that haven't changed SHOULD not
be transmitted.
If none of the options are specified the packet MUST be ignored and
considered a noop.
Fields that are omitted are considered unchanged,
fields that haven't changed SHOULD not be transmitted.
##### PoW Requirement Field
When PoW Requirement is updated, peers MUST NOT deliver envelopes with PoW lower than the PoW Requirement specified.
When PoW Requirement is updated,
peers MUST NOT deliver envelopes with PoW lower than the PoW Requirement specified.
PoW is defined as average number of iterations, required to find the current BestBit (the number of leading zero bits in the hash), divided by envelope size and TTL:
PoW is defined as average number of iterations,
required to find the current BestBit
(the number of leading zero bits in the hash), divided by envelope size and TTL:
PoW = (2**BestBit) / (size * TTL)
PoW = (2**BestBit) / (size * TTL)
PoW calculation:
fn short_rlp(envelope) = rlp of envelope, excluding env_nonce field.
fn pow_hash(envelope, env_nonce) = sha3(short_rlp(envelope) ++ env_nonce)
fn pow(pow_hash, size, ttl) = 2**leading_zeros(pow_hash) / (size * ttl)
fn short_rlp(envelope) = rlp of envelope, excluding env_nonce field.
fn pow_hash(envelope, env_nonce) = sha3(short_rlp(envelope) ++ env_nonce)
fn pow(pow_hash, size, ttl) = 2**leading_zeros(pow_hash) / (size * ttl)
where size is the size of the RLP-encoded envelope, excluding `env_nonce` field (size of `short_rlp(envelope)`).
where size is the size of the RLP-encoded envelope,
excluding `env_nonce` field (size of `short_rlp(envelope)`).
##### Bloom Filter Field
The bloom filter is used to identify a number of topics to a peer without compromising (too much) privacy over precisely what topics are of interest. Precise control over the information content (and thus efficiency of the filter) may be maintained through the addition of bits.
The bloom filter is used to identify a number of topics to a peer without compromising
(too much) privacy over precisely what topics are of interest.
Precise control over the information content (and thus efficiency of the filter)
may be maintained through the addition of bits.
Blooms are formed by the bitwise OR operation on a number of bloomed topics. The bloom function takes the topic and projects them onto a 512-bit slice. At most, three bits are marked for each bloomed topic.
Blooms are formed by the bitwise OR operation on a number of bloomed topics.
The bloom function takes the topic and projects them onto a 512-bit slice.
At most, three bits are marked for each bloomed topic.
The projection function is defined as a mapping from a 4-byte slice S to a 512-bit slice D; for ease of explanation, S will dereference to bytes, whereas D will dereference to bits.
The projection function is defined as a mapping
from a 4-byte slice S to a 512-bit slice D; for ease of explanation,
S will dereference to bytes, whereas D will dereference to bits.
LET D[*] = 0
FOREACH i IN { 0, 1, 2 } DO
LET n = S[i]
IF S[3] & (2 ** i) THEN n += 256
D[n] = 1
END FOR
LET D[*] = 0
FOREACH i IN { 0, 1, 2 } DO
LET n = S[i]
IF S[3] & (2 ** i) THEN n += 256
D[n] = 1
END FOR
A full bloom filter (all the bits set to 1) means that the node is to be considered a `Full Node` and it will accept any topic.
A full bloom filter (all the bits set to 1)
means that the node is to be considered a `Full Node` and it will accept any topic.
If both topic interest and bloom filter are specified, topic interest always takes precedence and bloom filter MUST be ignored.
If both topic interest and bloom filter are specified,
topic interest always takes precedence and bloom filter MUST be ignored.
If only bloom filter is specified, the current topic interest MUST be discarded and only the updated bloom filter MUST be used when forwarding or posting envelopes.
If only bloom filter is specified,
the current topic interest MUST be discarded and
only the updated bloom filter MUST be used when forwarding or posting envelopes.
A bloom filter with all bits set to 0 signals that the node is not currently interested in receiving any envelope.
A bloom filter with all bits set to 0 signals that the node is not currently
interested in receiving any envelope.
##### Topic Interest Field
Topic interest is used to share a node's interest in envelopes with specific topics. It does this in a more bandwidth considerate way, at the expense of some metadata protection. Peers MUST only send envelopes with specified topics.
Topic interest is used to share a node's interest in envelopes with specific topics.
It does this in a more bandwidth considerate way,
at the expense of some metadata protection.
Peers MUST only send envelopes with specified topics.
It is currently bounded to a maximum of 10000 topics.
If you are interested in more topics than that, this is currently underspecified
and likely requires updating it. The constant is subject to change.
It is currently bounded to a maximum of 10000 topics. If you are interested in more topics than that, this is currently underspecified and likely requires updating it. The constant is subject to change.
If only topic interest is specified, the current bloom filter MUST be discarded and
only the updated topic interest MUST be used when forwarding or posting envelopes.
If only topic interest is specified, the current bloom filter MUST be discarded and only the updated topic interest MUST be used when forwarding or posting envelopes.
An empty array signals that the node is not currently interested in receiving any envelope.
An empty array signals that the node is not currently interested in receiving
any envelope.
##### Rate Limits Field
Rate limits is used to inform other nodes of their self defined rate limits.
In order to provide basic Denial-of-Service attack protection, each node SHOULD define its own rate limits. The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
In order to provide basic Denial-of-Service attack protection,
each node SHOULD define its own rate limits.
The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
Each node MAY decide to whitelist, i.e. do not rate limit, selected IPs or peer IDs.
If a peer exceeds node's rate limits, the connection between them MAY be dropped.
Each node SHOULD broadcast its rate limits to its peers using the `status-update` packet. The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD broadcast its rate limits to its peers
using the `status-update` packet.
The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD respect rate limits advertised by its peers. The number of packets SHOULD be throttled in order not to exceed peer's rate limits. If the limit gets exceeded, the connection MAY be dropped by the peer.
Each node SHOULD respect rate limits advertised by its peers.
The number of packets SHOULD be throttled in order not to exceed peer's rate limits.
If the limit gets exceeded, the connection MAY be dropped by the peer.
Two rate limits strategies are applied:
@@ -318,193 +396,320 @@ The size limit SHOULD be greater or equal than the maximum packet size.
##### Light Node Field
When the node's `light-node` field is set to true, the node SHOULD NOT forward Envelopes from its peers.
When the node's `light-node` field is set to true,
the node SHOULD NOT forward Envelopes from its peers.
A node connected to a peer with the `light-node` field set to true MUST NOT depend on the peer for forwarding Envelopes.
A node connected to a peer with the `light-node` field set to true
MUST NOT depend on the peer for forwarding Envelopes.
##### Confirmations Enabled Field
When the node's `confirmations-enabled` field is set to true, the node SHOULD send [message confirmations](#batch-ack-and-message-response) to its peers.
When the node's `confirmations-enabled` field is set to true,
the node SHOULD send [message confirmations](#batch-ack-and-message-response)
to its peers.
#### Batch Ack and Message Response
Message confirmations tell a node that an envelope originating from it has been received by its peers, allowing a node to know whether an envelope has or has not been received.
Message confirmations tell a node that an envelope
originating from it has been received by its peers,
allowing a node to know whether an envelope has or has not been received.
A node MAY send a message confirmation for any batch of envelopes received with a Messages packet (`0x01`).
A node MAY send a message confirmation for any batch of envelopes
received with a Messages packet (`0x01`).
A message confirmation is sent using Batch Ack packet (`0x0B`) or Message Response packet (`0x0C`). The message confirmation is specified in the [ABNF specification](#abnf-specification).
A message confirmation is sent using Batch Ack packet (`0x0B`) or
Message Response packet (`0x0C`).
The message confirmation is specified in the [ABNF specification](#abnf-specification).
The current `version` in the `confirmation` is `1`.
The supported error codes:
- `1`: time sync error which happens when an envelope is too old or was created in the future (typically because of an unsynchronized clock of a node).
The drawback of sending message confirmations is that it increases the noise in the network because for each sent envelope, a corresponding confirmation is broadcast by one or more peers.
- `1`: time sync error which happens when an envelope is too old or
was created in the future (typically because of an unsynchronized clock of a node).
The drawback of sending message confirmations is that it increases the noise
in the network because for each sent envelope,
a corresponding confirmation is broadcast by one or more peers.
#### P2P Request
This packet is used for sending Dapp-level peer-to-peer requests, e.g. Waku Mail Client requesting historic (expired) envelopes from the [Waku Mail Server](../8/mail.md).
This packet is used for sending Dapp-level peer-to-peer requests,
e.g. Waku Mail Client requesting historic (expired)
envelopes from the [Waku Mail Server](../8/mail.md).
#### P2P Message
This packet is used for sending the peer-to-peer envelopes, which are not supposed to be forwarded any further. E.g. it might be used by the Waku Mail Server for delivery of historic (expired) envelopes, which is otherwise not allowed.
This packet is used for sending the peer-to-peer envelopes,
which are not supposed to be forwarded any further.
E.g. it might be used by the Waku Mail Server for delivery of historic (expired)
envelopes, which is otherwise not allowed.
#### P2P Request Complete
This packet is used to indicate that all envelopes, requested earlier with a P2P Request packet (`0x7E`), have been sent via one or more P2P Message packets (`0x7F`).
This packet is used to indicate that all envelopes,
requested earlier with a P2P Request packet (`0x7E`),
have been sent via one or more P2P Message packets (`0x7F`).
The content of the packet is explained in the [Waku Mail Server](../8/mail.md) specification.
### Payload Encryption
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme with SECP-256k1 public key.
Asymmetric encryption uses the standard Elliptic Curve Integrated Encryption Scheme
with SECP-256k1 public key.
Symmetric encryption uses AES GCM algorithm with random 96-bit nonce.
### Packet code Rationale
Packet codes `0x00` and `0x01` are already used in all Waku / Whisper versions. Packet code `0x02` and `0x03` were previously used in Whisper but are deprecated as of Waku v0.4
Packet codes `0x00` and `0x01` are already used in all Waku / Whisper versions.
Packet code `0x02` and `0x03` were previously used in Whisper but
are deprecated as of Waku v0.4
Packet code `0x22` is used to dynamically change the settings of a node.
Packet codes `0x7E` and `0x7F` may be used to implement Waku Mail Server and Client. Without the P2P Message packet it would be impossible to deliver the historic envelopes, since they will be recognized as expired, and the peer will be disconnected for violating the Waku protocol. They might be useful for other purposes when it is not possible to spend time on PoW, e.g. if a stock exchange will want to provide live feed about the latest trades.
Packet codes `0x7E` and `0x7F` may be used to implement Waku Mail Server and
Client.
Without the P2P Message packet it would be impossible to deliver the historic envelopes,
since they will be recognized as expired, and
the peer will be disconnected for violating the Waku protocol.
They might be useful for other purposes
when it is not possible to spend time on PoW,
e.g. if a stock exchange will want to provide live feed about the latest trades.
## Additional capabilities
Waku supports multiple capabilities. These include light node, rate limiting and bridging of traffic. Here we list these capabilities, how they are identified, what properties they have and what invariants they must maintain.
Waku supports multiple capabilities.
These include light node, rate limiting and bridging of traffic.
Here we list these capabilities, how they are identified,
what properties they have and what invariants they must maintain.
Additionally there is the capability of a mailserver which is documented in its on [specification](../8/mail.md).
Additionally,
there is the capability of a mailserver which is documented in its on [specification](../8/mail.md).
### Light node
The rationale for light nodes is to allow for interaction with waku on resource restricted devices as bandwidth can often be an issue.
The rationale for light nodes is to allow for interaction with waku
on resource restricted devices as bandwidth can often be an issue.
Light nodes MUST NOT forward any incoming envelopes, they MUST only send their own envelopes. When light nodes happen to connect to each other, they SHOULD disconnect. As this would result in envelopes being dropped between the two.
Light nodes MUST NOT forward any incoming envelopes,
they MUST only send their own envelopes.
When light nodes happen to connect to each other, they SHOULD disconnect.
As this would result in envelopes being dropped between the two.
Light nodes are identified by the `light_node` value in the Status packet.
### Accounting for resources (experimental)
Nodes MAY implement accounting, keeping track of resource usage. It is heavily inspired by Swarm's [SWAP protocol](https://www.bokconsulting.com.au/wp-content/uploads/2016/09/tron-fischer-sw3.pdf), and works by doing pairwise accounting for resources.
Nodes MAY implement accounting, keeping track of resource usage.
It is heavily inspired by Swarm's [SWAP protocol](https://www.bokconsulting.com.au/wp-content/uploads/2016/09/tron-fischer-sw3.pdf),
and works by doing pairwise accounting for resources.
Each node keeps track of resource usage with all other nodes. Whenever an envelope is received from a node that is expected (fits bloom filter or topic interest, is legal, etc) this is tracked.
Each node keeps track of resource usage with all other nodes.
Whenever an envelope is received from a node that is expected
(fits bloom filter or topic interest, is legal, etc) this is tracked.
Every epoch (say, every minute or every time an event happens) statistics SHOULD be aggregated and saved by the client:
Every epoch (say, every minute or every time an event happens)
statistics SHOULD be aggregated and saved by the client:
| peer | sent | received |
|-------|------|----------|
| peer1 | 0 | 123 |
| peer2 | 10 | 40 |
In later versions this will be amended by nodes communication thresholds, settlements and disconnect logic.
In later versions this will be amended by nodes communication thresholds,
settlements and disconnect logic.
## Upgradability and Compatibility
### General principles and policy
The currently advertised capability is `waku/1`. This needs to be advertised in the `hello` `ÐΞVp2p` [packet](https://ethereum.gitbooks.io/frontier-guide/devp2p.html).
If a node supports multiple versions of `waku`, those needs to be explicitly advertised. For example if both `waku/0` and `waku/1` are supported, both `waku/0` and `waku/1` MUST be advertised.
The currently advertised capability is `waku/1`.
This needs to be advertised in the `hello` `ÐΞVp2p` [packet](https://ethereum.gitbooks.io/frontier-guide/devp2p.html).
If a node supports multiple versions of `waku`, those needs to be explicitly advertised.
For example if both `waku/0` and `waku/1` are supported,
both `waku/0` and `waku/1` MUST be advertised.
These are policies that guide how we make decisions when it comes to upgradability, compatibility, and extensibility:
These are policies that guide how we make decisions when it comes to upgradability,
compatibility, and extensibility:
1. Waku aims to be compatible with previous and future versions.
2. In cases where we want to break this compatibility, we do so gracefully and as a single decision point.
2. In cases where we want to break this compatibility,
we do so gracefully and as a single decision point.
3. To achieve this, we employ the following two general strategies:
- a) Accretion (including protocol negotiation) over changing data
- b) When we want to change things, we give it a new name (for example, a version number).
- b) When we want to change things, we give it a new name
(for example, a version number).
Examples:
- We enable bridging between `shh/6` and `waku/1` until such a time as when we are ready to gracefully drop support for `shh/6` (1, 2, 3).
- When we add parameter fields, we (currently) do so by accreting them in a list, so old clients can ignore new fields (dynamic list) and new clients can use new capabilities (1, 3).
- To better support (2) and (3) in the future, we will likely release a new version that gives better support for open, growable maps (association lists or native map type) (3)
- When we we want to provide a new set of packets that have different requirements, we do so under a new protocol version and employ protocol versioning. This is a form of accretion at a level above - it ensures a client can support both protocols at once and drop support for legacy versions gracefully. (1,2,3)
- We enable bridging between `shh/6` and
`waku/1` until such a time as when we are ready to gracefully drop support
for `shh/6` (1, 2, 3).
- When we add parameter fields,
we (currently) do so by accreting them in a list,
so old clients can ignore new fields (dynamic list) and
new clients can use new capabilities (1, 3).
- To better support (2) and (3) in the future,
we will likely release a new version that gives better support for open,
growable maps (association lists or native map type) (3)
- When we we want to provide a new set of packets that have different requirements,
we do so under a new protocol version and employ protocol versioning.
This is a form of accretion at a level above -
it ensures a client can support both protocols at once and
drop support for legacy versions gracefully. (1,2,3)
### Backwards Compatibility
Waku is a different subprotocol from Whisper so it isn't directly compatible. However, the data format is the same, so compatibility can be achieved by the use of a bridging mode as described below. Any client which does not implement certain packet codes should gracefully ignore the packets with those codes. This will ensure the forward compatibility.
Waku is a different subprotocol from Whisper so it isn't directly compatible.
However, the data format is the same,
so compatibility can be achieved by the use of a bridging mode as described below.
Any client which does not implement certain packet codes
should gracefully ignore the packets with those codes.
This will ensure the forward compatibility.
### Waku-Whisper bridging
`waku/1` and `shh/6` are different DevP2P subprotocols, however they share the same data format making their envelopes compatible. This means we can bridge the protocols naively, this works as follows.
`waku/1` and `shh/6` are different DevP2P subprotocols,
however they share the same data format making their envelopes compatible.
This means we can bridge the protocols naively, this works as follows.
**Roles:**
- Waku client A, only Waku capability
- Whisper client B, only Whisper capability
- WakuWhisper bridge C, both Waku and Whisper capability
**Flow:**
1. A posts envelope; B posts envelope.
2. C picks up envelope from A and B and relays them both to Waku and Whisper.
3. A receives envelope on Waku; B on Whisper.
**Note**: This flow means if another bridge C1 is active, we might get duplicate relaying for a envelope between C1 and C2. I.e. Whisper(<>Waku<>Whisper)<>Waku, A-C1-C2-B. Theoretically this bridging chain can get as long as TTL permits.
**Note**: This flow means if another bridge C1 is active,
we might get duplicate relaying for a envelope between C1 and C2.
I.e. Whisper(<>Waku<>Whisper)<>Waku, A-C1-C2-B.
Theoretically this bridging chain can get as long as TTL permits.
### Forward Compatibility
It is desirable to have a strategy for maintaining forward compatibility between `waku/1` and future version of waku. Here we outline some concerns and strategy for this.
It is desirable to have a strategy for maintaining forward compatibility
between `waku/1` and future version of waku.
Here we outline some concerns and strategy for this.
- **Connecting to nodes with multiple versions:** The way this SHOULD be accomplished is by negotiating the versions of subprotocols, within the `hello` packet nodes transmit their capabilities along with a version. The highest common version should then be used.
- **Adding new packet codes:** New packet codes can be added easily due to the available packet codes. Unknown packet codes SHOULD be ignored. Upgrades that add new packet codes SHOULD implement some fallback mechanism if no response was received for nodes that do not yet understand this packet.
- **Adding new options in `status-options`:** New options can be added to the `status-options` association list in the `status` and `status-update` packet as options are OPTIONAL and unknown option keys SHOULD be ignored. A node SHOULD NOT disconnect from a peer when receiving `status-options` with unknown option keys.
- **Connecting to nodes with multiple versions:**
The way this SHOULD be accomplished is by negotiating the versions of subprotocols,
within the `hello` packet nodes transmit their capabilities along with a version.
The highest common version should then be used.
- **Adding new packet codes:**
New packet codes can be added easily due to the available packet codes.
Unknown packet codes SHOULD be ignored.
Upgrades that add new packet codes SHOULD implement some fallback mechanism
if no response was received for nodes that do not yet understand this packet.
- **Adding new options in `status-options`:**
New options can be added to the `status-options` association list in the `status`
and `status-update` packet as options are OPTIONAL and
unknown option keys SHOULD be ignored.
A node SHOULD NOT disconnect from a peer when receiving `status-options`
with unknown option keys.
## Appendix A: Security considerations
There are several security considerations to take into account when running Waku. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used. The security considerations for extra capabilities such as [mailservers](../8/mail.md#security-considerations) can be found in their respective specifications.
There are several security considerations to take into account when running Waku.
Chief among them are: scalability, DDoS-resistance and privacy.
These also vary depending on what capabilities are used.
The security considerations for extra capabilities,
such as [mailservers](../8/mail.md#security-considerations)
can be found in their respective specifications.
### Scalability and UX
#### Bandwidth usage:
#### Bandwidth usage
In version 0 of Waku, bandwidth usage is likely to be an issue. For more investigation into this, see the theoretical scaling model described [here](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
In version 0 of Waku, bandwidth usage is likely to be an issue.
For more investigation into this,
see the theoretical scaling model described
[here](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
#### Gossip-based routing:
#### Gossip-based routing
Use of gossip-based routing doesn't necessarily scale. It means each node can see an envelope multiple times, and having too many light nodes can cause propagation probability that is too low. See [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/) for more and a possible Kademlia based alternative.
Use of gossip-based routing doesn't necessarily scale.
It means each node can see an envelope multiple times, and
having too many light nodes can cause propagation probability that is too low.
See [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/)
for more and a possible Kademlia based alternative.
#### Lack of incentives:
#### Lack of incentives
Waku currently lacks incentives to run nodes, which means node operators are more likely to create centralized choke points.
Waku currently lacks incentives to run nodes,
which means node operators are more likely to create centralized choke points.
### Privacy
#### Light node privacy:
#### Light node privacy
The main privacy concern with a light node is that it has to reveal its topic interests (in addition to its IP/ID) to its directed peers. This is because when a light node publishes an envelope, its directed peers will know that the light node owns that envelope (as light nodes do not relay other envelopes). Therefore, the directed peers of a light node can make assumptions about what envelopes (topics) the light node is interested in.
The main privacy concern with a light node
is that it has to reveal its topic interests
(in addition to its IP/ID) to its directed peers.
This is because when a light node publishes an envelope,
its directed peers will know that the light node owns that envelope
(as light nodes do not relay other envelopes).
Therefore, the directed peers of a light node can make assumptions about what envelopes
(topics) the light node is interested in.
#### Mailserver client privacy
#### Mailserver client privacy:
A mailserver client fetches archival envelopes from a mailserver
through a direct connection.
In this direct connection,
the client discloses its IP/ID as well as the topics/ bloom filter
it is interested in to the mailserver.
The collection of such information allows the mailserver to link clients' IP/IDs
to their topic interests and build a profile for each client over time.
As such, the mailserver client has to trust the mailserver with this level of information.
A mailserver client fetches archival envelopes from a mailserver through a direct connection.
In this direct connection, the client discloses its IP/ID as well as the topics/ bloom filter it is interested in to the mailserver.
The collection of such information allows the mailserver to link clients' IP/IDs to their topic interests and build a profile for each client over time.
As such, the mailserver client has to trust the mailserver with this level of information.
#### Bloom filter privacy
#### Bloom filter privacy:
By having a bloom filter where only the topics you are interested in are set,
you reveal which envelopes you are interested in.
This is a fundamental tradeoff between bandwidth usage and privacy,
though the tradeoff space is likely suboptimal in terms of the [Anonymity](https://eprint.iacr.org/2017/954.pdf)
[trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
By having a bloom filter where only the topics you are interested in are set, you reveal which envelopes you are interested in. This is a fundamental tradeoff between bandwidth usage and privacy, though the tradeoff space is likely suboptimal in terms of the [Anonymity](https://eprint.iacr.org/2017/954.pdf) [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
#### Privacy guarantees not rigorous
#### Privacy guarantees not rigorous:
Privacy for Whisper / Waku haven't been studied rigorously for various threat models
like global passive adversary, local active attacker, etc.
This is unlike e.g. Tor and mixnets.
Privacy for Whisper / Waku haven't been studied rigorously for various threat models like global passive adversary, local active attacker, etc. This is unlike e.g. Tor and mixnets.
#### Topic hygiene
#### Topic hygiene:
Similar to bloom filter privacy, if you use a very specific topic you reveal more information. See scalability model linked above.
Similar to bloom filter privacy,
if you use a very specific topic you reveal more information.
See scalability model linked above.
### Spam resistance
**PoW bad for heterogeneous devices:**
Proof of work is a poor spam prevention mechanism. A mobile device can only have a very low PoW in order not to use too much CPU / burn up its phone battery. This means someone can spin up a powerful node and overwhelm the network.
Proof of work is a poor spam prevention mechanism.
A mobile device can only have a very low PoW
in order not to use too much CPU / burn up its phone battery.
This means someone can spin up a powerful node and overwhelm the network.
### Censorship resistance
**Devp2p TCP port blockable:**
By default Devp2p runs on port `30303`, which is not commonly used for any other service. This means it is easy to censor, e.g. airport WiFi. This can be mitigated somewhat by running on e.g. port `80` or `443`, but there are still outstanding issues. See libp2p and Tor's Pluggable Transport for how this can be improved.
By default Devp2p runs on port `30303`,
which is not commonly used for any other service.
This means it is easy to censor, e.g. airport WiFi.
This can be mitigated somewhat by running on e.g. port `80` or `443`,
but there are still outstanding issues.
See libp2p and Tor's Pluggable Transport for how this can be improved.
## Appendix B: Implementation Notes
@@ -519,17 +724,23 @@ By default Devp2p runs on port `30303`, which is not commonly used for any other
Notes useful for implementing Waku mode.
1. Avoid duplicate envelopes
1.Avoid duplicate envelopes
To avoid duplicate envelopes, only connect to one Waku node. Benign duplicate envelopes is an intrinsic property of Whisper which often leads to a N factor increase in traffic, where N is the number of peers you are connected to.
To avoid duplicate envelopes,
only connect to one Waku node.
Benign duplicate envelopes is an intrinsic property of Whisper
which often leads to a N factor increase in traffic,
where N is the number of peers you are connected to.
2. Topic specific recommendations
2.Topic specific recommendations
Consider partition topics based on some usage, to avoid too much traffic on a single topic.
Consider partition topics based on some usage,
to avoid too much traffic on a single topic.
### Node discovery
Resource restricted devices SHOULD use [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459) to discover nodes.
Resource restricted devices SHOULD use [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459)
to discover nodes.
Known static nodes MAY also be used.
@@ -567,16 +778,21 @@ Released [April 21,2020](https://github.com/vacp2p/specs/commit/9e650995f2417984
Released [March 17,2020](https://github.com/vacp2p/specs/commit/7b9dc562bc50c6bb844ac575cb221ec9cda2530a)
- Clarify the preferred way of handling unknown keys in the `status-options` association list.
- Correct spec/implementation mismatch: Change RLP keys to be the their int values in order to reflect production behavior
- Clarify the preferred way of handling unknown keys
in the `status-options` association list.
- Correct spec/implementation mismatch:
Change RLP keys to be the their int values in order to reflect production behavior
### Version 0.4
Released [February 21, 2020](https://github.com/vacp2p/specs/commit/17bd066e317bbe33af07146b721d73f24de47e88).
- Simplify implementation matrix with latest state
- Introduces a new required packet code Status Code (`0x22`) for communicating option changes
- Deprecates the following packet codes: PoW Requirement (`0x02`), Bloom Filter (`0x03`), Rate limits (`0x20`), Topic interest (`0x21`) - all superseded by the new Status Code (`0x22`)
- Introduces a new required packet code Status Code (`0x22`)
for communicating option changes
- Deprecates the following packet codes: PoW Requirement (`0x02`),
Bloom Filter (`0x03`), Rate limits (`0x20`), Topic interest (`0x21`) -
all superseded by the new Status Code (`0x22`)
- Increased `topic-interest` capacity from 1000 to 10000
### Version 0.3
@@ -586,7 +802,8 @@ Released [February 13, 2020](https://github.com/vacp2p/specs/commit/73138d6ba954
- Recommend DNS based node discovery over other Discovery methods.
- Mark spec as Draft mode in terms of its lifecycle.
- Simplify Changelog and misc formatting.
- Handshake/Status packet not compatible with shh/6 nodes; specifying options as association list.
- Handshake/Status packet not compatible with shh/6 nodes;
specifying options as association list.
- Include topic-interest in Status handshake.
- Upgradability policy.
- `topic-interest` packet code.
@@ -602,7 +819,8 @@ Released [December 10, 2019](https://github.com/vacp2p/specs/blob/waku-0.2.0/wak
- More details on handshake modifications.
- Accounting for resources mode (experimental)
- Appendix with security considerations: scalability and UX, privacy, and spam resistance.
- Appendix with implementation notes and implementation matrix across various clients with breakdown per capability.
- Appendix with implementation notes and
implementation matrix across various clients with breakdown per capability.
- More details on handshake and parameters.
- Describe rate limits in more detail.
- More details on mailserver and mail client API.
@@ -629,7 +847,6 @@ confirmations-enabled and rate-limits
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
## Footnotes
[^1]: Felix Lange et al. [The RLPx Transport Protocol](https://github.com/ethereum/devp2p/blob/master/rlpx.md). Ethereum.

View File

@@ -9,24 +9,29 @@ contributors:
- Kim De Mey <kimdemey@status.im>
---
This specification describes the encryption, decryption and signing of the content in the [data field used in Waku](../6/waku1.md/#abnf-specification).
This specification describes the encryption,
decryption and signing of the content in the [data field used in Waku](../6/waku1.md/#abnf-specification).
## Specification
The `data` field is used within the `waku envelope`, the field MUST contain the encrypted payload of the envelope.
The `data` field is used within the `waku envelope`,
the field MUST contain the encrypted payload of the envelope.
The fields that are concatenated and encrypted as part of the `data` field are:
- flags
- auxiliary field
- payload
- padding
- signature
In case of symmetric encryption, a `salt` (a.k.a. AES Nonce, 12 bytes) field MUST be appended.
- flags
- auxiliary field
- payload
- padding
- signature
In case of symmetric encryption, a `salt`
(a.k.a. AES Nonce, 12 bytes) field MUST be appended.
### ABNF
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234) we have the following format:
Using [Augmented Backus-Naur form (ABNF)](https://tools.ietf.org/html/rfc5234)
we have the following format:
```abnf
; 1 byte; first two bits contain the size of auxiliary field,
@@ -53,11 +58,23 @@ data = flags auxiliary-field payload padding [signature] [salt]
### Signature
Those unable to decrypt the envelope data are also unable to access the signature. The signature, if provided, is the ECDSA signature of the Keccak-256 hash of the unencrypted data using the secret key of the originator identity. The signature is serialized as the concatenation of the `R`, `S` and `V` parameters of the SECP-256k1 ECDSA signature, in that order. `R` and `S` MUST be big-endian encoded, fixed-width 256-bit unsigned. `V` MUST be an 8-bit big-endian encoded, non-normalized and should be either 27 or 28.
Those unable to decrypt the envelope data are also unable to access the signature.
The signature, if provided,
is the ECDSA signature of the Keccak-256 hash of the unencrypted data
using the secret key of the originator identity.
The signature is serialized as the concatenation of the `R`, `S` and
`V` parameters of the SECP-256k1 ECDSA signature, in that order.
`R` and `S` MUST be big-endian encoded, fixed-width 256-bit unsigned.
`V` MUST be an 8-bit big-endian encoded,
non-normalized and should be either 27 or 28.
### Padding
The padding field is used to align data size, since data size alone might reveal important metainformation. Padding can be arbitrary size. However, it is recommended that the size of Data Field (excluding the Salt) before encryption (i.e. plain text) SHOULD be factor of 256 bytes.
The padding field is used to align data size,
since data size alone might reveal important metainformation.
Padding can be arbitrary size.
However, it is recommended that the size of Data Field (excluding the Salt)
before encryption (i.e. plain text) SHOULD be factor of 256 bytes.
## Copyright

View File

@@ -12,19 +12,30 @@ contributors:
## Abstract
In this specification, we describe Mailservers. These are nodes responsible for archiving envelopes and delivering them to peers on-demand.
In this specification, we describe Mailservers.
These are nodes responsible for archiving envelopes and
delivering them to peers on-demand.
## Specification
A node which wants to provide mailserver functionality MUST store envelopes from incoming Messages packets (Waku packet-code `0x01`). The envelopes can be stored in any format, however they MUST be serialized and deserialized to the Waku envelope format.
A node which wants to provide mailserver functionality MUST store envelopes
from incoming Messages packets (Waku packet-code `0x01`).
The envelopes can be stored in any format,
however they MUST be serialized and deserialized to the Waku envelope format.
A mailserver SHOULD store envelopes for all topics to be generally useful for any peer, however for specific use cases it MAY store envelopes for a subset of topics.
A mailserver SHOULD store envelopes for all topics
to be generally useful for any peer,
however for specific use cases it MAY store envelopes for a subset of topics.
### Requesting Historic Envelopes
In order to request historic envelopes, a node MUST send a packet P2P Request (`0x7e`) to a peer providing mailserver functionality. This packet requires one argument which MUST be a Waku envelope.
In order to request historic envelopes,
a node MUST send a packet P2P Request (`0x7e`)
to a peer providing mailserver functionality.
This packet requires one argument which MUST be a Waku envelope.
In the Waku envelope's payload section, there MUST be RLP-encoded information about the details of the request:
In the Waku envelope's payload section,
there MUST be RLP-encoded information about the details of the request:
```abnf
; UNIX time in seconds; oldest requested envelope's creation time
@@ -55,21 +66,44 @@ payload-with-topic = "[" lower upper bloom limit cursor [ topics ] "]"
payload = payload-with-topic | payload-without-topic
```
The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit` so that the requester can send another request using the obtained `Cursor` value. What exactly is in the `Cursor` is up to the implementation. The requester SHOULD NOT use a `Cursor` obtained from one mailserver in a request to another mailserver because the format or the result MAY be different.
The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and
`Upper` is greater than `Limit` so that the requester can send another request
using the obtained `Cursor` value.
What exactly is in the `Cursor` is up to the implementation.
The requester SHOULD NOT use a `Cursor` obtained from one mailserver in a request
to another mailserver because the format or the result MAY be different.
The envelope MUST be encrypted with a symmetric key agreed between the requester and Mailserver.
The envelope MUST be encrypted with a symmetric key agreed between the requester
and Mailserver.
If `Topics` is used the `Cursor` field MUST be specified for the argument order to be unambiguous. However, it MAY be set to `null`. `Topics` is used to specify which topics a node is interested in. If `Topics` is not empty, a mailserver MUST only send envelopes that belong to a topic from `Topics` list and `Bloom` value MUST be ignored.
If `Topics` is used the `Cursor` field MUST be specified
for the argument order to be unambiguous.
However, it MAY be set to `null`.
`Topics` is used to specify which topics a node is interested in.
If `Topics` is not empty,
a mailserver MUST only send envelopes that belong to a topic from `Topics` list and
`Bloom` value MUST be ignored.
### Receiving Historic Envelopes
Historic envelopes MUST be sent to a peer as a packet with a P2P Message code (`0x7f`) followed by an array of Waku envelopes. A Mailserver MUST limit the amount of messages sent, either by the `Limit` specified in the request or limited to the maximum [RLPx packet size](./waku#maximum-packet-size), whichever limit comes first.
Historic envelopes MUST be sent to a peer as a packet with a P2P Message code (`0x7f`)
followed by an array of Waku envelopes.
A Mailserver MUST limit the amount of messages sent,
either by the `Limit` specified in the request or
limited to the maximum [RLPx packet size](./waku#maximum-packet-size),
whichever limit comes first.
In order to receive historic envelopes from a mailserver, a node MUST trust the selected mailserver, that is allow to receive expired packets with the P2P Message code. By default, such packets are discarded.
In order to receive historic envelopes from a mailserver,
a node MUST trust the selected mailserver,
that is allow to receive expired packets with the P2P Message code.
By default, such packets are discarded.
Received envelopes MUST be passed through the Whisper envelope pipelines so that they are picked up by registered filters and passed to subscribers.
Received envelopes MUST be passed through the Whisper envelope pipelines
so that they are picked up by registered filters and passed to subscribers.
For a requester, to know that all envelopes have been sent by mailserver, it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by a list with:
For a requester, to know that all envelopes have been sent by mailserver,
it SHOULD handle P2P Request Complete code (`0x7d`).
This code is followed by a list with:
```abnf
; array with a Keccak-256 hash of the envelope containing the original request.
@@ -84,27 +118,39 @@ cursor = *OCTET
payload = "[" request-id last-envelope-hash [ cursor ] "]"
```
If `Cursor` is not empty, it means that not all envelopes were sent due to the set `Limit` in the request. One or more consecutive requests MAY be sent with `Cursor` field filled in in order to receive the rest of the envelopes.
If `Cursor` is not empty,
it means that not all envelopes were sent due to the set `Limit` in the request.
One or more consecutive requests MAY be sent with `Cursor` field filled
in order to receive the rest of the envelopes.
### Security considerations
There are several security considerations to take into account when running or interacting with Mailservers. Chief among them are: scalability, DDoS-resistance and privacy.
There are several security considerations to take into account when running or
interacting with Mailservers.
Chief among them are: scalability, DDoS-resistance and privacy.
**Mailserver High Availability requirement:**
A mailserver has to be online to receive envelopes for other nodes, this puts a high availability requirement on it.
A mailserver has to be online to receive envelopes for other nodes,
this puts a high availability requirement on it.
**Mailserver client privacy:**
A mailserver client fetches archival envelopes from a mailserver through a direct connection.
In this direct connection, the client discloses its IP/ID as well as the topics/ bloom filter it is interested in to the mailserver.
The collection of such information allows the mailserver to link clients' IP/IDs to their topic interests and build a profile for each client over time.
A mailserver client fetches archival envelopes from a mailserver
through a direct connection.
In this direct connection,
the client discloses its IP/ID as well as the topics/ bloom filter
it is interested in to the mailserver.
The collection of such information allows the mailserver to link clients' IP/IDs
to their topic interests and build a profile for each client over time.
As such, the mailserver client has to trust the mailserver with this level of information.
A similar concern exists for the light nodes and their direct peers which is discussed in the security considerations of [6/WAKU1](../6/waku1.md).
A similar concern exists for the light nodes and
their direct peers which is discussed in the security considerations of [6/WAKU1](../6/waku1.md).
**Mailserver trusted connection:**
A mailserver has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning mailserver can overwhelm an individual node.
A mailserver has a direct TCP connection, which means they are trusted to send traffic.
This means a malicious or malfunctioning mailserver can overwhelm an individual node.
## Changelog

View File

@@ -9,8 +9,10 @@ contributors:
- Oskar Thorén <oskarth@titanproxy.com>
---
This specification describes the RPC API that Waku nodes MAY adhere to. The unified API allows clients to easily
be able to connect to any node implementation. The API described is privileged as a node stores the keys of clients.
This specification describes the RPC API that Waku nodes MAY adhere to.
The unified API allows clients to easily
be able to connect to any node implementation.
The API described is privileged as a node stores the keys of clients.
## Introduction
@@ -20,7 +22,10 @@ This API is based off the [Whisper V6 RPC API](https://github.com/ethereum/go-et
### Transport
Nodes SHOULD expose a [JSON RPC](https://www.jsonrpc.org/specification) API that can be accessed. The JSON RPC version SHOULD be `2.0`. Below is an example request:
Nodes SHOULD expose a [JSON RPC](https://www.jsonrpc.org/specification) API
that can be accessed.
The JSON RPC version SHOULD be `2.0`.
Below is an example request:
```json
{
@@ -46,7 +51,11 @@ In this section you will find objects used throughout the JSON RPC API.
#### Message
The message object represents a Waku message. Below you will find the description of the attributes contained in the message object. A message is the decrypted payload and padding of an [envelope](../7/data.md) along with all of its metadata and other extra information such as the hash.
The message object represents a Waku message.
Below you will find the description of the attributes contained in the message object.
A message is the decrypted payload and
padding of an [envelope](../7/data.md) along with all of its metadata and
other extra information such as the hash.
| Field | Type | Description |
| ----: | :--: | ----------- |
@@ -62,7 +71,8 @@ The message object represents a Waku message. Below you will find the descriptio
#### Filter
The filter object represents filters that can be applied to retrieve messages. Below you will find the description of the attributes contained in the filter object.
The filter object represents filters that can be applied to retrieve messages.
Below you will find the description of the attributes contained in the filter object.
| Field | Type | Description |
| ----: | :--: | ----------- |
@@ -73,7 +83,9 @@ The filter object represents filters that can be applied to retrieve messages. B
| `topics` | array | Array of possible topics, this can also contain partial topics |
| `allowP2P` | boolean | Indicates if this filter allows processing of direct peer-to-peer messages |
All fields are optional, however `symKeyID` or `privateKeyID` must be present, it cannot be both. Additionally, the `topics` field is only optional when an asymmetric key is used.
All fields are optional, however `symKeyID` or `privateKeyID` must be present,
it cannot be both.
Additionally, the `topics` field is only optional when an asymmetric key is used.
### Methods
@@ -93,11 +105,11 @@ none
The `waku_info` method returns information about a Waku node.
##### Parameters
Parameters
none
##### Response
Response
The response is an `Object` containing the following fields:
@@ -108,202 +120,231 @@ The response is an `Object` containing the following fields:
#### `waku_setMaxEnvelopeSize`
Sets the maximum envelope size allowed by this node. Any envelopes larger than this size both incoming and outgoing will be rejected. The envelope size can never exceed the underlying envelope size of `10mb`.
Sets the maximum envelope size allowed by this node.
Any envelopes larger than this size both incoming and outgoing will be rejected.
The envelope size can never exceed the underlying envelope size of `10mb`.
##### Parameters
Parameters
- **number** - The message size in bytes.
- **number** - The message size in bytes.
##### Response
Response
- **bool** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_setMinPoW`
Sets the minimal PoW required by this node.
##### Parameters
Parameters
- **number** - The new PoW requirement.
- **number** - The new PoW requirement.
##### Response
Response
- **bool** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_markTrustedPeer`
Marks a specific peer as trusted allowing it to send expired messages.
##### Parameters
Parameters
- **string** - `enode` of the peer.
##### Response
Response
- **bool** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_newKeyPair`
Generates a keypair used for message encryption and decryption.
##### Parameters
Parameters
none
##### Response
Response
- **string** - Key ID on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** -
Key ID on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_addPrivateKey`
Stores a key and returns its ID.
##### Parameters
Parameters
- **string** - Private key as hex bytes.
##### Response
Response
- **string** - Key ID on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - Key ID on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_deleteKeyPair`
Deletes a specific key if it exists.
##### Parameters
Parameters
- **string** - ID of the Key pair.
##### Response
Response
- **bool** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** - `true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_hasKeyPair`
Checks if the node has a private key of a key pair matching the given ID.
##### Parameters
Parameters
- **string** - ID of the Key pair.
##### Response
Response
- **bool** - `true` or `false` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** -
`true` or
`false` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_getPublicKey`
Returns the public key for an ID.
##### Parameters
Parameters
- **string** - ID of the Key.
##### Response
Response
- **string** - The public key or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - The public key or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_getPrivateKey`
Returns the private key for an ID.
##### Parameters
Parameters
- **string** - ID of the Key.
##### Response
Response
- **string** - The private key or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - The private key or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_newSymKey`
Generates a random symmetric key and stores it under an ID. This key can be used to encrypt and decrypt messages where the key is known to both parties.
Generates a random symmetric key and stores it under an ID.
This key can be used to encrypt and
decrypt messages where the key is known to both parties.
##### Parameters
Parameters
none
##### Response
Response
- **string** - The key ID or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - The key ID or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_addSymKey`
Stores the key and returns its ID.
##### Parameters
Parameters
- **string** - The raw key for symmetric encryption hex encoded.
##### Response
Response
- **string** - The key ID or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - The key ID or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_generateSymKeyFromPassword`
Generates the key from a password and stores it.
##### Parameters
Parameters
- **string** - The password.
- **string** - The password.
##### Response
Response
- **string** - The key ID or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - The key ID or an [error](https://www.jsonrpc.org/specification#error_object)
on failure.
#### `waku_hasSymKey`
Returns whether there is a key associated with the ID.
##### Parameters
Parameters
- **string** - ID of the Key.
##### Response
Response
- **bool** - `true` or `false` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** - `true` or `false` or an [error](https://www.jsonrpc.org/specification#error_object)
on failure.
#### `waku_getSymKey`
Returns the symmetric key associated with an ID.
##### Parameters
Parameters
- **string** - ID of the Key.
##### Response
Response
- **string** - Raw key on success or an [error](https://www.jsonrpc.org/specification#error_object) of failure.
- **string** - Raw key on success or
an [error](https://www.jsonrpc.org/specification#error_object) of failure.
#### `waku_deleteSymKey`
Deletes the key associated with an ID.
##### Parameters
Parameters
- **string** - ID of the Key.
##### Response
Response
- **bool** - `true` or `false` or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** -
`true` or `false` or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_subscribe`
Creates and registers a new subscription to receive notifications for inbound Waku messages.
Creates and
registers a new subscription to receive notifications for inbound Waku messages.
##### Parameters
Parameters
The parameters for this request is an array containing the following fields:
1. **string** - The ID of the function call, in case of Waku this must contain the value "messages".
1. **string** - The ID of the function call,
in case of Waku this must contain the value "messages".
2. **object** - The [message filter](#filter).
##### Response
Response
- **string** - ID of the subscription or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **string** - ID of the subscription or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
###### Notifications
Notifications
Notifications received by the client contain a [message](#message) matching the filter. Below is an example notification:
Notifications received by the client contain a [message](#message) matching the filter.
Below is an example notification:
```json
{
@@ -328,81 +369,90 @@ Notifications received by the client contain a [message](#message) matching the
#### `waku_unsubscribe`
Cancels and removes an existing subscription. The node MUST stop sending the client notifications.
Cancels and removes an existing subscription.
The node MUST stop sending the client notifications.
##### Parameters
Parameters
- **string** - The subscription ID.
- **string** - The subscription ID.
##### Response
Response
- **bool** - `true` or `false`
#### `waku_newMessageFilter`
Creates a new message filter within the node. This filter can be used to poll for new messages that match the criteria.
Creates a new message filter within the node.
This filter can be used to poll for new messages that match the criteria.
##### Parameters
Parameters
The request must contain a [message filter](#filter) as its parameter.
The request must contain a [message filter](#filter) as its parameter.
##### Response
Response
- **string** - The ID of the filter.
- **string** - The ID of the filter.
#### `waku_deleteMessageFilter`
Removes a message filter from the node.
##### Parameters
Parameters
- **string** - ID of the filter created with [`waku_newMessageFilter`](#waku_newMessageFilter).
- **string** - ID of the filter created with [`waku_newMessageFilter`](#waku_newmessagefilter).
##### Response
Response
- **bool** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
- **bool** - `true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_getFilterMessages`
Retrieves messages that match a filter criteria and were received after the last time this function was called.
Retrieves messages that match a filter criteria and
were received after the last time this function was called.
##### Parameters
Parameters
- **string** - ID of the filter created with [`waku_newMessageFilter`](#waku_newMessageFilter).
- **string** - ID of the filter created with [`waku_newMessageFilter`](#waku_newmessagefilter).
##### Response
Response
The response contains an array of [messages](#messages) or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
The response contains an array of [messages](#message) or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
#### `waku_post`
The `waku_post` method creates a waku envelope and propagates it to the network.
##### Parameters
Parameters
The parameters is an `Object` containing the following fields:
- **`symKeyID` [string]** `optional` - The ID of the symmetric key used for encryption
- **`pubKey` [string]** `optional` - The public key for message encryption.
- **`sig` [string]** `optional` - The ID of the signing key.
- **`ttl` [number]** - The time-to-live in seconds.
- **`topic` [string]** - 4 bytes message topic.
- **`payload` [string]** - The payload to be encrypted.
- **`padding` [string]** `optional` - The padding, a byte array of arbitrary length.
- **`powTime` [number]** - Maximum time in seconds to be spent on the proof of work.
- **`powTarget` [number]** - Minimal PoW target required for this message.
- **`targetPeer` [string]** `optional` - The optional peer ID for peer-to-peer messages.
*Either the **`symKeyID`** or the **`pubKey`** need to be present. It can not be both.*
#### Response
- **`symKeyID` [string]** `optional` - The ID of the symmetric key used for encryption
- **`pubKey` [string]** `optional` - The public key for message encryption.
- **`sig` [string]** `optional` - The ID of the signing key.
- **`ttl` [number]** - The time-to-live in seconds.
- **`topic` [string]** - 4 bytes message topic.
- **`payload` [string]** - The payload to be encrypted.
- **`padding` [string]** `optional` - The padding, a byte array of arbitrary length.
- **`powTime` [number]** - Maximum time in seconds to be spent on the proof of work.
- **`powTarget` [number]** - Minimal PoW target required for this message.
- **`targetPeer` [string]** `optional` - The optional peer ID for peer-to-peer messages.
- **bool** - `true` on success or an [error](https://www.jsonrpc.org/specification#error_object) on failure.
*Either the **`symKeyID`** or the **`pubKey`** need to be present.
It can not be both.*
Response
- **bool** -
`true` on success or
an [error](https://www.jsonrpc.org/specification#error_object) on failure.
## Changelog
| Version | Comment |
| :--------------------------------------------------------------------------------------:| ---------------- |
| [1.0.0](https://github.com/vacp2p/specs/commit/bc7e75ebb2e45d2cbf6ab27352c113e666df37c8)| Initial release. |
| [1.0.0](https://github.com/vacp2p/specs/commit/bc7e75ebb2e45d2cbf6ab27352c113e666df37c8)| Initial release. |
## Copyright