mirror of
https://github.com/vacp2p/rfc-index.git
synced 2026-01-11 08:38:19 -05:00
Compare commits
18 Commits
community_
...
zerokit-ap
| Author | SHA1 | Date | |
|---|---|---|---|
|
df73e89dfc | ||
|
|
8854180f9e | ||
|
|
6157167a75 | ||
|
|
376b60efcd | ||
|
|
444eea0876 | ||
|
|
fd9f767fcc | ||
|
|
99a11e7e08 | ||
|
|
3cd37b4538 | ||
|
|
b9a08305bb | ||
|
|
bf198face6 | ||
|
|
dabc31786b | ||
|
|
b2f35644a4 | ||
|
|
4f54254706 | ||
|
|
7f1df32779 | ||
|
|
e742cd5192 | ||
|
|
9d11a22901 | ||
|
|
aaf158aa59 | ||
|
|
e39d2884fe |
File diff suppressed because it is too large
Load Diff
@@ -1,377 +0,0 @@
|
||||
---
|
||||
title: CODEX-COMMUNITY-HISTORY
|
||||
name: Codex Community History
|
||||
status: raw
|
||||
tags: codex
|
||||
editor:
|
||||
contributors:
|
||||
- Jimmy Debe <jimmy@status.im>
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
This document describes how nodes in Status Communities archive historical message data of their communities.
|
||||
Not requiring to follow the time range limit provided by [13/WAKU2-STORE](../../waku/standards/core/13/store.md)
|
||||
nodes using the [BitTorrent protocol](https://www.bittorrent.org/beps/bep_0003.html).
|
||||
It also describes how the archives are distributed to community members via the [Status network](https://status.network/),
|
||||
so they can fetch them and
|
||||
gain access to a complete message history.
|
||||
|
||||
## Background
|
||||
|
||||
Messages are stored permanently by [13/WAKU2-STORE](../../waku/standards/core/13/store.md) nodes for a configurable time range,
|
||||
which is limited by the overall storage provided by a [13/WAKU2-STORE](../../waku/standards/core/13/store.md) nodes.
|
||||
Messages older than that period are no longer provided by [13/WAKU2-STORE](../../waku/standards/core/13/store.md) nodes,
|
||||
making it impossible for other nodes to request historical messages that go beyond that time range.
|
||||
This raises issues in the case of Status communities,
|
||||
where recently joined members of a community are not able to request complete message histories of the community channels.
|
||||
|
||||
### Terminology
|
||||
|
||||
| Name | Description |
|
||||
| ---- | -------------- |
|
||||
| Waku node | A [10/WAKU2](../../waku/standards/core/10/waku.md) node that implements [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md) |
|
||||
| Store node | A [10/WAKU2](../../waku/standards/core/10/waku.md) node that implements [13/WAKU2-STORE](../../waku/standards/core/13/store.md) |
|
||||
| Waku network | A group of [10/WAKU2](../../waku/standards/core/10/waku.md) nodes forming a graph, connected via [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md) |
|
||||
| Status user | A Status account that is used in a Status consumer product, such as Status Mobile or Status Desktop |
|
||||
| Status node | A Status client run by a Status application |
|
||||
| Control node| A Status node that owns the private key for a Status community |
|
||||
| Community member | A Status user that is part of a Status community, not owning the private key of the community|
|
||||
| Community member node | A Status node with message archive capabilities enabled, run by a community member |
|
||||
| Live messages | [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) received through the Waku network |
|
||||
| BitTorrent client | A program implementing the BitTorrent protocol |
|
||||
| Torrent/Torrent file | A file containing metadata about data to be downloaded by BitTorrent clients |
|
||||
| Magnet link | A link encoding the metadata provided by a torrent file (Magnet URI scheme) |
|
||||
|
||||
## Specification
|
||||
|
||||
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”,
|
||||
“SHOULD NOT”, “RECOMMENDED”, “MAY”, and
|
||||
“OPTIONAL” in this document are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
|
||||
|
||||
### Message History Archive
|
||||
|
||||
Message history archives are represented as `WakuMessageArchive` and
|
||||
created from a [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) exported from the local database.
|
||||
The following describes the protocol buffer for `WakuMessageArchive` :
|
||||
|
||||
``` protobuf
|
||||
|
||||
syntax = "proto3";
|
||||
|
||||
message WakuMessageArchiveMetadata {
|
||||
uint8 version = 1;
|
||||
uint64 from = 2;
|
||||
uint64 to = 3;
|
||||
repeated string content_Topic = 4;
|
||||
}
|
||||
|
||||
message WakuMessageArchive {
|
||||
uint8 version = 1;
|
||||
WakuMessageArchiveMetadata metadata = 2;
|
||||
repeated WakuMessage messages = 3; // `WakuMessage` is provided by 14/WAKU2-MESSAGE
|
||||
bytes padding = 4;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
The `from` field SHOULD contain a `timestamp` of the time range's lower bound.
|
||||
This type parallels to the `timestamp` of a `WakuMessage`.
|
||||
The `to` field SHOULD contain a `timestamp` of the time range's the higher bound.
|
||||
The `contentTopic` field MUST contain a list of all community channel `contentTopic`s.
|
||||
The `messages` field MUST contain all messages that belong in the archive, given its `from`,
|
||||
`to`, and `contentTopic` fields.
|
||||
|
||||
The `padding` field MUST contain the amount of zero bytes needed for the protobuf encoded `WakuMessageArchive`.
|
||||
The overall byte size MUST be a multiple of the `pieceLength` used to divide the data into pieces.
|
||||
This is needed for seamless encoding and
|
||||
decoding of archival data when interacting with BitTorrent,
|
||||
as explained in [creating message archive torrents](#creating-message-archive-torrents).
|
||||
|
||||
#### Message History Archive Index
|
||||
|
||||
Control nodes MUST provide message archives for the entire community history.
|
||||
The entire history consists of a set of `WakuMessageArchive`,
|
||||
where each archive contains a subset of historical `WakuMessage` for a time range of seven days.
|
||||
All the `WakuMessageArchive` are concatenated into a single file as a byte string, see [Ensuring reproducible data pieces](#ensuring-reproducible-data-pieces).
|
||||
|
||||
Control nodes MUST create a message history archive index,
|
||||
`WakuMessageArchiveIndex` with metadata,
|
||||
that allows receiving nodes to only fetch the message history archives they are interested in.
|
||||
|
||||
##### WakuMessageArchiveIndex
|
||||
|
||||
``` protobuf
|
||||
|
||||
syntax = "proto3"
|
||||
|
||||
message WakuMessageArchiveIndexMetadata {
|
||||
uint8 version = 1
|
||||
WakuMessageArchiveMetadata metadata = 2
|
||||
uint64 offset = 3
|
||||
uint64 num_pieces = 4
|
||||
}
|
||||
|
||||
message WakuMessageArchiveIndex {
|
||||
map<string, WakuMessageArchiveIndexMetadata> archives = 1
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
A `WakuMessageArchiveIndex` is a map where the key is the KECCAK-256 hash of the `WakuMessageArchiveIndexMetadata`,
|
||||
is derived from a 7-day archive
|
||||
and the value is an instance of that `WakuMessageArchiveIndexMetadata` corresponding to that archive.
|
||||
|
||||
The `offset` field MUST contain the position at which the message history archive starts in the byte string
|
||||
of the total message archive data.
|
||||
This MUST be the sum of the length of all previously created message archives in bytes, see [creating message archive torrents](#creating-message-archive-torrents).
|
||||
|
||||
The control node MUST update the `WakuMessageArchiveIndex` every time it creates one or
|
||||
more `WakuMessageArchive`s and bundle it into a new torrent.
|
||||
For every created `WakuMessageArchive`,
|
||||
there MUST be a `WakuMessageArchiveIndexMetadata` entry in the archives field `WakuMessageArchiveIndex`.
|
||||
|
||||
### Creating Message Archive Torrents
|
||||
|
||||
Control nodes MUST create a .torrent file containing metadata for all message history archives.
|
||||
To create a .torrent file, and
|
||||
later serve the message archive data on the BitTorrent network,
|
||||
control nodes MUST store the necessary data in dedicated files on the file system.
|
||||
|
||||
A torrent's source folder MUST contain the following two files:
|
||||
|
||||
- `data`: Contains all protobuf encoded `WakuMessageArchive`'s (as bit strings)
|
||||
concatenated in ascending order based on their time
|
||||
- `index`: Contains the protobuf encoded `WakuMessageArchiveIndex`
|
||||
|
||||
Control nodes SHOULD store these files in a dedicated folder that is identifiable via a community identifier.
|
||||
|
||||
### Ensuring Reproducible Data Pieces
|
||||
|
||||
The control node MUST ensure that the byte string from the protobuf encoded data
|
||||
is equal to the byte string data from the previously generated message archive torrent.
|
||||
Including the data of the latest seven days worth of messages encoded as `WakuMessageArchive`.
|
||||
Therefore, the size of data grows every seven days as it's append-only.
|
||||
|
||||
Control nodes MUST ensure that the byte size,
|
||||
for every individual `WakuMessageArchive` encoded protobuf,
|
||||
is a multiple of `pieceLength` using the padding field.
|
||||
If the `WakuMessageArchive` is not a multiple of `pieceLength`,
|
||||
its padding field MUST be filled with zero bytes and
|
||||
the `WakuMessageArchive` MUST be re-encoded until its size becomes a multiple of `pieceLength`.
|
||||
|
||||
This is necessary because the content of the data file will be split into pieces of `pieceLength` when the torrent file is created,
|
||||
and the SHA1 hash of every piece is then stored in the torrent file and
|
||||
later used by other nodes to request the data for each individual data piece.
|
||||
|
||||
By fitting message archives into a multiple of `pieceLength` and
|
||||
ensuring they fill the possible remaining space with zero bytes,
|
||||
control nodes prevent the next message archive from occupying that remaining space of the last piece,
|
||||
which will result in a different SHA1 hash for that piece.
|
||||
|
||||
Example: Without padding
|
||||
Let `WakuMessageArchive` "A1" be of size 20 bytes:
|
||||
|
||||
``` text
|
||||
0 11 22 33 44 55 66 77 88 99
|
||||
10 11 12 13 14 15 16 17 18 19
|
||||
```
|
||||
|
||||
With a `pieceLength` of 10 bytes, A1 will fit into 20 / 10 = 2 pieces:
|
||||
|
||||
```text
|
||||
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
|
||||
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
|
||||
```
|
||||
|
||||
Example: With padding
|
||||
Let `WakuMessageArchive` "A2" be of size 21 bytes:
|
||||
|
||||
```text
|
||||
0 11 22 33 44 55 66 77 88 99
|
||||
10 11 12 13 14 15 16 17 18 19
|
||||
20
|
||||
```
|
||||
|
||||
With a `pieceLength` of 10 bytes,
|
||||
A2 will fit into 21 / 10 = 2 pieces.
|
||||
|
||||
The remainder will introduce a third piece:
|
||||
|
||||
```text
|
||||
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
|
||||
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
|
||||
20 // piece[2] SHA1: 0x789
|
||||
```
|
||||
|
||||
The next `WakuMessageArchive` "A3" will be appended ("#3") to the existing data and
|
||||
occupy the remaining space of the third data piece.
|
||||
|
||||
The piece at index 2 will now produce a different SHA1 hash:
|
||||
|
||||
```text
|
||||
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
|
||||
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
|
||||
20 #3 #3 #3 #3 #3 #3 #3 #3 #3 // piece[2] SHA1: 0xeef
|
||||
#3 #3 #3 #3 #3 #3 #3 #3 #3 #3 // piece[3]
|
||||
```
|
||||
|
||||
By filling up the remaining space of the third piece with A2 using its padding field,
|
||||
it is guaranteed that its SHA1 will stay the same:
|
||||
|
||||
```text
|
||||
0 11 22 33 44 55 66 77 88 99 // piece[0] SHA1: 0x123
|
||||
10 11 12 13 14 15 16 17 18 19 // piece[1] SHA1: 0x456
|
||||
20 0 0 0 0 0 0 0 0 0 // piece[2] SHA1: 0x999
|
||||
#3 #3 #3 #3 #3 #3 #3 #3 #3 #3 // piece[3]
|
||||
#3 #3 #3 #3 #3 #3 #3 #3 #3 #3 // piece[4]
|
||||
```
|
||||
|
||||
### Seeding Message History Archives
|
||||
|
||||
The control node MUST seed the generated torrent until a new `WakuMessageArchive` is created.
|
||||
|
||||
The control node SHOULD NOT seed torrents for older message history archives.
|
||||
Only one torrent at a time SHOULD be seeded.
|
||||
|
||||
### Creating Magnet Links
|
||||
|
||||
Once a torrent file for all message archives is created,
|
||||
the control node MUST derive a magnet link,
|
||||
following the Magnet URI scheme using the underlying [BitTorrent protocol](https://www.bittorrent.org/beps/bep_0003.html) client.
|
||||
|
||||
#### Message Archive Distribution
|
||||
|
||||
Message archives are available via the BitTorrent network as they are being seeded by the control node.
|
||||
Other community member nodes will download the message archives, from the BitTorrent network,
|
||||
after receiving a magnet link that contains a message archive index.
|
||||
|
||||
The control node MUST send magnet links containing message archives and
|
||||
the message archive index to a special community channel.
|
||||
The `content_Topic` of that special channel follows the following format:
|
||||
|
||||
``` text
|
||||
|
||||
/{application-name}/{version-of-the-application}/{content-topic-name}/{encoding}
|
||||
```
|
||||
|
||||
All messages sent with this special channel's `content_Topic` MUST be instances of `ApplicationMetadataMessage`,
|
||||
with a [62/STATUS-PAYLOADS](../../status/62/payloads.md) of `CommunityMessageArchiveIndex`.
|
||||
|
||||
Only the control node MAY post to the special channel.
|
||||
Other messages on this specified channel MUST be ignored by clients.
|
||||
Community members MUST NOT have permission to send messages to the special channel.
|
||||
However, community member nodes MUST subscribe to a special channel,
|
||||
to receive a [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) containing magnet links for message archives.
|
||||
|
||||
#### Canonical Message Histories
|
||||
|
||||
Only control nodes are allowed to distribute messages with magnet links,
|
||||
via the special channel for magnet link exchange.
|
||||
Status nodes MUST ignore all messages in the special channel that aren't signed by a control node.
|
||||
Since the magnet links are created from the control node's database
|
||||
(and previously distributed archives),
|
||||
the message history provided by the control node becomes the canonical message history and
|
||||
single source of truth for the community.
|
||||
|
||||
Community member nodes MUST replace messages in their local database with the messages extracted from archives
|
||||
within the same time range.
|
||||
Messages that the control node didn't receive MUST be removed and
|
||||
are no longer part of the message history of interest,
|
||||
even if it already existed in a community member node's database.
|
||||
|
||||
### Fetching Message History Archives
|
||||
|
||||
The process of fetching message history:
|
||||
|
||||
1. Receive message archive index magnet link as described in [Message archive distribution](#message-archive-distribution),
|
||||
2. Download the index file from the torrent, then determine which message archives to download
|
||||
3. Download individual archives
|
||||
|
||||
Community member nodes subscribe to the special channel of the control nodes that publish magnet links for message history archives.
|
||||
Two RECOMMENDED scenarios in which community member nodes can receive such a magnet link message from the special channel:
|
||||
|
||||
1. The member node receives it via live messages, by listening to the special channel.
|
||||
2. The member node requests messages for a time range of up to 30 days from store nodes
|
||||
(this is the case when a new community member joins a community.)
|
||||
3. Downloading message archives
|
||||
|
||||
When community member nodes receive a message with a `CommunityMessageHistoryArchive` [62/STATUS-PAYLOADS](../../status/62/payloads.md),
|
||||
they MUST extract the `magnet_uri`.
|
||||
Then SHOULD pass it to their underlying BitTorrent client to fetch the latest message history archive index,
|
||||
which is the index file of the torrent, see [Creating message archive torrents].
|
||||
|
||||
Due to the nature of distributed systems,
|
||||
there's no guarantee that a received message is the "last" message.
|
||||
This is especially true when community member nodes request historical messages from store nodes.
|
||||
Therefore, community member nodes MUST wait for 20 seconds after receiving the last `CommunityMessageArchive`,
|
||||
before they start extracting the magnet link to fetch the latest archive index.
|
||||
|
||||
Once a message history archive index is downloaded and
|
||||
parsed back into `WakuMessageArchiveIndex`,
|
||||
community member nodes use a local lookup table to determine which of the listed archives are missing,
|
||||
using the KECCAK-256 hashes stored in the index.
|
||||
|
||||
For this lookup to work,
|
||||
member nodes MUST store the KECCAK-256 hashes,
|
||||
of the `WakuMessageArchiveIndexMetadata` provided by the index file,
|
||||
for all of the message history archives that have been downloaded into their local database.
|
||||
|
||||
Given a `WakuMessageArchiveIndex`, member nodes can access individual `WakuMessageArchiveIndexMetadata` to download individual archives.
|
||||
|
||||
Community member nodes MUST choose one of the following options:
|
||||
|
||||
1. Download all archives: Request and download all data pieces for the data provided by the torrent
|
||||
(this is the case for new community member nodes that haven't downloaded any archives yet.)
|
||||
2. Download only the latest archive: Request and
|
||||
download all pieces starting at the offset of the latest `WakuMessageArchiveIndexMetadata`
|
||||
(this is the case for any member node that already has downloaded all previous history and
|
||||
is now interested in only the latest archive).
|
||||
3. Download specific archives: Look into from and
|
||||
to fields of every `WakuMessageArchiveIndexMetadata` and
|
||||
determine the pieces for archives of a specific time range
|
||||
(can be the case for member nodes that have recently joined the network and
|
||||
are only interested in a subset of the complete history).
|
||||
|
||||
#### Storing Historical Messages
|
||||
|
||||
When message archives are fetched,
|
||||
community member nodes MUST unwrap the resulting `WakuMessage` instances into `ApplicationMetadataMessage` instances
|
||||
and store them in their local database.
|
||||
Community member nodes SHOULD NOT store the wrapped `WakuMessage` messages.
|
||||
|
||||
All messages within the same time range MUST be replaced with the messages provided by the message history archive.
|
||||
|
||||
Community members' nodes MUST ignore the expiration state of each archive message.
|
||||
|
||||
### Security Considerations
|
||||
|
||||
#### Multiple Community Owners
|
||||
|
||||
It is possible for control nodes to export the private key of their owned community and
|
||||
pass it to other users so they become control nodes as well.
|
||||
This means it's possible for multiple control nodes to exist for one community.
|
||||
|
||||
This might conflict with the assumption that the control node serves as a single source of truth.
|
||||
Multiple control nodes can have different message histories.
|
||||
Not only will multiple control nodes multiply the amount of archive index messages being distributed to the network,
|
||||
but they might also contain different sets of magnet links and their corresponding hashes.
|
||||
Even if just a single message is missing in one of the histories,
|
||||
the hashes presented in the archive indices will look completely different,
|
||||
resulting in the community member node downloading the corresponding archive.
|
||||
This might be identical to an archive that was already downloaded,
|
||||
except for that one message.
|
||||
|
||||
## Copyright
|
||||
|
||||
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
|
||||
|
||||
## References
|
||||
|
||||
- [13/WAKU2-STORE](../../waku/standards/core/13/store.md)
|
||||
- [BitTorrent protocol](https://www.bittorrent.org/beps/bep_0003.html)
|
||||
- [Status network](https://status.network/)
|
||||
- [10/WAKU2](../../waku/standards/core/10/waku.md)
|
||||
- [11/WAKU2-RELAY](../../waku/standards/core/11/relay.md)
|
||||
- [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md)
|
||||
- [62/STATUS-PAYLOADS](../../status/62/payloads.md)
|
||||
@@ -54,11 +54,15 @@ are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.tx
|
||||
|
||||
The three roles used in de-MLS is as follows:
|
||||
|
||||
- `node`: Nodes are members of network without being in any secure group messaging.
|
||||
- `node`: Nodes are participants in the network that are not currently members
|
||||
of any secure group messaging session but remain available as potential candidates for group membership.
|
||||
- `member`: Members are special nodes in the secure group messaging who
|
||||
obtains current group key of secure group messaging.
|
||||
- `steward`: Stewards are special and transparent members in secure group
|
||||
messaging who organizes the changes upon the voted-proposals.
|
||||
Each node is assigned a unique identity represented as a 20-byte value named `member id`.
|
||||
- `steward`: Stewards are special and transparent members in the secure group
|
||||
messaging who organize the changes by releasing commit messages upon the voted proposals.
|
||||
There are two special subsets of steward as epoch and backup steward,
|
||||
which are defined in the section de-MLS Objects.
|
||||
|
||||
## MLS Background
|
||||
|
||||
@@ -78,7 +82,7 @@ manage the `keyPackage` of the users where the `keyPackage` is the objects
|
||||
|
||||
Following section presents the MLS objects and components that used in this RFC:
|
||||
|
||||
`Epoch`: Fixed time intervals that changes the state that is defined by members,
|
||||
`Epoch`: Time intervals that changes the state that is defined by members,
|
||||
section 3.4 in [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/).
|
||||
|
||||
`MLS proposal message:` Members MUST receive the proposal message prior to the
|
||||
@@ -90,6 +94,7 @@ Here, the add and remove proposals are used.
|
||||
This is restricted by [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/) as if there is pending proposal,
|
||||
the application message should be cut.
|
||||
Note that: Since the MLS is based on servers, this delay between proposal and commit messages are very small.
|
||||
|
||||
`Commit message:` After members receive the proposals regarding group changes,
|
||||
the committer, who may be any member of the group, as specified in [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/),
|
||||
generates the necessary key material for the next epoch, including the appropriate welcome messages
|
||||
@@ -97,21 +102,48 @@ for new joiners and new entropy for removed members. In this RFC, the committers
|
||||
|
||||
### de-MLS Objects
|
||||
|
||||
`Voting proposal` Similar to MLS proposals, but processed only if approved through a voting process.
|
||||
This section presents the de-MLS objects:
|
||||
|
||||
`Voting proposal`: Similar to MLS proposals, but processed only if approved through a voting process.
|
||||
They function as application messages in the MLS group,
|
||||
allowing the steward to collect them without halting the protocol.
|
||||
There are three types of `voting proposal` according to the type of consensus as in shown Consensus Types section,
|
||||
these are, `commit proposal`, `steward election proposal` and `emergency criteria proposal`.
|
||||
|
||||
`Epoch steward`: The steward assigned to commit in `epoch E` according to the steward list.
|
||||
Holds the primary responsibility for creating commit in that epoch.
|
||||
|
||||
`Backup steward`: The steward next in line after the `epoch steward` on the `steward list` in `epoch E`.
|
||||
Only becomes active if the `epoch steward` is malicious or fails,
|
||||
in which case it completes the commitment phase.
|
||||
If unused in `epoch E`, it automatically becomes the `epoch steward` in `epoch E+1`.
|
||||
|
||||
`Steward list`: It is an ordered list that contains the `member id`s of authorized stewards.
|
||||
Each steward in the list becomes main responsible for creating the commit message when its turn arrives,
|
||||
according to this order for each epoch.
|
||||
For example, suppose there are two stewards in the list `steward A` first and `steward B` last in the list.
|
||||
`steward A` is responsible for creating the commit message for first epoch.
|
||||
Similarly, `steward B` is for the last epoch.
|
||||
Since the `epoch steward` is the primary committer for an epoch,
|
||||
it holds the main responsibility for producing the commit.
|
||||
However, other stewards MAY also generate a commit within the same epoch to preserve liveness
|
||||
in case the epoch steward is inactive or slow.
|
||||
Duplicate commits are not re-applied and only the single valid commit for the epoch is accepted by the group,
|
||||
as in described in section filtering proposals against the multiple comitting.
|
||||
|
||||
Therefore, if a malicious steward occurred, the `backup steward` will be charged with committing.
|
||||
Lastly, the size of the list named as `sn`, which also shows the epoch interval for steward list determination.
|
||||
|
||||
## Flow
|
||||
|
||||
General flow is as follows:
|
||||
|
||||
- A steward initializes a group just once, and then sends out Group Announcements (GA) periodically.
|
||||
|
||||
- Meanwhile, each`node`creates and sends their`credential` includes `keyPackage`.
|
||||
- Each `member`creates `voting proposals` sends them to from MLS group during epoch E.
|
||||
- Meanwhile, each `node` creates and sends their `credential` includes `keyPackage`.
|
||||
- Each `member` creates `voting proposals` sends them to from MLS group during `epoch E`.
|
||||
- Meanwhile, the `steward` collects finalized `voting proposals` from MLS group and converts them into
|
||||
`MLS proposals` then sends them with correspondng `commit messages`
|
||||
- Evantually, with the commit messages, all members starts the next epoch E+1.
|
||||
`MLS proposals` then sends them with corresponding `commit messages`
|
||||
- Evantually, with the commit messages, all members starts the next `epoch E+1`.
|
||||
|
||||
## Creating Voting Proposal
|
||||
|
||||
@@ -179,57 +211,220 @@ This is mostly similar with the general flow and specified in voting proposal an
|
||||
as in section 8.1. Group Context in [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/).
|
||||
2. `steward` creates a group anouncement (GA) according to the previous step and
|
||||
broadcast it to the all network periodically. GA message is visible in network to all `nodes`.
|
||||
3. The each `node` who wants to be a member needs to obtain this anouncement and create `credential`
|
||||
3. The each `node` who wants to be a `member` needs to obtain this anouncement and create `credential`
|
||||
includes `keyPackage` that is specified in [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/) section 10.
|
||||
4. The `steward` aggregates all `KeyPackages` utilizes them to provision group additions for new members,
|
||||
4. The `node` send the `KeyPackages` in plaintext with its signature with current `steward` public key which
|
||||
anounced in welcome topic. This step is crucial for security, ensuring that malicious nodes/stewards
|
||||
cannot use others' `KeyPackages`.
|
||||
It also provides flexibility for liveness in multi-steward settings,
|
||||
allowing more than one steward to obtain `KeyPackages` to commit.
|
||||
5. The `steward` aggregates all `KeyPackages` utilizes them to provision group additions for new members,
|
||||
based on the outcome of the voting process.
|
||||
5. Any `member` start to create `voting proposals` for adding or removing users,
|
||||
6. Any `member` start to create `voting proposals` for adding or removing users,
|
||||
and present them to the voting in the MLS group as an application message.
|
||||
|
||||
However, unlimited use of `voting proposals` within the group may be misused by
|
||||
malicious or overly active members.
|
||||
Therefore, an application-level constraint can be introduced to limit the number
|
||||
or frequency of proposals initiated by each member to prevent spam or abuse.
|
||||
6. Meanwhile, the `steward` collects finalized `voting proposals` with in epoch `E`,
|
||||
7. Meanwhile, the `steward` collects finalized `voting proposals` with in epoch `E`,
|
||||
that have received affirmative votes from members via application messages.
|
||||
Otherwise, the `steward` discards proposals that did not receive a majority of "YES" votes.
|
||||
Since voting proposals are transmitted as application messages, omitting them does not affect
|
||||
the protocol’s correctness or consistency.
|
||||
7. The `steward` converts all approved `voting proposals` into
|
||||
8. The `steward` converts all approved `voting proposals` into
|
||||
corresponding `MLS proposals` and `commit message`, and
|
||||
transmits both in a single operation as in [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/) section 12.4,
|
||||
including welcome messages for the new members. Therefore, the `commit message` ends the previous epoch and create new ones.
|
||||
including welcome messages for the new members.
|
||||
Therefore, the `commit message` ends the previous epoch and create new ones.
|
||||
9. The `members` applied the incoming `commit message` by checking the signatures and `voting proposals`
|
||||
and synchronized with the upcoming epoch.
|
||||
|
||||
## Multi stewards
|
||||
|
||||
Decentralization has already been achieved in the previous section.
|
||||
However, to improve availability and ensure censorship resistance,
|
||||
the single-steward protocol is extended to a multi-steward architecture.
|
||||
the single steward protocol is extended to a multi steward architecture.
|
||||
In this design, each epoch is coordinated by a designated steward,
|
||||
operating under the same protocol as the single-steward model.
|
||||
Thus, the multi-steward approach primarily defines how steward roles
|
||||
operating under the same protocol as the single steward model.
|
||||
Thus, the multi steward approach primarily defines how steward roles
|
||||
rotate across epochs while preserving the underlying structure and logic of the original protocol.
|
||||
Two variants of the multi-steward design are introduced to address different system requirements.
|
||||
Two variants of the multi steward design are introduced to address different system requirements.
|
||||
|
||||
### Multi steward with single consensus
|
||||
### Consensus Types
|
||||
|
||||
Consensus is agnostic with its payload; therefore, it can be used for various purposes.
|
||||
Note that each message for the consensus of proposals is an `application message` in the MLS object section.
|
||||
It is used in three ways as follows:
|
||||
|
||||
1. `Commit proposal`: It is the proposal instance that is specified in Creating Voting Proposal section
|
||||
with `Proposal.payload` MUST show the commit request from `members`.
|
||||
Any member MAY create this proposal in any epoch and `epoch steward` MUST collect and commit YES voted proposals.
|
||||
This is the only proposal type common to both single steward and multi steward designs.
|
||||
2. `Steward election proposal`: This is the process that finalizes the `steward list`,
|
||||
which sets and orders stewards responsible for creating commits over a predefined number of range in (`sn_min`,`sn_max`).
|
||||
The validity of the choosen `steward list` ends when the last steward in the list (the one at the final index) completes its commit.
|
||||
At that point, a new `steward election proposal` MUST be initiated again by any member during the corresponding epoch.
|
||||
The `Proposal.payload` field MUST represent the ordered identities of the proposed stewards.
|
||||
Each steward election proposal MUST be verified and finalized through the consensus process
|
||||
so that members can identify which steward will be responsible in each epoch
|
||||
and detect any unauthorized steward commits.
|
||||
3. `Emergency criteria proposal`: If there is a malicious member or steward,
|
||||
this event MUST be voted on to finalize it.
|
||||
If this returns YES, the next epoch MUST include the removal of the member or steward.
|
||||
In a specific case where a steward is removed from the group, causing the total number of stewards to fall below `sn_min`,
|
||||
it is required to repeat the `steward election proposal`.
|
||||
`Proposal.payload` MUST consist of the evidence of the dishonesty as described in the Steward violation list,
|
||||
and the identifier of the malicious member or steward.
|
||||
This proposal can be created by any member in any epoch.
|
||||
|
||||
The order of consensus proposal messages is important to achieving a consistent result.
|
||||
Therefore, messages MUST be prioritized by type in the following order, from highest to lowest priority:
|
||||
|
||||
- `Emergency criteria proposal`
|
||||
|
||||
- `Steward election proposal`
|
||||
|
||||
- `Commit proposal`
|
||||
|
||||
This means that if a higher-priority consensus proposal is present in the network,
|
||||
lower-priority messages MUST be withheld from transmission until the higher-priority proposals have been finalized.
|
||||
|
||||
### Steward list creation
|
||||
|
||||
The `steward list` consists of steward nominees who will become actual stewards if the `steward election proposal` is finalized with YES,
|
||||
is arbitrarily chosen from `member` and OPTIONALLY adjusted depending on the needs of the implementation.
|
||||
The `steward list` size, defined by the minimum `sn_min` and maximum `sn_max` bounds,
|
||||
is determined at the time of group creation.
|
||||
The `sn_min` requirement is applied only when the total number of members exceeds `sn_min`;
|
||||
if the number of available members falls below this threshold,
|
||||
the list size automatically adjusts to include all existing members.
|
||||
|
||||
The actual size of the list MAY vary within this range as `sn`, with the minimum value being at least 1.
|
||||
|
||||
The index of the slots shows epoch info and value of index shows `member id`s.
|
||||
The next in line steward for the `epoch E` is named as `epoch steward`, which has index E.
|
||||
And the subsequent steward in the `epoch E` is named as the `backup steward`.
|
||||
For example, let's assume steward list is (S3, S2, S1) if in the previous epoch the roles were
|
||||
(`backup steward`: S2, `epoch steward`: S1), then in the next epoch they become
|
||||
(`backup steward`: S3, `epoch steward`: S2) by shifting.
|
||||
|
||||
If the `epoch steward` is honest, the `backup steward` does not involve the process in epoch,
|
||||
and the `backup steward` will be the `epoch steward` within the `epoch E+1`.
|
||||
|
||||
If the `epoch steward` is malicious, the `backup steward` is involved in the commitment phase in `epoch E`
|
||||
and the former steward becomes the `backup steward` in `epoch E`.
|
||||
|
||||
Liveness criteria:
|
||||
|
||||
Once the active `steward list` has completed its assigned epochs,
|
||||
|
||||
members MUST proceed to elect the next set of stewards
|
||||
(which MAY include some or all of the previous members).
|
||||
This election is conducted through a type 2 consensus procedure, `steward election proposal`.
|
||||
|
||||
A `Steward election proposal` is considered valid only if the resulting `steward list`
|
||||
is produced through a deterministic process that ensures an unbiased distribution of steward assignments,
|
||||
since allowing bias could enable a malicious participant to manipulate the list
|
||||
and retain control within a favored group for multiple epochs.
|
||||
|
||||
The list MUST consist of at least `sn_min` members, including retained previous stewards,
|
||||
sorted according to the ascending value of `SHA256(epoch E || member id || group id)`,
|
||||
where `epoch E` is the epoch in which the election proposal is initiated,
|
||||
and `group id` for shuffling the list across the different groups.
|
||||
Any proposal with a list that does not adhere to this generation method MUST be rejected by all members.
|
||||
|
||||
We assume that there are no recurring entries in `SHA256(epoch E || member id || group id)`, since the SHA256 outputs are unique
|
||||
when there is no repetition in the `member id` values, against the conflicts on sorting issues.
|
||||
|
||||
### Multi steward with big consensuses
|
||||
|
||||
In this model, all group modifications, such as adding or removing members,
|
||||
must be approved through consensus by all participants,
|
||||
including the steward assigned for epoch `E`.
|
||||
including the steward assigned for `epoch E`.
|
||||
A configuration with multiple stewards operating under a shared consensus protocol offers
|
||||
increased decentralization and stronger protection against censorship.
|
||||
However, this benefit comes with reduced operational efficiency.
|
||||
The model is therefore best suited for small groups that value
|
||||
decentralization and censorship resistance more than performance.
|
||||
|
||||
### Multi steward with two consensuses
|
||||
To create a multi steward with a big consensus,
|
||||
the group is initialized with a single steward as specified as follows:
|
||||
|
||||
The two-consensus model offers improved efficiency with a trade-off in decentralization.
|
||||
1. The steward initialized the group with the config file.
|
||||
This config file MUST contain (`sn_min`,`sn_max`) as the `steward list` size range.
|
||||
2. The steward adds the members as a centralized way till the number of members reaches the `sn_min`.
|
||||
Then, members propose lists by voting proposal with size `sn`
|
||||
as a consensus among all members, as mentioned in the consensus section 2, according to the checks:
|
||||
the size of the proposed list `sn` is in the interval (`sn_min`,`sn_max`).
|
||||
Note that if the total number of members is below `sn_min`,
|
||||
then the steward list size MUST be equal to the total member count.
|
||||
3. After the voting proposal ends up with a `steward list`,
|
||||
and group changes are ready to be committed as specified in single steward section
|
||||
with a difference which is members also check the committed steward is `epoch steward` or `backup steward`,
|
||||
otherwise anyone can create `emergency criteria proposal`.
|
||||
4. If the `epoch steward` violates the changing process as mentioned in the section Steward violation list,
|
||||
one of the members MUST initialize the `emergency criteria proposal` to remove the malicious Steward.
|
||||
Then `backup steward` fulfills the epoch by committing again correctly.
|
||||
|
||||
A large consensus group provides better decentralization, but it requires significant coordination,
|
||||
which MAY not be suitable for groups with more than 1000 members.
|
||||
|
||||
### Multi steward with small consensuses
|
||||
|
||||
The small consensus model offers improved efficiency with a trade-off in decentralization.
|
||||
In this design, group changes require consensus only among the stewards, rather than all members.
|
||||
Regular members participate by periodically selecting the stewards but do not take part in each decision.
|
||||
Regular members participate by periodically selecting the stewards by `steward election proposal`
|
||||
but do not take part in commit decision by `commit proposal`.
|
||||
This structure enables faster coordination since consensus is achieved within a smaller group of stewards.
|
||||
It is particularly suitable for large user groups, where involving every member in each decision would be impractical.
|
||||
|
||||
The flow is similar to the big consensus including the `steward list` finalization with all members consensus
|
||||
only the difference here, the commit messages requires `commit proposal` only among the stewards.
|
||||
|
||||
## Filtering proposals against the multiple comitting
|
||||
|
||||
Since stewards are allowed to produce a commit even when they are not the designated `epoch steward`,
|
||||
multiple commits may appear within the same epoch, often reflecting recurring versions of the same proposal.
|
||||
To ensure a consistent outcome, the valid commit for the epoch SHOULD be selected as the one derived
|
||||
from the longest proposal chain, ordered by the ascending value of each proposal as `SHA256(proposal)`.
|
||||
All other cases, such as invalid commits or commits based on proposals that were not approved through voting,
|
||||
can be easily detected and discarded by the members.
|
||||
|
||||
## Steward violation list
|
||||
|
||||
A steward’s activity is called a violation if the action is one or more of the following:
|
||||
|
||||
1. Broken commit: The steward releases a different commit message from the voted `commit proposal`.
|
||||
This activity is identified by the `members` since the [MLS RFC 9420](https://datatracker.ietf.org/doc/rfc9420/) provides the methods
|
||||
that members can use to identify the broken commit messages that are possible in a few situations,
|
||||
such as commit and proposal incompatibility. Specifically, the broken commit can arise as follows:
|
||||
1. The commit belongs to the earlier epoch.
|
||||
2. The commit message should equal the latest epoch
|
||||
3. The commit needs to be compatible with the previous epoch’s `MLS proposal`.
|
||||
2. Broken MLS proposal: The steward prepares a different `MLS proposal` for the corresponding `voting proposal`.
|
||||
This activity is identified by the `members` since both `MLS proposal` and `voting proposal` are visible
|
||||
and can be identified by checking the hash of `Proposal.payload` and `MLSProposal.payload` is the same as RFC9240 section 12.1. Proposals.
|
||||
3. Censorship and inactivity: The situation where there is a voting proposal that is visible for every member,
|
||||
and the Steward does not provide an MLS proposal and commit.
|
||||
This activity is again identified by the `members`since `voting proposals` are visible to every member in the group,
|
||||
therefore each member can verify that there is no `MLS proposal` corresponding to `voting proposal`.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
In this section, the security considerations are shown as de-MLS assurance.
|
||||
|
||||
1. Malicious Steward: A Malicious steward can act maliciously,
|
||||
as in the Steward violation list section.
|
||||
Therefore, de-MLS enforces that any steward only follows the protocol under the consensus order
|
||||
and commits without emergency criteria application.
|
||||
2. Malicious Member: A member is only marked as malicious
|
||||
when the member acts by releasing a commit message.
|
||||
3. Steward list election bias: Although SHA256 is used together with two global variables
|
||||
to shuffle stewards in a deterministic and verifiable manner,
|
||||
this approach only minimizes election bias; it does not completely eliminate it.
|
||||
This design choice is intentional, in order to preserve the efficiency advantages provided by the MLS mechanism.
|
||||
|
||||
## Copyright
|
||||
|
||||
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/)
|
||||
|
||||
1268
vac/raw/logos-capability-discovery.md
Normal file
1268
vac/raw/logos-capability-discovery.md
Normal file
File diff suppressed because it is too large
Load Diff
1705
vac/raw/mix.md
1705
vac/raw/mix.md
File diff suppressed because it is too large
Load Diff
429
vac/raw/zerokit-api.md
Normal file
429
vac/raw/zerokit-api.md
Normal file
@@ -0,0 +1,429 @@
|
||||
---
|
||||
title: Zerokit-API
|
||||
name: Zerokit API
|
||||
status: raw
|
||||
category: Standards Track
|
||||
tags: [zerokit, rln, api]
|
||||
editor: Vac Team
|
||||
contributors:
|
||||
- Vinh Trinh <vinh@status.im>
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
This document specifies the Zerokit API, an implementation of the RLN-V2 protocol.
|
||||
The specification covers the unified interface exposed through native Rust,
|
||||
C-compatible Foreign Function Interface (FFI) bindings,
|
||||
and WebAssembly (WASM) bindings.
|
||||
|
||||
## Motivation
|
||||
|
||||
The main goal of this RFC is to define the API contract,
|
||||
serialization formats,
|
||||
and architectural guidance for integrating the Zerokit library
|
||||
across all supported platforms.
|
||||
Zerokit is the reference implementation of the RLN-V2 protocol.
|
||||
|
||||
## Format Specification
|
||||
|
||||
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
|
||||
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document
|
||||
are to be interpreted as described in [2119](https://www.ietf.org/rfc/rfc2119.txt).
|
||||
|
||||
### Important Note
|
||||
|
||||
All terms and parameters used remain the same as in [RLN-V2](./rln-v2.md) and [RLN-V1](../32/rln-v1.md#technical-overview).
|
||||
|
||||
### Architecture Overview
|
||||
|
||||
Zerokit follows a layered architecture where
|
||||
the core RLN logic is implemented once in Rust and
|
||||
exposed through platform-specific bindings.
|
||||
The protocol layer handles zero-knowledge proof generation and verification,
|
||||
Merkle tree operations, and cryptographic primitives.
|
||||
This core is wrapped by three interface layers:
|
||||
native Rust for direct library integration,
|
||||
FFI for C-compatible bindings consumed by languages (such as C and Nim),
|
||||
and WASM for browser and Node.js environments.
|
||||
All three interfaces maintain functional parity and
|
||||
share identical serialization formats for inputs and outputs.
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Application Layer │
|
||||
└──────────┬───────────────┬───────────────┬──────────┘
|
||||
│ │ │
|
||||
┌──────▼───────┐ ┌─────▼─────┐ ┌───────▼─────┐
|
||||
│ FFI API │ │ WASM API │ │ Rust API │
|
||||
│ (C/Nim/..) │ │ (Browser) │ │ (Native) │
|
||||
└──────┬───────┘ └─────┬─────┘ └───────┬─────┘
|
||||
└───────────────┼───────────────┘
|
||||
│
|
||||
┌─────────▼─────────┐
|
||||
│ RLN Protocol │
|
||||
│ (Rust Core) │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
### Supported Features
|
||||
|
||||
Zerokit provides compile-time feature flags that
|
||||
control Merkle tree storage backends,
|
||||
operational modes,
|
||||
and parallelization.
|
||||
|
||||
#### Merkle Tree Backends
|
||||
|
||||
`fullmerkletree` allocates the complete tree structure in memory.
|
||||
This backend provides the fastest performance but consumes the most memory.
|
||||
|
||||
`optimalmerkletree` uses sparse HashMap storage that only allocates nodes as needed.
|
||||
This backend balances performance and memory efficiency.
|
||||
|
||||
`pmtree` persists the tree to disk using a sled database.
|
||||
This backend enables state durability across process restarts.
|
||||
|
||||
#### Operational Modes
|
||||
|
||||
`stateless` disables the internal Merkle tree.
|
||||
Applications MUST provide the Merkle root and
|
||||
membership proof externally when generating proofs.
|
||||
|
||||
When `stateless` is not enabled,
|
||||
the library operates in stateful mode and
|
||||
requires one of the Merkle tree backends.
|
||||
|
||||
#### Parallelization
|
||||
|
||||
`parallel` enables rayon-based parallel computation for
|
||||
proof generation and tree operations.
|
||||
|
||||
This flag SHOULD be enabled for end-user clients where
|
||||
fastest individual proof generation time is required.
|
||||
For server-side proof services handling multiple concurrent requests,
|
||||
this flag SHOULD be disabled and
|
||||
applications SHOULD use dedicated worker threads per proof instead.
|
||||
The worker thread approach provides significantly higher throughput for
|
||||
concurrent proof generation.
|
||||
|
||||
## The API
|
||||
|
||||
### Overview
|
||||
|
||||
The API exposes functional interfaces with strongly-typed parameters.
|
||||
All three platform bindings share the same function signatures,
|
||||
differing only in language-specific conventions.
|
||||
Function signatures documented below are from the Rust perspective.
|
||||
|
||||
- Rust: <https://github.com/vacp2p/zerokit/blob/master/rln/src/public.rs>
|
||||
- FFI: <https://github.com/vacp2p/zerokit/tree/master/rln/src/ffi>
|
||||
- WASM: <https://github.com/vacp2p/zerokit/tree/master/rln-wasm>
|
||||
|
||||
### Error Handling
|
||||
|
||||
Error handling differs across platform bindings.
|
||||
|
||||
For native Rust,
|
||||
functions return `Result<T, RLNError>` where `RLNError` is an enum
|
||||
representing specific error conditions.
|
||||
The enum variants provide type-safe error handling and
|
||||
pattern matching capabilities.
|
||||
|
||||
For WASM and FFI bindings,
|
||||
errors are returned as human-readable string messages.
|
||||
This simplifies cross-language error propagation at
|
||||
the cost of type safety.
|
||||
Applications consuming these bindings SHOULD parse error strings or
|
||||
use error message prefixes to distinguish error types when needed.
|
||||
|
||||
### Initialization
|
||||
|
||||
Functions with the same name but different signatures are conditional compilation variants.
|
||||
Only one variant exists at compile time based on enabled feature flags.
|
||||
|
||||
`RLN::new(tree_depth, tree_config)` creates a new RLN instance by loading circuit resources from the default folder.
|
||||
The `tree_config` parameter accepts multiple types via the `TreeConfigInput` trait: a JSON string,
|
||||
a direct config object (with pmtree feature), or an empty string for defaults.
|
||||
Not available in WASM. Not available when `stateless` feature is enabled.
|
||||
|
||||
`RLN::new()` creates a new stateless RLN instance by loading circuit resources from the default folder.
|
||||
Only available when `stateless` feature is enabled. Not available in WASM.
|
||||
|
||||
`RLN::new_with_params(tree_depth, zkey_data, graph_data, tree_config)` creates a new RLN instance
|
||||
with pre-loaded circuit parameters passed as byte vectors.
|
||||
The `tree_config` parameter accepts multiple types via the `TreeConfigInput` trait.
|
||||
Not available in WASM. Not available when `stateless` feature is enabled.
|
||||
|
||||
`RLN::new_with_params(zkey_data, graph_data)` creates a new stateless RLN instance with pre-loaded circuit parameters.
|
||||
Only available when `stateless` feature is enabled. Not available in WASM.
|
||||
|
||||
`RLN::new_with_params(zkey_data)` creates a new stateless RLN instance for WASM with pre-loaded zkey data.
|
||||
Graph data is not required as witness calculation is handled externally in WASM environments (e.g., using [witness_calculator.js](https://github.com/vacp2p/zerokit/blob/master/rln-wasm/resources/witness_calculator.js)).
|
||||
Only available in WASM with `stateless` feature enabled.
|
||||
|
||||
### Key Generation
|
||||
|
||||
`keygen()` generates a random identity keypair returning `(identity_secret, id_commitment)`.
|
||||
|
||||
`seeded_keygen(seed)` generates a deterministic identity keypair
|
||||
from a seed returning `(identity_secret, id_commitment)`.
|
||||
|
||||
`extended_keygen()` generates a random extended identity keypair
|
||||
returning `(identity_trapdoor, identity_nullifier, identity_secret, id_commitment)`.
|
||||
|
||||
`extended_seeded_keygen(seed)` generates a deterministic extended identity keypair
|
||||
from a seed returning `(identity_trapdoor, identity_nullifier, identity_secret, id_commitment)`.
|
||||
|
||||
### Merkle Tree Management
|
||||
|
||||
All tree management functions are only available when
|
||||
`stateless` feature is NOT enabled.
|
||||
|
||||
`set_tree(tree_depth)` initializes the internal Merkle tree with the specified depth.
|
||||
Leaves are set to the default zero value.
|
||||
|
||||
`set_leaf(index, leaf)` sets a leaf value at the specified index.
|
||||
|
||||
`get_leaf(index)` returns the leaf value at the specified index.
|
||||
|
||||
`set_leaves_from(index, leaves)` sets multiple leaves starting from the specified index.
|
||||
Updates `next_index` to `max(next_index, index + n)`.
|
||||
If n leaves are passed, they will be set at positions `index`, `index+1`, ..., `index+n-1`.
|
||||
|
||||
`init_tree_with_leaves(leaves)` resets the tree state to default and initializes it
|
||||
with the provided leaves starting from index 0.
|
||||
This resets the internal `next_index` to 0 before setting the leaves.
|
||||
|
||||
`atomic_operation(index, leaves, indices)` atomically inserts leaves starting from index
|
||||
and removes leaves at the specified indices.
|
||||
Updates `next_index` to `max(next_index, index + n)` where n is the number of leaves inserted.
|
||||
|
||||
`set_next_leaf(leaf)` sets a leaf at the next available index and increments `next_index`.
|
||||
The leaf is set at the current `next_index` value, then `next_index` is incremented.
|
||||
|
||||
`delete_leaf(index)` sets the leaf at the specified index to the default zero value.
|
||||
Does not change the internal `next_index` value.
|
||||
|
||||
`leaves_set()` returns the number of leaves that have been set in the tree.
|
||||
|
||||
`get_root()` returns the current Merkle tree root.
|
||||
|
||||
`get_subtree_root(level, index)` returns the root of a subtree at the specified level and index.
|
||||
|
||||
`get_merkle_proof(index)` returns the Merkle proof for the leaf at the specified index as `(path_elements, identity_path_index)`.
|
||||
|
||||
`get_empty_leaves_indices()` returns indices of leaves set to zero up to the final leaf that was set.
|
||||
|
||||
`set_metadata(metadata)` stores arbitrary metadata in the RLN object for application use.
|
||||
This metadata is not used by the RLN module.
|
||||
|
||||
`get_metadata()` returns the metadata stored in the RLN object.
|
||||
|
||||
`flush()` closes the connection to the Merkle tree database.
|
||||
Should be called before dropping the RLN object when using persistent storage.
|
||||
|
||||
### Witness Construction
|
||||
|
||||
`RLNWitnessInput::new(identity_secret, user_message_limit, message_id, path_elements, identity_path_index, x, external_nullifier)` constructs
|
||||
a witness input for proof generation. Validates that `message_id < user_message_limit`.
|
||||
|
||||
### Witness Calculation
|
||||
|
||||
For native (non-WASM) environments, witness calculation is handled internally by the proof generation functions.
|
||||
The circuit witness is computed from the `RLNWitnessInput` and passed to the zero-knowledge proof system.
|
||||
|
||||
For WASM environments, witness calculation must be performed externally using a JavaScript witness calculator.
|
||||
The workflow is:
|
||||
|
||||
1. Create a `WasmRLNWitnessInput` with the required parameters
|
||||
2. Export to JSON format using `toBigIntJson()` method
|
||||
3. Pass the JSON to an external JavaScript witness calculator
|
||||
4. Use the calculated witness with `generate_rln_proof_with_witness`
|
||||
|
||||
The witness calculator computes all intermediate values required by the RLN circuit.
|
||||
|
||||
### Proof Generation
|
||||
|
||||
`generate_zk_proof(witness)` generates a Groth16 zkSNARK proof from a witness.
|
||||
Extract proof values separately using `proof_values_from_witness`.
|
||||
Not available in WASM.
|
||||
|
||||
`generate_rln_proof(witness)` generates a complete RLN proof returning both the zkSNARK proof and proof values as `(proof, proof_values)`.
|
||||
This combines proof generation and proof values extraction.
|
||||
Not available in WASM.
|
||||
|
||||
`generate_rln_proof_with_witness(calculated_witness, witness)` generates an RLN proof using
|
||||
a pre-calculated witness from an external witness calculator.
|
||||
The `calculated_witness` should be a `Vec<BigInt>` obtained from the external witness calculator.
|
||||
Returns `(proof, proof_values)`.
|
||||
This is the primary proof generation method for WASM where witness calculation is handled by JavaScript.
|
||||
|
||||
### Proof Verification
|
||||
|
||||
`verify_zk_proof(proof, proof_values)` verifies only the zkSNARK proof without root or signal validation.
|
||||
Returns `true` if the proof is valid.
|
||||
|
||||
`verify_rln_proof(proof, proof_values, x)` verifies the proof against the internal Merkle tree root and
|
||||
validates that `x` matches the proof signal.
|
||||
Returns an error if verification fails (invalid proof, invalid root, or invalid signal).
|
||||
Only available when `stateless` feature is NOT enabled.
|
||||
|
||||
`verify_with_roots(proof, proof_values, x, roots)` verifies the proof against a set of acceptable roots and
|
||||
validates the signal.
|
||||
If the roots slice is empty, root verification is skipped. Returns an error if verification fails.
|
||||
|
||||
### Slashing
|
||||
|
||||
`recover_id_secret(proof_values_1, proof_values_2)` recovers the identity secret from two proof values
|
||||
that share the same external nullifier.
|
||||
Used to detect and penalize rate limit violations.
|
||||
|
||||
### Hash Utilities
|
||||
|
||||
`poseidon_hash(inputs)` computes the Poseidon hash of the input field elements.
|
||||
|
||||
`hash_to_field_le(input)` hashes arbitrary bytes to a field element using little-endian byte order.
|
||||
|
||||
`hash_to_field_be(input)` hashes arbitrary bytes to a field element using big-endian byte order.
|
||||
|
||||
### Serialization Utilities
|
||||
|
||||
`rln_witness_to_bytes_le` / `rln_witness_to_bytes_be` serializes an RLN witness to bytes.
|
||||
|
||||
`bytes_le_to_rln_witness` / `bytes_be_to_rln_witness` deserializes bytes to an RLN witness.
|
||||
|
||||
`rln_proof_to_bytes_le` / `rln_proof_to_bytes_be` serializes an RLN proof to bytes.
|
||||
|
||||
`bytes_le_to_rln_proof` / `bytes_be_to_rln_proof` deserializes bytes to an RLN proof.
|
||||
|
||||
`rln_proof_values_to_bytes_le` / `rln_proof_values_to_bytes_be` serializes proof values to bytes.
|
||||
|
||||
`bytes_le_to_rln_proof_values` / `bytes_be_to_rln_proof_values` deserializes bytes to proof values.
|
||||
|
||||
`fr_to_bytes_le` / `fr_to_bytes_be` serializes a field element to 32 bytes.
|
||||
|
||||
`bytes_le_to_fr` / `bytes_be_to_fr` deserializes 32 bytes to a field element.
|
||||
|
||||
`vec_fr_to_bytes_le` / `vec_fr_to_bytes_be` serializes a vector of field elements to bytes.
|
||||
|
||||
`bytes_le_to_vec_fr` / `bytes_be_to_vec_fr` deserializes bytes to a vector of field elements.
|
||||
|
||||
### WASM-Specific Notes
|
||||
|
||||
WASM bindings wrap the Rust API with JavaScript-compatible types. Key differences:
|
||||
|
||||
- Field elements are wrapped as `WasmFr` with `fromBytesLE`, `fromBytesBE`, `toBytesLE`, `toBytesBE` methods.
|
||||
- Vectors of field elements use `VecWasmFr` with `push`, `get`, `length` methods.
|
||||
- Identity generation uses `Identity.generate()` and `Identity.generateSeeded(seed)` static methods.
|
||||
- Extended identity uses `ExtendedIdentity.generate()` and `ExtendedIdentity.generateSeeded(seed)`.
|
||||
- Witness input uses `WasmRLNWitnessInput` constructor and `toBigIntJson()` for witness calculator integration.
|
||||
- Proof generation requires external witness calculation via `generateRLNProofWithWitness(calculatedWitness, witness)`.
|
||||
- When `parallel` feature is enabled, call `initThreadPool()` to initialize the thread pool.
|
||||
- Errors are returned as JavaScript strings that can be caught via try-catch blocks.
|
||||
|
||||
### FFI-Specific Notes
|
||||
|
||||
FFI bindings use C-compatible types with the `ffi_` prefix. Key differences:
|
||||
|
||||
- Field elements are wrapped as `CFr` with corresponding conversion functions.
|
||||
- Results use `CResult` or `CBoolResult` structs with `ok` and `err` fields.
|
||||
- Errors are returned as C-compatible strings in the `err` field of result structs.
|
||||
- Memory must be explicitly freed using `ffi_*_free` functions.
|
||||
- Vectors use `repr_c::Vec` with `ffi_vec_*` helper functions.
|
||||
- Configuration is passed via file path to a JSON configuration file.
|
||||
|
||||
## Usage Patterns
|
||||
|
||||
This section describes common deployment scenarios and
|
||||
the recommended API combinations for each.
|
||||
|
||||
### Stateful with Changing Root
|
||||
|
||||
Applies when membership changes over time with members joining and slashing continuously.
|
||||
|
||||
Applications MUST maintain a sliding window of recent roots externally.
|
||||
When members are added or removed via `set_leaf`, `delete_leaf`, or `atomic_operation`,
|
||||
capture the new root using `get_root` and append it to the history buffer.
|
||||
Verify incoming proofs using `verify_with_roots` with the root history buffer,
|
||||
accepting proofs valid against any recent root.
|
||||
|
||||
The window size depends on network propagation delays and epoch duration.
|
||||
|
||||
### Stateful with Fixed Root
|
||||
|
||||
Applies when membership is established once and remains static during an operation period.
|
||||
|
||||
Initialize the tree using `init_tree_with_leaves` with the complete membership set.
|
||||
No root history is required.
|
||||
Verify proofs using `verify_rln_proof` which checks against the internal tree root directly.
|
||||
|
||||
### Stateless
|
||||
|
||||
Applies when membership state is managed externally,
|
||||
such as by a smart contract or relay network.
|
||||
|
||||
Enable the `stateless` feature flag.
|
||||
Obtain Merkle proofs and valid roots from the external source.
|
||||
Pass externally provided `path_elements` and `identity_path_index` to `RLNWitnessInput::new`.
|
||||
Verify using `verify_with_roots` with externally provided roots.
|
||||
|
||||
### WASM Browser Integration
|
||||
|
||||
WASM environments require external witness calculation.
|
||||
Use `WasmRLNWitnessInput::toBigIntJson()` to export the witness for
|
||||
JavaScript witness calculators,
|
||||
then pass the result to `generateRLNProofWithWitness`.
|
||||
|
||||
When `parallel` feature is enabled,
|
||||
call `initThreadPool()` before proof operations.
|
||||
This requires COOP/COEP headers for SharedArrayBuffer support.
|
||||
|
||||
#### Epoch and Rate Limit Configuration
|
||||
|
||||
The external nullifier is computed as `poseidon_hash([epoch, rln_identifier])`.
|
||||
The `rln_identifier` is a field element that uniquely identifies your application (e.g., a hash of your app name).
|
||||
|
||||
Each application SHOULD use a unique `rln_identifier` to
|
||||
prevent cross-application nullifier collisions.
|
||||
|
||||
The `user_message_limit` in the rate commitment determines messages allowed per epoch.
|
||||
The `message_id` must be less than `user_message_limit` and
|
||||
should increment with each message.
|
||||
Applications MUST persist the `message_id` counter to avoid violations after restarts.
|
||||
|
||||
## Security/Privacy Considerations
|
||||
|
||||
The security of Zerokit depends on the correct implementation of the RLN-V2 protocol
|
||||
and the underlying zero-knowledge proof system.
|
||||
Applications MUST ensure that:
|
||||
|
||||
- Identity secrets are kept confidential and never transmitted or logged
|
||||
- The `message_id` counter is properly persisted to prevent accidental rate limit violations
|
||||
- External nullifiers are constructed correctly to prevent cross-application attacks
|
||||
- Merkle tree roots are validated when using stateless mode
|
||||
- Circuit parameters (zkey and graph data) are obtained from trusted sources
|
||||
|
||||
When using the `parallel` feature in WASM,
|
||||
applications MUST serve content with appropriate COOP/COEP headers to
|
||||
enable SharedArrayBuffer support securely.
|
||||
|
||||
The slashing mechanism exposes identity secrets when rate limits are violated.
|
||||
Applications SHOULD educate users about this risk and
|
||||
implement safeguards to prevent accidental violations.
|
||||
|
||||
## References
|
||||
|
||||
### Normative
|
||||
|
||||
- [RLN-V1 Specification](../32/rln-v1.md) - Rate Limit Nullifier V1 protocol
|
||||
|
||||
### Informative
|
||||
|
||||
- [Zerokit GitHub Repository](https://github.com/vacp2p/zerokit) - Reference implementation
|
||||
- [RLN-V2 Specification](./rln-v2.md) - Rate Limit Nullifier V2 protocol
|
||||
- [Sled Database](https://sled.rs) - Embedded database for persistent Merkle tree storage
|
||||
- [Witness Calculator](https://github.com/vacp2p/zerokit/blob/master/rln-wasm/resources/witness_calculator.js) - JavaScript witness calculator for WASM environments
|
||||
|
||||
## Copyright
|
||||
|
||||
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
|
||||
Reference in New Issue
Block a user