github/research.logos.co
1
1
Fork 0
mirror of https://github.com/vacp2p/research.logos.co.git synced 2026-02-19 11:57:37 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
gh-pages
research.logos.co/rlog/rss.xml
Anton Iakimov 5f72a4e7d3 branch history squashed
2024-05-16 16:26:43 +02:00

1779 lines
933 KiB
XML
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
<channel>
<title>Vac Research Blog</title>
<link>https://vac.dev/rlog</link>
<description>Vac Research Blog</description>
<lastBuildDate>Mon, 13 May 2024 12:00:00 GMT</lastBuildDate>
<docs>https://validator.w3.org/feed/docs/rss2.html</docs>
<generator>https://github.com/jpmonette/feed</generator>
<language>en</language>
<item>
<title><![CDATA[RLN-v3: Towards a Flexible and Cost-Efficient Implementation]]></title>
<link>https://vac.dev/rlog/rln-v3</link>
<guid>https://vac.dev/rlog/rln-v3</guid>
<pubDate>Mon, 13 May 2024 12:00:00 GMT</pubDate>
<description><![CDATA[Improving on the previous version of RLN by allowing dynamic epoch sizes.]]></description>
<content:encoded><![CDATA[<p>Improving on the previous version of RLN by allowing dynamic epoch sizes.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Recommended previous reading: <a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a>.</p><p>The premise of RLN-v3 is to have a variable message rate per variable epoch,
which can be explained in the following way:</p><ul><li><p><strong>RLN-v1:</strong> “Alice can send 1 message per global epoch”</p><p>Practically, this is <code>1 msg/second</code></p></li><li><p><strong>RLN-v2:</strong> “Alice can send <code>x</code> messages per global epoch”</p><p>Practically, this is <code>x msg/second</code></p></li><li><p><strong>RLN-v3:</strong> “Alice can send <code>x</code> messages within a time interval <code>y</code> chosen by herself.
The funds she has to pay are affected by both the number of messages and the chosen time interval.
Other participants can choose different time intervals fitting their specific needs.</p><p>Practically, this is <code>x msg/y seconds</code></p></li></ul><p>RLN-v3 allows higher flexibility and ease of payment/stake for users who have more predictable usage patterns and therefore,
more predictable bandwidth usage on a p2p network (Waku, etc.).</p><p>For example:</p><ul><li>An AMM that broadcasts bids, asks, and fills over Waku may require a lot of throughput in the smallest epoch possible and hence may register an RLN-v3 membership of <code>10000 msg/1 second</code>.
They could do this with RLN-v2, too.</li><li>Alice, a casual user of a messaging app built on Waku, who messages maybe 3-4 people infrequently during the day, may register an RLN-v3 membership of <code>100 msg/hour</code>,
which would not be possible in RLN-v2 considering the <code>global epoch</code> was set to <code>1 second</code>.
With RLN-v2, Alice would have to register with a membership of <code>1 msg/sec</code>,
which would translate to <code>3600 msg/hour</code>. This is much higher than her usage and would
result in her overpaying to stake into the membership set.</li><li>A sync service built over Waku,
whose spec defines that it MUST broadcast a set of public keys every hour,
may register an RLN-v3 membership of <code>1 msg/hour</code>,
cutting down the costs to enter the membership set earlier.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="theory">Theory<a href="#theory" class="hash-link" aria-label="Direct link to Theory" title="Direct link to Theory">​</a></h2><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modification-to-leaves-set-in-the-membership-merkle-tree">Modification to leaves set in the membership Merkle tree<a href="#modification-to-leaves-set-in-the-membership-merkle-tree" class="hash-link" aria-label="Direct link to Modification to leaves set in the membership Merkle tree" title="Direct link to Modification to leaves set in the membership Merkle tree">​</a></h3><p>To ensure that a user’s epoch size (<code>user_epoch_limit</code>) is included within their membership we must modify the user’s commitment/leaf in the tree to contain it.
A user’s commitment/leaf in the tree is referred to as a <code>rate_commitment</code>,
which was previously derived from their public key (<code>identity_commitment</code>)
and their variable message rate (<code>user_message_limit</code>).</p><p>In <strong>RLN-v2:</strong></p><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>r</mi><mi>a</mi><mi>t</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo separator="true">,</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>m</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi>a</mi><mi>g</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">rate\_commitment = poseidon([identity\_commitment, user\_message\_limit])</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">m</span><span class="mord mathnormal">ess</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mclose">])</span></span></span></span></span></div><p>In <strong>RLN-v3:</strong></p><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>r</mi><mi>a</mi><mi>t</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo separator="true">,</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>m</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi>a</mi><mi>g</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo separator="true">,</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">rate\_commitment = poseidon([identity\_commitment, user\_message\_limit, user\_epoch\_limit])</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">m</span><span class="mord mathnormal">ess</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mclose">])</span></span></span></span></span></div><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modification-to-circuit-inputs">Modification to circuit inputs<a href="#modification-to-circuit-inputs" class="hash-link" aria-label="Direct link to Modification to circuit inputs" title="Direct link to Modification to circuit inputs">​</a></h3><p>To detect double signaling,
we make use of a circuit output <code>nullifier</code>,
which remains the same if a user generates a proof with the same <code>message_id</code> and <code>external_nullifier</code>,
where the <code>external_nullifier</code> and <code>nullifier</code> are defined as:</p><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>e</mi><mi>x</mi><mi>t</mi><mi>e</mi><mi>r</mi><mi>n</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>n</mi><mi>u</mi><mi>l</mi><mi>l</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mo separator="true">,</mo><mi>r</mi><mi>l</mi><mi>n</mi><mi mathvariant="normal">_</mi><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo><mspace linebreak="newline"></mspace><mi>n</mi><mi>u</mi><mi>l</mi><mi>l</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>e</mi><mi>c</mi><mi>r</mi><mi>e</mi><mi>t</mi><mo separator="true">,</mo><mi>e</mi><mi>x</mi><mi>t</mi><mi>e</mi><mi>r</mi><mi>n</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>n</mi><mi>u</mi><mi>l</mi><mi>l</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo separator="true">,</mo><mi>m</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi>a</mi><mi>g</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>i</mi><mi>d</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">external\_nullifier = poseidon([epoch, rln\_identifier]) \\ nullifier = poseidon([identity\_secret, external\_nullifier, message\_id])</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">x</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">na</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">n</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mclose">])</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">secre</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">x</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">na</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">m</span><span class="mord mathnormal">ess</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mclose">])</span></span></span></span></span></div><p>Where:</p><ul><li><code>epoch</code> is defined as the Unix epoch timestamp with seconds precision.</li><li><code>rln_identifier</code> uniquely identifies an application for which a user submits a proof.</li><li><code>identity_secret</code> is the private key of the user.</li><li><code>message_id</code> is the sequence number of the user’s message within <code>user_message_limit</code> in an epoch.</li></ul><p>In RLN-v2, the global epoch was 1 second,
hence we did not need to perform any assertions to the epoch’s value inside the circuit,
and the validation of the epoch was handled off-circuit (i.e., too old, too large, bad values, etc.).</p><p>In RLN-v3, we propose that the <code>epoch</code> that is passed into the circuit
must be a valid multiple of <code>user_epoch_limit</code>
since the user may pass in values of the <code>epoch</code> which do not directly correlate with the <code>user_epoch_limit</code>.</p><p>For example:</p><ul><li>A user with <code>user_epoch_limit</code> of 120
passes in an epoch of <code>237</code>
generates <code>user_message_limit</code> proofs with it,
can increment the epoch by <code>1</code>,
and generate <code>user_message_limit</code> proofs with it,
thereby allowing them to bypass the message per epoch restriction.</li></ul><p>One could say that we could perform this validation outside of the circuit,
but we maintain the <code>user_epoch_limit</code> as a private input to the circuit so that the user is not deanonymized by the anonymity set connected to that <code>user_epoch_limit</code>.
Since <code>user_epoch_limit</code> is kept private,
the verifier does not have access to that value and cannot perform validation on it.</p><p>If we ensure that the <code>epoch</code> is a multiple of <code>user_epoch_limit</code>,
we have the following scenarios:</p><ul><li>A user with <code>user_epoch_limit</code> of 120
passes in an epoch of <code>237</code>.
Proof generation fails since the epoch is not a multiple of <code>user_epoch_limit</code>.</li><li>A user with <code>user_epoch_limit</code> of 120
passes in an epoch of <code>240</code> and
can generate <code>user_message_limit</code> proofs without being slashed.</li></ul><p>Since we perform operations on the <code>epoch</code>, we must include it as a circuit input (previously, it was removed from the circuit inputs to RLN-v2).</p><p>Therefore, the new circuit inputs are as follows:</p><div class="language-c codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-c codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">// unchanged</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private identity_secret</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private user_message_limit</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private message_id</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private pathElements</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private pathIndices</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">public x </span><span class="token comment" style="color:rgb(98, 114, 164)">// messageHash</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">// new/changed</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private user_epoch_limit</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private user_epoch_quotient </span><span class="token comment" style="color:rgb(98, 114, 164)">// epoch/user_epoch_limit to assert within circuit</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">public epoch</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">public rln_identifier</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>The circuit outputs remain the same.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="additional-circuit-constraints">Additional circuit constraints<a href="#additional-circuit-constraints" class="hash-link" aria-label="Direct link to Additional circuit constraints" title="Direct link to Additional circuit constraints">​</a></h3><ol><li><p>Since we accept the <code>epoch</code>, <code>user_epoch_quotient</code>, and <code>user_epoch_limit</code>,
we must ensure that the relation between these 3 values is preserved. I.e.:</p><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mo>=</mo><mo>=</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo>∗</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>q</mi><mi>u</mi><mi>o</mi><mi>t</mi><mi>i</mi><mi>e</mi><mi>n</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">epoch == user\_epoch\_limit * user\_epoch\_quotient</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">==</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span></span></span></span></span></div></li><li><p>To ensure no overflows/underflows occur in the above multiplication,
we must constrain the inputs of <code>epoch</code>, <code>user_epoch_quotient</code>, and <code>user_epoch_limit</code>.
We have assumed <code>3600</code> to be the maximum valid size of the <code>user_epoch_quotient</code>.</p></li></ol><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo stretchy="false">(</mo><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mo stretchy="false">)</mo><mo>≤</mo><mn>64</mn><mtext>&nbsp;</mtext><mi>b</mi><mi>i</mi><mi>t</mi><mi>s</mi><mspace linebreak="newline"></mspace><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo stretchy="false">(</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo stretchy="false">)</mo><mo>≤</mo><mn>12</mn><mtext>&nbsp;</mtext><mi>b</mi><mi>i</mi><mi>t</mi><mi>s</mi><mspace linebreak="newline"></mspace><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo>≤</mo><mn>3600</mn><mspace linebreak="newline"></mspace><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo>≤</mo><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mspace linebreak="newline"></mspace><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>q</mi><mi>u</mi><mi>o</mi><mi>t</mi><mi>i</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>&lt;</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">size(epoch) \leq 64\ bits \\ size(user\_epoch\_limit) \leq 12\ bits \\ user\_epoch\_limit \leq 3600 \\ user\_epoch\_limit \leq epoch \\ user\_epoch\_quotient &lt; user\_epoch\_limit</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mopen">(</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">64</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">bi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">s</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mopen">(</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">12</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">bi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">s</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">3600</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">&lt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span></span></span></span></span></div><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modifications-to-external-epoch-validation-waku-etc">Modifications to external epoch validation (Waku, etc.)<a href="#modifications-to-external-epoch-validation-waku-etc" class="hash-link" aria-label="Direct link to Modifications to external epoch validation (Waku, etc.)" title="Direct link to Modifications to external epoch validation (Waku, etc.)">​</a></h3><p>For receivers of an RLN-v3 proof
to detect if a message is too old, we must use the higher bound of the <code>user_epoch_limit</code>, which has been set to <code>3600</code>.
The <strong>trade-off</strong> here is that we allow hour-old messages to propagate within the network.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modifications-to-double-signaling-detection-scheme-waku-etc">Modifications to double signaling detection scheme (Waku, etc.)<a href="#modifications-to-double-signaling-detection-scheme-waku-etc" class="hash-link" aria-label="Direct link to Modifications to double signaling detection scheme (Waku, etc.)" title="Direct link to Modifications to double signaling detection scheme (Waku, etc.)">​</a></h3><p>For verifiers of RLN-v1/v2 proofs,
a log of nullifiers seen in the last epoch is maintained,
and if there is a match with a pre-existing nullifier,
double signaling has been detected and the verifier MAY proceed to slash the spamming user.</p><p>With the RLN-v3 scheme,
we need to increase the size of the nullifier log used,
which previously cleared itself every second to the higher bound of the <code>user_epoch_limit</code>, which is <code>3600</code>.
Now, the RLN proof verifier must clear the nullifier log every <code>3600</code> seconds to satisfactorily detect double signaling.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-implementation">The implementation<a href="#the-implementation" class="hash-link" aria-label="Direct link to The implementation" title="Direct link to The implementation">​</a></h2><p>An implementation of the RLN-v3 scheme in <a href="https://docs.gnark.consensys.io/" target="_blank" rel="noopener noreferrer">gnark</a> can be found <a href="https://github.com/vacp2p/gnark-rln/blob/9b05eddc89901a06d8f41b093ce8ce12fd0bb4e0/rln/rln.go" target="_blank" rel="noopener noreferrer">here</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="comments-on-performance">Comments on performance<a href="#comments-on-performance" class="hash-link" aria-label="Direct link to Comments on performance" title="Direct link to Comments on performance">​</a></h2><ul><li>Hardware: Macbook Air M2, 16GB RAM</li><li>Circuit: <a href="https://github.com/vacp2p/gnark-rln/blob/9b05eddc89901a06d8f41b093ce8ce12fd0bb4e0/rln/rln.go" target="_blank" rel="noopener noreferrer">RLN-v3</a></li><li>Proving system: <a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer"><code>Groth16</code></a></li><li>Framework: <a href="https://docs.gnark.consensys.io/" target="_blank" rel="noopener noreferrer"><code>gnark</code></a></li><li>Elliptic curve: <a href="https://eprint.iacr.org/2013/879.pdf" target="_blank" rel="noopener noreferrer"><code>bn254</code></a> (aka bn128) (not to be confused with the 254-bit Weierstrass curve)</li><li>Finite field: Prime-order subgroup of the group of points on the <code>bn254</code> curve</li><li>Default Merkle tree height: <code>20</code></li><li>Hashing algorithm: <a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer"><code>Poseidon</code></a></li><li>Merkle tree: <a href="https://github.com/rate-limiting-nullifier/pmtree" target="_blank" rel="noopener noreferrer"><code>Sparse Indexed Merkle Tree</code></a></li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proving">Proving<a href="#proving" class="hash-link" aria-label="Direct link to Proving" title="Direct link to Proving">​</a></h3><p>The proving time for the RLN-v3 circuit is <code>90ms</code> for a single proof.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="verification">Verification<a href="#verification" class="hash-link" aria-label="Direct link to Verification" title="Direct link to Verification">​</a></h3><p>The verification time for the RLN-v3 circuit is <code>1.7ms</code> for a single proof.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2><p>The RLN-v3 scheme introduces a new epoch-based message rate-limiting scheme to the RLN protocol.
It enhances the user's flexibility in setting their message limits and cost-optimizes their stake.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><ul><li>Implementing the RLN-v3 scheme in <a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li><li>Implementing the RLN-v3 scheme in <a href="https://github.com/waku-org/nwaku" target="_blank" rel="noopener noreferrer">Waku</a></li><li>Formal security analysis of the RLN-v3 scheme</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a></li><li><a href="https://github.com/rate-limiting-nullifier/circom-rln" target="_blank" rel="noopener noreferrer">RLN Circuits</a></li><li><a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer">Groth16</a></li><li><a href="https://docs.gnark.consensys.io/" target="_blank" rel="noopener noreferrer">Gnark</a></li><li><a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer">Poseidon Hash</a></li><li><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li><li><a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">RLN-v1 RFC</a></li><li><a href="https://rfc.vac.dev/spec/58/" target="_blank" rel="noopener noreferrer">RLN-v2 RFC</a></li><li><a href="https://waku.org" target="_blank" rel="noopener noreferrer">Waku</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Verifying RLN Proofs in Light Clients with Subtrees]]></title>
<link>https://vac.dev/rlog/rln-light-verifiers</link>
<guid>https://vac.dev/rlog/rln-light-verifiers</guid>
<pubDate>Fri, 03 May 2024 12:00:00 GMT</pubDate>
<description><![CDATA[How resource-restricted devices can verify RLN proofs fast and efficiently.]]></description>
<content:encoded><![CDATA[<p>How resource-restricted devices can verify RLN proofs fast and efficiently.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Recommended previous reading: <a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a>.</p><p>This post expands upon ideas described in the previous post,
focusing on how resource-restricted devices can verify RLN proofs fast and efficiently.</p><p>Previously, it was required to fetch all the memberships from the smart contract,
construct the merkle tree locally,
and derive the merkle root,
which is subsequently used to verify RLN proofs.</p><p>This process is not feasible for resource-restricted devices since it involves a lot of RPC calls, computation and fault tolerance.
One cannot expect a mobile phone to fetch all the memberships from the smart contract and construct the merkle tree locally.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="constraints-and-requirements">Constraints and requirements<a href="#constraints-and-requirements" class="hash-link" aria-label="Direct link to Constraints and requirements" title="Direct link to Constraints and requirements">​</a></h2><p>An alternative solution to the one proposed in this post is to construct the merkle tree on-chain,
and have the root accessible with a single RPC call.
However, this approach increases gas costs for inserting new memberships and <em>may</em> not be feasible until it is optimized further with batching mechanisms, etc.</p><p>The other methods have been explored in more depth <a href="https://hackmd.io/@rymnc/rln-tree-storages" target="_blank" rel="noopener noreferrer">here</a>.</p><p>Following are the requirements and constraints for the solution proposed in this post:</p><ol><li>Cheap membership insertions.</li><li>As few RPC calls as possible to reduce startup time.</li><li>Merkle root of the tree is available on-chain.</li><li>No centralized services to sequence membership insertions.</li><li>Map inserted commitments to the block in which they were inserted.</li></ol><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="metrics-on-sync-time-for-a-tree-with-2653-leaves">Metrics on sync time for a tree with 2,653 leaves<a href="#metrics-on-sync-time-for-a-tree-with-2653-leaves" class="hash-link" aria-label="Direct link to Metrics on sync time for a tree with 2,653 leaves" title="Direct link to Metrics on sync time for a tree with 2,653 leaves">​</a></h2><p>The following metrics are based on the current implementation of RLN in the Waku gen0 network.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="test-bench">Test bench<a href="#test-bench" class="hash-link" aria-label="Direct link to Test bench" title="Direct link to Test bench">​</a></h3><ul><li>Hardware: Macbook Air M2, 16GB RAM</li><li>Network: 120 Megabits/sec</li><li>Nwaku commit: <a href="https://github.com/waku-org/nwaku/tree/e61e4ff90a235657a7dc4248f5be41b6e031e98c" target="_blank" rel="noopener noreferrer">e61e4ff</a></li><li>RLN membership set contract: <a href="https://sepolia.etherscan.io/address/0xF471d71E9b1455bBF4b85d475afb9BB0954A29c4#code" target="_blank" rel="noopener noreferrer">0xF471d71E9b1455bBF4b85d475afb9BB0954A29c4</a></li><li>Deployed block number: 4,230,716</li><li>RLN Membership set depth: 20</li><li>Hash function: PoseidonT3 (which is a gas guzzler)</li><li>Max size of the membership set: 2^20 = 1,048,576 leaves</li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="metrics">Metrics<a href="#metrics" class="hash-link" aria-label="Direct link to Metrics" title="Direct link to Metrics">​</a></h3><ul><li>Time to sync the whole tree: 4 minutes</li><li>RPC calls: 702</li><li>Number of leaves: 2,653</li></ul><p>One can argue that the time to sync the tree at the current state is not <em>that</em> bad.
However, the number of RPC calls is a concern,
which scales linearly with the number of blocks since the contract was deployed
This is because the implementation fetches all events from the contract,
chunking 2,000 blocks at a time.
This is done to avoid hitting the block limit of 10,000 events per call,
which is a limitation of popular RPC providers.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proposed-solution">Proposed solution<a href="#proposed-solution" class="hash-link" aria-label="Direct link to Proposed solution" title="Direct link to Proposed solution">​</a></h2><p>From a theoretical perspective,
one could construct the merkle tree on-chain,
in a view call, in-memory.
However, this is not feasible due to the gas costs associated with it.</p><p>To compute the root of a Merkle tree with <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>20</mn></msup></mrow><annotation encoding="application/x-tex">2^{20}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span></span></span></span></span> leaves it costs approximately 2 billion gas.
With Infura and Alchemy capping the gas limit to 350M and 550M gas respectively,
it is not possible to compute the root of the tree in a single call.</p><p>Acknowledging that <a href="https://polygon.technology/blog/polygon-miden-state-model" target="_blank" rel="noopener noreferrer">Polygon Miden</a> and <a href="https://penumbra.zone/blog/tiered-commitment-tree/" target="_blank" rel="noopener noreferrer">Penumbra</a> both make use of a tiered commitment tree,
we propose a similar approach for RLN.</p><p>A tiered commitment tree is a tree which is sharded into multiple smaller subtrees,
each of which is a tree in itself.
This allows scaling in terms of the number of leaves,
as well as reducing state bloat by just storing the root of a subtree when it is full instead of all its leaves.</p><p>Here, the question arises:
What is the maximum number of leaves in a subtree with which the root can be computed in a single call?</p><p>It costs approximately 217M gas to compute the root of a Merkle tree with <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span></span></span></span></span></span></span></span></span> leaves.</p><p>This is a feasible number for a single call,
and hence we propose a tiered commitment tree with a maximum of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span></span></span></span></span></span></span></span></span> leaves in a subtree and the number of subtrees is <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span></span></span></span></span></span></span></span></span>.
Therefore, the maximum number of leaves in the tree is <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>20</mn></msup></mrow><annotation encoding="application/x-tex">2^{20}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span></span></span></span></span> (the same as the current implementation).</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="img" src="/assets/images/light-rln-verifiers-f801999160884be6a1223ee7d76cebcf.png" width="631" height="381" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="insertion">Insertion<a href="#insertion" class="hash-link" aria-label="Direct link to Insertion" title="Direct link to Insertion">​</a></h3><p>When a commitment is inserted into the tree it is first inserted into the first subtree.
When the first subtree is full the next insertions go into the second subtree and so on.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="syncing">Syncing<a href="#syncing" class="hash-link" aria-label="Direct link to Syncing" title="Direct link to Syncing">​</a></h3><p>When syncing the tree,
one only needs to fetch the roots of the subtrees.
The root of the full tree can be computed in-memory or on-chain.</p><p>This allows us to derive the following relation:</p><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>n</mi><mi>u</mi><mi>m</mi><mi>b</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>o</mi><mi>f</mi><mi mathvariant="normal">_</mi><mi>r</mi><mi>p</mi><mi>c</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>a</mi><mi>l</mi><mi>l</mi><mi>s</mi><mo>=</mo><mi>n</mi><mi>u</mi><mi>m</mi><mi>b</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>o</mi><mi>f</mi><mi mathvariant="normal">_</mi><mi>f</mi><mi>i</mi><mi>l</mi><mi>l</mi><mi>e</mi><mi>d</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi>s</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">number\_of\_rpc\_calls = number\_of\_filled\_subtrees + 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal">mb</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">p</span><span class="mord mathnormal">c</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">c</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal">mb</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">e</span><span class="mord mathnormal">d</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal">rees</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span></span></div><p>This is a significant improvement over the current implementation,
which requires fetching all the memberships from the smart contract.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gas-costs">Gas costs<a href="#gas-costs" class="hash-link" aria-label="Direct link to Gas costs" title="Direct link to Gas costs">​</a></h3><p>The gas costs for inserting a commitment into the tree are the same as the current implementation except it consists of an extra SSTORE operation to store the <code>shardIndex</code> of the commitment.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="events">Events<a href="#events" class="hash-link" aria-label="Direct link to Events" title="Direct link to Events">​</a></h3><p>The events emitted by the contract are the same as the current implementation,
appending the <code>shardIndex</code> of the commitment.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proof-of-concept">Proof of concept<a href="#proof-of-concept" class="hash-link" aria-label="Direct link to Proof of concept" title="Direct link to Proof of concept">​</a></h3><p>A proof of concept implementation of the tiered commitment tree is available <a href="https://github.com/vacp2p/rln-contract/pull/37" target="_blank" rel="noopener noreferrer">here</a>,
and is deployed on Sepolia at <a href="https://sepolia.etherscan.io/address/0xE7987c70B54Ff32f0D5CBbAA8c8Fc1cAf632b9A5" target="_blank" rel="noopener noreferrer">0xE7987c70B54Ff32f0D5CBbAA8c8Fc1cAf632b9A5</a>.</p><p>It is compatible with the current implementation of the RLN verifier.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><ol><li>Optimize the gas costs of the tiered commitment tree.</li><li>Explore using different number of leaves under a given node in the tree (currently set to 2).</li></ol><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2><p>The tiered commitment tree is a promising approach to reduce the number of RPC calls required to sync the tree and reduce the gas costs associated with computing the root of the tree.
Consequently, it allows for a more scalable and efficient RLN verifier.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://github.com/rate-limiting-nullifier/circom-rln" target="_blank" rel="noopener noreferrer">RLN Circuits</a></li><li><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li><li><a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">RLN-V1 RFC</a></li><li><a href="https://rfc.vac.dev/spec/58/" target="_blank" rel="noopener noreferrer">RLN-V2 RFC</a></li><li><a href="https://hackmd.io/7cBCMU5hS5OYv8PTaW2wAQ?view" target="_blank" rel="noopener noreferrer">RLN Implementers guide</a></li><li><a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku]]></title>
<link>https://vac.dev/rlog/rln-anonymous-dos-prevention</link>
<guid>https://vac.dev/rlog/rln-anonymous-dos-prevention</guid>
<pubDate>Tue, 07 Nov 2023 12:00:00 GMT</pubDate>
<description><![CDATA[Rate Limiting Nullifiers in practice, applied to an anonymous p2p network, like Waku.]]></description>
<content:encoded><![CDATA[<p>Rate Limiting Nullifiers in practice, applied to an anonymous p2p network, like Waku.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Rate Limiting Nullifier (RLN) is a zero-knowledge gadget that allows users to prove 2 pieces of information,</p><ol><li>They belong to a permissioned membership set</li><li>Their rate of signaling abides by a fixed number that has been previously declared</li></ol><p>The "membership set" introduced above, is in the form of a sparse, indexed merkle tree.
This membership set can be maintained on-chain, off-chain or as a hybrid depending on the network's storage costs.
Waku makes use of a hybrid membership set,
where insertions are tracked in a smart contract.
In addition, each Waku node maintains a local copy of the tree,
which is updated upon each insertion.</p><p>Users register themselves with a hash of a locally generated secret,
which is then inserted into the tree at the next available index.
After having registered, users can prove their membership by proving their knowledge of the pre-image of the respective leaf in the tree.
The leaf hashes are also referred to as commitments of the respective users.
The actual proof is done by a <a href="https://ethereum.org/en/developers/tutorials/merkle-proofs-for-offline-data-integrity/" target="_blank" rel="noopener noreferrer">Merkle Inclusion Proof</a>, which is a type of ZK proof.</p><p>The circuit ensures that the user's secret does indeed hash to a leaf in the tree,
and that the provided Merkle proof is valid.</p><p>After a User generates this Merkle proof,
they can transmit it to other users,
who can verify the proof.
Including a message's hash within the proof generation,
additionally guarantees integrity of that message.</p><p>A malicious user could generate multiple proofs per epoch.
they generate multiple proofs per epoch.
However, when multiple proofs are generated per epoch,
the malicious user's secret is exposed, which strongly disincentivizes this attack.
This mechanism is further described in <a href="#malicious-user-secret-interpolation-mechanism">malicious User secret interpolation mechanism</a></p><p>Note: This blog post describes rln-v1, which excludes the range check in favor of a global rate limit for all users,
which is once per time window. This version is currently in use in waku-rln-relay.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-protocol-parameters">RLN Protocol parameters<a href="#rln-protocol-parameters" class="hash-link" aria-label="Direct link to RLN Protocol parameters" title="Direct link to RLN Protocol parameters">​</a></h2><p>Given below is the set of cryptographic primitives,
and constants that are used in the RLN protocol. </p><ol><li>Proving System: <a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer"><code>groth16</code></a></li><li>Elliptic Curve: <a href="https://eprint.iacr.org/2013/879.pdf" target="_blank" rel="noopener noreferrer"><code>bn254</code></a> (aka bn128) (not to be confused with the 254 bit Weierstrass curve)</li><li>Finite Field: Prime-order subgroup of the group of points on the <code>bn254</code> curve</li><li>Default Merkle Tree Height: <code>20</code></li><li>Hashing algorithm: <a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer"><code>Poseidon</code></a></li><li>Merkle Tree: <a href="https://github.com/rate-limiting-nullifier/pmtree" target="_blank" rel="noopener noreferrer"><code>Sparse Indexed Merkle Tree</code></a></li><li>Messages per epoch: <code>1</code></li><li>Epoch duration: <code>10 seconds</code></li></ol><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="malicious-user-secret-interpolation-mechanism">Malicious User secret interpolation mechanism<a href="#malicious-user-secret-interpolation-mechanism" class="hash-link" aria-label="Direct link to Malicious User secret interpolation mechanism" title="Direct link to Malicious User secret interpolation mechanism">​</a></h2><blockquote><p>note: all the parameters mentioned below are elements in the finite field mentioned above.</p></blockquote><p>The private inputs to the circuit are as follows: -</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">identitySecret: the randomly generated secret of the user</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">identityPathIndex: the index of the commitment derived from the secret</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">pathElements: elements included in the path to the index of the commitment</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>Following are the public inputs to the circuit -</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">x: hash of the signal to the finite field</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">rlnIdentifier: application-specific identifier which this proof is being generated for</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">epoch: the timestamp which this proof is being generated for</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>The outputs of the circuit are as follows: -</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">y: result of Shamir's secret sharing calculation</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">root: root of the Merkle tree obtained after applying the inclusion proof</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">nullifier: uniquely identifies a message, derived from rlnIdentifier, epoch, and the user's secret</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>With the above data in mind, following is the circuit pseudocode -</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">identityCommitment = Poseidon([identitySecret])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">root = MerkleInclusionProof(identityCommitment, identityPathIndex, pathElements)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">externalNullifier = Poseidon([epoch, rlnIdentifier])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">a1 = Poseidon([identitySecret, externalNullifier])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">y = identitySecret + a1 * x</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">nullifier = Poseidon([a1])</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>To interpolate the secret of a user who has sent multiple signals during the same epoch to the same rln-based application, we may make use of the following formula -</p><p><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>a</mi><mn>1</mn></msub><mo>=</mo><mfrac><mrow><mo stretchy="false">(</mo><msub><mi>y</mi><mn>1</mn></msub><mo>−</mo><msub><mi>y</mi><mn>2</mn></msub><mo stretchy="false">)</mo></mrow><mrow><mo stretchy="false">(</mo><msub><mi>x</mi><mn>1</mn></msub><mo>−</mo><msub><mi>x</mi><mn>2</mn></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">a_1 = {(y_1 - y_2) \over (x_1 - x_2)}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.53em;vertical-align:-0.52em"></span><span class="mord"><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.01em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mbin mtight">−</span><span class="mord mtight"><span class="mord mathnormal mtight">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mclose mtight">)</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.485em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mbin mtight">−</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mclose mtight">)</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.52em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span></span></p><p>where <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>, <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">y_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">x_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>, <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">y_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> are shares from different messages</p><p>subsequently, we may use one pair of the shares, <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">y_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> to obtain the <code>identitySecret</code></p><p><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi>S</mi><mi>e</mi><mi>c</mi><mi>r</mi><mi>e</mi><mi>t</mi><mo>=</mo><msub><mi>y</mi><mn>1</mn></msub><mo>−</mo><msub><mi>a</mi><mn>1</mn></msub><mo>∗</mo><mi>x</mi></mrow><annotation encoding="application/x-tex">identitySecret = y_1 - a_1 * x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mord mathnormal">ecre</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7778em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6153em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span></span></p><p>This enables RLN to be used for rate limiting with a <em>global</em> limit. For arbitrary limits,
please refer to an article written by @curryrasul, <a href="https://mirror.xyz/privacy-scaling-explorations.eth/iCLmH1JVb7fDqp6Mms2NR001m2_n5OOSHsLF2QrxDnQ" target="_blank" rel="noopener noreferrer">rln-v2</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="wakus-problem-with-dos">Waku's problem with DoS<a href="#wakus-problem-with-dos" class="hash-link" aria-label="Direct link to Waku's problem with DoS" title="Direct link to Waku's problem with DoS">​</a></h2><p>In a decentralized, privacy focused messaging system like <a href="https://waku.org" target="_blank" rel="noopener noreferrer">Waku</a>,
Denial of Service (DoS) vulnerabilities are very common, and must be addressed to promote network scale and optimal bandwidth utilization.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dos-prevention-with-user-metadata">DoS prevention with user metadata<a href="#dos-prevention-with-user-metadata" class="hash-link" aria-label="Direct link to DoS prevention with user metadata" title="Direct link to DoS prevention with user metadata">​</a></h3><p>There are a couple of ways a user can be rate-limited, either -</p><ol><li>IP Logging</li><li>KYC Logging</li></ol><p>Both IP and KYC logging prevent systems from being truly anonymous, and hence, cannot be used as a valid DoS prevention mechanism for Waku.</p><p>RLN can be used as an alternative, which provides the best of both worlds, i.e a permissioned membership set, as well as anonymous signaling.
However, we are bound by k-anonymity rules of the membership set.</p><p><a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">Waku-RLN-Relay</a> is a <a href="https://libp2p.io" target="_blank" rel="noopener noreferrer">libp2p</a> pubsub validator that verifies if a proof attached to a given message is valid.
In case the proof is valid, the message is relayed.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="performance-analysis">Performance analysis<a href="#performance-analysis" class="hash-link" aria-label="Direct link to Performance analysis" title="Direct link to Performance analysis">​</a></h2><blockquote><p>Test bench specs: AMD EPYC 7502P 32-Core, 4x32GB DDR4 Reg.ECC Memory </p></blockquote><p>This simulation was conducted by @alrevuelta, and is described in more detail <a href="https://github.com/waku-org/research/issues/23" target="_blank" rel="noopener noreferrer">here</a>.</p><p>The simulation included 100 waku nodes running in parallel.</p><p>Proof generation times -
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="img" src="/assets/images/proof_generation_time-195632e4864fa4c5f883895f2ea9e9e3.png" width="1547" height="1096" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Proof verification times -
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="img" src="/assets/images/proof_verification_time-c95708ef2a4fc0470114fbceebc6bc30.png" width="1564" height="1214" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>A spammer node publishes 3000 msg/epoch, which is detected by all connected nodes, and subsequently disconnect to prevent further spam -
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="img" src="/assets/images/spam_prevention_in_action-50221f227e3d94be5aeae45193cc04ea.png" width="1574" height="1108" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="security-analysis">Security analysis<a href="#security-analysis" class="hash-link" aria-label="Direct link to Security analysis" title="Direct link to Security analysis">​</a></h2><p><a href="https://doi.org/10.1007/s00145-018-9280-5" target="_blank" rel="noopener noreferrer">Barbulescu and Duquesne</a>
conclude that that the <code>bn254</code> curve has only 100 bits of security.
Since the bn254 curve has a small embedding degree,
it is vulnerable to the <a href="https://en.wikipedia.org/wiki/MOV_attack" target="_blank" rel="noopener noreferrer">MOV attack</a>.
However, the MOV attack is only applicable to pairings,
and not to the elliptic curve itself.
It is acceptable to use the bn254 curve for RLN,
since the circuit does not make use of pairings.</p><p><a href="https://github.com/vacp2p/research/issues/155" target="_blank" rel="noopener noreferrer">An analysis</a> on the number of rounds in the Poseidon hash function was done,
which concluded that the hashing rounds should <em>not</em> be reduced, </p><p>The <a href="https://github.com/vacp2p/rln-contract" target="_blank" rel="noopener noreferrer">smart contracts</a> have <em>not</em> been audited, and are not recommended for real world deployments <em>yet</em>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="storage-analysis">Storage analysis<a href="#storage-analysis" class="hash-link" aria-label="Direct link to Storage analysis" title="Direct link to Storage analysis">​</a></h2><div class="math math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><mn>32</mn><mtext>&nbsp;</mtext><mi>b</mi><mi>y</mi><mi>t</mi><mi>e</mi><mi>s</mi><mspace linebreak="newline"></mspace><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>h</mi><mi>e</mi><mi>i</mi><mi>g</mi><mi>h</mi><mi>t</mi><mo>=</mo><mn>20</mn><mspace linebreak="newline"></mspace><mi>t</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>e</mi><mi>a</mi><mi>v</mi><mi>e</mi><mi>s</mi><mo>=</mo><msup><mn>2</mn><mn>20</mn></msup><mspace linebreak="newline"></mspace><mi>m</mi><mi>a</mi><mi>x</mi><mi mathvariant="normal">_</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><mi>t</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>e</mi><mi>a</mi><mi>v</mi><mi>e</mi><mi>s</mi><mo>∗</mo><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mspace linebreak="newline"></mspace><mi>m</mi><mi>a</mi><mi>x</mi><mi mathvariant="normal">_</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><msup><mn>2</mn><mn>20</mn></msup><mo>∗</mo><mn>32</mn><mo>=</mo><mn>33</mn><mo separator="true">,</mo><mn>554</mn><mo separator="true">,</mo><mn>432</mn><mspace linebreak="newline"></mspace><mo>∴</mo><mi>m</mi><mi>a</mi><mi>x</mi><mi mathvariant="normal">_</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><mn>33.55</mn><mtext>&nbsp;</mtext><mi>m</mi><mi>e</mi><mi>g</mi><mi>a</mi><mi>b</mi><mi>y</mi><mi>t</mi><mi>e</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">commitment\_size = 32\ bytes \\ tree\_height =20 \\ total\_leaves = 2^{20} \\ max\_tree\_size = total\_leaves * commitment\_size \\ max\_tree\_size = 2^{20} * 32 = 33,554,432 \\ ∴max\_tree\_size = 33.55\ megabytes</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord">32</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord mathnormal">t</span><span class="mord mathnormal">es</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">h</span><span class="mord mathnormal">e</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">h</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">20</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">t</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">e</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">es</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8641em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8641em"><span style="top:-3.113em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">ma</span><span class="mord mathnormal">x</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">t</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">e</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">es</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">ma</span><span class="mord mathnormal">x</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8641em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8641em"><span style="top:-3.113em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">32</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em"></span><span class="mord">33</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">554</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">432</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.6922em"></span><span class="mrel amsrm">∴</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">ma</span><span class="mord mathnormal">x</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord">33.55</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">ab</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord mathnormal">t</span><span class="mord mathnormal">es</span></span></span></span></span></div><p>The storage overhead introduced by RLN is minimal.
RLN only requires 34 megabytes of storage, which poses no problem on most end-user hardware, with the exception of IoT/microcontrollers.
Still, we are working on further optimizations allowing proof generation without having to store the full tree.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-bare-minimum-requirements-to-run-rln">The bare minimum requirements to run RLN<a href="#the-bare-minimum-requirements-to-run-rln" class="hash-link" aria-label="Direct link to The bare minimum requirements to run RLN" title="Direct link to The bare minimum requirements to run RLN">​</a></h2><p>With proof generation time in sub-second latency, along with low storage overhead for the tree,
it is possible for end users to generate and verify RLN proofs on a modern smartphone.</p><p>Following is a demo provided by @rramos that demonstrates
<a href="https://drive.google.com/file/d/1ITLYrDOQrHQX2_3Q6O5EqKPYJN8Ye2gF/view?usp=sharing" target="_blank" rel="noopener noreferrer">waku-rln-relay used in react native</a>.</p><blockquote><p>Warning: The react native sdk will be deprecated soon, and the above demo should serve as a PoC for RLN on mobiles</p></blockquote><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-usage-guide">RLN usage guide<a href="#rln-usage-guide" class="hash-link" aria-label="Direct link to RLN usage guide" title="Direct link to RLN usage guide">​</a></h2><p><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a> implements api's that allow users to handle operations to the tree,
as well as generate/verify RLN proofs.</p><p>Our main implementation of RLN can be accessed via this Rust <a href="https://crates.io/crates/rln" target="_blank" rel="noopener noreferrer">crate</a>,
which is documented <a href="https://docs.rs/rln/0.4.1/rln/public/struct.RLN.html" target="_blank" rel="noopener noreferrer">here</a>.
It can used in other langugages via the FFI API, which is documented <a href="https://docs.rs/rln/0.4.1/rln/ffi/index.html" target="_blank" rel="noopener noreferrer">here</a>.
The usage of RLN in Waku is detailed in our <a href="https://hackmd.io/7cBCMU5hS5OYv8PTaW2wAQ?view" target="_blank" rel="noopener noreferrer">RLN Implementers guide</a>,
which provides step-by-step instructions on how to run Waku-RLN-Relay.</p><p>Following is a diagram that will help understand the dependency tree -</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="rln-dep-tree" src="/assets/images/rln_dep_tree-0bf1837513daecde1a3de4deb9a8855f.jpg" width="631" height="552" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><ul><li>Optimizations to zerokit for proof generation time.</li><li>Incrementing tree depth from 20 to 32, to allow more memberships.</li><li>Optimizations to the smart contract.</li><li>An ability to signal validity of a message in different time windows.</li><li>Usage of proving systems other than Groth16.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://github.com/rate-limiting-nullifier/circom-rln" target="_blank" rel="noopener noreferrer">RLN Circuits</a></li><li><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li><li><a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">RLN-V1 RFC</a></li><li><a href="https://rfc.vac.dev/spec/58/" target="_blank" rel="noopener noreferrer">RLN-V2 RFC</a></li><li><a href="https://hackmd.io/7cBCMU5hS5OYv8PTaW2wAQ?view" target="_blank" rel="noopener noreferrer">RLN Implementers guide</a></li><li><a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer">groth16</a></li><li><a href="https://eprint.iacr.org/2013/879.pdf" target="_blank" rel="noopener noreferrer">bn254</a></li><li><a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer">Poseidon Hash</a></li><li><a href="https://github.com/rate-limiting-nullifier/pmtree" target="_blank" rel="noopener noreferrer">Sparse Indexed Merkle Tree</a></li><li><a href="https://doi.org/10.1007/s00145-018-9280-5" target="_blank" rel="noopener noreferrer">Updating key size estimations for pairings</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[GossipSub Improvements: Evolution of Overlay Design and Message Dissemination in Unstructured P2P Networks]]></title>
<link>https://vac.dev/rlog/GossipSub Improvements</link>
<guid>https://vac.dev/rlog/GossipSub Improvements</guid>
<pubDate>Mon, 06 Nov 2023 12:00:00 GMT</pubDate>
<description><![CDATA[GossipSub Improvements: Evolution of Overlay Design and Message Dissemination in Unstructured P2P Networks]]></description>
<content:encoded><![CDATA[<p>GossipSub Improvements: Evolution of Overlay Design and Message Dissemination in Unstructured P2P Networks </p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="motivitation">Motivitation<a href="#motivitation" class="hash-link" aria-label="Direct link to Motivitation" title="Direct link to Motivitation">​</a></h2><p>We have been recently working on analyzing and improving the performance of the GossipSub protocol for large messages,
as in the case of Ethereum Improvement Proposal <a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">EIP-4844</a>.
This work led to a comprehensive study of unstructured P2P networks.
The intention was to identify the best practices that can serve as guidelines for performance improvement and scalability of P2P networks.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Nodes in an unstructured p2p network form self-organizing overlay(s) on top of the IP infrastructure to facilitate different services like information dissemination,
query propagation, file sharing, etc. The overlay(s) can be as optimal as a tree-like structure or as enforcing as a fully connected mesh. </p><p>Due to peer autonomy and a trustless computing environment, some peers may deviate from the expected operation or even leave the network.
At the same time, the underlying IP layer is unreliable. </p><p>Therefore, tree-like overlays are not best suited for reliable information propagation.
Moreover, tree-based solutions usually result in significantly higher message dissemination latency due to suboptimal branches. </p><p>Flooding-based solutions, on the other hand, result in maximum resilience against adversaries and achieve minimal message dissemination latency because the message propagates through all (including the optimal) paths.
Redundant transmissions help maintain the integrity and security of the network in the presence of adversaries and high node failure but significantly increase network-wide bandwidth utilization, cramming the bottleneck links. </p><p>An efficient alternative is to lower the number of redundant transmissions by D-regular broadcasting, where a peer will likely receive (or relay) a message from up to <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> random peers.
Publishing through a D-regular overlay triggers approximately <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">N \times D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> transmissions.
Reducing <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> reduces the redundant transmissions but compromises reachability and latency.
Sharing metadata through a K-regular overlay (where <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>&gt;</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">K &gt; D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">&gt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span>) allows nodes to pull missing messages. </p><p>GossipSub [<a href="https://arxiv.org/pdf/2007.02754.pdf" target="_blank" rel="noopener noreferrer">1</a>] benefits from full-message (D-regular) and metadata-only (k-regular) overlays.
Alternatively, a metadata-only overlay can be used, requiring a pull-based operation that significantly minimizes bandwidth utilization at the cost of increased latency. </p><p>Striking the right balance between parameters like <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo separator="true">,</mo><mi>K</mi></mrow><annotation encoding="application/x-tex">D, K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span></span>, pull-based operation, etc., can yield application-specific performance tuning, but scalability remains a problem.</p><p>At the same time, many other aspects can significantly contribute to the network's performance and scalability.
One option is to realize peers' suitability and continuously changing capabilities while forming overlays. </p><p>For instance, a low-bandwidth link near a publisher can significantly demean the entire network's performance.
Reshuffling of peering links according to the changing network conditions can lead to superior performance. </p><p>Laying off additional responsibilities to more capable nodes (super nodes) can alleviate peer cramming, but it makes the network susceptible to adversaries/peer churn.
Grouping multiple super nodes to form virtual node(s) can solve this problem. </p><p>Similarly, flat (single-tier) overlays cannot address the routing needs in large (geographically dispersed) networks.</p><p>Hierarchical (Multi-tier) overlays with different intra/inter-overlay routing solutions can better address these needs.
Moreover, using message aggregation schemes for grouping multiple messages can save bandwidth and provide better resilience against adversaries/peer churn.</p><p>This article's primary objective is to investigate the possible choices that can empower an unstructured P2P network to achieve superior performance for the broadest set of applications.
We look into different constraints imposed by application-specific needs (performance goals) and investigate various choices that can augment the network's performance.
We explore overlay designs/freshness, peer selection approaches, message-relaying mechanisms, and resilience against adversaries/peer churn.
We consider GossipSub a baseline protocol to explore various possibilities and decisively commit to the ones demonstrating superior performance.
We also discuss the current state and, where applicable, propose a strategic plan for embedding new features to the GossipSub protocol. </p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal1-low-latency-operation">GOAL1: Low Latency Operation<a href="#goal1-low-latency-operation" class="hash-link" aria-label="Direct link to GOAL1: Low Latency Operation" title="Direct link to GOAL1: Low Latency Operation">​</a></h2><p>Different applications, like blockchain, streaming, etc., impose strict time bounds on network-wide message dissemination latency.
A message delivered after the imposed time bounds is considered as dropped.
An early message delivery in applications like live streaming can further enhance the viewing quality.</p><p>The properties and nature of the overlay network topology significantly impact the performance of services and applications executed on top of them.
Studying and devising mechanisms for better overlay design and message dissemination is paramount to achieving superior performance.</p><p>Interestingly, shortest-path message delivery trees have many limitations: </p><p>1) Changing network dynamics requires a quicker and continuous readjustment of the multicast tree.
2) The presence of resource-constrained (bandwidth/compute, etc.) nodes in the overlay can result in congestion.
3) Node failure can result in partitions, making many segments unreachable.
4) Assuring a shortest-path tree-like structure requires a detailed view of the underlying (and continuously changing) network topology. </p><p>Solutions involve creating multiple random trees to add redundancy [<a href="https://ieeexplore.ieee.org/abstract/document/6267905" target="_blank" rel="noopener noreferrer">2</a>].
Alternatives involve building an overlay mesh and forwarding messages through the multicast delivery tree (eager push). </p><p>Metadata is shared through the overlay links so that the nodes can ask for missing messages (lazy push or pull-based operation) through the overlay links.
New nodes are added from the overlay on node failure, but it requires non-faulty node selection.</p><p>GossipSub uses eager push (through overlay mesh) and lazy push (through IWANT messages). </p><p>The mesh degree <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>L</mi><mi>o</mi><mi>w</mi></mrow></msub><mo>≤</mo><mi>D</mi><mo>≤</mo><msub><mi>D</mi><mrow><mi>H</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{Low} \leq D \leq D_{High}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">L</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8193em;vertical-align:-0.136em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.08125em">H</span><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span> is crucial in deciding message dissemination latency.
A smaller value for <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> results in higher latency due to increased rounds, whereas a higher <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> reduces latency on the cost of increased bandwidth.
At the same time, keeping <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> independent of the growing network size (<span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span></span></span></span></span>) may increase network-wide message dissemination latency.
Adjusting <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> with <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span></span></span></span></span> maintains similar latency on the cost of increased workload for peers.
Authors in [<a href="https://infoscience.epfl.ch/record/83478/files/EugGueKerMas04IEEEComp.pdf" target="_blank" rel="noopener noreferrer">3</a>] suggest only a logarithmic increase in <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> to maintain a manageable workload for peers.
In [<a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">4</a>], it is reported that the average mesh degree should not exceed <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub><mo>=</mo><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>+</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">D_{avg} = \ln(N) + C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span></span> for an optimal operation,
where <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi></mrow><annotation encoding="application/x-tex">C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span></span> is a small constant.</p><p>Moreover, quicker shuffling of peers results in better performance in the presence of resource-constrained nodes or node failure [<a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">4</a>].</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal2-considering-heterogeneity-in-overlay-design">GOAL2: Considering Heterogeneity In Overlay Design<a href="#goal2-considering-heterogeneity-in-overlay-design" class="hash-link" aria-label="Direct link to GOAL2: Considering Heterogeneity In Overlay Design" title="Direct link to GOAL2: Considering Heterogeneity In Overlay Design">​</a></h2><p>Random peering connections in P2P overlays represent a stochastic process. It is inherently difficult to precisely model the performance of such systems.
Most of the research on P2P networks provides simulation results assuming nodes with similar capabilities.
The aspect of dissimilar capabilities and resource-constrained nodes is less explored.</p><p>It is discussed in GOAL1 that overlay mesh results in better performance if <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{avg}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span> does not exceed <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>+</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">\ln(N) + C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span></span>.
Enforcing all the nodes to have approximately <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>+</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">\ln(N) + C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span></span> peers makes resource-rich nodes under-utilized, while resource-constrained nodes are overloaded.
At the same time, connecting high-bandwidth nodes through a low-bandwidth node undermines the network's performance.
Ideally, the workload on any node should not exceed its available resources.
A better solution involves a two-phased operation:</p><ol><li><p>Every node computes its available bandwidth and selects a node degree <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> proportional to its available bandwidth [<a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">4</a>].
Different bandwidth estimation approaches are suggested in literature [<a href="https://ieeexplore.ieee.org/abstract/document/1224454" target="_blank" rel="noopener noreferrer">5</a>,<a href="https://ieeexplore.ieee.org/abstract/document/1248658" target="_blank" rel="noopener noreferrer">6</a>].
Simple bandwidth estimation approaches like variable packet size probing [<a href="https://ieeexplore.ieee.org/abstract/document/1248658" target="_blank" rel="noopener noreferrer">6</a>] yield similar results with less complexity.
It is also worth mentioning that many nodes may want to allocate only a capped share of their bandwidth to the network.
Lowering <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> according to the available bandwidth can still prove helpful.
Additionally, bandwidth preservation at the transport layer through approaches like µTP can be useful.
To further conform to the suggested mesh-degree average <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{avg}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span>, every node tries achieving this average within its neighborhood, resulting in an overall similar <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{avg}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span>.</p></li><li><p>From the available local view, every node tries connecting peers with the lowest latency until <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> connections are made.
We suggest referring to the peering solution discussed in GOAL5 to avoid network partitioning.</p></li></ol><p>The current GossipSub design considers homogeneous peers, and every node tries maintaining <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>L</mi><mi>o</mi><mi>w</mi></mrow></msub><mo>≤</mo><mi>D</mi><mo>≤</mo><msub><mi>D</mi><mrow><mi>H</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{Low} \leq D \leq D_{High}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">L</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8193em;vertical-align:-0.136em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.08125em">H</span><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span> connections. </p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal3-bandwidth-optimization">GOAL3: Bandwidth Optimization<a href="#goal3-bandwidth-optimization" class="hash-link" aria-label="Direct link to GOAL3: Bandwidth Optimization" title="Direct link to GOAL3: Bandwidth Optimization">​</a></h2><p>Redundant message transmissions are essential for handling adversaries/node failure. However, these transmissions result in traffic bursts, cramming many overlay links.
This not only adds to the network-wide message dissemination latency but a significant share of the network's bandwidth is wasted on (usually) unnecessary transmissions.
It is essential to explore solutions that can minimize the number of redundant transmissions while assuring resilience against node failures. </p><p>Many efforts have been made to minimize the impact of redundant transmissions.
These solutions include multicast delivery trees, metadata sharing to enable pull-based operation, in-network information caching, etc. [<a href="https://dl.acm.org/doi/abs/10.1145/945445.945473" target="_blank" rel="noopener noreferrer">7</a>,<a href="https://link.springer.com/chapter/10.1007/11558989_12" target="_blank" rel="noopener noreferrer">8</a>].
GossipSub employs a hybrid of eager push (message dissemination through the overlay) and lazy push (a pull-based operation by the nodes requiring information through IWANT messages). </p><p>A better alternative to simple redundant transmission is to use message aggregation [<a href="https://ieeexplore.ieee.org/abstract/document/8737576" target="_blank" rel="noopener noreferrer">9</a>,<a href="https://dl.acm.org/doi/abs/10.1145/1993636.1993676" target="_blank" rel="noopener noreferrer">10</a>,<a href="https://ieeexplore.ieee.org/abstract/document/4276446" target="_blank" rel="noopener noreferrer">11</a>] for the GossipSub protocol.
As a result, redundant message transmissions can serve as a critical advantage of the GossipSub protocol.
Suppose that we have three equal-length messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mn>1</mn><mo separator="true">,</mo><mi>x</mi><mn>2</mn><mo separator="true">,</mo><mi>x</mi><mn>3</mn></mrow><annotation encoding="application/x-tex">x1, x2, x3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em"></span><span class="mord mathnormal">x</span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">x</span><span class="mord">3</span></span></span></span></span>. Assuming an XOR coding function,
we know two trivial properties: <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mn>1</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mo>=</mo><mi>x</mi><mn>1</mn></mrow><annotation encoding="application/x-tex">x1 \oplus x2 \oplus x2 = x1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">x</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord mathnormal">x</span><span class="mord">1</span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><mi>x</mi><mn>1</mn><mi mathvariant="normal">∣</mi><mo>=</mo><mi mathvariant="normal">∣</mi><mi>x</mi><mn>1</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">\vert x1 \vert = \vert x1 \oplus x2 \oplus x2 \vert</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">∣</span><span class="mord mathnormal">x</span><span class="mord">1∣</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">∣</span><span class="mord mathnormal">x</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">x</span><span class="mord">2∣</span></span></span></span></span>. </p><p>This implies that instead of sending messages individually, we can encode and transmit composite message(s) to the network.
The receiver can reconstruct the original message from encoded segments.
As a result, fewer transmissions are sufficient for sending more messages to the network. </p><p>However, sharing linear combinations of messages requires organizing messages in intervals,
and devising techniques to identify all messages belonging to each interval.
In addition, combining messages from different publishers requires more complex arrangements,
involving embedding publisher/message IDs, delayed forwarding (to accommodate more messages), and mechanisms to ensure the decoding of messages at all peers.
Careful application-specific need analysis can help decide the benefits against the added complexity. </p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal4-handling-large-messages">GOAL4: Handling Large Messages<a href="#goal4-handling-large-messages" class="hash-link" aria-label="Direct link to GOAL4: Handling Large Messages" title="Direct link to GOAL4: Handling Large Messages">​</a></h2><p>Many applications require transferring large messages for their successful operation. For instance, database/blockchain transactions [<a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">12</a>].
This introduces two challenges: </p><p>1) Redundant large message transmissions result in severe network congestion.
2) Message transmissions follow a store/forward process at all peers, which is inefficient in the case of large messages. </p><p>The above-mentioned challenges result in a noticeable increase in message dissemination latency and bandwidth wastage.
Most of the work done for handling large messages involves curtailing redundant transmissions using multicast delivery trees,
reducing the number of fanout nodes, employing in-network message caching, pull-based operation, etc.</p><p>Approaches like message aggregation also prove helpful in minimizing bandwidth wastage.</p><p>Our recent work on GossipSub improvements (still a work in progress) suggests the following solutions to deal with large message transmissions: </p><ol><li><p>Using IDontWant message proposal [<a href="https://github.com/libp2p/specs/pull/413" target="_blank" rel="noopener noreferrer">13</a>] and staggered sending. </p><p>IDontWant message helps curtail redundant transmissions by letting other peers know we have already received the message.
Staggered sending enables relaying the message to a short subset of peers in each round.
We argue that simultaneously relaying a message to all peers hampers the effectiveness of the IDontWant message.
Therefore, using the IDontWant message with staggered sending can yield better results by allowing timely reception and processing of IDontWant messages.</p></li><li><p>Message transmissions follow a store/forward process at all peers that is inefficient in the case of large messages.
We can parallelize message transmission by partitioning large messages into smaller fragments, letting intermediate peers relay these fragments as soon as they receive them.</p></li></ol><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal5-scalability">GOAL5: Scalability<a href="#goal5-scalability" class="hash-link" aria-label="Direct link to GOAL5: Scalability" title="Direct link to GOAL5: Scalability">​</a></h2><p>P2P networks are inherently scalable because every incoming node brings in bandwidth and compute resources.
In other words, we can keep adding nodes to the network as long as every incoming node brings at-least <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">R \times D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> bandwidth,
where <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi></mrow><annotation encoding="application/x-tex">R</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span></span></span></span></span> is average data arrival rate.
It is worth mentioning that network-wide message dissemination requires at-least <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">⌈</mo><msub><mrow><mi>log</mi><mo>⁡</mo></mrow><mi>D</mi></msub><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo stretchy="false">⌉</mo></mrow><annotation encoding="application/x-tex">\lceil \log_D (N) \rceil</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">⌈</span><span class="mop"><span class="mop">lo<span style="margin-right:0.01389em">g</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2342em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2441em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)⌉</span></span></span></span></span> hops.
Therefore, increasing network size increases message dissemination latency, assuming D is independent of the network size.</p><p>Additionally, problems like peer churn, adversaries, heterogeneity, distributed operation, etc., significantly hamper the network's performance.
Most efforts for bringing scalability to the P2P systems have focused on curtailing redundant transmissions and flat overlay adjustments.
Hierarchical overlay designs, on the other hand, are less explored.</p><p>Placing a logical structure in unstructured P2P systems can help scale P2P networks. </p><p>One possible solution is to use a hierarchical overlay inspired by the approaches [<a href="https://link.springer.com/article/10.1007/s12083-016-0460-5" target="_blank" rel="noopener noreferrer">14</a>,<a href="https://link.springer.com/chapter/10.1007/978-3-030-19223-5_16" target="_blank" rel="noopener noreferrer">15</a>,<a href="https://ieeexplore.ieee.org/abstract/document/9826458" target="_blank" rel="noopener noreferrer">16</a>].
An abstract operation of such overlay design is provided below:</p><ol><li><p>Clustering nodes based on locality, assuming that such peers will have relatively lower intra-cluster latency and higher bandwidth.
For this purpose, every node tries connecting peers with the lowest latency until <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></span> connections are made or the cluster limit is reached.</p></li><li><p>A small subset of nodes having the highest bandwidth and compute resources is selected from each cluster.
These super nodes form a fully connected mesh and jointly act as a virtual node,
mitigating the problem of peer churn among super nodes.</p></li><li><p>Virtual nodes form a fully connected mesh to construct a hierarchical overlay.
Each virtual node is essentially a collection of super nodes;
a link to any of the constituent super nodes represents a link to the virtual node.</p></li><li><p>One possible idea is to use GossipSub for intra-cluster message dissemination and FloodSub for inter-cluster message dissemination.</p></li></ol><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary">Summary<a href="#summary" class="hash-link" aria-label="Direct link to Summary" title="Direct link to Summary">​</a></h2><p>Overlay acts as a virtual backbone for a P2P network. A flat overlay is more straightforward and allows effortless readjustment to application needs.
On the other hand, a hierarchical overlay can bring scalability at the cost of increased complexity.
Regardless of the overlay design, a continuous readjustment to appropriate peering links is essential for superior performance.
At the same time, bandwidth preservation (through message aggregation, caching at strategic locations, metadata sharing, pull-based operation, etc.) can help minimize latency.
However, problems like peer churn and in-network adversaries can be best alleviated through balanced redundant coverage, and frequent reshuffling of the peering links.</p><h1>References</h1><ul><li>[1]<!-- --> D. Vyzovitis, Y. Napora, D. McCormick, D. Dias, and Y. Psaras, “Gossipsub: Attack-resilient message propagation in the filecoin and eth2. 0 networks,” arXiv preprint arXiv:2007.02754, 2020. Retrieved from <a href="https://arxiv.org/pdf/2007.02754.pdf" target="_blank" rel="noopener noreferrer">https://arxiv.org/pdf/2007.02754.pdf</a></li><li>[2]<!-- --> M. Matos, V. Schiavoni, P. Felber, R. Oliveira, and E. Riviere, “Brisa: Combining efficiency and reliability in epidemic data dissemination,” in 2012 IEEE 26th International Parallel and Distributed Processing Symposium. IEEE, 2012, pp. 983–994. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/6267905" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/6267905</a></li><li>[3]<!-- --> P. T. Eugster, R. Guerraoui, A. M. Kermarrec, and L. Massouli, “Epidemic information dissemination in distributed systems,” IEEE Computer, vol. 37, no. 5, 2004. Retrieved from <a href="https://infoscience.epfl.ch/record/83478/files/EugGueKerMas04IEEEComp.pdf" target="_blank" rel="noopener noreferrer">https://infoscience.epfl.ch/record/83478/files/EugGueKerMas04IEEEComp.pdf</a></li><li>[4]<!-- --> D. Frey, “Epidemic protocols: From large scale to big data,” Ph.D. dissertation, Universite De Rennes 1, 2019. Retrieved from <a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">https://inria.hal.science/tel-02375909/document</a></li><li>[5]<!-- --> M. Jain and C. Dovrolis, “End-to-end available bandwidth: measurement methodology, dynamics, and relation with tcp throughput,” IEEE/ACM Transactions on networking, vol. 11, no. 4, pp. 537–549, 2003. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/1224454" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/1224454</a></li><li>[6]<!-- --> R. Prasad, C. Dovrolis, M. Murray, and K. Claffy, “Bandwidth estimation: metrics, measurement techniques, and tools,” IEEE network, vol. 17, no. 6, pp. 27–35, 2003. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/1248658" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/1248658</a></li><li>[7]<!-- --> D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat, “Bullet: High bandwidth data dissemination using an overlay mesh,” in Proceedings of the nineteenth ACM symposium on Operating systems principles, 2003, pp. 282–297. Retrieved from <a href="https://dl.acm.org/doi/abs/10.1145/945445.945473" target="_blank" rel="noopener noreferrer">https://dl.acm.org/doi/abs/10.1145/945445.945473</a></li><li>[8]<!-- --> V. Pai, K. Kumar, K. Tamilmani, V. Sambamurthy, and A. E. Mohr, “Chainsaw: Eliminating trees from overlay multicast,” in Peer-to-Peer Systems IV: 4th International Workshop, IPTPS 2005, Ithaca, NY, USA, February 24-25, 2005. Revised Selected Papers 4. Springer, 2005, pp. 127–140. Retrieved from <a href="https://link.springer.com/chapter/10.1007/11558989_12" target="_blank" rel="noopener noreferrer">https://link.springer.com/chapter/10.1007/11558989_12</a></li><li>[9]<!-- --> Y.-D. Bromberg, Q. Dufour, and D. Frey, “Multisource rumor spreading with network coding,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019, pp. 2359–2367. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/8737576" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/8737576</a></li><li>[10]<!-- --> B. Haeupler, “Analyzing network coding gossip made easy,” in Proceedings of the forty-third annual ACM symposium on Theory of computing, 2011, pp. 293–302. Retrieved from <a href="https://dl.acm.org/doi/abs/10.1145/1993636.1993676" target="_blank" rel="noopener noreferrer">https://dl.acm.org/doi/abs/10.1145/1993636.1993676</a></li><li>[11]<!-- --> S. Yu and Z. Li, “Massive data delivery in unstructured peer-to-peer networks with network coding,” in 6th IEEE/ACIS International Conference on Computer and Information Science (ICIS 2007). IEEE, 2007, pp. 592–597. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/4276446" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/4276446</a></li><li>[12]<!-- --> V. Buterin, D. Feist, D. Loerakker, G. Kadianakis, M. Garnett, M. Taiwo, and A. Dietrichs, “Eip-4844: Shard blob transactions scale data-availability of ethereum in a simple, forwards-compatible manner,” 2022. Retrieved from <a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">https://eips.ethereum.org/EIPS/eip-4844</a></li><li>[13]<!-- --> A. Manning, “Gossipsub extension for epidemic meshes (v1.2.0),” 2022. Retrieved from <a href="https://github.com/libp2p/specs/pull/413" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/pull/413</a></li><li>[14]<!-- --> Z. Duan, C. Tian, M. Zhou, X. Wang, N. Zhang, H. Du, and L. Wang, “Two-layer hybrid peer-to-peer networks,” Peer-to-Peer Networking and Applications, vol. 10, pp. 1304–1322, 2017. Retrieved from <a href="https://link.springer.com/article/10.1007/s12083-016-0460-5" target="_blank" rel="noopener noreferrer">https://link.springer.com/article/10.1007/s12083-016-0460-5</a></li><li>[15]<!-- --> W. Hao, J. Zeng, X. Dai, J. Xiao, Q. Hua, H. Chen, K.-C. Li, and H. Jin, “Blockp2p: Enabling fast blockchain broadcast with scalable peer-to-peer network topology,” in Green, Pervasive, and Cloud Computing: 14th International Conference, GPC 2019, Uberlandia, Brazil, May 26–28, 2019, Proceedings 14. Springer, 2019, pp. 223–237. Retrieved from <a href="https://link.springer.com/chapter/10.1007/978-3-030-19223-5_16" target="_blank" rel="noopener noreferrer">https://link.springer.com/chapter/10.1007/978-3-030-19223-5_16</a></li><li>[16]<!-- --> H. Qiu, T. Ji, S. Zhao, X. Chen, J. Qi, H. Cui, and S. Wang, “A geography-based p2p overlay network for fast and robust blockchain systems,” IEEE Transactions on Services Computing, 2022. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/9826458" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/9826458</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Nescience - A zkVM leveraging hiding properties]]></title>
<link>https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties</link>
<guid>https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties</guid>
<pubDate>Mon, 28 Aug 2023 12:00:00 GMT</pubDate>
<description><![CDATA[Nescience, a privacy-first blockchain zkVM.]]></description>
<content:encoded><![CDATA[<p>Nescience, a privacy-first blockchain zkVM. </p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Nescience is a privacy-first blockchain project that aims to enable private transactions and provide a general-purpose execution environment for classical applications.
The goals include creating a state separation architecture for public/private computation,
designing a versatile virtual machine based on mainstream instruction sets,
creating proofs for private state updates, implementing a kernel-based architecture for correct execution of private functions,
and implementing core DeFi protocols such as AMMs and staking from a privacy perspective. </p><p>It intends to create a user experience that is similar to public blockchains, but with additional privacy features that users can leverage at will.
To achieve this goal, Nescience will implement a versatile virtual machine that can be used to implement existing blockchain applications,
while also enabling the development of privacy-centric protocols such as private staking and private DEXs.</p><p>To ensure minimal trust assumptions and prevent information leakage, Nescience proposes a proof system that allows users to create proofs for private state updates,
while the verification of the proofs and the execution of the public functions inside the virtual machine can be delegated to an external incentivised prover. </p><p>It also aims to implement a seamless interaction between public and private state, enabling composability between contracts, and private and public functions.
Finally, Nescience intends to implement permissive licensing, which means that the source code will be open-source,
and developers will be able to use and modify the code without any restriction.</p><p>Our primary objective is the construction of the Zero-Knowledge Virtual Machine (zkVM). This document serves as a detailed exploration of the multifaceted challenges,
potential solutions, and alternatives that lay ahead. Each step is a testament to our commitment to thoroughness;
we systematically test various possibilities and decisively commit to the one that demonstrates paramount performance and utility.
For instance, as we progress towards achieving Goal 2, we are undertaking a rigorous benchmarking of the Nova proof system against its contemporaries.
Should Nova showcase superior performance metrics, we stand ready to integrate it as our proof system of choice. Through such meticulous approaches,
we not only reinforce the foundation of our project but also ensure its scalability and robustness in the ever-evolving landscape of blockchain technology.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-1-create-a-state-separation-architecture">Goal 1: Create a State Separation Architecture<a href="#goal-1-create-a-state-separation-architecture" class="hash-link" aria-label="Direct link to Goal 1: Create a State Separation Architecture" title="Direct link to Goal 1: Create a State Separation Architecture">​</a></h2><p>The initial goal revolves around crafting a distinctive architecture that segregates public and private computations,
employing an account-based framework for the public state and a UTXO-based structure for the private state. </p><p>The UTXO model [<a href="https://bitcoin.org/bitcoin.pdf" target="_blank" rel="noopener noreferrer">1</a>,<a href="https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/" target="_blank" rel="noopener noreferrer">2</a>], notably utilized in Bitcoin, generates new UTXOs to serve future transactions,
while the account-based paradigm assigns balances to accounts that transactions can modify.
Although the UTXO model bolsters privacy by concealing comprehensive balances,
the pursuit of a dual architecture mandates a meticulous synchronization of these state models,
ensuring that private transactions remain inconspicuous in the wider public network state. </p><p>This task is further complicated by the divergent transaction processing methods intrinsic to each model,
necessitating a thoughtful and innovative approach to harmonize their functionality.
To seamlessly bring together the dual architecture, harmonizing the account-based model for public state with the UTXO-based model for private state,
a comprehensive strategy is essential.</p><p>The concept of blending an account-based structure with a UTXO-based model for differentiating between public and private states is intriguing.
It seeks to leverage the strengths of both models: the simplicity and directness of the account-based model with the privacy enhancements of the UTXO model.</p><p>Here's a breakdown and a potential strategy for harmonizing these models:</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-rationale-behind-the-dual-architecture-"><ins> Rationale Behind the Dual Architecture: </ins><a href="#-rationale-behind-the-dual-architecture-" class="hash-link" aria-label="Direct link to -rationale-behind-the-dual-architecture-" title="Direct link to -rationale-behind-the-dual-architecture-">​</a></h3><ul><li><p><strong>Account-Based Model:</strong> This model is intuitive and easy to work with. Every participant has an account,
and transactions directly modify the balances of these accounts. It's conducive for smart contracts and a broad range of applications.</p></li><li><p><strong>UTXO-Based Model:</strong> This model treats every transaction as a new output, which can then be used as an input for future transactions.
By not explicitly associating transaction outputs with user identities, it offers a degree of privacy.</p></li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-harmonizing-the-two-systems-"><ins> Harmonizing the Two Systems: </ins><a href="#-harmonizing-the-two-systems-" class="hash-link" aria-label="Direct link to -harmonizing-the-two-systems-" title="Direct link to -harmonizing-the-two-systems-">​</a></h3><ol><li><p>Translation Layer</p><ul><li><p>Role: Interface between UTXO and account-based states.</p></li><li><p><em>UTXO-to-Account Adapter:</em> When UTXOs are spent, the adapter can translate these into the corresponding account balance modifications.
This could involve creating a temporary 'pseudo-account' that mirrors the
UTXO's attributes.</p></li><li><p><em>Account-to-UTXO Adapter:</em> When an account wishes to make a private transaction,
it would initiate a process converting a part of its balance to a UTXO, facilitating a privacy transaction.</p></li></ul></li><li><p>Unified Identity Management</p><ul><li><p>Role: Maintain a unified identity (or address) system that works across both state models,
allowing users to easily manage their public and private states without requiring separate identities.</p></li><li><p><em>Deterministic Wallets:</em> Use Hierarchical Deterministic (HD) wallets [<a href="https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14" target="_blank" rel="noopener noreferrer">3</a>,<a href="https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki" target="_blank" rel="noopener noreferrer">4</a>], enabling users to generate multiple addresses (both UTXO and account-based) from a single seed.
This ensures privacy while keeping management centralized for the user.</p></li></ul></li></ol><ol start="3"><li><p>State Commitments</p><ul><li><p>Role: Use cryptographic commitments to commit to the state of both models. This can help in efficiently validating cross-model transactions.</p></li><li><p><em>Verkle Trees:</em> Verkle Trees combine Vector Commitment and the KZG polynomial commitment scheme to produce a structure that's efficient in terms of both proofs and verification.
Verkle proofs are considerably small in size (less data to store and transmit), where Transaction and state verifications can be faster due to the smaller proof sizes and computational efficiencies.</p></li><li><p><em>Mimblewimble-style Aggregation</em> [<a href="https://github.com/mimblewimble/grin/blob/master/doc/intro.md" target="_blank" rel="noopener noreferrer">5</a>]: For UTXOs, techniques similar to those used in Mimblewimble can be used to aggregate transactions, keeping the state compact and enhancing privacy.</p></li></ul></li></ol><ol start="4"><li><p>Batch Processing &amp; Anonymity Sets</p><ul><li><p>Role: Group several UTXO-based private transactions into a single public account-based transaction.
This can provide a level of obfuscation and can make synchronization between the two models more efficient.</p></li><li><p><em>CoinJoin Technique</em> [<a href="https://en.bitcoin.it/wiki/CoinJoin" target="_blank" rel="noopener noreferrer">6</a>]: As seen in Bitcoin, multiple users can combine their UTXO transactions into one, enhancing privacy.</p></li><li><p><em>Tornado Cash Principle</em> [<a href="https://github.com/tornadocash/tornado-classic-ui" target="_blank" rel="noopener noreferrer">7</a>]: For account-based systems wanting to achieve privacy, methods like those used in Tornado Cash can be implemented,
providing zk-SNARKs-based private transactions.</p></li></ul></li><li><p>Event Hooks &amp; Smart Contracts</p><ul><li><p>Role: Implement event-driven mechanisms that trigger specific actions in one model based on events in the other.
For instance, a private transaction (UTXO-based) can trigger a corresponding public notification or event in the account-based model.</p></li><li><p><em>Conditional Execution:</em> Smart contracts could be set to execute based on events in the UTXO system. For instance,
a smart contract might release funds (account-based) once a specific UTXO is spent.</p></li><li><p><em>Privacy Smart Contracts:</em> Using zk-SNARKs or zk-STARKs to bring privacy to the smart contract layer,
allowing for private logic execution.</p></li></ul></li></ol><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-challenges-and-solutions-"><ins> Challenges and Solutions </ins><a href="#-challenges-and-solutions-" class="hash-link" aria-label="Direct link to -challenges-and-solutions-" title="Direct link to -challenges-and-solutions-">​</a></h3><ol><li><p>Synchronization Overhead</p><ul><li><p>Challenge: Combining two distinct transaction models creates an inherent synchronization challenge.</p></li><li><p>State Channels: By allowing transactions to be conducted off-chain between participants, state channels can alleviate synchronization stresses.
Only the final state needs to be settled on-chain, drastically reducing the amount of data and frequency of updates required.</p></li><li><p>Sidechains: These act as auxiliary chains to the main blockchain. Transactions can be processed on the sidechain and then periodically synced with the main chain.
This structure helps reduce the immediate load on the primary system.</p></li><li><p>Checkpointing: Introduce periodic checkpoints where the two systems' states are verified and harmonized.
This can ensure consistency without constant synchronization.</p></li></ul></li><li><p>Double Spending</p><ul><li><p>Challenge: With two models operating in tandem, there's an increased risk of double-spending attacks.</p></li><li><p>Multi-Signature Transactions: Implementing transactions that require signatures from both systems can prevent unauthorized movements.</p></li><li><p>Cross-Verification Mechanisms: Before finalizing a transaction, it undergoes verification in both UTXO and account-based systems.
If discrepancies arise, the transaction can be halted.</p></li><li><p>Timestamping: By attaching a timestamp to each transaction, it's possible to order them sequentially, making it easier to spot and prevent double spending.</p></li></ul></li><li><p>Complexity in User Experience</p><ul><li><p>Challenge: The dual model, while powerful, is inherently complex.</p></li><li><p>Abstracted User Interfaces: Design UIs that handle the complexity behind the scenes,
allowing users to make transactions without needing to understand the nuances of the dual model.</p></li><li><p>Guided Tutorials: Offer onboarding tutorials to acquaint users with the system's features,
especially emphasizing when and why they might choose one transaction type over the other.</p></li><li><p>Feedback Systems: Implement systems where users can provide feedback on any complexities or challenges they encounter.
This real-time feedback can be invaluable for iterative design improvements.</p></li></ul></li><li><p>Security</p><ul><li><p>Challenge: Merging two systems can introduce unforeseen vulnerabilities.</p></li><li><p>Threat Modeling: Regularly conduct threat modeling exercises to anticipate potential attack vectors,
especially those that might exploit the interaction between the two systems.</p></li><li><p>Layered Security Protocols: Beyond regular audits, introduce multiple layers of security checks.
Each layer can act as a fail-safe if a potential threat bypasses another.</p></li><li><p>Decentralized Watchtowers: These are third-party services that monitor the network for malicious activities.
If any suspicious activity is detected, they can take corrective measures or raise alerts.</p></li></ul></li><li><p>Gas &amp; Fee Management:</p><ul><li><p>Challenge: A dual model can lead to convoluted fee structures.</p></li><li><p>Dynamic Fee Adjustment: Implement algorithms that adjust fees based on network congestion and transaction type.
This can ensure fairness and prevent network abuse.</p></li><li><p>Fee Estimation Tools: Provide tools that can estimate fees before a transaction is initiated.
This helps users understand potential costs upfront.</p></li><li><p>Unified Gas Stations: Design platforms where users can purchase or allocate gas for both transaction types simultaneously,
simplifying the gas acquisition process.</p></li></ul></li></ol><p>By addressing these challenges head-on with a detailed and systematic approach, it's possible to unlock the full potential of a dual-architecture system,
combining the strengths of both UTXO and account-based models without their standalone limitations.</p><table><thead><tr><th>Aspect</th><th>Details</th></tr></thead><tbody><tr><td><strong>Harmony</strong></td><td>- <strong>Advanced VM Development:</strong> Design tailored for private smart contracts. - <strong>Leverage Established Architectures:</strong> Use WASM or RISC-V to harness their versatile and encompassing nature suitable for zero-knowledge applications. - <strong>Support for UTXO &amp; Account-Based Models:</strong> Enhance adaptability across various blockchain structures.</td></tr><tr><td><strong>Challenges</strong></td><td>- <strong>Adaptation Concerns:</strong> WASM and RISC-V weren't designed with zero-knowledge proofs as a primary focus, posing integration challenges. - <strong>Complexities with Newer Systems:</strong> Systems like (Super)Nova, STARKs, and Sangria are relatively nascent, adding another layer of intricacy to the integration. - <strong>Optimization Concerns:</strong> Ensuring that these systems are optimized for zero-knowledge proofs.</td></tr><tr><td><strong>Proposed Solutions</strong></td><td>- <strong>Integration of Nova:</strong> Consider Nova's proof system for its potential alignment with project goals. - <strong>Comprehensive Testing:</strong> Rigorously test and benchmark against alternatives like Halo2, Plonky, and Starky to validate choices. - <strong>Poseidon Recursion Technique:</strong> To conduct exhaustive performance tests, providing insights into each system's efficiency and scalability.</td></tr></tbody></table><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-2-virtual-machine-creation">Goal 2: Virtual Machine Creation<a href="#goal-2-virtual-machine-creation" class="hash-link" aria-label="Direct link to Goal 2: Virtual Machine Creation" title="Direct link to Goal 2: Virtual Machine Creation">​</a></h2><p>The second goal entails the creation of an advanced virtual machine by leveraging established mainstream instruction sets like WASM or RISC-V.
Alternatively, the objective involves pioneering a new, specialized instruction set meticulously optimized for Zero-Knowledge applications.</p><p>This initiative seeks to foster a versatile and efficient environment for executing computations within the privacy-focused context of the project.
Both WASM and RISC-V exhibit adaptability to both UTXO and account-based models due to their encompassing nature as general-purpose instruction set architectures.</p><p> <em>WASM</em>, operating as a low-level virtual machine, possesses the capacity to execute code derived from a myriad of high-level programming languages,
and boasts seamless integration across diverse blockchain platforms. </p><p> Meanwhile, <em>RISC-V</em> emerges as a versatile option, accommodating both models, and can be seamlessly integrated with secure enclaves like SGX or TEE,
elevating the levels of security and privacy. However, it is crucial to acknowledge that employing WASM or RISC-V might present challenges,
given their original design without specific emphasis on optimizing for Zero-Knowledge Proofs (ZKPs). </p><p> Further complexity arises with the consideration of more potent proof systems like (Super)Nova, STARKs, and Sangria, which,
while potentially addressing optimization concerns, necessitate extensive research and testing due to their relatively nascent status within the field.
This accentuates the need for a judicious balance between established options and innovative solutions in pursuit of an architecture harmoniously amalgamating privacy, security, and performance.</p><p>The ambition to build a powerful virtual machine tailored to zero-knowledge (ZK) applications is both commendable and intricate.
The combination of two renowned instruction sets, WASM and RISC-V, in tandem with ZK, is an innovation that could redefine privacy standards in blockchain.
Let's dissect the challenges and possibilities inherent in this goal:</p><ol><li><p>Established Mainstream Instruction Sets - WASM and RISC-V</p><ul><li><p>Strengths:</p><ul><li><p><em>WASM</em>: Rooted in its ability to execute diverse high-level language codes, its potential for cross-chain compatibility makes it a formidable contender.
Serving as a low-level virtual machine, its role in the blockchain realm is analogous to that of the Java Virtual Machine in the traditional computing landscape.</p></li><li><p><em>RISC-V</em>: This open-standard instruction set architecture has made waves due to its customizable nature.
Its adaptability to both UTXO and account-based structures coupled with its compatibility with trusted execution environments like SGX and TEE augments its appeal,
especially in domains that prioritize security and privacy.</p></li></ul></li><li><p>Challenges: Neither WASM nor RISC-V was primarily designed with ZKPs in mind. While they offer flexibility,
they might lack the necessary optimizations for ZK-centric tasks. Adjustments to these architectures might demand intensive R&amp;D efforts.</p></li></ul></li></ol><ol start="2"><li><p>Pioneering a New, Specialized Instruction Set</p><ul><li><p>Strengths: A bespoke instruction set can be meticulously designed from the ground up with ZK in focus,
potentially offering unmatched performance and optimizations tailored to the project's requirements.</p></li><li><p>Challenges: Crafting a new instruction set is a monumental task requiring vast resources, including expertise, time, and capital.
It would also need to garner community trust and support over time.</p></li></ul></li></ol><ol start="3"><li><p>Contemporary Proof Systems - (Super)Nova, STARKs, Sangria</p><ul><li><p>Strengths: These cutting-edge systems, being relatively new, might offer breakthrough cryptographic efficiencies that older systems lack: designed with modern challenges in mind,
they could potentially bridge the gap where WASM and RISC-V might falter in terms of ZKP optimization.</p></li><li><p>Challenges: Their nascent nature implies a dearth of exhaustive testing, peer reviews, and potentially limited community support.
The unknowns associated with these systems could introduce unforeseen vulnerabilities or complexities.
While they could offer optimizations that address challenges presented by WASM and RISC-V, their young status demands rigorous vetting and testing.</p></li></ul></li></ol><center><table><thead><tr><th align="center"></th><th align="center">Mainstream (WASM, RISC-V)</th><th align="center">ZK-optimized (New Instruction Set)</th></tr></thead><tbody><tr><td align="center">Existing Tooling</td><td align="center">YES</td><td align="center">NO</td></tr><tr><td align="center">Blockchain-focused</td><td align="center">NO</td><td align="center">YES</td></tr><tr><td align="center">Performant</td><td align="center">DEPENDS</td><td align="center">YES</td></tr></tbody></table></center><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-optimization-concerns-for-wasm-and-risc-v-"><ins> Optimization Concerns for WASM and RISC-V: </ins><a href="#-optimization-concerns-for-wasm-and-risc-v-" class="hash-link" aria-label="Direct link to -optimization-concerns-for-wasm-and-risc-v-" title="Direct link to -optimization-concerns-for-wasm-and-risc-v-">​</a></h3><ul><li><p><em>Cryptography Libraries</em>: ZKP applications rely heavily on specific cryptographic primitives. Neither WASM nor RISC-V natively supports all of these primitives.
Thus, a comprehensive library of cryptographic functions, optimized for these platforms, needs to be developed.</p></li><li><p><em>Parallel Execution</em>: Given the heavy computational demands of ZKPs, leveraging parallel processing capabilities can optimize the time taken.
Both WASM and RISC-V would need modifications to handle parallel execution of ZKP processes efficiently.</p></li><li><p><em>Memory Management</em>: ZKP computations can sometimes require significant amounts of memory, especially during the proof generation phase.
Fine-tuned memory management mechanisms are essential to prevent bottlenecks.</p></li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-emerging-zkp-optimized-systems-considerations-"><ins> Emerging ZKP Optimized Systems Considerations: </ins><a href="#-emerging-zkp-optimized-systems-considerations-" class="hash-link" aria-label="Direct link to -emerging-zkp-optimized-systems-considerations-" title="Direct link to -emerging-zkp-optimized-systems-considerations-">​</a></h3><ul><li><p><em>Proof Size</em>: Different systems generate proofs of varying sizes. A smaller proof size is preferable for blockchain applications to save on storage and bandwidth.
The trade-offs between proof size, computational efficiency, and security need to be balanced.</p></li><li><p><em>Universality</em>: Some systems can support any computational statement (universal), while others might be tailored to specific tasks.
A universal system can be more versatile for diverse applications on the blockchain.</p></li><li><p><em>Setup Requirements</em>: Certain ZKP systems, like zk-SNARKs, require a trusted setup, which can be a security concern.
Alternatives like zk-STARKs don't have this requirement but come with other trade-offs.</p></li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-strategies-for-integration-"><ins> Strategies for Integration: </ins><a href="#-strategies-for-integration-" class="hash-link" aria-label="Direct link to -strategies-for-integration-" title="Direct link to -strategies-for-integration-">​</a></h3><ul><li><p><em>Iterative Development</em>: Given the complexities, an iterative development approach can be beneficial.
Start with a basic integration of WASM or RISC-V for general tasks and gradually introduce specialized ZKP functionalities.</p></li><li><p><em>Benchmarking</em>: Establish benchmark tests specifically for ZKP operations. This will provide continuous feedback on the performance of the system as modifications are made, ensuring optimization.</p></li><li><p><em>External Audits &amp; Research</em>: Regular checks from cryptographic experts and collaboration with academic researchers can help in staying updated and ensuring secure implementations.</p></li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-3-proofs-creation-and-verification">Goal 3: Proofs Creation and Verification<a href="#goal-3-proofs-creation-and-verification" class="hash-link" aria-label="Direct link to Goal 3: Proofs Creation and Verification" title="Direct link to Goal 3: Proofs Creation and Verification">​</a></h2><p>The process of generating proofs for private state updates is vested in the hands of the user, aligning with our commitment to minimizing trust assumptions and enhancing privacy.
Concurrently, the responsibility of verifying these proofs and executing public functions within the virtual machine can be effectively delegated to an external prover,
a role that is incentivized to operate with utmost honesty and integrity. This intricate balance seeks to safeguard against information leakage,
preserving the confidentiality of private transactions. Integral to this mechanism is the establishment of a robust incentivization framework.</p><p>To ensure the prover’s steadfast commitment to performing tasks with honesty, we should introduce a mechanism that facilitates both rewards for sincere behavior and penalties for any deviation from the expected standards.
This two-pronged approach serves as a compelling deterrent against dishonest behavior and fosters an environment of accountability.
In addition to incentivization, a crucial consideration is the economic aspect of verification and execution.
The verification process has been intentionally designed to be more cost-effective than execution. </p><p>This strategic approach prevents potential malicious actors from exploiting the system by flooding it with spurious proofs, a scenario that could arise when the costs align favorably.
By maintaining a cost balance that favors verification, we bolster the system’s resilience against fraudulent activities while ensuring its efficiency.
In sum, our multifaceted approach endeavors to strike an intricate equilibrium between user-initiated proof creation, external verification, and incentivization.
This delicate interplay of mechanisms ensures a level of trustworthiness that hinges on transparency, accountability, and economic viability.</p><p>As a result, we are poised to cultivate an ecosystem where users’ privacy is preserved, incentives are aligned,
and the overall integrity of the system is fortified against potential adversarial actions. To achieve the goals of user-initiated proof creation,
external verification, incentivization, and cost-effective verification over execution, several options and mechanisms can be employed:</p><ol><li><p><strong>User-Initiated Proof Creation:</strong> Users are entrusted with the generation of proofs for private state updates, thus ensuring greater privacy and reducing trust dependencies.</p><ul><li><p>Challenges:</p><ul><li><p>Maintaining the quality and integrity of the proofs generated by users.</p></li><li><p>Ensuring that users have the tools and knowledge to produce valid proofs.</p></li></ul></li><li><p>Solutions:</p><ul><li><p>Offer extensive documentation, tutorials, and user-friendly tools to streamline the proof-generation process.</p></li><li><p>Implement checks at the verifier's end to ensure the quality of proofs.</p></li></ul></li></ul></li></ol><ol start="2"><li><p><strong>External Verification by Provers:</strong> An external prover verifies the proofs and executes public functions within the virtual machine.</p><ul><li><p>Challenges:</p><ul><li><p>Ensuring that the external prover acts honestly.</p></li><li><p>Avoiding centralized points of failure.</p></li></ul></li><li><p>Solutions:</p><ul><li><p>Adopt a decentralized verification approach, with multiple provers cross-verifying each other’s work.</p></li><li><p>Use reputation systems to rank provers based on their past performances, creating a trust hierarchy.</p></li></ul></li></ul></li><li><p><strong> Incentivization Framework:</strong> A system that rewards honesty and penalizes dishonest actions, ensuring provers' commitment to the task.</p><ul><li><p>Challenges:</p><ul><li><p>Determining the right balance of rewards and penalties.</p></li><li><p>Ensuring that the system cannot be gamed for undue advantage.</p></li></ul></li><li><p>Solutions<sup id="fnref-1-c890aa"><a href="#fn-1-c890aa" class="footnote-ref">1</a></sup>: </p><ul><li><p>Implement a dynamic reward system that adjusts based on network metrics and provers' performance.</p></li><li><p>Use a staking mechanism where provers need to lock up a certain amount of assets.
Honest behavior earns rewards, while dishonest behavior could lead to loss of staked assets.</p></li></ul></li></ul></li><li><p><strong>Economic Viability through Cost Dynamics:</strong> Making verification more cost-effective than execution to deter spamming and malicious attacks.</p><ul><li><p>Challenges: </p><ul><li><p>Setting the right cost metrics for both verification and execution.</p></li><li><p>Ensuring that genuine users aren’t priced out of the system.</p></li></ul></li><li><p>Solutions:</p><ul><li><p>Use a dynamic pricing model, adjusting costs in real-time based on network demand.</p></li><li><p>Implement gas-like mechanisms to differentiate operation costs and ensure fairness.</p></li></ul></li></ul></li><li><p><strong> Maintaining Trustworthiness:</strong> Create a system that's transparent, holds all actors accountable, and is economically sound.</p><ul><li><p>Challenges: </p><ul><li><p>Keeping the balance where users feel their privacy is intact, while provers feel incentivized.</p></li><li><p>Ensuring the system remains resilient against adversarial attacks.</p></li></ul></li><li><p>Solutions:</p><ul><li><p>Implement layered checks and balances.</p></li><li><p>Foster community involvement, allowing them to participate in decision-making, potentially through a decentralized autonomous organization (DAO).</p></li></ul></li></ul></li></ol><p>Each of these options can be combined or customized to suit the specific requirements of your project, striking a balance between user incentives,
cost dynamics, and verification integrity. A thoughtful combination of these mechanisms ensures that the system remains robust, resilient,
and conducive to the objectives of user-initiated proof creation, incentivized verification, and cost- effective validation.</p><center><table><thead><tr><th>Aspect</th><th>Details</th></tr></thead><tbody><tr><td><strong>Design Principle</strong></td><td>- <strong>User Responsibility:</strong> Generating proofs for private state updates. - <strong>External Prover:</strong> Delegated the task of verifying proofs and executing public VM functions.</td></tr><tr><td><strong>Trust &amp; Privacy</strong></td><td>- <strong>Minimized Trust Assumptions:</strong> Place proof generation in users' hands. - <strong>Enhanced Privacy:</strong> Ensure confidentiality of private transactions and prevent information leakage.</td></tr><tr><td><strong>Incentivization Framework</strong></td><td>- <strong>Rewards:</strong> Compensate honest behavior. - <strong>Penalties:</strong> Deter and penalize dishonest behavior.</td></tr><tr><td><strong>Economic Considerations</strong></td><td>- <strong>Verification vs. Execution:</strong> Make verification more cost-effective than execution to prevent spurious proofs flooding. - <strong>Cost Balance:</strong> Strengthen resilience against fraudulent activities and maintain efficiency.</td></tr><tr><td><strong>Outcome</strong></td><td>An ecosystem where: - Users' privacy is paramount. - Incentives are appropriately aligned. - The system is robust against adversarial actions.</td></tr></tbody></table></center><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-4-kernel-based-architecture-implementation">Goal 4: Kernel-based Architecture Implementation<a href="#goal-4-kernel-based-architecture-implementation" class="hash-link" aria-label="Direct link to Goal 4: Kernel-based Architecture Implementation" title="Direct link to Goal 4: Kernel-based Architecture Implementation">​</a></h2><p>This goal centers on the establishment of a kernel-based architecture, akin to the model observed in ZEXE, to facilitate the attestation of accurate private function executions.
This innovative approach employs recursion to construct a call stack, which is then validated through iterative recursive computations.
At its core, this technique harnesses a recursive Succinct Non-Interactive Argument of Knowledge (SNARK) mechanism, where each function call’s proof accumulates within the call stack.</p><p>The subsequent verification of this stack’s authenticity leverages recursive SNARK validation.
While this method offers robust verification of private function executions, it’s essential to acknowledge its associated intricacies.</p><p>The generation of SNARK proofs necessitates a substantial computational effort, which, in turn, may lead to elevated gas fees for users.
Moreover, the iterative recursive computations could potentially exhibit computational expansion as the depth of recursion increases.
This calls for a meticulous balance between the benefits of recursive verification and the resource implications it may entail.</p><p>In essence, Goal 4 embodies a pursuit of enhanced verification accuracy through a kernel-based architecture.
By weaving recursion and iterative recursive computations into the fabric of our system, we aim to establish a mechanism that accentuates the trustworthiness of private function executions,
while conscientiously navigating the computational demands that ensue.</p><p>To accomplish the goal of implementing a kernel-based architecture for recursive verification of private function executions,
several strategic steps and considerations can be undertaken: recursion handling and depth management.</p><ins> Recursion Handling </ins><ul><li><p><em>Call Stack Management:</em> </p><ul><li>Implement a data structure to manage the call stack, recording each recursive function call’s details, parameters, and state.</li></ul></li><li><p><em>Proof Accumulation: </em></p><ul><li><p>Design a mechanism to accumulate proof data for each function call within the call stack.
This includes cryptographic commitments, intermediate results, and cryptographic challenges.</p></li><li><p>Ensure that the accumulated proof data remains secure and tamper-resistant throughout the recursion process.</p></li></ul></li><li><p><em>Intermediary SNARK Proofs:</em></p><ul><li><p>Develop an intermediary SNARK proof for each function call’s correctness within the call stack.
This proof should demonstrate that the function executed correctly and produced expected outputs.</p></li><li><p>Ensure that the intermediary SNARK proof for each recursive call can be aggregated and verified together, maintaining the integrity of the entire call stack.</p></li></ul></li></ul><ins> Depth management </ins><ul><li><p><em>Depth Limitation:</em></p><ul><li><p>Define a threshold for the maximum allowable recursion depth based on the system’s computational capacity, gas limitations, and performance considerations.</p></li><li><p>Implement a mechanism to prevent further recursion beyond the defined depth limit, safeguarding against excessive computational growth.</p></li></ul></li><li><p><em>Graceful Degradation:</em></p><ul><li><p>Design a strategy for graceful degradation when the recursion depth approaches or reaches the defined limit.
This may involve transitioning to alternative execution modes or optimization techniques.</p></li><li><p>Communicate the degradation strategy to users and ensure that the system gracefully handles scenarios where recursion must be curtailed.</p></li></ul></li><li><p><em>Resource Monitoring:</em></p><ul><li>Develop tools to monitor resource consumption (such as gas usage and computational time) as recursion progresses.
Provide real-time feedback to users about the cost and impact of recursive execution.</li></ul></li><li><p><em>Dynamic Depth Adjustment:</em></p><ul><li><p>Consider implementing adaptive depth management that dynamically adjusts the recursion depth based on network conditions, transaction fees, and available resources.</p></li><li><p>Utilize algorithms to assess the optimal recursion depth for efficient execution while adhering to gas cost constraints.</p></li></ul></li><li><p><em>Fallback Mechanisms:</em></p><ul><li>Create fallback mechanisms that activate if the recursion depth limit is reached or if the system encounters resource constraints.
These mechanisms could involve alternative verification methods or delayed execution.</li></ul></li><li><p><em>User Notifications:</em></p><ul><li>Notify users when the recursion depth limit is approaching, enabling them to make informed decisions about the complexity of their transactions and potential resource usage.</li></ul></li></ul><p>Goal 4 underscores the project's ambition to integrate the merits of a kernel-based architecture with recursive verifications to bolster the reliability of private function executions.
While the approach promises robust outcomes, it's pivotal to maneuver through its intricacies with astute strategies, ensuring computational efficiency and economic viability.
By striking this balance, the architecture can realize its full potential in ensuring trustworthy and efficient private function executions.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-5-seamless-interaction-design">Goal 5: Seamless Interaction Design<a href="#goal-5-seamless-interaction-design" class="hash-link" aria-label="Direct link to Goal 5: Seamless Interaction Design" title="Direct link to Goal 5: Seamless Interaction Design">​</a></h2><p>Goal 5 revolves around the meticulous design of a seamless interaction between public and private states within the blockchain ecosystem.
This objective envisions achieving not only composability between contracts but also the harmonious integration of private and public functions.</p><p>A notable challenge in this endeavor lies in the intricate interplay between public and private states,
wherein the potential linkage of a private transaction to a public one raises concerns about unintended information leakage.</p><p>The essence of this goal entails crafting an architecture that facilitates the dynamic interaction of different states while ensuring that the privacy and confidentiality of private transactions remain unbreached.
This involves the formulation of mechanisms that enable secure composability between contracts, guaranteeing the integrity of interactions across different layers of functionality.</p><p>A key focus of this goal is to surmount the challenge of information leakage by implementing robust safeguards.
The solution involves devising strategies to mitigate the risk of revealing private transaction details when connected to corresponding public actions.
By creating a nuanced framework that com- partmentalizes private and public interactions, the architecture aims to uphold privacy while facilitating seamless interoperability.</p><p>Goal 5 encapsulates a multifaceted undertaking, calling for the creation of an intricate yet transparent framework that empowers users to confidently engage in both public and private functions,
without compromising the confidentiality of private transactions. The successful realization of this vision hinges on a delicate blend of architectural ingenuity, cryptographic sophistication, and user-centric design.</p><p>To achieve seamless interaction between public and private states, composability, and privacy preservation, a combination of solutions and approaches can be employed.
In the table below, a comprehensive list of solutions that address these objectives:</p><center><table><thead><tr><th align="center"><strong>Solution Category</strong></th><th align="center"><strong>Description</strong></th></tr></thead><tbody><tr><td align="center"><strong>Layer 2 Solutions</strong></td><td align="center">Employ zk-Rollups, Optimistic Rollups, and state channels to handle private interactions off-chain and settle them on-chain periodically. Boost scalability and cut transaction costs.</td></tr><tr><td align="center"><strong>Intermediary Smart Contracts</strong></td><td align="center">Craft smart contracts as intermediaries for secure public-private interactions. Use these to manage data exchange confidentially.</td></tr><tr><td align="center"><strong>Decentralized Identity &amp; Pseudonymity</strong></td><td align="center">Implement decentralized identity systems for pseudonymous interactions. Validate identity using cryptographic proofs.</td></tr><tr><td align="center"><strong>Confidential Sidechains &amp; Cross-Chain</strong></td><td align="center">Set up confidential sidechains and employ cross-chain protocols to ensure private and composability across blockchains.</td></tr><tr><td align="center"><strong>Temporal Data Structures</strong></td><td align="center">Create chronological data structures for secure interactions. Utilize cryptographic methods for data integrity and privacy.</td></tr><tr><td align="center"><strong>Homomorphic Encryption &amp; MPC</strong></td><td align="center">Apply homomorphic encryption and MPC for computations on encrypted data and interactions between state layers.</td></tr><tr><td align="center"><strong>Commit-Reveal Schemes</strong></td><td align="center">Introduce commit-reveal mechanisms for private transactions, revealing data only post necessary public actions.</td></tr><tr><td align="center"><strong>Auditability &amp; Verifiability</strong></td><td align="center">Use on-chain tools for auditing and verifying interactions. Utilize cryptographic commitments for third-party validation.</td></tr><tr><td align="center"><strong>Data Fragmentation &amp; Sharding</strong></td><td align="center">Fragment data across shards for private interactions and curtailed data exposure. Bridge shards securely with cryptography.</td></tr><tr><td align="center"><strong>Ring Signatures &amp; CoinJoin</strong></td><td align="center">Incorporate ring signatures and CoinJoin protocols to mask transaction details and mix transactions collaboratively.</td></tr></tbody></table></center><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-6-integration-of-defi-protocols-with-a-privacy-preserving-framework">Goal 6: Integration of DeFi Protocols with a Privacy-Preserving Framework<a href="#goal-6-integration-of-defi-protocols-with-a-privacy-preserving-framework" class="hash-link" aria-label="Direct link to Goal 6: Integration of DeFi Protocols with a Privacy-Preserving Framework" title="Direct link to Goal 6: Integration of DeFi Protocols with a Privacy-Preserving Framework">​</a></h2><p>The primary aim of Goal 6 is to weave key DeFi protocols, such as AMMs and staking, into a user-centric environment that accentuates privacy.
This endeavor comes with inherent challenges, especially considering the heterogeneity of existing DeFi protocols, predominantly built on Ethereum.
These variations in programming languages and VMs exacerbate the quest for interoperability. Furthermore, the success and functionality of DeFi protocols is closely tied to liquidity,
which in turn is influenced by user engagement and the amount of funds locked into the system.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-strategic-roadmap-for-goal-6-"><ins> Strategic Roadmap for Goal 6 </ins><a href="#-strategic-roadmap-for-goal-6-" class="hash-link" aria-label="Direct link to -strategic-roadmap-for-goal-6-" title="Direct link to -strategic-roadmap-for-goal-6-">​</a></h2><ol><li><p><em><strong> Pioneering Privacy-Centric DeFi Models: </strong></em> Initiate the development of AMMs and staking solutions that are inherently protective of users' transactional privacy and identity.</p></li><li><p><em><strong> Specialized Smart Contracts with Privacy: </strong></em> Architect distinct smart contracts infused with privacy elements, setting the stage for secure user interactions within this new, confidential DeFi landscape.</p></li><li><p><em><strong> Optimized User Interfaces: </strong></em> Craft interfaces that resonate with user needs, simplifying the journey through the private DeFi space without compromising on security.</p></li><li><p><em><strong> Tackling Interoperability: </strong></em> </p><ul><li><p>Deploy advanced bridge technologies and middleware tools to foster efficient data exchanges and guarantee operational harmony across a spectrum of programming paradigms and virtual environments.</p></li><li><p>Design and enforce universal communication guidelines that bridge the privacy-centric DeFi entities with the larger DeFi world seamlessly.</p></li></ul></li></ol><ol start="5"><li><p><em><strong> Enhancing and Sustaining Liquidity: </strong></em></p><ul><li><p>Unveil innovative liquidity stimuli and yield farming incentives, compelling users to infuse liquidity into the private DeFi space.</p></li><li><p>Incorporate adaptive liquidity frameworks that continually adjust based on the evolving market demands, ensuring consistent liquidity.</p></li><li><p>Forge robust alliances with other DeFi stalwarts, jointly maximizing liquidity stores and honing sustainable token distribution strategies.</p></li></ul></li></ol><ol start="6"><li><em><strong> Amplifying Community Engagement:</strong></em> Design and roll out enticing incentive schemes to rally users behind privacy-focused AMMs and staking systems,
thereby nurturing a vibrant, privacy-advocating DeFi community.</li></ol><p>Through the integration of these approaches, we aim to achieve Goal 6, providing users with a privacy-focused platform for engaging effortlessly in core DeFi functions such as AMMs and staking,
all while effectively overcoming the obstacles related to interoperability and liquidity concerns.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary-of-the-architecture">Summary of the Architecture<a href="#summary-of-the-architecture" class="hash-link" aria-label="Direct link to Summary of the Architecture" title="Direct link to Summary of the Architecture">​</a></h2><p>In our quest to optimize privacy, we're proposing a Zero-Knowledge Virtual Machine (Zkvm) that harnesses the power of Zero-Knowledge Proofs (ZKPs).
These proofs ensure that while private state data remains undisclosed, public state transitions can still be carried out and subsequently verified by third parties.
This blend of public and private state is envisaged to be achieved through a state tree representing the public state, while the encrypted state leaves stand for the private state.
Each user's private state indicates validity through the absence of a corresponding nullifier.
A nullifier is a unique cryptographic value generated in privacy-preserving blockchain transactions to prevent double-spending,
ensuring that each private transaction is spent only once without revealing its details.</p><p>Private functions' execution mandates users to offer a proof underscoring the accurate execution of all encapsulated private calls.
For validating a singular private function call, we're leaning into the kernel-based model inspired by the ZEXE protocol.
Defined as kernel circuits, these functions validate the correct execution of each private function call.
Due to their recursive circuit structure, a succession of private function calls can be executed by calculating proofs in an iterative manner.
Execution-relevant data, like private and public call stacks and additions to the state tree, are incorporated as public inputs.</p><p>Our method integrates the verification keys for these functions within a merkle tree. Here's the innovation: a user's ZKP showcases the existence of the verification key in this tree, yet keeps the executed function concealed.
The unique function identifier can be presented as the verification key, with all contracts merkleized for hiding functionalities.</p><p>We suggest a nuanced shift from the ZEXE protocol's identity function, which crafts an identity for smart contracts delineating its behavior, access timeframes, and other functionalities.
Instead of the ZEXE protocol's structure, our approach pivots to a method anchored in the
security of a secret combined with the uniqueness from hashing with the contract address.
The underlying rationale is straightforward: the sender, equipped with a unique nonce and salt for the transaction, hashes the secret, payload, nonce, and salt.
This result is then hashed with the contract address for the final value. The hash function's unidirectional nature ensures that the input cannot be deduced easily from its output.
A specific concern, however, is the potential repetition of secret and payload values across transactions, which could jeopardize privacy.
Yet, by embedding the function's hash within the hash of the contract address, users can validate a specific function's execution without divulging the function, navigating this limitation.</p><p>Alternative routes do exist: We could employ signature schemes like ECDSA, focusing on uniqueness and authenticity, albeit at the cost of complex key management.
Fully Homomorphic Encryption (FHE) offers another pathway, enabling function execution on encrypted data, or Multi-Party Computation (MPC) which guarantees non-disclosure of function or inputs.
Yet, integrating ZKPs with either FHE or MPC presents a challenge. Combining cryptographic functions like SHA-3 and BLAKE2 can also bolster security and uniqueness.
It's imperative to entertain these alternatives, especially when hashing might not serve large input/output functions effectively or might fall short in guaranteeing uniqueness.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="current-state">Current State<a href="#current-state" class="hash-link" aria-label="Direct link to Current State" title="Direct link to Current State">​</a></h2><p>Our aim is to revolutionize the privacy and security paradigms through Nescience.
As we strive to set milestones and achieve groundbreaking advancements,
our current focus narrows onto the realization of Goal 2 and Goal 3.</p><p>Our endeavors to build a powerful virtual machine tailored for Zero-Knowledge applications have led us down the path of rigorous exploration and testing.
We believe that integrating the right proof system is pivotal to our project's success, which brings us to Nova [<a href="https://eprint.iacr.org/2021/370" target="_blank" rel="noopener noreferrer">8</a>].
In our project journey, we have opted to integrate the Nova proof system, recognizing its potential alignment with our overarching goals.
However, as part of our meticulous approach to innovation and optimization, we acknowledge the need to thoroughly examine Nova’s performance capabilities,
particularly due to its status as a pioneering and relatively unexplored proof system.</p><p>This critical evaluation entails a comprehensive process of benchmarking and comparative analysis <a href="https://github.com/vacp2p/zk-explorations" target="_blank" rel="noopener noreferrer">[9]</a>,
pitting Nova against other prominent proof systems in the field, including Halo2 [<a href="https://electriccoin.co/blog/explaining-halo-2/" target="_blank" rel="noopener noreferrer">10</a>],
Plonky2 [<a href="https://polygon.technology/blog/introducing-plonky2" target="_blank" rel="noopener noreferrer">11</a>], and Starky [<a href="https://eprint.iacr.org/2021/582" target="_blank" rel="noopener noreferrer">12</a>].
This ongoing and methodical initiative is designed to ensure a fair and impartial assessment, enabling us to draw meaningful conclusions about Nova’s strengths and limitations in relation to its counterparts.
By leveraging the Poseidon recursion technique, we are poised to conduct an exhaustive performance test that delves into intricate details.
Through this testing framework, we aim to discern whether Nova possesses the potential to outshine its contemporaries in terms of efficiency, scalability, and overall performance.
The outcome of this rigorous evaluation will be pivotal in shaping our strategic decisions moving forward.
Armed with a comprehensive understanding of Nova’s performance metrics vis-à-vis other proof systems,
we can confidently chart a course that maximizes the benefits of our project’s optimization efforts.</p><p>Moreover, as we ambitiously pursue the establishment of a robust mechanism for proof creation and verification, our focus remains resolute on preserving user privacy,
incentivizing honest behaviour, and ensuring the cost-effective verification of transactions.
At the heart of this endeavor is our drive to empower users by allowing them the autonomy of generating proofs for private state updates,
thereby reducing dependencies and enhancing privacy.
We would like to actively work on providing comprehensive documentation, user-friendly tools,
and tutorials to aid users in this intricate process.</p><p>Parallelly, we're looking into decentralized verification processes, harnessing the strength of multiple external provers that cross-verify each other's work.
Our commitment is further cemented by our efforts to introduce a dynamic reward system that adjusts based on network metrics and prover performance.
This intricate balance, while challenging, aims to fortify our system against potential adversarial actions, aligning incentives, and preserving the overall integrity of the project.</p><h1>References</h1><p>[1]<!-- --> Nakamoto, S. (2008). Bitcoin: A Peer-to-Peer Electronic Cash System. Retrieved from <a href="https://bitcoin.org/bitcoin.pdf" target="_blank" rel="noopener noreferrer">https://bitcoin.org/bitcoin.pdf</a></p><p>[2]<!-- --> Sanchez, F. (2021). Cardano’s Extended UTXO accounting model. Retrived from <a href="https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/" target="_blank" rel="noopener noreferrer">https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/</a></p><p>[3]<!-- --> Morgan, D. (2020). HD Wallets Explained: From High Level to Nuts and Bolts. Retrieved from <a href="https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14" target="_blank" rel="noopener noreferrer">https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14</a></p><p>[4]<!-- --> Wuille, P. (012). Bitcoin Improvement Proposal (BIP) 44. Retrieved from <a href="https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki" target="_blank" rel="noopener noreferrer">https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki</a></p><p>[5]<!-- --> Jedusor, T. (2020). Introduction to Mimblewimble and Grin. Retrieved from <a href="https://github.com/mimblewimble/grin/blob/master/doc/intro.md" target="_blank" rel="noopener noreferrer">https://github.com/mimblewimble/grin/blob/master/doc/intro.md</a></p><p>[6]<!-- --> Bitcoin's official wiki overview of the CoinJoin method. Retrieved from <a href="https://en.bitcoin.it/wiki/CoinJoin" target="_blank" rel="noopener noreferrer">https://en.bitcoin.it/wiki/CoinJoin</a></p><p>[7]<!-- --> TornadoCash official Github page. Retrieved from <a href="https://github.com/tornadocash/tornado-classic-ui" target="_blank" rel="noopener noreferrer">https://github.com/tornadocash/tornado-classic-ui</a></p><p>[8]<!-- --> Kothapalli, A., Setty, S., Tzialla, I. (2021). Nova: Recursive Zero-Knowledge Arguments from Folding Schemes. Retrieved from <a href="https://eprint.iacr.org/2021/370" target="_blank" rel="noopener noreferrer">https://eprint.iacr.org/2021/370</a></p><p>[9]<!-- --> ZKvm Github page. Retrieved from <a href="https://github.com/vacp2p/zk-explorations" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/zk-explorations</a></p><p>[10]<!-- --> Electric Coin Company (2020). Explaining Halo 2. Retrieved from <a href="https://electriccoin.co/blog/explaining-halo-2/" target="_blank" rel="noopener noreferrer">https://electriccoin.co/blog/explaining-halo-2/</a></p><p>[11]<!-- --> Polygon Labs (2022). Introducing Plonky2. Retrieved from <a href="https://polygon.technology/blog/introducing-plonky2" target="_blank" rel="noopener noreferrer">https://polygon.technology/blog/introducing-plonky2</a></p><p>[12]<!-- --> StarkWare (2021). ethSTARK Documentation. Retrieved from <a href="https://eprint.iacr.org/2021/582" target="_blank" rel="noopener noreferrer">https://eprint.iacr.org/2021/582</a></p><div class="footnotes"><hr><ol><li id="fn-1-c890aa"><p>Incentive Mechanisms:</p><ul><li><p>Token Rewards: Design a token-based reward system where honest provers are compensated with tokens for their verification services.
This incentivizes participation and encourages integrity.</p></li><li><p>Staking and Slashing: Introduce a staking mechanism where provers deposit tokens as collateral.
Dishonest behavior results in slashing (partial or complete loss) of the staked tokens, while honest actions are rewarded.</p></li><li><p>Proof of Work/Proof of Stake: Implement a proof-of-work or proof-of- stake consensus mechanism for verification,
aligning incentives with the blockchain’s broader consensus mechanism.</p></li></ul><p><a href="#fnref-1-c890aa" class="footnote-backref">↩</a></p></li></ol></div>]]></content:encoded>
</item>
<item>
<title><![CDATA[Device Pairing in Js-waku and Go-waku]]></title>
<link>https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku</link>
<guid>https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku</guid>
<pubDate>Mon, 24 Apr 2023 12:00:00 GMT</pubDate>
<description><![CDATA[Device pairing and secure message exchange using Waku and noise protocol.]]></description>
<content:encoded><![CDATA[<p>Device pairing and secure message exchange using Waku and noise protocol.</p><p>As the world becomes increasingly connected through the internet, the need for secure and reliable communication becomes paramount. In <a href="https://vac.dev/wakuv2-noise" target="_blank" rel="noopener noreferrer">this article</a> it is described how the Noise protocol can be used as a key-exchange mechanism for Waku.</p><p>Recently, this feature was introduced in <a href="https://github.com/waku-org/js-noise" target="_blank" rel="noopener noreferrer">js-waku</a> and <a href="https://github.com/waku-org/go-waku" target="_blank" rel="noopener noreferrer">go-waku</a>, providing a simple API for developers to implement secure communication protocols using the Noise Protocol framework. These open-source libraries provide a solid foundation for building secure and decentralized applications that prioritize data privacy and security.</p><p>This functionality is designed to be simple and easy to use, even for developers who are not experts in cryptography. The library offers a clear and concise API that abstracts away the complexity of the Noise Protocol framework and provides an straightforward interface for developers to use. Using this, developers can effortlessly implement secure communication protocols on top of their JavaScript and Go applications, without having to worry about the low-level details of cryptography.</p><p>One of the key benefits of using Noise is that it provides end-to-end encryption, which means that the communication between two parties is encrypted from start to finish. This is essential for ensuring the security and privacy of sensitive information</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="device-pairing">Device Pairing<a href="#device-pairing" class="hash-link" aria-label="Direct link to Device Pairing" title="Direct link to Device Pairing">​</a></h3><p>In today's digital world, device pairing has become an integral part of our lives. Whether it's connecting our smartphones with other computers or web applications, the need for secure device pairing has become more crucial than ever. With the increasing threat of cyber-attacks and data breaches, it's essential to implement secure protocols for device pairing to ensure data privacy and prevent unauthorized access.</p><p>To demonstrate how device pairing can be achieved using Waku and Noise, we have examples available at <a href="https://examples.waku.org/noise-js/" target="_blank" rel="noopener noreferrer">https://examples.waku.org/noise-js/</a>. You can try pairing different devices, such as mobile and desktop, via a web application. This can be done by scanning a QR code or opening a URL that contains the necessary data for a secure handshake.</p><p>The process works as follows:</p><p>Actors:</p><ul><li>Alice the initiator</li><li>Bob the responder</li></ul><ol><li>The first step in achieving secure device pairing using Noise and Waku is for Bob generate the pairing information which could be transmitted out-of-band. For this, Bob opens <a href="https://examples.waku.org/noise-js/" target="_blank" rel="noopener noreferrer">https://examples.waku.org/noise-js/</a> and a QR code is generated, containing the data required to do the handshake. This pairing QR code is timeboxed, meaning that after 2 minutes, it will become invalid and a new QR code must be generated</li><li>Alice scans the QR code using a mobile phone. This will open the app with the QR code parameters initiating the handshake process which is described in <a href="https://rfc.vac.dev/spec/43/#protocol-flow" target="_blank" rel="noopener noreferrer">43/WAKU2-DEVICE-PAIRING</a>. These messages are exchanged between two devices over Waku to establish a secure connection. The handshake messages consist of three main parts: the initiator's message, the responder's message, and the final message, which are exchanged to establish a secure connection. While using js-noise, the developer is abstracted of this process, since the messaging happens automatically depending on the actions performed by the actors in the pairing process.</li><li>Both Alice and Bob will be asked to verify each other's identity. This is done by confirming if an 8-digits authorization code match in both devices. If both actors confirm that the authorization code is valid, the handshake concludes succesfully</li><li>Alice and Bob receive a set of shared keys that can be used to start exchanging encrypted messages. The shared secret keys generated during the handshake process are used to encrypt and decrypt messages sent between the devices. This ensures that the messages exchanged between the devices are secure and cannot be intercepted or modified by an attacker.</li></ol><p>The above example demonstrates device pairing using js-waku. Additionally, You can also try building and experimenting with other noise implementations like nwaku, or go-waku, with an example available at <a href="https://github.com/waku-org/go-waku/tree/master/examples/noise" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/go-waku/tree/master/examples/noise</a> in which the same flow described before is done with Bob (the receiver) using go-waku instead of js-waku.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h3><p>With its easy to use API built on top of the Noise Protocol framework and the LibP2P networking stack, if you are a developer looking to implement secure messaging in their applications that are both decentralized and censorship resistant, Waku is definitely an excellent choice worth checking out!</p><p>Waku is also Open source with a MIT and APACHEv2 licenses, which means that developers are encouraged to contribute code, report bugs, and suggest improvements to make it even better.</p><p>Don't hesitate to try the live example at <a href="https://examples.waku.org/noise-js" target="_blank" rel="noopener noreferrer">https://examples.waku.org/noise-js</a> and build your own webapp using <a href="https://github.com/waku-org/js-noise" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/js-noise</a>, <a href="https://github.com/waku-org/js-waku" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/js-waku</a> and <a href="https://github.com/waku-org/go-waku" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/go-waku</a>. This will give you a hands-on experience of implementing secure communication protocols using the Noise Protocol framework in a practical setting. Happy coding!</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h3><ul><li><a href="https://vac.dev/wakuv2-noise" target="_blank" rel="noopener noreferrer">Noise handshakes as key-exchange mechanism for Waku</a></li><li><a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">Noise Protocols for Waku Payload Encryption</a></li><li><a href="https://rfc.vac.dev/spec/37/" target="_blank" rel="noopener noreferrer">Session Management for Waku Noise</a></li><li><a href="https://rfc.vac.dev/spec/43/" target="_blank" rel="noopener noreferrer">Device pairing and secure transfers with Noise</a></li><li><a href="https://github.com/waku-org/go-waku/tree/master/examples/noise" target="_blank" rel="noopener noreferrer">go-waku Noise's example</a></li><li><a href="https://github.com/waku-org/js-waku-examples/tree/master/examples/noise-js" target="_blank" rel="noopener noreferrer">js-waku Noise's example</a></li><li><a href="https://github.com/waku-org/js-noise/" target="_blank" rel="noopener noreferrer">js-noise</a></li><li><a href="https://github.com/waku-org/js-noise/" target="_blank" rel="noopener noreferrer">go-noise</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[The Future of Waku Network: Scaling, Incentivization, and Heterogeneity]]></title>
<link>https://vac.dev/rlog/future-of-waku-network</link>
<guid>https://vac.dev/rlog/future-of-waku-network</guid>
<pubDate>Mon, 03 Apr 2023 00:00:00 GMT</pubDate>
<description><![CDATA[Learn how the Waku Network is evolving through scaling, incentivization, and diverse ecosystem development and what the future might look like.]]></description>
<content:encoded><![CDATA[<p>Learn how the Waku Network is evolving through scaling, incentivization, and diverse ecosystem development and what the future might look like.</p><p>Waku is preparing for production with a focus on the Status Communities use case. In this blog post, we will provide an
overview of recent discussions and research outputs, aiming to give you a better understanding of how the Waku network
may look like in terms of scaling and incentivization.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dos-mitigation-for-status-communities">DOS Mitigation for Status Communities<a href="#dos-mitigation-for-status-communities" class="hash-link" aria-label="Direct link to DOS Mitigation for Status Communities" title="Direct link to DOS Mitigation for Status Communities">​</a></h2><p>Waku is actively exploring DOS mitigation mechanisms suitable for Status Communities. While RLN
(Rate Limiting Nullifiers) remains the go-to DOS protection solution due to its privacy-preserving and
censorship-resistant properties, there is still more work to be done. We are excited to collaborate with PSE
(Privacy &amp; Scaling Explorations) in this endeavor. Learn more about their latest progress in this <a href="https://twitter.com/CPerezz19/status/1640373940634939394?s=20" target="_blank" rel="noopener noreferrer">tweet</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-heterogeneous-waku-network">A Heterogeneous Waku Network<a href="#a-heterogeneous-waku-network" class="hash-link" aria-label="Direct link to A Heterogeneous Waku Network" title="Direct link to A Heterogeneous Waku Network">​</a></h2><p>As we noted in a previous <a href="https://forum.vac.dev/t/waku-payment-models/166/3" target="_blank" rel="noopener noreferrer">forum post</a>, Waku's protocol
incentivization model needs to be flexible to accommodate various business models. Flexibility ensures that projects
can choose how they want to use Waku based on their specific needs.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="reversing-the-incentivization-question">Reversing the Incentivization Question<a href="#reversing-the-incentivization-question" class="hash-link" aria-label="Direct link to Reversing the Incentivization Question" title="Direct link to Reversing the Incentivization Question">​</a></h3><p>Traditionally, the question of incentivization revolves around how to incentivize operators to run nodes. We'd like to
reframe the question and instead ask, "How do we pay for the infrastructure?"</p><p>Waku does not intend to offer a free lunch.
Ethereum's infrastructure is supported by transaction fees and inflation, with validators receiving rewards from both sources.
However, this model does not suit a communication network like Waku.
Users and platforms would not want to pay for every single message they send. Additionally, Waku aims to support instant
ephemeral messages that do not require consensus or long-term storage.</p><p>Projects that use Waku to enable user interactions, whether for chat messages, gaming, private DeFi, notifications, or
inter-wallet communication, may have different value extraction models. Some users might provide services for the
project and expect to receive value by running nodes, while others may pay for the product or run infrastructure to
contribute back. Waku aims to support each of these use cases, which means there will be various ways to "pay for the
infrastructure."</p><p>In <a href="https://vac.dev/building-privacy-protecting-infrastructure" target="_blank" rel="noopener noreferrer">his talk</a>, Oskar addressed two strategies: RLN and service credentials.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-and-service-credentials">RLN and Service Credentials<a href="#rln-and-service-credentials" class="hash-link" aria-label="Direct link to RLN and Service Credentials" title="Direct link to RLN and Service Credentials">​</a></h3><p>RLN enables DOS protection across the network in a privacy-preserving and permission-less manner: stake in a contract,
and you can send messages.</p><p>Service credentials establish a customer-provider relationship. Users might pay to have messages they are interested in
stored and served by a provider. Alternatively, a community owner could pay a service provider to host their community.</p><p>Providers could offer trial or limited free services to Waku users, similar to Slack or Discord. Once a trial is expired or outgrown,
a community owner could pay for more storage or bandwidth, similar to Slack's model.
Alternatively, individual users could contribute financially, akin to Discord's Server Boost, or by sharing their own
resources with their community.</p><p>We anticipate witnessing various scenarios across the spectrum: from users sharing resources to users paying for access to the network and everything in between.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-network-ethereum-or-cosmos">Waku Network: Ethereum or Cosmos?<a href="#waku-network-ethereum-or-cosmos" class="hash-link" aria-label="Direct link to Waku Network: Ethereum or Cosmos?" title="Direct link to Waku Network: Ethereum or Cosmos?">​</a></h2><p>Another perspective is to consider whether the Waku network will resemble Ethereum or Cosmos.</p><p>For those not familiar with the difference between both, in a very concise manner:</p><ul><li>Ethereum is a set of protocols and software that are designed to operate on one common network and infrastructure</li><li>Cosmos is a set of protocols and software (SDKs) designed to be deployed in separate yet interoperable networks and infrastructures by third parties</li></ul><p>We want Waku to be decentralized to provide censorship resistance and privacy-preserving communication.
If each application has to deploy its own network, we will not achieve this goal.
Therefore, we aim Waku to be not only an open source set of protocols, but also a shared infrastructure that anyone can leverage to build applications on top, with some guarantees in terms of decentralization and anonymity.
This approach is closer in spirit to Ethereum than Cosmos.
Do note that, similarly to Ethereum, anyone is free to take Waku software and protocols and deploy their own network.</p><p>Yet, because of the difference in the fee model, the Waku Network is unlikely to be as unified as Ethereum's.
We currently assume that there will be separate gossipsub networks with different funding models.
Since there is no consensus on Waku, each individual operator can decide which network to support, enabling Waku to maintain its permission-less property.</p><p>Most likely, the Waku network will be heterogeneous, and node operators will choose the incentivization model they prefer.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="scalability-and-discovery-protocols">Scalability and Discovery Protocols<a href="#scalability-and-discovery-protocols" class="hash-link" aria-label="Direct link to Scalability and Discovery Protocols" title="Direct link to Scalability and Discovery Protocols">​</a></h2><p>To enable scalability, the flow of messages in the Waku network will be divided in shards,
so that not every node has to forward every message of the whole network.
Discovery protocols will facilitate users connecting to the right nodes to receive the messages they are interested in.</p><p>Different shards could be subject to a variety of rate limiting techniques (globally, targeted to that shard or something in-between).</p><p>Marketplace protocols may also be developed to help operators understand how they can best support the network and where
their resources are most needed. However, we are still far from establishing or even assert that such a marketplace will be needed.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="open-problems">Open Problems<a href="#open-problems" class="hash-link" aria-label="Direct link to Open Problems" title="Direct link to Open Problems">​</a></h2><p>Splitting traffic between shards reduces bandwidth consumption for every Waku Relay node.
This improvement increases the likelihood that users with home connections can participate and contribute to the gossipsub network without encountering issues.</p><p>However, it does not cap traffic.
There are still open problems regarding how to guarantee that someone can use Waku with lower Internet bandwidth or run critical services, such as a validation node, on the same connection.</p><p>We have several ongoing initiatives:</p><ul><li>Analyzing the Status Community protocol to confirm efficient usage of Waku <a href="https://github.com/vacp2p/research/issues/177" target="_blank" rel="noopener noreferrer">[4]</a></li><li>Simulating the Waku Network to measure actual bandwidth usage <a href="https://github.com/waku-org/pm/issues/2" target="_blank" rel="noopener noreferrer">[5]</a></li><li>Segregating chat messages from control and media messages <a href="https://rfc.vac.dev/spec/57/#control-message-shards" target="_blank" rel="noopener noreferrer">[6]</a></li></ul><p>The final solution will likely be a combination of protocols that reduce bandwidth usage or mitigate the risk of DOS attacks, providing flexibility for users and platforms to enable the best experience.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-evolving-waku-network">The Evolving Waku Network<a href="#the-evolving-waku-network" class="hash-link" aria-label="Direct link to The Evolving Waku Network" title="Direct link to The Evolving Waku Network">​</a></h2><p>The definition of the "Waku Network" will likely change over time. In the near future, it will transition from a single
gossipsub network to a sharded set of networks unified by a common discovery layer. This change will promote scalability
and allow various payment models to coexist within the Waku ecosystem.</p><p>In conclusion, the future of Waku Network entails growth, incentivization, and heterogeneity while steadfastly
maintaining its core principles. As Waku continues to evolve, we expect it to accommodate a diverse range of use cases
and business models, all while preserving privacy, resisting censorship, avoiding surveillance, and remaining accessible
to devices with limited resources.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ol><li><a href="https://rfc.vac.dev/spec/51/" target="_blank" rel="noopener noreferrer">51/WAKU2-RELAY-SHARDING</a></li><li><a href="https://rfc.vac.dev/spec/57/" target="_blank" rel="noopener noreferrer">57/STATUS-Simple-Scaling</a></li><li><a href="https://rfc.vac.dev/spec/58/" target="_blank" rel="noopener noreferrer">58/RLN-V2</a></li><li><a href="https://github.com/vacp2p/research/issues/177" target="_blank" rel="noopener noreferrer">Scaling Status Communities: Potential Problems</a></li><li><a href="https://github.com/waku-org/pm/issues/2" target="_blank" rel="noopener noreferrer">Waku Network Testing</a></li><li><a href="https://rfc.vac.dev/spec/57/#control-message-shards" target="_blank" rel="noopener noreferrer">51/WAKU2-RELAY-SHARDING: Control Message Shards</a></li></ol>]]></content:encoded>
</item>
<item>
<title><![CDATA[Waku for All Decentralized Applications and Infrastructures]]></title>
<link>https://vac.dev/rlog/waku-for-all</link>
<guid>https://vac.dev/rlog/waku-for-all</guid>
<pubDate>Tue, 08 Nov 2022 00:00:00 GMT</pubDate>
<description><![CDATA[Waku is an open communication protocol and network. Decentralized apps and infrastructure can use Waku for their]]></description>
<content:encoded><![CDATA[<p>Waku is an open communication protocol and network. Decentralized apps and infrastructure can use Waku for their
communication needs. It is designed to enable dApps and decentralized infrastructure projects to have secure, private,
scalable communication. Waku is available in several languages and platforms, from Web to mobile to desktop to cloud.
Initially, We pushed Waku adoption to the Web ecosystem, we learned that Waku is usable in a variety of complex applications
and infrastructure projects. We have prioritized our effort to make Waku usable on various platforms and environments.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background">Background<a href="#background" class="hash-link" aria-label="Direct link to Background" title="Direct link to Background">​</a></h2><p>We have built Waku to be the communication layer for Web3. Waku is a collection of protocols to chose from for your
messaging needs. It enables secure, censorship-resistant, privacy-preserving, spam-protected communication for its user.
It is designed to run on any device, from mobile to the cloud.</p><p>Waku is available on many systems and environments and used by several applications and SDKs for decentralized communications.</p><p>This involved research efforts in various domains: conversational security, protocol incentivization, zero-knowledge,
etc.</p><p>Waku uses novel technologies. Hence, we knew that early dogfooding of Waku was necessary. Even if research
was still <em>in progress</em> <a href="#references">[1]</a>. Thus, as soon as Waku protocols and software were usable, we started to push
for the adoption of Waku. This started back in 2021.</p><p>Waku is the communication component of the Web3 trifecta. This trifecta was Ethereum (contracts), Swarm
(storage) and Whisper (communication). Hence, it made sense to first target dApps which already uses one of the pillars:
Ethereum.</p><p>As most dApps are web apps, we started the development of <a href="https://vac.dev/presenting-js-waku" target="_blank" rel="noopener noreferrer">js-waku for the browser</a>.</p><p>Once ready, we reached out to dApps to integrate Waku, added <a href="https://twitter.com/waku_org/status/1451400128791605254?s=20&amp;t=Zhc0BEz6RVLkE_SeE6UyFA" target="_blank" rel="noopener noreferrer">prizes to hackathons</a>
and gave <a href="https://docs.wakuconnect.dev/docs/presentations/" target="_blank" rel="noopener noreferrer">talks</a>.</p><p>We also assumed we would see patterns in the usage of Waku, that we would facilitate with the help of
<a href="https://github.com/status-im/wakuconnect-vote-poll-sdk" target="_blank" rel="noopener noreferrer">SDKs</a>.</p><p>Finally, we created several web apps:
<a href="https://docs.wakuconnect.dev/docs/examples/" target="_blank" rel="noopener noreferrer">examples</a>
and <a href="https://github.com/status-iM/gnosis-safe-waku" target="_blank" rel="noopener noreferrer">PoCs</a>.</p><p>By discussing with Waku users and watching it being used, we learned a few facts:</p><ol><li>The potential use cases for Waku are varied and many:</li></ol><ul><li>Wallet <!-- -->&lt;<!-- -->&gt;<!-- --> dApp communication: <a href="https://medium.com/walletconnect/walletconnect-v2-0-protocol-whats-new-3243fa80d312" target="_blank" rel="noopener noreferrer">WalletConnect</a>, <a href="https://xmtp.org/docs/dev-concepts/architectural-overview/" target="_blank" rel="noopener noreferrer">XMTP</a></li><li>Off-chain (and private) marketplace:
<a href="https://twitter.com/RAILGUN_Project/status/1556780629848727552?s=20&amp;t=NEKQJiJAfg5WJqvuF-Ym_Q" target="_blank" rel="noopener noreferrer">RAILGUN</a> &amp;
<a href="https://twitter.com/TheBojda/status/1455557282318721026" target="_blank" rel="noopener noreferrer">Decentralized Uber</a></li><li>Signature exchange for a multi-sign wallet: <a href="https://github.com/status-im/gnosis-safe-waku" target="_blank" rel="noopener noreferrer">Gnosis Safe x Waku</a></li><li>Off-chain Game moves/actions: <a href="https://showcase.ethglobal.com/ethonline2021/super-card-game" target="_blank" rel="noopener noreferrer">Super Card Game (EthOnline 2021)</a></li><li>Decentralized Pastebin: <a href="https://debin.io/" target="_blank" rel="noopener noreferrer">Debin</a></li></ul><ol start="2"><li>Many projects are interested in having an embedded chat in their dApp,</li><li>There are complex applications that need Waku as a solution. Taking RAILGUN as an example:</li></ol><ul><li>Web wallet</li><li>+<!-- --> React Native mobile wallet</li><li>+<!-- --> NodeJS node/backend.</li></ul><p>(1) means that it is not that easy to create SDKs for common use cases.</p><p>(2) was a clear candidate for an SDK. Yet, building a chat app is a complex task. Hence, the Status app team tackled
this in the form of <a href="https://github.com/status-im/status-web/" target="_blank" rel="noopener noreferrer">Status Web</a>.</p><p>Finally, (3) was the most important lesson. We learned that multi-tier applications need Waku for decentralized and
censorship-resistant communications. For these projects, js-waku is simply not enough. They need Waku to work in their
Golang backend, Unity desktop game and React Native mobile app.</p><p>We understood that we should see the whole Waku software suite
(<a href="https://github.com/waku-org/js-waku" target="_blank" rel="noopener noreferrer">js-waku</a>,
<a href="https://github.com/status-im/nwaku" target="_blank" rel="noopener noreferrer">nwaku</a>,
<a href="https://github.com/status-im/go-waku" target="_blank" rel="noopener noreferrer">go-waku</a>,
<a href="https://github.com/waku-org/waku-react-native" target="_blank" rel="noopener noreferrer">waku-react-native</a>,
<a href="https://github.com/waku-org" target="_blank" rel="noopener noreferrer">etc</a>) as an asset for its success.
That we should not limit outreach, marketing, documentation efforts to the web, but target all platforms.</p><p>From a market perspective, we identified several actors:</p><ul><li>platforms: Projects that uses Waku to handle communication,</li><li>operators: Operators run Waku nodes and are incentivized to do so,</li><li>developers: Developers are usually part of a platforms or solo hackers learning Web3,</li><li>contributors: Developers and researchers with interests in decentralization, privacy, censorship-resistance,
zero-knowledge, etc.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-for-all-decentralized-applications-and-infrastructures">Waku for All Decentralized Applications and Infrastructures<a href="#waku-for-all-decentralized-applications-and-infrastructures" class="hash-link" aria-label="Direct link to Waku for All Decentralized Applications and Infrastructures" title="Direct link to Waku for All Decentralized Applications and Infrastructures">​</a></h2><p>In 2022, we shifted our focus to make the various Waku implementations <strong>usable and used</strong>.</p><p>We made Waku <a href="https://github.com/status-im/go-waku/tree/master/examples" target="_blank" rel="noopener noreferrer">multi-plaform</a>.</p><p>We shifted Waku positioning to leverage all Waku implementations and better serve the user's needs:</p><ul><li>Running a node for your projects and want to use Waku? Use <a href="https://github.com/status-im/nwaku" target="_blank" rel="noopener noreferrer">nwaku</a>.</li><li>Going mobile? Use <a href="https://github.com/status-im/waku-react-native" target="_blank" rel="noopener noreferrer">Waku React Native</a>.</li><li>C++ Desktop Game? Use <a href="https://github.com/status-im/go-waku/tree/master/examples/c-bindings" target="_blank" rel="noopener noreferrer">go-waku's C-Bindings</a>.</li><li>Web app? Use <a href="https://github.com/status-im/js-waku" target="_blank" rel="noopener noreferrer">js-waku</a>.</li></ul><p>We are consolidating the documentation for all implementations on a single website (<a href="https://github.com/waku-org/waku.org/issues/15" target="_blank" rel="noopener noreferrer">work in progress</a>)
to improve developer experience.</p><p>This year, we also started the <em>operator outreach</em> effort to push for users to run their own Waku nodes. We have
recently concluded our <a href="https://github.com/status-im/nwaku/issues/828" target="_blank" rel="noopener noreferrer">first operator trial run</a>.
<a href="https://vac.dev/introducing-nwaku" target="_blank" rel="noopener noreferrer">Nwaku</a>'s documentation, stability and performance has improved. It is now easier to
run your <a href="https://github.com/status-im/nwaku/tree/master/docs/operators" target="_blank" rel="noopener noreferrer">own Waku node</a>.</p><p>Today, operator wannabes most likely run their own nodes to support or use the Waku network.
We are <a href="https://twitter.com/oskarth/status/1582027828295790593?s=20&amp;t=DPEP6fXK6KWbBjV5EBCBMA" target="_blank" rel="noopener noreferrer">dogfooding</a>
<a href="https://github.com/status-im/nwaku/issues/827" target="_blank" rel="noopener noreferrer">Waku RLN</a>, our novel economic spam protection protocol,
and looking at <a href="https://github.com/vacp2p/research/issues/99" target="_blank" rel="noopener noreferrer">incentivizing the Waku Store protocol</a>.
This way, we are adding reasons to run your own Waku node.</p><p>For those who were following us in 2021, know that we are retiring the <em>Waku Connect</em> branding in favour of the <em>Waku</em>
branding.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-for-your-project">Waku for Your Project<a href="#waku-for-your-project" class="hash-link" aria-label="Direct link to Waku for Your Project" title="Direct link to Waku for Your Project">​</a></h2><p>As discussed, Waku is now available on various platforms. The question remains: How can Waku benefit <strong>your</strong> project?</p><p>Here are a couple of use cases we recently investigated:</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="layer-2-decentralization">Layer-2 Decentralization<a href="#layer-2-decentralization" class="hash-link" aria-label="Direct link to Layer-2 Decentralization" title="Direct link to Layer-2 Decentralization">​</a></h2><p>Most (<a href="#references">[2] [3]</a> roll-ups use a centralized sequencer or equivalent. Running several sequencers is not as straightforward as running several execution nodes.
Waku can help:</p><ul><li>Provide a neutral marketplace for a mempool: If sequencers compete for L2 tx fees, they may not be incentivized to
share transactions with other sequencers. Waku nodes can act as a neutral network to enable all sequences to access
transactions.</li><li>Enable censorship-resistant wallet<!-- -->&lt;<!-- -->&gt;<!-- -->L2 communication,</li><li>Provide rate limiting mechanism for spam protection: Using <a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">RLN</a> to prevent DDOS.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="device-pairing-and-communication">Device pairing and communication<a href="#device-pairing-and-communication" class="hash-link" aria-label="Direct link to Device pairing and communication" title="Direct link to Device pairing and communication">​</a></h2><p>With <a href="https://rfc.vac.dev/spec/43/" target="_blank" rel="noopener noreferrer">Waku Device Pairing</a>, a user can setup a secure encrypted communication channel
between their devices. As this channel would operate over Waku, it would be censorship-resistant and privacy preserving.
These two devices could be:</p><ul><li>Ethereum node and mobile phone to access a remote admin panel,</li><li>Alice's phone and Bob's phone for any kind of secure communication,</li><li>Mobile wallet and desktop/browser dApp for transaction and signature exchange.</li></ul><p>Check <a href="https://github.com/waku-org/js-waku/issues/950" target="_blank" rel="noopener noreferrer">js-waku#950</a> for the latest update on this.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="get-involved">Get Involved<a href="#get-involved" class="hash-link" aria-label="Direct link to Get Involved" title="Direct link to Get Involved">​</a></h2><p>Developer? Grab any of the Waku implementations and integrate it in your app: <a href="https://waku.org/platform" target="_blank" rel="noopener noreferrer">https://waku.org/platform</a>.</p><p>Researcher? See <a href="https://vac.dev/contribute" target="_blank" rel="noopener noreferrer">https://vac.dev/contribute</a> to participate in Waku research.</p><p>Tech-savvy? Try to run your own node: <a href="https://waku.org/operator" target="_blank" rel="noopener noreferrer">https://waku.org/operator</a>.</p><p>Otherwise, play around with the various <a href="https://github.com/waku-org/js-waku-examples#readme" target="_blank" rel="noopener noreferrer">web examples</a>.</p><p>If you want to help, we are <a href="https://jobs.status.im/" target="_blank" rel="noopener noreferrer">hiring</a>!</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="moving-forward">Moving Forward<a href="#moving-forward" class="hash-link" aria-label="Direct link to Moving Forward" title="Direct link to Moving Forward">​</a></h2><p>What you can expect next:</p><ul><li><a href="https://forum.vac.dev/t/waku-v2-scalability-studies/142/9" target="_blank" rel="noopener noreferrer">Scalability and performance studies</a> and improvement across Waku software,</li><li><a href="https://github.com/waku-org/waku.org/issues/15" target="_blank" rel="noopener noreferrer">New websites</a> to easily find documentation about Waku and its implementations,</li><li>New Waku protocols implemented in all code bases and cross client PoCs
(<a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">noise</a>, <a href="https://rfc.vac.dev/spec/37/" target="_blank" rel="noopener noreferrer">noise-sessions</a>,
<a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">waku-rln-relay</a>, etc),</li><li>Easier to <a href="https://github.com/status-im/nwaku/issues/828" target="_blank" rel="noopener noreferrer">run your own waku node</a>, more operator trials,</li><li>Dogfooding and Improvement of existing protocols (e.g. <a href="https://github.com/vacp2p/rfc/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc++12%2FWAKU2-FILTER" target="_blank" rel="noopener noreferrer">Waku Filter</a>),</li><li>Continue our focus Waku portability: Browser,
<a href="https://twitter.com/richardramos_me/status/1574405469912932355?s=20&amp;t=DPEP6fXK6KWbBjV5EBCBMA" target="_blank" rel="noopener noreferrer">Raspberry Pi Zero</a> and other restricted-resource environments,</li><li>More communication &amp; marketing effort around Waku and the Waku developer community.</li></ul><hr><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li>[<!-- -->1<!-- -->]<!-- --> Waku is modular; it is a suite of protocols; hence some Waku protocols may be mature, while
new protocols are still being designed. Which means that research continues to be <em>ongoing</em> while
Waku is already used in production.</li><li><a href="https://community.optimism.io/docs/how-optimism-works/#block-production" target="_blank" rel="noopener noreferrer">[2]</a> The Optimism Foundation runs the only block produce on the Optimism network.</li><li><a href="https://l2beat.com/" target="_blank" rel="noopener noreferrer">[3]</a> Top 10 L2s are documented has having a centralized operator.</li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Building Privacy-Protecting Infrastructure]]></title>
<link>https://vac.dev/rlog/building-privacy-protecting-infrastructure</link>
<guid>https://vac.dev/rlog/building-privacy-protecting-infrastructure</guid>
<pubDate>Fri, 04 Nov 2022 12:00:00 GMT</pubDate>
<description><![CDATA[What is privacy-protecting infrastructure? Why do we need it and how we can build it? We'll look at Waku, the communication layer for Web3. We'll see how it uses ZKPs to incentivize and protect the Waku network. We'll also look at Zerokit, a library that makes it easier to use ZKPs in different environments. After reading this, I hope you'll better understand the importance of privacy-protecting infrastructure and how we can build it.]]></description>
<content:encoded><![CDATA[<p>What is privacy-protecting infrastructure? Why do we need it and how we can build it? We'll look at Waku, the communication layer for Web3. We'll see how it uses ZKPs to incentivize and protect the Waku network. We'll also look at Zerokit, a library that makes it easier to use ZKPs in different environments. After reading this, I hope you'll better understand the importance of privacy-protecting infrastructure and how we can build it.</p><p><em>This write-up is based on a talk given at DevCon 6 in Bogota, a video can be found <a href="https://www.youtube.com/watch?v=CW1DYJifdhs" target="_blank" rel="noopener noreferrer">here</a></em></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="intro">Intro<a href="#intro" class="hash-link" aria-label="Direct link to Intro" title="Direct link to Intro">​</a></h3><p>In this write-up, we are going to talk about building privacy-protecting
infrastructure. What is it, why do we need it and how can we build it?</p><p>We'll look at Waku, the communication layer for Web3. We'll look at how we are
using Zero Knowledge (ZK) technology to incentivize and protect the Waku
network. We'll also look at Zerokit, a library we are writing to make ZKP easier
to use in different environments.</p><p>At the end of this write-up, I hope you'll come away with an understanding of
the importance of privacy-protecting infrastructure and how we can build it.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="about">About<a href="#about" class="hash-link" aria-label="Direct link to About" title="Direct link to About">​</a></h3><p>First, briefly about Vac. We build public good protocols for the decentralized
web, with a focus on privacy and communication. We do applied research based on
which we build protocols, libraries and publications. We are also the custodians
of protocols that reflect a set of principles.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Principles" src="/assets/images/building_private_infra_principles-699c52e62e0e4de0843ddb23ffbed365.png" width="1204" height="356" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>It has its origins in the <a href="https://status.im/" target="_blank" rel="noopener noreferrer">Status app</a> and trying to improve
the underlying protocols and infrastructure. We build <a href="https://waku.org/" target="_blank" rel="noopener noreferrer">Waku</a>,
among other things.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="why-build-privacy-protecting-infrastructure">Why build privacy-protecting infrastructure?<a href="#why-build-privacy-protecting-infrastructure" class="hash-link" aria-label="Direct link to Why build privacy-protecting infrastructure?" title="Direct link to Why build privacy-protecting infrastructure?">​</a></h3><p>Privacy is the power to selectively reveal yourself. It is a requirement for
freedom and self-determination.</p><p>Just like you need decentralization in order to get censorship-resistance, you
need privacy to enable freedom of expression.</p><p>To build applications that are decentralized and privacy-protecting, you need
the base layer, the infrastructure itself, to have those properties.</p><p>We see this a lot. It is easier to make trade-offs at the application layer than
doing them at the base layer. You can build custodial solutions on top of a
decentralized and non-custodial network where participants control their own
keys, but you can't do the opposite.</p><p>If you think about it, buildings can be seen as a form of privacy-protecting
infrastructure. It is completely normal and obvious in many ways, but when it
comes to the digital realm our mental models and way of speaking about it hasn't
caught up yet for most people.</p><p>I'm not going too much more into the need for privacy or what happens when you
don't have it, but suffice to say it is an important property for any open
society.</p><p>When we have conversations, true peer-to-peer offline conversations, we can talk
privately. If we use cash to buy things we can do commerce privately.</p><p>On the Internet, great as it is, there are a lot of forces that makes this
natural state of things not the default. Big Tech has turned users into a
commodity, a product, and monetized user's attention for advertising. To
optimize for your attention they need to surveil your habits and activities, and
hence breach your privacy. As opposed to more old-fashioned models, where
someone is buying a useful service from a company and the incentives are more
aligned.</p><p>We need to build credibly neutral infrastructure that protects your privacy at
the base layer, in order to truly enable applications that are
censorship-resistant and encourage meaningful freedom of expression.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="web3-infrastructure">Web3 infrastructure<a href="#web3-infrastructure" class="hash-link" aria-label="Direct link to Web3 infrastructure" title="Direct link to Web3 infrastructure">​</a></h3><p>Infrastructure is what lies underneath. Many ways of looking at this but I'll
keep it simple as per the original Web3 vision. You had Ethereum for
compute/consensus, Swarm for storage, and Whisper for messaging. Waku has taken
over the mantle from Whisper and is a lot more
<a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">usable</a> today than Whisper ever was,
for many reasons.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Web3 Infrastructure" src="/assets/images/web3_holy_trinity-fd2023ba2271927950dc70bb56f3c615.png" width="1408" height="826" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>On the privacy-front, we see how Ethereum is struggling. It is a big UX problem,
especially when you try to add privacy back "on top". It takes a lot of effort
and it is easier to censor. We see this with recent action around Tornado Cash.
Compare this with something like Zcash or Monero, where privacy is there by
default.</p><p>There are also problems when it comes to the p2p networking side of things, for
example with Ethereum validator privacy and hostile actors and jurisdictions. If
someone can easily find out where a certain validator is physically located,
that's a problem in many parts of the world. Being able to have stronger
privacy-protection guarantees would be very useful for high-value targets.</p><p>This doesn't begin to touch on the so called "dapps" that make a lot of
sacrifices in how they function, from the way domains work, to how websites are
hosted and the reliance on centralized services for communication. We see this
time and time again, where centralized, single points of failure systems work
for a while, but then eventually fail.</p><p>In many cases an individual user might not care enough though, and for platforms
the lure to take shortcuts is strong. That is why it is important to be
principled, but also pragmatic in terms of the trade-offs that you allow on top.
We'll touch more on this in the design goals around modularity that Waku has.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="zk-for-privacy-protecting-infrastructure">ZK for privacy-protecting infrastructure<a href="#zk-for-privacy-protecting-infrastructure" class="hash-link" aria-label="Direct link to ZK for privacy-protecting infrastructure" title="Direct link to ZK for privacy-protecting infrastructure">​</a></h3><p>ZKPs are a wonderful new tool. Just like smart contracts enables programmable
money, ZKPs allow us to express fundamentally new things. In line with the great
tradition of trust-minimization, we can prove statement while revealing the
absolute minimum information necessary. This fits the definition of privacy, the
power to selectively reveal yourself, perfectly. I'm sure I don't need to tell
anyone reading this but this is truly revolutionary. The technology is advancing
extremely fast and often it is our imagination that is the limit.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Zero knowledge" src="/assets/images/building_private_infra_zk-61dc3331f70705c672242b894bc35ab8.png" width="1412" height="930" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku">Waku<a href="#waku" class="hash-link" aria-label="Direct link to Waku" title="Direct link to Waku">​</a></h3><p>What is Waku? It is a set of modular protocols for p2p communication. It has a
focus on privacy, security and being able to run anywhere. It is the spiritual
success to Whisper.</p><p>By modular we mean that you can pick and choose protocols and how you use them
depending on constraints and trade-offs. For example, bandwidth usage vs
privacy.</p><p>It is designed to work in resource restricted environments, such as mobile
phones and in web browsers. It is important that infrastructure meets users
where they are and supports their real-world use cases. Just like you don't need
your own army and a castle to have your own private bathroom, you shouldn't need
to have a powerful always-on node to get reasonable privacy and
censorship-resistance. We might call this self-sovereignty.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku---adaptive-nodes">Waku - adaptive nodes<a href="#waku---adaptive-nodes" class="hash-link" aria-label="Direct link to Waku - adaptive nodes" title="Direct link to Waku - adaptive nodes">​</a></h3><p>One way of looking at Waku is as an open service network. There are nodes with
varying degrees of capabilities and requirements. For example when it comes to
bandwidth usage, storage, uptime, privacy requirements, latency requirements,
and connectivity restrictions.</p><p>We have a concept of adaptive nodes that can run a variety of protocols. A node
operator can choose which protocols they want to run. Naturally, there'll be
some nodes that do more consumption and other nodes that do more provisioning.
This gives rise to the idea of a service network, where services are provided
for and consumed.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Adaptive Nodes" src="/assets/images/building_private_infra_adaptive-69974a7e087e209572e1c2faf162e5d5.png" width="1408" height="1098" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku---protocol-interactions">Waku - protocol interactions<a href="#waku---protocol-interactions" class="hash-link" aria-label="Direct link to Waku - protocol interactions" title="Direct link to Waku - protocol interactions">​</a></h3><p>There are many protocols that interact. Waku Relay protocol is based on libp2p
GossipSub for p2p messaging. We have filter for bandwidth-restricted nodes to
only receive subset of messages. Lightpush for nodes with short connection
windows to push messages into network. Store for nodes that want to retrieve
historical messages.</p><p>On the payload layer, we provide support for Noise handshakes/key-exchanges.
This means that as a developers, you can get end-to-end encryption and expected
guarantees out of the box. We have support for setting up a secure channel from
scratch, and all of this paves the way for providing Signal's Double Ratchet at
the protocol level much easier. We also have experimental support for
multi-device usage. Similar features have existed in for example the Status app
for a while, but with this we make it easier for any platform using Waku to use
it.</p><p>There are other protocols too, related to peer discovery, topic usage, etc. See
<a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">specs</a> for more details.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Protocol Interactions" src="/assets/images/building_private_infra_interactions-8e9eee38a9a67973c36d11d111bdc384.png" width="1166" height="1131" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku---network">Waku - Network<a href="#waku---network" class="hash-link" aria-label="Direct link to Waku - Network" title="Direct link to Waku - Network">​</a></h3><p>For the Waku network, there are a few problems. For example, when it comes to
network spam and incentivizing service nodes. We want to address these while
keeping privacy-guarantees of the base layer. I'm going to go into both of
these.</p><p>The spam problem arises on the gossip layer when anyone can overwhelm the
network with messages. The service incentivization is a problem when nodes don't
directly benefit from the provisioning of a certain service. This can happen if
they are not using the protocol directly themselves as part of normal operation,
or if they aren't socially inclined to provide a certain service. This depends a
lot on how an individual platform decides to use the network.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Waku Network" src="/assets/images/building_private_infra_network-43aa536967aee45b44a1e2a6673b6941.png" width="1860" height="980" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dealing-with-network-spam-and-rln-relay">Dealing with network spam and RLN Relay<a href="#dealing-with-network-spam-and-rln-relay" class="hash-link" aria-label="Direct link to Dealing with network spam and RLN Relay" title="Direct link to Dealing with network spam and RLN Relay">​</a></h3><p>Since the p2p relay network is open to anyone, there is a problem with spam. If
we look at existing solutions for dealing with spam in traditional messaging
systems, a lot of entities like Google, Facebook, Twitter, Telegram, Discord use
phone number verification. While this is largely sybil-resistant, it is
centralized and not private at all.</p><p>Historically, Whisper used PoW which isn't good for heterogenerous networks.
Peer scoring is open to sybil attacks and doesn't directly address spam
protection in an anonymous p2p network.</p><p>The key idea here is to use RLN for private economic spam protection using
zkSNARKs.</p><p>I'm not going to go into too much detail of RLN here. If you are interested, I
gave a <a href="https://www.youtube.com/watch?v=g41nHQ0mLoA" target="_blank" rel="noopener noreferrer">talk</a> in Amsterdam at
Devconnect about this. We have some write-ups on RLN
<a href="https://vac.dev/rln-relay" target="_blank" rel="noopener noreferrer">here</a> by Sanaz who has been pushing a lot of this
from our side. There's also another talk at Devcon by Tyler going into RLN in
more detail. Finally, here's the <a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">RLN spec</a>.</p><p>I'll briefly go over what it is, the interface and circuit and then talk about
how it is used in Waku.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln---overview-and-flow">RLN - Overview and Flow<a href="#rln---overview-and-flow" class="hash-link" aria-label="Direct link to RLN - Overview and Flow" title="Direct link to RLN - Overview and Flow">​</a></h3><p>RLN stands for Rate Limiting Nullifier. It is an anonyomous rate limiting
mechanism based on zkSNARKs. By rate limiting we mean you can only send N
messages in a given period. By anonymity we mean that you can't link message to
a publisher. We can think of it as a voting booth, where you are only allowed to
vote once every election.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Voting Booth" src="/assets/images/building_private_infra_vote-a5992b54f4076642acc8e20ac716c750.png" width="703" height="479" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>It can be used for spam protection in p2p messaging systems, and also rate
limiting in general, such as for a decentralized captcha.</p><p>There are three parts to it. You register somewhere, then you can signal and
finally there's a verification/slashing phase. You put some capital at risk,
either economic or social, and if you double signal you get slashed.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln---circuit">RLN - Circuit<a href="#rln---circuit" class="hash-link" aria-label="Direct link to RLN - Circuit" title="Direct link to RLN - Circuit">​</a></h3><p>Here's what the private and public inputs to the circuit look like. The identity
secret is generated locally, and we create an identity commitment that is
inserted into a Merkle tree. We then use Merkle proofs to prove membership.
Registered member can only signal once for a given epoch or external nullifier,
for example every ten seconds in Unix time. RLN identifer is for a specific RLN
app.</p><p>We also see what the circuit output looks like. This is calculated locally. <code>y</code>
is a share of the secret equation, and the (internal) nullifier acts as a unique
fingerprint for a given app/user/epoch combination. How do we calculate <code>y</code> and
the internal nullifier?</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">// Private input</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal input identity_secret;</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal input path_elements[n_levels][1];</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal input identity_path_index[n_levels];</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">// Public input</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal input x; // signal_hash</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal input epoch; // external_nullifier</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal input rln_identifier;</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">// Circuit output</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal output y;</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal output root;</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">signal output nullifier;</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln---shamirs-secret-sharing">RLN - Shamir's secret sharing<a href="#rln---shamirs-secret-sharing" class="hash-link" aria-label="Direct link to RLN - Shamir's secret sharing" title="Direct link to RLN - Shamir's secret sharing">​</a></h3><p>This is done using <a href="https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing" target="_blank" rel="noopener noreferrer">Shamir's secret
sharing</a>. Shamir’s
secret sharing is based on idea of splitting a secret into shares. This is how
we enable slashing of funds.</p><p>In this case, we have two shares. If a given identity <code>a0</code> signals twice in
epoch/external nullifier, <code>a1</code> is the same. For a given RLN app,
<code>internal_nullifier</code> then stays the same. <code>x</code> is signal hash which is different,
and <code>y</code> is public, so we can reconstruct <code>identity_secret</code>. With the identity
secret revealed, this gives access to e.g. financial stake.</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">a_0 = identity_secret // secret S</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">a_1 = poseidonHash([a0, external_nullifier])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">y = a_0 + x * a_1</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">internal_nullifier = poseidonHash([a_1, rln_identifier])</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Shamir&amp;#39;s secret sharing" src="/assets/images/building_private_infra_shamir-8f4c8e31d2eaa86b62392514a411b999.png" width="936" height="704" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-relay">RLN Relay<a href="#rln-relay" class="hash-link" aria-label="Direct link to RLN Relay" title="Direct link to RLN Relay">​</a></h3><p>This is how RLN is used with Relay/GossipSub protocol. A node registers and
locks up funds, and after that it can send messages. It publishes a message
containing the Zero Knowledge proof and some other details.</p><p>Each relayer node listens to the membership contract for new members, and it
also keeps track of relevant metadata and merkle tree. Metadata is needed to be
able to detect double signaling and perform slashing.</p><p>Before forwarding a message, it does some verification checks to ensure there
are no duplicate messages, ZKP is valid and no double signaling has occured. It
is worth noting that this can be combined with peer scoring, for example for
duplicate messages or invalid ZK proofs.</p><p>In line of Waku's goals of modularity, RLN Relay is applied on a specific subset
of pubsub and content topics. You can think of it as an extra secure channel.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="RLN Relay" src="/assets/images/building_private_infra_rlnrelay-4823f37fce52d9d44d72ca73028fa9b8.png" width="1874" height="995" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-relay-cross-client-testnet">RLN Relay cross-client testnet<a href="#rln-relay-cross-client-testnet" class="hash-link" aria-label="Direct link to RLN Relay cross-client testnet" title="Direct link to RLN Relay cross-client testnet">​</a></h3><p>Where are we with RLN Relay deployment? We've recently launched our second
testnet. This is using RLN Relay with a smart contract on Goerli. It integrates
with our example p2p chat application, and it does so through three different
clients, nwaku, go-waku and js-waku for browsers. This is our first p2p
cross-client testnet for RLN Relay.</p><p>Here's a <a href="https://www.youtube.com/watch?v=-vVrJWW0fls" target="_blank" rel="noopener noreferrer">video</a> that shows a user
registering in a browser, signaling through JS-Waku. It then gets relayed to a
nwaku node, that verifies the proof. The second
<a href="https://www.youtube.com/watch?v=Xz5q2ZhkFYs" target="_blank" rel="noopener noreferrer">video</a> shows what happens in the
spam case. when more than one message is sent in a given epoch, it detects it as
spam and discards it. Slashing hasn't been implemented fully yet in the client
and is a work in progress.</p><p>If you are curious and want to participate, you can join the effort on our <a href="https://discord.gg/PQFdubGt6d" target="_blank" rel="noopener noreferrer">Vac
Discord</a>. We also have
<a href="https://github.com/status-im/nwaku/blob/master/docs/tutorial/rln-chat-cross-client.md" target="_blank" rel="noopener noreferrer">tutorials</a>
setup for all clients so you can play around with it.</p><p>As part of this, and to make it work in multiple different environments, we've
also been developing a new library called Zerokit. I'll talk about this a bit
later.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="private-settlement--service-credentials">Private settlement / Service credentials<a href="#private-settlement--service-credentials" class="hash-link" aria-label="Direct link to Private settlement / Service credentials" title="Direct link to Private settlement / Service credentials">​</a></h3><p>Going back to the service network idea, let's talk about service credentials.
The idea behind service credentials and private settlement is to enable two
actors to pay for and provide services without compromising their privacy. We do
not want the payment to create a direct public link between the service provider
and requester.</p><p>Recall the Waku service network illustration with adaptive nodes that choose
which protocols they want to run. Many of these protocols aren't very heavy and
just work by default. For example the relay protocol is enabled by default.
Other protocols are much heavier to provide, such as storing historical
messages.</p><p>It is desirable to have additional incentives for this, especially for platforms
that aren't community-based where some level of altruism can be assumed (e.g.
Status Communities, or WalletConnect cloud infrastructure).</p><p>You have a node Alice that is often offline and wants to consume historical
messages on some specific content topics. You have another node Bob that runs a
server at home where they store historical messages for the last several weeks.
Bob is happy to provide this service for free because he's excited about running
privacy-preserving infrastructure and he's using it himself, but his node is
getting overwhelmed by freeloaders and he feels like he should be paid something
for continuing to provide this service.</p><p>Alice deposits some funds in a smart contract which registers it in a tree,
similar to certain other private settlement mechanisms. A fee is taken or
burned. In exchange, she gets a set of tokens or service credentials. When she
wants to do a query with some criteria, she sends this to Bob. Bob responds with
size of response, cost, and receiver address. Alice then sends a proof of
delegation of a service token as a payment. Bob verifies the proof and resolves
the query.</p><p>The end result is that Alice has consumed some service from Bob, and Bob has
received payment for this. There's no direct transaction link between Alice and
Bob, and gas fees can be minimized by extending the period before settling on
chain.</p><p>This can be complemented with altruistic service provisioning, for example by
splitting the peer pool into two slots, or only providing a few cheap queries
for free.</p><p>The service provisioning is general, and can be generalized for any kind of
request/response service provisoning that we want to keep private.</p><p>This isn't a perfect solution, but it is an incremental improvement on top of
the status quo. It can be augmented with more advanced techniques such as better
non-repudiable node reputation, proof of correct service provisioning, etc.</p><p>We are currently in the raw spec / proof of concept stage of this. We expect to
launch a testnet of this later this year or early next year.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Service credentials flow" src="/assets/images/building_private_infra_servicecred-b022d763d66e89fb610d8d4552355e3c.png" width="1414" height="1022" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="zerokit">Zerokit<a href="#zerokit" class="hash-link" aria-label="Direct link to Zerokit" title="Direct link to Zerokit">​</a></h3><p><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a> is a set of Zero Knowledge modules,
written in Rust and designed to be used in many different environments. The
initial goal is to get the best of both worlds with Circom/Solidity/JS and
Rust/ZK ecosystem. This enables people to leverage Circom-based constructs from
non-JS environments.</p><p>For the RLN module, it is using Circom circuits via ark-circom and Rust for
scaffolding. It exposes a C FFI API that can be used through other system
programming environments, like Nim and Go. It also exposes an experimental WASM
API that can be used through web browsers.</p><p>Waku is p2p infrastructure running in many different environments, such as
Nim/JS/Go/Rust, so this a requirement for us.</p><p>Circom and JS strengths are access to Dapp developers, tooling, generating
verification code, circuits etc. Rust strengths is that it is systems-based and
easy to interface with other language runtime such as Nim, Go, Rust, C. It also
gives access to other Rust ZK ecosystems such as arkworks. This opens door for
using other constructs, such as Halo2. This becomes especially relevant for
constructs where you don't want to do a trusted setup or where circuits are more
complex/custom and performance requirements are higher.</p><p>In general with Zerokit, we want to make it easy to build and use ZKP in a
multitude of environments, such as mobile phones and web browsers. Currently it
is too complex to write privacy-protecting infrastructure with ZKPs considering
all the languages and tools you have to learn, from JS, Solidity and Circom to
Rust, WASM and FFI. And that isn't even touching on things like secure key
storage or mobile dev. Luckily more and more projects are working on this,
including writing DSLs etc. It'd also be exciting if we can make a useful
toolstack for JS-less ZK dev to reduce cognitive overhead, similar to what we
have with something like Foundry.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="other-research">Other research<a href="#other-research" class="hash-link" aria-label="Direct link to Other research" title="Direct link to Other research">​</a></h3><p>I also want to mention a few other things we are doing. One thing is
<a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">protocol specifications</a>. We think this is very important
for p2p infra, and we see a lot of other projects that claim to do it p2p
infrastructure but they aren't clear about guarantees or how stable something
is. That makes it hard to have multiple implementations, to collaborate across
different projects, and to analyze things objectively.</p><p>Related to that is publishing <a href="https://vac.dev/publications" target="_blank" rel="noopener noreferrer">papers</a>. We've put
out three so far, related to Waku and RLN-Relay. This makes it easier to
interface with academia. There's a lot of good researchers out there and we want
to build a better bridge between academia and industry.</p><p>Another thing is <a href="https://vac.dev/wakuv2-relay-anon" target="_blank" rel="noopener noreferrer">network</a>
<a href="https://github.com/vacp2p/research/issues/107" target="_blank" rel="noopener noreferrer">privacy</a>. Waku is modular with
respect to privacy guarantees, and there are a lot of knobs to turn here
depending on specific deployments. For example, if you are running the full
relay protocol you currently have much stronger receiver anonymity than if you
are running filter protocol from a bandwidth or connectivity-restricted node.</p><p>We aim to make this pluggable depending on user needs. E.g. mixnets such as Nym
come with some trade-offs but are a useful tool in the arsenal. A good mental
model to keep in mind is the anonymity trilemma, where you can only pick 2/3 out
of low latency, low bandwidth usage and strong anonymity.</p><p>We are currently exploring <a href="https://github.com/vacp2p/research/issues/119" target="_blank" rel="noopener noreferrer">Dandelion-like
additions</a> to the relay/gossip
protocol, which would provide for stronger sender anonymity, especially in a
multi-node/botnet attacker model. As part of this we are looking into different
parameters choices and general possibilities for lower latency usage. This could
make it more amenable for latency sensitive environments, such as validator
privacy, for specific threat models. The general theme here is we want to be
rigorous with the guarantees we provide, under what conditions and for what
threat models.</p><p>Another thing mentioned earlier is <a href="https://vac.dev/wakuv2-noise" target="_blank" rel="noopener noreferrer">Noise payload
encryption</a>, and specifically things like allowing
for pairing different devices with e.g. QR codes. This makes it easier for
developers to provide secure messaging in many realistic scenarios in a
multi-device world.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Other research" src="/assets/images/building_private_infra_misc-16721ea7c68873dbb0276ae7fe665ae5.png" width="1662" height="1156" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary">Summary<a href="#summary" class="hash-link" aria-label="Direct link to Summary" title="Direct link to Summary">​</a></h3><p>We've gone over what privacy-protecting infrastructure is, why we want it and
how we can build it. We've seen how ZK is a fundamental building block for this.
We've looked at Waku, the communication layer for Web3, and how it uses Zero
Knowledge proofs to stay private and function better. We've also looked at
Zerokit and how we can make it easier to do ZKP in different environments.</p><p>Finally we also looked at some other research we've been doing. All of the
things mentioned in this article, and more, is available as
<a href="https://vac.dev/research" target="_blank" rel="noopener noreferrer">write-ups</a>, <a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">specs</a>, or
discussions on our <a href="/rlog/forum.vac.dev/">forum</a> or <a href="/rlog/github.com/vacp2p/">Github</a>.</p><p>If you find any of this exciting to work on, feel free to reach out on our
Discord. We are also <a href="https://jobs.status.im/" target="_blank" rel="noopener noreferrer">hiring</a>, and we have started
expanding into other privacy infrastructure tech like private and provable
computation with ZK-WASM.</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Waku Privacy and Anonymity Analysis Part I: Definitions and Waku Relay]]></title>
<link>https://vac.dev/rlog/wakuv2-relay-anon</link>
<guid>https://vac.dev/rlog/wakuv2-relay-anon</guid>
<pubDate>Fri, 22 Jul 2022 10:00:00 GMT</pubDate>
<description><![CDATA[Introducing a basic threat model and privacy/anonymity analysis for the Waku v2 relay protocol.]]></description>
<content:encoded><![CDATA[<p>Introducing a basic threat model and privacy/anonymity analysis for the Waku v2 relay protocol.</p><p><a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Waku v2</a> enables secure, privacy preserving communication using a set of modular P2P protocols.
Waku v2 also aims at protecting the user's anonymity.
This post is the first in a series about Waku v2 security, privacy, and anonymity.
The goal is to eventually have a full privacy and anonymity analysis for each of the Waku v2 protocols, as well as covering the interactions of various Waku v2 protocols.
This provides transparency with respect to Waku's current privacy and anonymity guarantees, and also identifies weak points that we have to address.</p><p>In this post, we first give an informal description of security, privacy and anonymity in the context of Waku v2.
For each definition, we summarize Waku's current guarantees regarding the respective property.
We also provide attacker models, an attack-based threat model, and a first anonymity analysis of <a href="https://rfc.vac.dev/spec/11/" target="_blank" rel="noopener noreferrer">Waku v2 relay</a> within the respective models.</p><p>Waku comprises many protocols that can be combined in a modular way.
For our privacy and anonymity analysis, we start with the relay protocol because it is at the core of Waku v2 enabling Waku's publish subscribe approach to P2P messaging.
In its current form, Waku relay is a minor extension of <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md" target="_blank" rel="noopener noreferrer">libp2p GossipSub</a>.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 1: The Waku v2 relay mesh is based on the [GossipSub mesh](https://docs.libp2p.io/concepts/publish-subscribe#types-of-peering)" src="/assets/images/libp2p_gossipsub_types_of_peering-d0772153a5d11dea7b24c0bdc307a93d.png" width="800" height="305" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="informal-definitions-security-privacy-and-anonymity">Informal Definitions: Security, Privacy, and Anonymity<a href="#informal-definitions-security-privacy-and-anonymity" class="hash-link" aria-label="Direct link to Informal Definitions: Security, Privacy, and Anonymity" title="Direct link to Informal Definitions: Security, Privacy, and Anonymity">​</a></h2><p>The concepts of security, privacy, and anonymity are linked and have quite a bit of overlap.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="security">Security<a href="#security" class="hash-link" aria-label="Direct link to Security" title="Direct link to Security">​</a></h3><p>Of the three, <a href="https://en.wikipedia.org/wiki/Information_security" target="_blank" rel="noopener noreferrer">Security</a> has the clearest agreed upon definition,
at least regarding its key concepts: <em>confidentiality</em>, <em>integrity</em>, and <em>availability</em>.</p><ul><li>confidentiality: data is not disclosed to unauthorized entities.</li><li>integrity: data is not modified by unauthorized entities.</li><li>availability: data is available, i.e. accessible by authorized entities.</li></ul><p>While these are the key concepts, the definition of information security has been extended over time including further concepts,
e.g. <a href="https://en.wikipedia.org/wiki/Authentication" target="_blank" rel="noopener noreferrer">authentication</a> and <a href="https://en.wikipedia.org/wiki/Non-repudiation" target="_blank" rel="noopener noreferrer">non-repudiation</a>.
We might cover these in future posts.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="privacy">Privacy<a href="#privacy" class="hash-link" aria-label="Direct link to Privacy" title="Direct link to Privacy">​</a></h3><p>Privacy allows users to choose which data and information</p><ul><li>they want to share</li><li>and with whom they want to share it.</li></ul><p>This includes data and information that is associated with and/or generated by users.
Protected data also comprises metadata that might be generated without users being aware of it.
This means, no further information about the sender or the message is leaked.
Metadata that is protected as part of the privacy-preserving property does not cover protecting the identities of sender and receiver.
Identities are protected by the <a href="#anonymity">anonymity property</a>.</p><p>Often privacy is realized by the confidentiality property of security.
This neither makes privacy and security the same, nor the one a sub category of the other.
While security is abstract itself (its properties can be realized in various ways), privacy lives on a more abstract level using security properties.
Privacy typically does not use integrity and availability.
An adversary who has no access to the private data, because the message has been encrypted, could still alter the message.</p><p>Waku offers confidentiality via secure channels set up with the help of the <a href="https://noiseprotocol.org/" target="_blank" rel="noopener noreferrer">Noise Protocol Framework</a>.
Using these secure channels, message content is only disclosed to the intended receivers.
They also provide good metadata protection properties.
However, we do not have a metadata protection analysis as of yet,
which is part of our privacy/anonymity roadmap.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="anonymity">Anonymity<a href="#anonymity" class="hash-link" aria-label="Direct link to Anonymity" title="Direct link to Anonymity">​</a></h3><p>Privacy and anonymity are closely linked.
Both the identity of a user and data that allows inferring a user's identity should be part of the privacy policy.
For the purpose of analysis, we want to have a clearer separation between these concepts.</p><p>We define anonymity as <em>unlinkablity of users' identities and their shared data and/or actions</em>.</p><p>We subdivide anonymity into <em>receiver anonymity</em> and <em>sender anonymity</em>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="receiver-anonymity">Receiver Anonymity<a href="#receiver-anonymity" class="hash-link" aria-label="Direct link to Receiver Anonymity" title="Direct link to Receiver Anonymity">​</a></h4><p>We define receiver anonymity as <em>unlinkability of users' identities and the data they receive and/or related actions</em>.
The data transmitted via Waku relay must be a <a href="https://rfc.vac.dev/spec/14/" target="_blank" rel="noopener noreferrer">Waku message</a>, which contains a content topic field.
Because each message is associated with a content topic, and each receiver is interested in messages with specific content topics,
receiver anonymity in the context of Waku corresponds to <em>subscriber-topic unlinkability</em>.
An example for the "action" part of our receiver anonymity definition is subscribing to a specific topic.</p><p>The Waku message's content topic is not related to the libp2p pubsub topic.
For now, Waku uses a single libp2p pubsub topic, which means messages are propagated via a single mesh of peers.
With this, the receiver discloses its participation in Waku on the gossipsub layer.
We will leave the analysis of libp2p gossipsub to a future article within this series, and only provide a few hints and pointers here.</p><p>Waku offers k-anonymity regarding content topic interest in the global adversary model.
<a href="https://en.wikipedia.org/wiki/K-anonymity" target="_blank" rel="noopener noreferrer">K-anonymity</a> in the context of Waku means an attacker can link receivers to content topics with a maximum certainty of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>k</mi></mrow><annotation encoding="application/x-tex">1/k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1/</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span>.
The larger <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span>, the less certainty the attacker gains.
Receivers basically hide in a pool of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span> content topics, any subset of which could be topics they subscribed to.
The attacker does not know which of those the receiver actually subscribed to,
and the receiver enjoys <a href="https://en.wikipedia.org/wiki/Plausible_deniability#Use_in_cryptography" target="_blank" rel="noopener noreferrer">plausible deniability</a> regarding content topic subscription.
Assuming there are <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span></span> Waku content topics, a receiver has <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span></span>-anonymity with respect to association to a specific content topic.</p><p>Technically, Waku allows distributing messages over several libp2p pubsub topics.
This yields <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span>-anonymity, assuming <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span> content topics share the same pubsub topic.
However, if done wrongly, such sharding of pubsub topics can breach anonymity.
A formal specification of anonymity-preserving topic sharding building on the concepts of <a href="https://specs.status.im/spec/10#partitioned-topic" target="_blank" rel="noopener noreferrer">partitioned topics</a> is part of our roadmap.</p><p>Also, Waku is not directly concerned with 1:1 communication, so for this post, 1:1 communication is out of scope.
Channels for 1:1 communication can be implemented on top of Waku relay.
In the future, a 1:1 communication protocol might be added to Waku.
Similar to topic sharding, it would maintain receiver anonymity leveraging <a href="https://specs.status.im/spec/10#partitioned-topic" target="_blank" rel="noopener noreferrer">partitioned topics</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="sender-anonymity">Sender Anonymity<a href="#sender-anonymity" class="hash-link" aria-label="Direct link to Sender Anonymity" title="Direct link to Sender Anonymity">​</a></h4><p>We define sender anonymity as <em>unlinkability of users' identities and the data they send and/or related actions</em>.
Because the data in the context of Waku is Waku messages, sender anonymity corresponds to <em>sender-message unlinkability</em>.</p><p>In summary, Waku offers weak sender anonymity because of <a href="https://rfc.vac.dev/spec/11/#signature-policy" target="_blank" rel="noopener noreferrer">Waku's strict no sign policy</a>,
which has its origins in the <a href="https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#why-are-we-using-the-strictnosign-signature-policy" target="_blank" rel="noopener noreferrer">Ethereum consensus specs</a>.
<a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">17/WAKU-RLN-RELAY</a> and <a href="https://rfc.vac.dev/spec/18/" target="_blank" rel="noopener noreferrer">18/WAKU2-SWAP</a> mitigate replay and injection attacks.</p><p>Waku currently does not offer sender anonymity in stronger attacker models, as well as cannot protect against targeted attacks in weaker attacker models like the single or multi node attacker.
We will cover this in more detail in later sections.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="anonymity-trilemma">Anonymity Trilemma<a href="#anonymity-trilemma" class="hash-link" aria-label="Direct link to Anonymity Trilemma" title="Direct link to Anonymity Trilemma">​</a></h3><p><a href="https://freedom.cs.purdue.edu/projects/trilemma.html" target="_blank" rel="noopener noreferrer">The Anonymity trilemma</a> states that only two out of <em>strong anonymity</em>, <em>low bandwidth</em>, and <em>low latency</em> can be guaranteed in the global on-net attacker model.
Waku's goal, being a modular set of protocols, is to offer any combination of two out of these three properties, as well as blends.
An example for blending is an adjustable number of pubsub topics and peers in the respective pubsub topic mesh; this allows tuning the trade-off between anonymity and bandwidth.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 2: Anonymity Trilemma: pick two. " src="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjwhLS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgoKPHN2ZwogICB3aWR0aD0iMjEuMzEyMTEzbW0iCiAgIGhlaWdodD0iMjQuODY2NzIybW0iCiAgIHZpZXdCb3g9IjAgMCAyMS4zMTIxMTMgMjQuODY2NzIyIgogICB2ZXJzaW9uPSIxLjEiCiAgIGlkPSJzdmc1IgogICB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciCiAgIHhtbG5zOnN2Zz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxkZWZzCiAgICAgaWQ9ImRlZnMyIiAvPgogIDxnCiAgICAgaWQ9ImxheWVyMSIKICAgICB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtMy42NTM0ODM0LC02LjE1OTM5ODQpIj4KICAgIDxlbGxpcHNlCiAgICAgICBzdHlsZT0iZmlsbDojYzgwMDAwO2ZpbGwtb3BhY2l0eTowLjI7ZmlsbC1ydWxlOmV2ZW5vZGQ7c3Ryb2tlLXdpZHRoOjAuMjY0NTgzIgogICAgICAgaWQ9InBhdGgzOS0zIgogICAgICAgY3g9Ii04LjA5NDkxMTYiCiAgICAgICBjeT0iMTUuMzc4NTM4IgogICAgICAgcng9IjkuMzkyNzA3OCIKICAgICAgIHJ5PSIzLjk2ODc1IgogICAgICAgdHJhbnNmb3JtPSJyb3RhdGUoLTYwLjczMzYxMykiIC8+CiAgICA8ZWxsaXBzZQogICAgICAgc3R5bGU9ImZpbGw6IzAwMDA2NDtmaWxsLW9wYWNpdHk6MC4yO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgIGlkPSJwYXRoMzktNiIKICAgICAgIGN4PSIxNC4xMTAyNDMiCiAgICAgICBjeT0iMjIuNDY4NjI4IgogICAgICAgcng9IjkuMzkyNzA3OCIKICAgICAgIHJ5PSIzLjk2ODc1IiAvPgogICAgPGVsbGlwc2UKICAgICAgIHN0eWxlPSJmaWxsOiMyODAwMjg7ZmlsbC1vcGFjaXR5OjAuMztmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4xOTU5MDEiCiAgICAgICBpZD0icGF0aDM5LTciCiAgICAgICBjeD0iMTQuMTIzMzI5IgogICAgICAgY3k9IjI5LjA3NDIzNiIKICAgICAgIHJ4PSIxMC40Njk4NDYiCiAgICAgICByeT0iMS45NTE4ODUxIiAvPgogICAgPGVsbGlwc2UKICAgICAgIHN0eWxlPSJmaWxsOiMwMDY0MDA7ZmlsbC1vcGFjaXR5OjAuMjtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICBpZD0iZWxsaXBzZTc4NyIKICAgICAgIGN4PSItMjIuMTEyMzEyIgogICAgICAgY3k9Ii05LjYzNDU1MDEiCiAgICAgICByeD0iOS4zOTI3MDc4IgogICAgICAgcnk9IjMuOTY4NzUiCiAgICAgICB0cmFuc2Zvcm09Im1hdHJpeCgtMC40ODg4NzA3NiwtMC44NzIzNTYyMiwtMC44NzIzNTYyMiwwLjQ4ODg3MDc2LDAsMCkiIC8+CiAgICA8dGV4dAogICAgICAgeG1sOnNwYWNlPSJwcmVzZXJ2ZSIKICAgICAgIHN0eWxlPSJmb250LXN0eWxlOm5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7Zm9udC1zaXplOjEuNTQxNTVweDtmb250LWZhbWlseTpzYW5zLXNlcmlmO2ZpbGw6IzAwMDAwMDtmaWxsLW9wYWNpdHk6MTtzdHJva2U6bm9uZTtzdHJva2Utd2lkdGg6MC4wMzg1Mzg2IgogICAgICAgeD0iLTEwLjYzMjMwNSIKICAgICAgIHk9IjE2LjM2NjU3OSIKICAgICAgIGlkPSJ0ZXh0NDI1OSIKICAgICAgIHRyYW5zZm9ybT0icm90YXRlKC01NS4wMjk4MDcpIj48dHNwYW4KICAgICAgICAgaWQ9InRzcGFuNDI1NyIKICAgICAgICAgc3R5bGU9InN0cm9rZS13aWR0aDowLjAzODUzODYiCiAgICAgICAgIHg9Ii0xMC42MzIzMDUiCiAgICAgICAgIHk9IjE2LjM2NjU3OSI+bG93IGxhdGVuY3k8L3RzcGFuPjwvdGV4dD4KICAgIDx0ZXh0CiAgICAgICB4bWw6c3BhY2U9InByZXNlcnZlIgogICAgICAgc3R5bGU9ImZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtmb250LXNpemU6MS42MDUxN3B4O2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWY7ZmlsbDojMDAwMDAwO2ZpbGwtb3BhY2l0eToxO3N0cm9rZTpub25lO3N0cm9rZS13aWR0aDowLjA0MDEyOTQiCiAgICAgICB4PSIxNy4xMzU3NCIKICAgICAgIHk9Ii04Ljc1MjMyNiIKICAgICAgIGlkPSJ0ZXh0MTAzMjMiCiAgICAgICB0cmFuc2Zvcm09InJvdGF0ZSg1OC4wMjkxOSkiPjx0c3BhbgogICAgICAgICBpZD0idHNwYW4xMDMyMSIKICAgICAgICAgc3R5bGU9InN0cm9rZS13aWR0aDowLjA0MDEyOTQiCiAgICAgICAgIHg9IjE3LjEzNTc0IgogICAgICAgICB5PSItOC43NTIzMjYiPmxvdyBiYW5kd2lkdGg8L3RzcGFuPjwvdGV4dD4KICAgIDx0ZXh0CiAgICAgICB4bWw6c3BhY2U9InByZXNlcnZlIgogICAgICAgc3R5bGU9ImZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtmb250LXNpemU6MS41NTM0NnB4O2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWY7ZmlsbDojMDAwMDAwO2ZpbGwtb3BhY2l0eToxO3N0cm9rZTpub25lO3N0cm9rZS13aWR0aDowLjAzODgzNjciCiAgICAgICB4PSI3LjQ3NTA3ODYiCiAgICAgICB5PSIyMi45MzU0IgogICAgICAgaWQ9InRleHQxMjk0MyI+PHRzcGFuCiAgICAgICAgIGlkPSJ0c3BhbjEyOTQxIgogICAgICAgICBzdHlsZT0ic3Ryb2tlLXdpZHRoOjAuMDM4ODM2NyIKICAgICAgICAgeD0iNy40NzUwNzg2IgogICAgICAgICB5PSIyMi45MzU0Ij5zdHJvbmcgYW5vbnltaXR5PC90c3Bhbj48L3RleHQ+CiAgICA8dGV4dAogICAgICAgeG1sOnNwYWNlPSJwcmVzZXJ2ZSIKICAgICAgIHN0eWxlPSJmb250LXN0eWxlOm5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7Zm9udC1zaXplOjEuNDk2MjFweDtmb250LWZhbWlseTpzYW5zLXNlcmlmO2ZpbGw6IzAwMDAwMDtmaWxsLW9wYWNpdHk6MTtzdHJva2U6bm9uZTtzdHJva2Utd2lkdGg6MC4wMzc0MDUzIgogICAgICAgeD0iNi4xNjc4NDEiCiAgICAgICB5PSIyOS42ODc5MjUiCiAgICAgICBpZD0idGV4dDE0MzgzIj48dHNwYW4KICAgICAgICAgaWQ9InRzcGFuMTQzODEiCiAgICAgICAgIHN0eWxlPSJzdHJva2Utd2lkdGg6MC4wMzc0MDUzIgogICAgICAgICB4PSI2LjE2Nzg0MSIKICAgICAgICAgeT0iMjkuNjg3OTI1Ij5mcmVxdWVuY3kgLyBwYXR0ZXJuPC90c3Bhbj48L3RleHQ+CiAgPC9nPgo8L3N2Zz4K" width="81" height="94" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>A fourth factor that influences <a href="https://freedom.cs.purdue.edu/projects/trilemma.html" target="_blank" rel="noopener noreferrer">the anonymity trilemma</a> is <em>frequency and patterns</em> of messages.
The more messages there are, and the more randomly distributed they are, the better the anonymity protection offered by a given anonymous communication protocol.
So, incentivising users to use the protocol, for instance by lowering entry barriers, helps protecting the anonymity of all users.
The frequency/patterns factor is also related to the above described k-anonymity.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="censorship-resistance">Censorship Resistance<a href="#censorship-resistance" class="hash-link" aria-label="Direct link to Censorship Resistance" title="Direct link to Censorship Resistance">​</a></h3><p>Another security related property that Waku aims to offer is censorship resistance.
Censorship resistance guarantees that users can participate even if an attacker tries to deny them access.
So, censorship resistance ties into the availability aspect of security.
In the context of Waku that means users should be able to send messages as well as receive all messages they are interested in,
even if an attacker tries to prevent them from disseminating messages or tries to deny them access to messages.</p><p>Currently, Waku only guarantees censorship resistance in the weak single node attacker model.
While currently employed secure channels mitigate targeted censorship, e.g. blocking specific content topics,
general censorship resistance in strong attacker models is part of our roadmap.
Among other options, we will investigate <a href="https://www.pluggabletransports.info/about/" target="_blank" rel="noopener noreferrer">Pluggable Transports</a> in future articles.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="attacker-types">Attacker Types<a href="#attacker-types" class="hash-link" aria-label="Direct link to Attacker Types" title="Direct link to Attacker Types">​</a></h2><p>The following lists various attacker types with varying degrees of power.
The more power an attacker has, the more difficult it is to gain the respective attacker position.</p><p>Each attacker type comes in a passive and an active variant.
While a passive attacker can stay hidden and is not suspicious,
the respective active attacker has more (or at least the same) deanonymization power.</p><p>We also distinguish between internal and external attackers.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="internal">Internal<a href="#internal" class="hash-link" aria-label="Direct link to Internal" title="Direct link to Internal">​</a></h3><p>With respect to Waku relay, an internal attacker participates in the same pubsub topic as its victims.
Without additional measures on higher layer protocols, access to an internal position is easy to get.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="single-node">Single Node<a href="#single-node" class="hash-link" aria-label="Direct link to Single Node" title="Direct link to Single Node">​</a></h4><p>This attacker controls a single node.
Because this position corresponds to normal usage of Waku relay, it is trivial to obtain.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="multi-node">Multi Node<a href="#multi-node" class="hash-link" aria-label="Direct link to Multi Node" title="Direct link to Multi Node">​</a></h4><p>This attacker controls several nodes. We assume a smaller static number of controlled nodes.
The multi node position can be achieved relatively easily by setting up multiple nodes.
Botnets might be leveraged to increase the number of available hosts.
Multi node attackers could use <a href="https://en.wikipedia.org/wiki/Sybil_attack" target="_blank" rel="noopener noreferrer">Sybil attacks</a> to increase the number of controlled nodes.
A countermeasure is for nodes to only accept libp2p gossipsub graft requests from peers with different IP addresses, or even different subnets.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="linearly-scaling-nodes">Linearly Scaling Nodes<a href="#linearly-scaling-nodes" class="hash-link" aria-label="Direct link to Linearly Scaling Nodes" title="Direct link to Linearly Scaling Nodes">​</a></h4><p>This attacker controls a number of nodes that scales linearly with the number of nodes in the network.
This attacker is especially interesting to investigate in the context of DHT security,
which Waku uses for ambient peer discovery.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="external">External<a href="#external" class="hash-link" aria-label="Direct link to External" title="Direct link to External">​</a></h3><p>An external attacker can only see encrypted traffic (protected by a secure channel set up with <a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">Noise</a>).
Because an internal position can be easily obtained,
in practice external attackers would mount combined attacks that leverage both internal an external attacks.
We cover this more below when describing attacks.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="local">Local<a href="#local" class="hash-link" aria-label="Direct link to Local" title="Direct link to Local">​</a></h4><p>A local attacker has access to communication links in a local network segment.
This could be a rogue access point (with routing capability).</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="as">AS<a href="#as" class="hash-link" aria-label="Direct link to AS" title="Direct link to AS">​</a></h4><p>An AS attacker controls a single AS (autonomous system).
A passive AS attacker can listen to traffic on arbitrary links within the AS.
An active AS attacker can drop, inject, and alter traffic on arbitrary links within the AS.</p><p>In practice, a malicious ISP would be considered as an AS attacker.
A malicious ISP could also easily setup a set of nodes at specific points in the network,
gaining internal attack power similar to a strong multi node attacker.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="global-on-net">Global On-Net<a href="#global-on-net" class="hash-link" aria-label="Direct link to Global On-Net" title="Direct link to Global On-Net">​</a></h4><p>A global on-net attacker has complete overview over the whole network.
A passive global attacker can listen to traffic on all links,
while the active global attacker basically carries the traffic: it can freely drop, inject, and alter traffic at all positions in the network.
This basically corresponds to the <a href="https://en.wikipedia.org/wiki/Dolev%E2%80%93Yao_model" target="_blank" rel="noopener noreferrer">Dolev-Yao model</a>.</p><p>An entity with this power would, in practice, also have the power of the internal linearly scaling nodes attacker.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="attack-based-threat-analysis">Attack-based Threat Analysis<a href="#attack-based-threat-analysis" class="hash-link" aria-label="Direct link to Attack-based Threat Analysis" title="Direct link to Attack-based Threat Analysis">​</a></h2><p>The following lists various attacks including the weakest attacker model in which the attack can be successfully performed.
The respective attack can be performed in all stronger attacker models as well.</p><p>An attack is considered more powerful if it can be successfully performed in a weaker attacker model.</p><p>If not stated otherwise, we look at these attacks with respect to their capability to deanonymize the message sender.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="scope">Scope<a href="#scope" class="hash-link" aria-label="Direct link to Scope" title="Direct link to Scope">​</a></h3><p>In this post, we introduce a simple tightly scoped threat model for Waku v2 Relay, which will be extended in the course of this article series.</p><p>In this first post, we will look at the relay protocol in isolation.
Even though many threats arise from layers Waku relay is based on, and layers that in turn live on top of relay,
we want to first look at relay in isolation because it is at the core of Waku v2.
Addressing and trying to solve all security issues of a complex system at once is an overwhelming task, which is why we focus on the soundness of relay first.</p><p>This also goes well with the modular design philosophy of Waku v2, as layers of varying levels of security guarantees can be built on top of relay, all of which can relay on the guarantees that Waku provides.
Instead of looking at a multiplicative explosion of possible interactions, we look at the core in this article, and cover the most relevant combinations in future posts.</p><p>Further restricting the scope, we will look at the data field of a relay message as a black box.
In a second article on Waku v2 relay, we will look into the data field, which according to the <a href="https://rfc.vac.dev/spec/11/#message-fields" target="_blank" rel="noopener noreferrer">specification of Waku v2 relay</a> must be a <a href="https://rfc.vac.dev/spec/14/" target="_blank" rel="noopener noreferrer">Waku v2 message</a>.
We only consider messages with version field <code>2</code>, which indicates that the payload has to be encoded using <a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">35/WAKU2-NOISE</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="prerequisite-get-a-specific-position-in-the-network">Prerequisite: Get a Specific Position in the Network<a href="#prerequisite-get-a-specific-position-in-the-network" class="hash-link" aria-label="Direct link to Prerequisite: Get a Specific Position in the Network" title="Direct link to Prerequisite: Get a Specific Position in the Network">​</a></h3><p>Some attacks require the attacker node(s) to be in a specific position in the network.
In most cases, this corresponds to trying to get into the mesh peer list for the desired pubsub topic of the victim node.</p><p>In libp2p gossipsub, and by extension Waku v2 relay, nodes can simply send a graft message for the desired topic to the victim node.
If the victim node still has open slots, the attacker gets the desired position.
This only requires the attacker to know the gossipsub multiaddress of the victim node.</p><p>A linearly scaling nodes attacker can leverage DHT based discovery systems to boost the probability of malicious nodes being returned, which in turn significantly increases the probability of attacker nodes ending up in the peer lists of victim nodes.
<a href="https://vac.dev/wakuv2-apd" target="_blank" rel="noopener noreferrer">Waku v2 discv5</a> will employ countermeasures that mitigate the amplifying effect this attacker type can achieve.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="replay-attack">Replay Attack<a href="#replay-attack" class="hash-link" aria-label="Direct link to Replay Attack" title="Direct link to Replay Attack">​</a></h3><p>In the scope we defined above, Waku v2 is resilient against replay attacks.
GossipSub nodes, and by extension Waku relay nodes, feature a <code>seen</code> cache, and only relay messages they have not seen before.
Further, replay attacks will be punished by <a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">RLN</a> and <a href="https://rfc.vac.dev/spec/18/" target="_blank" rel="noopener noreferrer">SWAP</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="neighbourhood-surveillance">Neighbourhood Surveillance<a href="#neighbourhood-surveillance" class="hash-link" aria-label="Direct link to Neighbourhood Surveillance" title="Direct link to Neighbourhood Surveillance">​</a></h3><p>This attack can be performed by a single node attacker that is connected to all peers of the victim node <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> with respect to a specific topic mesh.
The attacker also has to be connected to <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.
In this position, the attacker will receive messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">m_v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">v</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> sent by <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> both on the direct path from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>, and on indirect paths relayed by peers of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.
It will also receive messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">m_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> that are not sent by <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>. These messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">m_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> are relayed by both <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> and the peers of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.
Messages that are received (significantly) faster from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> than from any other of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>'s peers are very likely messages that <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> sent,
because for these messages the attacker is one hop closer to the source.</p><p>The attacker can (periodically) measure latency between itself and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>, and between itself and the peers of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> to get more accurate estimates for the expected timings.
An AS attacker (and if the topology allows, even a local attacker) could also learn the latency between <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> and its well-behaving peers.
An active AS attacker could also increase the latency between <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> and its peers to make the timing differences more prominent.
This, however, might lead to <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> switching to other peers.</p><p>This attack cannot (reliably) distinguish messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">m_v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">v</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> sent by <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> from messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">m_y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span> relayed by peers of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> the attacker is not connected to.
Still, there are hop-count variations that might be leveraged.
Messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">m_v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">v</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> always have a hop-count of 1 on the path from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> to the attacker, while all other paths are longer.
Messages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">m_y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span> might have the same hop-count on the path from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> as well as on other paths.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="controlled-neighbourhood">Controlled Neighbourhood<a href="#controlled-neighbourhood" class="hash-link" aria-label="Direct link to Controlled Neighbourhood" title="Direct link to Controlled Neighbourhood">​</a></h3><p>If a multi node attacker manages to control all peers of the victim node, it can trivially tell which messages originated from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="observing-messages">Observing Messages<a href="#observing-messages" class="hash-link" aria-label="Direct link to Observing Messages" title="Direct link to Observing Messages">​</a></h3><p>If Waku relay was not protected with Noise, the AS attacker could simply check for messages leaving <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> which have not been relayed to <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.
These are the messages sent by <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.
Waku relay protects against this attack by employing secure channels setup using Noise.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="correlation">Correlation<a href="#correlation" class="hash-link" aria-label="Direct link to Correlation" title="Direct link to Correlation">​</a></h3><p>Monitoring all traffic (in an AS or globally), allows the attacker to identify traffic correlated with messages originating from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span>.
This (alone) does not allow an external attacker to learn which message <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span></span> sent, but it allows identifying the respective traffic propagating through the network.
The more traffic in the network, the lower the success rate of this attack.</p><p>Combined with just a few nodes controlled by the attacker, the actual message associated with the correlated traffic can eventually be identified.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dos">DoS<a href="#dos" class="hash-link" aria-label="Direct link to DoS" title="Direct link to DoS">​</a></h3><p>An active single node attacker could run a disruption attack by</p><ul><li>(1) dropping messages that should be relayed</li><li>(2) flooding neighbours with bogus messages</li></ul><p>While (1) has a negative effect on availability, the impact is not significant.
A linearly scaling botnet attacker, however, could significantly disrupt the network with such an attack.
(2) is thwarted by <a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">RLN</a>.
Also <a href="https://rfc.vac.dev/spec/18/" target="_blank" rel="noopener noreferrer">SWAP</a> helps mitigating DoS attacks.</p><p>A local attacker can DoS Waku by dropping all Waku traffic within its controlled network segment.
An AS attacker can DoS Waku within its authority, while a global attacker can DoS the whole network.
A countermeasure are censorship resistance techniques like <a href="https://www.pluggabletransports.info/about/" target="_blank" rel="noopener noreferrer">Pluggable Transports</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary-and-future-work">Summary and Future Work<a href="#summary-and-future-work" class="hash-link" aria-label="Direct link to Summary and Future Work" title="Direct link to Summary and Future Work">​</a></h2><p>Currently, Waku v2 relay offers k-anonymity with respect to receiver anonymity.
This also includes k-anonymity towards legitimate members of the same topic.</p><p>Waku v2 relay offers sender anonymity in the single node attacker model with its <a href="https://rfc.vac.dev/spec/11/#signature-policy" target="_blank" rel="noopener noreferrer">strict no sign policy</a>.
Currently, Waku v2 does not guarantee sender anonymity in the multi node and stronger attacker models.
However, we are working on modular anonymity-preserving protocols and building blocks as part of our privacy/anonymity roadmap.
The goal is to allow tunable anonymity with respect to trade offs between <em>strong anonymity</em>, <em>low bandwidth</em>, and <em>low latency</em>.
All of these cannot be fully guaranteed as the <a href="https://freedom.cs.purdue.edu/projects/trilemma.html" target="_blank" rel="noopener noreferrer">the anonymity trilemma</a> states.
Some applications have specific requirements, e.g. low latency, which require a compromise on anonymity.
Anonymity-preserving mechanisms we plan to investigate and eventually specify as pluggable anonymity protocols for Waku comprise</p><ul><li><a href="https://arxiv.org/abs/1805.11060" target="_blank" rel="noopener noreferrer">Dandelion++</a> for lightweight anonymity;</li><li><a href="https://en.wikipedia.org/wiki/Onion_routing" target="_blank" rel="noopener noreferrer">onion routing</a> as a building block adding a low latency anonymization layer;</li><li><a href="https://en.wikipedia.org/wiki/Mix_network" target="_blank" rel="noopener noreferrer">a mix network</a> for providing strong anonymity (on top of onion routing) even in the strongest attacker model at the cost of higher latency.</li></ul><p>These pluggable anonymity-preserving protocols will form a sub-set of the Waku v2 protocol set.
As an intermediate step, we might directly employ Tor for onion-routing, and <a href="https://nymtech.net/" target="_blank" rel="noopener noreferrer">Nym</a> as a mix-net layer.</p><p>In future research log posts, we will cover further Waku v2 protocols and identify anonymity problems that will be added to our roadmap.
These protocols comprise</p><ul><li><a href="https://rfc.vac.dev/spec/13/" target="_blank" rel="noopener noreferrer">13/WAKU2-STORE</a>, which can violate receiver anonymity as it allows filtering by content topic.
A countermeasure is using the content topic exclusively for local filters.</li><li><a href="https://rfc.vac.dev/spec/12/" target="_blank" rel="noopener noreferrer">12/WAKU2-FILTER</a>, which discloses nodes' interest in topics;</li><li><a href="https://rfc.vac.dev/spec/19/" target="_blank" rel="noopener noreferrer">19/WAKU2-LIGHTPUSH</a>, which also discloses nodes' interest in topics and links the lightpush client as the sender of a message to the lightpush service node;</li><li><a href="https://rfc.vac.dev/spec/21/" target="_blank" rel="noopener noreferrer">21/WAKU2-FTSTORE</a>, which discloses nodes' interest in specific time ranges allowing to infer information like online times.</li></ul><p>While these protocols are not necessary for the operation of Waku v2, and can be seen as pluggable features,
we aim to provide alternatives without the cost of lowering the anonymity level.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">10/WAKU2</a></li><li><a href="https://rfc.vac.dev/spec/11/" target="_blank" rel="noopener noreferrer">11/WAKU2-RELAY</a></li><li><a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md" target="_blank" rel="noopener noreferrer">libp2p GossipSub</a></li><li><a href="https://en.wikipedia.org/wiki/Information_security" target="_blank" rel="noopener noreferrer">Security</a></li><li><a href="https://en.wikipedia.org/wiki/Authentication" target="_blank" rel="noopener noreferrer">Authentication</a></li><li><a href="https://en.wikipedia.org/wiki/Non-repudiation" target="_blank" rel="noopener noreferrer">Non-repudiation</a></li><li><a href="https://noiseprotocol.org/" target="_blank" rel="noopener noreferrer">Noise Protocol Framework</a></li><li><a href="https://en.wikipedia.org/wiki/Plausible_deniability#Use_in_cryptography" target="_blank" rel="noopener noreferrer">plausible deniability</a></li><li><a href="https://rfc.vac.dev/spec/14/" target="_blank" rel="noopener noreferrer">Waku v2 message</a></li><li><a href="https://specs.status.im/spec/10#partitioned-topic" target="_blank" rel="noopener noreferrer">partitioned topics</a></li><li><a href="https://en.wikipedia.org/wiki/Sybil_attack" target="_blank" rel="noopener noreferrer">Sybil attack</a></li><li><a href="https://en.wikipedia.org/wiki/Dolev%E2%80%93Yao_model" target="_blank" rel="noopener noreferrer">Dolev-Yao model</a></li><li><a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">35/WAKU2-NOISE</a></li><li><a href="https://vac.dev/wakuv2-apd" target="_blank" rel="noopener noreferrer">33/WAKU2-DISCV5</a></li><li><a href="https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#why-are-we-using-the-strictnosign-signature-policy" target="_blank" rel="noopener noreferrer">strict no sign policy</a></li><li><a href="https://rfc.vac.dev/spec/11/#signature-policy" target="_blank" rel="noopener noreferrer">Waku v2 strict no sign policy</a></li><li><a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">17/WAKU-RLN-RELAY</a></li><li><a href="https://freedom.cs.purdue.edu/projects/trilemma.html" target="_blank" rel="noopener noreferrer">anonymity trilemma</a></li><li><a href="https://rfc.vac.dev/spec/18/" target="_blank" rel="noopener noreferrer">18/WAKU2-SWAP</a></li><li><a href="https://www.pluggabletransports.info/about/" target="_blank" rel="noopener noreferrer">Pluggable Transports</a></li><li><a href="https://nymtech.net/" target="_blank" rel="noopener noreferrer">Nym</a></li><li><a href="https://arxiv.org/abs/1805.11060" target="_blank" rel="noopener noreferrer">Dandelion++</a></li><li><a href="https://rfc.vac.dev/spec/13/" target="_blank" rel="noopener noreferrer">13/WAKU2-STORE</a></li><li><a href="https://rfc.vac.dev/spec/12/" target="_blank" rel="noopener noreferrer">12/WAKU2-FILTER</a></li><li><a href="https://rfc.vac.dev/spec/19/" target="_blank" rel="noopener noreferrer">19/WAKU2-LIGHTPUSH</a></li><li><a href="https://rfc.vac.dev/spec/21/" target="_blank" rel="noopener noreferrer">21/WAKU2-FTSTORE</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Noise handshakes as key-exchange mechanism for Waku]]></title>
<link>https://vac.dev/rlog/wakuv2-noise</link>
<guid>https://vac.dev/rlog/wakuv2-noise</guid>
<pubDate>Tue, 17 May 2022 10:00:00 GMT</pubDate>
<description><![CDATA[We provide an overview of the Noise Protocol Framework as a tool to design efficient and secure key-exchange mechanisms in Waku2.]]></description>
<content:encoded><![CDATA[<p>We provide an overview of the Noise Protocol Framework as a tool to design efficient and secure key-exchange mechanisms in Waku2.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>In this post we will provide an overview of how <a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Waku v2</a> users can adopt <a href="http://www.noiseprotocol.org/noise.html" target="_blank" rel="noopener noreferrer">Noise handshakes</a> to agree on cryptographic keys used to securely encrypt messages.</p><p>This process belongs to the class of <em>key-exchange</em> mechanisms, consisting of all those protocols that, with different levels of complexity and security guarantees, allow two parties to publicly agree on a secret without letting anyone else know what this secret is.</p><p>But why do we need key-exchange mechanisms in the first place?</p><p>With the advent of <a href="https://en.wikipedia.org/wiki/Public-key_cryptography" target="_blank" rel="noopener noreferrer">public-key cryptography</a>, it become possible to decouple encryption from decryption through use of two distinct cryptographic keys: one <em>public</em>, used to encrypt information and that can be made available to anyone, and one <em>private</em> (kept secret), which enables decryption of messages encrypted with its corresponding public key. The same does not happen in the case of <a href="https://en.wikipedia.org/wiki/Symmetric-key_algorithm" target="_blank" rel="noopener noreferrer">symmetric encryption schemes</a> where, instead, the same key is used for both encryption and decryption operations and hence cannot be publicly revealed as for public keys.</p><p>In order to address specific application needs, many different public, symmetric and hybrid cryptographic schemes were designed: <a href="https://rfc.vac.dev/spec/6/" target="_blank" rel="noopener noreferrer">Waku v1</a> and <a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Waku v2</a>, which inherits part of their design from the Ethereum messaging protocol <a href="https://ethereum.org/en/developers/docs/networking-layer/#whisper" target="_blank" rel="noopener noreferrer">Whisper</a>, provide <a href="https://rfc.vac.dev/spec/26/" target="_blank" rel="noopener noreferrer">support</a> to both public-key primitives (<a href="https://en.wikipedia.org/wiki/Integrated_Encryption_Scheme" target="_blank" rel="noopener noreferrer"><code>ECIES</code></a>, <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm" target="_blank" rel="noopener noreferrer"><code>ECDSA</code></a>) and symmetric primitives (<a href="https://en.wikipedia.org/wiki/Galois/Counter_Mode" target="_blank" rel="noopener noreferrer"><code>AES-256-GCM</code></a>, <a href="https://en.wikipedia.org/wiki/SHA-3" target="_blank" rel="noopener noreferrer"><code>KECCAK-256</code></a>), used to sign, hash, encrypt and decrypt exchanged messages.</p><p>In principle, when communications employ public-key based encryption schemes (<code>ECIES</code>, in the case of Waku), there is no need for a key-agreement among parties: messages can be directly encrypted using the recipient's public-key before being sent over the network. However, public-key encryption and decryption primitives are usually very inefficient in processing large amount of data, and this may constitute a bottleneck for many of today's applications. Symmetric encryption schemes such as <code>AES-256-GCM</code>, on the other hand, are much more efficient, but the encryption/decryption key needs to be shared among users beforehand any encrypted messages is exchanged.</p><p>To counter the downsides given by each of these two approaches while taking advantage of their strengths, hybrid constructions were designed. In these, public-key primitives are employed to securely agree on a secret key which, in turn, is used with a symmetric cipher for encrypting messages. In other words, such constructions specify a (public-key based) key-agreement mechanism!</p><p>Waku, up to <a href="https://rfc.vac.dev/spec/14/#payload-encryption" target="_blank" rel="noopener noreferrer">payload version 1</a>, does not implement nor recommend any protocol for exchanging symmetric ciphers' keys, leaving such task to the application layer. It is important to note that the kind of key-agreement employed has a direct impact on the security properties that can be granted on later encrypted messages, while security requirements usually depend on the specific application for which encryption is needed in the first place.</p><p>In this regard, <a href="https://status.im" target="_blank" rel="noopener noreferrer">Status</a>, which builds on top of Waku, <a href="https://specs.status.im/spec/5" target="_blank" rel="noopener noreferrer">implements</a> a custom version of the <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener noreferrer">X3DH</a> key-agreement protocol, in order to allow users to instantiate end-to-end encrypted communication channels. However, although such a solution is optimal when applied to (distributed) E2E encrypted chats, it is not flexible enough to fit or simplify the variety of applications Waku aims to address.
Hence, proposing and implementing one or few key-agreements which provide certain (presumably <em>strong</em>) security guarantees, would inevitably degrade performances of all those applications for which, given their security requirements, more tailored and efficient key-exchange mechanisms can be employed.</p><p>Guided by different examples, in the following sections we will overview Noise, a protocol framework we are <a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">currently integrating</a> in Waku, for building secure key-agreements between two parties. One of the great advantage of using Noise is that it is possible to add support to new key-exchanges by just specifying users' actions from a predefined list, requiring none to minimal modifications to existing implementations. Furthermore, Noise provides a framework to systematically analyze protocols' security properties and the corresponding attacker threat models. This allows not only to easily design new key-agreements eventually optimized for specific applications we want to address, but also to easily analyze or even <a href="https://noiseexplorer.com/" target="_blank" rel="noopener noreferrer">formally verify</a> any of such custom protocol!</p><p>We believe that with its enormous flexibility and features, Noise represents a perfect candidate for bringing key-exchange mechanisms in Waku.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-diffie-hellman-key-exchange">The Diffie-Hellman Key-exchange<a href="#the-diffie-hellman-key-exchange" class="hash-link" aria-label="Direct link to The Diffie-Hellman Key-exchange" title="Direct link to The Diffie-Hellman Key-exchange">​</a></h2><p>The formalization of modern public-key cryptography started with the pioneering work of Whitefield Diffie and Martin Hellman, who detailed one of the earliest known key-agreement protocols: the famous <a href="https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange" target="_blank" rel="noopener noreferrer">Diffie-Hellman Key-Exchange</a>.</p><p>Diffie-Hellman (DH) key-exchange is largely used today and represents the main cryptographic building block on which Noise handshakes' security is based.</p><p>In turn, the security of DH is based on a mathematical problem called <a href="https://en.wikipedia.org/wiki/Discrete_logarithm" target="_blank" rel="noopener noreferrer">discrete logarithm</a> which is believed to be hard when the agreement is practically instantiated using certain <a href="https://en.wikipedia.org/wiki/Elliptic_curve" target="_blank" rel="noopener noreferrer">elliptic curves</a> <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi></mrow><annotation encoding="application/x-tex">E</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.05764em">E</span></span></span></span></span> defined over finite fields <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="double-struck">F</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">\mathbb{F}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">F</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span>.</p><p>Informally, a DH exchange between Alice and Bob proceeds as follows:</p><ul><li>Alice picks a secret scalar <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>A</mi></msub><mo>∈</mo><msub><mi mathvariant="double-struck">F</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">s_A\in\mathbb{F}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">F</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span> and computes, using the underlying <a href="https://en.wikipedia.org/wiki/Elliptic_curve_point_multiplication" target="_blank" rel="noopener noreferrer">curve's arithmetic</a>, the point <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>A</mi></msub><mo>=</mo><msub><mi>s</mi><mi>A</mi></msub><mo>⋅</mo><mi>P</mi><mo>∈</mo><mi>E</mi><mo stretchy="false">(</mo><msub><mi mathvariant="double-struck">F</mi><mi>p</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P_A = s_A\cdot P\in E(\mathbb{F}_p)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0361em;vertical-align:-0.2861em"></span><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="mopen">(</span><span class="mord"><span class="mord mathbb">F</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span></span> for a certain pre-agreed public generator <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi></mrow><annotation encoding="application/x-tex">P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> of the elliptic curve <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mo stretchy="false">(</mo><msub><mi mathvariant="double-struck">F</mi><mi>p</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">E(\mathbb{F}_p)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0361em;vertical-align:-0.2861em"></span><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="mopen">(</span><span class="mord"><span class="mord mathbb">F</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span></span>. She then sends <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">P_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> to Bob.</li><li>Similarly, Bob picks a secret scalar <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>B</mi></msub><mo>∈</mo><msub><mi mathvariant="double-struck">F</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">s_B\in\mathbb{F}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">F</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></span>, computes <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>B</mi></msub><mo>=</mo><msub><mi>s</mi><mi>B</mi></msub><mo>⋅</mo><mi>P</mi><mo>∈</mo><mi>E</mi><mo stretchy="false">(</mo><msub><mi mathvariant="double-struck">F</mi><mi>p</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P_B = s_B\cdot P\in E(\mathbb{F}_p)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0361em;vertical-align:-0.2861em"></span><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="mopen">(</span><span class="mord"><span class="mord mathbb">F</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span></span> and sends <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">P_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> to Alice.</li><li>By commutativity of scalar multiplication, both Alice and Bob can now compute the point <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub><mo>=</mo><msub><mi>s</mi><mi>A</mi></msub><msub><mi>s</mi><mi>B</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">P_{AB} = s_As_B\cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span>, using the elliptic curve point received from the other party and their secret scalar.</li></ul><p>The assumed hardness of computing discrete logarithms in the elliptic curve, ensures that it is not possible to compute <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">s_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> or <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">s_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">P_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">P_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>, respectively. Another security assumption (named <a href="https://en.wikipedia.org/wiki/Computational_Diffie%E2%80%93Hellman_assumption" target="_blank" rel="noopener noreferrer">Computational Diffie-Hellman assumption</a>) ensures that it is not possible to compute <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub></mrow><annotation encoding="application/x-tex">P_{AB}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> from <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi></mrow><annotation encoding="application/x-tex">P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span>, <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">P_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">P_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>. Hence the point <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub></mrow><annotation encoding="application/x-tex">P_{AB}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> shared by Alice and Bob at the end of the above protocol cannot be efficiently computed by an attacker intercepting <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">P_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">P_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>, and can then be used to generate a secret to be later employed, for example, as a symmetric encryption key.</p><p>On a side note, this protocol shows the interplay between two components typical to public-key based schemes: the scalars <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">s_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">s_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> can be seen as <em>private keys</em> associated to the <em>public keys</em> <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">P_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">P_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>, respectively, which allow Alice and Bob only to compute the shared secret point <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub></mrow><annotation encoding="application/x-tex">P_{AB}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="ephemeral-and-static-public-keys">Ephemeral and Static Public Keys<a href="#ephemeral-and-static-public-keys" class="hash-link" aria-label="Direct link to Ephemeral and Static Public Keys" title="Direct link to Ephemeral and Static Public Keys">​</a></h2><p>Although we assumed that it is practically impossible for an attacker to compute the randomly picked secret scalar from the corresponding public elliptic curve point, it may happen that such scalar gets compromised or can be guessed due to a faulty employed random number generator. In such cases, an attacker will be able to recover the final shared secret and all encryption keys eventually derived from that, with clear catastrophic consequences for the privacy of exchanged messages.</p><p>To mitigate such issues, multiple DH operations can be combined using two different types of exchanged elliptic curve points or, better, <em>public keys</em>: <em>ephemeral keys</em>, that is random keys used only once in a DH operation, and long-term <em>static keys</em>, used mainly for authentication purposes since employed multiple times.</p><p>Just to provide an example, let us suppose Alice and Bob perform the following custom DH-based key-exchange protocol:</p><ul><li>Alice generates an ephemeral key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mi>A</mi></msub><mo>=</mo><msub><mi>e</mi><mi>A</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">E_A=e_A\cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> by picking a random scalar <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>e</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">e_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and sends <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">E_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> to Bob;</li><li>Similarly, Bob generates an ephemeral key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mi>B</mi></msub><mo>=</mo><msub><mi>e</mi><mi>B</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">E_B=e_B\cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> and sends <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">E_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> to Alice;</li><li>Alice and Bob computes <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub><mo>=</mo><msub><mi>e</mi><mi>A</mi></msub><msub><mi>e</mi><mi>B</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">E_{AB} = e_Ae_B \cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> and from it derive a secret encryption key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span>.</li><li>Bob sends to Alice his static key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mi>B</mi></msub><mo>=</mo><msub><mi>s</mi><mi>B</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">S_B = s_B\cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> encrypted with <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span>.</li><li>Alice encrypts with <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span> her static key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mi>A</mi></msub><mo>=</mo><msub><mi>s</mi><mi>A</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">S_A = s_A\cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> and sends it to Bob.</li><li>Alice and Bob decrypt the received static keys, compute the secret <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub><mo>=</mo><msub><mi>s</mi><mi>A</mi></msub><msub><mi>s</mi><mi>B</mi></msub><mo>⋅</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">S_{AB} = s_As_B \cdot P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5945em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span></span> and use it together with <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub></mrow><annotation encoding="application/x-tex">E_{AB}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> to derive a new encryption key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>k</mi><mo>~</mo></mover></mrow><annotation encoding="application/x-tex">\tilde{k}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9313em"></span><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9313em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span><span style="top:-3.6134em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.25em"><span class="mord">~</span></span></span></span></span></span></span></span></span></span></span> to be later used with a symmetric cipher.</li></ul><p>In this protocol, if Alice's and/or Bob's static keys get compromised, it would not possible to derive the final secret key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>k</mi><mo>~</mo></mover></mrow><annotation encoding="application/x-tex">\tilde{k}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9313em"></span><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9313em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span><span style="top:-3.6134em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.25em"><span class="mord">~</span></span></span></span></span></span></span></span></span></span></span>, since at least one ephemeral key among <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mi>A</mi></msub></mrow><annotation encoding="application/x-tex">E_A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">A</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mi>B</mi></msub></mrow><annotation encoding="application/x-tex">E_B</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span> has to be compromised too in order to recover the secret <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>E</mi><mrow><mi>A</mi><mi>B</mi></mrow></msub></mrow><annotation encoding="application/x-tex">E_{AB}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">A</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span>. Furthermore, since Alice's and Bob's long-term static keys are encrypted, an attacker intercepting exchanged (encrypted) public keys will not be able to link such communication to Alice or Bob, unless one of the ephemeral key is compromised (and, even in such case, none of the messages encrypted under the key <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>k</mi><mo>~</mo></mover></mrow><annotation encoding="application/x-tex">\tilde{k}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9313em"></span><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9313em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span><span style="top:-3.6134em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.25em"><span class="mord">~</span></span></span></span></span></span></span></span></span></span></span> can be decrypted).</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-noise-protocol-framework">The Noise Protocol Framework<a href="#the-noise-protocol-framework" class="hash-link" aria-label="Direct link to The Noise Protocol Framework" title="Direct link to The Noise Protocol Framework">​</a></h2><p>In previous section we gave a small intuition on how multiple DH operations over ephemeral and static users' public keys can be combined to create different key-exchange protocols.</p><p>The <a href="http://www.noiseprotocol.org/noise.html" target="_blank" rel="noopener noreferrer">Noise Protocol Framework</a>, defines various rules for building custom key-exchange protocols while allowing easy analysis of the security properties and threat models provided given the type and order of the DH operations employed.</p><p>In Noise terminology, a key-agreement or <em>Noise protocol</em> consists of one or more <em>Noise handshakes</em>. During a Noise handshake, Alice and Bob exchange multiple (handshake) messages containing their ephemeral keys and/or static keys. These public keys are then used to perform a handshake-dependent sequence of Diffie-Hellman operations, whose results are all hashed into a shared secret key. Similarly as we have seen above, after a handshake is complete, each party will use the derived secret key to send and receive <a href="https://en.wikipedia.org/wiki/Authenticated_encryption" target="_blank" rel="noopener noreferrer">authenticated encrypted data</a> by employing a symmetric cipher.</p><p>Depending on the <em>handshake pattern</em> adopted, different security guarantees can be provided on messages encrypted using a handshake-derived key.</p><p>The Noise handshakes we support in Waku all provide the following security properties:</p><ul><li><strong>Confidentiality</strong>: the adversary should not be able to learn what data is being sent between Alice and Bob.</li><li><strong>Strong forward secrecy</strong>: an active adversary cannot decrypt messages nor infer any information on the employed encryption key, even in the case he has access to Alice's and Bob's long-term private keys (during or after their communication).</li><li><strong>Authenticity</strong>: the adversary should not be able to cause either Alice or Bob to accept messages coming from a party different than their original senders.</li><li><strong>Integrity</strong>: the adversary should not be able to cause Alice or Bob to accept data that has been tampered with.</li><li><strong>Identity-hiding</strong>: once a secure communication channel is established, a passive adversary should not be able to link exchanged encrypted messages to their corresponding sender and recipient by knowing their long-term static keys.</li></ul><p>We refer to <a href="http://www.noiseprotocol.org/noise.html" target="_blank" rel="noopener noreferrer">Noise specification</a> for more formal security definitions and precise threat models relative to Waku <a href="#Supported-Noise-Handshakes-in-Waku">supported Noise Handshake patterns</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="message-patterns">Message patterns<a href="#message-patterns" class="hash-link" aria-label="Direct link to Message patterns" title="Direct link to Message patterns">​</a></h2><p>Noise handshakes involving DH operations over ephemeral and static keys can be succinctly sketched using the following set of <em>handshake message tokens</em>: <code>e</code>,<code>s</code>,<code>ee</code>,<code>se</code>,<code>es</code>,<code>ss</code>.</p><p>Tokens employing single letters denote (the type of) users' public keys: <code>e</code> refers to randomly generated ephemeral key(s), while <code>s</code> indicates the users' long-term static key(s).</p><p>Two letters tokens, instead, denotes DH operations over the two users' public keys the token refers to, given that the left token letter refers to the handshake <em>initiator's</em> public key, while the right token letter indicates the used <em>responder's</em> public key. Thus, if Alice started a handshake with Bob, the <code>es</code> token will shortly represent a DH operation among Alice's ephemeral key <code>e</code> and Bob's static key <code>s</code>.</p><p>Since, in order to perform any DH operations users need to share (or pre-share) the corresponding public keys, Noise compactly represents messages' exchanges using the two direction <code>-&gt;</code> and <code>&lt;-</code>, where the <code>-&gt;</code> denotes a message (arbitrary and/or DH public key) from the initiator to the responder, while <code>&lt;-</code> the opposite.</p><p>Hence a <em>message pattern</em> consisting of a direction and one or multiple tokens such as <code>&lt;- e, s, es</code> has to be interpreted one token at a time: in this example, the responder is sending his ephemeral and static key to the initiator and is then executing a DH operation over the initiator's ephemeral key <code>e</code> (shared in a previously exchanged message pattern) and his static key <code>s</code>. On the other hand, such message indicates also that the initiator received the responder's ephemeral and static keys <code>e</code> and <code>s</code>, respectively, and performed a DH operation over his ephemeral key and the responder's just received static key <code>s</code>. In this way, both parties will be able to derive at the end of each message pattern processed the same shared secret, which is eventually used to update any derived symmetric encryption keys computed so far.</p><p>In some cases, DH public keys employed in a handshake are pre-shared before the handshake itself starts. In order to chronologically separate exchanged keys and DH operations performed before and during a handshake, Noise employs the <code>...</code> delimiter.</p><p>For example, the following message patterns</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">&lt;- e</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">...</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-&gt; e, ee</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>indicates that the initiator knew the responder's ephemeral key before he sends his own ephemeral key and executes a DH operation between both parties ephemeral keys (similarly, the responder receives the initiator's ephemeral key and does a <code>ee</code> DH operation).</p><p>At this point it should be clear how such notation is able to compactly represent a large variety of DH based key-agreements. Nevertheless, we can easily define additional tokens and processing rules in order to address specific applications and security requirements, such as the <a href="http://www.noiseprotocol.org/noise.html#handshake-tokens" target="_blank" rel="noopener noreferrer"><code>psk</code></a> token used to process arbitrary pre-shared key material.</p><p>As an example of Noise flexibility, the custom protocol we detailed <a href="#Ephemeral-and-Static-Public-Keys">above</a> can be shortly represented as <em>(Alice is on the left)</em>:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">-&gt; e</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">&lt;- e, ee, s</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-&gt; s, ss</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>where after each DH operation an encryption key is derived (along with the secrets computed by all previously executed DH operations) in order to encrypt/decrypt any subsequent sent/received message.</p><p>Another example is given by the possibility to replicate within Noise the well established Signal's <a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener noreferrer">X3DH</a> key-agreement protocols, thus making the latter a general framework to design and study security of many practical and widespread DH-based key-exchange protocols.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-noise-state-objects">The Noise State Objects<a href="#the-noise-state-objects" class="hash-link" aria-label="Direct link to The Noise State Objects" title="Direct link to The Noise State Objects">​</a></h2><p>We mentioned multiple times that parties derive an encryption key each time they perform a DH operation, but how does this work in more details?</p><p>Noise defines three <em>state object</em>: a <em>Handshake State</em>, a <em>Symmetric State</em> and a <em>Cipher State</em>, each encapsulated into each other and instantiated during the execution of a handshake.</p><p>The Handshake State object stores the user's and other party's received ephemeral and static keys (if any) and embeds a Symmetric State object.</p><p>The Symmetric State, instead, stores a handshake hash value <code>h</code>, iteratively updated with any message read/received and DH secret computed, and a chaining key <code>ck</code>, updated using a key derivation function every time a DH secret is computed. This object further embeds a Cipher State.</p><p>Lastly, the Cipher State stores a symmetric encryption <code>k</code> key and a counter <code>n</code> used to encrypt and decrypt messages exchanged during the handshake (not only static keys, but also arbitrary payloads). These key and counter are refreshed every time the chaining key is updated.</p><p>While processing each handshake's message pattern token, all these objects are updated according to some specific <em>processing rules</em> which employ a combination of public-key primitives, hash and key-derivation functions and symmetric ciphers. It is important to note, however, that at the end of each processed message pattern, the two users will share the same Symmetric and Cipher State embedded in their respective Handshake States.</p><p>Once a handshake is complete, users derive two new Cipher States and can then discard the Handshake State object (and, thus, the embedded Symmetric State and Cipher State objects)
employed during the handshake.</p><p>These two Cipher states are used to encrypt and decrypt all outbound and inbound after-handshake messages, respectively, and only to these will be granted the confidentiality, authenticity, integrity and identity-hiding properties we detailed above.</p><p>For more details on processing rules, we refer to <a href="http://www.noiseprotocol.org/noise.html" target="_blank" rel="noopener noreferrer">Noise specifications</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="supported-noise-handshakes-in-waku">Supported Noise Handshakes in Waku<a href="#supported-noise-handshakes-in-waku" class="hash-link" aria-label="Direct link to Supported Noise Handshakes in Waku" title="Direct link to Supported Noise Handshakes in Waku">​</a></h2><p>The Noise handshakes we provided support to in Waku address four typical scenarios occurring when an encrypted communication channel between Alice and Bob is going to be created:</p><ul><li>Alice and Bob know each others' static key.</li><li>Alice knows Bob's static key;</li><li>Alice and Bob share no key material and they don't know each others' static key.</li><li>Alice and Bob share some key material, but they don't know each others' static key.</li></ul><p>The possibility to have handshakes based on the reciprocal knowledge parties have of each other, allows designing Noise handshakes that can quickly reach the desired level of security on exchanged encrypted messages while keeping the number of interactions between Alice and Bob minimum.</p><p>Nonetheless, due to the pure <em>token-based</em> nature of handshake processing rules, implementations can easily add support to any custom handshake pattern with minor modifications, in case more specific application use-cases need to be addressed.</p><p>On a side note, we already mentioned that identity-hiding properties can be guaranteed against a passive attacker that only reads the communication occurring between Alice and Bob. However, an active attacker who compromised one party's static key and actively interferes with the parties' exchanged messages, may lower the identity-hiding security guarantees provided by some handshake patterns. In our security model we exclude such adversary, but, for completeness, in the following we report a summary of possible de-anonymization attacks that can be performed by such an active attacker.</p><p>For more details on supported handshakes and on how these are implemented in Waku, we refer to <a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">35/WAKU2-NOISE</a> RFC.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-k1k1-handshake">The K1K1 Handshake<a href="#the-k1k1-handshake" class="hash-link" aria-label="Direct link to The K1K1 Handshake" title="Direct link to The K1K1 Handshake">​</a></h3><p>If Alice and Bob know each others' static key (e.g., these are public or were already exchanged in a previous handshake) , they MAY execute a <code>K1K1</code> handshake. In Noise notation <em>(Alice is on the left)</em> this can be sketched as:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> K1K1:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; s</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> &lt;- s</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ...</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; e</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> &lt;- e, ee, es</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; se</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>We note that here only ephemeral keys are exchanged. This handshake is useful in case Alice needs to instantiate a new separate encrypted communication channel with Bob, e.g. opening multiple parallel connections, file transfers, etc.</p><p><strong>Security considerations on identity-hiding (active attacker)</strong>: no static key is transmitted, but an active attacker impersonating Alice can check candidates for Bob's static key.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-xk1-handshake">The XK1 Handshake<a href="#the-xk1-handshake" class="hash-link" aria-label="Direct link to The XK1 Handshake" title="Direct link to The XK1 Handshake">​</a></h3><p>Here, Alice knows how to initiate a communication with Bob and she knows his public static key: such discovery can be achieved, for example, through a publicly accessible register of users' static keys, smart contracts, or through a previous public/private advertisement of Bob's static key.</p><p>A Noise handshake pattern that suits this scenario is <code>XK1</code>:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> XK1:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> &lt;- s</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ...</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; e</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> &lt;- e, ee, es</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; s, se</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>Within this handshake, Alice and Bob reciprocally authenticate their static keys <code>s</code> using ephemeral keys <code>e</code>. We note that while Bob's static key is assumed to be known to Alice (and hence is not transmitted), Alice's static key is sent to Bob encrypted with a key derived from both parties ephemeral keys and Bob's static key.</p><p><strong>Security considerations on identity-hiding (active attacker)</strong>: Alice's static key is encrypted with forward secrecy to an authenticated party. An active attacker initiating the handshake can check candidates for Bob's static key against recorded/accepted exchanged handshake messages.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-xx-and-xxpsk0-handshakes">The XX and XXpsk0 Handshakes<a href="#the-xx-and-xxpsk0-handshakes" class="hash-link" aria-label="Direct link to The XX and XXpsk0 Handshakes" title="Direct link to The XX and XXpsk0 Handshakes">​</a></h3><p>If Alice is not aware of any static key belonging to Bob (and neither Bob knows anything about Alice), she can execute an <code>XX</code> handshake, where each party tran<strong>X</strong>mits to the other its own static key.</p><p>The handshake goes as follows:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> XX:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; e</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> &lt;- e, ee, s, es</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; s, se</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>We note that the main difference with <code>XK1</code> is that in second step Bob sends to Alice his own static key encrypted with a key obtained from an ephemeral-ephemeral Diffie-Hellman exchange.</p><p>This handshake can be slightly changed in case both Alice and Bob pre-shares some secret <code>psk</code> which can be used to strengthen their mutual authentication during the handshake execution. One of the resulting protocol, called <code>XXpsk0</code>, goes as follow:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain"> XXpsk0:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; psk, e</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> &lt;- e, ee, s, es</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> -&gt; s, se</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>The main difference with <code>XX</code> is that Alice's and Bob's static keys, when transmitted, would be encrypted with a key derived from <code>psk</code> as well.</p><p><strong>Security considerations on identity-hiding (active attacker)</strong>: Alice's static key is encrypted with forward secrecy to an authenticated party for both <code>XX</code> and <code>XXpsk0</code> handshakes. In <code>XX</code>, Bob's static key is encrypted with forward secrecy but is transmitted to a non-authenticated user which can then be an active attacker. In <code>XXpsk0</code>, instead, Bob's secret key is protected by forward secrecy to a partially authenticated party (through the pre-shared secret <code>psk</code> but not through any static key), provided that <code>psk</code> was not previously compromised (in such case identity-hiding properties provided by the <code>XX</code> handshake applies).</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="session-management-and-multi-device-support">Session Management and Multi-Device Support<a href="#session-management-and-multi-device-support" class="hash-link" aria-label="Direct link to Session Management and Multi-Device Support" title="Direct link to Session Management and Multi-Device Support">​</a></h2><p>When two users complete a Noise handshake, an encryption/decryption session - or <em>Noise session</em> - consisting of two Cipher States is instantiated.</p><p>By identifying Noise session with a <code>session-id</code> derived from the handshake's cryptographic material, we can take advantage of the <a href="https://github.com/libp2p/specs/tree/master/pubsub" target="_blank" rel="noopener noreferrer">PubSub/GossipSub</a> protocols used by Waku for relaying messages in order to manage instantiated Noise sessions.</p><p>The core idea is to exchange after-handshake messages (encrypted with a Cipher State specific to the Noise session), over a content topic derived from the (secret) <code>session-id</code> the corresponding session refers to.</p><p>This allows to decouple the handshaking phase from the actual encrypted communication, thus improving users' identity-hiding capabilities.</p><p>Furthermore, by publicly revealing a value derived from <code>session-id</code> on the corresponding session content topic, a Noise session can be marked as <em>stale</em>, enabling peers to save resources by discarding any eventually <a href="https://rfc.vac.dev/spec/13/" target="_blank" rel="noopener noreferrer">stored</a> message sent to such content topic.</p><p>One relevant aspect in today's applications is the possibility for users to employ different devices in their communications. In some cases, this is non-trivial to achieve since, for example, encrypted messages might be required to be synced on different devices which do not necessarily share the necessary key material for decryption and may be temporarily offline.</p><p>We address this by requiring each user's device to instantiate multiple Noise sessions either with all user's other devices which, in turn, all together share a Noise session with the other party, or by directly instantiating a Noise session with all other party's devices.</p><p>We named these two approaches <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mn>11</mn><mi>M</mi></mrow><annotation encoding="application/x-tex">N11M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mord">11</span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mi>M</mi></mrow><annotation encoding="application/x-tex">NM</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">NM</span></span></span></span></span>, respectively, which are in turn loosely based on the paper <a href="https://eprint.iacr.org/2019/1363.pdf" target="_blank" rel="noopener noreferrer">“Multi-Device for Signal”</a> and <a href="https://signal.org/docs/specifications/sesame/" target="_blank" rel="noopener noreferrer">Signal’s Sesame Algorithm</a>.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/N11M-87240745a0c2c63e625cab407b6f0439.png" width="1321" height="934" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Informally, in the <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mn>11</mn><mi>M</mi></mrow><annotation encoding="application/x-tex">N11M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mord">11</span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span></span> session management scheme, once the first Noise session between any of Alice’s and Bob’s device is instantiated, its session information is securely propagated to all other devices using previously instantiated Noise sessions. Hence, all devices are able to send and receive new messages on the content topic associated to such session.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/NM-81c4d4cfe2ecb8a75dbdbfa450ebafa2.png" width="1321" height="934" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>In the <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mi>M</mi></mrow><annotation encoding="application/x-tex">NM</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">NM</span></span></span></span></span> session management scheme, instead, all pairs of Alice's and Bob's devices have a distinct Noise session: a message is then sent from the currently-in-use sender’s device to all recipient’s devices, by properly encrypting and sending it to the content topics of each corresponding Noise session. If sent messages should be available on all sender’s devices as well, we require each pair of sender’s devices to instantiate a Noise session used for syncing purposes.</p><p>For more technical details on how Noise sessions are instantiated and managed within these two mechanisms and the different trade-offs provided by the latter, we refer to <a href="https://rfc.vac.dev/spec/37/" target="_blank" rel="noopener noreferrer">37/WAKU2-NOISE-SESSIONS</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusions">Conclusions<a href="#conclusions" class="hash-link" aria-label="Direct link to Conclusions" title="Direct link to Conclusions">​</a></h2><p>In this post we provided an overview of Noise, a protocol framework for designing Diffie-Hellman based key-exchange mechanisms allowing systematic security and threat model analysis.</p><p>The flexibility provided by Noise components allows not only to fully replicate with same security guarantees well established key-exchange primitives such as X3DH, currently employed by Status <a href="https://specs.status.im/spec/5" target="_blank" rel="noopener noreferrer">5/TRANSPORT-SECURITY</a>, but enables also optimizations based on the reciprocal knowledge parties have of each other while allowing easier protocols' security analysis and (formal) verification.</p><p>Furthermore, different handshakes can be combined and executed one after each other, a particularly useful feature to authenticate multiple static keys employed by different applications but also to ease keys revocation.</p><p>The possibility to manage Noise sessions over multiple devices and the fact that handshakes can be concretely instantiated using modern, fast and secure cryptographic primitives such as <a href="https://datatracker.ietf.org/doc/html/rfc7539" target="_blank" rel="noopener noreferrer">ChaChaPoly</a> and <a href="https://datatracker.ietf.org/doc/html/rfc7693" target="_blank" rel="noopener noreferrer">BLAKE2b</a>, make Noise one of the best candidates for efficiently and securely address the many different needs of applications built on top of Waku requiring key-agreement.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-steps">Future steps<a href="#future-steps" class="hash-link" aria-label="Direct link to Future steps" title="Direct link to Future steps">​</a></h2><p>The available <a href="https://github.com/status-im/nwaku/tree/master/waku/v2/waku_noise" target="_blank" rel="noopener noreferrer">implementation</a> of Noise in <code>nwaku</code>, although mostly complete, is still in its testing phase. As future steps we would like to:</p><ul><li>have an extensively tested and robust Noise implementation;</li><li>formalize, implement and test performances of the two proposed <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mn>11</mn><mi>M</mi></mrow><annotation encoding="application/x-tex">N11M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mord">11</span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span></span> and <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mi>M</mi></mrow><annotation encoding="application/x-tex">NM</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">NM</span></span></span></span></span> session management mechanisms and their suitability for common use-case scenarios;</li><li>provide Waku network nodes a native protocol to readily support key-exchanges, strongly-encrypted communication and multi-device session management mechanisms with none-to-little interaction besides applications' connection requests.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://rfc.vac.dev/spec/6/" target="_blank" rel="noopener noreferrer">6/WAKU1</a></li><li><a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">10/WAKU2</a></li><li><a href="https://rfc.vac.dev/spec/13/" target="_blank" rel="noopener noreferrer">13/WAKU2-STORE</a></li><li><a href="https://rfc.vac.dev/spec/26/" target="_blank" rel="noopener noreferrer">26/WAKU-PAYLOAD</a></li><li><a href="https://rfc.vac.dev/spec/35/" target="_blank" rel="noopener noreferrer">35/WAKU2-NOISE</a></li><li><a href="https://rfc.vac.dev/spec/37/" target="_blank" rel="noopener noreferrer">37/WAKU2-NOISE-SESSIONS</a></li><li><a href="https://specs.status.im/spec/5" target="_blank" rel="noopener noreferrer">5/TRANSPORT-SECURITY</a></li><li><a href="https://github.com/libp2p/specs/tree/master/pubsub" target="_blank" rel="noopener noreferrer">The PubSub/GossipSub Protocols</a></li><li><a href="http://www.noiseprotocol.org/noise.html" target="_blank" rel="noopener noreferrer">The Noise Protocol Framework</a></li><li><a href="https://signal.org/docs/specifications/x3dh/" target="_blank" rel="noopener noreferrer">The X3DH Key-agreement Protocol</a></li><li><a href="https://eprint.iacr.org/2019/1363.pdf" target="_blank" rel="noopener noreferrer">“Multi-Device for Signal”</a></li><li><a href="https://signal.org/docs/specifications/sesame/" target="_blank" rel="noopener noreferrer">Signal’s Sesame Algorithm</a>.</li><li><a href="https://en.wikipedia.org/wiki/Public-key_cryptography" target="_blank" rel="noopener noreferrer">Public-key cryptography</a></li><li><a href="https://en.wikipedia.org/wiki/Elliptic_curve" target="_blank" rel="noopener noreferrer">Elliptic curves</a></li><li><a href="https://en.wikipedia.org/wiki/Elliptic_curve_point_multiplication" target="_blank" rel="noopener noreferrer">Elliptic Curve point multiplication</a></li><li><a href="https://en.wikipedia.org/wiki/Symmetric-key_algorithm" target="_blank" rel="noopener noreferrer">Symmetric key algorithm</a></li><li><a href="https://en.wikipedia.org/wiki/Authenticated_encryption" target="_blank" rel="noopener noreferrer">Authenticated encryption</a></li><li><a href="https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange" target="_blank" rel="noopener noreferrer">Diffie-Hellman Key-Exchange</a></li><li><a href="https://en.wikipedia.org/wiki/Discrete_logarithm" target="_blank" rel="noopener noreferrer">The Discrete Logarithm Problem</a></li><li><a href="https://en.wikipedia.org/wiki/Computational_Diffie%E2%80%93Hellman_assumption" target="_blank" rel="noopener noreferrer">Computational Diffie-Hellman Assumption</a></li><li><a href="https://en.wikipedia.org/wiki/Integrated_Encryption_Scheme" target="_blank" rel="noopener noreferrer">The ECIES Encryption Algorithm</a></li><li><a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm" target="_blank" rel="noopener noreferrer">The ECDSA Signature Algorithm</a></li><li><a href="https://en.wikipedia.org/wiki/Galois/Counter_Mode" target="_blank" rel="noopener noreferrer">The Galois Counter Mode mode of operation</a></li><li><a href="https://datatracker.ietf.org/doc/html/rfc7539" target="_blank" rel="noopener noreferrer">The ChaChaPoly AEAD Cipher</a></li><li><a href="https://datatracker.ietf.org/doc/html/rfc7693" target="_blank" rel="noopener noreferrer">The BLAKE2b Hash Function</a></li><li><a href="https://en.wikipedia.org/wiki/SHA-3" target="_blank" rel="noopener noreferrer">The SHA-3 Hash Function</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Waku v2 Ambient Peer Discovery]]></title>
<link>https://vac.dev/rlog/wakuv2-apd</link>
<guid>https://vac.dev/rlog/wakuv2-apd</guid>
<pubDate>Mon, 09 May 2022 10:00:00 GMT</pubDate>
<description><![CDATA[Introducing and discussing ambient peer discovery methods currently used by Waku v2, as well as future plans in this area.]]></description>
<content:encoded><![CDATA[<p>Introducing and discussing ambient peer discovery methods currently used by Waku v2, as well as future plans in this area.</p><p><a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Waku v2</a> comprises a set of modular protocols for secure, privacy preserving communication.
Avoiding centralization, these protocols exchange messages over a P2P network layer.
In order to build a P2P network, participating nodes first have to discover peers within this network.
This is where <a href="https://docs.libp2p.io/concepts/publish-subscribe/#discovery" target="_blank" rel="noopener noreferrer"><em>ambient peer discovery</em></a> comes into play:
it allows nodes to find peers, making it an integral part of any decentralized application.</p><p>In this post the term <em>node</em> to refers to <em>our</em> endpoint or the endpoint that takes action,
while the term <em>peer</em> refers to other endpoints in the P2P network.
These endpoints can be any device connected to the Internet: e.g. servers, PCs, notebooks, mobile devices, or applications like a browser.
As such, nodes and peers are the same. We use these terms for the ease of explanation without loss of generality.</p><p>In Waku's modular design, ambient peer discovery is an umbrella term for mechanisms that allow nodes to find peers.
Various ambient peer discovery mechanisms are supported, and each is specified as a separate protocol.
Where do these protocols fit into Waku's protocol stack?
The P2P layer of Waku v2 builds on <a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/README.md" target="_blank" rel="noopener noreferrer">libp2p gossipsub</a>.
Nodes participating in a gossipsub protocol manage a mesh network that is used for routing messages.
This mesh network is an <a href="https://en.wikipedia.org/wiki/Peer-to-peer#Unstructured_networks" target="_blank" rel="noopener noreferrer">unstructured P2P network</a>
offering high robustness and resilience against attacks.
Gossipsub implements many improvements overcoming the shortcomings typically associated with unstructured P2P networks, e.g. inefficient flooding based routing.
The gossipsub mesh network is managed in a decentralized way, which requires each node to know other participating peers.
Waku v2 may use any combination of its ambient discovery protocols to find appropriate peers.</p><p>Summarizing, Waku v2 comprises a <em>peer management layer</em> based on libp2p gossipsub,
which manages the peers of nodes, and an <em>ambient peer discovery layer</em>,
which provides information about peers to the peer management layer.</p><p>We focus on ambient peer discovery methods that are in line with our goal of building a fully decentralized, generalized, privacy-preserving and censorship-resistant messaging protocol.
Some of these protocols still need adjustments to adhere to our privacy and anonymity requirements. For now, we focus on operational stability and feasibility.
However, when choosing techniques, we pay attention to selecting mechanisms that can feasibly be tweaked for privacy in future research efforts.
Because of the modular design and the fact that Waku v2 has several discovery methods at its disposal, we could even remove a protocol in case future evaluation deems it not fitting our standards.</p><p>This post covers the current state and future considerations of ambient peer discovery for Waku v2,
and gives reason for changes and modifications we made or plan to make.
The ambient peer discovery protocols currently supported by Waku v2 are a modified version of Ethereum's <a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5.md" target="_blank" rel="noopener noreferrer">Discovery v5</a>
and <a href="https://vac.dev/dns-based-discovery" target="_blank" rel="noopener noreferrer">DNS-based discovery</a>.
Waku v2 further supports <a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange" target="_blank" rel="noopener noreferrer">gossipsub's peer exchange protocol</a>.
In addition, we plan to introduce protocols for general peer exchange and capability discovery, respectively.
The former allows resource restricted nodes to outsource querying for peers to stronger peers,
the latter allows querying peers for their supported capabilities.
Besides these new protocols, we are working on integrating capability discovery in our existing ambient peer discovery protocols.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="static-node-lists">Static Node Lists<a href="#static-node-lists" class="hash-link" aria-label="Direct link to Static Node Lists" title="Direct link to Static Node Lists">​</a></h2><p>The simplest method of learning about peers in a P2P network is via static node lists.
These can be given to nodes as start-up parameters or listed in a config-file.
They can also be provided in a script-parseable format, e.g. in JSON.
While this method of providing bootstrap nodes is very easy to implement, it requires static peers, which introduce centralized elements.
Also, updating static peer information introduces significant administrative overhead:
code and/or config files have to be updated and released.
Typically, static node lists only hold a small number of bootstrap nodes, which may lead to high load on these nodes.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dns-based-discovery">DNS-based Discovery<a href="#dns-based-discovery" class="hash-link" aria-label="Direct link to DNS-based Discovery" title="Direct link to DNS-based Discovery">​</a></h2><p>Compared to static node lists,
<a href="https://vac.dev/dns-based-discovery" target="_blank" rel="noopener noreferrer">DNS-based discovery</a> (specified in <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a>)
provides a more dynamic way of discovering bootstrap nodes.
It is very efficient, can easily be handled by resource restricted devices and provides very good availability.
In addition to a naive DNS approach, Ethereum's DNS-based discovery introduces efficient authentication leveraging <a href="https://en.wikipedia.org/wiki/Merkle_tree" target="_blank" rel="noopener noreferrer">Merkle trees</a>.</p><p>A further advantage over static node lists is the separation of code/release management and bootstrap node management.
However, changing and updating the list of bootstrap nodes still requires administrative privileges because DNS records have to be added or updated.</p><p>While this method of discovery still requires centralized elements,
node list management can be delegated to various DNS zones managed by other entities mitigating centralization.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="discovery-v5">Discovery V5<a href="#discovery-v5" class="hash-link" aria-label="Direct link to Discovery V5" title="Direct link to Discovery V5">​</a></h2><p>A much more dynamic method of ambient peer discovery is <a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5.md" target="_blank" rel="noopener noreferrer">Discovery v5</a>, which is Ethereum's peer discovery protocol.
It is based on the <a href="https://en.wikipedia.org/wiki/Kademlia" target="_blank" rel="noopener noreferrer">Kademlia</a> distributed hashtable (DHT).
An <a href="https://vac.dev/kademlia-to-discv5" target="_blank" rel="noopener noreferrer">introduction to discv5 and its history</a>, and a <a href="https://vac.dev/feasibility-discv5" target="_blank" rel="noopener noreferrer">discv5 Waku v2 feasibility study</a>
can be found in previous posts on this research log.</p><p>We use Discovery v5 as an ambient peer discovery method for Waku v2 because it is decentralized, efficient, actively researched, and has web3 as its main application area.
Discv5 also offers mitigation techniques for various attacks, which we cover later in this post.</p><p>Using a DHT (structured P2P network) as a means for ambient peer discovery, while using the gossipsub mesh network (unstructured P2P network) for transmitting actual messages,
Waku v2 leverages advantages from both worlds.
One of the main benefits of DHTs is offering a global view over participating nodes.
This, in turn, allows sampling random sets of nodes which is important for equally distributing load.
Gossipsub, on the other hand, offers great robustness and resilience against attacks.
Even if discv5 discovery should not work in advent of a DoS attack, Waku v2 can still operate switching to different discovery methods.</p><p>Discovery methods that use separate P2P networks still depend on bootstrapping,
which Waku v2 does via parameters on start-up or via DNS-based discovery.
This might raise the question of why such discovery methods are beneficial.
The answer lies in the aforementioned global view of DHTs. Without discv5 and similar methods, the bootstrap nodes are used as part of the gossipsub mesh.
This might put heavy load on these nodes and further, might open pathways to inference attacks.
Discv5, on the other hand, uses the bootstrap nodes merely as an entry to the discovery network and can provide random sets of nodes (sampled from a global view)
for bootstrapping or expanding the mesh.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dht-background">DHT Background<a href="#dht-background" class="hash-link" aria-label="Direct link to DHT Background" title="Direct link to DHT Background">​</a></h3><p>Distributed Hash Tables are a class of structured P2P overlay networks.
A DHT can be seen as a distributed node set of which each node is responsible for a part of the hash space.
In contrast to unstructured P2P networks, e.g. the mesh network maintained by gossipsub,
DHTs have a global view over the node set and the hash space (assuming the participating nodes behave well).</p><p>DHTs are susceptible to various kinds of attacks, especially <a href="https://en.wikipedia.org/wiki/Sybil_attack" target="_blank" rel="noopener noreferrer">Sybil attacks</a>
and <a href="https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/heilman" target="_blank" rel="noopener noreferrer">eclipse attacks</a>.
While security aspects have been addressed in various research papers, general practical solutions are not available.
However, discv5 introduced various practical mitigation techniques.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="random-walk-discovery">Random Walk Discovery<a href="#random-walk-discovery" class="hash-link" aria-label="Direct link to Random Walk Discovery" title="Direct link to Random Walk Discovery">​</a></h3><p>While discv5 is based on the Kademlia DHT, it only uses the <em>distributed node set</em> aspect of DHTs.
It does not map values (items) into the distributed hash space.
This makes sense, because the main purpose of discv5 is discovering other nodes that support discv5, which are expected to be Ethereum nodes.
Ethereum nodes that want to discover other Ethereum nodes simply query the discv5 network for a random set of peers.
If Waku v2 would do the same, only a small subset of the retrieved nodes would support Waku v2.</p><p>A first naive solution for Waku v2 discv5 discovery is</p><ul><li>retrieve a random node set, which is achieved by querying for a set of randomly chosen node IDs</li><li>filter the returned nodes on the query path based on Waku v2 capability via the <a href="https://rfc.vac.dev/spec/31/" target="_blank" rel="noopener noreferrer">Waku v2 ENR</a></li><li>repeat until enough Waku v2 capable nodes are found</li></ul><p>This query process boils down to random walk discovery, which is very resilient against attacks, but also very inefficient if the number of nodes supporting the desired capability is small.
We refer to this as the needle-in-the-haystack problem.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="random-walk-performance-estimation">Random Walk Performance Estimation<a href="#random-walk-performance-estimation" class="hash-link" aria-label="Direct link to Random Walk Performance Estimation" title="Direct link to Random Walk Performance Estimation">​</a></h3><p>This subsection provides a rough estimation of the overhead introduced by random walk discovery.</p><p>Given the following parameters:</p><ul><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span></span> number of total nodes participating in discv5</li><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span></span> percentage of nodes supporting Waku</li><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi></mrow><annotation encoding="application/x-tex">W</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span></span></span></span></span> the event of having at least one Waku node in a random sample</li><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></span> the size of a random sample (default = 16)</li><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span></span></span></span></span> the number of parallel queries started</li><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">b</span></span></span></span></span> bits per hop</li><li><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>q</mi></mrow><annotation encoding="application/x-tex">q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">q</span></span></span></span></span> the number of queries</li></ul><p>A query takes <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi><mi>o</mi><msub><mi>g</mi><msup><mn>2</mn><mi>b</mi></msup></msub><mi>n</mi></mrow><annotation encoding="application/x-tex">log_{2^b}n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8971em;vertical-align:-0.2026em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">o</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.4974em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mtight">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.782em"><span style="top:-2.786em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight">b</span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2026em"><span></span></span></span></span></span></span><span class="mord mathnormal">n</span></span></span></span></span> hops to retrieve a random sample of nodes.</p><p><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><mi>W</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>−</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><mi mathvariant="normal">/</mi><mn>100</mn><msup><mo stretchy="false">)</mo><mi>k</mi></msup></mrow><annotation encoding="application/x-tex">P(W) = 1 - (1-p/100)^k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord mathnormal">p</span><span class="mord">/100</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span></span> is the probability of having at least one Waku node in the sample.</p><p><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><msup><mi>W</mi><mi>q</mi></msup><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>−</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><mi mathvariant="normal">/</mi><mn>100</mn><msup><mo stretchy="false">)</mo><mrow><mi>k</mi><mi>q</mi></mrow></msup></mrow><annotation encoding="application/x-tex">P(W^q) = 1 - (1-p/100)^{kq}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord mathnormal">p</span><span class="mord">/100</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span></span></span></span></span></span> is the probability of having at least one Waku node in the union of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>q</mi></mrow><annotation encoding="application/x-tex">q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">q</span></span></span></span></span> samples.</p><p>Expressing this in terms of <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>q</mi></mrow><annotation encoding="application/x-tex">q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">q</span></span></span></span></span>, we can write:
<span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><msup><mi>W</mi><mi>q</mi></msup><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>−</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><mi mathvariant="normal">/</mi><mn>100</mn><msup><mo stretchy="false">)</mo><mrow><mi>k</mi><mi>q</mi></mrow></msup><mtext>  </mtext><mo>⟺</mo><mtext>  </mtext><mi>q</mi><mo>=</mo><mi>l</mi><mi>o</mi><msub><mi>g</mi><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><mi mathvariant="normal">/</mi><mn>100</mn><msup><mo stretchy="false">)</mo><mi>k</mi></msup></mrow></msub><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>P</mi><mo stretchy="false">(</mo><msup><mi>W</mi><mi>q</mi></msup><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">P(W^q) = 1 - (1-p/100)^{kq} \iff q = log_{(1-p/100)^k}(1-P(W^q))</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord mathnormal">p</span><span class="mord">/100</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">⟺</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.1276em;vertical-align:-0.3776em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">o</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.4974em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mtight">1</span><span class="mbin mtight">−</span><span class="mord mathnormal mtight">p</span><span class="mord mtight">/100</span><span class="mclose mtight"><span class="mclose mtight">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.782em"><span style="top:-2.786em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3776em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span><span class="mclose">))</span></span></span></span></span></p><p>Figure 1 shows a log-log plot for <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><msup><mi>W</mi><mi>q</mi></msup><mo stretchy="false">)</mo><mo>=</mo><mn>90</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">P(W^q) = 90\%</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8056em;vertical-align:-0.0556em"></span><span class="mord">90%</span></span></span></span></span>.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 1: log-log plot showing the number of queries necessary to retrieve a Waku v2 node with a probability of 90% in relation to the Waku v2 node concentration in the network." src="/assets/images/waku_v2_discv5_random_walk_estimation-671ba3d44404d97d719de8853a3cbbbb.svg" width="960" height="576" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Assuming <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.1</mn></mrow><annotation encoding="application/x-tex">p=0.1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">0.1</span></span></span></span></span>, we would need</p><p><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.9</mn><mo>=</mo><mn>1</mn><mo>−</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mn>0.1</mn><mi mathvariant="normal">/</mi><mn>100</mn><msup><mo stretchy="false">)</mo><mrow><mn>16</mn><mi>q</mi></mrow></msup><mo>=</mo><mo>&gt;</mo><mi>q</mi><mo>≈</mo><mn>144</mn></mrow><annotation encoding="application/x-tex">0.9 = 1 - (1-0.1/100)^{16q} =&gt; q \approx 144</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">0.9</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0641em;vertical-align:-0.25em"></span><span class="mord">0.1/100</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">16</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=&gt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6776em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">144</span></span></span></span></span></p><p>queries to get a Waku node with 90% probability, which leads to <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>≈</mo><mn>144</mn><mo>∗</mo><mn>18</mn><mo>=</mo><mn>2592</mn></mrow><annotation encoding="application/x-tex">\approx 144 * 18 = 2592</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4831em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">144</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">18</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">2592</span></span></span></span></span> overlay hops.
Choosing <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">b=3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">b</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">3</span></span></span></span></span> would reduce the number to <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>≈</mo><mn>144</mn><mo>∗</mo><mn>6</mn><mo>=</mo><mn>864</mn></mrow><annotation encoding="application/x-tex">\approx 144 * 6 = 864</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4831em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">144</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">6</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">864</span></span></span></span></span>.
Even when choosing <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>10</mn></mrow><annotation encoding="application/x-tex">\alpha = 10</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">10</span></span></span></span></span> we would have to wait at least 80 RTTs.
This effort is just for retrieving a single Waku node. Ideally, we want at least 3 Waku nodes for bootstrapping a Waku relay.</p><p><a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5-theory.md#ad-placement-and-topic-radius" target="_blank" rel="noopener noreferrer">The discv5 doc</a> roughly estimates <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">p=1%</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span></span> to be the threshold for acceptably efficient random walk discovery.
This is in line with our estimation:</p><p><span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.9</mn><mo>=</mo><mn>1</mn><mo>−</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mn>1</mn><mi mathvariant="normal">/</mi><mn>100</mn><msup><mo stretchy="false">)</mo><mrow><mn>16</mn><mi>q</mi></mrow></msup><mo>=</mo><mo>&gt;</mo><mi>q</mi><mo>≈</mo><mn>14</mn></mrow><annotation encoding="application/x-tex">0.9 = 1 - (1-1/100)^{16q} =&gt; q \approx 14</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">0.9</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0641em;vertical-align:-0.25em"></span><span class="mord">1/100</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">16</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=&gt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6776em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">14</span></span></span></span></span></p><p>The number of necessary queries is linearly dependent on the percentage <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span></span> of Waku nodes.
The number of hops per query is logarithmically dependent on <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span></span>.
Thus, random walk searching is inefficient for small percentages <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span></span>.
Still, random walks are more resilient against attacks.</p><p>We can conclude that a Waku node concentration below 1% renders vanilla discv5 unfit for our needs.
Our current solution and future plans for solving this issue are covered in the next subsections.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="simple-solution-separate-discovery-network">Simple Solution: Separate Discovery Network<a href="#simple-solution-separate-discovery-network" class="hash-link" aria-label="Direct link to Simple Solution: Separate Discovery Network" title="Direct link to Simple Solution: Separate Discovery Network">​</a></h3><p>The simple solution we currently use for <a href="https://rfc.vac.dev/spec/33/" target="_blank" rel="noopener noreferrer">Waku v2 discv5</a> is a separate discv5 network.
All (well behaving) nodes in this network support Waku v2, resulting in a very high query efficiency.
However, this solution reduces resilience because the difficulty of attacking a DHT scales with the number of participating nodes.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="discv5-topic-discovery">Discv5 Topic Discovery<a href="#discv5-topic-discovery" class="hash-link" aria-label="Direct link to Discv5 Topic Discovery" title="Direct link to Discv5 Topic Discovery">​</a></h3><p>We did not base our solution on the <a href="https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement" target="_blank" rel="noopener noreferrer">current version of discv5 topic discovery</a>,
because, similar to random walk discovery, it suffers from poor performance for relatively rare capabilities/topics.</p><p>However, there is <a href="https://github.com/harnen/service-discovery-paper" target="_blank" rel="noopener noreferrer">ongoing research</a> in discv5 topic discovery which is close to ideas we explored when pondering efficient and resilient Waku discv5 solutions.
We keep a close eye on this research, give feedback, and make suggestions, as we plan to switch to this version of topic discovery in the future.</p><p>In a nutshell, topic discovery will manage separate routing tables for each topic.
These topic specific tables are initialized with nodes from the discv5 routing table.
While the buckets of the discv5 routing table represent distance intervals from the node's <code>node ID</code>, the topic table buckets represent distance intervals from <code>topic ID</code>s.</p><p>Nodes that want to register a topic try to register that topic at one random peer per bucket.
This leads to registering the topic at peers in closer and closer neighbourhoods around the topic ID, which
yields a very efficient and resilient compromise between random walk discovery and DHT discovery.
Peers in larger neighbourhoods around the topic ID are less efficient to discover, however more resilient against eclipse attacks and vice versa.</p><p>Further, this works well with the overload and DoS protection discv5 employs.
Discv5 limits the amount of nodes registered per topic on a single peer. Further, discv5 enforces a waiting time before nodes can register topics at peers.
So, for popular topics, a node might fail to register the topic in a close neighbourhood.
However, because the topic is popular (has a high occurrence percentage <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span></span>), it can still be efficiently discovered.</p><p>In the future, we also plan to integrate Waku v2 capability discovery, which will not only allow asking for nodes that support Waku v2,
but asking for Waku v2 nodes supporting specific Waku v2 protocols like filter or store.
For the store protocol we envision sub-capabilities reflecting message topics and time frames of messages.
We will also investigate related security implications.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="attacks-on-dhts">Attacks on DHTs<a href="#attacks-on-dhts" class="hash-link" aria-label="Direct link to Attacks on DHTs" title="Direct link to Attacks on DHTs">​</a></h3><p>In this post, we only briefly describe common attacks on DHTs.
These attacks are mainly used for denial of service (DoS),
but can also used as parts of more sophisticated attacks, e.g. deanonymization attacks.
A future post on this research log will cover security aspects of ambient peer discovery with a focus on privacy and anonymity.</p><p><em>Sybil Attack</em></p><p>The power of an attacker in a DHT is proportional to the number of controlled nodes.
Controlling nodes comes at a high resource cost and/or requires controlling a botnet via a preliminary attack.</p><p>In a Sybil attack, an attacker generates lots of virtual node identities.
This allows the attacker to control a large portion of the ID space in a DHT at a relatively low cost.
Sybil attacks are especially powerful when the attacker can freely choose the IDs of generated nodes,
because this allows positioning at chosen points in the DHT.</p><p>Because Sybil attacks amplify the power of many attacks against DHTs,
making Sybil attacks as difficult as possible is the basis for resilient DHT operation.
The typical abstract mitigation approach is binding node identities to physical network interfaces.
To some extend, this can be achieved by introducing IP address based limits.
Further, generating node IDs can be bound by proof of work (PoW),
which, however, comes with a set of shortcomings, e.g. relatively high costs on resource restricted devices.
<a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5-rationale.md#sybil-and-eclipse-attacks" target="_blank" rel="noopener noreferrer">The discv5 doc</a>
describes both Sybil and eclipse attacks, as well as concrete mitigation techniques employed by discv5.</p><p><em>Eclipse Attack</em></p><p>In an eclipse attack, nodes controlled by the attacker poison the routing tables of other nodes in a way that parts of the DHT become eclipsed, i.e. invisible.
When a controlled node is asked for the next step in a path,
it provides another controlled node as the next step,
effectively navigating the querying node around or away from certain areas of the DHT.
While several mitigation techniques have been researched, there is no definitive protection against eclipse attacks available as of yet.
One mitigation technique is increasing <span class="math math-inline"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span></span></span></span></span>, the number of parallel queries, and following each concurrent path independently for the lookup.</p><p>The eclipse attack becomes very powerful in combination with a successful Sybil attack;
especially when the attacker can freely choose the position of the Sybil nodes.</p><p>The aforementioned new topic discovery of discv5 provides a good balance between protection against eclipse attacks and query performance.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="peer-exchange-protocol">Peer Exchange Protocol<a href="#peer-exchange-protocol" class="hash-link" aria-label="Direct link to Peer Exchange Protocol" title="Direct link to Peer Exchange Protocol">​</a></h2><p>While discv5 based ambient peer discovery has many desirable properties, resource restricted nodes and nodes behind restrictive NAT setups cannot run discv5 satisfactory.
With these nodes in mind, we started working on a simple <em>peer exchange protocol</em> based on ideas proposed <a href="https://github.com/libp2p/specs/issues/222" target="_blank" rel="noopener noreferrer">here</a>.
The peer exchange protocol will allow nodes to ask peers for additional peers.
Similar to discv5, the peer exchange protocol will also support capability discovery.</p><p>The new peer exchange protocol can be seen as a simple replacement for the <a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/rendezvous/README.md" target="_blank" rel="noopener noreferrer">Rendezvous protocol</a>, which Waku v2 does not support.
While the rendezvous protocol involves nodes registering at rendezvous peers, the peer exchange protocol simply allows nodes to ask any peer for a list of peers (with a certain set of capabilities).
Rendezvous tends to introduce centralized elements as rendezvous peers have a super-peer role.</p><p>In the future, we will investigate resource usage of <a href="https://rfc.vac.dev/spec/33/" target="_blank" rel="noopener noreferrer">Waku v2 discv5</a> and provide suggestions for minimal resources nodes should have to run discv5 satisfactory.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="further-protocols-related-to-discovery">Further Protocols Related to Discovery<a href="#further-protocols-related-to-discovery" class="hash-link" aria-label="Direct link to Further Protocols Related to Discovery" title="Direct link to Further Protocols Related to Discovery">​</a></h2><p>Waku v2 comprises further protocols related to ambient peer discovery. We shortly mention them for context, even though they are not strictly ambient peer discovery protocols.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gossipsub-peer-exchange-protocol">Gossipsub Peer Exchange Protocol<a href="#gossipsub-peer-exchange-protocol" class="hash-link" aria-label="Direct link to Gossipsub Peer Exchange Protocol" title="Direct link to Gossipsub Peer Exchange Protocol">​</a></h3><p>Gossipsub provides an integrated <a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange" target="_blank" rel="noopener noreferrer">peer exchange</a> mechanism which is also supported by Waku v2.
Gossipsub peer exchange works in a <em>push</em> manner. Nodes send peer lists to peers they prune from the active mesh.
This pruning is part of the gossipsub peer management, blurring the boundaries of <em>peer management</em> and <em>ambient peer discovery</em>.</p><p>We will investigate anonymity implications of this protocol and might disable it in favour of more anonymity-preserving protocols.
Sending a list of peers discloses information about the sending node.
We consider restricting these peer lists to cached peers that are currently not used in the active gossipsub mesh.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="capability-negotiation">Capability Negotiation<a href="#capability-negotiation" class="hash-link" aria-label="Direct link to Capability Negotiation" title="Direct link to Capability Negotiation">​</a></h3><p>Some of the ambient peer discovery methods used by Waku2 will support capability discovery.
This allows to narrow down the set of retrieved peers to peers that support specific capabilities.
This is efficient because it avoids establishing connections to nodes that we are not interested in.</p><p>However, the ambient discovery interface does not require capability discovery, which will lead to nodes having peers with unknown capabilities in their peer lists.
We work on a <em>capability negotiation protocol</em> which allows nodes to ask peers</p><ul><li>for their complete list of capabilities, and</li><li>whether they support a specific capability</li></ul><p>We will investigate security implications, especially when sending full capability lists.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="nat-traversal">NAT traversal<a href="#nat-traversal" class="hash-link" aria-label="Direct link to NAT traversal" title="Direct link to NAT traversal">​</a></h2><p>For <a href="https://docs.libp2p.io/concepts/nat/" target="_blank" rel="noopener noreferrer">NAT traversal</a>, Waku v2 currently supports the port mapping protocols <a href="https://en.wikipedia.org/wiki/Universal_Plug_and_Play" target="_blank" rel="noopener noreferrer">UPnP</a> and <a href="https://datatracker.ietf.org/doc/html/rfc6886" target="_blank" rel="noopener noreferrer">NAT-PMP</a> / <a href="https://datatracker.ietf.org/doc/html/rfc6887" target="_blank" rel="noopener noreferrer">PCP</a>.</p><p>In the future, we plan to add support for parts of <a href="https://datatracker.ietf.org/doc/html/rfc8445" target="_blank" rel="noopener noreferrer">ICE</a>, e.g. <a href="https://datatracker.ietf.org/doc/html/rfc7350" target="_blank" rel="noopener noreferrer">STUN</a>.
We do not plan to support <a href="https://www.rfc-editor.org/rfc/rfc5928" target="_blank" rel="noopener noreferrer">TURN</a> because TURN relays would introduce a centralized element.
A modified decentralized version of TURN featuring incentivization might be an option in the future;
strong peers could offer a relay service similar to TURN.</p><p>There are <a href="https://github.com/ethereum/devp2p/issues/199" target="_blank" rel="noopener noreferrer">plans to integrate more NAT traversal into discv5</a>, in which we might participate.
So far, the only traversal technique supported by discv5 is nodes receiving their external IP address in pong messages.</p><p>While NAT traversal is very important, adding more NAT traversal techniques is not a priority at the moment.
Nodes behind restrictive symmetric NAT setups cannot be discovered, but they can still discover peers in less restrictive setups.
While we wish to have as many nodes as possible to be discoverable via ambient peer discovery, two nodes behind a restrictive symmetric NAT can still exchange Waku v2 messages if they discovered a shared peer.
This is one of the nice resilience related properties of flooding based routing algorithms.</p><p>For mobile nodes, which suffer from changing IP addresses and double NAT setups, we plan using the peer exchange protocol to ask peers for more peers.
Besides saving resources on resource restricted devices, this approach works as long as peers are in less restrictive environments.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion-and-future-prospects">Conclusion and Future Prospects<a href="#conclusion-and-future-prospects" class="hash-link" aria-label="Direct link to Conclusion and Future Prospects" title="Direct link to Conclusion and Future Prospects">​</a></h2><p><em>Ambient peer discovery</em> is an integral part of decentralized applications. It allows nodes to learn about peers in the network.
As of yet, Waku v2 supports DNS-based discovery and a slightly modified version of discv5.
We are working on further protocols, including a peer exchange protocol that allows resource restricted nodes to ask stronger peers for peer lists.
Further, we are working on adding capability discovery to our ambient discovery protocols, allowing nodes to find peers with desired properties.</p><p>These protocols can be combined in a modular way and allow Waku v2 nodes to build a strong and resilient mesh network,
even if some discovery methods are not available in a given situation.</p><p>We will investigate security properties of these discovery mechanisms with a focus on privacy and anonymity in a future post on this research log.
As an outlook we can already state that DHT approaches typically allow inferring information about the querying node.
Further, sending peer lists allows inferring the position of a node within the mesh, and by extension information about the node.
Waku v2 already provides some mitigation, because the mesh for transmitting actual messages, and the peer discovery network are separate.
To mitigate information leakage by transmitting peer lists, we plan to only reply with lists of peers that nodes do not use in their active meshes.</p><hr><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Waku v2</a></li><li><a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/README.md" target="_blank" rel="noopener noreferrer">libp2p gossipsub</a></li><li><a href="https://en.wikipedia.org/wiki/Peer-to-peer#Unstructured_networks" target="_blank" rel="noopener noreferrer">unstructured P2P network</a></li><li><a href="https://docs.libp2p.io/concepts/publish-subscribe/#discovery" target="_blank" rel="noopener noreferrer">ambient peer discovery</a></li><li><a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5.md" target="_blank" rel="noopener noreferrer">Discovery v5</a></li><li><a href="https://en.wikipedia.org/wiki/Kademlia" target="_blank" rel="noopener noreferrer">Kademlia</a></li><li><a href="https://vac.dev/kademlia-to-discv5" target="_blank" rel="noopener noreferrer">Discv5 history</a></li><li><a href="https://vac.dev/feasibility-discv5" target="_blank" rel="noopener noreferrer">Discv5 Waku v2 feasibility study</a></li><li><a href="https://vac.dev/dns-based-discovery" target="_blank" rel="noopener noreferrer">DNS-based discovery</a></li><li><a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a></li><li><a href="https://en.wikipedia.org/wiki/Merkle_tree" target="_blank" rel="noopener noreferrer">Merkle trees</a></li><li><a href="https://en.wikipedia.org/wiki/Sybil_attack" target="_blank" rel="noopener noreferrer">Sybil attack</a></li><li><a href="https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/heilman" target="_blank" rel="noopener noreferrer">eclipse attack</a></li><li><a href="https://rfc.vac.dev/spec/31/" target="_blank" rel="noopener noreferrer">Waku v2 ENR</a></li><li><a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5-theory.md#ad-placement-and-topic-radius" target="_blank" rel="noopener noreferrer">Discv5 topic discovery</a></li><li><a href="https://github.com/harnen/service-discovery-paper" target="_blank" rel="noopener noreferrer">Discv5 paper</a></li><li><a href="https://github.com/ethereum/devp2p/blob/6b0abc3d956a626c28dce1307ee9f546db17b6bd/discv5/discv5-rationale.md#sybil-and-eclipse-attacks" target="_blank" rel="noopener noreferrer">Discv5 vs Sybil and eclipse attacks</a></li><li><a href="https://github.com/libp2p/specs/issues/222" target="_blank" rel="noopener noreferrer">peer exchange idea</a></li><li><a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/rendezvous/README.md" target="_blank" rel="noopener noreferrer">Rendezvous protocol</a></li><li><a href="https://rfc.vac.dev/spec/33/" target="_blank" rel="noopener noreferrer">Waku v2 discv5</a></li><li><a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange" target="_blank" rel="noopener noreferrer">Gossipsub peer exchange</a></li><li><a href="https://docs.libp2p.io/concepts/nat/" target="_blank" rel="noopener noreferrer">NAT traversal</a></li><li><a href="https://en.wikipedia.org/wiki/Universal_Plug_and_Play" target="_blank" rel="noopener noreferrer">UPnP</a></li><li><a href="https://datatracker.ietf.org/doc/html/rfc6886" target="_blank" rel="noopener noreferrer">NAT-PMP</a></li><li><a href="https://datatracker.ietf.org/doc/html/rfc6887" target="_blank" rel="noopener noreferrer">PCP</a>.</li><li><a href="https://github.com/ethereum/devp2p/issues/199" target="_blank" rel="noopener noreferrer">Discv5 topic efficiency issue</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Introducing nwaku]]></title>
<link>https://vac.dev/rlog/introducing-nwaku</link>
<guid>https://vac.dev/rlog/introducing-nwaku</guid>
<pubDate>Tue, 12 Apr 2022 10:00:00 GMT</pubDate>
<description><![CDATA[Introducing nwaku, a Nim-based Waku v2 client, including a summary of recent developments and preview of current and future focus areas.]]></description>
<content:encoded><![CDATA[<p>Introducing nwaku, a Nim-based Waku v2 client, including a summary of recent developments and preview of current and future focus areas.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background">Background<a href="#background" class="hash-link" aria-label="Direct link to Background" title="Direct link to Background">​</a></h2><p>If you've been following our <a href="https://vac.dev/research-log/" target="_blank" rel="noopener noreferrer">research log</a>,
you'll know that many things have happened in the world of Waku v2 since <a href="/waku-v2-ethereum-coscup">our last general update</a>.
In line with our <a href="https://vac.dev/#about" target="_blank" rel="noopener noreferrer">long term goals</a>,
we've introduced new protocols,
tweaked our existing protocols
and expanded our team.
We've also shown <a href="/waku-v1-v2-bandwidth-comparison">in a series of practical experiments</a> that Waku v2 does indeed deliver on some of the <a href="/waku-v2-plan">theoretical advantages</a> it was designed to have over its predecessor, Waku v1.
A <a href="https://forum.vac.dev/t/vac-sustainability-and-business-workshop/116" target="_blank" rel="noopener noreferrer">sustainability and business workshop</a> led to the formulation of a clearer vision for Vac as a team.</p><p>From the beginning, our protocol development has been complemented by various client implementations of these protocols,
first in <a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer">Nim</a>,
but later also in <a href="https://github.com/status-im/js-waku" target="_blank" rel="noopener noreferrer">JavaScript</a>
and <a href="https://github.com/status-im/go-waku" target="_blank" rel="noopener noreferrer">Go</a>.
A follow-up post will clarify the purposes, similarities and differences between these three clients.
The <a href="https://github.com/status-im/nim-waku/tree/d2fccb5220144893f994a67f2cc26661247f101f/waku/v2" target="_blank" rel="noopener noreferrer">Nim client</a>, is our reference implementation,
developed by the research team in parallel with the specs
and building on a home-grown implementation of <a href="https://github.com/status-im/nim-libp2p" target="_blank" rel="noopener noreferrer"><code>libp2p</code></a>.
The Nim client is suitable to run as <a href="/waku-update">a standalone adaptive node</a>,
managed by individual operators
or as an encapsulated service node in other applications.
This post looks at some recent developments within the Nim client.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-nim-waku-is-now-known-as-nwaku">1. <em><strong>nim-waku</strong></em> is now known as <em><strong>nwaku</strong></em><a href="#1-nim-waku-is-now-known-as-nwaku" class="hash-link" aria-label="Direct link to 1-nim-waku-is-now-known-as-nwaku" title="Direct link to 1-nim-waku-is-now-known-as-nwaku">​</a></h2><p>Pronounced NWHA-koo.
You may already have seen us refer to "<code>nwaku</code>" on Vac communication channels,
but it is now official:
The <code>nim-waku</code> Waku v2 client has been named <code>nwaku</code>.
Why? Well, we needed a recognizable name for our client that could easily be referred to in everyday conversations
and <code>nim-waku</code> just didn't roll off the tongue.
We've followed the example of the closely related <a href="https://github.com/status-im/nimbus-eth2" target="_blank" rel="noopener noreferrer"><code>nimbus</code> project</a> to find a punchier name
that explicitly links the client to both the Waku set of protocols and the Nim language.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-improvements-in-stability-and-performance">2. Improvements in stability and performance<a href="#2-improvements-in-stability-and-performance" class="hash-link" aria-label="Direct link to 2. Improvements in stability and performance" title="Direct link to 2. Improvements in stability and performance">​</a></h2><p>The initial implementation of Waku v2 demonstrated how the suite of protocols can be applied
to form a generalized, peer-to-peer messaging network,
while addressing a wide range of adaptive requirements.
This allowed us to lift several protocol <a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">specifications</a> from <code>raw</code> to <code>draft</code> status,
indicating that a reference implementation exists for each.
However, as internal dogfooding increased and more external applications started using <code>nwaku</code>,
we stepped up our focus on the client's stability and performance.
This is especially true where we want <code>nwaku</code> to run unsupervised in a production environment
without any degradation in the services it provides.</p><p>Some of the more significant productionization efforts over the last couple of months included:</p><ol><li><p>Reworking the <code>store</code> implementation to maintain stable memory usage
while storing historical messages
and serving multiple clients querying history simultaneously.
Previously, a <code>store</code> node would see gradual service degradation
due to inefficient memory usage when responding to history queries.
Queries that often took longer than 8 mins now complete in under 100 ms.</p></li><li><p>Improved peer management.
For example, <code>filter</code> nodes will now remove unreachable clients after a number of connection failures,
whereas they would previously keep accumulating dead peers.</p></li><li><p>Improved disk usage.
<code>nwaku</code> nodes that persist historical messages on disk now manage their own storage size based on the <code>--store-capacity</code>.
This can significantly improve node start-up times.</p></li></ol><p>More stability issues may be addressed in future as <code>nwaku</code> matures,
but we've noticed a marked improvement in the reliability of running <code>nwaku</code> nodes.
These include environments where <code>nwaku</code> nodes are expected to run with a long uptime.
Vac currently operates two long-running fleets of <code>nwaku</code> nodes, <code>wakuv2.prod</code> and <code>wakuv2.test</code>,
for internal dogfooding and
to serve as experimental bootstrapping nodes.
Status has also recently deployed similar fleets for production and testing based on <code>nwaku</code>.
Our goal is to have <code>nwaku</code> be stable, performant and flexible enough
to be an attractive option for operators to run and maintain their own Waku v2 nodes.
See also the <a href="#future-work">future work</a> section below for more on our general goal of <em><code>nwaku</code> for operators</em>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-improvements-in-interoperability">3. Improvements in interoperability<a href="#3-improvements-in-interoperability" class="hash-link" aria-label="Direct link to 3. Improvements in interoperability" title="Direct link to 3. Improvements in interoperability">​</a></h2><p>We've implemented several features that improve <code>nwaku</code>'s usability in different environments
and its interoperability with other Waku v2 clients.
One major step forward here was adding support for both secure and unsecured WebSocket connections as <code>libp2p</code> transports.
This allows direct connectivity with <code>js-waku</code>
and paves the way for native browser usage.
We've also added support for parsing and resolving DNS-type <code>multiaddrs</code>,
i.e. multiaddress protocol schemes <a href="https://github.com/multiformats/multiaddr/blob/b746a7d014e825221cc3aea6e57a92d78419990f/protocols.csv#L8-L11" target="_blank" rel="noopener noreferrer"><code>dns</code>, <code>dns4</code>, <code>dns6</code> and <code>dnsaddr</code></a>.
A <code>nwaku</code> node can now also be <a href="https://github.com/status-im/nim-waku/tree/d2fccb5220144893f994a67f2cc26661247f101f/waku/v2#configuring-a-domain-name" target="_blank" rel="noopener noreferrer">configured with its own IPv4 DNS domain name</a>
allowing dynamic IP address allocation without impacting a node's reachability by its peers.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-peer-discovery">4. Peer discovery<a href="#4-peer-discovery" class="hash-link" aria-label="Direct link to 4. Peer discovery" title="Direct link to 4. Peer discovery">​</a></h2><p><em>Peer discovery</em> is the method by which nodes become aware of each other’s existence.
The question of peer discovery in a Waku v2 network has been a focus area since the protocol was first conceptualized.
Since then several different approaches to discovery have been proposed and investigated.
We've implemented three discovery mechanisms in <code>nwaku</code> so far:</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dns-based-discovery">DNS-based discovery<a href="#dns-based-discovery" class="hash-link" aria-label="Direct link to DNS-based discovery" title="Direct link to DNS-based discovery">​</a></h3><p><code>nwaku</code> nodes can retrieve an authenticated, updateable list of peers via DNS to bootstrap connection to a Waku v2 network.
Our implementation is based on <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gossipsub-peer-exchange">GossipSub peer exchange<a href="#gossipsub-peer-exchange" class="hash-link" aria-label="Direct link to GossipSub peer exchange" title="Direct link to GossipSub peer exchange">​</a></h3><p><a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange" target="_blank" rel="noopener noreferrer">GossipSub Peer Exchange (PX)</a> is a GossipSub v1.1 mechanism
whereby a pruning peer may provide a pruned peer with a set of alternative peers
where it can connect to reform its mesh.
This is a very suitable mechanism to gradually discover more peers
from an initial connection to a small set of bootstrap peers.
It is enabled in a <code>nwaku</code> node by default.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-node-discovery-protocol-v5">Waku Node Discovery Protocol v5<a href="#waku-node-discovery-protocol-v5" class="hash-link" aria-label="Direct link to Waku Node Discovery Protocol v5" title="Direct link to Waku Node Discovery Protocol v5">​</a></h3><p>This is a DHT-based discovery mechanism adapted to store and relay <em>node records</em>.
Our implementation is based on <a href="https://github.com/ethereum/devp2p/blob/fa6428ada7385c13551873b2ae6ad2457c228eb8/discv5/discv5-theory.md" target="_blank" rel="noopener noreferrer">Ethereum's Discovery v5 protocol</a>
with some <a href="https://rfc.vac.dev/spec/33/" target="_blank" rel="noopener noreferrer">minor modifications</a> to isolate our discovery network from that of Ethereum.
The decision to separate the Waku Discovery v5 network from Ethereum's was made on considerations of lookup efficiency.
This comes at a possible tradeoff in network resilience.
We are considering merging with the Ethereum Discovery v5 network in future,
or even implement a hybrid solution.
<a href="https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121/8" target="_blank" rel="noopener noreferrer">This post</a> explains the decision and future steps.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-spam-protection-using-rln">5. Spam protection using RLN<a href="#5-spam-protection-using-rln" class="hash-link" aria-label="Direct link to 5. Spam protection using RLN" title="Direct link to 5. Spam protection using RLN">​</a></h2><p>An early addition to our suite of protocols was <a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">an extension of <code>11/WAKU-RELAY</code></a>
that provided spam protection using <a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">Rate Limiting Nullifiers (RLN)</a>.
The <code>nwaku</code> client now contains a working demonstration and integration of RLN relay.
Check out <a href="https://github.com/status-im/nim-waku/blob/ee96705c7fbe4063b780ac43b7edee2f6c4e351b/docs/tutorial/rln-chat2-live-testnet.md" target="_blank" rel="noopener noreferrer">this tutorial</a> to see the protocol in action using a toy chat application built on <code>nwaku</code>.
We'd love for people to join us in dogfooding RLN spam protection as part of our operator incentive testnet.
Feel free to join our <a href="https://discord.gg/KNj3ctuZvZ" target="_blank" rel="noopener noreferrer">Vac Discord</a> server
and head to the <code>#rln</code> channel for more information.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><p>As we continue working towards our goal of a fully decentralized, generalized and censorship-resistant messaging protocol,
these are some of the current and future focus areas for <code>nwaku</code>:</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="reaching-out-to-operators">Reaching out to operators:<a href="#reaching-out-to-operators" class="hash-link" aria-label="Direct link to Reaching out to operators:" title="Direct link to Reaching out to operators:">​</a></h3><p>We are starting to push for operators to run and maintain their own Waku v2 nodes,
preferably contributing to the default Waku v2 network as described by the default pubsub topic (<code>/waku/2/default-waku/proto</code>).
Amongst other things, a large fleet of stable operator-run Waku v2 nodes will help secure the network,
provide valuable services to a variety of applications
and ensure the future sustainability of both Vac as a research organization and the Waku suite of protocols.</p><p>We are targeting <code>nwaku</code> as the main option for operator-run nodes.<br>
<!-- -->Specifically, we aim to provide through <code>nwaku</code>:</p><ol><li>a lightweight and robust Waku v2 client.
This client must be first in line to support innovative and new Waku v2 protocols,
but configurable enough to serve the adaptive needs of various operators.</li><li>an easy-to-follow guide for operators to configure,
set up and maintain their own nodes</li><li>a set of operator-focused tools to monitor and maintain a running node</li></ol><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="better-conversational-security-layer-guarantees">Better conversational security layer guarantees<a href="#better-conversational-security-layer-guarantees" class="hash-link" aria-label="Direct link to Better conversational security layer guarantees" title="Direct link to Better conversational security layer guarantees">​</a></h3><p>Conversational security guarantees in Waku v2 are currently designed around the Status application.
Developers building their own applications on top of Waku would therefore
either have to reimplement a set of tools similar to Status
or build their own security solutions on the application layer above Waku.
We are working on <a href="https://github.com/vacp2p/research/issues/97" target="_blank" rel="noopener noreferrer">a set of features</a> built into Waku
that will provide the general security properties Waku users may desire
and do so in a modern and simple way.
This is useful for applications outside of Status that want similar security guarantees.
As a first step, we've already made good progress toward <a href="https://forum.vac.dev/t/noise-handshakes-as-key-exchange-mechanism-for-waku2/130" target="_blank" rel="noopener noreferrer">integrating noise handshakes</a> as a key exchange mechanism in Waku v2.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="protocol-incentivization">Protocol incentivization<a href="#protocol-incentivization" class="hash-link" aria-label="Direct link to Protocol incentivization" title="Direct link to Protocol incentivization">​</a></h3><p>We want to design incentivization around our protocols to encourage desired behaviors in the Waku network,
rewarding nodes providing costly services
and punishing adversarial actions.
This will increase the overall security of the network
and encourage operators to run their own Waku nodes.
In turn, the sustainability of Vac as an organization will be better guaranteed.
As such, protocol incentivization was a major focus in our recent <a href="https://forum.vac.dev/t/vac-sustainability-and-business-workshop/" target="_blank" rel="noopener noreferrer">Vac Sustainability and Business Workshop</a>.
Our first step here is to finish integrating RLN relay into Waku
with blockchain interaction to manage members,
punish spammers
and reward spam detectors.
After this, we want to design monetary incentivization for providers of <code>store</code>, <code>lightpush</code> and <code>filter</code> services.
This may also tie into a reputation mechanism for service nodes based on a network-wide consensus on service quality.
A big challenge for protocol incentivization is doing it in a private fashion,
so we can keep similar metadata protection guarantees as the Waku base layer.
This ties into our focus on <a href="https://forum.vac.dev/t/vac-3-zk/97" target="_blank" rel="noopener noreferrer">Zero Knowledge tech</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="improved-store-capacity">Improved store capacity<a href="#improved-store-capacity" class="hash-link" aria-label="Direct link to Improved store capacity" title="Direct link to Improved store capacity">​</a></h3><p>The <code>nwaku</code> store currently serves as an efficient in-memory store for historical messages,
dimensioned by the maximum number of messages the store node is willing to keep.
This makes the <code>nwaku</code> store appropriate for keeping history over a short term
without any time-based guarantees,
but with the advantage of providing fast responses to history queries.
Some applications, such as Status, require longer-term historical message storage
with time-based dimensioning
to guarantee that messages will be stored for a specified minimum period.
Because of the relatively high cost of memory compared to disk space,
a higher capacity store, with time guarantees, should operate as a disk-only database of historical messages.
This is an ongoing effort.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="multipurpose-discovery">Multipurpose discovery<a href="#multipurpose-discovery" class="hash-link" aria-label="Direct link to Multipurpose discovery" title="Direct link to Multipurpose discovery">​</a></h3><p>In addition to <a href="#4-peer-discovery">the three discovery methods</a> already implemented in <code>nwaku</code>,
we are working on improving discovery on at least three fronts:</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="capability-discovery"><em>Capability discovery:</em><a href="#capability-discovery" class="hash-link" aria-label="Direct link to capability-discovery" title="Direct link to capability-discovery">​</a></h4><p>Waku v2 nodes may be interested in peers with specific capabilities, for example:</p><ol><li>peers within a specific pubsub topic mesh,</li><li>peers with <strong>store</strong> capability,</li><li><strong>store</strong> peers with x days of history for a specific content topic, etc.</li></ol><p>Capability discovery entails mechanisms by which such capabilities can be advertised and discovered/negotiated.
One major hurdle to overcome is the increased complexity of finding a node with specific capabilities within the larger network (a needle in a haystack).
See the <a href="https://github.com/vacp2p/rfc/issues/429" target="_blank" rel="noopener noreferrer">original problem statement</a> for more.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="improvements-in-discovery-v5"><em>Improvements in Discovery v5</em><a href="#improvements-in-discovery-v5" class="hash-link" aria-label="Direct link to improvements-in-discovery-v5" title="Direct link to improvements-in-discovery-v5">​</a></h4><p>Of the implemented discovery methods,
Discovery v5 best addresses our need for a decentralized and scalable discovery mechanism.
With the basic implementation done,
there are some improvements planned for Discovery v5,
including methods to increase security such as merging with the Ethereum Discovery v5 network,
introducing explicit NAT traversal
and utilizing <a href="https://github.com/ethereum/devp2p/blob/fa6428ada7385c13551873b2ae6ad2457c228eb8/discv5/discv5-theory.md#topic-advertisement" target="_blank" rel="noopener noreferrer">topic advertisement</a>.
The <a href="https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121" target="_blank" rel="noopener noreferrer">Waku v2 Discovery v5 Roadmap</a> contains more details.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="generalized-peer-exchange"><em>Generalized peer exchange</em><a href="#generalized-peer-exchange" class="hash-link" aria-label="Direct link to generalized-peer-exchange" title="Direct link to generalized-peer-exchange">​</a></h4><p><code>nwaku</code> already implements <a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange" target="_blank" rel="noopener noreferrer">GossipSub peer exchange</a>.
We now need a general request-response mechanism outside of GossipSub
by which a node may learn about other Waku v2 nodes
by requesting and receiving a list of peers from a neighbor.
This could, for example, be a suitable way for resource-restricted devices to request a stronger peer
to perform a random Discovery v5 lookup on their behalf
or simply to be informed of a subset of the peers known to that neighbor.
See <a href="https://github.com/vacp2p/rfc/issues/495" target="_blank" rel="noopener noreferrer">this issue</a> for more.</p><hr><p>This concludes a general outline of some of the main recent developments in the <code>nwaku</code> client
and a summary of the current and future focus areas.
Much more is happening behind the scenes, of course,
so for more information, or to join the conversation,
feel free to join our <a href="https://discord.gg/KNj3ctuZvZ" target="_blank" rel="noopener noreferrer">Vac Discord</a> server
or to check out the <a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer"><code>nwaku</code> repo on Github</a>.
You can also view the changelog for past releases <a href="https://github.com/status-im/nim-waku/releases" target="_blank" rel="noopener noreferrer">here</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">17/WAKU-RLN-RELAY</a></li><li><a href="https://rfc.vac.dev/spec/32/" target="_blank" rel="noopener noreferrer">32/RLN</a></li><li><a href="https://rfc.vac.dev/spec/33/" target="_blank" rel="noopener noreferrer">33/WAKU2-DISCV5</a></li><li><a href="https://github.com/vacp2p/rfc/issues/429" target="_blank" rel="noopener noreferrer">Capabilities advertising</a></li><li><a href="https://github.com/status-im/nim-waku/tree/d2fccb5220144893f994a67f2cc26661247f101f/waku/v2#configuring-a-domain-name" target="_blank" rel="noopener noreferrer">Configuring a domain name</a></li><li><a href="https://github.com/vacp2p/research/issues/97" target="_blank" rel="noopener noreferrer">Conversational security</a></li><li><a href="https://github.com/ethereum/devp2p/blob/fa6428ada7385c13551873b2ae6ad2457c228eb8/discv5/discv5-theory.md#topic-advertisement" target="_blank" rel="noopener noreferrer">Discovery v5 Topic Advertisement</a></li><li><a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a></li><li><a href="https://github.com/libp2p/specs/blob/10712c55ab309086a52eec7d25f294df4fa96528/pubsub/gossipsub/gossipsub-v1.1.md#prune-backoff-and-peer-exchange" target="_blank" rel="noopener noreferrer">GossipSub Peer Exchange</a></li><li><a href="https://github.com/status-im/go-waku" target="_blank" rel="noopener noreferrer">go-waku</a></li><li><a href="https://github.com/status-im/js-waku" target="_blank" rel="noopener noreferrer">js-waku</a></li><li><a href="https://github.com/multiformats/multiaddr/blob/b746a7d014e825221cc3aea6e57a92d78419990f/protocols.csv#L8-L11" target="_blank" rel="noopener noreferrer"><code>multiaddr</code> formats</a></li><li><a href="https://github.com/status-im/nimbus-eth2" target="_blank" rel="noopener noreferrer">nimbus-eth2</a></li><li><a href="https://github.com/status-im/nim-libp2p" target="_blank" rel="noopener noreferrer">nim-libp2p</a></li><li><a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer">nim-waku</a></li><li><a href="https://github.com/status-im/nim-waku/releases" target="_blank" rel="noopener noreferrer">nim-waku releases</a></li><li><a href="https://github.com/ethereum/devp2p/blob/fa6428ada7385c13551873b2ae6ad2457c228eb8/discv5/discv5-theory.md" target="_blank" rel="noopener noreferrer">Node Discovery Protocol v5 - Theory</a></li><li><a href="https://forum.vac.dev/t/noise-handshakes-as-key-exchange-mechanism-for-waku2/130" target="_blank" rel="noopener noreferrer">Noise handshakes</a></li><li><a href="https://github.com/status-im/nim-waku/blob/ee96705c7fbe4063b780ac43b7edee2f6c4e351b/docs/tutorial/rln-chat2-live-testnet.md" target="_blank" rel="noopener noreferrer">RLN tutorial</a></li><li><a href="https://forum.vac.dev/t/vac-3-zk/97" target="_blank" rel="noopener noreferrer">Vac &lt;3 ZK</a></li><li><a href="https://vac.dev/#about" target="_blank" rel="noopener noreferrer">Vac About page</a></li><li><a href="https://vac.dev/research-log/" target="_blank" rel="noopener noreferrer">Vac Research log</a></li><li><a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">Vac RFC site</a></li><li><a href="https://forum.vac.dev/t/vac-sustainability-and-business-workshop/" target="_blank" rel="noopener noreferrer">Vac Sustainability and Business Workshop</a></li><li><a href="/waku-update">Waku Update</a></li><li><a href="/waku-v1-v2-bandwidth-comparison">Waku v1 vs Waku v2: Bandwidth Comparison</a></li><li><a href="https://github.com/vacp2p/rfc/issues/495" target="_blank" rel="noopener noreferrer">Waku v2 Peer Exchange</a></li><li><a href="https://forum.vac.dev/t/waku-v2-discv5-roadmap-discussion/121" target="_blank" rel="noopener noreferrer">Waku v2 Discovery v5 Roadmap</a></li><li><a href="/waku-v2-plan">What's the Plan for Waku v2?</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Opinion: Pseudo-ethics in the Surveillance Tech Industry]]></title>
<link>https://vac.dev/rlog/ethics-surveillance-tech</link>
<guid>https://vac.dev/rlog/ethics-surveillance-tech</guid>
<pubDate>Fri, 03 Dec 2021 10:00:00 GMT</pubDate>
<description><![CDATA[A look at typical ethical shortfalls in the global surveillance tech industry.]]></description>
<content:encoded><![CDATA[<p>A look at typical ethical shortfalls in the global surveillance tech industry.</p><p><em>This is an opinion piece by pseudonymous contributor, circe.</em></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="preface">Preface<a href="#preface" class="hash-link" aria-label="Direct link to Preface" title="Direct link to Preface">​</a></h2><p>The Vac team aims to provide a public good in the form of freely available, open source tools and protocols for decentralized communication.
As such, we value our independence and the usefulness of our protocols for a wide range of applications.
At the same time, we realize that all technical development, including ours, has a moral component.
As a diverse team we are guided by a shared devotion to the principles of human rights and liberty.
This explains why we place such a high premium on security, censorship-resistance and privacy -
a stance we <a href="https://our.status.im/our-principles/" target="_blank" rel="noopener noreferrer">share with the wider Status Network</a>.
The post below takes a different approach from our usual more technical analyses,
by starting to peel back the curtain on the ethical shortfalls of the global surveillance tech industry.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="spotlight-on-an-industry">Spotlight on an industry<a href="#spotlight-on-an-industry" class="hash-link" aria-label="Direct link to Spotlight on an industry" title="Direct link to Spotlight on an industry">​</a></h2><p><a href="https://www.apple.com/newsroom/2021/11/apple-sues-nso-group-to-curb-the-abuse-of-state-sponsored-spyware/" target="_blank" rel="noopener noreferrer">Apple's announcement</a> of their lawsuit against Israel's NSO Group
marks the latest in a series of recent setbacks for the surveillance tech company.
In early November, the <a href="https://public-inspection.federalregister.gov/2021-24123.pdf" target="_blank" rel="noopener noreferrer">United States blacklisted the firm</a>,
citing concerns about the use of their spyware by foreign governments targeting civilians such as "journalists, businesspeople, activists" and more.
The company is already <a href="https://www.reuters.com/article/us-facebook-cyber-whatsapp-nsogroup-idUSKBN1X82BE" target="_blank" rel="noopener noreferrer">embroiled in a lawsuit with Whatsapp</a>
over their exploit of the chat app's video calling service to install malware on target devices.
NSO Group's most infamous product, <a href="https://forbiddenstories.org/case/the-pegasus-project/" target="_blank" rel="noopener noreferrer">Pegasus</a>, operates as a hidden exploit installed on victims' mobile phones,
sometimes without even requiring as much as an unguarded click on a malicious link.
It has the potential to lay bare, and report to its owners, <em>everything</em> within the reach of the infected device.
For most people this amounts to a significant portion of their private lives and thoughts.
Pegasus can read your private messages (even encrypted), collect your passwords, record calls, track your location and access your device's microphone and camera.
No activity or application on an infected phone would be hidden.</p><p>The latest controversies are perhaps less because of the novelty of the revelations -
the existence of Pegasus has been known to civil activists <a href="https://www.bbc.com/news/technology-37192670" target="_blank" rel="noopener noreferrer">since at least 2016</a>.
Rather, the public was reminded again of the potential scope of surveillance tech
in the indiscriminate use of Pegasus on private citizens.
This has far-reaching implications for human freedoms worldwide.
Earlier this year, a <a href="https://www.theguardian.com/world/2021/jul/18/revealed-leak-uncovers-global-abuse-of-cyber-surveillance-weapon-nso-group-pegasus" target="_blank" rel="noopener noreferrer">leaked list of over 50,000 targets</a>, or possible targets, of Pegasus included
the phone numbers of human rights advocates, independent journalists, lawyers and political activists.
This should have come as no surprise.
The type of autocratically inclined agents, and governments, who would venture to buy and use such invasive cyber-arms often target those they find politically inconvenient.
Pegasus, and similar technologies, simply extend the reach and capacity of such individuals and governments -
no border or distance, no political rank or social advantage, no sanctity of profession or regard for dignity,
provide any indemnity from becoming a victim.
Your best hope is to remain uninteresting enough to escape consideration.</p><p>The NSO Group has, of course, denied allegations of culpability and questions the authenticity of the list.
At this stage, the latter is almost beside the point:
Amnesty International's cybersecurity team, Security Lab, <em>did</em> find <a href="https://www.amnesty.org/en/latest/research/2021/07/forensic-methodology-report-how-to-catch-nso-groups-pegasus/#_ftn1" target="_blank" rel="noopener noreferrer">forensic evidence of Pegasus</a> on the phones of several volunteers whose numbers appeared on the original list,
including those of journalists and human rights activists.
(Security Lab has since opened up their <a href="https://github.com/mvt-project/mvt" target="_blank" rel="noopener noreferrer">infection finding tool</a> to the public.)
French intelligence has similarly <a href="https://www.theguardian.com/news/2021/aug/02/pegasus-spyware-found-on-journalists-phones-french-intelligence-confirms" target="_blank" rel="noopener noreferrer">inspected and confirmed</a> infection of at least three devices belonging to journalists.
The phones of several people who were close to the Saudi-American journalist, Jamal Khashoggi, were <a href="https://www.bbc.com/news/world-57891506" target="_blank" rel="noopener noreferrer">confirmed hacked</a>
both before and after Khashoggi's brutal murder at the Saudi embassy in Istanbul in 2018.
<a href="https://www.theguardian.com/news/2021/sep/21/hungary-journalist-daniel-nemeth-phones-infected-with-nso-pegasus-spyware" target="_blank" rel="noopener noreferrer">More reports</a> of confirmed Pegasus hacks are still published with some regularity.
It is now an open secret that many authoritarian governments have bought Pegasus.
It's not difficult to extrapolate from existing reports and such clients' track records
what the potential injuries to human freedoms are that they can inflict with access to such a powerful cyberweapon.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-typical-response">A typical response<a href="#a-typical-response" class="hash-link" aria-label="Direct link to A typical response" title="Direct link to A typical response">​</a></h2><p><a href="https://www.theguardian.com/news/2021/jul/18/response-from-nso-and-governments" target="_blank" rel="noopener noreferrer">NSO's response</a> to the allegations follows a textbook approach
of avoiding earnest ethical introspection on the manufacturing, and selling, of cyber-arms.
Firstly, shift ethical responsibility to a predetermined process, a list of checkboxes of your own making.
The Group, for example, claims to sell only to "vetted governments", following a classification process
of which they have now <a href="https://www.nsogroup.com/wp-content/uploads/2021/06/ReportBooklet.pdf" target="_blank" rel="noopener noreferrer">published some procedural details</a> but no tangible criteria.
The next step is to reaffirm continuously, and repetitively, your dedication to the <em>legal</em> combat against crime,
<a href="https://www.nsogroup.com/wp-content/uploads/2021/06/ReportBooklet.pdf" target="_blank" rel="noopener noreferrer">"legitimate law enforcement agencies"</a> (note the almost tautological phrasing),
adherence to international arms trade laws,
compliance clauses in customer contracts, etc.
Thirdly, having been absolved of any moral suspicions that might exist about product and process,
from conception to engineering to trade,
distance yourself from the consequences of its use in the world.
<a href="https://www.theguardian.com/news/2021/jul/18/response-from-nso-and-governments" target="_blank" rel="noopener noreferrer">"NSO does not operate its technology, does not collect, nor possesses, nor has any access to any kind of data of its customers."</a>
It is interesting that directly after this statement they claim with contradictory confidence that
their "technology was not associated in any way with the heinous murder of Jamal Khashoggi".
The unapologetic tone seems hardly appropriate when the same document confirms that the Group had to
shut down customers' systems due to "confirmed misuse" and have had to do so "multiple times" in the past.
Given all this, the response manages to evade any serious interrogation of the "vetting" process itself,
which forced the company to reject "approximately 15% of potential new opportunities for Pegasus" in one year.
Courageous.</p><p>We have heard this all before.
There exists a multi-billion dollar industry of private companies and engineering firms <a href="https://www.economist.com/business/2019/12/12/offering-software-for-snooping-to-governments-is-a-booming-business" target="_blank" rel="noopener noreferrer">thriving on proceeds</a> from
selling surveillance tools and cyber-arms to dubious agencies and foreign governments.
In turn, the most power-hungry and oppressive regimes often <em>rely</em> on such technological innovations -
for which they lack the in-country engineering expertise -
to maintain control, suppress uprisings, intimidate opposing journalists, and track their citizens.
It's a lucrative business opportunity, and resourceful companies have sprung up everywhere to supply this demand,
often in countries where citizens, including employees of the company, would be horrified if they were similarly subject to the oppressions of their own products.
When, in 2014, Italy's <em>HackingTeam</em> were pulsed by the United Nations about their (then alleged) selling of spyware to Sudan,
which would have been a contravention of the UN's weapon export ban,
they simply replied that their product was not controlled as a weapon and therefore not subject to such scrutiny.
They remained within their legal bounds, technically.
Furthermore, they similarly shifted ethical responsibility to external standards of legitimacy,
claiming their <a href="https://citizenlab.ca/2014/02/mapping-hacking-teams-untraceable-spyware/" target="_blank" rel="noopener noreferrer">"software is not sold to governments that are blacklisted by the EU, the US, NATO, and similar international organizations"</a>.
When the company themselves were <a href="https://www.wired.com/2015/07/hacking-team-breach-shows-global-spying-firm-run-amok/" target="_blank" rel="noopener noreferrer">hacked in 2015</a>,
revelations (confirmations, that is) of widespread misuse by repressive governments were damaging enough to force them to disappear and rebrand as Memento Labs.
<a href="https://www.mem3nt0.com/en/" target="_blank" rel="noopener noreferrer">Their website</a> boasts an impressive list of statutes, regulations, procedures, export controls and legal frameworks,
all of which the rebranded hackers proudly comply with.
Surely no further ethical scrutiny is necessary?</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="ethics--the-law">Ethics != the law<a href="#ethics--the-law" class="hash-link" aria-label="Direct link to Ethics != the law" title="Direct link to Ethics != the law">​</a></h2><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-law-is-trailing-behind">The law is trailing behind<a href="#the-law-is-trailing-behind" class="hash-link" aria-label="Direct link to The law is trailing behind" title="Direct link to The law is trailing behind">​</a></h3><p>Such recourse to the <em>legality</em> of your action as ethical justification is moot for several reasons.
The first is glaringly obvious -
our laws are ill-equipped to address the implications of modern technology.
Legal systems are a cumbersome inheritance built over generations.
This is especially true of the statutes and regulations governing international trade, behind which these companies so often hide.
Our best legal systems are trailing miles behind the technology for which we seek guidelines.
Legislators are still struggling to make sense of technologies like face recognition,
the repercussions of smart devices acting "on their own" and biases in algorithms.
To claim you are performing ethical due diligence by resorting to an outdated and incomplete system of legal codes is disingenuous.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-law-depends-on-ethics">The law depends on ethics<a href="#the-law-depends-on-ethics" class="hash-link" aria-label="Direct link to The law depends on ethics" title="Direct link to The law depends on ethics">​</a></h3><p>The second reason is more central to my argument,
and an important flaw in these sleight of hand justifications appearing from time to time in the media.
Ethics can in no way be confused as synonymous with legality or legitimacy.
These are incommensurable concepts.
In an ideal world, of course, the law is meant to track the minimum standards of ethical conduct in a society.
Laws are often drafted exactly from some ethical, and practical, impulse to minimize harmful conduct
and provide for corrective and punitive measures where transgressions do occur.
The law, however, has a much narrower scope than ethics.
It can be just or unjust.
In fact, it is in need of ethics to constantly reform.
Ethics and values are born out of collective self-reflection.
It develops in our conversation with ourselves and others about the type of society we strive for.
As such, an ethical worldview summarizes our deepest intuitions about how we should live and measure our impact on the world.
For this reason, ethics is primarily enforced by social and internal pressures, not legal boundaries -
our desire to do what <em>ought</em> to be done, however we define that.
Ethics is therefore a much grander scheme than global legal systems
and the diplomatic frameworks that grants legitimacy to governments.
These are but one limited outflow of the human aspiration to form societies in accordance with our ideologies and ethics.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="international-law-is-vague-and-exploitable">International law is vague and exploitable<a href="#international-law-is-vague-and-exploitable" class="hash-link" aria-label="Direct link to International law is vague and exploitable" title="Direct link to International law is vague and exploitable">​</a></h3><p>Of course, the cyber-arms trade has a favorite recourse, <em>international</em> law, which is even more limited.
Since such products are seldomly sold to governments and agencies within the country of production,
it enables a further distancing from consequences.
Many private surveillance companies are based in fairly liberal societies with (seemingly) strict emphases on human rights in their domestic laws.
International laws are much more complicated - for opportunists a synonym for "more grey areas in which to hide".
Company conduct can now be governed, and excused, by a system that follows
the whims of autocrats with exploitative intent and vastly different ethical conceptions from the company's purported aims.
International law, and the ways it is most often enforced by way of, say, UN-backed sanctions,
have long been shaped by the compromises of international diplomacy.
To be blunt: these laws are weak and subject to exactly the sort of narrow interests behind which mercenaries have always hidden.
The surveillance tech industry is no exception.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2><p>My point is simple:
selling cyber-arms with the potential to become vast tools of oppression to governments and bodies with blatant histories of human rights violations,
and all but the publicly announced intention to continue operating in this way,
is categorically unconscionable.
This seems obvious no matter what ethics system you argue from,
provided it harbors any consideration for human dignity and freedom.
It is a sign of poor moral discourse that such recourses to law and legitimacy are often considered synonymous with ethical justification.
"<em>I have acted within the bounds of law</em>", <em>"We supply only to legitimate law enforcement agencies"</em>, etc. are no substitutes.
Ethical conduct requires an honest evaluation of an action against some conception of "the good",
however you define that.
Too often the surveillance tech industry precisely sidesteps this question,
both in internal processes and external rationalisations to a concerned public.</p><p>John Locke, he of the life-liberty-and-property, articulated the idea that government exists solely through the consent of the governed.
Towards the end of the 17th century, he wrote in his <em>Second Treatise on Civil Government</em>,
"<!-- -->[w]<!-- -->henever legislators endeavor to take away,
and destroy the property of the people, or to reduce them to slavery under arbitrary power,
they put themselves in a state of war with the people, who are thereupon absolved from any further obedience".
The inference is straightforward and humanist in essence:
legitimacy is not something that is conferred by governments and institutions.
Rather, they derive their legitimacy from us, their citizens, holding them to standards of ethics and societal ideals.
This legitimacy only remains in tact as long as this mandate is honored and continuously extended by a well-informed public.
This is the principle of informed consent on which all reciprocal ethics is based.</p><p>The surveillance tech industry may well have nothing more or less noble in mind than profit-making within legal bounds
when developing and selling their products.
However, when such companies are revealed again and again to have supplied tools of gross human rights violations to known human rights violators,
they will do well to remember that ethics always <em>precedes</em> requirements of legality and legitimacy.
It is a fallacy to take normative guidance from the concept of "legitimacy"
if the concept itself depends on such normative guidelines for definition.
Without examining the ethical standards by which institutions, governments, and laws, were created,
no value-judgements about their legitimacy can be made.
Hiding behind legal compliance as substitute for moral justification is not enough.
Targets of increasingly invasive governmental snooping are too often chosen precisely to suppress the mechanisms from which the legitimacy of such governments flow -
the consent of ordinary civilians.
Free and fair elections, free speech, free media, freedom of thought are all at risk.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://our.status.im/our-principles/" target="_blank" rel="noopener noreferrer">Status Principles</a></li><li><a href="https://public-inspection.federalregister.gov/2021-24123.pdf" target="_blank" rel="noopener noreferrer">Federal Register: Addition of Certain Entities to the Entity List</a></li><li><a href="https://forbiddenstories.org/case/the-pegasus-project/" target="_blank" rel="noopener noreferrer">forbiddenstories.org: The Pegasus Project</a></li><li><a href="https://www.theguardian.com/news/series/pegasus-project" target="_blank" rel="noopener noreferrer">theguardian.com: The Pegasus Project</a></li><li><a href="https://www.amnesty.org/en/latest/research/2021/07/forensic-methodology-report-how-to-catch-nso-groups-pegasus/#_ftn1" target="_blank" rel="noopener noreferrer">amnesty.org Forensic Methodology Report: How to catch NSO Group’s Pegasus</a></li><li><a href="https://www.apple.com/newsroom/2021/11/apple-sues-nso-group-to-curb-the-abuse-of-state-sponsored-spyware/" target="_blank" rel="noopener noreferrer">Apple sues NSO Group to curb the abuse of state-sponsored spyware</a></li><li><a href="https://www.bbc.com/news/technology-37192670" target="_blank" rel="noopener noreferrer">bbc.com: Who are the hackers who cracked the iPhone?</a></li><li><a href="https://www.bbc.com/news/world-57891506" target="_blank" rel="noopener noreferrer">bbc.com: Pegasus: Who are the alleged victims of spyware targeting?</a></li><li><a href="https://citizenlab.ca/2014/02/mapping-hacking-teams-untraceable-spyware/" target="_blank" rel="noopener noreferrer">citizenlab.ca: Mapping Hacking Team’s “Untraceable” Spyware</a></li><li><a href="https://www.economist.com/business/2019/12/12/offering-software-for-snooping-to-governments-is-a-booming-business" target="_blank" rel="noopener noreferrer">economist.com: Offering software for snooping to governments is a booming business</a></li><li><a href="https://www.mem3nt0.com/en/" target="_blank" rel="noopener noreferrer">Memento Labs</a></li><li><a href="https://github.com/mvt-project/mvt" target="_blank" rel="noopener noreferrer">Mobile Verification Toolkit to identify compromised devices</a></li><li><a href="https://www.nsogroup.com/wp-content/uploads/2021/06/ReportBooklet.pdf" target="_blank" rel="noopener noreferrer">NSO Group: Transparency and Responsibility Report 2021</a></li><li><a href="https://www.reuters.com/article/us-facebook-cyber-whatsapp-nsogroup-idUSKBN1X82BE" target="_blank" rel="noopener noreferrer">reuters.com: WhatsApp sues Israel's NSO for allegedly helping spies hack phones around the world</a></li><li><a href="https://www.wired.com/2015/07/hacking-team-breach-shows-global-spying-firm-run-amok/" target="_blank" rel="noopener noreferrer">wired.com: Hacking Team Breach Shows a Global Spying Firm Run Amok</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[Waku v1 vs Waku v2: Bandwidth Comparison]]></title>
<link>https://vac.dev/rlog/waku-v1-v2-bandwidth-comparison</link>
<guid>https://vac.dev/rlog/waku-v1-v2-bandwidth-comparison</guid>
<pubDate>Wed, 03 Nov 2021 10:00:00 GMT</pubDate>
<description><![CDATA[A local comparison of bandwidth profiles showing significantly improved scalability in Waku v2 over Waku v1.]]></description>
<content:encoded><![CDATA[<p>A local comparison of bandwidth profiles showing significantly improved scalability in Waku v2 over Waku v1.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background">Background<a href="#background" class="hash-link" aria-label="Direct link to Background" title="Direct link to Background">​</a></h2><p>The <a href="https://vac.dev/waku-v2-plan" target="_blank" rel="noopener noreferrer">original plan</a> for Waku v2 suggested theoretical improvements in resource usage over Waku v1,
mainly as a result of the improved amplification factors provided by GossipSub.
In its turn, <a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">Waku v1 proposed improvements</a> over its predecessor, Whisper.</p><p>Given that Waku v2 is aimed at resource restricted environments,
we are specifically interested in its scalability and resource usage characteristics.
However, the theoretical performance improvements of Waku v2 over Waku v1,
has never been properly benchmarked and tested.</p><p>Although we're working towards a full performance evaluation of Waku v2,
this would require significant planning and resources,
if it were to simulate "real world" conditions faithfully and measure bandwidth and resource usage across different network connections,
robustness against attacks/losses, message latencies, etc.
(There already exists a fairly comprehensive <a href="https://research.protocol.ai/publications/gossipsub-v1.1-evaluation-report/vyzovitis2020.pdf" target="_blank" rel="noopener noreferrer">evaluation of GossipSub v1.1</a>,
on which <a href="https://rfc.vac.dev/spec/11/" target="_blank" rel="noopener noreferrer"><code>11/WAKU2-RELAY</code></a> is based.)</p><p>As a starting point,
this post contains a limited and local comparison of the <em>bandwidth</em> profile (only) between Waku v1 and Waku v2.
It reuses and adapts existing network simulations for <a href="https://github.com/status-im/nim-waku/blob/master/waku/v1/node/quicksim.nim" target="_blank" rel="noopener noreferrer">Waku v1</a> and <a href="https://github.com/status-im/nim-waku/blob/master/waku/v2/node/quicksim2.nim" target="_blank" rel="noopener noreferrer">Waku v2</a>
and compares bandwidth usage for similar message propagation scenarios.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="theoretical-improvements-in-waku-v2">Theoretical improvements in Waku v2<a href="#theoretical-improvements-in-waku-v2" class="hash-link" aria-label="Direct link to Theoretical improvements in Waku v2" title="Direct link to Theoretical improvements in Waku v2">​</a></h2><p>Messages are propagated in Waku v1 using <a href="https://en.wikipedia.org/wiki/Flooding_(computer_networking)" target="_blank" rel="noopener noreferrer">flood routing</a>.
This means that every peer will forward every new incoming message to all its connected peers (except the one it received the message from).
This necessarily leads to unnecessary duplication (termed <em>amplification factor</em>),
wasting bandwidth and resources.
What's more, we expect this effect to worsen the larger the network becomes,
as each <em>connection</em> will receive a copy of each message,
rather than a single copy per peer.</p><p>Message routing in Waku v2 follows the <code>libp2p</code> <em>GossipSub</em> protocol,
which lowers amplification factors by only sending full message contents to a subset of connected peers.
As a Waku v2 network grows, each peer will limit its number of full-message ("mesh") peerings -
<code>libp2p</code> suggests a maximum of <code>12</code> such connections per peer.
This allows much better scalability than a flood-routed network.
From time to time, a Waku v2 peer will send metadata about the messages it has seen to other peers ("gossip" peers).</p><p>See <a href="https://hackmd.io/@vac/main/%2FYYlZYBCURFyO_ZG1EiteWg#11WAKU2-RELAY-gossipsub" target="_blank" rel="noopener noreferrer">this explainer</a> for a more detailed discussion.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="methodology">Methodology<a href="#methodology" class="hash-link" aria-label="Direct link to Methodology" title="Direct link to Methodology">​</a></h2><p>The results below contain only some scenarios that provide an interesting contrast between Waku v1 and Waku v2.
For example, <a href="https://en.wikipedia.org/wiki/Star_network" target="_blank" rel="noopener noreferrer">star network topologies</a> do not show a substantial difference between Waku v1 and Waku v2.
This is because each peer relies on a single connection to the central node for every message,
which barely requires any routing:
each connection receives a copy of every message for both Waku v1 and Waku v2.
Hybrid topologies similarly show only a difference between Waku v1 and Waku v2 for network segments with <a href="https://en.wikipedia.org/wiki/Mesh_networking" target="_blank" rel="noopener noreferrer">mesh-like connections</a>,
where routing decisions need to be made.</p><p>For this reason, the following approach applies to all iterations:</p><ol><li>Simulations are run <strong>locally</strong>.
This limits the size of possible scenarios due to local resource constraints,
but is a way to quickly get an approximate comparison.</li><li>Nodes are treated as a <strong>blackbox</strong> for which we only measure bandwidth,
using an external bandwidth monitoring tool.
In other words, we do not consider differences in the size of the envelope (for v1) or the message (for v2).</li><li>Messages are published at a rate of <strong>50 new messages per second</strong> to each network,
except where explicitly stated otherwise.</li><li>Each message propagated in the network carries <strong>8 bytes</strong> of random payload, which is <strong>encrypted</strong>.
The same symmetric key cryptographic algorithm (with the same keys) are used in both Waku v1 and v2.</li><li>Traffic in each network is <strong>generated from 10 nodes</strong> (randomly-selected) and published in a round-robin fashion to <strong>10 topics</strong> (content topics for Waku v2).
In practice, we found no significant difference in <em>average</em> bandwidth usage when tweaking these two parameters (the number of traffic generating nodes and the number of topics).</li><li>Peers are connected in a decentralized <strong>full mesh topology</strong>,
i.e. each peer is connected to every other peer in the network.
Waku v1 is expected to flood all messages across all existing connections.
Waku v2 gossipsub will GRAFT some of these connections for full-message peerings,
with the rest being gossip-only peerings.</li><li>After running each iteration, we <strong>verify that messages propagated to all peers</strong> (comparing the number of published messages to the metrics logged by each peer).</li></ol><p>For Waku v1, nodes are configured as "full" nodes (i.e. with full bloom filter),
while Waku v2 nodes are <code>relay</code> nodes, all subscribing and publishing to the same PubSub topic.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="network-size-comparison">Network size comparison<a href="#network-size-comparison" class="hash-link" aria-label="Direct link to Network size comparison" title="Direct link to Network size comparison">​</a></h2><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iteration-1-10-nodes">Iteration 1: 10 nodes<a href="#iteration-1-10-nodes" class="hash-link" aria-label="Direct link to Iteration 1: 10 nodes" title="Direct link to Iteration 1: 10 nodes">​</a></h3><p>Let's start with a small network of 10 nodes only and see how Waku v1 bandwidth usage compares to that of Waku v2.
At this small scale we don't expect to see improved bandwidth usage in Waku v2 over Waku v1,
since all connections, for both Waku v1 and Waku v2, will be full-message connections.
The number of connections is low enough that Waku v2 nodes will likely GRAFT all connections to full-message peerings,
essentially flooding every message on every connection in a similar fashion to Waku v1.
If our expectations are confirmed, it helps validate our methodology,
showing that it gives more or less equivalent results between Waku v1 and Waku v2 networks.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-10-nodes-fcc807080c17463099e65069a7580532.png" width="1243" height="734" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Sure enough, the figure shows that in this small-scale setup,
Waku v1 actually has a lower per-peer bandwidth usage than Waku v2.
One reason for this may be the larger overall proportion of control messages in a gossipsub-routed network such as Waku v2.
These play a larger role when the total network traffic is comparatively low, as in this iteration.
Also note that the average bandwidth remains more or less constant as long as the rate of published messages remains stable.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iteration-2-30-nodes">Iteration 2: 30 nodes<a href="#iteration-2-30-nodes" class="hash-link" aria-label="Direct link to Iteration 2: 30 nodes" title="Direct link to Iteration 2: 30 nodes">​</a></h3><p>Now, let's run the same scenario for a larger network of highly-connected nodes, this time consisting of 30 nodes.
At this point, the Waku v2 nodes will start pruning some connections to limit the number of full-message peerings (to a maximum of <code>12</code>),
while the Waku v1 nodes will continue flooding messages to all connected peers.
We therefore expect to see a somewhat improved bandwidth usage in Waku v2 over Waku v1.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-30-nodes-dc614f343b395e41c450f67e7e753881.png" width="1240" height="733" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Bandwidth usage in Waku v2 has increased only slightly from the smaller network of 10 nodes (hovering between 2000 and 3000 kbps).
This is because there are only a few more full-message peerings than before.
Compare this to the much higher increase in bandwidth usage for Waku v1, which now requires more than 4000 kbps on average.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iteration-3-50-nodes">Iteration 3: 50 nodes<a href="#iteration-3-50-nodes" class="hash-link" aria-label="Direct link to Iteration 3: 50 nodes" title="Direct link to Iteration 3: 50 nodes">​</a></h3><p>For an even larger network of 50 highly connected nodes,
the divergence between Waku v1 and Waku v2 is even larger.
The following figure shows comparative average bandwidth usage for a throughput of 50 messages per second.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-50-nodes-7facf8c03b87e2ebb2dd3967fac6e6a0.png" width="1241" height="735" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Average bandwidth usage (for the same message rate) has remained roughly the same for Waku v2 as it was for 30 nodes,
indicating that the number of full-message peerings per node has not increased.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iteration-4-85-nodes">Iteration 4: 85 nodes<a href="#iteration-4-85-nodes" class="hash-link" aria-label="Direct link to Iteration 4: 85 nodes" title="Direct link to Iteration 4: 85 nodes">​</a></h3><p>We already see a clear trend in the bandwidth comparisons above,
so let's confirm by running the test once more for a network of 85 nodes.
Due to local resource constraints, the effective throughput for Waku v1 falls to below 50 messages per second,
so the v1 results below have been normalized and are therefore approximate.
The local Waku v2 simulation maintains the message throughput rate without any problems.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-85-nodes-a4c56b478f03d471ae25bc36d6087bbf.png" width="1237" height="732" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iteration-5-150-nodes">Iteration 5: 150 nodes<a href="#iteration-5-150-nodes" class="hash-link" aria-label="Direct link to Iteration 5: 150 nodes" title="Direct link to Iteration 5: 150 nodes">​</a></h3><p>Finally, we simulate message propagation in a network of 150 nodes.
Due to local resource constraints, we run this simulation at a lower rate -
35 messages per second -
and for a shorter amount of time.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-150-nodes-aee6ff3d7b339b78fd56cc52eb86268e.png" width="1361" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Notice how the Waku v1 bandwidth usage is now more than 10 times worse than that of Waku v2.
This is to be expected, as each Waku v1 node will try to flood each new message to 149 other peers,
while the Waku v2 nodes limit their full-message peerings to no more than 12.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="discussion">Discussion<a href="#discussion" class="hash-link" aria-label="Direct link to Discussion" title="Direct link to Discussion">​</a></h3><p>Let's summarize average bandwidth growth against network growth for a constant message propagation rate.
Since we are particularly interested in how Waku v1 compares to Waku v2 in terms of bandwidth usage,
the results are normalised to the Waku v2 average bandwidth usage for each network size.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-overall-network-size-42e3912f3895367f3458c92e3e46ea47.png" width="1238" height="731" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Extrapolation is a dangerous game,
but it's safe to deduce that the divergence will only grow for even larger network topologies.
Although control signalling contributes more towards overall bandwidth for Waku v2 networks,
this effect becomes less noticeable for larger networks.
For network segments with more than ~18 densely connected nodes,
the advantage of using Waku v2 above Waku v1 becomes clear.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="network-traffic-comparison">Network traffic comparison<a href="#network-traffic-comparison" class="hash-link" aria-label="Direct link to Network traffic comparison" title="Direct link to Network traffic comparison">​</a></h2><p>The analysis above controls the average message rate while network size grows.
In reality, however, active users (and therefore message rates) are likely to grow in conjunction with the network.
This will have an effect on bandwidth for both Waku v1 and Waku v2, though not in equal measure.
Consider the impact of an increasing rate of messages in a network of constant size:</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku1-vs-waku2-overall-message-rate-aa6f7884f804b62ba14a6b4e369eaee3.png" width="1129" height="672" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>The <em>rate</em> of increase in bandwidth for Waku v2 is slower than that for Waku v1 for a corresponding increase in message propagation rate.
In fact, for a network of 30 densely-connected nodes,
if the message propagation rate increases by 1 per second,
Waku v1 requires an increased average bandwidth of almost 70kbps at each node.
A similar traffic increase in Waku v2 requires on average 40kbps more bandwidth per peer, just over half that of Waku v1.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusions">Conclusions<a href="#conclusions" class="hash-link" aria-label="Direct link to Conclusions" title="Direct link to Conclusions">​</a></h2><ul><li><strong>Waku v2 scales significantly better than Waku v1 in terms of average bandwidth usage</strong>,
especially for densely connected networks.</li><li>E.g. for a network consisting of <strong>150</strong> or more densely connected nodes,
Waku v2 provides more than <strong>10x</strong> better average bandwidth usage rates than Waku v1.</li><li>As the network continues to scale, both in absolute terms (number of nodes) and in network traffic (message rates) the disparity between Waku v2 and Waku v1 becomes even larger.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><p>Now that we've confirmed that Waku v2's bandwidth improvements over its predecessor matches theory,
we can proceed to a more in-depth characterisation of Waku v2's resource usage.
Some questions that we want to answer include:</p><ul><li>What proportion of Waku v2's bandwidth usage is used to propagate <em>payload</em> versus bandwidth spent on <em>control</em> messaging to maintain the mesh?</li><li>To what extent is message latency (time until a message is delivered to its destination) affected by network size and message rate?</li><li>How <em>reliable</em> is message delivery in Waku v2 for different network sizes and message rates?</li><li>What are the resource usage profiles of other Waku v2 protocols (e.g.<a href="https://rfc.vac.dev/spec/12/" target="_blank" rel="noopener noreferrer"><code>12/WAKU2-FILTER</code></a> and <a href="https://rfc.vac.dev/spec/19/" target="_blank" rel="noopener noreferrer"><code>19/WAKU2-LIGHTPUSH</code></a>)?</li></ul><p>Our aim is to get ever closer to a "real world" understanding of Waku v2's performance characteristics,
identify and fix vulnerabilities
and continually improve the efficiency of our suite of protocols.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><ul><li><a href="https://research.protocol.ai/publications/gossipsub-v1.1-evaluation-report/vyzovitis2020.pdf" target="_blank" rel="noopener noreferrer">Evaluation of GossipSub v1.1</a></li><li><a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">Fixing Whisper with Waku</a></li><li><a href="https://hackmd.io/@vac/main/%2FYYlZYBCURFyO_ZG1EiteWg#11WAKU2-RELAY-gossipsub" target="_blank" rel="noopener noreferrer">GossipSub vs flood routing</a></li><li><a href="https://www.techopedia.com/definition/13335/star-topology#:~:text=Star%20topology%20is%20a%20network,known%20as%20a%20star%20network." target="_blank" rel="noopener noreferrer">Network topologies: star</a></li><li><a href="https://en.wikipedia.org/wiki/Mesh_networking" target="_blank" rel="noopener noreferrer">Network topologies: mesh</a></li><li><a href="https://vac.dev/waku-v2-plan" target="_blank" rel="noopener noreferrer">Waku v2 original plan</a></li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[[Talk at COSCUP] Vac, Waku v2 and Ethereum Messaging]]></title>
<link>https://vac.dev/rlog/waku-v2-ethereum-coscup</link>
<guid>https://vac.dev/rlog/waku-v2-ethereum-coscup</guid>
<pubDate>Fri, 06 Aug 2021 12:00:00 GMT</pubDate>
<description><![CDATA[Learn more about Waku v2, its origins, goals, protocols, implementation and ongoing research. Understand how it is used and how it can be useful for messaging in Ethereum.]]></description>
<content:encoded><![CDATA[<p>Learn more about Waku v2, its origins, goals, protocols, implementation and ongoing research. Understand how it is used and how it can be useful for messaging in Ethereum.</p><p><em>This is the English version of a talk originally given in Chinese at COSCUP in Taipei.</em></p><p><a href="https://www.youtube.com/watch?v=s0ATpQ4_XFc" target="_blank" rel="noopener noreferrer">video recording with Chinese and English subtitles.</a></p><hr><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Hi everyone!</p><p>Today I'll talk to you about Waku v2. What it is, what problems it is solving,
and how it can be useful for things such as messaging in Ethereum. First, let me
start with some brief background.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="brief-history-and-background">Brief history and background<a href="#brief-history-and-background" class="hash-link" aria-label="Direct link to Brief history and background" title="Direct link to Brief history and background">​</a></h2><p>Back when Ethereum got started, there used to be this concept of the "holy
trinity". You had Ethereum for compute/consensus, Swarm for storage, and Whisper
for messaging. This is partly where the term Web3 comes from.</p><p>Status started out as an app with the goal of being a window onto Ethereum and
a secure messenger. As one of the few, if not the only, apps using Whisper in
production, not to mention on a mobile phone, we quickly realized there were
problems with the underlying protocols and infrastructure. Protocols such as
Whisper weren't quite ready for prime time yet when it came to things such as
scalability and working in the real world.</p><p>As we started addressing some of these challenges, and moved from app
developement to focusing on protocols, research and infrastructure, we created
Vac. Vac is an r&amp;d unit doing protocol research focused on creating modular p2p
messaging protocols for private, secure, censorship resistant communication.</p><p>I won't go into too much detail on the issues with Whisper, if you are
interested in this check out this talk
<a href="https://www.youtube.com/watch?v=6lLT33tsJjs" target="_blank" rel="noopener noreferrer">here</a> or this
<a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">article</a>.</p><p>In a nutshell, we forked Whisper to address immediate shortcomings and this
became Waku v1. Waku v2 is complete re-thought implementation from scratch on top
of libp2p. This will be the subject of today's talk.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-v2">Waku v2<a href="#waku-v2" class="hash-link" aria-label="Direct link to Waku v2" title="Direct link to Waku v2">​</a></h2><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="overview">Overview<a href="#overview" class="hash-link" aria-label="Direct link to Overview" title="Direct link to Overview">​</a></h3><p>Waku v2 is a privacy-preserving peer-to-peer messaging protocol for resource
restricted devices. We can look at Waku v2 as several things:</p><ul><li>Set of protocols</li><li>Set of implementations</li><li>Network of nodes</li></ul><p>Let's first look at what the goals are.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goals">Goals<a href="#goals" class="hash-link" aria-label="Direct link to Goals" title="Direct link to Goals">​</a></h3><p>Waku v2 provides a PubSub based messaging protocol with the following
characteristics:</p><ol><li><strong>Generalized messaging</strong>. Applications that require a messaging protocol to
communicate human to human, machine to machine, or a mix.</li><li><strong>Peer-to-peer</strong>. For applications that require a p2p solution.</li><li><strong>Resource restricted</strong>. For example, running with limited bandwidth, being
mostly-offline, or in a browser.</li><li><strong>Privacy</strong>. Applications that have privacy requirements, such as pseudonymity,
metadata protection, etc.</li></ol><p>And to provide these properties in a modular fashion, where applications can
choose their desired trade-offs.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="protocols">Protocols<a href="#protocols" class="hash-link" aria-label="Direct link to Protocols" title="Direct link to Protocols">​</a></h3><p>Waku v2 consists of several protocols. Here we highlight a few of the most
important ones:</p><ul><li>10/WAKU2 - main specification, details how all the pieces fit together</li><li>11/RELAY - thin layer on top of GossipSub for message dissemination</li><li>13/STORE - fetching of historical messages</li><li>14/MESSAGE - message payload</li></ul><p>This is the recommended subset for a minimal Waku v2 client.</p><p>In addition to this there are many other types of specifications at various
stages of maturity, such as: content based filtering, bridge mode to Waku v1,
JSON RPC API, zkSNARKS based spam protection with RLN, accounting and
settlements with SWAP, fault-tolerant store nodes, recommendations around topic
usage, and more.</p><p>See <a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">https://rfc.vac.dev/</a> for a full overview.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="implementations">Implementations<a href="#implementations" class="hash-link" aria-label="Direct link to Implementations" title="Direct link to Implementations">​</a></h3><p>Waku v2 consists of multiple implementations. This allows for client diversity,
makes it easier to strengthen the protocols, and allow people to use Waku v2 in
different contexts.</p><ul><li>nim-waku - the reference client written in Nim, most full-featured.</li><li>js-waku - allow usage of Waku v2 from browsers, focus on interacting with dapps.</li><li>go-waku - subset of Waku v2 to ease integration into the Status app.</li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="testnet-huilong-and-dogfooding">Testnet Huilong and dogfooding<a href="#testnet-huilong-and-dogfooding" class="hash-link" aria-label="Direct link to Testnet Huilong and dogfooding" title="Direct link to Testnet Huilong and dogfooding">​</a></h3><p>In order to test the protocol we have setup a testnet across all implementations
called Huilong. Yes, that's the Taipei subway station!</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/huilong-8b4f76f3e4117a3d34d3e90c0baef066.jpg" width="3264" height="2448" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Among us core devs we have disabled the main #waku Discord channel used for
development, and people run their own node connected to this toy chat application.</p><p>Feel free to join and say hi! Instructions can be found here:</p><ul><li><p><a href="https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md" target="_blank" rel="noopener noreferrer">nim-waku chat</a></p></li><li><p><a href="https://status-im.github.io/js-waku/" target="_blank" rel="noopener noreferrer">js-waku chat</a></p></li><li><p><a href="https://github.com/status-im/go-waku/tree/master/examples/chat2" target="_blank" rel="noopener noreferrer">go-waku chat</a></p></li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="research">Research<a href="#research" class="hash-link" aria-label="Direct link to Research" title="Direct link to Research">​</a></h3><p>While Waku v2 is being used today, we are actively researching improvements.
Since the design is modular, we can gracefully introduce new capabilities. Some
of these research areas are:</p><ul><li>Privacy-preserving spam protection using zkSNARKs and RLN</li><li>Accounting and settlement of resource usage to incentivize nodes to provide services with SWAP</li><li>State synchronization for store protocol to make it easier to run a store node without perfect uptime</li><li>Better node discovery</li><li>More rigorous privacy analysis</li><li>Improving interaction with wallets and dapp</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="use-cases">Use cases<a href="#use-cases" class="hash-link" aria-label="Direct link to Use cases" title="Direct link to Use cases">​</a></h2><p>Let's look at where Waku v2 is and can be used.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="prelude-topics-in-waku-v2">Prelude: Topics in Waku v2<a href="#prelude-topics-in-waku-v2" class="hash-link" aria-label="Direct link to Prelude: Topics in Waku v2" title="Direct link to Prelude: Topics in Waku v2">​</a></h3><p>To give some context, there are two different types of topics in Waku v2. One is
a PubSub topic, for routing. The other is a content topic, which is used for
content based filtering. Here's an example of the default PubSub topic:</p><p><code>/waku/2/default-waku/proto</code></p><p>This is recommended as it increases privacy for participants and it is stored by
default, however this is up to the application.</p><p>The second type of topic is a content topic, which is application specific. For
example, here's the content topic used in our testnet:</p><p><code>/toychat/2/huilong/proto</code></p><p>For more on topics, see <a href="https://rfc.vac.dev/spec/23/" target="_blank" rel="noopener noreferrer">https://rfc.vac.dev/spec/23/</a></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="status-app">Status app<a href="#status-app" class="hash-link" aria-label="Direct link to Status app" title="Direct link to Status app">​</a></h3><p>In the Status protocol, content topics - topics in Whisper/Waku v1 - are used for several things:</p><ul><li>Contact code topic to discover X3DH bundles for perfect forward secrecy<ul><li>Partitioned into N (currently 5000) content topics to balance privacy with efficiency</li></ul></li><li>Public chats correspond to hash of the plaintext name</li><li>Negotiated topic for 1:1 chat with DHKE derived content topic</li></ul><p>See more here <a href="https://specs.status.im/spec/10" target="_blank" rel="noopener noreferrer">https://specs.status.im/spec/10</a></p><p>Currently, Status app is in the process of migrating to and testing Waku v2.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dappconnect-ethereum-messaging">DappConnect: Ethereum messaging<a href="#dappconnect-ethereum-messaging" class="hash-link" aria-label="Direct link to DappConnect: Ethereum messaging" title="Direct link to DappConnect: Ethereum messaging">​</a></h3><p>It is easy to think of Waku as being for human messaging, since that's how it is
primarily used in the Status app, but the goal is to be useful for generalized
messaging, which includes Machine-To-Machine (M2M) messaging.</p><p>Recall the concept of the holy trinity with Ethereum/Swarm/Whisper and Web3 that
we mentioned in the beginning. Messaging can be used as a building block for
dapps, wallets, and users to communicate with each other. It can be used for
things such as:</p><ul><li>Multisig and DAO vote transactions only needing one on-chain operation</li><li>Giving dapps ability to send push notifications to users</li><li>Giving users ability to directly respond to requests from dapps</li><li>Decentralized WalletConnect</li><li>Etc</li></ul><p>Basically anything that requires communication and doesn't have to be on-chain.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="walletconnect-v2">WalletConnect v2<a href="#walletconnect-v2" class="hash-link" aria-label="Direct link to WalletConnect v2" title="Direct link to WalletConnect v2">​</a></h3><p>WalletConnect is an open protocol for connecting dapps to wallets with a QR
code. Version 2 is using Waku v2 as a communication channel to do so in a
decentralized and private fashion.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/walletconnect-62fa376651fb831e64e9a247d495bf53.png" width="809" height="452" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>See for more: <a href="https://docs.walletconnect.org/v/2.0/tech-spec" target="_blank" rel="noopener noreferrer">https://docs.walletconnect.org/v/2.0/tech-spec</a></p><p>WalletConnect v2 is currently in late alpha using Waku v2.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="more-examples">More examples<a href="#more-examples" class="hash-link" aria-label="Direct link to More examples" title="Direct link to More examples">​</a></h3><ul><li>Gasless voting and vote aggregation off-chain</li><li>Dapp games using Waku as player discovery mechanism</li><li>Send encrypted message to someone with an Ethereum key</li><li>&lt;<!-- -->Your dapp here<!-- -->&gt;</li></ul><p>These are all things that are in progress / proof of concept stage.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="contribute">Contribute<a href="#contribute" class="hash-link" aria-label="Direct link to Contribute" title="Direct link to Contribute">​</a></h2><p>We'd love to see contributions of any form!</p><ul><li>You can play with it here: <a href="https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md" target="_blank" rel="noopener noreferrer">nim-waku chat</a> (/ <a href="https://status-im.github.io/js-waku/" target="_blank" rel="noopener noreferrer">js-waku browser chat</a>)</li><li>Use Waku to build a dapp: <a href="https://status-im.github.io/js-waku/docs/" target="_blank" rel="noopener noreferrer">js-waku docs</a></li><li>Contribute to code: <a href="https://github.com/status-im/js-waku" target="_blank" rel="noopener noreferrer">js-waku</a> / <a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer">nim-waku</a></li><li>Contribute to specs: <a href="https://github.com/vacp2p/rfc" target="_blank" rel="noopener noreferrer">vacp2p/rfc</a></li><li>We are hiring: Wallet &amp; Dapp Integration Developer, Distributed Systems Engineer, Protocol Engineer, Protocol Researcher - all <a href="https://status.im/our_team/jobs.html" target="_blank" rel="noopener noreferrer">job listings</a></li><li>Join our new <a href="https://discord.gg/bJCTqS5H" target="_blank" rel="noopener noreferrer">Discord</a></li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2><p>In this talk we've gone over the original vision for Web3 and how Waku came to
be. We've also looked at what Waku v2 aims to do. We looked at its protocols,
implementations, the current testnet as well as briefly on some ongoing
research for Vac.</p><p>We've also looked at some specific use cases for Waku. First we looked at how
Status uses it with different topics. Then we looked at how it can be useful for
messaging in Ethereum, including for things like WalletConnect.</p><p>I hope this talk gives you a better idea of what Waku is, why it exists, and
that it inspires you to contribute, either to Waku itself or by using it in your
own project!</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Presenting JS-Waku: Waku v2 in the Browser]]></title>
<link>https://vac.dev/rlog/presenting-js-waku</link>
<guid>https://vac.dev/rlog/presenting-js-waku</guid>
<pubDate>Fri, 04 Jun 2021 12:00:00 GMT</pubDate>
<description><![CDATA[JS-Waku is bringing Waku v2 to the browser. Learn what we achieved so far and what is next in our pipeline!]]></description>
<content:encoded><![CDATA[<p>JS-Waku is bringing Waku v2 to the browser. Learn what we achieved so far and what is next in our pipeline!</p><p>For the past 3 months, we have been working on bringing Waku v2 to the browser.
Our aim is to empower dApps with Waku v2, and it led to the creation of a new library.
We believe now is good time to introduce it!</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-v2">Waku v2<a href="#waku-v2" class="hash-link" aria-label="Direct link to Waku v2" title="Direct link to Waku v2">​</a></h2><p>First, let's review what Waku v2 is and what problem it is trying to solve.</p><p>Waku v2 comes from a need to have a more scalable, better optimised solution for the Status app to achieve decentralised
communications on resource restricted devices (i.e., mobile phones).</p><p>The Status chat feature was initially built over Whisper.
However, Whisper has a number of caveats which makes it inefficient for mobile phones.
For example, with Whisper, all devices are receiving all messages which is not ideal for limited data plans.</p><p>To remediate this, a Waku mode (then Waku v1), based on devp2p, was introduced.
To further enable web and restricted resource environments, Waku v2 was created based on libp2p.
The migration of the Status chat feature to Waku v2 is currently in progress.</p><p>We see the need of such solution in the broader Ethereum ecosystem, beyond Status.
This is why we are building Waku v2 as a decentralised communication platform for all to use and build on.
If you want to read more about Waku v2 and what it aims to achieve,
checkout <a href="/waku-v2-plan">What's the Plan for Waku v2?</a>.</p><p>Since last year, we have been busy defining and implementing Waku v2 protocols in <a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer">nim-waku</a>,
from which you can build <a href="https://github.com/status-im/nim-waku#wakunode" target="_blank" rel="noopener noreferrer">wakunode2</a>.
Wakunode2 is an adaptive and modular Waku v2 node,
it allows users to run their own node and use the Waku v2 protocols they need.
The nim-waku project doubles as a library, that can be used to add Waku v2 support to native applications.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-v2-in-the-browser">Waku v2 in the browser<a href="#waku-v2-in-the-browser" class="hash-link" aria-label="Direct link to Waku v2 in the browser" title="Direct link to Waku v2 in the browser">​</a></h2><p>We believe that dApps and wallets can benefit from the Waku network in several ways.
For some dApps, it makes sense to enable peer-to-peer communications.
For others, machine-to-machine communications would be a great asset.
For example, in the case of a DAO,
Waku could be used for gas-less voting.
Enabling the DAO to notify their users of a new vote,
and users to vote without interacting with the blockchain and spending gas.</p><p><a href="https://github.com/status-im/murmur" target="_blank" rel="noopener noreferrer">Murmur</a> was the first attempt to bring Whisper to the browser,
acting as a bridge between devp2p and libp2p.
Once Waku v2 was started and there was a native implementation on top of libp2p,
a <a href="https://github.com/vacp2p/waku-web-chat" target="_blank" rel="noopener noreferrer">chat POC</a> was created to demonstrate the potential of Waku v2
in web environment.
It showed how using js-libp2p with few modifications enabled access to the Waku v2 network.
There was still some unresolved challenges.
For example, nim-waku only support TCP connections which are not supported by browser applications.
Hence, to connect to other node, the POC was connecting to a NodeJS proxy application using websockets,
which in turn could connect to wakunode2 via TCP.</p><p>However, to enable dApp and Wallet developers to easily integrate Waku in their product,
we need to give them a library that is easy to use and works out of the box:
introducing <a href="https://github.com/status-im/js-waku" target="_blank" rel="noopener noreferrer">JS-Waku</a>.</p><p>JS-Waku is a JavaScript library that allows your dApp, wallet or other web app to interact with the Waku v2 network.
It is available right now on <a href="https://www.npmjs.com/package/js-waku" target="_blank" rel="noopener noreferrer">npm</a>:</p><p><code>npm install js-waku</code>.</p><p>As it is written in TypeScript, types are included in the npm package to allow easy integration with TypeScript, ClojureScript and other typed languages that compile to JavaScript.</p><p>Key Waku v2 protocols are already available:
<a href="https://rfc.vac.dev/spec/14/" target="_blank" rel="noopener noreferrer">message</a>, <a href="https://rfc.vac.dev/spec/13/" target="_blank" rel="noopener noreferrer">store</a>, <a href="https://rfc.vac.dev/spec/11/" target="_blank" rel="noopener noreferrer">relay</a> and <a href="https://rfc.vac.dev/spec/19/" target="_blank" rel="noopener noreferrer">light push</a>,
enabling your dApp to:</p><ul><li>Send and receive near-instant messages on the Waku network (relay),</li><li>Query nodes for messages that may have been missed, e.g. due to poor cellular network (store),</li><li>Send messages with confirmations (light push).</li></ul><p>JS-Waku needs to operate in the same context from which Waku v2 was born:
a restricted environment were connectivity or uptime are not guaranteed;
JS-Waku brings Waku v2 to the browser.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="achievements-so-far">Achievements so far<a href="#achievements-so-far" class="hash-link" aria-label="Direct link to Achievements so far" title="Direct link to Achievements so far">​</a></h2><p>We focused the past month on developing a <a href="https://status-im.github.io/js-waku/" target="_blank" rel="noopener noreferrer">ReactJS Chat App</a>.
The aim was to create enough building blocks in JS-Waku to enable this showcase web app that
we now <a href="https://github.com/status-im/nim-waku/issues/399" target="_blank" rel="noopener noreferrer">use for dogfooding</a> purposes.</p><p>Most of the effort was on getting familiar with the <a href="https://github.com/libp2p/js-libp2p" target="_blank" rel="noopener noreferrer">js-libp2p</a> library
that we heavily rely on.
JS-Waku is the second implementation of Waku v2 protocol,
so a lot of effort on interoperability was needed.
For example, to ensure compatibility with the nim-waku reference implementation,
we run our <a href="https://github.com/status-im/js-waku/blob/90c90dea11dfd1277f530cf5d683fb92992fe141/src/lib/waku_relay/index.spec.ts#L137" target="_blank" rel="noopener noreferrer">tests against wakunode2</a> as part of the CI.</p><p>This interoperability effort helped solidify the current Waku v2 specifications:
By clarifying the usage of topics
(<a href="https://github.com/vacp2p/rfc/issues/327" target="_blank" rel="noopener noreferrer">#327</a>, <a href="https://github.com/vacp2p/rfc/pull/383" target="_blank" rel="noopener noreferrer">#383</a>),
fix discrepancies between specs and nim-waku
(<a href="https://github.com/status-im/nim-waku/issues/418" target="_blank" rel="noopener noreferrer">#418</a>, <a href="https://github.com/status-im/nim-waku/issues/419" target="_blank" rel="noopener noreferrer">#419</a>)
and fix small nim-waku &amp; nim-libp2p bugs
(<a href="https://github.com/status-im/nim-waku/issues/411" target="_blank" rel="noopener noreferrer">#411</a>, <a href="https://github.com/status-im/nim-waku/issues/439" target="_blank" rel="noopener noreferrer">#439</a>).</p><p>To fully access the waku network, JS-Waku needs to enable web apps to connect to nim-waku nodes.
A standard way to do so is using secure websockets as it is not possible to connect directly to a TCP port from the browser.
Unfortunately websocket support is not yet available in <a href="https://github.com/status-im/nim-libp2p/issues/407" target="_blank" rel="noopener noreferrer">nim-libp2p</a> so
we ended up deploying <a href="https://github.com/novnc/websockify" target="_blank" rel="noopener noreferrer">websockify</a> alongside wakunode2 instances.</p><p>As we built the <a href="https://github.com/status-im/js-waku/tree/main/examples/web-chat" target="_blank" rel="noopener noreferrer">web chat app</a>,
we were able to fine tune the API to provide a simple and succinct interface.
You can start a node, connect to other nodes and send a message in less than ten lines of code:</p><div class="language-javascript codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-javascript codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token keyword module" style="color:rgb(189, 147, 249);font-style:italic">import</span><span class="token plain"> </span><span class="token imports punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token imports"> </span><span class="token imports maybe-class-name">Waku</span><span class="token imports"> </span><span class="token imports punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"> </span><span class="token keyword module" style="color:rgb(189, 147, 249);font-style:italic">from</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'js-waku'</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">const</span><span class="token plain"> waku </span><span class="token operator">=</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(189, 147, 249);font-style:italic">await</span><span class="token plain"> </span><span class="token maybe-class-name">Waku</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token method function property-access" style="color:rgb(80, 250, 123)">create</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">const</span><span class="token plain"> nodes </span><span class="token operator">=</span><span class="token plain"> </span><span class="token keyword control-flow" style="color:rgb(189, 147, 249);font-style:italic">await</span><span class="token plain"> </span><span class="token function" style="color:rgb(80, 250, 123)">getStatusFleetNodes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword control-flow" style="color:rgb(189, 147, 249);font-style:italic">await</span><span class="token plain"> </span><span class="token known-class-name class-name">Promise</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token method function property-access" style="color:rgb(80, 250, 123)">all</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">nodes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token method function property-access" style="color:rgb(80, 250, 123)">map</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token parameter">addr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"> </span><span class="token arrow operator">=&gt;</span><span class="token plain"> waku</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token method function property-access" style="color:rgb(80, 250, 123)">dial</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">addr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword" style="color:rgb(189, 147, 249);font-style:italic">const</span><span class="token plain"> msg </span><span class="token operator">=</span><span class="token plain"> </span><span class="token maybe-class-name">WakuMessage</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token method function property-access" style="color:rgb(80, 250, 123)">fromUtf8String</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'Here is a message!'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'/my-cool-app/1/my-use-case/proto'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token keyword control-flow" style="color:rgb(189, 147, 249);font-style:italic">await</span><span class="token plain"> waku</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token property-access">relay</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token method function property-access" style="color:rgb(80, 250, 123)">send</span><span class="token punctuation" style="color:rgb(248, 248, 242)">(</span><span class="token plain">msg</span><span class="token punctuation" style="color:rgb(248, 248, 242)">)</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>We have also put a bounty at <a href="https://0xhack.dev/" target="_blank" rel="noopener noreferrer">0xHack</a> for using JS-Waku
and running a <a href="https://www.youtube.com/watch?v=l77j0VX75QE" target="_blank" rel="noopener noreferrer">workshop</a>.
We were thrilled to have a couple of hackers create new software using our libraries.
One of the projects aimed to create a decentralised, end-to-end encrypted messenger app,
similar to what the <a href="https://rfc.vac.dev/spec/20/" target="_blank" rel="noopener noreferrer">ETH-DM</a> protocol aims to achieve.
Another project was a decentralised Twitter platform.
Such projects allow us to prioritize the work on JS-Waku and understand how DevEx can be improved.</p><p>As more developers use JS-Waku, we will evolve the API to allow for more custom and fine-tune usage of the network
while preserving this out of the box experience.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="whats-next">What's next?<a href="#whats-next" class="hash-link" aria-label="Direct link to What's next?" title="Direct link to What's next?">​</a></h2><p>Next, we are directing our attention towards <a href="https://github.com/status-im/js-waku/issues/68" target="_blank" rel="noopener noreferrer">Developer Experience</a>.
We already have <a href="https://www.npmjs.com/package/js-waku" target="_blank" rel="noopener noreferrer">documentation</a> available but we want to provide more:
<a href="https://github.com/status-im/js-waku/issues/56" target="_blank" rel="noopener noreferrer">Tutorials</a>, various examples
and showing how <a href="https://github.com/status-im/js-waku/issues/72" target="_blank" rel="noopener noreferrer">JS-Waku can be used with Web3</a>.</p><p>By prioritizing DevEx we aim to enable JS-Waku integration in dApps and wallets.
We think JS-Waku builds a strong case for machine-to-machine (M2M) communications.
The first use cases we are looking into are dApp notifications:
Enabling dApp to notify their user directly in their wallets!
Leveraging Waku as a decentralised infrastructure and standard so that users do not have to open their dApp to be notified
of events such as DAO voting.</p><p>We already have some POC in the pipeline to enable voting and polling on the Waku network,
allowing users to save gas by <strong>not</strong> broadcasting each individual vote on the blockchain.</p><p>To facilitate said applications, we are looking at improving integration with Web3 providers by providing examples
of signing, validating, encrypting and decrypting messages using Web3.
Waku is privacy conscious, so we will also provide signature and encryption examples decoupled from users' Ethereum identity.</p><p>As you can read, we have grand plans for JS-Waku and Waku v2.
There is a lot to do, and we would love some help so feel free to
check out the new role in our team:
<a href="https://status.im/our_team/jobs.html?gh_jid=3157894" target="_blank" rel="noopener noreferrer">js-waku: Wallet &amp; Dapp Integration Developer</a>.
We also have a number of <a href="https://status.im/our_team/jobs.html" target="_blank" rel="noopener noreferrer">positions</a> open to work on Waku protocol and nim-waku.</p><p>If you are as excited as us by JS-Waku, why not build a dApp with it?
You can find documentation on the <a href="https://www.npmjs.com/package/js-waku" target="_blank" rel="noopener noreferrer">npmjs page</a>.</p><p>Whether you are a developer, you can come chat with us using <a href="https://status-im.github.io/js-waku/" target="_blank" rel="noopener noreferrer">WakuJS Web Chat</a>
or <a href="https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md" target="_blank" rel="noopener noreferrer">chat2</a>.
You can get support in #dappconnect-support on <a href="https://discord.gg/j5pGbn7MHZ" target="_blank" rel="noopener noreferrer">Vac Discord</a> or <a href="https://t.me/dappconnectsupport" target="_blank" rel="noopener noreferrer">Telegram</a>.
If you have any ideas on how Waku could enable a specific dapp or use case, do share, we are always keen to hear it.</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Privacy-preserving p2p economic spam protection in Waku v2]]></title>
<link>https://vac.dev/rlog/rln-relay</link>
<guid>https://vac.dev/rlog/rln-relay</guid>
<pubDate>Fri, 05 Mar 2021 12:00:00 GMT</pubDate>
<description><![CDATA[This post is going to give you an overview of how spam protection can be achieved in Waku Relay through rate-limiting nullifiers. We will cover a summary of spam-protection methods in centralized and p2p systems, and the solution overview and details of the economic spam-protection method. The open issues and future steps are discussed in the end.]]></description>
<content:encoded><![CDATA[<p>This post is going to give you an overview of how spam protection can be achieved in Waku Relay through rate-limiting nullifiers. We will cover a summary of spam-protection methods in centralized and p2p systems, and the solution overview and details of the economic spam-protection method. The open issues and future steps are discussed in the end.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>This post is going to give you an overview of how spam protection can be achieved in Waku Relay protocol<sup id="fnref-2-3deade"><a href="#fn-2-3deade" class="footnote-ref">2</a></sup> through Rate-Limiting Nullifiers<sup id="fnref-3-3deade"><a href="#fn-3-3deade" class="footnote-ref">3</a></sup> <sup id="fnref-4-3deade"><a href="#fn-4-3deade" class="footnote-ref">4</a></sup> or RLN for short.</p><p>Let me give a little background about Waku(v2)<sup id="fnref-1-3deade"><a href="#fn-1-3deade" class="footnote-ref">1</a></sup>. Waku is a privacy-preserving peer-to-peer (p2p) messaging protocol for resource-restricted devices. Being p2p means that Waku relies on <strong>No</strong> central server. Instead, peers collaboratively deliver messages in the network. Waku uses GossipSub<sup id="fnref-16-3deade"><a href="#fn-16-3deade" class="footnote-ref">16</a></sup> as the underlying routing protocol (as of the writeup of this post). At a high level, GossipSub is based on publisher-subscriber architecture. That is, <em>peers, congregate around topics they are interested in and can send messages to topics. Each message gets delivered to all peers subscribed to the topic</em>. In GossipSub, a peer has a constant number of direct connections/neighbors. In order to publish a message, the author forwards its message to a subset of neighbors. The neighbors proceed similarly till the message gets propagated in the network of the subscribed peers. The message publishing and routing procedures are part of the Waku Relay<sup id="fnref-17-3deade"><a href="#fn-17-3deade" class="footnote-ref">17</a></sup> protocol.
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 1: An overview of privacy-preserving p2p economic spam protection in Waku v2 RLN-Relay protocol." src="/assets/images/rln-relay-overview-7a73a646ea1b6b9bdfc62d96d7de296e.png" width="1920" height="1080" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="what-do-we-mean-by-spamming">What do we mean by spamming?<a href="#what-do-we-mean-by-spamming" class="hash-link" aria-label="Direct link to What do we mean by spamming?" title="Direct link to What do we mean by spamming?">​</a></h2><p>In centralized messaging systems, a spammer usually indicates an entity that uses the messaging system to send an unsolicited message (spam) to large numbers of recipients. However, in Waku with a p2p architecture, spam messages not only affect the recipients but also all the other peers involved in the routing process as they have to spend their computational power/bandwidth/storage capacity on processing spam messages. As such, we define a spammer as an entity that uses the messaging system to publish a large number of messages in a short amount of time. The messages issued in this way are called spam. In this definition, we disregard the intention of the spammer as well as the content of the message and the number of recipients.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="possible-solutions">Possible Solutions<a href="#possible-solutions" class="hash-link" aria-label="Direct link to Possible Solutions" title="Direct link to Possible Solutions">​</a></h2><p>Has the spamming issue been addressed before? Of course yes! Here is an overview of the spam protection techniques with their trade-offs and use-cases. In this overview, we distinguish between protection techniques that are targeted for centralized messaging systems and those for p2p architectures.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="centralized-messaging-systems">Centralized Messaging Systems<a href="#centralized-messaging-systems" class="hash-link" aria-label="Direct link to Centralized Messaging Systems" title="Direct link to Centralized Messaging Systems">​</a></h3><p>In traditional centralized messaging systems, spam usually signifies unsolicited messages sent in bulk or messages with malicious content like malware. Protection mechanisms include</p><ul><li>authentication through some piece of personally identifiable information e.g., phone number</li><li>checksum-based filtering to protect against messages sent in bulk</li><li>challenge-response systems</li><li>content filtering on the server or via a proxy application</li></ul><p>These methods exploit the fact that the messaging system is centralized and a global view of the users' activities is available based on which spamming patterns can be extracted and defeated accordingly. Moreover, users are associated with an identifier e.g., a username which enables the server to profile each user e.g., to detect suspicious behavior like spamming. Such profiling possibility is against the user's anonymity and privacy.</p><p>Among the techniques enumerated above, authentication through phone numbers is a some-what economic-incentive measure as providing multiple valid phone numbers will be expensive for the attacker. Notice that while using an expensive authentication method can reduce the number of accounts owned by a single spammer, cannot address the spam issue entirely. This is because the spammer can still send bulk messages through one single account. For this approach to be effective, a centralized mediator is essential. That is why such a solution would not fit the p2p environments where no centralized control exists.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="p2p-systems">P2P Systems<a href="#p2p-systems" class="hash-link" aria-label="Direct link to P2P Systems" title="Direct link to P2P Systems">​</a></h3><p>What about spam prevention in p2p messaging platforms? There are two techniques, namely <em>Proof of Work</em><sup id="fnref-8-3deade"><a href="#fn-8-3deade" class="footnote-ref">8</a></sup> deployed by Whisper<sup id="fnref-9-3deade"><a href="#fn-9-3deade" class="footnote-ref">9</a></sup> and <em>Peer scoring</em><sup id="fnref-6-3deade"><a href="#fn-6-3deade" class="footnote-ref">6</a></sup> method (namely reputation-based approach) adopted by LibP2P. However, each of these solutions has its own shortcomings for real-life use-cases as explained below.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proof-of-work">Proof of work<a href="#proof-of-work" class="hash-link" aria-label="Direct link to Proof of work" title="Direct link to Proof of work">​</a></h4><p>The idea behind the Proof Of Work i.e., POW<sup id="fnref-8-3deade"><a href="#fn-8-3deade" class="footnote-ref">8</a></sup> is to make messaging a computationally costly operation hence lowering the messaging rate of <strong>all</strong> the peers including the spammers. In specific, the message publisher has to solve a puzzle and the puzzle is to find a nonce such that the hash of the message concatenated with the nonce has at least z leading zeros. z is known as the difficulty of the puzzle. Since the hash function is one-way, peers have to brute-force to find a nonce. Hashing is a computationally-heavy operation so is the brute-force. While solving the puzzle is computationally expensive, it is comparatively cheap to verify the solution.</p><p>POW is also used as the underlying mining algorithm in Ethereum and Bitcoin blockchain. There, the goal is to contain the mining speed and allow the decentralized network to come to a consensus, or agree on things like account balances and the order of transactions.</p><p>While the use of POW makes perfect sense in Ethereum / Bitcoin blockchain, it shows practical issues in heterogeneous p2p messaging systems with resource-restricted peers. Some peers won't be able to carry the designated computation and will be effectively excluded. Such exclusion showed to be practically an issue in applications like Status, which used to rely on POW for spam-protection, to the extent that the difficulty level had to be set close to zero.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="peer-scoring">Peer Scoring<a href="#peer-scoring" class="hash-link" aria-label="Direct link to Peer Scoring" title="Direct link to Peer Scoring">​</a></h4><p>The peer scoring method<sup id="fnref-6-3deade"><a href="#fn-6-3deade" class="footnote-ref">6</a></sup> that is utilized by libp2p is to limit the number of messages issued by a peer in connection to another peer. That is each peer monitors all the peers to which it is directly connected and adjusts their messaging quota i.e., to route or not route their messages depending on their past activities. For example, if a peer detects its neighbor is sending more than x messages per month, can drop its quota to z.x where z is less than one. The shortcoming of this solution is that scoring is based on peers' local observations and the concept of the score is defined in relation to one single peer. This leaves room for an attack where a spammer can make connections to k peers in the system and publishes k.(x-1) messages by exploiting all of its k connections. Another attack scenario is through botnets consisting of a large number of e.g., a million bots. The attacker rents a botnet and inserts each of them as a legitimate peer to the network and each can publish x-1 messages per month<sup id="fnref-7-3deade"><a href="#fn-7-3deade" class="footnote-ref">7</a></sup>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="economic-incentive-spam-protection">Economic-Incentive Spam protection<a href="#economic-incentive-spam-protection" class="hash-link" aria-label="Direct link to Economic-Incentive Spam protection" title="Direct link to Economic-Incentive Spam protection">​</a></h4><p>Is this the end of our spam-protection journey? Shall we simply give up and leave spammers be? Certainly not!
Waku RLN-Relay gives us a p2p spam-protection method which:</p><ul><li>suits <strong>p2p</strong> systems and does not rely on any central entity.</li><li>is <strong>efficient</strong> i.e., with no unreasonable computational, storage, memory, and bandwidth requirement! as such, it fits the network of <strong>heterogeneous</strong> peers.</li><li>respects users <strong>privacy</strong> unlike reputation-based and centralized methods.</li><li>deploys <strong>economic-incentives</strong> to contain spammers' activity. Namely, there is a financial sacrifice for those who want to spam the system. How? follow along ...</li></ul><p>We devise a general rule to save everyone's life and that is</p><p><strong>No one can publish more than M messages per epoch without being financially charged!</strong></p><p>We set M to 1 for now, but this can be any arbitrary value. You may be thinking "This is too restrictive! Only one per epoch?". Don't worry, we set the epoch to a reasonable value so that it does not slow down the communication of innocent users but will make the life of spammers harder! Epoch here can be every second, as defined by UTC date-time +-20s.</p><p>The remainder of this post is all about the story of how to enforce this limit on each user's messaging rate as well as how to impose the financial cost when the limit gets violated. This brings us to the Rate Limiting Nullifiers and how we integrate this technique into Waku v2 (in specific the Waku Relay protocol) to protect our valuable users against spammers.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="technical-terms">Technical Terms<a href="#technical-terms" class="hash-link" aria-label="Direct link to Technical Terms" title="Direct link to Technical Terms">​</a></h2><p><strong>Zero-knowledge proof</strong>: Zero-knowledge proof (ZKP)<sup id="fnref-14-3deade"><a href="#fn-14-3deade" class="footnote-ref">14</a></sup> allows a <em>prover</em> to show a <em>verifier</em> that they know something, without revealing what that something is. This means you can do the trust-minimized computation that is also privacy-preserving. As a basic example, instead of showing your ID when going to a bar you simply give them proof that you are over 18, without showing the doorman your id. In this write-up, by ZKP we essentially mean zkSNARK<sup id="fnref-15-3deade"><a href="#fn-15-3deade" class="footnote-ref">15</a></sup> which is one of the many types of ZKPs.</p><p><strong>Threshold Secret Sharing Scheme</strong>: (m,n) Threshold secret-sharing is a method by which you can split a secret value s into n pieces in a way that the secret s can be reconstructed by having m pieces (m &lt;= n). The economic-incentive spam protection utilizes a (2,n) secret sharing realized by Shamir Secret Sharing Scheme<sup id="fnref-13-3deade"><a href="#fn-13-3deade" class="footnote-ref">13</a></sup>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="overview-economic-incentive-spam-protection-through-rate-limiting-nullifiers">Overview: Economic-Incentive Spam protection through Rate Limiting Nullifiers<a href="#overview-economic-incentive-spam-protection-through-rate-limiting-nullifiers" class="hash-link" aria-label="Direct link to Overview: Economic-Incentive Spam protection through Rate Limiting Nullifiers" title="Direct link to Overview: Economic-Incentive Spam protection through Rate Limiting Nullifiers">​</a></h2><p><strong>Context</strong>: We started the idea of economic-incentive spam protection more than a year ago and conducted a feasibility study to identify blockers and unknowns. The results are published in our prior <a href="https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks" target="_blank" rel="noopener noreferrer">post</a>. Since then major progress has been made and the prior identified blockers that are listed below are now addressed. Kudos to <a href="https://github.com/barryWhiteHat" target="_blank" rel="noopener noreferrer">Barry WhiteHat</a>, <a href="https://github.com/kilic" target="_blank" rel="noopener noreferrer">Onur Kilic</a>, <a href="https://github.com/weijiekoh/perpetualpowersoftau" target="_blank" rel="noopener noreferrer">Koh Wei Jie</a> for all of their hard work, research, and development which made this progress possible.</p><ul><li>the proof time<sup id="fnref-22-3deade"><a href="#fn-22-3deade" class="footnote-ref">22</a></sup> which was initially in the order of minutes ~10 mins and now is almost 0.5 seconds</li><li>the prover key size<sup id="fnref-21-3deade"><a href="#fn-21-3deade" class="footnote-ref">21</a></sup> which was initially ~110MB and now is ~3.9MB</li><li>the lack of Shamir logic<sup id="fnref-19-3deade"><a href="#fn-19-3deade" class="footnote-ref">19</a></sup> which is now implemented and part of the RLN repository<sup id="fnref-4-3deade"><a href="#fn-4-3deade" class="footnote-ref">4</a></sup></li><li>the concern regarding the potential multi-party computation for the trusted setup of zkSNARKs which got resolved<sup id="fnref-20-3deade"><a href="#fn-20-3deade" class="footnote-ref">20</a></sup></li><li>the lack of end-to-end integration that now we made it possible, have it implemented, and are going to present it in this post. New blockers are also sorted out during the e2e integration which we will discuss in the <a href="#feasibility-and-open-issues">Feasibility and Open Issues</a> section.</li></ul><p>Now that you have more context, let's see how the final solution works. The fundamental point is to make it economically costly to send more than your share of messages and to do so in a privacy-preserving and e2e fashion. To do that we have the following components:</p><ul><li>1- <strong>Group</strong>: We manage all the peers inside a large group (later we can split peers into smaller groups, but for now consider only one). The group management is done via a smart contract which is devised for this purpose and is deployed on the Ethereum blockchain.</li><li>2- <strong>Membership</strong>: To be able to send messages and in specific for the published messages to get routed by all the peers, publishing peers have to register to the group. Membership involves setting up public and private key pairs (think of it as the username and password). The private key remains at the user side but the public key becomes a part of the group information on the contract (publicly available) and everyone has access to it. Public keys are not human-generated (like usernames) and instead they are random numbers, as such, they do not reveal any information about the owner (think of public keys as pseudonyms). Registration is mandatory for the users who want to publish a message, however, users who only want to listen to the messages are more than welcome and do not have to register in the group.</li><li><strong>Membership fee</strong>: Membership is not for free! each peer has to lock a certain amount of funds during the registration (this means peers have to have an Ethereum account with sufficient balance for this sake). This fund is safely stored on the contract and remains intact unless the peer attempts to break the rules and publish more than one message per epoch.</li><li><strong>Zero-knowledge Proof of membership</strong>: Do you want your message to get routed to its destination, fine, but you have to prove that you are a member of the group (sorry, no one can escape the registration phase!). Now, you may be thinking that should I attach my public key to my message to prove my membership? Absolutely Not! we said that our solution respects privacy! membership proofs are done in a zero-knowledge manner that is each message will carry cryptographic proof asserting that "the message is generated by one of the current members of the group", so your identity remains private and your anonymity is preserved!</li><li><strong>Slashing through secret sharing</strong>: Till now it does not seem like we can catch spammers, right? yes, you are right! now comes the exciting part, detecting spammers and slashing them. The core idea behind the slashing is that each publishing peer (not routing peers!) has to integrate a secret share of its private key inside the message. The secret share is deterministically computed over the private key and the current epoch. The content of this share is harmless for the peer's privacy (it looks random) unless the peer attempts to publish more than one message in the same epoch hence disclosing more than one secret share of its private key. Indeed two distinct shares of the private key under the same epoch are enough to reconstruct the entire private key. Then what should you do with the recovered private key? hurry up! go to the contract and withdraw the private key and claim its fund and get rich!! Are you thinking what if spammers attach junk values instead of valid secret shares? Of course, that wouldn't be cool! so, there is a zero-knowledge proof for this sake as well where the publishing peer has to prove that the secret shares are generated correctly.</li></ul><p>A high-level overview of the economic spam protection is shown in Figure 1.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="flow">Flow<a href="#flow" class="hash-link" aria-label="Direct link to Flow" title="Direct link to Flow">​</a></h2><p>In this section, we describe the flow of the economic-incentive spam detection mechanism from the viewpoint of a single peer. An overview of this flow is provided in Figure 3.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="setup-and-registration">Setup and Registration<a href="#setup-and-registration" class="hash-link" aria-label="Direct link to Setup and Registration" title="Direct link to Setup and Registration">​</a></h2><p>A peer willing to publish a message is required to register. Registration is moderated through a smart contract deployed on the Ethereum blockchain. The state of the contract contains the list of registered members' public keys. An overview of registration is illustrated in Figure 2.</p><p>For the registration, a peer creates a transaction that sends x amount of Ether to the contract. The peer who has the "private key" <code>sk</code> associated with that deposit would be able to withdraw x Ether by providing valid proof. Note that <code>sk</code> is initially only known by the owning peer however it may get exposed to other peers in case the owner attempts spamming the system i.e., sending more than one message per epoch.
The following relation holds between the <code>sk</code> and <code>pk</code> i.e., <code>pk = H(sk)</code> where <code>H</code> denotes a hash function.
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 2: Registration" src="/assets/images/rln-relay-704966b8b4e9245d45ff49dee43f1c05.png" width="1952" height="892" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="maintaining-the-membership-merkle-tree">Maintaining the membership Merkle Tree<a href="#maintaining-the-membership-merkle-tree" class="hash-link" aria-label="Direct link to Maintaining the membership Merkle Tree" title="Direct link to Maintaining the membership Merkle Tree">​</a></h2><p>The ZKP of membership that we mentioned before relies on the representation of the entire group as a <a href="/">Merkle Tree</a>. The tree construction and maintenance is delegated to the peers (the initial idea was to keep the tree on the chain as part of the contract, however, the cost associated with member deletion and insertion was high and unreasonable, please see <a href="#Feasibility-and-Open-Issues">Feasibility and Open Issues</a> for more details). As such, each peer needs to build the tree locally and sync itself with the contract updates (peer insertion and deletion) to mirror them on its tree.
Two pieces of information of the tree are important as they enable peers to generate zero-knowledge proofs. One is the root of the tree and the other is the membership proof (or the authentication path). The tree root is public information whereas the membership proof is private data (or more precisely the index of the peer in the tree).</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="publishing">Publishing<a href="#publishing" class="hash-link" aria-label="Direct link to Publishing" title="Direct link to Publishing">​</a></h2><p>In order to publish at a given epoch, each message must carry a proof i.e., a zero-knowledge proof signifying that the publishing peer is a registered member, and has not exceeded the messaging rate at the given epoch.</p><p>Recall that the enforcement of the messaging rate was through associating a secret shared version of the peer's <code>sk</code> into the message together with a ZKP that the secret shares are constructed correctly. As for the secret sharing part, the peer generates the following data:</p><ol><li><code>shareX</code></li><li><code>shareY</code></li><li><code>nullifier</code></li></ol><p>The pair (<code>shareX</code>, <code>shareY</code>) is the secret shared version of <code>sk</code> that are generated using Shamir secret sharing scheme. Having two such pairs for an identical <code>nullifier</code> results in full disclosure of peer's <code>sk</code> and hence burning the associated deposit. Note that the <code>nullifier</code> is a deterministic value derived from <code>sk</code> and <code>epoch</code> therefore any two messages issued by the same peer (i.e., using the same <code>sk</code>) for the same <code>epoch</code> are guaranteed to have identical <code>nullifier</code>s.</p><p>Finally, the peer generates a zero-knowledge proof <code>zkProof</code> asserting the membership of the peer in the group and the correctness of the attached secret share (<code>shareX</code>, <code>shareY</code>) and the <code>nullifier</code>. In order to generate a valid proof, the peer needs to have two private inputs i.e., its <code>sk</code> and its authentication path. Other inputs are the tree root, epoch, and the content of the message.</p><p><strong>Privacy Hint:</strong> Note that the authentication path of each peer depends on the recent list of members (hence changes when new peers register or leave). As such, it is recommended (and necessary for privacy/anonymity) that the publisher updates her authentication path based on the latest status of the group and attempts the proof using the updated version.</p><p>An overview of the publishing procedure is provided in Figure 3.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="routing">Routing<a href="#routing" class="hash-link" aria-label="Direct link to Routing" title="Direct link to Routing">​</a></h2><p>Upon the receipt of a message, the routing peer needs to decide whether to route it or not. This decision relies on the following factors:</p><ol><li>If the epoch value attached to the message has a non-reasonable gap with the routing peer's current epoch then the message must be dropped (this is to prevent a newly registered peer spamming the system by messaging for all the past epochs).</li><li>The message MUST contain valid proof that gets verified by the routing peer.
If the preceding checks are passed successfully, then the message is relayed. In case of an invalid proof, the message is dropped. If spamming is detected, the publishing peer gets slashed (see <a href="#Spam-Detection-and-Slashing">Spam Detection and Slashing</a>).</li></ol><p>An overview of the routing procedure is provided in Figure 3.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="spam-detection-and-slashing">Spam Detection and Slashing<a href="#spam-detection-and-slashing" class="hash-link" aria-label="Direct link to Spam Detection and Slashing" title="Direct link to Spam Detection and Slashing">​</a></h3><p>In order to enable local spam detection and slashing, routing peers MUST record the <code>nullifier</code>, <code>shareX</code>, and <code>shareY</code> of any incoming message conditioned that it is not spam and has valid proof. To do so, the peer should follow the following steps.</p><ol><li>The routing peer first verifies the <code>zkProof</code> and drops the message if not verified.</li><li>Otherwise, it checks whether a message with an identical <code>nullifier</code> has already been relayed.<ul><li>a) If such message exists and its <code>shareX</code> and <code>shareY</code> components are different from the incoming message, then slashing takes place (if the <code>shareX</code> and <code>shareY</code> fields of the previously relayed message is identical to the incoming message, then the message is a duplicate and shall be dropped).</li><li>b) If none found, then the message gets relayed.</li></ul></li></ol><p>An overview of the slashing procedure is provided in Figure 3.
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 3: Publishing, Routing and Slashing workflow." src="/assets/images/rln-message-verification-9fece24cdde2c518766ede67364c958f.png" width="1632" height="2680" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="feasibility-and-open-issues">Feasibility and Open Issues<a href="#feasibility-and-open-issues" class="hash-link" aria-label="Direct link to Feasibility and Open Issues" title="Direct link to Feasibility and Open Issues">​</a></h2><p>We've come a long way since a year ago, blockers resolved, now we have implemented it end-to-end. We learned lot and could identify further issues and unknowns some of which are blocking getting to production. The summary of the identified issues are presented below.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="storage-overhead-per-peer">Storage overhead per peer<a href="#storage-overhead-per-peer" class="hash-link" aria-label="Direct link to Storage overhead per peer" title="Direct link to Storage overhead per peer">​</a></h2><p>Currently, peers are supposed to maintain the entire tree locally and it imposes storage overhead which is linear in the size of the group (see this <a href="https://github.com/vacp2p/research/issues/57" target="_blank" rel="noopener noreferrer">issue</a><sup id="fnref-11-3deade"><a href="#fn-11-3deade" class="footnote-ref">11</a></sup> for more details). One way to cope with this is to use the light-node and full-node paradigm in which only a subset of peers who are more resourceful retain the tree whereas the light nodes obtain the necessary information by interacting with the full nodes. Another way to approach this problem is through a more storage efficient method (as described in this research issue<sup id="fnref-12-3deade"><a href="#fn-12-3deade" class="footnote-ref">12</a></sup>) where peers store a partial view of the tree instead of the entire tree. Keeping the partial view lowers the storage complexity to O(log(N)) where N is the size of the group. There are still unknown unknowns to this solution, as such, it must be studied further to become fully functional.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="cost-effective-way-of-member-insertion-and-deletion">Cost-effective way of member insertion and deletion<a href="#cost-effective-way-of-member-insertion-and-deletion" class="hash-link" aria-label="Direct link to Cost-effective way of member insertion and deletion" title="Direct link to Cost-effective way of member insertion and deletion">​</a></h2><p>Currently, the cost associated with RLN-Relay membership is around 30 USD<sup id="fnref-10-3deade"><a href="#fn-10-3deade" class="footnote-ref">10</a></sup>. We aim at finding a more cost-effective approach. Please feel free to share with us your solution ideas in this regard in this <a href="https://github.com/vacp2p/research/issues/56" target="_blank" rel="noopener noreferrer">issue</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="exceeding-the-messaging-rate-via-multiple-registrations">Exceeding the messaging rate via multiple registrations<a href="#exceeding-the-messaging-rate-via-multiple-registrations" class="hash-link" aria-label="Direct link to Exceeding the messaging rate via multiple registrations" title="Direct link to Exceeding the messaging rate via multiple registrations">​</a></h2><p>While the economic-incentive solution has an economic incentive to discourage spamming, we should note that there is still <strong>expensive attack(s)</strong><sup id="fnref-23-3deade"><a href="#fn-23-3deade" class="footnote-ref">23</a></sup> that a spammer can launch to break the messaging rate limit. That is, the attacker can pay for multiple legit registrations e.g., k, hence being able to publish k messages per epoch. We believe that the higher the membership fee is, the less probable would be such an attack, hence a stronger level of spam-protection can be achieved. Following this argument, the high fee associated with the membership (which we listed above as an open problem) can indeed be contributing to a better protection level.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion-and-future-steps">Conclusion and Future Steps<a href="#conclusion-and-future-steps" class="hash-link" aria-label="Direct link to Conclusion and Future Steps" title="Direct link to Conclusion and Future Steps">​</a></h2><p>As discussed in this post, Waku RLN Relay can achieve a privacy-preserving economic spam protection through rate-limiting nullifiers. The idea is to financially discourage peers from publishing more than one message per epoch. In specific, exceeding the messaging rate results in a financial charge. Those who violate this rule are called spammers and their messages are spam. The identification of spammers does not rely on any central entity. Also, the financial punishment of spammers is cryptographically guaranteed.
In this solution, privacy is guaranteed since: 1) Peers do not have to disclose any piece of personally identifiable information in any phase i.e., neither in the registration nor in the messaging phase 2) Peers can prove that they have not exceeded the messaging rate in a zero-knowledge manner and without leaving any trace to their membership accounts.
Furthermore, all the computations are light hence this solution fits the heterogenous p2p messaging system. Note that the zero-knowledge proof parts are handled through zkSNARKs and the benchmarking result can be found in the RLN benchmark report<sup id="fnref-5-3deade"><a href="#fn-5-3deade" class="footnote-ref">5</a></sup>.</p><p><strong>Future steps</strong>:</p><p>We are still at the PoC level, and the development is in progress. As our future steps,</p><ul><li>we would like to evaluate the running time associated with the Merkle tree operations. Indeed, the need to locally store Merkle tree on each peer was one of the unknowns discovered during this PoC and yet the concrete benchmarking result in this regard is not available.</li><li>We would also like to pursue our storage-efficient Merkle Tree maintenance solution in order to lower the storage overhead of peers.</li><li>In line with the storage optimization, the full-node light-node structure is another path to follow.</li><li>Another possible improvement is to replace the membership contract with a distributed group management scheme e.g., through distributed hash tables. This is to address possible performance issues that the interaction with the Ethereum blockchain may cause. For example, the registration transactions are subject to delay as they have to be mined before being visible in the state of the membership contract. This means peers have to wait for some time before being able to publish any message.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="acknowledgement">Acknowledgement<a href="#acknowledgement" class="hash-link" aria-label="Direct link to Acknowledgement" title="Direct link to Acknowledgement">​</a></h2><p>Thanks to Onur Kılıç for his explanation and pointers and for assisting with development and runtime issues. Also thanks to Barry Whitehat for his time and insightful comments. Special thanks to Oskar Thoren for his constructive comments and his guides during the development of this PoC and the writeup of this post.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><div class="footnotes"><hr><ol><li id="fn-2-3deade">RLN-Relay specification: <a href="https://rfc.vac.dev/spec/17/" target="_blank" rel="noopener noreferrer">https://rfc.vac.dev/spec/17/</a><a href="#fnref-2-3deade" class="footnote-backref">↩</a></li><li id="fn-3-3deade">RLN documentation: <a href="https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?both" target="_blank" rel="noopener noreferrer">https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?both</a><a href="#fnref-3-3deade" class="footnote-backref">↩</a></li><li id="fn-4-3deade">RLN repositories: <a href="https://github.com/kilic/RLN" target="_blank" rel="noopener noreferrer">https://github.com/kilic/RLN</a> and <a href="https://github.com/kilic/rlnapp" target="_blank" rel="noopener noreferrer">https://github.com/kilic/rlnapp</a><a href="#fnref-4-3deade" class="footnote-backref">↩</a></li><li id="fn-1-3deade">Waku v2: <a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">https://rfc.vac.dev/spec/10/</a><a href="#fnref-1-3deade" class="footnote-backref">↩</a></li><li id="fn-16-3deade">GossipSub: <a href="https://docs.libp2p.io/concepts/publish-subscribe/" target="_blank" rel="noopener noreferrer">https://docs.libp2p.io/concepts/publish-subscribe/</a><a href="#fnref-16-3deade" class="footnote-backref">↩</a></li><li id="fn-17-3deade">Waku Relay: <a href="https://rfc.vac.dev/spec/11/" target="_blank" rel="noopener noreferrer">https://rfc.vac.dev/spec/11/</a><a href="#fnref-17-3deade" class="footnote-backref">↩</a></li><li id="fn-8-3deade">Proof of work: <a href="http://www.infosecon.net/workshop/downloads/2004/pdf/clayton.pdf" target="_blank" rel="noopener noreferrer">http://www.infosecon.net/workshop/downloads/2004/pdf/clayton.pdf</a> and <a href="https://link.springer.com/content/pdf/10.1007/3-540-48071-4_10.pdf" target="_blank" rel="noopener noreferrer">https://link.springer.com/content/pdf/10.1007/3-540-48071-4_10.pdf</a><a href="#fnref-8-3deade" class="footnote-backref">↩</a></li><li id="fn-9-3deade">EIP-627 Whisper: <a href="https://eips.ethereum.org/EIPS/eip-627" target="_blank" rel="noopener noreferrer">https://eips.ethereum.org/EIPS/eip-627</a><a href="#fnref-9-3deade" class="footnote-backref">↩</a></li><li id="fn-6-3deade">Peer Scoring: <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#peer-scoring" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#peer-scoring</a><a href="#fnref-6-3deade" class="footnote-backref">↩</a></li><li id="fn-7-3deade">Peer scoring security issues: <a href="https://github.com/vacp2p/research/issues/44" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/44</a><a href="#fnref-7-3deade" class="footnote-backref">↩</a></li><li id="fn-14-3deade">Zero Knowledge Proof: <a href="https://dl.acm.org/doi/abs/10.1145/3335741.3335750" target="_blank" rel="noopener noreferrer">https://dl.acm.org/doi/abs/10.1145/3335741.3335750</a> and <a href="https://en.wikipedia.org/wiki/Zero-knowledge_proof" target="_blank" rel="noopener noreferrer">https://en.wikipedia.org/wiki/Zero-knowledge_proof</a><a href="#fnref-14-3deade" class="footnote-backref">↩</a></li><li id="fn-15-3deade">zkSNARKs: <a href="https://link.springer.com/chapter/10.1007/978-3-662-49896-5_11" target="_blank" rel="noopener noreferrer">https://link.springer.com/chapter/10.1007/978-3-662-49896-5_11</a> and <a href="https://coinpare.io/whitepaper/zcash.pdf" target="_blank" rel="noopener noreferrer">https://coinpare.io/whitepaper/zcash.pdf</a><a href="#fnref-15-3deade" class="footnote-backref">↩</a></li><li id="fn-13-3deade">Shamir Secret Sharing Scheme: <a href="https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing" target="_blank" rel="noopener noreferrer">https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing</a><a href="#fnref-13-3deade" class="footnote-backref">↩</a></li><li id="fn-22-3deade">zkSNARKs proof time: <a href="https://github.com/vacp2p/research/issues/7" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/7</a><a href="#fnref-22-3deade" class="footnote-backref">↩</a></li><li id="fn-21-3deade">Prover key size: <a href="https://github.com/vacp2p/research/issues/8" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/8</a><a href="#fnref-21-3deade" class="footnote-backref">↩</a></li><li id="fn-19-3deade">The lack of Shamir secret sharing in zkSNARKs: <a href="https://github.com/vacp2p/research/issues/10" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/10</a><a href="#fnref-19-3deade" class="footnote-backref">↩</a></li><li id="fn-20-3deade">The MPC required for zkSNARKs trusted setup: <a href="https://github.com/vacp2p/research/issues/9" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/9</a><a href="#fnref-20-3deade" class="footnote-backref">↩</a></li><li id="fn-11-3deade">Storage overhead per peer: <a href="https://github.com/vacp2p/research/issues/57" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/57</a><a href="#fnref-11-3deade" class="footnote-backref">↩</a></li><li id="fn-12-3deade">Storage-efficient Merkle Tree maintenance: <a href="https://github.com/vacp2p/research/pull/54" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/pull/54</a><a href="#fnref-12-3deade" class="footnote-backref">↩</a></li><li id="fn-10-3deade">Cost-effective way of member insertion and deletion: <a href="https://github.com/vacp2p/research/issues/56" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/research/issues/56</a><a href="#fnref-10-3deade" class="footnote-backref">↩</a></li><li id="fn-23-3deade">Attack on the messaging rate: <a href="https://github.com/vacp2p/specs/issues/251" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/specs/issues/251</a><a href="#fnref-23-3deade" class="footnote-backref">↩</a></li><li id="fn-5-3deade">RLN Benchmark: <a href="https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Benchmarks" target="_blank" rel="noopener noreferrer">https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?view#Benchmarks</a><a href="#fnref-5-3deade" class="footnote-backref">↩</a></li></ol></div>]]></content:encoded>
</item>
<item>
<title><![CDATA[[Talk] Vac, Waku v2 and Ethereum Messaging]]></title>
<link>https://vac.dev/rlog/waku-v2-ethereum-messaging</link>
<guid>https://vac.dev/rlog/waku-v2-ethereum-messaging</guid>
<pubDate>Tue, 10 Nov 2020 12:00:00 GMT</pubDate>
<description><![CDATA[Talk from Taipei Ethereum Meetup. Read on to find out about our journey from Whisper to Waku v2, as well as how Waku v2 can be useful for Etherum Messaging.]]></description>
<content:encoded><![CDATA[<p>Talk from Taipei Ethereum Meetup. Read on to find out about our journey from Whisper to Waku v2, as well as how Waku v2 can be useful for Etherum Messaging.</p><p><em>The following post is a transcript of the talk given at the <a href="https://www.meetup.com/Taipei-Ethereum-Meetup/events/274033344/" target="_blank" rel="noopener noreferrer">Taipei Ethereum meetup, November 5</a>. There is also a <a href="https://www.youtube.com/watch?v=lUDy1MoeYnI" target="_blank" rel="noopener noreferrer">video recording</a>.</em></p><hr><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="0-introduction">0. Introduction<a href="#0-introduction" class="hash-link" aria-label="Direct link to 0. Introduction" title="Direct link to 0. Introduction">​</a></h2><p>Hi! My name is Oskar and I'm the protocol research lead at Vac. This talk will be divided into two parts. First I'll talk about the journey from Whisper, to Waku v1 and now to Waku v2. Then I'll talk about messaging in Ethereum. After this talk, you should have an idea of what Waku v2 is, the problems it is trying to solve, as well as where it can be useful for messaging in Ethereum.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="part-1---vac-and-the-journey-from-whisper-to-waku-v1-to-waku-v2">PART 1 - VAC AND THE JOURNEY FROM WHISPER TO WAKU V1 TO WAKU V2<a href="#part-1---vac-and-the-journey-from-whisper-to-waku-v1-to-waku-v2" class="hash-link" aria-label="Direct link to PART 1 - VAC AND THE JOURNEY FROM WHISPER TO WAKU V1 TO WAKU V2" title="Direct link to PART 1 - VAC AND THE JOURNEY FROM WHISPER TO WAKU V1 TO WAKU V2">​</a></h2><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-vac-intro">1. Vac intro<a href="#1-vac-intro" class="hash-link" aria-label="Direct link to 1. Vac intro" title="Direct link to 1. Vac intro">​</a></h2><p>First, what is Vac? Vac grew out of our efforts Status to create a window on to Ethereum and secure messenger. Vac is modular protocol stack for p2p secure messaging, paying special attention to resource restricted devices, privacy and censorship resistance.</p><p>Today we are going to talk mainly about Waku v2, which is the transport privacy / routing aspect of the Vac protocol stack. It sits "above" the p2p overlay, such as libp2p dealing with transports etc, and below a conversational security layer dealing with messaging encryption, such as using Double Ratchet etc.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-whisper-to-waku-v1">2. Whisper to Waku v1<a href="#2-whisper-to-waku-v1" class="hash-link" aria-label="Direct link to 2. Whisper to Waku v1" title="Direct link to 2. Whisper to Waku v1">​</a></h2><p>In the beginning, there was Whisper. Whisper was part of the holy trinity of Ethereum. You had Ethereum for consensus/computation, Whisper for messaging, and Swarm for storage.</p><p>However, for various reasons, Whisper didn't get the attention it deserved. Development dwindled, it promised too much and it suffered from many issues, such as being extremely inefficient and not being suitable for running on e.g. mobile phone. Despite this, Status used it in its app from around 2017 to 2019. As far as I know, it was one of very few, if not the only, production uses of Whisper.</p><p>In an effort to solve some of its immediate problems, we forked Whisper into Waku and formalized it with a proper specification. This solved immediate bandwidth issues for light nodes, introduced rate limiting for better spam protection, improved historical message support, etc.</p><p>If you are interested in this journey, checkout the <a href="https://www.youtube.com/watch?v=6lLT33tsJjs" target="_blank" rel="noopener noreferrer">EthCC talk Dean and I gave in Paris earlier this year</a>.</p><p>Status upgraded to Waku v1 early 2020. What next?</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-waku-v1-to-v2">3. Waku v1 to v2<a href="#3-waku-v1-to-v2" class="hash-link" aria-label="Direct link to 3. Waku v1 to v2" title="Direct link to 3. Waku v1 to v2">​</a></h2><p>We were far from done. The changes we had made were quite incremental and done in order to get tangible improvements as quickly as possible. This meant we couldn't address more fundamental issues related to full node routing scalability, running with libp2p for more transports, better security, better spam protection and incentivization.</p><p>This kickstarted Waku v2 efforts, which is what we've been working on since July. This work was and is initally centered around a few pieces:</p><p>(a) Moving to libp2p</p><p>(b) Better routing</p><p>(c) Accounting and user-run nodes</p><p>The general theme was: making the Waku network more scalable and robust.</p><p>We also did a scalability study to show at what point the network would run into issues, due to the inherent lack of routing that Whisper and Waku v1 provided.</p><p>You can read more about this <a href="https://vac.dev/waku-v2-plan" target="_blank" rel="noopener noreferrer">here</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="35-waku-v2---design-goals">3.5 Waku v2 - Design goals<a href="#35-waku-v2---design-goals" class="hash-link" aria-label="Direct link to 3.5 Waku v2 - Design goals" title="Direct link to 3.5 Waku v2 - Design goals">​</a></h2><p>Taking a step back, what problem does Waku v2 attempt to solve compared to all the other solutions that exists out there? What type of applications should use it and why? We have the following design goals:</p><ol><li><p><strong>Generalized messaging</strong>. Many applications requires some form of messaging protocol to communicate between different subsystems or different nodes. This messaging can be human-to-human or machine-to-machine or a mix.</p></li><li><p><strong>Peer-to-peer</strong>. These applications sometimes have requirements that make them suitable for peer-to-peer solutions.</p></li><li><p><strong>Resource restricted</strong>. These applications often run in constrained environments, where resources or the environment is restricted in some fashion. E.g.:</p><ul><li>limited bandwidth, CPU, memory, disk, battery, etc</li><li>not being publicly connectable</li><li>only being intermittently connected; mostly-offline</li></ul></li><li><p><strong>Privacy</strong>. These applications have a desire for some privacy guarantees, such as pseudonymity, metadata protection in transit, etc.</p></li></ol><p>As well as to do so in a modular fashion. Meaning you can find a reasonable trade-off depending on your exact requirements. For example, you usually have to trade off some bandwidth to get metadata protection, and vice versa.</p><p>The concept of designing for resource restricted devices also leads to the concept of adaptive nodes, where you have more of a continuum between full nodes and light nodes. For example, if you switch your phone from mobile data to WiFi you might be able to handle more bandwidth, and so on.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-waku-v2---breakdown">4. Waku v2 - Breakdown<a href="#4-waku-v2---breakdown" class="hash-link" aria-label="Direct link to 4. Waku v2 - Breakdown" title="Direct link to 4. Waku v2 - Breakdown">​</a></h2><p>Where is Waku v2 at now, and how is it structured?</p><p>It is running over libp2p and we had our second internal testnet last week or so. As a side note, we name our testnets after subway stations in Taipei, the first one being Nangang, and the most recent one being Dingpu.</p><p>The main implementation is written in Nim using nim-libp2p, which is also powering Nimbus, an Ethereum 2 client. There is also a PoC for running Waku v2 in the browser. On a spec level, we have the following specifications that corresponds to the components that make up Waku v2:</p><ul><li>Waku v2 - this is the main spec that explains the goals of providing generalized messaging, in a p2p context, with a focus on privacy and running on resources restricted devices.</li><li>Relay - this is the main PubSub spec that provides better routing. It builds on top of GossipSub, which is what Eth2 heavily relies on as well.</li><li>Store - this is a 1-1 protocol for light nodes to get historical messages, if they are mostly-offline.</li><li>Filter - this is a 1-1 protocol for light nodes that are bandwidth restricted to only (or mostly) get messages they care about.</li><li>Message - this explains the payload, to get some basic encryption and content topics. It corresponds roughly to envelopes in Whisper/Waku v1.</li><li>Bridge - this explains how to do bridging between Waku v1 and Waku v2 for compatibility.</li></ul><p>Right now, all protocols, with the exception of bridge, are in draft mode, meaning they have been implemented but are not yet being relied upon in production.</p><p>You can read more about the breakdown in this <a href="https://vac.dev/waku-v2-update" target="_blank" rel="noopener noreferrer">update</a> though some progress has been made since then, as well was in the <a href="https://rfc.vac.dev/spec/10" target="_blank" rel="noopener noreferrer">main Waku v2 spec</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-waku-v2---upcoming">5. Waku v2 - Upcoming<a href="#5-waku-v2---upcoming" class="hash-link" aria-label="Direct link to 5. Waku v2 - Upcoming" title="Direct link to 5. Waku v2 - Upcoming">​</a></h2><p>What's coming up next? There are a few things.</p><p>For Status to use it in production, it needs to be integrated into the main app using the Nim Node API. The bridge also needs to be implemented and tested.</p><p>For other users, we are currently overhauling the API to allow usage from a browser, e.g. To make this experience great, there are also a few underlying infrastructure things that we need in nim-libp2p, such as a more secure HTTP server in Nim, Websockets and WebRTC support.</p><p>There are also some changes we made to at what level content encryption happens, and this needs to be made easier to use in the API. This means you can use a node without giving your keys to it, which is useful in some environments.</p><p>More generally, beyond getting to production-ready use, there are a few bigger pieces that we are working on or will work on soon. These are things like:</p><ul><li>Better scaling, by using topic sharding.</li><li>Accounting and user-run nodes, to account for and incentives full nodes.</li><li>Stronger and more rigorous privacy guarantees, e.g. through study of GossipSub, unlinkable packet formats, etc.</li><li>Rate Limit Nullifier for privacy preserving spam protection, a la what Barry Whitehat has presented before.</li></ul><p>As well as better support for Ethereum M2M Messaging. Which is what I'll talk about next.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="part-2---ethereum-messaging">PART 2 - ETHEREUM MESSAGING<a href="#part-2---ethereum-messaging" class="hash-link" aria-label="Direct link to PART 2 - ETHEREUM MESSAGING" title="Direct link to PART 2 - ETHEREUM MESSAGING">​</a></h2><p>A lot of what follows is inspired by exploratory work that John Lea has done at Status, previously Head of UX Architecture at Ubuntu.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="6-ethereum-messaging---why">6. Ethereum Messaging - Why?<a href="#6-ethereum-messaging---why" class="hash-link" aria-label="Direct link to 6. Ethereum Messaging - Why?" title="Direct link to 6. Ethereum Messaging - Why?">​</a></h2><p>It is easy to think that Waku v2 is only for human to human messaging, since that's how Waku is currently primarily used in the Status app. However, the goal is to be useful for generalized messaging, which includes other type of information as well as machine to machine messaging.</p><p>What is Ethereum M2M messaging? Going back to the Holy Trinity of Ethereum/Whisper/Swarm, the messaging component was seen as something that could facilitate messages between dapps and acts as a building block. This can help with things such as:</p><ul><li>Reducing on-chain transactions</li><li>Reduce latency for operations</li><li>Decentralize centrally coordinated services (like WalletConnect)</li><li>Improve UX of dapps</li><li>Broadcast live information</li><li>A message transport layer for state channels</li></ul><p>And so on.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="7-ethereum-messaging---why-cont">7. Ethereum Messaging - Why? (Cont)<a href="#7-ethereum-messaging---why-cont" class="hash-link" aria-label="Direct link to 7. Ethereum Messaging - Why? (Cont)" title="Direct link to 7. Ethereum Messaging - Why? (Cont)">​</a></h2><p>What are some examples of practical things Waku as used for Ethereum Messaging could solve?</p><ul><li>Multisig transfers only needing one on chain transaction</li><li>DAO votes only needing one one chain transaction</li><li>Giving dapps ability to direct push notifications to users</li><li>Giving users ability to directly respond to requests from daps</li><li>Decentralized Wallet Connect</li></ul><p>Etc.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="8-whats-needed-to-deliver-this">8. What's needed to deliver this?<a href="#8-whats-needed-to-deliver-this" class="hash-link" aria-label="Direct link to 8. What's needed to deliver this?" title="Direct link to 8. What's needed to deliver this?">​</a></h2><p>We can break it down into our actors:</p><ul><li>Decentralized M2M messaging system (Waku)</li><li>Native wallets (Argent, Metamask, Status, etc)</li><li>Dapps that benefit from M2M messaging</li><li>Users whose problems are being solved</li></ul><p>Each of these has a bunch of requirements in turn. The messaging system needs to be decentralized, scalable, robust, etc. Wallets need support for messaging layer, dapps need to integrate this, etc.</p><p>This is a lot! Growing adoption is a challenge. There is a catch 22 in terms of justifying development efforts for wallets, when no dapps need it, and likewise for dapps when no wallets support Waku. In addition to this, there must be proven usage of Waku before it can be relied on, etc. How can we break this up into smaller pieces of work?</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="9-breaking-up-the-problem-and-a-high-level-roadmap">9. Breaking up the problem and a high level roadmap<a href="#9-breaking-up-the-problem-and-a-high-level-roadmap" class="hash-link" aria-label="Direct link to 9. Breaking up the problem and a high level roadmap" title="Direct link to 9. Breaking up the problem and a high level roadmap">​</a></h2><p>We can start small. It doesn't and need to be used for critical features first. A more hybrid approach can be taken where it acts more as nice-to-haves.</p><ol><li>Forking Whisper and solving scalablity, spam etc issues with it.
This is a work in progress. What we talked about in part 1.</li><li>Expose messaging API for Dapp developers.</li><li>Implement decentralized version of WalletConnect.
Currently wallets connect ot dapps with centralized service. Great UX.</li><li>Solve DAO/Multi-Sig coordination problem.
E.g. send message to wallet-derived key when it is time to sign a transaction.</li><li>Extend dapp-to-user and user-to-dapp communication to more dapps.
Use lessons learned and examples to drive adoptation for wallets/dapps.</li></ol><p>And then build up from there.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="10-we-are-hiring">10. We are hiring!<a href="#10-we-are-hiring" class="hash-link" aria-label="Direct link to 10. We are hiring!" title="Direct link to 10. We are hiring!">​</a></h2><p>A lot of this will happen in Javascript and browsers, since that's the primarily environment for a lot of wallets and dapps. We are currently hiring for a Waku JS Wallet integration lead to help push this effort further.</p><p>Come talk to me after or <a href="https://status.im/our_team/open_positions.html?gh_jid=2385338" target="_blank" rel="noopener noreferrer">apply here</a>.</p><p>That's it! You can find us on Status, Telegram, vac.dev. I'm on twitter <a href="https://twitter.com/oskarth" target="_blank" rel="noopener noreferrer">here</a>.</p><p>Questions?</p><hr>]]></content:encoded>
</item>
<item>
<title><![CDATA[Waku v2 Update]]></title>
<link>https://vac.dev/rlog/waku-v2-update</link>
<guid>https://vac.dev/rlog/waku-v2-update</guid>
<pubDate>Mon, 28 Sep 2020 12:00:00 GMT</pubDate>
<description><![CDATA[A research log. Read on to find out what is going on with Waku v2, a messaging protocol. What has been happening? What is coming up next?]]></description>
<content:encoded><![CDATA[<p>A research log. Read on to find out what is going on with Waku v2, a messaging protocol. What has been happening? What is coming up next?</p><p>It has been a while since the last post. It is time for an update on Waku v2. Aside from getting more familiar with libp2p (specifically nim-libp2p) and some vacation, what have we been up to? In this post we'll talk about what we've gotten done since last time, and briefly talk about immediate next steps and future. But first, a recap.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="recap">Recap<a href="#recap" class="hash-link" aria-label="Direct link to Recap" title="Direct link to Recap">​</a></h2><p>In the last post (<a href="https://vac.dev/waku-v2-plan" target="_blank" rel="noopener noreferrer">Waku v2 plan</a>) we explained the rationale of Waku v2 - the current Waku network is fragile and doesn't scale. To solve this, Waku v2 aims to reduce amplification factors and get more user run nodes. We broke the work down into three separate tracks.</p><ol><li>Track 1 - Move to libp2p</li><li>Track 2 - Better routing</li><li>Track 3 - Accounting and user-run nodes</li></ol><p>As well as various rough components for each track. The primary initial focus is track 1. This means things like: moving to FloodSub, simplify the protocol, core integration, topic interest behavior, historical message caching, and Waku v1&lt;<!-- -->&gt;<!-- -->v2 bridge.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="current-state">Current state<a href="#current-state" class="hash-link" aria-label="Direct link to Current state" title="Direct link to Current state">​</a></h2><p>Let's talk about the state of specs and our main implementation nim-waku. Then we'll go over our recent testnet, Nangang, and finish off with a Web PoC.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="specs">Specs<a href="#specs" class="hash-link" aria-label="Direct link to Specs" title="Direct link to Specs">​</a></h2><p>After some back and forth on how to best structure things, we ended up breaking down the specs into a few pieces. While Waku v2 is best thought of as a cohesive whole in terms of its capabilities, it is made up of several protocols. Here's a list of the current specs and their status:</p><ul><li><a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Main spec</a> (draft)</li><li><a href="https://rfc.vac.dev/spec/11/" target="_blank" rel="noopener noreferrer">Relay protocol spec</a> (draft)</li><li><a href="https://rfc.vac.dev/spec/12" target="_blank" rel="noopener noreferrer">Filter protocol spec</a> (raw)</li><li><a href="https://rfc.vac.dev/spec/13" target="_blank" rel="noopener noreferrer">Store protocol spec</a> (raw)</li><li><a href="https://rfc.vac.dev/spec/15/" target="_blank" rel="noopener noreferrer">Bridge spec</a> (raw)</li></ul><p>Raw means there is not yet an implementation that corresponds fully to the spec, and draft means there is an implementation that corresponds to the spec. In the interest of space, we won't go into too much detail on the specs here except to note a few things:</p><ul><li>The relay spec is essentially a thin wrapper on top of PubSub/FloodSub/GossipSub</li><li>The filter protocol corresponds to previous light client mode in Waku v1</li><li>The store protocol corresponds to the previous mailserver construct in Waku v1</li></ul><p>The filter and store protocol allow for adaptive nodes, i.e. nodes that have various capabilities. For example, a node being mostly offline, or having limited bandwidth capacity. The bridge spec outlines how to bridge the Waku v1 and v2 networks.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="implementation">Implementation<a href="#implementation" class="hash-link" aria-label="Direct link to Implementation" title="Direct link to Implementation">​</a></h2><p>The main implementation we are working on is <a href="https://github.com/status-im/nim-waku/" target="_blank" rel="noopener noreferrer">nim-waku</a>. This builds on top of libraries such as <a href="https://github.com/status-im/nim-libp2p" target="_blank" rel="noopener noreferrer">nim-libp2p</a> and others that the <a href="https://nimbus.team/" target="_blank" rel="noopener noreferrer">Nimbus team</a> have been working on as part of their Ethereum 2.0 client.</p><p>Currently nim-waku implements the relay protocol, and is close to implementing filter and store protocol. It also exposes a <a href="https://github.com/status-im/nim-waku/blob/master/docs/api/v2/node.md" target="_blank" rel="noopener noreferrer">Nim Node API</a> that allows libraries such as <a href="https://github.com/status-im/status-nim" target="_blank" rel="noopener noreferrer">nim-status</a> to use it. Additionally, there is also a rudimentary JSON RPC API for command line scripting.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="nangang-testnet">Nangang testnet<a href="#nangang-testnet" class="hash-link" aria-label="Direct link to Nangang testnet" title="Direct link to Nangang testnet">​</a></h2><p>Last week we launched a very rudimentary internal testnet called Nangang. The goal was to test basic connectivity and make sure things work end to end. It didn't have things like: client integration, encryption, bridging, multiple clients, store/filter protocol, or even a real interface. What it did do is allow Waku developers to "chat" via RPC calls and looking in the log output. Doing this meant we exposed and fixed a few blockers, such as connection issues, deployment, topic subscription management, protocol and node integration, and basic scripting/API usage. After this, we felt confident enough to upgrade the main and relay spec to "draft" status.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-web-poc">Waku Web PoC<a href="#waku-web-poc" class="hash-link" aria-label="Direct link to Waku Web PoC" title="Direct link to Waku Web PoC">​</a></h2><p>As a bonus, we wanted to see what it'd take to get Waku running in a browser. This is a very powerful capability that enables a lot of use cases, and something that libp2p enables with its multiple transport support.</p><p>Using the current stack, with nim-waku, would require quite a lot of ground work with WASM, WebRTC, Websockets support etc. Instead, we decided to take a shortcut and hack together a JS implementation called <a href="https://github.com/vacp2p/waku-web-chat/" target="_blank" rel="noopener noreferrer">Waku Web Chat</a>. This quick hack wouldn't be possible without the people behind <a href="https://github.com/libp2p/js-libp2p-examples/" target="_blank" rel="noopener noreferrer">js-libp2p-examples</a> and <a href="https://github.com/libp2p/js-libp2p" target="_blank" rel="noopener noreferrer">js-libp2p</a> and all its libraries. These are people like Jacob Heun, Vasco Santos, and Cayman Nava. Thanks!</p><p>It consists of a brower implementation, a NodeJS implementation and a bootstrap server that acts as a signaling server for WebRTC. It is largely a bastardized version of GossipSub, and while it isn't completely to spec, it does allow messages originating from a browser to eventually end up at a nim-waku node, and vice versa. Which is pretty cool.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="coming-up">Coming up<a href="#coming-up" class="hash-link" aria-label="Direct link to Coming up" title="Direct link to Coming up">​</a></h2><p>Now that we know what the current state is, what is still missing? what are the next steps?</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="things-that-are-missing">Things that are missing<a href="#things-that-are-missing" class="hash-link" aria-label="Direct link to Things that are missing" title="Direct link to Things that are missing">​</a></h2><p>While we are getting closer to closing out work for track 1, there are still a few things missing from the initial scope:</p><ol><li><p>Store and filter protocols need to be finished. This means basic spec, implementation, API integration and proven to work in a testnet. All of these are work in progress and expected to be done very soon. Once the store protocol is done in a basic form, it needs further improvements to make it production ready, at least on a spec/basic implementation level.</p></li><li><p>Core integration was mentioned in scope for track 1 initially. This work has stalled a bit, largely due to organizational bandwidth and priorities. While there is a Nim Node API that in theory is ready to be used, having it be used in e.g. Status desktop or mobile app is a different matter. The team responsible for this at Status (<a href="https://github.com/status-im/status-nim" target="_blank" rel="noopener noreferrer">status-nim</a> has been making progress on getting nim-waku v1 integrated, and is expected to look into nim-waku v2 integration soon. One thing that makes this a especially tricky is the difference in interface between Waku v1 and v2, which brings
us too...</p></li><li><p>Companion spec for encryption. As part of simplifying the protocol, the routing is decoupled from the encryption in v2 (<a href="https://github.com/vacp2p/specs/issues/158" target="_blank" rel="noopener noreferrer">1</a>, <a href="https://github.com/vacp2p/specs/issues/181" target="_blank" rel="noopener noreferrer">2</a>). There are multiple layers of encryption at play here, and we need to figure out a design that makes sense for various use cases (dapps using Waku on their own, Status app, etc).</p></li><li><p>Bridge implementation. The spec is done and we know how it should work, but it needs to be implemented.</p></li><li><p>General tightening up of specs and implementation.</p></li></ol><p>While this might seem like a lot, a lot has been done already, and the majority of the remaining tasks are more amendable to be pursued in parallel with other efforts. It is also worth mentioning that part of track 2 and 3 have been started, in the form of moving to GossipSub (amplification factors) and basics of adaptive nodes (multiple protocols). This is in addition to things like Waku Web which were not part of the initial scope.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="upcoming">Upcoming<a href="#upcoming" class="hash-link" aria-label="Direct link to Upcoming" title="Direct link to Upcoming">​</a></h2><p>Aside from the things mentioned above, what is coming up next? There are a few areas of interest, mentioned in no particular order. For track 2 and 3, see previous post for more details.</p><ol><li><p>Better routing (track 2). While we are already building on top of GossipSub, we still need to explore things like topic sharding in more detail to further reduce amplification factors.</p></li><li><p>Accounting and user-run nodes (track 3). With store and filter protocol getting ready, we can start to implement accounting and light connection game for incentivization in a bottom up and iterative manner.</p></li><li><p>Privacy research. Study better and more rigorous privacy guarantees. E.g. how FloodSub/GossipSub behaves for common threat models, and how custom packet
format can improve things like unlinkability.</p></li><li><p>zkSnarks RLN for spam protection and incentivization. We studied this <a href="https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks" target="_blank" rel="noopener noreferrer">last year</a> and recent developments have made this relevant to study again. Create an <a href="https://github.com/vacp2p/specs/issues/189" target="_blank" rel="noopener noreferrer">experimental spec/PoC</a> as an extension to the relay protocol. Kudos to Barry Whitehat and others like Kobi Gurkan and Koh Wei Jie for pushing this!</p></li><li><p>Ethereum M2M messaging. Being able to run in the browser opens up a lot of doors, and there is an opportunity here to enable things like a decentralized WalletConnect, multi-sig transactions, voting and similar use cases. This was the original goal of Whisper, and we'd like to deliver on that.</p></li></ol><p>As you can tell, quite a lot of thing! Luckily, we have two people joining as protocol engineers soon, which will bring much needed support for the current team of ~2-2.5 people. More details to come in further updates.</p><hr><p>If you are feeling adventurous and want to use early stage alpha software, check out the <a href="https://github.com/status-im/nim-waku/tree/master/docs" target="_blank" rel="noopener noreferrer">docs</a>. If you want to read the specs, head over to <a href="https://rfc.vac.dev/spec/10/" target="_blank" rel="noopener noreferrer">Waku spec</a>. If you want to talk with us, join us on <a href="https://get.status.im/chat/public/vac" target="_blank" rel="noopener noreferrer">Status</a> or on <a href="https://t.me/vacp2p" target="_blank" rel="noopener noreferrer">Telegram</a> (they are bridged).</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[What's the Plan for Waku v2?]]></title>
<link>https://vac.dev/rlog/waku-v2-plan</link>
<guid>https://vac.dev/rlog/waku-v2-plan</guid>
<pubDate>Wed, 01 Jul 2020 12:00:00 GMT</pubDate>
<description><![CDATA[Read about our plans for Waku v2, moving to libp2p, better routing, adaptive nodes and accounting!]]></description>
<content:encoded><![CDATA[<p>Read about our plans for Waku v2, moving to libp2p, better routing, adaptive nodes and accounting!</p><p><strong>tldr: The Waku network is fragile and doesn't scale. Here's how to solve it.</strong></p><p><em>NOTE: This post was originally written with Status as a primary use case in mind, which reflects how we talk about some problems here. However, Waku v2 is a general-purpose private p2p messaging protocol, especially for people running in resource restricted environments.</em></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="problem">Problem<a href="#problem" class="hash-link" aria-label="Direct link to Problem" title="Direct link to Problem">​</a></h2><p>The Waku network is fragile and doesn't scale.</p><p>As <a href="https://status.im" target="_blank" rel="noopener noreferrer">Status</a> is moving into a user-acquisition phase and is improving retention rates for users they need the infrastructure to keep up, specifically when it comes to messaging.</p><p>Based on user acquisition models, the initial goal is to support 100k DAU in September, with demand growing from there.</p><p>With the Status Scaling Model we have studied the current bottlenecks as a function of concurrent users (CCU) and daily active users (DAU). Here are the conclusions.</p><p>*<strong>*<!-- -->1. Connection limits<!-- -->*<!-- -->*</strong>. With 100 full nodes we reach ~10k CCU based on connection limits. This can primarily be addressed by increasing the number of nodes (cluster or user operated). This assumes node discovery works. It is also worth investigating the limitations of max number of connections, though this is likely to be less relevant for user-operated nodes. For a user-operated network, this means 1% of users have to run a full node. See Fig 1-2.</p><p>*<strong>*<!-- -->2. Bandwidth as a bottleneck<!-- -->*<!-- -->*</strong>. We notice that memory usage appears to not be
the primary bottleneck for full nodes, and the bottleneck is still bandwidth. To support 10k DAU, and full nodes with an amplification factor of 25 the required Internet speed is ~50 Mbps, which is a fast home Internet connection. For ~100k DAU only cloud-operated nodes can keep up (500 Mbps). See Fig 3-5.</p><p>*<strong>*<!-- -->3. Amplification factors<!-- -->*<!-- -->*</strong>. Reducing amplification factors with better routing, would have a high impact, but it is likely we'd need additional measures as well, such as topic sharding or similar. See Fig 8-13.</p><p>Figure 1-5:</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig1-64444f5b246c9fd1014093b193fc3453.png" width="507" height="337" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig2-774c26bf57c92c1d1a30cc6eddc95180.png" width="512" height="337" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig3-47788396aa8aa03dcb9fda2dd4ac306f.png" width="513" height="339" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig4-f8aaace8b1a824bee26cc451699faa6b.png" width="528" height="328" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig5-ffdec86c00f77ab94306c7ae27e9c8c7.png" width="523" height="353" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>See <a href="https://colab.research.google.com/drive/1Fz-oxRxxAFPpM1Cowpnb0nT52V1-yeRu#scrollTo=Yc3417FUJJ_0" target="_blank" rel="noopener noreferrer">https://colab.research.google.com/drive/1Fz-oxRxxAFPpM1Cowpnb0nT52V1-yeRu#scrollTo=Yc3417FUJJ_0</a> for the full report.</p><p>What we need to do is:</p><ol><li>Reduce amplification factors</li><li>Get more user-run full nodes</li></ol><p>Doing this means the Waku network will be able to scale, and doing so in the right way, in a robust fashion. What would a fragile way of scaling be? Increasing our reliance on a Status Pte Ltd operated cluster which would paint us in a corner where we:</p><ul><li>keep increasing requirements for Internet speed for full nodes</li><li>are vulnerable to censorship and attacks</li><li>have to control the topology in an artifical manner to keep up with load</li><li>basically re-invent a traditional centralized client-server app with extra steps</li><li>deliberately ignore most of our principles</li><li>risk the network being shut down when we run out of cash</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="appetite">Appetite<a href="#appetite" class="hash-link" aria-label="Direct link to Appetite" title="Direct link to Appetite">​</a></h2><p>Our initial risk appetite for this is 6 weeks for a small team.</p><p>The idea is that we want to make tangible progress towards the goal in a limited period of time, as opposed to getting bogged down in trying to find a theoretically perfect generalized solution. Fixed time, variable scope.</p><p>It is likely some elements of a complete solution will be done separately. See later sections for that.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="solution">Solution<a href="#solution" class="hash-link" aria-label="Direct link to Solution" title="Direct link to Solution">​</a></h2><p>There are two main parts of the solution. One is to reduce amplification factors, and the other is incentivization to get more user run full nodes with desktop, etc.</p><p>What does a full node provide? It provides connectivity to the network, can act as a bandwidth "barrier" and be high or reasonably high availability. What this means right now is essentially topic interest and storing historical messages.</p><p>The goal is here to improve the status quo, not get a perfect solution from the get go. All of this can be iterated on further, for stronger guarantees, as well as replaced by other new modules.</p><p>Let's first look at the baseline, and then go into some of the tracks and their phases. Track 1 is best done first, after which track 2 and 3 can be executed in parallel. Track 1 gives us more options for track 2 and 3. The work in track 1 is currently more well-defined, so it is likely the specifics of track 2 and 3 will get refined at a later stage.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="baseline">Baseline<a href="#baseline" class="hash-link" aria-label="Direct link to Baseline" title="Direct link to Baseline">​</a></h2><p>Here's where we are at now. In reality, the amplification factor are likely even worse than this (15 in the graph below), up to 20-30. Especially with an open network, where we can't easily control connectivity and availability of nodes. Left unchecked, with a full mesh, it could even go as high x100, though this is likely excessive and can be dialed down. See scaling model for more details.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku_v1_routing_small-65bf881ec98bcded566accbbc4f2262d.png" width="1500" height="798" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="track-1---move-to-libp2p">Track 1 - Move to libp2p<a href="#track-1---move-to-libp2p" class="hash-link" aria-label="Direct link to Track 1 - Move to libp2p" title="Direct link to Track 1 - Move to libp2p">​</a></h2><p>Moving to PubSub over libp2p wouldn't improve amplification per se, but it would be stepping stone. Why? It paves the way for GossipSub, and would be a checkpoint on this journey. Additionally, FloodSub and GossipSub are compatible, and very likely other future forms of PubSub such as GossipSub 1.1 (hardened/more secure), EpiSub, forwarding Kademlia / PubSub over Kademlia, etc. Not to mention security This would also give us access to the larger libp2p ecosystem (multiple protocols, better encryption, quic, running in the browser, security audits, etc, etc), as well as be a joint piece of infrastructured used for Eth2 in Nimbus. More wood behind fewer arrows.</p><p>See more on libp2p PubSub here: <a href="https://docs.libp2p.io/concepts/publish-subscribe/" target="_blank" rel="noopener noreferrer">https://docs.libp2p.io/concepts/publish-subscribe/</a></p><p>As part of this move, there are a few individual pieces that are needed.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-floodsub">1. FloodSub<a href="#1-floodsub" class="hash-link" aria-label="Direct link to 1. FloodSub" title="Direct link to 1. FloodSub">​</a></h3><p>This is essentially what Waku over libp2p would look like in its most basic form.</p><p>One difference that is worth noting is that the app topics would <strong>not</strong> be the same as Waku topics. Why? In Waku we currently don't use topics for routing between full nodes, but only for edge/light nodes in the form of topic interest. In FloodSub, these topics are used for routing.</p><p>Why can't we use Waku topics for routing directly? PubSub over libp2p isn't built for rare and ephemeral topics, and nodes have to explicitly subscribe to a topic. See topic sharding section for more on this.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku_v2_routing_flood_small-1b9dc4447d094efc9f29656c534ca3b9.png" width="777" height="399" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Moving to FloodSub over libp2p would also be an opportunity to clean up and simplify some components that are no longer needed in the Waku v1 protocol, see point below.</p><p>Very experimental and incomplete libp2p support can be found in the nim-waku repo under v2: <a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer">https://github.com/status-im/nim-waku</a></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-simplify-the-protocol">2. Simplify the protocol<a href="#2-simplify-the-protocol" class="hash-link" aria-label="Direct link to 2. Simplify the protocol" title="Direct link to 2. Simplify the protocol">​</a></h3><p>Due to Waku's origins in Whisper, devp2p and as a standalone protocol, there are a lot of stuff that has accumulated (<a href="https://rfc.vac.dev/spec/6/" target="_blank" rel="noopener noreferrer">https://rfc.vac.dev/spec/6/</a>). Not all of it serves it purpose anymore. For example, do we still need RLP here when we have Protobuf messages? What about extremely low PoW when we have peer scoring? What about key management / encryption when have encryption at libp2p and Status protocol level?</p><p>Not everything has to be done in one go, but being minimalist at this stage will the protocol lean and make us more adaptable.</p><p>The essential characteristic that has to be maintained is that we don't need to change the upper layers, i.e. we still deal with (Waku) topics and some envelope like data unit.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-core-integration">3. Core integration<a href="#3-core-integration" class="hash-link" aria-label="Direct link to 3. Core integration" title="Direct link to 3. Core integration">​</a></h3><p>As early as possible we want to integrate with Core via Stimbus in order to mitigate risk and catch integration issues early in the process. What this looks like in practice is some set of APIs, similar to how Whisper and Waku were working in parallel, and experimental feature behind a toggle in core/desktop.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-topic-interest-behavior">4. Topic interest behavior<a href="#4-topic-interest-behavior" class="hash-link" aria-label="Direct link to 4. Topic interest behavior" title="Direct link to 4. Topic interest behavior">​</a></h3><p>While we target full node traffic here, we want to make sure we maintain the existing bandwidth requirements for light nodes that Waku v1 addressed (<a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">https://vac.dev/fixing-whisper-with-waku</a>). This means implementing topic-interest in the form of Waku topics. Note that this would be separate from app topics notes above.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-historical-message-caching">5. Historical message caching<a href="#5-historical-message-caching" class="hash-link" aria-label="Direct link to 5. Historical message caching" title="Direct link to 5. Historical message caching">​</a></h3><p>Basically what mailservers are currently doing. This likely looks slightly different in a libp2p world. This is another opportunity to simplify things with a basic REQ-RESP architecture, as opposed to the roundabout way things are now. Again, not everything has to be done in one go but there's no reason to reimplement a poor API if we don't have to.</p><p>Also see section below on adaptive nodes and capabilities.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="6-waku-v1--libp2p-bridge">6. Waku v1 &lt;<!-- -->&gt;<!-- --> Libp2p bridge<a href="#6-waku-v1--libp2p-bridge" class="hash-link" aria-label="Direct link to 6-waku-v1--libp2p-bridge" title="Direct link to 6-waku-v1--libp2p-bridge">​</a></h3><p>To make the transition complete, there has to a be bridge mode between current Waku and libp2p. This is similar to what was done for Whisper and Waku, and allows any nodes in the network to upgrade to Waku v2 at their leisure. For example, this would likely look different for Core, Desktop, Research and developers.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="track-2---better-routing">Track 2 - Better routing<a href="#track-2---better-routing" class="hash-link" aria-label="Direct link to Track 2 - Better routing" title="Direct link to Track 2 - Better routing">​</a></h2><p>This is where we improve the amplification factors.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-gossipsub">1. GossipSub<a href="#1-gossipsub" class="hash-link" aria-label="Direct link to 1. GossipSub" title="Direct link to 1. GossipSub">​</a></h3><p>This is a subprotocol of FloodSub in the libp2p world. Moving to GossipSub would allow traffic between full nodes to go from an amplification factor of ~25 to ~6. This basically creates a mesh of stable bidirectional connections, together with some gossiping capabilities outside of this view.</p><p>Explaining how GossipSub works is out of scope of this document. It is implemented in nim-libp2p and used by Nimbus as part of Eth2. You can read the specs here in more detail if you are interested: <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md</a> and <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md</a></p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku_v2_routing_gossip_small-f8d8b32be158ac22c72dc2199b2b8a5a.png" width="874" height="654" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig8-0ebc4f49d1fcf101e3bb712af814a2b0.png" width="523" height="333" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig9-7b81a7e5843abd908d28b9e5915b6deb.png" width="539" height="336" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig10-d1535b8a2d4061f75fe94d85002c9d31.png" width="521" height="337" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig11-1b7a924152d9305d69fe266f18812a9f.png" width="553" height="343" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>While we technically could implement this over existing Waku, we'd have to re-implement it, and we'd lose out on all the other benefits libp2p would provide, as well as the ecosystem of people and projects working on improving the scalability and security of these protocols.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-topic-sharding">2. Topic sharding<a href="#2-topic-sharding" class="hash-link" aria-label="Direct link to 2. Topic sharding" title="Direct link to 2. Topic sharding">​</a></h3><p>This one is slightly more speculative in terms of its ultimate impact. The basic idea is to split the application topic into N shards, say 10, and then each full node can choose which shards to listen to. This can reduce amplification factors by another factor of 10.</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku_v2_routing_sharding_small-b830b07d0e5c1ba984dda0e21481328b.png" width="922" height="222" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig12-e590a87c5895c90a527a9d21ac1e7fcf.png" width="541" height="335" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div>
<div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/status_scaling_model_fig13-3eacca1ad71d153bfa9f55c665946e60.png" width="541" height="338" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Note that this means a light node that listens to several topics would have to be connected to more full nodes to get connectivity. For a more exotic version of this, see <a href="https://forum.vac.dev/t/rfc-topic-propagation-extension-to-libp2p-pubsub/47" target="_blank" rel="noopener noreferrer">https://forum.vac.dev/t/rfc-topic-propagation-extension-to-libp2p-pubsub/47</a></p><p>This is orthogonal from the choice of FloodSub or GossipSub, but due to GossipSub's more dynamic nature it is likely best combined with it.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-other-factors">3. Other factors<a href="#3-other-factors" class="hash-link" aria-label="Direct link to 3. Other factors" title="Direct link to 3. Other factors">​</a></h3><p>Not a primary focus, but worth a look. Looking at the scaling model, there might be other easy wins to improve overall bandwidth consumption between full nodes. For example, can we reduce envelope size by a significant factor?</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="track-3---accounting-and-user-run-nodes">Track 3 - Accounting and user-run nodes<a href="#track-3---accounting-and-user-run-nodes" class="hash-link" aria-label="Direct link to Track 3 - Accounting and user-run nodes" title="Direct link to Track 3 - Accounting and user-run nodes">​</a></h2><p>This is where we make sure the network isn't fragile, become a true p2p app, get our users excited and engaged, and allow us to scale the network without creating an even bigger cluster.</p><p>To work in practice, this has a soft dependency on node discovery such as DNS based discovery (<a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">https://eips.ethereum.org/EIPS/eip-1459</a>) or Discovery v5 (<a href="https://vac.dev/feasibility-discv5" target="_blank" rel="noopener noreferrer">https://vac.dev/feasibility-discv5</a>).</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-adaptive-nodes-and-capabilities">1. Adaptive nodes and capabilities<a href="#1-adaptive-nodes-and-capabilities" class="hash-link" aria-label="Direct link to 1. Adaptive nodes and capabilities" title="Direct link to 1. Adaptive nodes and capabilities">​</a></h3><p>We want to make the gradation between light nodes, full nodes, storing (partial set of) historical messages, only acting for a specific shard, etc more flexible and explicit. This is required to identify and discover the nodes you want. See <a href="https://github.com/vacp2p/specs/issues/87" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/specs/issues/87</a></p><p>Depending on how the other tracks come together, this design should allow for a desktop node to identify as a full relaying node for some some app topic shard, but also express waku topic interest and retrieve historical messages itself.</p><p>E.g. Disc v5 can be used to supply node properties through ENR.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-accounting">2. Accounting<a href="#2-accounting" class="hash-link" aria-label="Direct link to 2. Accounting" title="Direct link to 2. Accounting">​</a></h3><p>This is based on a few principles:</p><ol><li>Some nodes contribute a lot more than other nodes in the network</li><li>We can account for the difference in contribution in some fashion</li><li>We want to incentivize nodes to tell the true, and be incentivized not to lie</li></ol><p>Accounting here is a stepping stone, where accounting is the raw data upon which some settlement later occurs. It can have various forms of granularity. See <a href="https://forum.vac.dev/t/accounting-for-resources-in-waku-and-beyond/31" target="_blank" rel="noopener noreferrer">https://forum.vac.dev/t/accounting-for-resources-in-waku-and-beyond/31</a> for discussion.</p><p>We also note that in GossipSub, the mesh is bidrectional. Additionally, it doesn't appears to be a high priority issue in terms of nodes misreporting. What is an issue is having people run full nodes in the first place. There are a few points to that. It has to be possible in the end-user UX, nodes have to be discovered, and it has to be profitable/visible that you are contributing. UX and discovery are out of scope for this work, whereas visibility/accounting is part of this scope. Settlement is a stretch goal here.</p><p>The general shape of the solution is inspired by the Swarm model, where we do accounting separate from settlement. It doesn't require any specific proofs, but nodes are incentivized to tell the truth in the following way:</p><ol><li>Both full node and light node do accounting in a pairwise, local fashion</li><li>If a light node doesn't ultimately pay or lie about reporting, they get disconnected (e.g.)</li><li>If a full node doesn't provide its service the light node may pick another full node (e.g.)</li></ol><p>While accounting for individual resource usage is useful, for the ultimate end user experience we can ideally account for other things such as:</p><ul><li>end to end delivery</li><li>online time</li><li>completeness of storage</li></ul><p>This can be gradually enhanced and strengthened, for example with proofs, consistency checks, Quality of Service, reputation systems. See <a href="https://discuss.status.im/t/network-incentivisation-first-draft/1037" target="_blank" rel="noopener noreferrer">https://discuss.status.im/t/network-incentivisation-first-draft/1037</a> for one attempt to provide stronger guarantees with periodic consistency checks and a shared fund mechanism. And <a href="https://forum.vac.dev/t/incentivized-messaging-using-validity-proofs/51" target="_blank" rel="noopener noreferrer">https://forum.vac.dev/t/incentivized-messaging-using-validity-proofs/51</a> for using validity proofs and removing liveness requirement for settlement.</p><p>All of this is optional at this stage, because our goal here is to improve the status quo for user run nodes. Accounting at this stage should be visible and correspond to the net benefit a node provides to another.</p><p>As a concrete example: a light node has some topic interest and cares about historical messages on some topic. A full node communicates envelopes as they come in, communicates their high availability (online time) and stores/forward stored messages. Both nodes have this information, and if they agree settlement (initially just a mock message) can be sending a payment to an address at some time interval / over some defined volume. See future sections for how this can be improved upon.</p><p>Also see below in section 4, using constructs such as eigentrust as a local reputation mechanism.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-relax-high-availability-requirement">3. Relax high availability requirement<a href="#3-relax-high-availability-requirement" class="hash-link" aria-label="Direct link to 3. Relax high availability requirement" title="Direct link to 3. Relax high availability requirement">​</a></h3><p>If we want desktop nodes to participate in the storing of historical messages, high availability is a problem. It is a problem for any node, especially if they lie about it, but assuming they are honest it is still an issue.</p><p>By being connected to multiple nodes, we can get an overlapping online window. Then these can be combined together to get consistency. This is obviously experimental and would need to be tested before being deployed, but if it works it'd be very useful.</p><p>Additionally or alternatively, instead of putting a high requirement on message availability, focus on detection of missing information. This likely requires re-thinking how we do data sync / replication.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-incentivize-light-and-full-nodes-to-tell-the-truth-policy-etc">4. Incentivize light and full nodes to tell the truth (policy, etc)<a href="#4-incentivize-light-and-full-nodes-to-tell-the-truth-policy-etc" class="hash-link" aria-label="Direct link to 4. Incentivize light and full nodes to tell the truth (policy, etc)" title="Direct link to 4. Incentivize light and full nodes to tell the truth (policy, etc)">​</a></h3><p>In accounting phase it is largely assumed nodes are honest. What happens when they lie, and how do we incentivize them to be honest? In the case of Bittorrent this is done with tit-for-tat, however this is a different kind of relationship. What follows are some examples of how this can be done.</p><p>For light nodes:</p><ul><li>if they don't, they get disconnected</li><li>prepayment (especially to "high value" nodes)</li></ul><p>For full nodes:</p><ul><li>multiple nodes reporting to agree, where truth becomes a shelling point</li><li>use eigentrust</li><li>staking for discovery visibility with slashing</li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-settlement-poc">5. Settlement PoC<a href="#5-settlement-poc" class="hash-link" aria-label="Direct link to 5. Settlement PoC" title="Direct link to 5. Settlement PoC">​</a></h3><p>Can be done after phase 2 if so desired. Basically integrate payments based on accounting and policy.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="out-of-scope">Out of scope<a href="#out-of-scope" class="hash-link" aria-label="Direct link to Out of scope" title="Direct link to Out of scope">​</a></h2><ol><li>We assume the Status Base model requirements are accurate.</li><li>We assume Core will improve retention rates.</li><li>We assume the Stimbus production team will enable integration of nim-waku.</li><li>We assume Discovery mechanisms such as DNS and Discovery v5 will be worked on separately.</li><li>We assume Core will, at some point, provide an UX for integrating payment of services.</li><li>We assume the desktop client is sufficiently usable.</li><li>We assume Core and Infra will investigate ways of improving MaxPeers.</li></ol>]]></content:encoded>
</item>
<item>
<title><![CDATA[Feasibility Study: Discv5]]></title>
<link>https://vac.dev/rlog/feasibility-discv5</link>
<guid>https://vac.dev/rlog/feasibility-discv5</guid>
<pubDate>Mon, 27 Apr 2020 12:00:00 GMT</pubDate>
<description><![CDATA[Looking at discv5 and the theoretical numbers behind finding peers.]]></description>
<content:encoded><![CDATA[<p>Looking at discv5 and the theoretical numbers behind finding peers.</p><blockquote><p>Disclaimer: some of the numbers found in this write-up could be inaccurate. They are based on the current understanding of theoretical parts of the protocol itself by the author and are meant to provide a rough overview rather than bindable numbers.</p></blockquote><p>This post serves as a more authoritative overview of the discv5 study, for a discussionary post providing more context make sure to check out the corresponding <a href="https://discuss.status.im/t/discv5-feasibility-study/1632" target="_blank" rel="noopener noreferrer">discuss post</a>. Additionally, if you are unfamiliar with discv5, check out my previous write-up: <a href="https://vac.dev/kademlia-to-discv5" target="_blank" rel="noopener noreferrer">"From Kademlia to Discv5"</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="motivating-problem">Motivating Problem<a href="#motivating-problem" class="hash-link" aria-label="Direct link to Motivating Problem" title="Direct link to Motivating Problem">​</a></h2><p>The discovery method currently used by <a href="https://status.im" target="_blank" rel="noopener noreferrer">Status</a>, is made up of various components and grew over time to solve a mix of problems. We want to simplify this while maintaining some of the properties we currently have.</p><p>Namely, we want to ensure censorship resistance to state-level adversaries. One of the issues Status had which caused us them add to their discovery method was the fact that addresses from providers like AWS and GCP were blocked both in Russia and China. Additionally, one of the main factors required is the ability to function on resource restricted devices.</p><p>Considering we are talking about resource restricted devices, let's look at the implications and what we need to consider:</p><ul><li><strong>Battery consumption</strong> - constant connections like websockets consume a lot of battery life.</li><li><strong>CPU usage</strong> - certain discovery methods may be CPU incentive, slowing an app down and making it unusable.</li><li><strong>Bandwidth consumption</strong> - a lot of users will be using data plans, the discovery method needs to be efficient in order to accommodate those users without using up significant portions of their data plans.</li><li><strong>Short connection windows</strong> - the discovery algorithm needs to be low latency, that means it needs to return results fast. This is because many users will only have the app open for a short amount of time.</li><li><strong>Not publicly connectable</strong> - There is a good chance that most resource restricted devices are not publicly connectable.</li></ul><p>For a node to be able to participate as both a provider, and a consumer in the discovery method. Meaning a node both reads from other nodes' stored DHTs and hosts the DHT for other nodes to read from, it needs to be publically connectable. This means another node must be able to connect to some public IP of the given node.</p><p>With devices that are behind a NAT, this is easier said than done. Especially mobile devices, that when connected to 4G LTE networks are often stuck behind a symmetric NAT, drastically reducing the the succeess rate of NAT traversal. Keeping this in mind, it becomes obvious that most resource restricted devices will be consumers rather than providers due to this technical limitation.</p><p>In order to answer our questions, we formulated the problem with a simple method for testing. The "needle in a haystack" problem was formulated to figure out how easily a specific node can be found within a given network. This issue was fully formulated in <a href="https://github.com/vacp2p/research/issues/15" target="_blank" rel="noopener noreferrer">vacp2p/research#15</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="overview">Overview<a href="#overview" class="hash-link" aria-label="Direct link to Overview" title="Direct link to Overview">​</a></h2><p>The main things we wanted to investigate was the overhead on finding a peer. This means we wanted to look at both the bandwidth, latency and effectiveness of this. There are 2 methods which we can use to find a peer:</p><ul><li>We can find a peer with a specific ID, using normal lookup methods as documented by Kademlia.</li><li>We can find a peer that advertises a capability, this is possible using either capabilities advertised in the ENR or through <a href="https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement" target="_blank" rel="noopener noreferrer">topic tables</a>.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="feasbility">Feasbility<a href="#feasbility" class="hash-link" aria-label="Direct link to Feasbility" title="Direct link to Feasbility">​</a></h2><p>To be able to investigate the feasibility of discv5, we used various methods including rough calculations which can be found in the <a href="https://vac.dev/discv5-notebook/" target="_blank" rel="noopener noreferrer">notebook</a>, and a simulation isolated in <a href="https://github.com/vacp2p/research/pull/19" target="_blank" rel="noopener noreferrer">vacp2p/research#19</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="cpu--memory-usage">CPU &amp; Memory Usage<a href="#cpu--memory-usage" class="hash-link" aria-label="Direct link to CPU &amp; Memory Usage" title="Direct link to CPU &amp; Memory Usage">​</a></h3><p>The experimental discv5 has already been used within Status, however what was noticed was that the CPU and memory usage was rather high. It therefore should be investiaged if this is still the case, and if it is, it should be isolated where this stems from. Additionally it is worth looking at whether or not this is the case with both the go and nim implementation.</p><p>See details: <a href="https://github.com/vacp2p/research/issues/31" target="_blank" rel="noopener noreferrer">vacp2p/research#31</a></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="nat-on-cellular-data">NAT on Cellular Data<a href="#nat-on-cellular-data" class="hash-link" aria-label="Direct link to NAT on Cellular Data" title="Direct link to NAT on Cellular Data">​</a></h3><p>If a peer is not publically connectable it can not participate in the DHT both ways. A lot of mobile phones are behind symmetric NATs which UDP hole-punching close to impossible. It should be investigated whether or not mobile phones will be able to participate both ways and if there are good methods for doing hole-punching.</p><p>See details: <a href="https://github.com/vacp2p/research/issues/29" target="_blank" rel="noopener noreferrer">vacp2p/research#29</a></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="topic-tables">Topic Tables<a href="#topic-tables" class="hash-link" aria-label="Direct link to Topic Tables" title="Direct link to Topic Tables">​</a></h3><p>Topic Tables allow us the ability to efficiently find nodes given a specific topic. However, they are not implemented in the <a href="https://github.com/status-im/nim-eth/" target="_blank" rel="noopener noreferrer">status-im/nim-eth</a> implementation nor are they fully finalized in the spec. These are important if the network grows past a size where the concentration of specific nodes is relatively low making them hard to find.</p><p>See details: <a href="https://github.com/vacp2p/research/issues/26" target="_blank" rel="noopener noreferrer">vacp2p/research#26</a></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="finding-a-node">Finding a node<a href="#finding-a-node" class="hash-link" aria-label="Direct link to Finding a node" title="Direct link to Finding a node">​</a></h3><p>It is important to note, that given a network is relatively small sized, eg 100-500 nodes, then finding a node given a specific address is relatively managable. Additionally, if the concentration of a specific capability in a network is reasonable, then finding a node advertising its capabilities using an ENR rather than the topic table is also managable. A reasonable concentration for example would be 10%, which would give us an 80% chance of getting a node with that capability in the first lookup request. This can be explored more using our <a href="https://vac.dev/discv5-notebook/#Needle-in-a-haystack-with-ENR-records-indicating-capabilities" target="_blank" rel="noopener noreferrer">discv5 notebook</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="results">Results<a href="#results" class="hash-link" aria-label="Direct link to Results" title="Direct link to Results">​</a></h2><p>Research has shown that finding a node in the DHT has a relatively low effect on bandwidth, both inbound and outbound. For example when trying to find a node in a network of 100 nodes, it would take roughly 5668 bytes total. Additionally if we assume 100ms latency per request it would range at ≈ 300ms latency, translating to 3 requests to find a specific node.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="general-thoughts">General Thoughts<a href="#general-thoughts" class="hash-link" aria-label="Direct link to General Thoughts" title="Direct link to General Thoughts">​</a></h2><p>One of the main blockers right now is figuring out what the CPU and memory usage of discv5 is on mobile phones, this is a large blocker as it affects one of the core problems for us. We need to consider whether discv5 is an upgrade as it allows us to simplify our current discovery process or if it is too much of an overhead for resource restricted devices. The topic table feature could largely enhance discovery however it is not yet implemented. Given that CPU and memory isn't too high, discv5 could probably be used as the other issues are more "features" than large scale issues. Implementing it would already reduce the ability for state level adversaries to censor our nodes.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="acknowledgements">Acknowledgements<a href="#acknowledgements" class="hash-link" aria-label="Direct link to Acknowledgements" title="Direct link to Acknowledgements">​</a></h2><ul><li>Oskar Thoren</li><li>Dmitry Shmatko</li><li>Kim De Mey</li><li>Corey Petty</li></ul>]]></content:encoded>
</item>
<item>
<title><![CDATA[What Would a WeChat Replacement Need?]]></title>
<link>https://vac.dev/rlog/wechat-replacement-need</link>
<guid>https://vac.dev/rlog/wechat-replacement-need</guid>
<pubDate>Thu, 16 Apr 2020 12:00:00 GMT</pubDate>
<description><![CDATA[What would a self-sovereign, private, censorship-resistant and open alternative to WeChat look like?]]></description>
<content:encoded><![CDATA[<p>What would a self-sovereign, private, censorship-resistant and open alternative to WeChat look like?</p><p>What would it take to replace WeChat? More specifically, what would a self-sovereign, private, censorship-resistant and open alternative look like? One that allows people to communicate, coordinate and transact freely.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background">Background<a href="#background" class="hash-link" aria-label="Direct link to Background" title="Direct link to Background">​</a></h2><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="what-wechat-provides-to-the-end-user">What WeChat provides to the end-user<a href="#what-wechat-provides-to-the-end-user" class="hash-link" aria-label="Direct link to What WeChat provides to the end-user" title="Direct link to What WeChat provides to the end-user">​</a></h3><p>Let's first look at some of the things that WeChat providers. It is a lot:</p><ul><li><strong>Messaging:</strong> 1:1 and group chat. Text, as well as voice and video. Post gifs. Share location.</li><li><strong>Group chat:</strong> Limited to 500 people; above 100 people people need to verify with a bank account. Also has group video chat and QR code to join a group.</li><li><strong>Timeline/Moments:</strong> Post comments with attachments and have people like/comment on it.</li><li><strong>Location Discovery:</strong> See WeChat users that are nearby.</li><li><strong>Profile:</strong> Nickname and profile picture; can alias people.</li><li><strong>"Broadcast" messages:</strong> Send one message to many contacts, up to 200 people (spam limited).</li><li><strong>Contacts:</strong> Max 5000 contacts (people get around it with multiple accounts and sim cards).</li><li><strong>App reach:</strong> Many diferent web apps, extensions, native apps, etc. Scan QR code to access web app from phone.</li><li><strong>Selective posting:</strong> Decide who can view your posts and who can view your comments on other people's post.</li><li><strong>Transact:</strong> Send money gifts through red envelopes.</li><li><strong>Transact:</strong> Use WeChat pay to transfer money to friends and businesses; linked account with Alipay that is connected to your bank account.</li><li><strong>Services:</strong> Find taxis and get notifications; book flights, train tickets, hotels etc.</li><li><strong>Mini apps:</strong> API for all kinds of apps that allow you to provide services etc.</li><li><strong>Picture in picture:</strong> allowing you to have a video call while using the app.</li></ul><p>And much more. Not going to through it all in detail, and there are probably many things I don't know about WeChat since I'm not a heavy user living in mainland China.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="how-wechat-works---a-toy-model">How WeChat works - a toy model<a href="#how-wechat-works---a-toy-model" class="hash-link" aria-label="Direct link to How WeChat works - a toy model" title="Direct link to How WeChat works - a toy model">​</a></h3><p>This is an overly simplistic model of how WeChat works, but it is sufficient for our purposes. This general design applies to most traditional client-server apps today.</p><p>To sign up for account you need a phone number or equivalent. To get access to some features you need to verify your identity further, for example with official ID and/or bank account.</p><p>When you signup this creates an entry in the WeChat server, from now on treated as a black box. You authenticate with that box, and thats where you get your messages from. If you go online the app asks that box for messages you have received while you were offline. If you login from a different app your contacts and conversations are synced from that box.</p><p>The box gives you an account, it deals with routing to your contacts, it stores messages and attachments and gives access to mini apps that people have uploaded. For transacting money, there is a partnership with a different company that has a different box which talks to your bank account.</p><p>This is done in a such a way that they can support a billion users with the features above, no sweat.</p><p>Whoever controls that box can sees who you are talking with and what the content of those messages are. There is no end to end encryption. If WeChat/Tencent disagrees with you for some reason they can ban you. This means you can't interact with the box under that name anymore.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="what-do-we-want">What do we want?<a href="#what-do-we-want" class="hash-link" aria-label="Direct link to What do we want?" title="Direct link to What do we want?">​</a></h2><p>We want something that is self-sovereign, private, censorship-resistant and open that allows individuals and groups of people to communicate and transact freely. To explore what this means in more detail, without getting lost in the weeds, we provide the following list of properties. A lot of these are tied together, and some fall out of the other requirements. Some of them stand in slight opposition to each other.</p><p><strong>Self-sovereignity identity</strong>. Exercises authority within your own sphere. If you aren't harming anyone, you should be able to have an account and communicate with other people.</p><p><strong>Pseudonymity, and ideally total anonymity</strong>. Not having your identity tied to your real name (e.g. through phone number, bank account, ID, etc). This allows people to act more freely without being overly worried about censorship and coercion in the real world. While total anonymity is even more desirable - especially to break multiple hops to a true-name action - real-world constraints sometimes makes this more challenging.</p><p><strong>Private and secure communication</strong>. Your communication and who you transact with should be for your eyes only. This includes transactions (transfer of value) as a form of communication.</p><p><strong>Censorship-resistance</strong>. Not being able to easily censor individuals on the platform. Both at an individual, group and collective level. Not having single points of failure that allow service to be disrupted.</p><p><strong>Decentralization</strong>. Partly falls out of censorship-resistance and other properties. If infrastructure isn't decentralized it means there's a single point of failure that can be disrupted. This is more of a tool than a goal on its own, but it is an important tool.</p><p><strong>Built for mass adoption</strong>. Includes scalabiltiy, UX (latency, reliability, bandwidth consumption, UI etc), and allowing for people to stick around. One way of doing this is to allow users to discover people they want to talk to.</p><p><strong>Scalability</strong>. Infrastructure needs to support a lot of users to be a viabile alternative. Like, a billion of them (eventually).</p><p><strong>Fundamentals in place to support great user experience</strong>. To be a viable alternative, aside from good UI and distribution, fundamentals such as latency, bandwidth usage, consistency etc must support great UX to be a viable alternative.</p><p><strong>Works for resource restricted devices, including smartphones</strong>. Most people will use a smartphone to use this. This means it has to work well on them and similar devices, without becoming a second-class citizen where we ignore properties such as censorship-resistance and privacy. Some concession to reality will be necessary due to additional constraints, which leads us to...</p><p><strong>Adaptive nodes</strong>. Nodes will have different capabilities, and perhaps at different times. To maintain a lot of the properties described here it is desirable if as many participants as possible are first-class citizens. If a phone is switching from a limited data plan to a WiFi network or from battery to AC power it can do more useful work, and so on. Likewise for a laptop with a lot of free disk space and spare compute power, etc.</p><p><strong>Sustainable</strong>. If there's no centralized, top down ad-driven model, this means all the infrastructure has to be sustainable somehow. Since these are individual entitites, this means it has to be paid for. While altruistic modes and similar can be used, this likely requires some form of incentivization scheme for useful services provided in the network. Related: free rider problem.</p><p><strong>Spam resistant</strong>. Relates to sustainability, scalability and built for mass adoption. Made more difficult by pseudonymous identity due to whitewashing attacks.</p><p><strong>Trust-minimized</strong>. To know that properties are provided for and aren't compromised, various ways of minimizing trust requirements are useful. This also related to mass adoption and social cohesion. Examples include: open and audited protocols, open source, reproducible builds, etc. This also relates to how mini apps are provided for, since we may not know their source but want to be able to use them anyway.</p><p><strong>Open source</strong>. Related to above, where we must be able to inspect the software to know that it functions as advertised and hasn't been compromised, e.g. by uploading private data to a third party.</p><p>Some of these are graded and a bit subtle, i.e.:</p><ul><li>Censorship resistance would ideally be able to absorb Internet shutdowns. This would require an extensive MANET/meshnet infrastructure, which while desirable, requires a lot of challenges to be overcome to be feasible.</li><li>Privacy would ideally make all actions (optionally) totally anoymous, though this may incur undue costs on bandwidth and latency, which impacts user experience.</li><li>Decentralization, certain topologies, such as DHTs, are efficient and quite decentralized but still have some centralized aspects, which makes it attackable in various ways. Ditto for blockchains compared with bearer instruments which requires some coordinating infrastructure, compared with naturally occuring assets such as precious metals.</li><li>"Discover people" and striving for "total anonymity" might initially seem incompatible. The idea is to provide for sane defaults, and then allow people to decide how much information they want to disclose. This is the essence of privacy.</li><li>Users often want <em>some</em> form of moderation to get a good user experience, which can be seen as a form of censorship. The idea to raise the bar on the basics, the fundamental infrastructure. If individuals or specific communities want certain moderation mechanisms, that is still a compatible requirement.</li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="counterpoint-1">Counterpoint 1<a href="#counterpoint-1" class="hash-link" aria-label="Direct link to Counterpoint 1" title="Direct link to Counterpoint 1">​</a></h3><p>We could refute the above by saying that the design goals are undesirable. We want a system where people can censor others, and where everyone is tied to their real identity. Or we could say something like, freedom of speech is a general concept, and it doesn't apply to Internet companies, even if they provide a vital service. You can survive without it and you should've read the terms of service. This roughly charactericizes the mainstream view.</p><p>Additional factor here is the idea that a group of people know more about what's good for you then you do, so they are protecting you.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="counterpoint-2">Counterpoint 2<a href="#counterpoint-2" class="hash-link" aria-label="Direct link to Counterpoint 2" title="Direct link to Counterpoint 2">​</a></h3><p>We could agree with all these design goals, but think they are too extreme in terms of their requirements. For example, we could operate as a non profit, take donations and volunteers, and then host the whole infrastructure ourselves. We could say we are in a friendly legislation, so we won't be a single point of failure. Since we are working on this and maybe even our designs are open, you can trust us and we'll provide service and infrastructure that gives you what you want without having to pay for it or solve all these complex decentralized computation and so on problems. If you don't trust us for some reason, you shouldn't use us regardless. Also, this is better than status quo. And we are more likely to survive by doing this, either by taking shortcuts or by being less ambituous in terms of scope.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="principal-components">Principal components<a href="#principal-components" class="hash-link" aria-label="Direct link to Principal components" title="Direct link to Principal components">​</a></h2><p>There are many ways to skin a cat, but this is one way of breaking down the problem. We have a general direction with the properties listed above, together with some understanding of how WeChat works for the everday user. Now the question is, what infrastructure do we need to support this? How do we achieve the above properties, or at least get closer to them? We want to figure out the necessary building blocks, and one of doing this is to map out likely necessary components.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background-ethereum-and-web3-stack">Background: Ethereum and Web3 stack<a href="#background-ethereum-and-web3-stack" class="hash-link" aria-label="Direct link to Background: Ethereum and Web3 stack" title="Direct link to Background: Ethereum and Web3 stack">​</a></h3><p>It is worth noting that a lot of the required infrastructure has been developed, at least as concepts, in the original Ethereum / Web3 vision. In it there is Ethereum for consensus/compute/transact, storage through Swarm, and communication through Whisper. That said, the main focus has been on the Ethereum blockchain itself, and a lot of things have happened in the last 5y+ with respect to technology around privacy and scalabilty. It is worth revisiting things from a fresh point of view, with the WeChat alternative in mind as a clear use case.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="account---self-sovereign-identity-and-the-perils-of-phone-numbers">Account - self-sovereign identity and the perils of phone numbers<a href="#account---self-sovereign-identity-and-the-perils-of-phone-numbers" class="hash-link" aria-label="Direct link to Account - self-sovereign identity and the perils of phone numbers" title="Direct link to Account - self-sovereign identity and the perils of phone numbers">​</a></h3><p>Starting from the most basic: what is an account and how do you get one? With most internet services today, WeChat and almost all popular messaging apps included, you need to signup with some centralized authority. Usually you also have to verify this with some data that ties this account to you as an individual. E.g. by requiring a phone number, which in most jurisdictions <sup id="fnref-1-367620"><a href="#fn-1-367620" class="footnote-ref">1</a></sup> means giving out your real ID. This also means you can be banned from using the service by a somewhat arbitrary process, with no due process.</p><p>Now, we could argue these app providers can do what they want. And they are right, in a very narrow sense. As apps like WeChat (and Google) become general-purpose platforms, they become more and more ingrained in our everyday lives. They start to provide utilities that we absolutely require to work to go about our day, such as paying for food or transportation. This means we need higher standard than this.</p><p>Justifications for requiring phone numbers are usually centered around three claims:</p><ol><li>Avoiding spam</li><li>Tying your account to your real name, for various reasons</li><li>Using as a commonly shared identifier as a social network discovery mechanism</li></ol><p>Of course, many services require more than phone numbers. E.g. email, other forms of personal data such as voice recording, linking a bank account, and so on.</p><p>In contrast, a self-sovereign system would allow you to "create an account" completely on your own. This can easily be done with public key cryptograpy, and it also paves the way for end-to-end encryption to make your messages private.</p><p>The main issue with this that you need to get more creative about avoiding spam (e.g. through white washing attacks), and ideally there is some other form of social discovery mechanism.</p><p>Just having a public key as an account isn't enough though. If it goes through a central server, then nothing is stopping that server from arbitrarly blocking requests related to that public key. Of course, this also depends on how transparent such requests are. Fundamentally, lest we rely completely on goodwill, there needs to be multiple actors by which you can use the service. This naturally points to decentralization as a requirement. See counterpoint.</p><p>Even so, if the system is closed source we don't know what it is doing. Perhaps the app communicating is also uploading data to another place, or somehow making it possible to see who is who and act accordingly.</p><p>You might notice that just one simple property, self-sovereign identity, leads to a slew of other requirements and properties. You might also notice that WeChat is far from alone in this, even if their identity requirements might be a bit stringent than, say, Telegram. Their control aspects are also a bit more extreme, at least for someone with western sensibilities <sup id="fnref-2-367620"><a href="#fn-2-367620" class="footnote-ref">2</a></sup>.</p><p>Most user facing applications have similar issues, Google Apps/FB/Twitter etc. For popular tools that have this built in, we can look at git - which is truly decentralized and have keypair at the bottom. It is for a very specific technical domain, and even then people rely on Github. Key management is fairly difficult even for technical people, and for normal people even more so. Banks are generally far behind on this tech, relying on arcane procedures and special purpose hardware for 2FA. That's another big issue.</p><p>Let's shift gears a bit and talk about some other functional requirements.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="routing---packets-from-a-to-b">Routing - packets from A to B<a href="#routing---packets-from-a-to-b" class="hash-link" aria-label="Direct link to Routing - packets from A to B" title="Direct link to Routing - packets from A to B">​</a></h3><p>In order to get a lot of the features WeChat provides, we need the ability to do three things: communicate, store data, and transact with people. We need a bit more than that, but let's focus on this for now.</p><p>To communicate with people, in the base case, we need to go from one phone to another phone that is separated by a large distance. This requires some form of routing. The most natural platform to build this on is the existing Internet, though not the only one. Most phones are resource restricted, and are only "on" for brief periods of time. This is needed to preserve battery and bandwidth. Additionally, Internet uses IPs as endpoints, which change as a phones move through space. NAT punching etc isn't always perfect either. This means we need a way to get a message from one public key to another, and through some intermediate nodes. We can think of these nodes as a form of service network. Similar to how a power grid works, or phone lines, or collection of ISPs.</p><p>One important property here is to ensure we don't end up in a situation like the centralized capture scenario above, something we've seen with centralized ISPs <sup id="fnref-3-367620"><a href="#fn-3-367620" class="footnote-ref">3</a></sup> <sup id="fnref-4-367620"><a href="#fn-4-367620" class="footnote-ref">4</a></sup> where they can choose which traffic is good and which is bad. We want to allow the use of different service nodes, just like if a restaurant gives you food poisioning you can go to the one next door and then the first one goes out of business after a while. And the circle of life continues.</p><p>We shouldn't be naive though, and think that this is something nodes are likely to do for free. They need to be adequately compensated for their services, in some of incentivization scheme. That can either be monetary, or as in the case of Bittorrent, more of a barter situation where you use game theory to coordinate with strangers <sup id="fnref-5-367620"><a href="#fn-5-367620" class="footnote-ref">5</a></sup>, and some form of reputation attached to it (for private trackers).</p><p>There are many ways of doing routing, and we won't go into too much technical detail here. Suffice to say is that you likely want both a structured and unstructured alternative, and that these comes with several trade-offs when it comes to efficiency, metadata protection, ability to incentivize, compatibility with existing topologies, and suitability for mobilephones (mostly offline, bandwidth restricted, not directly connectable). Expect more on this in a future article.</p><p>Some of these considerations naturally leads us into the storage and transaction components.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="storage---available-and-persistant-for-later">Storage - available and persistant for later<a href="#storage---available-and-persistant-for-later" class="hash-link" aria-label="Direct link to Storage - available and persistant for later" title="Direct link to Storage - available and persistant for later">​</a></h3><p>If mobile phones are mostly offline, we need some way to store these messages so they can be retrieved when online again. The same goes for various kinds attachments as well, and for when people are switching devices. A user might control their timeline, but in the WeChat case that timeline is stored on Tencent's servers, and queried from there as well. This naturally needs to happen by some other service nodes. In the WeChat case, and for most IMs, the way these servers are paid for is through some indirect ad mechanism. The entity controlling these ads and so on is the same one as the one operating the servers for storage. A more direct model with different entities would see these services being compensated for their work.</p><p>We also need storage for attachments, mini-apps, as well as a way of understanding the current state of consensus when it comes to the compute/transact module. In the WeChat case, this state is completely handled by the bank institution or one of their partners, such as Alibaba. When it comes to bearer instruments like cash, no state needs to be kept as that's a direct exchange in the physical world. This isn't directly compatible with transfering value over a distance.</p><p>All of this state requires availability and persistance. It should be done in a trust minimized fashion and decentralized, which requires some form of incentivization for keeping data around. If it isn't, you are relying on social cohesion which breaks down at very large scales.</p><p>Since data will be spread out across multiple nodes, you need a way to sync data and transfer it in the network. As well as being able to add and query data from it. All of this requires a routing component.</p><p>To make it more censorship resistant it might be better to keep it as a general-purpose store, i.e. individuals don't need to know what they storing. Otherwise, you naturally end up in a situation where individual nodes can be pressured to not store certain content.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="messaging---from-me-to-you-to-all-of-us-not-them">Messaging - from me to you to all of us (not them)<a href="#messaging---from-me-to-you-to-all-of-us-not-them" class="hash-link" aria-label="Direct link to Messaging - from me to you to all of us (not them)" title="Direct link to Messaging - from me to you to all of us (not them)">​</a></h3><p>This builds on top of routing, but it has a slightly different focus. The goal is to allow for individuals and groups to communicate in a private, secure and censorship-resistant manner.</p><p>It also needs to provide a decent interface to the end user, in terms of dealing seamlessly with offline messages, providing reliable and timely messaging.</p><p>In order to get closer to the ideal of total anonymity, it is useful to be able to hide metadata of who is talking to whom. This applies to both normal communication as well as for transactions. Ideally, no one but the parties involved can see who is taking part in a conversation. This can be achieved through various techniques such as mixnets, anonymous credentials, private information retrieval, and so on. Many of these techniques have a fundamental trade-off with latency and bandwidth, something that is a big concern for mobilephones. Being able to do some form of tuning, in an adaptive node manner, depending on your threat model and current capabilities is useful here.</p><p>The baseline here is pseudonymity, and having tools to allow individuals to "cut off" ties to their real world identity and transactions. People act different in different circles in the real world, and this should be mimicked online as well. Your company, family or government shouldn't be able to know what exactly you use your paycheck for, and who you are talking to.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="compute---transact-contract-and-settle">Compute - transact, contract and settle<a href="#compute---transact-contract-and-settle" class="hash-link" aria-label="Direct link to Compute - transact, contract and settle" title="Direct link to Compute - transact, contract and settle">​</a></h3><p>The most immediate need here is transaction from A to B. Direct exchange. There is also a more indirect need for private lawmaking and contracting.</p><p>We talked about routing and storage and how they likely need to be incentivized to work properly. How are they going to be compensated? While this could in theory work via existing banking system and so on, this would be rather heavy. It'd also very likely require tying your identifier to your legal name, something that goes against what we want to achieve. What we want is something that acts more as right-to-access, similar to the way cash functions in a society <sup id="fnref-6-367620"><a href="#fn-6-367620" class="footnote-ref">6</a></sup>. I pay for a fruit with something that is valuable to you and then I'm on my way.</p><p>While there might be other candidates, such as pre-paid debit cards and so on, this transaction mode pretty much requires a cryptocurrency component. The alternative is to do it on a reputation basis, which might work for small communities, due to social cohesion, but quickly detoriates for large ones <sup id="fnref-7-367620"><a href="#fn-7-367620" class="footnote-ref">7</a></sup>. Ad hoc models like private Bittorrent trackers are centralized and easy to censor.</p><p>Now, none of the existing cryptocurrency models are ideal. They also all suffer from lack of widespread use, and it is difficult to get onboarded to them in the first place. Transactions in Bitcoin are slow. Ethereum is faster and has more capabilities, but it still suffers from linking payments over time, which makes the privacy part of this more difficult. Zcash, Monero and similar are interesting, but also require more use. For Zcash, shielded transactions appear to only account for less than 2% of all transactions in 2019 <sup id="fnref-8-367620"><a href="#fn-8-367620" class="footnote-ref">8</a></sup> <sup id="fnref-9-367620"><a href="#fn-9-367620" class="footnote-ref">9</a></sup>.</p><p>Another dimension is what sets general purpose cryptocurrencies like Ethereum apart. Aside from just paying from A to B, you can encode rules about when something should be paid out and not. This is very useful for doing a form of private lawmaking, contracting, for setting up service agreements with these nodes. If there's no trivial recourse as in the meatspace world, where you know someone's name and you can sue them, you need a different kind of model.</p><p>What makes something like Zcash interesting is that it works more like digital cash. Instead of leaving a public trail for everyone, where someone can see where you got the initial money from and then trace you across various usage, for Zcash every hop is privacy preserving.</p><p>To fulfill the general goals of being censorship resistance and secure, it is also vital that the system being used stays online and can't be easily disrupted. That points to disintermediation, as opposed to using gateways and exchanges. This is a case where something like cash, or gold, is more direct, since no one can censor this transaction without being physically present where this direct exchange is taking place. However, like before, this doesn't work over distance.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="secure-chat---just-our-business">Secure chat - just our business<a href="#secure-chat---just-our-business" class="hash-link" aria-label="Direct link to Secure chat - just our business" title="Direct link to Secure chat - just our business">​</a></h3><p>Similar to the messaging module above. The distinction here is that we assume the network part has already taken place. Here we are interested in keeping the contents of messages private, so that means confidentiality/end-to-end encryption, integrity, authentication, as well as forward secrecy and plausible deniability. This means that even if there's some actor that gets some private key material, or confiscated your phone, there is some level of...ephemerality to your conversations. Another issue here in terms of scalable private group chat.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="extensible-mini-apps">Extensible mini apps<a href="#extensible-mini-apps" class="hash-link" aria-label="Direct link to Extensible mini apps" title="Direct link to Extensible mini apps">​</a></h3><p>This relates to the compute and storage module above. Essentially we want to provide mini apps as in WeChat, but to do so in a way that is compatible with what we want to achieve more generally. This allows individuals and small businesses to create small tools for various purposes, and coordinate with strangers. E.g. booking a cab or getting an insurance, and so on.</p><p>This has a higher dependency on the contracting/general computation aspect. I.e. often it isn't only a transaction, but you might want to encode some specific rules here that strangers can abide by without having too high trust requirements. As a simple example: escrows.</p><p>This also needs an open API that anyone can use. It should be properly secured, so using one doesn't compromise the rest of the system it is operating in. To be censorship resistant it requires the routing and storage component to work properly.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="where-are-we-now">Where are we now?<a href="#where-are-we-now" class="hash-link" aria-label="Direct link to Where are we now?" title="Direct link to Where are we now?">​</a></h2><p>Let's look back at some of desirable properties we set out in the beginning and see how close we are to building out the necessary components. Is it realistic at all or just a pipe dream? We'll see that there are many building blocks in place, and there's reason for hope.</p><p><strong>Self-sovereignity identity</strong>. Public key crypto and web of trust like constructs makes this possible.</p><p><strong>Pseudonymity, and ideally total anonymity</strong>. Pseudonymity can largely be achieved with public key crypto and open systems that allow for permissionless participation. For transactions, pseudonymity exists in most cryptocurrencies. The challenge is linkage across time, especially when interfacing with other "legacy" system. There are stronger constructs that are actively being worked on and are promising here, such as mixnets (Nym), mixers (Wasabi Wallet, Tornado.Cash) and zero knowledge proofs (Zcash, Ethereum, Starkware). This area of applied research has exploded over the last few years.</p><p><strong>Private and secure communication</strong>. Signal has pioneered a lot of this, following OTR. Double Ratchet, X3DH. E2EE is minimum these days, and properties like PFS and PD are getting better. For metadata protection, you have Tor, with its faults, and more active research on mixnets and private information retrieval, etc.</p><p><strong>Censorship-resistance</strong>. This covers a lot of ground across the spectrum. You have technologies like Bittorrent, Bitcoin/Ethereum, Tor obfuscated transports, E2EE by default, partial mesh networks in production, abilit to move/replicate host machines more quickly have all made this more of a reality than it used to be. this easier. Of course, techniques such as deep packet inspection and internet shutdowns have increased.</p><p><strong>Decentralization</strong>. Cryptocurrencies, projects like libp2p and IPFS. Need to be mindful here of many projects that claim decentralization but are still vulnerable to single points of failures, such as relying on gateways.</p><p><strong>Built for mass adoption</strong>. This one is more subjective. There's definitely a lot of work to be done here, both when it comes to fundamental performance, key management and things like social discoverability. Directionally these things are improving and becoming easier for the average person but there is a lot ot be done here.</p><p><strong>Scalability</strong>. With projects like Ethereum 2.0 and IPFS more and more resources are a being put into this, both at the consensus/compute layer as well as networking (gossip, scalable Kademlia) layer. Also various layer 2 solutions for transactions.</p><p><strong>Fundamentals in place to support great user experience</strong>. Similar to built for mass adoption. As scalability becomes more important, more applied research is being done in the p2p area to improve things like latency, bandwidth.</p><p><strong>Works for resource restricted devices, including smartphones</strong>. Work in progress and not enough focus here, generally an after thought. Also have stateless clients etc.</p><p><strong>Adaptive nodes</strong>. See above. With subprotocols and capabilities in Ethereum and libp2p, this is getting easier.</p><p><strong>Sustainable</strong>. Token economics is a thing. While a lot of it won't stay around, there are many more projects working on making themselves dispensable. Being open source, having an engaged community and enabling users run their own infrastructure. Users as stakeholders.</p><p><strong>Spam resistant</strong>. Tricky problem if you want to be pseudonymous, but some signs of hope with incentivization mechanisms, zero knowledge based signaling, etc. Together with various forms of rate limiting and better controlling of topology and network amplification. And just generally being battle-tested by real world attacks, such as historical Ethereum DDoS attacks.</p><p><strong>Trust minimized</strong>. Bitcoin. Zero knowledge provable computation. Open source. Reproducible builds. Signed binaries. Incentive compatible structures. Independent audits. Still a lot of work, but getting better.</p><p><strong>Open source</strong>. Big and only getting bigger. Including mainstream companies.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="whats-next">What's next?<a href="#whats-next" class="hash-link" aria-label="Direct link to What's next?" title="Direct link to What's next?">​</a></h2><p>We've look at what WeChat provides and what we'd like an alternative to look like. We've also seen a few principal modules that are necessary to achieve those goals. To achieve all of this is a daunting task, and one might call it overly ambitiuous. We've also seen how far we've come with some of the goals, and how a lot of the pieces are there, in one form or another. Then it is a question of putting them all together in the right mix.</p><p>The good news is that a lot of people are working all these building blocks and thinking about these problems. Compared to a few years ago we've come quite far when it comes to p2p infrastructure, privacy, security, scalability, and general developer mass and mindshare. If you want to join us in building some of these building blocks, and assembling them, check out our forum.</p><p>PS. We are <a href="https://status.im/our_team/open_positions.html" target="_blank" rel="noopener noreferrer">hiring protocol engineers</a>. DS</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="acknowledgements">Acknowledgements<a href="#acknowledgements" class="hash-link" aria-label="Direct link to Acknowledgements" title="Direct link to Acknowledgements">​</a></h2><p>Corey, Dean, Jacek.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2><div class="footnotes"><hr><ol><li id="fn-1-367620">Mandatory SIM card registration laws: <a href="https://privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws" target="_blank" rel="noopener noreferrer">https://privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws</a><a href="#fnref-1-367620" class="footnote-backref">↩</a></li><li id="fn-2-367620">On WeChat keyword censorship: <a href="https://citizenlab.ca/2016/11/wechat-china-censorship-one-app-two-systems/" target="_blank" rel="noopener noreferrer">https://citizenlab.ca/2016/11/wechat-china-censorship-one-app-two-systems/</a><a href="#fnref-2-367620" class="footnote-backref">↩</a></li><li id="fn-3-367620">Net Neutrality: <a href="https://www.eff.org/issues/net-neutrality" target="_blank" rel="noopener noreferrer">https://www.eff.org/issues/net-neutrality</a><a href="#fnref-3-367620" class="footnote-backref">↩</a></li><li id="fn-4-367620">ISP centralization: <a href="https://ilsr.org/repealing-net-neutrality-puts-177-million-americans-at-risk/" target="_blank" rel="noopener noreferrer">https://ilsr.org/repealing-net-neutrality-puts-177-million-americans-at-risk/</a><a href="#fnref-4-367620" class="footnote-backref">↩</a></li><li id="fn-5-367620">Incentives Build Robustness in BitTorrent bittorrent.org/bittorrentecon.pdf<a href="#fnref-5-367620" class="footnote-backref">↩</a></li><li id="fn-6-367620">The Case for Electronic Cash: <a href="https://coincenter.org/files/2019-02/the-case-for-electronic-cash-coin-center.pdf" target="_blank" rel="noopener noreferrer">https://coincenter.org/files/2019-02/the-case-for-electronic-cash-coin-center.pdf</a><a href="#fnref-6-367620" class="footnote-backref">↩</a></li><li id="fn-7-367620">Money, blockchains, and social scalability: <a href="http://unenumerated.blogspot.com/2017/02/money-blockchains-and-social-scalability.html" target="_blank" rel="noopener noreferrer">http://unenumerated.blogspot.com/2017/02/money-blockchains-and-social-scalability.html</a><a href="#fnref-7-367620" class="footnote-backref">↩</a></li><li id="fn-8-367620">Zcash private transactions (partial paywall): <a href="https://www.theblockcrypto.com/genesis/48413/an-analysis-of-zcashs-private-transactions" target="_blank" rel="noopener noreferrer">https://www.theblockcrypto.com/genesis/48413/an-analysis-of-zcashs-private-transactions</a><a href="#fnref-8-367620" class="footnote-backref">↩</a></li><li id="fn-9-367620">Shielded transactions usage (stats page 404s): <a href="https://z.cash/support/faq/" target="_blank" rel="noopener noreferrer">https://z.cash/support/faq/</a><a href="#fnref-9-367620" class="footnote-backref">↩</a></li></ol></div>]]></content:encoded>
</item>
<item>
<title><![CDATA[From Kademlia to Discv5]]></title>
<link>https://vac.dev/rlog/kademlia-to-discv5</link>
<guid>https://vac.dev/rlog/kademlia-to-discv5</guid>
<pubDate>Thu, 09 Apr 2020 16:00:00 GMT</pubDate>
<description><![CDATA[A quick history of discovery in peer-to-peer networks, along with a look into discv4 and discv5, detailing what they are, how they work and where they differ.]]></description>
<content:encoded><![CDATA[<p>A quick history of discovery in peer-to-peer networks, along with a look into discv4 and discv5, detailing what they are, how they work and where they differ.</p><p>If you've been working on Ethereum or adjacent technologies you've probably heard of <a href="https://github.com/ethereum/devp2p/blob/master/discv4.md" target="_blank" rel="noopener noreferrer">discv4</a> or <a href="https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md" target="_blank" rel="noopener noreferrer">discv5</a>. But what are they actually? How do they work and what makes them different? To answer these questions, we need to start at the beginning, so this post will assume that there is little knowledge on the subject so the post should be accessible for anyone.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-beginning">The Beginning<a href="#the-beginning" class="hash-link" aria-label="Direct link to The Beginning" title="Direct link to The Beginning">​</a></h2><p>Let's start right at the beginning: the problem of discovery and organization of nodes in peer-to-peer networks.</p><p>Early P2P file sharing technologies, such as Napster, would share information about who holds what file using a single server. A node would connect to the central server and give it a list of the files it owns. Another node would then connect to that central server, find a node that has the file it is looking for and contact that node. This however was a flawed system -- it was vulnerable to attacks and left a single party open to lawsuits.</p><p>It became clear that another solution was needed, and after years of research and experimentation, we were given the distributed hash table or DHT.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="distributed-hash-tables">Distributed Hash Tables<a href="#distributed-hash-tables" class="hash-link" aria-label="Direct link to Distributed Hash Tables" title="Direct link to Distributed Hash Tables">​</a></h2><p>In 2001 4 new protocols for such DHTs were conceived, <a href="https://pdos.csail.mit.edu/~strib/docs/tapestry/tapestry_jsac03.pdf" target="_blank" rel="noopener noreferrer">Tapestry</a>, <a href="https://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf" target="_blank" rel="noopener noreferrer">Chord</a>, <a href="https://people.eecs.berkeley.edu/~sylvia/papers/cans.pdf" target="_blank" rel="noopener noreferrer">CAN</a> and <a href="http://rowstron.azurewebsites.net/PAST/pastry.pdf" target="_blank" rel="noopener noreferrer">Pastry</a>, all of which made various trade-offs and changes in their core functionality, giving them unique characteristics.</p><p>But as said, they're all DHTs. So what is a DHT?</p><p>A distributed hash table (DHT) is essentially a distributed key-value list. Nodes participating in the DHT can easily retrieve the value for a key.</p><p>If we have a network with 9 key-value pairs and 3 nodes, ideally each node would store 3 (optimally 6 for redundancy) of those key-value pairs, meaning that if a key-value pair were to be updated, only part of the network would responsible for ensuring that it is. The idea is that any node in the network would know where to find the specific key-value pair it is looking for based on how things are distributed amongst the nodes.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="kademlia">Kademlia<a href="#kademlia" class="hash-link" aria-label="Direct link to Kademlia" title="Direct link to Kademlia">​</a></h2><p>So now that we know what DHTs are, let's get to Kademlia, the predecessor of discv4. Kademlia was created by Petar Maymounkov and David Mazières in 2002. I will naively say that this is probably one of the most popular and most used DHT protocols. It's quite simple in how it works, so let's look at it.</p><p>In Kademlia, nodes and values are arranged by distance (in a very mathematical definition). This distance is not a geographical one, but rather based on identifiers. It is calculated how far 2 identifiers are from eachother using some distance function.</p><p>Kademlia uses an <code>XOR</code> as its distance function. An <code>XOR</code> is a function that outputs <code>true</code> only when inputs differ. Here is an example with some binary identifiers:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">XOR 10011001</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 00110010</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> 10101011</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>The top in decimal numbers means that the distance between <code>153</code> and <code>50</code> is <code>171</code>.</p><p>There are several reasons why <code>XOR</code> was taken:</p><ol><li>The distance from one ID to itself will be <code>0</code>.</li><li>Distance is symmetric, A to B is the same as B to A.</li><li>Follows triangle inequality, if <code>A</code>, <code>B</code> and <code>C</code> are points on a triangle then the distance <code>A</code> to <code>B</code> is closer or equal to that of <code>A</code> to <code>C</code> plus the one from <code>B</code> to <code>C</code>.</li></ol><p>In summary, this distance function allows a node to decide what is "close" to it and make decisions based on that "closeness".</p><p>Kademlia nodes store a routing table. This table contains multiple lists. Each subsequent list contains nodes which are a little further distanced than the ones included in the previous list. Nodes maintain detailed knowledge about nodes closest to them, and the further away a node is, the less knowledge the node maintains about it.</p><p>So let's say I want to find a specific node. What I would do is go to any node which I already know and ask them for all their neighbours closest to my target. I repeat this process for the returned neighbours until I find my target.</p><p>The same thing happens for values. Values have a certain distance from nodes and their IDs are structured the same way so we can calculate this distance. If I want to find a value, I simply look for the neighbours closest to that value's key until I find the one storing said value.</p><p>For Kademlia nodes to support these functions, there are several messages with which the protocol communicates.</p><ul><li><code>PING</code> - Used to check whether a node is still running.</li><li><code>STORE</code> - Stores a value with a given key on a node.</li><li><code>FINDNODE</code> - Returns the closest nodes requested to a given ID.</li><li><code>FINDVALUE</code> - The same as <code>FINDNODE</code>, except if a node stores the specific value it will return it directly.</li></ul><p><em>This is a <strong>very</strong> simplified explanation of Kademlia and skips various important details. For the full description, make sure to check out the <a href="https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf" target="_blank" rel="noopener noreferrer">paper</a> or a more in-depth <a href="http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html" target="_blank" rel="noopener noreferrer">design specification</a></em></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="discv4">Discv4<a href="#discv4" class="hash-link" aria-label="Direct link to Discv4" title="Direct link to Discv4">​</a></h2><p>Now after that history lesson, we finally get to discv4 (which stands for discovery v4), Ethereum's current node discovery protocol. The protocol itself is essentially based off of Kademlia, however it does away with certain aspects of it. For example, it does away with any usage of the value part of the DHT.</p><p>Kademlia is mainly used for the organisation of the network, so we only use the routing table to locate other nodes. Due to the fact that discv4 doesn't use the value portion of the DHT at all, we can throw away the <code>FINDVALUE</code> and <code>STORE</code> commands described by Kademlia.</p><p>The lookup method previously described by Kademlia describes how a node gets its peers. A node contacts some node and asks it for the nodes closest to itself. It does so until it can no longer find any new nodes.</p><p>Additionally, discv4 adds mutual endpoint verification. This is meant to ensure that a peer calling <code>FINDNODE</code> also participates in the discovery protocol.</p><p>Finally, all discv4 nodes are expected to maintain up-to-date ENR records. These contain information about a node. They can be requested from any node using a discv4-specific packet called <code>ENRRequest</code>.</p><p><em>If you want some more details on ENRs, check out one of my posts <a href="https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/" target="_blank" rel="noopener noreferrer">"Network Addresses in Ethereum"</a></em></p><p>Discv4 comes with its own range of problems however. Let's look at a few of them.</p><p>Firstly, the way discv4 works right now, there is no way to differentiate between node sub-protocols. This means for example that an Ethereum node could add an Ethereum Classic Node, Swarm or Whisper node to its DHT without realizing that it is invalid until more communication has happened. This inability to differentiate sub-protocols makes it harder to find specific nodes, such as Ethereum nodes with light-client support.</p><p>Next, in order to prevent replay attacks, discv4 uses timestamps. This however can lead to various issues when a host's clock is wrong. For more details, see the <a href="https://github.com/ethereum/devp2p/blob/master/discv4.md#known-issues-in-the-current-version" target="_blank" rel="noopener noreferrer">"Known Issues"</a> section of the discv4 specification.</p><p>Finally, we have an issue with the way mutual endpoint verification works. Messages can get dropped and there is no way to tell if both peers have verified eachother. This means that we could consider our peer verified while it does not consider us so making them drop the <code>FINDNODE</code> packet.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="discv5">Discv5<a href="#discv5" class="hash-link" aria-label="Direct link to Discv5" title="Direct link to Discv5">​</a></h2><p>Finally, let's look at discv5. The next iteration of discv4 and the discovery protocol which will be used by Eth 2.0. It aims at fixing various issues present in discv4.</p><p>The first change is the way <code>FINDNODE</code> works. In traditional Kademlia as well as in discv5, we pass an identifier. However, in discv5 we instead pass the logarithmic distance, meaning that a <code>FINDNODE</code> request gets a response containing all nodes at the specified logarithmic distance from the called node.</p><p>Logarithmic distance means we first calculate the distance and then run it through our log base 2 function. See:</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">log2(A xor B)</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>And the second, more important change, is that discv5 aims at solving one of the biggest issues of discv4: the differentiation of sub-protocols. It does this by adding topic tables. Topic tables are <a href="https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)" target="_blank" rel="noopener noreferrer">first in first out</a> lists that contain nodes which have advertised that they provide a specific service. Nodes get themselves added to this list by registering <code>ads</code> on their peers.</p><p>As of writing, there is still an issue with this proposal. There is currently no efficient way for a node to place <code>ads</code> on multiple peers, since it would require separate requests for every peer which is inefficient in a large-scale network.</p><p>Additionally, it is unclear how many peers a node should place these <code>ads</code> on and exactly which peers to place them on. For more details, check out the issue <a href="https://github.com/ethereum/devp2p/issues/136" target="_blank" rel="noopener noreferrer">devp2p#136</a>.</p><p>There are a bunch more smaller changes to the protocol, but they are less important hence they were ommitted from this summary.</p><p>Nevertheless, discv5 still does not resolve a couple issues present in discv4, such as unreliable endpoint verification. As of writing this post, there is currently no new method in discv5 to improve the endpoint verification process.</p><p>As you can see discv5 is still a work in progress and has a few large challenges to overcome. However if it does, it will most likely be a large improvement to a more naive Kademlia implementations.</p><hr><p>Hopefully this article helped explain what these discovery protocols are and how they work. If you're interested in their full specifications you can find them on <a href="https://github.com/ethereum/devp2p" target="_blank" rel="noopener noreferrer">github</a>.</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Waku Update]]></title>
<link>https://vac.dev/rlog/waku-update</link>
<guid>https://vac.dev/rlog/waku-update</guid>
<pubDate>Fri, 14 Feb 2020 12:00:00 GMT</pubDate>
<description><![CDATA[A research log. What's the current state of Waku? How many users does it support? What are the bottlenecks? What's next?]]></description>
<content:encoded><![CDATA[<p>A research log. What's the current state of Waku? How many users does it support? What are the bottlenecks? What's next?</p><p>Waku is our fork of Whisper where we address the shortcomings of Whisper in an iterative manner. We've seen a in <a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">previous post</a> that Whisper doesn't scale, and why. In this post we'll talk about what the current state of Waku is, how many users it can support, and future plans.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="current-state">Current state<a href="#current-state" class="hash-link" aria-label="Direct link to Current state" title="Direct link to Current state">​</a></h2><p><strong>Specs:</strong></p><p>We released <a href="https://rfc.vac.dev/spec/6" target="_blank" rel="noopener noreferrer">Waku spec v0.3</a> this week! You can see the full changelog <a href="https://rfc.vac.dev/spec/6/#changelog" target="_blank" rel="noopener noreferrer">here</a>.</p><p>The main change from 0.2 is making the handshake more flexible. This enables us to communicate topic interest immediately without ambiguity. We also did the following:</p><ul><li>added recommendation for DNS based discovery</li><li>added an upgradability and compatibility policy</li><li>cut the spec up into several components</li></ul><p>We cut the spec up in several components to make Vac as modular as possible. The components right now are:</p><ul><li>Waku (main spec), currently in <a href="https://rfc.vac.dev/spec/6" target="_blank" rel="noopener noreferrer">version 0.3.0</a></li><li>Waku envelope data field, currently in <a href="https://rfc.vac.dev/spec/7" target="_blank" rel="noopener noreferrer">version 0.1.0</a></li><li>Waku mailserver, currently in <a href="https://rfc.vac.dev/spec/8" target="_blank" rel="noopener noreferrer">version 0.2.0</a></li></ul><p>We can probably factor these out further as the main spec is getting quite big, but this is good enough for now.</p><p><strong>Clients:</strong></p><p>There are currently two clients that implement Waku v0.3, these are <a href="https://github.com/status-im/nim-waku" target="_blank" rel="noopener noreferrer">Nimbus (Update: now nim-waku)</a> in Nim and <a href="https://github.com/status-im/status-go" target="_blank" rel="noopener noreferrer">status-go</a> in Go.</p><p>For more details on what each client support and don't, you can follow the <a href="https://github.com/vacp2p/pm/issues/7" target="_blank" rel="noopener noreferrer">work in progress checklist</a>.</p><p>Work is currently in progress to integrate it into the <a href="https://github.com/status-im/status-react/pull/9949" target="_blank" rel="noopener noreferrer">Status core app</a>. Waku is expected to be part of their upcoming 1.1 release (see <a href="https://trello.com/b/DkxQd1ww/status-app-roadmap" target="_blank" rel="noopener noreferrer">Status app roadmap (link deprecated)</a>).</p><p><strong>Simulation:</strong></p><p>We have a <a href="https://github.com/status-im/nim-waku/blob/master/waku/v1/node/quicksim.nim" target="_blank" rel="noopener noreferrer">simulation</a> that verifies - or rather, fails to falsify - our <a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">scalability model</a>. More on the simulation and what it shows below.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="how-many-users-does-waku-support">How many users does Waku support?<a href="#how-many-users-does-waku-support" class="hash-link" aria-label="Direct link to How many users does Waku support?" title="Direct link to How many users does Waku support?">​</a></h2><p>This is our current understanding of how many users a network running Waku can support. Specifically in the context of the Status chat app, since that's the most immediate consumer of Waku. It should generalize fairly well to most deployments.</p><p><strong>tl;dr (for Status app):</strong></p><ul><li>beta: 100 DAU</li><li>v1: 1k DAU</li><li>v1.1 (waku only): 10k DAU (up to x10 with deployment hotfixes)</li><li>v1.2 (waku+dns): 100k DAU (can optionally be folded into v1.1)</li></ul><p><em>Assuming 10 concurrent users = 100 DAU. Estimate uncertainty increases for each order of magnitude until real-world data is observed.</em></p><p>As far as we know right now, these are the bottlenecks we have:</p><ul><li>Immediate bottleneck - Receive bandwidth for end user clients (aka ‘Fixing Whisper with Waku’)</li><li>Very likely bottleneck - Nodes and cluster capacity (aka ‘DNS based node discovery’)</li><li>Conjecture but not unlikely to appear- Full node traffic (aka ‘the routing / partition problem’)</li></ul><p>We've already seen the first bottleneck being discussed in the initial post. Dean wrote a post on <a href="https://vac.dev/dns-based-discovery" target="_blank" rel="noopener noreferrer">DNS based discovery</a> which explains how we will address the likely second bottleneck. More on the third one in future posts.</p><p>For more details on these bottlenecks, see <a href="https://discuss.status.im/t/scalability-estimate-how-many-users-can-waku-and-the-status-app-support/1514" target="_blank" rel="noopener noreferrer">Scalability estimate: How many users can Waku and the Status app support?</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="simulation">Simulation<a href="#simulation" class="hash-link" aria-label="Direct link to Simulation" title="Direct link to Simulation">​</a></h2><p>The ultimate test is real-world usage. Until then, we have a simulation thanks to Kim De Mey from the Nimbus team!</p><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/waku_simulation-3923bb91799b72a73608687149068b40.jpeg" width="1131" height="637" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>We have two network topologies, Star and full mesh. Both networks have 6 full nodes, one traditional light node with bloom filter, and one Waku light node.</p><p>One of the full nodes sends 1 envelope over 1 of the 100 topics that the two light nodes subscribe to. After that, it sends 10000 envelopes over random topics.</p><p>For light node, bloom filter is set to almost 10% false positive (bloom filter: n=100, k=3, m=512). It shows the number of valid and invalid envelopes received for the different nodes.</p><p><strong>Star network:</strong></p><table><thead><tr><th>Description</th><th>Peers</th><th>Valid</th><th>Invalid</th></tr></thead><tbody><tr><td>Master node</td><td>7</td><td>10001</td><td>0</td></tr><tr><td>Full node 1</td><td>3</td><td>10001</td><td>0</td></tr><tr><td>Full node 2</td><td>1</td><td>10001</td><td>0</td></tr><tr><td>Full node 3</td><td>1</td><td>10001</td><td>0</td></tr><tr><td>Full node 4</td><td>1</td><td>10001</td><td>0</td></tr><tr><td>Full node 5</td><td>1</td><td>10001</td><td>0</td></tr><tr><td>Light node</td><td>2</td><td>815</td><td>0</td></tr><tr><td>Waku light node</td><td>2</td><td>1</td><td>0</td></tr></tbody></table><p><strong>Full mesh:</strong></p><table><thead><tr><th>Description</th><th>Peers</th><th>Valid</th><th>Invalid</th></tr></thead><tbody><tr><td>Full node 0</td><td>7</td><td>10001</td><td>20676</td></tr><tr><td>Full node 1</td><td>7</td><td>10001</td><td>9554</td></tr><tr><td>Full node 2</td><td>5</td><td>10001</td><td>23304</td></tr><tr><td>Full node 3</td><td>5</td><td>10001</td><td>11983</td></tr><tr><td>Full node 4</td><td>5</td><td>10001</td><td>24425</td></tr><tr><td>Full node 5</td><td>5</td><td>10001</td><td>23472</td></tr><tr><td>Light node</td><td>2</td><td>803</td><td>803</td></tr><tr><td>Waku light node</td><td>2</td><td>1</td><td>1</td></tr></tbody></table><p>Things to note:</p><ul><li>Whisper light node with ~10% false positive gets ~10% of total traffic</li><li>Waku light node gets ~1000x less envelopes than Whisper light node</li><li>Full mesh results in a lot more duplicate messages, expect for Waku light node</li></ul><p>Run the simulation yourself <a href="https://github.com/status-im/nim-waku/blob/master/waku/v1/node/quicksim.nim" target="_blank" rel="noopener noreferrer">here</a>. The parameters are configurable, and it is integrated with Prometheus and Grafana.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="difference-between-waku-and-whisper">Difference between Waku and Whisper<a href="#difference-between-waku-and-whisper" class="hash-link" aria-label="Direct link to Difference between Waku and Whisper" title="Direct link to Difference between Waku and Whisper">​</a></h2><p>Summary of main differences between Waku v0 spec and Whisper v6, as described in <a href="https://eips.ethereum.org/EIPS/eip-627" target="_blank" rel="noopener noreferrer">EIP-627</a>:</p><ul><li>Handshake/Status message not compatible with shh/6 nodes; specifying options as association list</li><li>Include topic-interest in Status handshake</li><li>Upgradability policy</li><li><code>topic-interest</code> packet code</li><li>RLPx subprotocol is changed from shh/6 to waku/0.</li><li>Light node capability is added.</li><li>Optional rate limiting is added.</li><li>Status packet has following additional parameters: light-node, confirmations-enabled and rate-limits</li><li>Mail Server and Mail Client functionality is now part of the specification.</li><li>P2P Message packet contains a list of envelopes instead of a single envelope.</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="next-steps-and-future-plans">Next steps and future plans<a href="#next-steps-and-future-plans" class="hash-link" aria-label="Direct link to Next steps and future plans" title="Direct link to Next steps and future plans">​</a></h2><p>Several challenges remain to make Waku a robust and suitable base
communication protocol. Here we outline a few challenges that we are addressing and will continue to work on:</p><ul><li>scalability of the network</li><li>incentived infrastructure and spam-resistance</li><li>build with resource restricted devices in mind, including nodes being mostly offline</li></ul><p>For the third bottleneck, a likely candidate for fixing this is Kademlia routing. This is similar to what is done in <a href="https://www.ethswarm.org/" target="_blank" rel="noopener noreferrer">Swarm's</a> PSS. We are in the early stages of experimenting with this over libp2p in <a href="https://github.com/status-im/nim-libp2p" target="_blank" rel="noopener noreferrer">nim-libp2p</a>. More on this in a future post!</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="acknowledgements">Acknowledgements<a href="#acknowledgements" class="hash-link" aria-label="Direct link to Acknowledgements" title="Direct link to Acknowledgements">​</a></h2><p><em>Image from "caged sky" by mh.xbhd.org is licensed under CC BY 2.0 (<a href="https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e" target="_blank" rel="noopener noreferrer">https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e</a>)</em></p>]]></content:encoded>
</item>
<item>
<title><![CDATA[DNS Based Discovery]]></title>
<link>https://vac.dev/rlog/dns-based-discovery</link>
<guid>https://vac.dev/rlog/dns-based-discovery</guid>
<pubDate>Fri, 07 Feb 2020 12:00:00 GMT</pubDate>
<description><![CDATA[A look at EIP-1459 and the benefits of DNS based discovery.]]></description>
<content:encoded><![CDATA[<p>A look at EIP-1459 and the benefits of DNS based discovery.</p><p>Discovery in p2p networks is the process of how nodes find each other and specific resources they are looking for. Popular discovery protocols, such as <a href="https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf" target="_blank" rel="noopener noreferrer">Kademlia</a> which utilizes a <a href="https://en.wikipedia.org/wiki/Distributed_hash_table" target="_blank" rel="noopener noreferrer">distributed hash table</a> or DHT, are highly inefficient for resource restricted devices. These methods use short connection windows, and it is quite battery intensive to keep establishing connections. Additionally, we cannot expect a mobile phone for example to synchronize an entire DHT using cellular data.</p><p>Another issue is how we do the initial bootstrapping. In other words, how does a client find its first node to then discover the rest of the network? In most applications, including Status right now, this is done with a <a href="https://specs.status.im/spec/1#bootstrapping" target="_blank" rel="noopener noreferrer">static list of nodes</a> that a client can connect to.</p><p>In summary, we have a static list that provides us with nodes we can connect to which then allows us to discover the rest of the network using something like Kademlia. But what we need is something that can easily be mutated, guarantees a certain amount of security, and is efficient for resource restricted devices. Ideally our solution would also be robust and scalable.</p><p>How do we do this?</p><p><a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP 1459: Node Discovery via DNS</a>, which is one of the strategies we are using for discovering waku nodes. <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a> is a DNS-based discovery protocol that stores <a href="https://en.wikipedia.org/wiki/Merkle_tree" target="_blank" rel="noopener noreferrer">merkle trees</a> in DNS records which contain connection information for nodes.</p><p><em>Waku is our fork of Whisper. Oskar recently wrote an <a href="https://vac.dev/fixing-whisper-with-waku" target="_blank" rel="noopener noreferrer">entire post</a> explaining it. In short, Waku is our method of fixing the shortcomings of Whisper in a more iterative fashion. You can find the specification <a href="https://rfc.vac.dev/spec/6/" target="_blank" rel="noopener noreferrer">here</a></em></p><p>DNS-based methods for bootstrapping p2p networks are quite popular. Even Bitcoin uses it, but it uses a concept called DNS seeds, which are just DNS servers that are configured to return a list of randomly selected nodes from the network upon being queried. This means that although these seeds are hardcoded in the client, the IP addresses of actual nodes do not have to be.</p><div class="language-console codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-console codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">&gt; dig dnsseed.bluematt.me +short</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">129.226.73.12</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">107.180.78.111</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">169.255.56.123</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">91.216.149.28</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">85.209.240.91</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">66.232.124.232</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">207.55.53.96</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">86.149.241.168</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">193.219.38.57</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">190.198.210.139</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">74.213.232.234</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">158.181.226.33</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">176.99.2.207</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">202.55.87.45</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">37.205.10.3</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">90.133.4.73</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">176.191.182.3</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">109.207.166.232</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">45.5.117.59</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">178.211.170.2</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">160.16.0.30</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>The above displays the result of querying on of these DNS seeds. All the nodes are stored as <a href="https://simpledns.plus/help/a-records" target="_blank" rel="noopener noreferrer"><code>A</code> records</a> for the given domain name. This is quite a simple solution which Bitcoin almost soley relies on since removing the <a href="https://en.bitcoin.it/wiki/Network#IRC" target="_blank" rel="noopener noreferrer">IRC bootstrapping method in v0.8.2</a>.</p><p>What makes this DNS based discovery useful? It allows us to have a mutable list of bootstrap nodes without needing to ship a new version of the client every time a list is mutated. It also allows for a more lightweight method of discovering nodes, something very important for resource restricted devices.</p><p>Additionally, DNS provides us with a robust and scalable infrastructure. This is due to its hierarchical architecture. This hierarchical architecture also already makes it distributed such that the failure of one DNS server does not result in us no longer being able to resolve our name.</p><p>As with every solution though, there is a trade-off. By storing the list in DNS name an adversary would simply need to censor the DNS records for a specific name. This would prevent any new client trying to join the network from being able to do so.</p><p>One thing you notice when looking at <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a> is that it is a lot more technically complex than Bitcoin's way of doing this. So if Bitcoin uses this simple method and has proven that it works, why did we need a new method?</p><p>There are multiple reasons, but the main one is <strong>security</strong>. In the Bitcoin example, an attacker could create a new list and no one querying would be able to tell. This is however mitigated in <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a> where we can verify the integrity of the entire returned list by storing an entire merkle tree in the DNS records.</p><p>Let's dive into this. Firstly, a client that is using these DNS records for discovery must know the public key corresponding to the private key controlled by the entity creating the list. This is because the entire list is signed using a secp256k1 private key, giving the client the ability to authenticate the list and know that it has not been tampered with by some external party.</p><p>So that already makes this a lot safer than the method Bitcoin uses. But how are these lists even stored? As previously stated they are stored using <strong>merkle trees</strong> as follows:</p><ul><li><p>The root of the tree is stored in a <a href="https://simpledns.plus/help/txt-records" target="_blank" rel="noopener noreferrer"><code>TXT</code> record</a>, this record contains the tree's root hash, a sequence number which is incremented every time the tree is updated and a signature as stated above.</p><p>Additionally, there is also a root hash to a second tree called a <strong>link tree</strong>, it contains the information to different lists. This link tree allows us to delegate trust and build a graph of multiple merkle trees stored across multiple DNS names.</p><p>The sequence number ensures that an attacker cannot replace a tree with an older version because when a client reads the tree, they should ensure that the sequence number is greater than the last synchronized version.</p></li><li><p>Using the root hash for the tree, we can find the merkle tree's first branch, the branch is also stored in a <code>TXT</code> record. The branch record contains all the hashes of the branch's leafs.</p></li><li><p>Once a client starts reading all the leafs, they can find one of two things: either a new branch record leading them further down the tree or an Ethereum Name Records (ENR) which means they now have the address of a node to connect to! To learn more about ethereum node records you can have a look at <a href="https://eips.ethereum.org/EIPS/eip-778" target="_blank" rel="noopener noreferrer">EIP-778</a>, or read a short blog post I wrote explaining them <a href="https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/#enr" target="_blank" rel="noopener noreferrer">here</a>.</p></li></ul><p>Below is the zone file taken from the <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a>, displaying how this looks in practice.</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">; name ttl class type content</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">@ 60 IN TXT enrtree-root:v1 e=JWXYDBPXYWG6FX3GMDIBFA6CJ4 l=C7HRFPF3BLGF3YR4DY5KX3SMBE seq=1 sig=o908WmNp7LibOfPsr4btQwatZJ5URBr2ZAuxvK4UWHlsB9sUOTJQaGAlLPVAhM__XJesCHxLISo94z5Z2a463gA</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">C7HRFPF3BLGF3YR4DY5KX3SMBE 86900 IN TXT enrtree://AM5FCQLWIZX2QFPNJAP7VUERCCRNGRHWZG3YYHIUV7BVDQ5FDPRT2@morenodes.example.org</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">JWXYDBPXYWG6FX3GMDIBFA6CJ4 86900 IN TXT enrtree-branch:2XS2367YHAXJFGLZHVAWLQD4ZY,H4FHT4B454P6UXFD7JCYQ5PWDY,MHTDO6TMUBRIA2XWG5LUDACK24</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">2XS2367YHAXJFGLZHVAWLQD4ZY 86900 IN TXT enr:-HW4QOFzoVLaFJnNhbgMoDXPnOvcdVuj7pDpqRvh6BRDO68aVi5ZcjB3vzQRZH2IcLBGHzo8uUN3snqmgTiE56CH3AMBgmlkgnY0iXNlY3AyNTZrMaECC2_24YYkYHEgdzxlSNKQEnHhuNAbNlMlWJxrJxbAFvA</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">H4FHT4B454P6UXFD7JCYQ5PWDY 86900 IN TXT enr:-HW4QAggRauloj2SDLtIHN1XBkvhFZ1vtf1raYQp9TBW2RD5EEawDzbtSmlXUfnaHcvwOizhVYLtr7e6vw7NAf6mTuoCgmlkgnY0iXNlY3AyNTZrMaECjrXI8TLNXU0f8cthpAMxEshUyQlK-AM0PW2wfrnacNI</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">MHTDO6TMUBRIA2XWG5LUDACK24 86900 IN TXT enr:-HW4QLAYqmrwllBEnzWWs7I5Ev2IAs7x_dZlbYdRdMUx5EyKHDXp7AV5CkuPGUPdvbv1_Ms1CPfhcGCvSElSosZmyoqAgmlkgnY0iXNlY3AyNTZrMaECriawHKWdDRk2xeZkrOXBQ0dfMFLHY4eENZwdufn1S1o</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>All of this has already been introduced into go-ethereum with the pull request <a href="https://github.com/ethereum/go-ethereum/pull/20094" target="_blank" rel="noopener noreferrer">#20094</a>, created by Felix Lange. There's a lot of tooling around it that already exists too which is really cool. So if your project is written in Golang and wants to use this, it's relatively simple! Additionally, here's a proof of concept that shows what this might look like with libp2p on <a href="https://github.com/decanus/dns-discovery" target="_blank" rel="noopener noreferrer">github</a>.</p><p>I hope this was a helpful explainer into DNS based discovery, and shows <a href="https://eips.ethereum.org/EIPS/eip-1459" target="_blank" rel="noopener noreferrer">EIP-1459</a>'s benefits over more traditional DNS-based discovery schemes.</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Fixing Whisper with Waku]]></title>
<link>https://vac.dev/rlog/fixing-whisper-with-waku</link>
<guid>https://vac.dev/rlog/fixing-whisper-with-waku</guid>
<pubDate>Tue, 03 Dec 2019 12:00:00 GMT</pubDate>
<description><![CDATA[A research log. Why Whisper doesn't scale and how to fix it.]]></description>
<content:encoded><![CDATA[<p>A research log. Why Whisper doesn't scale and how to fix it.</p><p>This post will introduce Waku. Waku is a fork of Whisper that attempts to
addresses some of Whisper's shortcomings in an iterative fashion. We will also
introduce a theoretical scaling model for Whisper that shows why it doesn't
scale, and what can be done about it.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2><p>Whisper is a gossip-based communication protocol or an ephemeral key-value store
depending on which way you look at it. Historically speaking, it is the
messaging pilllar of <a href="http://gavwood.com/dappsweb3.html" target="_blank" rel="noopener noreferrer">Web3</a>, together with
Ethereum for consensus and Swarm for storage.</p><p>Whisper, being a somewhat esoteric protocol and with some fundamental issues,
hasn't seen a lot of usage. However, applications such as Status are using it,
and have been making minor ad hoc modifications to it to make it run on mobile
devices.</p><p>What are these fundamental issues? In short:</p><ol><li>scalability, most immediately when it comes to bandwidth usage</li><li>spam-resistance, proof of work is a poor mechanism for heterogeneous nodes</li><li>no incentivized infrastructure, leading to centralized choke points</li><li>lack of formal and unambiguous specification makes it hard to analyze and implement</li><li>running over devp2p, which limits where it can run and how</li></ol><p>In this post, we'll focus on the first problem, which is scalability through bandwidth usage.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="whisper-theoretical-scalability-model">Whisper theoretical scalability model<a href="#whisper-theoretical-scalability-model" class="hash-link" aria-label="Direct link to Whisper theoretical scalability model" title="Direct link to Whisper theoretical scalability model">​</a></h2><p><em>(Feel free to skip this section if you want to get right to the results).</em></p><p>There's widespread implicit knowledge that Whisper "doesn't scale", but it is less understood exactly why. This theoretical model attempts to encode some characteristics of it. Specifically for use case such as one by Status (see <a href="https://specs.status.im/spec/3" target="_blank" rel="noopener noreferrer">Status Whisper usage
spec</a>).</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="caveats">Caveats<a href="#caveats" class="hash-link" aria-label="Direct link to Caveats" title="Direct link to Caveats">​</a></h3><p>First, some caveats: this model likely contains bugs, has wrong assumptions, or completely misses certain dimensions. However, it acts as a form of existence proof for unscalability, with clear reasons.</p><p>If certain assumptions are wrong, then we can challenge them and reason about them in isolation. It doesn’t mean things will definitely work as the model predicts, and that there aren’t unknown unknowns.</p><p>The model also only deals with receiving bandwidth for end nodes, uses mostly static assumptions of averages, and doesn’t deal with spam resistance, privacy guarantees, accounting, intermediate node or network wide failures.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goals">Goals<a href="#goals" class="hash-link" aria-label="Direct link to Goals" title="Direct link to Goals">​</a></h3><ol><li>Ensure network scales by being user or usage bound, as opposed to bandwidth growing in proportion to network size.</li><li>Staying with in a reasonable bandwidth limit for limited data plans.</li><li>Do the above without materially impacting existing nodes.</li></ol><p>It proceeds through various case with clear assumptions behind them, starting from the most naive assumptions. It shows results for 100 users, 10k users and 1m users.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="model">Model<a href="#model" class="hash-link" aria-label="Direct link to Model" title="Direct link to Model">​</a></h3><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 1. Only receiving messages meant for you [naive case]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A4. Only receiving messages meant for you.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 1000.0KB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 1000.0KB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 1000.0KB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 2. Receiving messages for everyone [naive case]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A5. Received messages for everyone.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 97.7MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 9.5GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 953.7GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 3. All private messages go over one discovery topic [naive case]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A6. Proportion of private messages (static): 0.5</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A7. Public messages only received by relevant recipients (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A8. All private messages are received by everyone (same topic) (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 49.3MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 4.8GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 476.8GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 4. All private messages are partitioned into shards [naive case]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A6. Proportion of private messages (static): 0.5</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A7. Public messages only received by relevant recipients (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A9. Private messages partitioned across partition shards (static), n=5000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 1000.0KB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 1.5MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 98.1MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 5. 4 + Bloom filter with false positive rate</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A6. Proportion of private messages (static): 0.5</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A7. Public messages only received by relevant recipients (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A9. Private messages partitioned across partition shards (static), n=5000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A10. Bloom filter size (m) (static): 512</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A11. Bloom filter hash functions (k) (static): 3</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A12. Bloom filter elements, i.e. topics, (n) (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A13. Bloom filter assuming optimal k choice (sensitive to m, n).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A14. Bloom filter false positive proportion of full traffic, p=0.1</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 10.7MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 978.0MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 95.5GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">NOTE: Traffic extremely sensitive to bloom false positives</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">This completely dominates network traffic at scale.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">With p=1% we get 10k users ~100MB/day and 1m users ~10gb/day)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 6. Case 5 + Benign duplicate receives</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A6. Proportion of private messages (static): 0.5</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A7. Public messages only received by relevant recipients (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A9. Private messages partitioned across partition shards (static), n=5000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A10. Bloom filter size (m) (static): 512</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A11. Bloom filter hash functions (k) (static): 3</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A12. Bloom filter elements, i.e. topics, (n) (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A13. Bloom filter assuming optimal k choice (sensitive to m, n).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A14. Bloom filter false positive proportion of full traffic, p=0.1</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A15. Benign duplicate receives factor (static): 2</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A16. No bad envelopes, bad PoW, expired, etc (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 21.5MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 1.9GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 190.9GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 7. 6 + Mailserver under good conditions; small bloom fp; mostly offline</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A6. Proportion of private messages (static): 0.5</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A7. Public messages only received by relevant recipients (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A9. Private messages partitioned across partition shards (static), n=5000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A10. Bloom filter size (m) (static): 512</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A11. Bloom filter hash functions (k) (static): 3</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A12. Bloom filter elements, i.e. topics, (n) (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A13. Bloom filter assuming optimal k choice (sensitive to m, n).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A14. Bloom filter false positive proportion of full traffic, p=0.1</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A15. Benign duplicate receives factor (static): 2</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A16. No bad envelopes, bad PoW, expired, etc (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A17. User is offline p% of the time (static) p=0.9</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A18. No bad request, dup messages for mailservers; overlap perfect (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A19. Mailserver requests can change false positive rate to be p=0.01</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 3.9MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 284.8MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 27.8GB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Case 8. No metadata protection w bloom filter; 1 node connected; static shard</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Aka waku mode.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Next step up is to either only use contact code, or shard more aggressively.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Note that this requires change of other nodes behavior, not just local node.</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">Assumptions:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A1. Envelope size (static): 1024kb</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A2. Envelopes / message (static): 10</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A3. Received messages / day (static): 100</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A6. Proportion of private messages (static): 0.5</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A7. Public messages only received by relevant recipients (static).</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">- A9. Private messages partitioned across partition shards (static), n=5000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 100 users, receiving bandwidth is 1000.0KB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 10k users, receiving bandwidth is 1.5MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">For 1m users, receiving bandwidth is 98.1MB/day</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">------------------------------------------------------------</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>See <a href="https://github.com/vacp2p/research/tree/master/whisper_scalability" target="_blank" rel="noopener noreferrer">source</a>
for more detail on the model and its assumptions.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="takeaways">Takeaways<a href="#takeaways" class="hash-link" aria-label="Direct link to Takeaways" title="Direct link to Takeaways">​</a></h3><ol><li>Whisper as it currently works doesn’t scale, and we quickly run into unacceptable bandwidth usage.</li><li>There are a few factors of this, but largely it boils down to noisy topics usage and use of bloom filters. Duplicate (e.g. see <a href="https://our.status.im/whisper-pss-comparison/" target="_blank" rel="noopener noreferrer">Whisper vs PSS</a>) and bad envelopes are also factors, but this depends a bit more on specific deployment configurations.</li><li>Waku mode (case 8) is an additional capability that doesn’t require other nodes to change, for nodes that put a premium on performance.</li><li>The next bottleneck after this is the partitioned topics (app/network specific), which either needs to gracefully (and potentially quickly) grow, or an alternative way of consuming those messages needs to be deviced.</li></ol><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" src="/assets/images/whisper_scalability-4450d316341c4305ffc6f580a776fc2d.png" width="1920" height="934" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>The results are summarized in the graph above. Notice the log-log scale. The
colored backgrounds correspond to the following bandwidth usage:</p><ul><li>Blue: &lt;10mb/d (&lt;~300mb/month)</li><li>Green: &lt;30mb/d (&lt;~1gb/month)</li><li>Yellow: &lt;100mb/d (&lt;~3gb/month)</li><li>Red: &gt;100mb/d (&gt;3gb/month)</li></ul><p>These ranges are somewhat arbitrary, but are based on <a href="https://github.com/status-im/status-react/issues/9081" target="_blank" rel="noopener noreferrer">user
requirements</a> for users
on a limited data plan, with comparable usage for other messaging apps.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introducing-waku">Introducing Waku<a href="#introducing-waku" class="hash-link" aria-label="Direct link to Introducing Waku" title="Direct link to Introducing Waku">​</a></h2><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="motivation-for-a-new-protocol">Motivation for a new protocol<a href="#motivation-for-a-new-protocol" class="hash-link" aria-label="Direct link to Motivation for a new protocol" title="Direct link to Motivation for a new protocol">​</a></h3><p>Apps such as Status will likely use something like Whisper for the forseeable
future, and we want to enable them to use it with more users on mobile devices
without bandwidth exploding with minimal changes.</p><p>Additionally, there's not a clear cut alternative that maps cleanly to the
desired use cases (p2p, multicast, privacy-preserving, open, etc).</p><p>We are actively researching, developing and collaborating with more greenfield
approaches. It is likely that Waku will either converge to those, or Waku will
lay the groundwork (clear specs, common issues/components) necessary to make
switching to another protocol easier. In this project we want to emphasize
iterative work with results on the order of weeks.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="briefly-on-waku-mode">Briefly on Waku mode<a href="#briefly-on-waku-mode" class="hash-link" aria-label="Direct link to Briefly on Waku mode" title="Direct link to Briefly on Waku mode">​</a></h3><ul><li>Doesn’t impact existing clients, it’s just a separate node and capability.</li><li>Other nodes can still use Whisper as is, like a full node.</li><li>Sacrifices metadata protection and incurs higher connectivity/availability requirements for scalbility</li></ul><p><strong>Requirements:</strong></p><ul><li>Exposes API to get messages from a set of list of topics (no bloom filter)</li><li>Way of being identified as a Waku node (e.g. through version string)</li><li>Option to statically encode this node in app, e.g. similar to custom bootnodes/mailserver</li><li>Only node that needs to be connected to, possibly as Whisper relay / mailserver hybrid</li></ul><p><strong>Provides:</strong></p><ul><li>likely provides scalability of up to 10k users and beyond</li><li>with some enhancements to partition topic logic, can possibly scale up to 1m users (app/network specific)</li></ul><p><strong>Caveats:</strong></p><ul><li>hasn’t been tested in a large-scale simulation</li><li>other network and intermediate node bottlenecks might become apparent (e.g. full bloom filter and private cluster capacity; can likely be dealt with in isolation using known techniques, e.g. load balancing) (deployment specific)</li></ul><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="progress-so-far">Progress so far<a href="#progress-so-far" class="hash-link" aria-label="Direct link to Progress so far" title="Direct link to Progress so far">​</a></h3><p>In short, we have a <a href="https://rfc.vac.dev/spec/5" target="_blank" rel="noopener noreferrer">Waku version 0 spec up</a> as well as a <a href="https://github.com/status-im/nim-eth/pull/120" target="_blank" rel="noopener noreferrer">PoC</a> for backwards compatibility. In the coming weeks, we are going to solidify the specs, get a more fully featured PoC for <a href="https://github.com/status-im/nim-eth/pull/114" target="_blank" rel="noopener noreferrer">Waku mode</a>. See <a href="https://github.com/vacp2p/pm/issues/5" target="_blank" rel="noopener noreferrer">rough roadmap</a>, project board <!-- -->[link deprecated]<!-- --> and progress thread on the <a href="https://forum.vac.dev/t/waku-project-and-progress/24" target="_blank" rel="noopener noreferrer">Vac forum</a>.</p><p>The spec has been rewrittten for clarity, with ABNF grammar and less ambiguous language. The spec also incorporates several previously <a href="https://rfc.vac.dev/spec/6/#additional-capabilities" target="_blank" rel="noopener noreferrer">ad hoc implemented features</a>, such as light nodes and mailserver/client support. This has already caught a few incompatibilities between the <code>geth</code> (Go), <code>status/whisper</code> (Go) and <code>nim-eth</code> (Nim) versions, specifically around light node usage and the handshake.</p><p>If you are interested in this effort, please check out <a href="https://forum.vac.dev/" target="_blank" rel="noopener noreferrer">our forum</a> for questions, comments and proposals. We already have some discussion for better <a href="https://forum.vac.dev/t/stake-priority-based-queuing/26" target="_blank" rel="noopener noreferrer">spam protection</a> (see <a href="https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks" target="_blank" rel="noopener noreferrer">previous post</a> for a more complex but privacy-preserving proposal), something that is likely going to be addressed in future versions of Waku, along with many other fixes and enhancement.</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Feasibility Study: Semaphore rate limiting through zkSNARKs]]></title>
<link>https://vac.dev/rlog/feasibility-semaphore-rate-limiting-zksnarks</link>
<guid>https://vac.dev/rlog/feasibility-semaphore-rate-limiting-zksnarks</guid>
<pubDate>Fri, 08 Nov 2019 12:00:00 GMT</pubDate>
<description><![CDATA[A research log. Zero knowledge signaling as a rate limiting mechanism to prevent spam in p2p networks.]]></description>
<content:encoded><![CDATA[<p>A research log. Zero knowledge signaling as a rate limiting mechanism to prevent spam in p2p networks.</p><p><strong>tldr: Moon math promising for solving spam in Whisper, but to get there we need to invest more in performance work and technical upskilling.</strong></p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="motivating-problem">Motivating problem<a href="#motivating-problem" class="hash-link" aria-label="Direct link to Motivating problem" title="Direct link to Motivating problem">​</a></h2><p>In open p2p networks for messaging, one big problem is spam-resistance. Existing solutions, such as Whisper's proof of work, are insufficient, especially for heterogeneous nodes. Other reputation-based approaches might not be desirable, due to issues around arbitrary exclusion and privacy.</p><p>One possible solution is to use a right-to-access staking-based method, where a node is only able to send a message, signal, at a certain rate, and otherwise they can be slashed. One problem with this is in terms of privacy-preservation, where we specifically don't want a user to be tied to a specific payment or unique fingerprint.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="related-problems">Related problems<a href="#related-problems" class="hash-link" aria-label="Direct link to Related problems" title="Direct link to Related problems">​</a></h3><p>In addition to above, there are a lot of related problems that share similarities in terms of their structure and proposed solution.</p><ul><li>Private transactions (<a href="https://z.cash/" target="_blank" rel="noopener noreferrer">Zcash</a>, <a href="https://www.aztecprotocol.com/" target="_blank" rel="noopener noreferrer">AZTEC</a>)</li><li>Private voting (<a href="https://github.com/kobigurk/semaphore" target="_blank" rel="noopener noreferrer">Semaphore</a>)</li><li>Private group membership (Semaphore)</li><li>Layer 2 scaling, poss layer 1 (<a href="https://ethresear.ch/t/on-chain-scaling-to-potentially-500-tx-sec-through-mass-tx-validation/3477" target="_blank" rel="noopener noreferrer">ZK Rollup</a>; StarkWare/Eth2-3)</li></ul><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="overview">Overview<a href="#overview" class="hash-link" aria-label="Direct link to Overview" title="Direct link to Overview">​</a></h2><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="basic-terminology">Basic terminology<a href="#basic-terminology" class="hash-link" aria-label="Direct link to Basic terminology" title="Direct link to Basic terminology">​</a></h2><p>A <em>zero-knowledge proof</em> allows a <em>prover</em> to show a <em>verifier</em> that they know something, without revealing what that something is. This means you can do trust-minimized computation that is also privacy preserving. As a basic example, instead of showing your ID when going to a bar you simply give them a proof that you are over 18, without showing the doorman your id.</p><p><em>zkSNARKs</em> is a form of zero-knowledge proofs. There are many types of zero-knowledge proofs, and the field is evolving rapidly. They come with various trade-offs in terms of things such as: trusted setup, cryptographic assumptions, proof/verification key size, proof/verification time, proof size, etc. See section below for more.</p><p><em>Semaphore</em> is a framework/library/construct on top of zkSNARks. It allows for zero-knowledge signaling, specifically on top of Ethereum. This means an approved user can broadcast some arbitrary string without revealing their identity, given some specific constraints. An approved user is someone who has been added to a certain merkle tree. See <a href="https://github.com/kobigurk/semaphore" target="_blank" rel="noopener noreferrer">current Github home</a> for more.</p><p><em>Circom</em> is a DSL for writing arithmetic circuits that can be used in zkSNARKs, similar to how you might write a NAND gate. See <a href="https://github.com/iden3/circom" target="_blank" rel="noopener noreferrer">Github</a> for more.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="basic-flow">Basic flow<a href="#basic-flow" class="hash-link" aria-label="Direct link to Basic flow" title="Direct link to Basic flow">​</a></h2><p>We start with a private voting example, and then extend it to the slashable rate limiting example.</p><ol><li><p>A user registers an identity (arbitrary keypair), along with a small fee, to a smart contract. This adds them to a merkle tree and allows them to prove that they are member of that group, without revealing who they are.</p></li><li><p>When a user wants to send a message, they compute a zero-knowledge proof. This ensures certain invariants, have some <em>public outputs</em>, and can be verified by anyone (including a smart contract).</p></li><li><p>Any node can verify the proof, including smart contracts on chain (as of Byzantinum HF). Additionally, a node can have rules for the public output. In the case of voting, one such rule is that a specific output hash has to be equal to some predefined value, such as "2020-01-01 vote on Foo Bar for president".</p></li><li><p>Because of how the proof is constructed, and the rules around output values, this ensures that: a user is part of the approved set of voters and that a user can only vote once.</p></li><li><p>As a consequence of above, we have a system where registered users can only vote once, no one can see who voted for what, and this can all be proven and verified.</p></li></ol><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rate-limiting-example">Rate limiting example<a href="#rate-limiting-example" class="hash-link" aria-label="Direct link to Rate limiting example" title="Direct link to Rate limiting example">​</a></h3><p>In the case of rate limiting, we do want nodes to send multiple messages. This changes step 3-5 above somewhat.</p><p><em>NOTE: It is a bit more involved than this, and if we precompute proofs the flow might look a bit different. But the general idea is the same</em>.</p><ol><li><p>Instead of having a rule that you can only vote once, we have a rule that you can only send a message per epoch. Epoch here can be every second, as defined by UTC date time +-20s.</p></li><li><p>Additionally, if a users sends more than one message per epoch, one of the public outputs is a random share of a private key. Using Shamir's Secret Sharing (similar to a multisig) and 2/3 key share as an example threshold: in the normal case only 1/3 private keys is revealed, which is insufficient to have access. In the case where two messages are sent in an epoch, probabilistically 2/3 shares is sufficient to have access to the key (unless you get the same random share of the key).</p></li><li><p>This means any untrusted user who detects a spamming user, can use it to access their private key corresponding to funds in the contract, and thus slash them.</p></li><li><p>As a consequence of above, we have a system where registered users can only messages X times per epoch, and no one can see who is sending what messages. Additionally, if a user is violating the above rate limit, they can be punished and any user can profit from it.</p></li></ol><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="briefly-on-scope-of-approved-users">Briefly on scope of 'approved users'<a href="#briefly-on-scope-of-approved-users" class="hash-link" aria-label="Direct link to Briefly on scope of 'approved users'" title="Direct link to Briefly on scope of 'approved users'">​</a></h3><p>In the case of an application like Status, this construct can either be a global StatusNetwork group, or one per chat, or network, etc. It can be applied both at the network and user level. There are no specific limitations on where or who deploys this, and it is thus more of a UX consideration.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="technical-details">Technical details<a href="#technical-details" class="hash-link" aria-label="Direct link to Technical details" title="Direct link to Technical details">​</a></h2><p>For a fairly self-contained set of examples above, see exploration in <a href="https://github.com/vacp2p/research/blob/master/zksnarks/semaphore/src/hello.js" target="_blank" rel="noopener noreferrer">Vac research repo</a>. Note that the Shamir secret sharing is not inside the SNARK, but out-of-band for now.</p><p>The <a href="https://github.com/kobigurk/semaphore" target="_blank" rel="noopener noreferrer">current version</a> of Semaphore is using NodeJS and <a href="https://github.com/iden3/circom" target="_blank" rel="noopener noreferrer">Circom</a> from Iden3 for Snarks.</p><p>For more on rate limiting idea, see <a href="https://ethresear.ch/t/semaphore-rln-rate-limiting-nullifier-for-spam-prevention-in-anonymous-p2p-setting/5009/" target="_blank" rel="noopener noreferrer">ethresearch post</a>.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="feasibility">Feasibility<a href="#feasibility" class="hash-link" aria-label="Direct link to Feasibility" title="Direct link to Feasibility">​</a></h2><p>The above repo was used to exercise the basic paths and to gain intution of feasibility. Based on it and related reading we outline a few blockers and things that require further study.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="technical-feasibility">Technical feasibility<a href="#technical-feasibility" class="hash-link" aria-label="Direct link to Technical feasibility" title="Direct link to Technical feasibility">​</a></h3><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proof-time">Proof time<a href="#proof-time" class="hash-link" aria-label="Direct link to Proof time" title="Direct link to Proof time">​</a></h4><p>Prove time for Semaphore (<a href="https://github.com/kobigurk/semaphore" target="_blank" rel="noopener noreferrer">https://github.com/kobigurk/semaphore</a>) zKSNARKs using circom, groth and snarkjs is currently way too long. It takes on the order of ~10m to generate a proof. With Websnark, it is likely to take 30s, which might still be too long. We should experiment with native code on mobile here.</p><p>See <a href="https://github.com/vacp2p/research/issues/7" target="_blank" rel="noopener noreferrer">details</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proving-key-size">Proving key size<a href="#proving-key-size" class="hash-link" aria-label="Direct link to Proving key size" title="Direct link to Proving key size">​</a></h4><p>Prover key size is ~110mb for Semaphore. Assuming this is embedded on mobile device, it bloats the APK a lot. Current APK size is ~30mb and even that might be high for people with limited bandwidth.</p><p>See <a href="https://github.com/vacp2p/research/issues/8" target="_blank" rel="noopener noreferrer">details</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="trusted-setup">Trusted setup<a href="#trusted-setup" class="hash-link" aria-label="Direct link to Trusted setup" title="Direct link to Trusted setup">​</a></h4><p>Using zkSNARKs a trusted setup is required to generate prover and verifier keys. As part of this setup, a toxic parameter lambda is generated. If a party gets access to this lambda, they can prove anything. This means people using zKSNARKs usually have an elaborate MPC ceremony to ensure this parameter doesn't get discovered.</p><p>See <a href="https://github.com/vacp2p/research/issues/9" target="_blank" rel="noopener noreferrer">details</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="shamir-logic-in-snark">Shamir logic in SNARK<a href="#shamir-logic-in-snark" class="hash-link" aria-label="Direct link to Shamir logic in SNARK" title="Direct link to Shamir logic in SNARK">​</a></h4><p>For <a href="https://ethresear.ch/t/semaphore-rln-rate-limiting-nullifier-for-spam-prevention-in-anonymous-p2p-setting/5009" target="_blank" rel="noopener noreferrer">Semaphore RLN</a> we need to embed the Shamir logic inside the SNARK in order to do slashing for spam. Currently the <a href="https://github.com/vacp2p/research/blob/master/zksnarks/semaphore/src/hello.js#L450" target="_blank" rel="noopener noreferrer">implementation</a> is trusted and very hacky.</p><p>See <a href="https://github.com/vacp2p/research/issues/10" target="_blank" rel="noopener noreferrer">details</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="end-to-end-integation">End to end integation<a href="#end-to-end-integation" class="hash-link" aria-label="Direct link to End to end integation" title="Direct link to End to end integation">​</a></h4><p><a href="https://github.com/vacp2p/research/blob/master/zksnarks/semaphore/src/hello.js" target="_blank" rel="noopener noreferrer">Currently</a> is standalone and doesn't touch multiple users, deployed contract with merkle tree and verification, actual transactions, a mocked network, add/remove members, etc. There are bound to be edge cases and unknown unknowns here.</p><p>See <a href="https://github.com/vacp2p/research/issues/11" target="_blank" rel="noopener noreferrer">details</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="licensing-issues">Licensing issues<a href="#licensing-issues" class="hash-link" aria-label="Direct link to Licensing issues" title="Direct link to Licensing issues">​</a></h4><p>Currently Circom <a href="https://github.com/iden3/circom/blob/master/COPYING" target="_blank" rel="noopener noreferrer">uses a GPL license</a>, which can get tricky when it comes to the App Store etc.</p><p>See <a href="https://github.com/vacp2p/research/issues/12" target="_blank" rel="noopener noreferrer">details</a>.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="alternative-zkps">Alternative ZKPs?<a href="#alternative-zkps" class="hash-link" aria-label="Direct link to Alternative ZKPs?" title="Direct link to Alternative ZKPs?">​</a></h4><p>Some of the isolated blockers for zKSNARKs (<a href="https://github.com/vacp2p/research/issues/7" target="_blank" rel="noopener noreferrer">#7</a>, <a href="https://github.com/vacp2p/research/issues/8" target="_blank" rel="noopener noreferrer">#8</a>, <a href="https://github.com/vacp2p/research/issues/9" target="_blank" rel="noopener noreferrer">#9</a>) might be mitigated by the use of other ZKP technology. However, they likely have their own issues.</p><p>See <a href="https://github.com/vacp2p/research/issues/13" target="_blank" rel="noopener noreferrer">details</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="social-feasibility">Social feasibility<a href="#social-feasibility" class="hash-link" aria-label="Direct link to Social feasibility" title="Direct link to Social feasibility">​</a></h3><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="technical-skill">Technical skill<a href="#technical-skill" class="hash-link" aria-label="Direct link to Technical skill" title="Direct link to Technical skill">​</a></h4><p>zkSNARKs and related technologies are quite new. To learn how they work and get an intuition for them requires individuals to dedicate a lot of time to studying them. This means we must make getting competence in these technologies if we wish to use them to our advantage.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="time-and-resources">Time and resources<a href="#time-and-resources" class="hash-link" aria-label="Direct link to Time and resources" title="Direct link to Time and resources">​</a></h4><p>In order for this and related projects (such as private transaction) to get anywhere, it must be made an explicit area of focus for an extend period of time.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="general-thoughts">General thoughts<a href="#general-thoughts" class="hash-link" aria-label="Direct link to General thoughts" title="Direct link to General thoughts">​</a></h2><p>Similar to Whisper, and in line with moving towards protocol and infrastructure, we need to upskill and invest resources into this. This doesn't mean developing all of the technologies ourselves, but gaining enough competence to leverage and extend existing solutions by the growing ZKP community.</p><p>For example, this might also include leveraging largely ready made solutions such as AZTEC for private transaction; more fundamental research into ZK rollup and similar; using Semaphore for private group membership and private voting; Nim based wrapper aronud Bellman, etc.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="acknowledgement">Acknowledgement<a href="#acknowledgement" class="hash-link" aria-label="Direct link to Acknowledgement" title="Direct link to Acknowledgement">​</a></h2><p>Thanks to Barry Whitehat for patient explanation and pointers. Thanks to WJ for helping with runtime issues.</p><p><em>Peacock header image from <!-- -->[Tonos]<!-- -->(&lt;<a href="https://en.wikipedia.org/wiki/File:Flickr" target="_blank" rel="noopener noreferrer">https://en.wikipedia.org/wiki/File:Flickr</a></em>-<em>lo.tangelini</em>-<em>Tonos</em>(1).jpg&gt;).<!-- -->_</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[P2P Data Sync with a Remote Log]]></title>
<link>https://vac.dev/rlog/remote-log</link>
<guid>https://vac.dev/rlog/remote-log</guid>
<pubDate>Fri, 04 Oct 2019 12:00:00 GMT</pubDate>
<description><![CDATA[A research log. Asynchronous P2P messaging? Remote logs to the rescue!]]></description>
<content:encoded><![CDATA[<p>A research log. Asynchronous P2P messaging? Remote logs to the rescue!</p><p>A big problem when doing end-to-end data sync between mobile nodes is that most devices are offline most of the time. With a naive approach, you quickly run into issues of 'ping-pong' behavior, where messages have to be constantly retransmitted. We saw some basic calculations of what this bandwidth multiplier looks like in a <a href="https://vac.dev/p2p-data-sync-for-mobile" target="_blank" rel="noopener noreferrer">previous post</a>.</p><p>While you could do some background processing, this is really battery-draining, and on iOS these capabilities are limited. A better approach instead is to loosen the constraint that two nodes need to be online at the same time. How do we do this? There are two main approaches, one is the <em>store and forward model</em>, and the other is a <em>remote log</em>.</p><p>In the <em>store and forward</em> model, we use an intermediate node that forward messages on behalf of the recipient. In the <em>remote log</em> model, you instead replicate the data onto some decentralized storage, and have a mutable reference to the latest state, similar to DNS. While both work, the latter is somewhat more elegant and "pure", as it has less strict requirements of an individual node's uptime. Both act as a highly-available cache to smoothen over non-overlapping connection windows between endpoints.</p><p>In this post we are going to describe how such a remote log schema could work. Specifically, how it enhances p2p data sync and takes care of the <a href="https://vac.dev/p2p-data-sync-for-mobile" target="_blank" rel="noopener noreferrer">following requirements</a>:</p><blockquote><ol start="3"><li>MUST allow for mobile-friendly usage. By mobile-friendly we mean devices
that are resource restricted, mostly-offline and often changing network.</li></ol></blockquote><blockquote><ol start="4"><li>MAY use helper services in order to be more mobile-friendly. Examples of
helper services are decentralized file storage solutions such as IPFS and
Swarm. These help with availability and latency of data for mostly-offline
devices.</li></ol></blockquote><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="remote-log">Remote log<a href="#remote-log" class="hash-link" aria-label="Direct link to Remote log" title="Direct link to Remote log">​</a></h2><p>A remote log is a replication of a local log. This means a node can read data from a node that is offline.</p><p>The spec is in an early draft stage and can be found <a href="https://github.com/vacp2p/specs/pull/16" target="_blank" rel="noopener noreferrer">here</a>. A very basic <a href="https://en.wikipedia.org/wiki/Spike_(software_development)" target="_blank" rel="noopener noreferrer">spike</a> / proof-of-concept can be found <a href="https://github.com/vacp2p/research/tree/master/remote_log" target="_blank" rel="noopener noreferrer">here</a>.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="definitions">Definitions<a href="#definitions" class="hash-link" aria-label="Direct link to Definitions" title="Direct link to Definitions">​</a></h3><table><thead><tr><th>Term</th><th>Definition</th></tr></thead><tbody><tr><td>CAS</td><td>Content-addressed storage. Stores data that can be addressed by its hash.</td></tr><tr><td>NS</td><td>Name system. Associates mutable data to a name.</td></tr><tr><td>Remote log</td><td>Replication of a local log at a different location.</td></tr></tbody></table><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="roles">Roles<a href="#roles" class="hash-link" aria-label="Direct link to Roles" title="Direct link to Roles">​</a></h3><p>There are four fundamental roles:</p><ol><li>Alice</li><li>Bob</li><li>Name system (NS)</li><li>Content-addressed storage (CAS)</li></ol><p>The <em>remote log</em> is the data format of what is stored in the name system.</p><p>"Bob" can represent anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="flow">Flow<a href="#flow" class="hash-link" aria-label="Direct link to Flow" title="Direct link to Flow">​</a></h3><p></p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Figure 1: Remote log data synchronization." src="/assets/images/remote-log-5781aa7290ab5e4b5dbf652f5792ef2e.png" width="1200" height="320" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="data-format">Data format<a href="#data-format" class="hash-link" aria-label="Direct link to Data format" title="Direct link to Data format">​</a></h3><p>The remote log lets receiving nodes know what data they are missing. Depending on the specific requirements and capabilities of the nodes and name system, the information can be referred to differently. We distinguish between three rough modes:</p><ol><li>Fully replicated log</li><li>Normal sized page with CAS mapping</li><li>"Linked list" mode - minimally sized page with CAS mapping</li></ol><p>A remote log is simply a mapping from message identifiers to their corresponding address in a CAS:</p><table><thead><tr><th>Message Identifier (H1)</th><th>CAS Hash (H2)</th></tr></thead><tbody><tr><td>H1_3</td><td>H2_3</td></tr><tr><td>H1_2</td><td>H2_2</td></tr><tr><td>H1_1</td><td>H2_1</td></tr><tr><td></td><td></td></tr><tr><td><em>address to next page</em></td><td></td></tr></tbody></table><p>The numbers here corresponds to messages. Optionally, the content itself can be included, just like it normally would be sent over the wire. This bypasses the need for a dedicated CAS and additional round-trips, with a trade-off in bandwidth usage.</p><table><thead><tr><th>Message Identifier (H1)</th><th>Content</th></tr></thead><tbody><tr><td>H1_3</td><td>C3</td></tr><tr><td>H1_2</td><td>C2</td></tr><tr><td>H1_1</td><td>C1</td></tr><tr><td></td><td></td></tr><tr><td><em>address to next page</em></td><td></td></tr></tbody></table><p>Both patterns can be used in parallel, e,g. by storing the last <code>k</code> messages directly and use CAS pointers for the rest. Together with the <code>next_page</code> page semantics, this gives users flexibility in terms of bandwidth and latency/indirection, all the way from a simple linked list to a fully replicated log. The latter is useful for things like backups on durable storage.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="interaction-with-mvds">Interaction with MVDS<a href="#interaction-with-mvds" class="hash-link" aria-label="Direct link to Interaction with MVDS" title="Direct link to Interaction with MVDS">​</a></h3><p><a href="https://rfc.vac.dev/spec/2/#payloads" target="_blank" rel="noopener noreferrer">vac.mvds.Message</a> payloads are the only payloads that MUST be uploaded. Other messages types MAY be uploaded, depending on the implementation.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><p>The spec is still in an early draft stage, so it is expected to change. Same with the proof of concept. More work is needed on getting a fully featured proof of concept with specific CAS and NAS instances. E.g. Swarm and Swarm Feeds, or IPFS and IPNS, or something else.</p><p>For data sync in general:</p><ul><li>Make consistency guarantees more explicit for app developers with support for sequence numbers and DAGs, as well as the ability to send non-synced messages. E.g. ephemeral typing notifications, linear/sequential history and casual consistency/DAG history</li><li>Better semantics and scalability for multi-user sync contexts, e.g. CRDTs and joining multiple logs together</li><li>Better usability in terms of application layer usage (data sync clients) and supporting more transports</li></ul><hr><p>PS1. Thanks everyone who submitted great <a href="https://explorer.bounties.network/bounty/3389" target="_blank" rel="noopener noreferrer">logo proposals</a> for Vac!</p><p>PPS2. Next week on October 10th decanus and I will be presenting Vac at <a href="https://devcon.org/agenda" target="_blank" rel="noopener noreferrer">Devcon</a>, come say hi :)</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[Vac - A Rough Overview]]></title>
<link>https://vac.dev/rlog/vac-overview</link>
<guid>https://vac.dev/rlog/vac-overview</guid>
<pubDate>Fri, 02 Aug 2019 12:00:00 GMT</pubDate>
<description><![CDATA[Vac is a modular peer-to-peer messaging stack, with a focus on secure messaging. Overview of terms, stack and open problems.]]></description>
<content:encoded><![CDATA[<p>Vac is a modular peer-to-peer messaging stack, with a focus on secure messaging. Overview of terms, stack and open problems.</p><p>Vac is a <strong>modular peer-to-peer messaging stack, with a focus on secure messaging</strong>. What does that mean? Let's unpack it a bit.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="basic-terms">Basic terms<a href="#basic-terms" class="hash-link" aria-label="Direct link to Basic terms" title="Direct link to Basic terms">​</a></h2><p><em>messaging stack</em>. While the initial focus is on <a href="https://vac.dev/p2p-data-sync-for-mobile" target="_blank" rel="noopener noreferrer">data sync</a>, we are concerned with all layers in the stack. That means all the way from underlying transports, p2p overlays and routing, to initial trust establishment and semantics for things like group chat. The ultimate goal is to give application developers the tools they need to provide secure messaging for their users, so they can focus on their domain expertise.</p><p><em>modular</em>. Unlike many other secure messaging applications, our goal is not to have a tightly coupled set of protocols, nor is it to reinvent the wheel. Instead, we aim to provide options at each layer in the stack, and build on the shoulders of giants, putting a premimum on interoperability. It's similar in philosophy to projects such as <a href="https://libp2p.io/" target="_blank" rel="noopener noreferrer">libp2p</a> or <a href="https://www.parity.io/substrate/" target="_blank" rel="noopener noreferrer">Substrate</a> in that regard. Each choice comes with different trade-offs, and these look different for different applications.</p><p><em>peer-to-peer</em>. The protocols we work on are pure p2p, and aim to minimize centralization. This too is in opposition to many initiatives in the secure messaging space.</p><p><em>messaging</em>. By messaging we mean messaging in a generalized sense. This includes both human to human communication, as well machine to machine communication. By messaging we also mean something more fundamental than text messages, we also include things like transactions (state channels, etc) under this moniker.</p><p><em>secure messaging</em>. Outside of traditional notions of secure messaging, such as ensuring end to end encryption, forward secrecy, avoiding MITM-attacks, etc, we are also concerned with two other forms of secure messaging. We call these <em>private messaging</em> and <em>censorship-resistance</em>. Private messaging means viewing privacy as a security property, with all that entails. Censorship resistance ties into being p2p, but also in terms of allowing for transports and overlays that can't easily be censored by port blocking, traffic analysis, and similar.</p><p><em>Vāc</em>. Is a Vedic goddess of speech. It also hints at being a vaccine.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="protocol-stack">Protocol stack<a href="#protocol-stack" class="hash-link" aria-label="Direct link to Protocol stack" title="Direct link to Protocol stack">​</a></h2><p>What does this stack look like? We take inspiration from <a href="https://tools.ietf.org/html/rfc793" target="_blank" rel="noopener noreferrer">core</a> <a href="https://www.ietf.org/rfc/rfc1122.txt" target="_blank" rel="noopener noreferrer">internet architecture</a>, existing <a href="https://css.csail.mit.edu/6.858/2020/readings/secure-messaging.pdf" target="_blank" rel="noopener noreferrer">survey work</a> and other <a href="https://code.briarproject.org/briar/briar/wikis/A-Quick-Overview-of-the-Protocol-Stack" target="_blank" rel="noopener noreferrer">efforts</a> that have been done to decompose the problem into orthogonal pieces. Each layer provides their own set of properties and only interact with the layers it is adjacent to. Note that this is a rough sketch.</p><table><thead><tr><th>Layer / Protocol</th><th>Purpose</th><th>Examples</th></tr></thead><tbody><tr><td>Application layer</td><td>End user semantics</td><td>1:1 chat, group chat</td></tr><tr><td>Data Sync</td><td>Data consistency</td><td>MVDS, BSP</td></tr><tr><td>Secure Transport</td><td>Confidentiality, PFS, etc</td><td>Double Ratchet, MLS</td></tr><tr><td>Transport Privacy</td><td>Transport and metadata protection</td><td>Whisper, Tor, Mixnet</td></tr><tr><td>P2P Overlay</td><td>Overlay routing, NAT traversal</td><td>devp2p, libp2p</td></tr><tr><td></td><td></td><td></td></tr><tr><td>Trust Establishment</td><td>Establishing end-to-end trust</td><td>TOFU, web of trust</td></tr></tbody></table><p>As an example, end user semantics such as group chat or moderation capabilities can largely work regardless of specific choices further down the stack. Similarly, using a mesh network or Tor doesn't impact the use of Double Ratchet at the Secure Transport layer.</p><p>Data Sync plays a similar role to what TCP does at the transport layer in a traditional Internet architecture, and for some applications something more like UDP is likely to be desirable.</p><p>In terms of specific properties and trade-offs at each layer, we'll go deeper down into them as we study them. For now, this is best treated as a rough sketch or mental map.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="problems-and-rough-priorities">Problems and rough priorities<a href="#problems-and-rough-priorities" class="hash-link" aria-label="Direct link to Problems and rough priorities" title="Direct link to Problems and rough priorities">​</a></h2><p>With all the pieces involved, this is quite an undertaking. Luckily, a lot of pieces are already in place and can be either incorporated as-is or iterated on. In terms of medium and long term, here's a rough sketch of priorities and open problems.</p><ol><li><strong>Better data sync.</strong> While the current <a href="https://rfc.vac.dev/spec/2/" target="_blank" rel="noopener noreferrer">MVDS</a> works, it is lacking in a few areas:</li></ol><ul><li>Lack of remote log for mostly-offline offline devices</li><li>Better scalability for multi-user chat contexts</li><li>Better usability in terms of application layer usage and supporting more transports</li></ul><ol start="2"><li><strong>Better transport layer support.</strong> Currently MVDS runs primarily over Whisper, which has a few issues:</li></ol><ul><li>scalability, being able to run with many nodes</li><li>spam-resistance, proof of work is a poor mechanism for heterogeneous devices</li><li>no incentivized infrastructure, leading to centralized choke points</li></ul><p>In addition to these most immediate concerns, there are other open problems. Some of these are overlapping with the above.</p><ol start="3"><li><p><strong>Adaptive nodes.</strong> Better support for resource restricted devices and nodes of varying capabilities. Light connection strategy for resources and guarantees. Security games to outsource processing with guarantees.</p></li><li><p><strong>Incentivized and spam-resistant messaging.</strong> Reasons to run infrastructure and not relying on altruistic nodes. For spam resistance, in p2p multicast spam is a big attack vector due to amplification. There are a few interesting directions here, such as EigenTrust, proof of burn with micropayments, and leveraging zero-knowledge proofs.</p></li><li><p><strong>Strong privacy guarantees at transport privacy layer</strong>. More rigorous privacy guarantees and explicit trade-offs for metadata protection. Includes Mixnet.</p></li><li><p><strong>Censorship-resistant and robust P2P overlay</strong>. NAT traversal; running in the browser; mesh networks; pluggable transports for traffic obfuscation.</p></li><li><p><strong>Scalable and decentralized secure conversational security.</strong> Strong security guarantees such as forward secrecy, post compromise security, for large group chats. Includes projects such MLS and extending Double Ratchet.</p></li><li><p><strong>Better trust establishment and key handling</strong>. Avoiding MITM attacks while still enabling a good user experience. Protecting against ghost users in group chat and providing better ways to do key handling.</p></li></ol><p>There is also a set of more general problems, that touch multiple layers:</p><ol start="9"><li><p><strong>Ensuring modularity and interoperability</strong>. Providing interfaces that allow for existing and new protocols to be at each layer of the stack.</p></li><li><p><strong>Better specifications</strong>. Machine-readable and formally verified specifications. More rigorous analysis of exact guarantees and behaviors. Exposing work in such a way that it can be analyzed by academics.</p></li><li><p><strong>Better simulations</strong>. Providing infrastructure and tooling to be able to test protocols in adverse environments and at scale.</p></li><li><p><strong>Enabling excellent user experience</strong>. A big reason for the lack of widespread adoption of secure messaging is the fact that more centralized, insecure methods provide a better user experience. Given that incentives can align better for users interested in secure messaging, providing an even better user experience should be doable.</p></li></ol><hr><p>We got some work to do. Come help us if you want. See you in the next update!</p>]]></content:encoded>
</item>
<item>
<title><![CDATA[P2P Data Sync for Mobile]]></title>
<link>https://vac.dev/rlog/p2p-data-sync-for-mobile</link>
<guid>https://vac.dev/rlog/p2p-data-sync-for-mobile</guid>
<pubDate>Fri, 19 Jul 2019 12:00:00 GMT</pubDate>
<description><![CDATA[A research log. Reliable and decentralized, pick two.]]></description>
<content:encoded><![CDATA[<p>A research log. Reliable and decentralized, pick two.</p><p>Together with decanus, I've been working on the problem of data sync lately.</p><p>In building p2p messaging systems, one problem you quickly come across is the problem of reliably transmitting data. If there's no central server with high availability guarantees, you can't meaningfully guarantee that data has been transmitted. One way of solving this problem is through a synchronization protocol.</p><p>There are many synchronization protocols out there and I won't go into detail of how they differ with our approach here. Some common examples are Git and Bittorrent, but there are also projects like IPFS, Swarm, Dispersy, Matrix, Briar, SSB, etc.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="problem-motivation">Problem motivation<a href="#problem-motivation" class="hash-link" aria-label="Direct link to Problem motivation" title="Direct link to Problem motivation">​</a></h2><p>Why do we want to do p2p sync for mobilephones in the first place? There are three components to that question. One is on the value of decentralization and peer-to-peer, the second is on why we'd want to reliably sync data at all, and finally why mobilephones and other resource restricted devices.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="why-p2p">Why p2p?<a href="#why-p2p" class="hash-link" aria-label="Direct link to Why p2p?" title="Direct link to Why p2p?">​</a></h3><p>For decentralization and p2p, there are both technical and social/philosophical reasons. Technically, having a user-run network means it can scale with the number of users. Data locality is also improved if you query data that's close to you, similar to distributed CDNs. The throughput is also improved if there are more places to get data from.</p><p>Socially and philosophically, there are several ways to think about it. Open and decentralized networks also relate to the idea of open standards, i.e. compare the longevity of AOL with IRC or Bittorrent. One is run by a company and is shut down as soon as it stops being profitable, the others live on. Additionally increasingly control of data and infrastructure is becoming a liability. By having a network with no one in control, everyone is. It's ultimately a form of democratization, more similar to organic social structures pre Big Internet companies. This leads to properties such as censorship resistance and coercion resistance, where we limit the impact a 3rd party might have a voluntary interaction between individuals or a group of people. Examples of this are plentiful in the world of Facebook, Youtube, Twitter and WeChat.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="why-reliably-sync-data">Why reliably sync data?<a href="#why-reliably-sync-data" class="hash-link" aria-label="Direct link to Why reliably sync data?" title="Direct link to Why reliably sync data?">​</a></h3><p>At risk of stating the obvious, reliably syncing data is a requirement for many problem domains. You don't get this by default in a p2p world, as it is unreliable with nodes permissionslessly join and leave the network. In some cases you can get away with only ephemeral data, but usually you want some kind of guarantees. This is a must for reliable group chat experience, for example, where messages are expected to arrive in a timely fashion and in some reasonable order. The same is true for messages there represent financial transactions, and so on.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="why-mobilephones">Why mobilephones?<a href="#why-mobilephones" class="hash-link" aria-label="Direct link to Why mobilephones?" title="Direct link to Why mobilephones?">​</a></h3><p>Most devices people use daily are mobile phones. It's important to provide the same or at least similar guarantees to more traditional p2p nodes that might run on a desktop computer or computer. The alternative is to rely on gateways, which shares many of the drawbacks of centralized control and prone to censorship, control and surveillence.</p><p>More generally, resource restricted devices can differ in their capabilities. One example is smartphones, but others are: desktop, routers, Raspberry PIs, POS systems, and so on. The number and diversity of devices are exploding, and it's useful to be able to leverage this for various types of infrastructure. The alternative is to centralize on big cloud providers, which also lends itself to lack of democratization and censorship, etc.</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="minimal-requirements">Minimal Requirements<a href="#minimal-requirements" class="hash-link" aria-label="Direct link to Minimal Requirements" title="Direct link to Minimal Requirements">​</a></h2><p>For requirements or design goals for a solution, here's what we came up with.</p><ol><li><p>MUST sync data reliably between devices. By reliably we mean having the ability to deal with messages being out of order, dropped, duplicated, or delayed.</p></li><li><p>MUST NOT rely on any centralized services for reliability. By centralized services we mean any single point of failure that isn’t one of the endpoint devices.</p></li><li><p>MUST allow for mobile-friendly usage. By mobile-friendly we mean devices that are resource restricted, mostly-offline and often changing network.</p></li><li><p>MAY use helper services in order to be more mobile-friendly. Examples of helper services are decentralized file storage solutions such as IPFS and Swarm. These help with availability and latency of data for mostly-offline devices.</p></li><li><p>MUST have the ability to provide casual consistency. By casual consistency we mean the commonly accepted definition in distributed systems literature. This means messages that are casually related can achieve a partial ordering.</p></li><li><p>MUST support ephemeral messages that don’t need replication. That is, allow for messages that don’t need to be reliabily transmitted but still needs to be transmitted between devices.</p></li><li><p>MUST allow for privacy-preserving messages and extreme data loss. By privacy-preserving we mean things such as exploding messages (self-destructing messages). By extreme data loss we mean the ability for two trusted devices to recover from a, deliberate or accidental, removal of data.</p></li><li><p>MUST be agnostic to whatever transport it is running on. It should not rely on specific semantics of the transport it is running on, nor be tightly coupled with it. This means a transport can be swapped out without loss of reliability between devices.</p></li></ol><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="mvds---a-minimium-viable-version">MVDS - a minimium viable version<a href="#mvds---a-minimium-viable-version" class="hash-link" aria-label="Direct link to MVDS - a minimium viable version" title="Direct link to MVDS - a minimium viable version">​</a></h2><p>The first minimum viable version is in an alpha stage, and it has a <a href="https://rfc.vac.dev/spec/2" target="_blank" rel="noopener noreferrer">specification</a>, <a href="https://github.com/vacp2p/mvds" target="_blank" rel="noopener noreferrer">implementation</a> and we have deployed it in a <a href="https://github.com/status-im/status-console-client" target="_blank" rel="noopener noreferrer">console client</a> for end to end functionality. It's heavily inspired by <a href="https://code.briarproject.org/briar/briar-spec/blob/master/protocols/BSP.md" target="_blank" rel="noopener noreferrer">Bramble Sync Protocol</a>.</p><p>The spec is fairly minimal. You have nodes that exchange records over some secure transport. These records are of different types, such as <code>OFFER</code>, <code>MESSAGE</code>, <code>REQUEST</code>, and <code>ACK</code>. A peer keep tracks of the state of message for each node it is interacting with. There's also logic for message retransmission with exponential delay. The positive ACK and retransmission model is quite similar to how TCP is designed.</p><p>There are two different modes of syncing, interactive and batch mode. See sequence diagrams below.</p><p>Interactive mode:
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Interactive mode" src="/assets/images/mvds_interactive-b04b5377d67c337013e72abbbd40ec69.png" width="656" height="436" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Batch mode:
</p><div class="wrapper_SWrM active_qZD5"><img loading="lazy" alt="Batch mode" src="/assets/images/mvds_batch-8bb753ee771b1f96610ba432fa7fcec3.png" width="656" height="278" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917ZM2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917V8.75Zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167v2.333Zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167Z" fill="#fff"></path></svg></div></button></div><p></p><p>Which mode should you choose? It's a tradeoff of latency and bandwidth. If you want to minimize latency, batch mode is better. If you care about preserving bandwidth interactive mode is better. The choice is up to each node.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="basic-simulation">Basic simulation<a href="#basic-simulation" class="hash-link" aria-label="Direct link to Basic simulation" title="Direct link to Basic simulation">​</a></h3><p>Initial ad hoc bandwidth and latency testing shows some issues with a naive approach. Running with the <a href="https://github.com/vacp2p/mvds/" target="_blank" rel="noopener noreferrer">default simulation settings</a>:</p><ul><li>communicating nodes: 2</li><li>nodes using interactive mode: 2</li><li>interval between messages: 5s</li><li>time node is offine: 90%</li><li>nodes each node is sharing with: 2</li></ul><p>we notice a <a href="https://notes.status.im/7QYa4b6bTH2wMk3HfAaU0w#" target="_blank" rel="noopener noreferrer">huge overhead</a>. More specifically, we see a ~5 minute latency overhead and a bandwidth multiplier of x100-1000, i.e. 2-3 orders of magnitude just for receiving a message with interactive mode, without acks.</p><p>Now, that seems terrible. A moment of reflection will reveal why that is. If each node is offline uniformly 90% of the time, that means that each record will be lost 90% of the time. Since interactive mode requires offer, request, payload (and then ack), that's three links just for Bob to receive the actual message.</p><p>Each failed attempt implies another retransmission. That means we have <code>(1/0.1)^3 = 1000</code> expected overhead to receive a message in interactive mode. The latency follows naturally from that, with the retransmission logic.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="mostly-offline-devices">Mostly-offline devices<a href="#mostly-offline-devices" class="hash-link" aria-label="Direct link to Mostly-offline devices" title="Direct link to Mostly-offline devices">​</a></h3><p>The problem above hints at the requirements 3 and 4 above. While we did get reliable syncing (requirement 1), it came at a big cost.</p><p>There are a few ways of getting around this issue. One is having a <em>store and forward</em> model, where some intermediary node picks up (encrypted) messages and forwards them to the recipient. This is what we have in production right now at Status.</p><p>Another, arguably more pure and robust, way is having a <em>remote log</em>, where the actual data is spread over some decentralized storage layer, and you have a mutable reference to find the latest messages, similar to DNS.</p><p>What they both have in common is that they act as a sort of highly-available cache to smooth over the non-overlapping connection windows between two endpoints. Neither of them are <em>required</em> to get reliable data transmission.</p><h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="basic-calculations-for-bandwidth-multiplier">Basic calculations for bandwidth multiplier<a href="#basic-calculations-for-bandwidth-multiplier" class="hash-link" aria-label="Direct link to Basic calculations for bandwidth multiplier" title="Direct link to Basic calculations for bandwidth multiplier">​</a></h3><p>While we do want better simulations, and this is a work in progress, we can also look at the above scenarios using some basic calculations. This allows us to build a better intuition and reason about the problem without having to write code. Let's start with some assumptions:</p><ul><li>two nodes exchanging a single message in batch mode</li><li>10% uniformly random uptime for each node</li><li>in HA cache case, 100% uptime of a piece of infrastructure C</li><li>retransmission every epoch (with constant or exponential backoff)</li><li>only looking at average (p50) case</li></ul><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="first-case-no-helper-services">First case, no helper services<a href="#first-case-no-helper-services" class="hash-link" aria-label="Direct link to First case, no helper services" title="Direct link to First case, no helper services">​</a></h4><p>A sends a message to B, and B acks it.</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">A message -&gt; B (10% chance of arrival)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">A &lt;- ack B (10% chance of arrival)</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>With a constant backoff, A will send messages at epoch <code>1, 2, 3, ...</code>. With exponential backoff and a multiplier of 2, this would be <code>1, 2, 4, 8, ...</code>. Let's assume constant backoff for now, as this is what will influence the success rate and thus the bandwidth multiplier.</p><p>There's a difference between <em>time to receive</em> and <em>time to stop sending</em>. Assuming each send attempt is independent, it takes on average 10 epochs for A's message to arrive with B. Furthermore:</p><ol><li>A will send messages until it receives an ACK.</li><li>B will send ACK if it receives a message.</li></ol><p>To get an average of one ack through, A needs to send 100 messages, and B send on average 10 acks. That's a multiplier of roughly a 100. That's roughly what we saw with the simulation above for receiving a message in interactive mode.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="second-case-high-availability-caching-layer">Second case, high-availability caching layer<a href="#second-case-high-availability-caching-layer" class="hash-link" aria-label="Direct link to Second case, high-availability caching layer" title="Direct link to Second case, high-availability caching layer">​</a></h4><p>Let's introduce a helper node or piece of infrastructure, C. Whenever A or B sends a message, it also sends it to C. Whenever A or B comes online, it queries for messages with C.</p><div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">A message -&gt; B (10% chance of arrival)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">A message -&gt; C (100% chance of arrival)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">B &lt;- req/res -&gt; C (100% chance of arrival)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">A &lt;- ack B (10% chance of arrival)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">C &lt;- ack B (100% chance of arrival)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">A &lt;- req/res -&gt; C (100% chance of arrival)</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg" role="img"><path d="M2.917 12.833c-.321 0-.596-.114-.825-.343a1.121 1.121 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166H2.917ZM5.25 10.5c-.32 0-.596-.114-.824-.343a1.121 1.121 0 0 1-.343-.824v-7c0-.32.115-.595.343-.824.229-.229.504-.343.824-.342h5.25c.32 0 .596.114.824.343.229.228.343.503.343.823v7c0 .321-.115.596-.343.825a1.121 1.121 0 0 1-.824.342H5.25Zm0-1.167h5.25v-7H5.25v7Z" fill="#fff"></path></svg></div></span></button></div></div></div><p>What's the probability that A's messages will arrive at B? Directly, it's still 10%. But we can assume it's 100% that C picks up the message. (Giving C a 90% chance success rate doesn't materially change the numbers).</p><p>B will pick up A's message from C after an average of 10 epochs. Then B will send ack to A, which will also be picked up by C 100% of the time. Once A comes online again, it'll query C and receive B's ack.</p><p>Assuming we use exponential backoff with a multiplier of 2, A will send a message directly to B at epoch <code>1, 2, 4, 8</code> (assuming it is online). At this point, epoch <code>10</code>, B will be online in the average case. These direct sends will likely fail, but B will pick the message up from C and send one ack, both directly to A and to be picked up by C. Once A comes online, it'll query C and receive the ack from B, which means it won't do any more retransmits.</p><p>How many messages have been sent? Not counting interactions with C, A sends 4 (at most) and B 1. Depending on if the interaction with C is direct or indirect (i.e. multicast), the factor for interaction with C will be ~2. This means the total bandwidth multiplier is likely to be <code>&lt;10</code>, which is a lot more acceptable.</p><p>Since the syncing semantics are end-to-end, this is without relying on the reliablity of C.</p><h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="caveat">Caveat<a href="#caveat" class="hash-link" aria-label="Direct link to Caveat" title="Direct link to Caveat">​</a></h4><p>Note that both of these are probabilistic argument. They are also based on heuristics. More formal analysis would be desirable, as well as better simulations to experimentally verify them. In fact, the calculations could very well be wrong!</p><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2><p>There are many enhancements that can be made and are desirable. Let's outline a few.</p><ol><li><p>Data sync clients. Examples of actual usage of data sync, with more interesting domain semantics. This also includes usage of sequence numbers and DAGs to know what content is missing and ought to be synced.</p></li><li><p>Remote log. As alluded to above, this is necessary. It needs a more clear specification and solid proof of concepts.</p></li><li><p>More efficient ways of syncing with large number of nodes. When the number of nodes goes up, the algorithmic complexity doesn't look great. This also touches on things such as ambient content discovery.</p></li><li><p>More robust simulations and real-world deployments. Exisiting simulation is ad hoc, and there are many improvements that can be made to gain more confidence and identify issues. Additionally, better formal analysis.</p></li><li><p>Example usage over multiple transports. Including things like sneakernet and meshnets. The described protocol is designed to work over unstructured, structured and private p2p networks. In some cases it can leverage differences in topology, such as multicast, or direct connections.</p></li></ol>]]></content:encoded>
</item>
</channel>
</rss>
Reference in New Issue View Git Blame Copy Permalink